Loading large mysql select statements into python

2018-11-08T19:02:42

I have two mysql databases called database A and database B. I want to use python to copy all the data from a table in database A to a table in database B. The datatable holds approximately 5M rows. I have written the following code:

connection = make_connection(source_db)
c = connection.cursor()
batch_size = 100000
while True:
    batch = c.fetchmany(batch_size)
    print('Extracted a batch.')
    if not batch:
        break
    insert_data(batch)

connection.close()

I ran this code and it takes minutes for fetchmany() to extract a single batch. I've tried to use fetchall() without the while True statement but this takes even longer.

Question: How do I successfully speed up this process/what is the best way to load large select statements into python?

Copyright License:
Author:「Mr. President」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.
Link to:https://stackoverflow.com/questions/53206405/loading-large-mysql-select-statements-into-python

About “Loading large mysql select statements into python” questions

I have two mysql databases called database A and database B. I want to use python to copy all the data from a table in database A to a table in database B. The datatable holds approximately 5M rows...
On a MyISAM engine MySQL server (5.1) We have some large query to make a report (1 million results). I notice that these select statements lock the tables. I don't find a real explanation for thi...
I'm trying to use MySQL connector as an alternative to pymysql as it supports multiple statements in one query for some updates I have to make (Here is my other question related to that) however it...
I'm trying to understand an issue I am having with a MySQL 5.5 server. This server hosts a number of databases. Each day at a certain time a process runs a series of inserts into TWO tables within...
I have a huge (1GB+) database dump that I want to load into new databases on other servers. I tried parsing it line by line and executing each into mysql, but it unfortunately doesn't split lines e...
I want to fetch classified details from a table. I have classified_ids in a csv format. I break them out from CSV and make independent queries for every classified_id. What I am doing right now is:
new to programming and trying to figure out an assignment. I have to do a nested loop with only select statements (no join statements) with Python. At a bit of a loss as to how to do so. The foll...
I have a series of MySql queries that need to be called with PDO. Many of them are using MySql variables. Surely, a solution would be to use PDO prepared statements but that's not always the case. ...
I'd like to start tinkering with large government data sets—in particular, I want to work with campaign contribution records and lobbying disclosure records. The Sunlight Foundation and the Center ...
I want to use a select statement with a variable within the where clause. Ive done resarch on this looking at How to use variables in SQL statement in Python? and Inserting Variables MySQL Using Py...

Copyright License:Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer.