Loading large mysql select statements into python
I have two mysql databases called database A and database B . I want to use python to copy all the data from a table in database A to a table in database B . The datatable holds approximately 5M rows. I have written the following code:
connection = make_connection(source_db)
c = connection.cursor()
batch_size = 100000
while True:
batch = c.fetchmany(batch_size)
print('Extracted a batch.')
if not batch:
break
insert_data(batch)
connection.close()
I ran this code and it takes minutes for fetchmany() to extract a single batch. I've tried to use fetchall() without the while True statement but this takes even longer.
Question: How do I successfully speed up this process/what is the best way to load large select statements into python?
Copyright License: Author:「Mr. President」,Reproduced under the CC 4.0 BY-SA copyright license with link to original source & disclaimer. Link to:https://stackoverflow.com/questions/53206405/loading-large-mysql-select-statements-into-python
About “Loading large mysql select statements into python” questions
I have two mysql databases called database A and database B. I want to use python to copy all the data from a table in database A to a table in database B. The datatable holds approximately 5M rows...
|
On a MyISAM engine MySQL server (5.1)
We have some large query to make a report (1 million results). I notice that these select statements lock the tables.
I don't find a real explanation for thi...
|
I'm trying to use MySQL connector as an alternative to pymysql as it supports multiple statements in one query for some updates I have to make (Here is my other question related to that) however it...
|
I'm trying to understand an issue I am having with a MySQL 5.5 server.
This server hosts a number of databases. Each day at a certain time a process runs a series of inserts into TWO tables within...
|
I have a huge (1GB+) database dump that I want to load into new databases on other servers. I tried parsing it line by line and executing each into mysql, but it unfortunately doesn't split lines e...
|
I want to fetch classified details from a table. I have classified_ids in a csv format. I break them out from CSV and make independent queries for every classified_id.
What I am doing right now is:
|
new to programming and trying to figure out an assignment. I have to do a nested loop with only select statements (no join statements) with Python. At a bit of a loss as to how to do so.
The foll...
|
I have a series of MySql queries that need to be called with PDO. Many of them are using MySql variables. Surely, a solution would be to use PDO prepared statements but that's not always the case. ...
|
I'd like to start tinkering with large government data sets—in particular, I want to work with campaign contribution records and lobbying disclosure records. The Sunlight Foundation and the Center ...
|
I want to use a select statement with a variable within the where clause. Ive done resarch on this looking at How to use variables in SQL statement in Python? and Inserting Variables MySQL Using Py...
|
|
|