python: read files in parallel and merge them

Given an iterable of csv files, pandas read and concatenate them

import pandas as pd
from multiprocessing import Pool

files = folderPath.glob('*.csv')
with Pool() as pool:
    df = pd.concat(pool.map(pd.read_csv, files))

To see progress, use tqdm to wrap the input files

files = list(folderPath.glob('*.csv'))
with Pool() as pool:
    df = pd.concat(pool.map(pd.read_csv, tqdm(files, total=len(files))))

You can use ThreadPoolExecutor to achieve the same thing.

from concurrent.futures import ThreadPoolExecutor

with ThreadPoolExecutor() as pool:
    df = pd.concat(pool.map(pd.read_csv, tqdm(files, total=len(files))))

• ThreadPoolExecutor: Utilizes threads for parallel execution. Threads share the same memory and are light weight compared to processes. ( file I/O, network operations, waiting time events )
• multiprocessing.Pool: Utilizes processes for parallel execution. Each process has its own memory, inter-process communication is more costly than thread communication. Useful for CPU-bound tasks that benefit from multiple CPU cores. ( CPU-bound tasks, heavy computation )

See all notes