Building a Custom CSV File Splitter: A Step-by-Step Guide Large CSV files can easily overwhelm spreadsheet software like Microsoft Excel or Google Sheets. When a file contains millions of rows, it slows down data pipelines and crashes local applications.
Building a custom CSV splitter allows you to break massive files into manageable chunks based on your exact needs. This guide will walk you through building a high-performance CSV splitter using Python. Why Build a Custom Splitter?
Standard file splitters chop documents by file size or strict line counts. A custom programmatic approach offers distinct advantages:
Preserves Headers: Automatically copies the header row to every new sub-file.
Memory Efficiency: Streams data line-by-line instead of loading gigabytes into RAM.
Smart Boundaries: Keeps related data rows together if needed. Prerequisites
To follow this tutorial, you only need Python installed on your system. We will use Python’s built-in modules, meaning no external dependencies or third-party libraries are required. Step 1: Set Up the Project Structure
Create a new directory for your project and open a file named csv_splitter.py. csv_splitter_project/ │ └── csv_splitter.py Use code with caution. Step 2: Write the Splitting Logic
Open csv_splitter.py and add the following code. This script reads a source CSV file and distributes its contents into smaller chunk files based on a maximum row threshold.
import os import csv def split_csv(source_filepath, output_dir, rows_per_file=50000): “”” Splits a large CSV file into multiple smaller files with headers preserved. “”” # Ensure the output directory exists if not os.path.exists(output_dir): os.makedirs(output_dir) # Extract base filename and extension base_name, ext = os.path.splitext(os.path.basename(source_filepath)) with open(source_filepath, mode=‘r’, newline=“, encoding=‘utf-8’) as source_file: reader = csv.reader(source_file) # Extract the header row try: header = next(reader) except StopIteration: print(“The source file is empty.”) return current_file_index = 1 current_row_count = 0 current_output_file = None writer = None for row in reader: # Open a new file if we hit the row limit or just started if current_output_file is None or current_row_count >= rows_per_file: if current_output_file: current_output_file.close() # Generate the new chunk filename output_filename = f”{base_name}part{current_file_index}{ext}” output_filepath = os.path.join(output_dir, output_filename) print(f”Creating: {output_filepath}“) current_output_file = open(output_filepath, mode=‘w’, newline=”, encoding=‘utf-8’) writer = csv.writer(current_output_file) # Write the header to the new file writer.writerow(header) current_file_index += 1 current_row_count = 0 # Write the data row writer.writerow(row) current_row_count += 1 # Clean up and close the final file if current_output_file: current_output_file.close() print(“Splitting process completed successfully.”) # Example usage: if name == “main”: # Replace with your actual file path large_file = “large_dataset.csv” output_folder = “split_files” # Split into chunks of 50,000 rows each split_csv(large_file, output_folder, rows_per_file=50000) Use code with caution. How It Works
Streamed Reading: The script uses csv.reader to fetch one row at a time. This keeps memory usage near zero, even for multi-gigabyte files.
Header Extraction: next(reader) grabs the very first line. The script saves this in memory and injects it at the top of every new chunk.
Dynamic File Creation: A counter tracks rows written to the active file. When it hits the rows_per_file limit, the script closes the current file and opens a sequentially numbered successor. Enhancing the Script
You can extend this basic script depending on your production requirements:
Command Line Interface (CLI): Wrap the script using Python’s built-in argparse module to pass file paths and row limits directly via the terminal.
Compression: Modify the file writer to compress output files instantly into .gz format using Python’s gzip library to save disk space. Conclusion
You now have a lightweight, robust tool to chop massive datasets into bite-sized pieces. By using Python’s standard library, your script remains highly portable and blazing fast.
Leave a Reply