This is a self-built demonstration project, not client work. It exists to show a practical approach to a common problem: exported spreadsheets often contain padded headers, blank rows, duplicate records, and columns nobody uses.
The problem
A business exports data from an internal system. The file opens with inconsistent headers, extra whitespace, blank rows, duplicate records, and at least one empty column. Before the data can be imported, reported on, or reviewed, someone has to clean it by hand.
What the tool does
- Detects comma, semicolon, tab, or pipe delimiters from the header row.
- Trims whitespace and normalises headers to lowercase snake_case.
- Gives duplicate headers unique names.
- Removes blank rows and exact duplicate records while preserving order.
- Drops columns with no data unless the user asks to keep them.
- Writes a cleaned CSV and an optional JSON summary report.
Example run
python clean_csv.py sample_input.csv -o sample_output.csv --report sample_report.json
4 input rows -> 2 output rows; 1 duplicates removed; 1 empty columns removed.The sample input has five columns. The cleaned output has four: name, email, phone, and notes. The unused column is removed, the duplicate row is dropped, and the notes field is normalised.
Implementation notes
The tool uses only the Python standard library: csv, argparse, json, and pathlib. That keeps the demonstration easy to run and audit. The test suite covers header normalisation, duplicate removal, empty-column handling, short-row padding, delimiter detection, and the command-line workflow.
Scope and limitations
- It does not validate business rules such as email formats, dates, currencies, or required fields.
- It removes exact duplicates after trimming whitespace; it does not fuzzy-match near-duplicates.
- It pads short rows and ignores extra cells beyond the header width. A production workflow should surface those cases to the user.
- It does not upload or transmit data anywhere; it runs locally on the files provided.
The full source, sample files, and tests live in the repository under demos/csv_cleanup/.
