Healthcare data engineering
NY SPARCS
From hospital CSVs to a reporting app.
I used Python and DuckDB to normalize New York’s annual discharge files, build reporting tables, and feed a Streamlit app.
Pipeline map
- SourceAnnual discharge CSVs
- TransformPython + DuckDB SQL
- StoreParquet reporting marts
- ExploreStreamlit dashboard
How I built it
- Source
- Public, de-identified discharge CSVs from New York. Field names and coding systems change between annual releases.
- Data flow
- CSV → header normalization → DuckDB staging → SQL transformations → Parquet marts → Streamlit.
- Checks
- Test fixtures check header cleanup and reconcile the reporting tables with the source. Small groups are suppressed in published comparisons.
- Decisions
- I used DuckDB and Parquet to keep the build local and easy to inspect. The downloader records source details and limits retries.
- Limits
- The data is annual, so it won’t answer real-time questions. Charges aren’t revenue. Small demo tables are included; this portfolio doesn’t run the full pipeline.