← All projects
AI / Data Coursework Lab /

Data engineering final

Olympic Data Pipeline

An end-to-end Olympic athlete pipeline combining real-time event processing with a reusable batch lane.

GitHub

About the project

An end-to-end Olympic athlete pipeline combining real-time event processing with a reusable batch lane.

Evidence & results

  • Joined MySQL athlete metadata with Kafka competition events in Spark.
  • Implemented streaming aggregation and batch processing through reusable Python modules.
  • Published processed statistics to Kafka and MySQL, with a monitor, logs, and screenshots as operational evidence.