All projects
Case study · Data Engineering · ETL
RepoInventory ETL & Alerting Pipeline
Scheduled ingestion, validation, and threshold alerting
Python
FastAPI
PostgreSQL
SQLAlchemy
Celery
Redis
Docker
How it works
extract → transform → validate → load → alert
live01
Extract
inventory feeds
pull raw
02
Transform
Pandas · normalize
clean
03
Validate
integrity checks
guard
04
Load
Supabase / PostgreSQL · 540K+
bulk
05
Alert
Celery + Redis scheduled
watch
Async workers (Celery + Redis) stage each record through every gate; only validated rows reach Postgres, and anomalies fire real-time threshold alerts.
01The problem
Inventory data arrives from multiple sources in inconsistent shapes. Loading it reliably, validating every record, and surfacing problems before they become stockouts is the whole job.
02Approach
- 01Split the pipeline into maintainable stages: ingestion, transformation, validation, persistence, and alerting.
- 02Used SQLAlchemy and structured database access for repeatable loads.
- 03Scheduled processing asynchronously with Celery and Redis so runs were repeatable rather than ad hoc.
- 04Generated alerts when inventory crossed configured thresholds.
03Outcomes
- A maintainable pipeline where each stage does one job.
- Repeatable asynchronous runs instead of one-off scripts.
- Threshold alerts surface operational conditions early.