Summary of one scheduled run of a pipeline that gathers daily price history for 13 items at three time intervals.
Example output. The requests, websites and records are invented.
| Step | What happens |
|---|---|
| Fetch | Each source is requested with automatic retries if it fails. |
| Standardise | Timestamps are converted to one time zone and one format. |
| Validate | Impossible values and duplicate records are detected before anything is saved. |
| Calculate | Derived figures such as moving averages are added to every series. |
| Load | Rows are written to the database so that re-running never creates duplicates. |
| Report | A summary of what was loaded and what failed is produced for every run. |