Über Open CoDE Software Wiki Diskussionen GitLab

Skip to content

Always replay responses but evict them based on age

Adam Reichold requested to merge age-based-replay into main

This is an alternative to per-source or per-group schedules, i.e. !45 (closed), which solves the underyling issue of not harvesting at a higher frequency than necessary in a different and more fine-grained manner. We always "harvest" all sources, meaning parse and extract their responses and process the resulting dataset, but we only fetch those responses from the network which are actually older than the defined per-source maximum age.

This avoids any troubles with our internal data model having to be kept stable for a whole harvest cycle like a week or a month while significantly reducing the load we create at the data providers and locally on our infrastructure. As an example, our slowest harvester for the WISIA systems is configured for "weekly" updates in this MR by setting the maximum age to 6d and 22h (to account for the 2h jitter of our daily harvester schedule).

Closes #2 (closed)

Merge request reports