Über Open CoDE Software Wiki Diskussionen GitLab

Skip to content

RFC: Bump concurrency and reduce max_age for suche-st

Adam Reichold requested to merge (removed):suche-st-config into main

Since this is a web offer which browsers often access using 8 or more connections concurrently and we are scraping it at night, I think bumping concurrency to 2 instead of 1 is reasonable and should ideally reduce the time to fully harvest this source from 7 hours to 4 hours.

Furthermore, since there few actual articles but the underyling web pages are likely updated quite regularly, the max_age is reduced to 1 month instead of 3.

Merge request reports