Reconsider whether fixed pagination sizes should be hard-coded or configured
The discussion from !394 (merged) should be addressed, i.e. we currently represent fixed pagination sizes that are determined by the source itself by configuring the source.batch_size
attribute via the harvester.toml
configuration file.
This means we can possibly just change the configuration to adjust the scraper when the pagination size changes at the source. On the other hand, this also means we can break the scraper by configuring a wrong pagination size.
Since multiple people are now taking ownership of this code base, we should reconsider the prvious decision and if we change it, do so for all affected (internal and external) harvesters.