Über Open CoDE Software Wiki Diskussionen GitLab

Skip to content

Keep batchsize out of configuration

Jakob Deller requested to merge keep-batchsize-out-of-configuration into main

As I stumbled upon this in the KliVO harvester, I would prefer to keep the batch_sizes out of harvester.toml if they are inherent to the design of the scraped page, so the changing of this parameter would lead to breaking the harvesting process.

In my opinion the parameters invites to being treated as a non-essential part of the configuration that might be changed to optimize harvesting speed.

Closes #242 (closed)

Edited by Adam Reichold

Merge request reports

Loading