Give up on quota exhaustion
Created by: garloff
When running out of quota, we produce lots of alarms without additional insight. This can happen due to failed cleanups. (The script already tries hard, so the obvious way to address this already has been exercised.)
Suggestion:
- api_monitor.sh: detect out of quota
- on detection: remember and slow down (sleep 120)
- on repeated detection: slow down and stop iterations
- run_in_loop.sh: it does already try to do extra cleanups, which might help
- add detection for run time < 1hr and sleep for the rest, to restart out alarming no more than once per hour