Über Open CoDE Software Wiki Diskussionen GitLab

Skip to content

Extend service functionality to also efficiently collect logs in an SQLite database.

Doing this ourselves instead of leaving it to systemd-journal-remote would have the following benefits which we need weigh against the additional maintenance burden:

  • More CPU efficient, e.g. ingesting my notebook's logs has systemd-journal-upload at 100% CPU while this service ingests at 10% CPU. (I think this is mainly due to zero-copy parsing and because Zstd is more efficient than XZ used by systemd-journald.)
  • More space efficient, e.g. ingesting my notebook's 600 MB of logs yields a 70 MB database that is actually indexed on the host, container and unit fields. (This is achieved using deduplication and column-oriented field storage to improve compressibility.)
  • Fine-grained time-based log eviction is built in, i.e. we can retire our hacky rotate-remote-journals.py script and get configurable granularity instead of weekly file-based eviction.
  • Better robustness, i.e. ungracefully restarting an instance of systemd-journal-upload will no longer yield corrupted journal files but just loose a few events.
  • More efficient matching of expected messages, e.g. we control the format and I opted to omit process PID (which the patterns would otherwise need to include) and explicitly do not match on timestamps (which the regular expression engine would otherwise have to skip). We also do the matching inline to the persistence, i.e. there is no more serialization/deserialization round-trip.

Still left to be done before this can be merged:

  • Split the code into more maintainable modules.
  • Finish the documentation to include the new functionality.
  • Install SQLite development files (headers, libraries, etc.) in the builder image.
  • Actually test this using our multi-host setup and ingest the existing logs.
  • Setup the units on the status VM to use jemalloc instead of the default dlmalloc.
  • Adjust the expected log messages patterns to match the updated formatting provided here.
Edited by Adam Reichold

Merge request reports