Allow cluster operators to use custom container registry

Created by: matofeder

As a cluster operator, I want to use a custom container registry that mirrors public registries e.g. to have a copy of container images under my own control or to avoid issues with pull rate limit from docker.io registry.

Current state

SCS K8s bootstrap scripts install various k8s components and software, a few of them use docker.io public registry as a source. The docker.io source is defined in their manifest files. Some of them are located in the k8s-cluster-api-provider repository, and some of them are downloaded on the fly from other git repositories.

Desired state

As was discussed in one SCS container team meeting we agreed that it is a good idea to use registry.scs.community as a mirror for docker.io and maybe also other container sources to avoid pull rate limit issues.

Open questions

1. Should we hard-code the SCS registry registry.scs.community instead of docker.io in related manifest files which are located in the k8s-cluster-api-provider repository and also patch downloaded manifest files before they are applied?

2. Should we introduce variables that allow cluster operators to override docker.io occurrences from 1.? It means: keeping docker.io as a default and allowing cluster operators to change this default.

3. Should we do the same (1. or 2.) for other registries like k8s.gcr.io (i.e. not only docker.io)?

4. User workload-related question: Should we set CRI (containerd) default registry to registry.scs.community? or make this value also configurable? It means: if the user does not specify the container registry in his/her manifest file (e.g. image: nginx), the registry.scs.community will be used as a default registry instead of docker.io

Your feedback is welcome, @jschoone, @garloff, @chess-knight