Hacking OKD: Changing an OpenShift Component and Building Your Own Release
OKD is built from hundreds of repositories under github.com/openshift. If you want to fix a bug, try out a feature, or understand how OKD works, the fastest way to learn is to change the code and run it on a real cluster.
This post walks through the developer loop for OKD on CentOS Stream CoreOS (SCOS):
- Build a component image for OKD (with the
scosbuild tag and an SCOS base image) - Create a new release payload that swaps in your custom image
- Install (or upgrade to) that release and test your change
In this example we'll use the machine-config-operator (MCO) as the example, but the same steps apply to (almost) any payload component.
The first half of this post is the commands you need to run. The second half, Under the hood, explains why they work and how to adapt them to other components.
Prerequisites
podman(ordocker) and the latestoc (openshift-client)- A container registry account that your cluster can pull from (in this blog we'll assume you are using quay.io, but any public registry will work)
- An environment to install OKD into (a cloud account, vSphere, bare metal, etc.)
Step 0: Find the code that is actually in your release
Pick a release from the OKD release controller i.e.
export FROM_RELEASE=quay.io/okd/scos-release:5.0.0-okd-scos.1
Find the image and the exact commit that went into that release for your component:
oc adm release info --image-for=machine-config-operator ${FROM_RELEASE}
oc adm release info --commit-urls ${FROM_RELEASE} | grep machine-config-operator
Clone the repository and check out the branch that matches your release (for example release-5.0 is for OKD 5.0, main is always for the
next release in development):
git clone https://github.com/openshift/machine-config-operator
cd machine-config-operator
git checkout release-5.0
Now make your change.
Step 1: Build the component for OKD (SCOS)
These steps work for most of the Go operators in the payload. A few components can't be rebuilt outside of OpenShift CI today, and some need extra changes. Check Components that need extra work before you start.
The MCO's Dockerfile.rhel7 has two FROM lines, and both point at images on the OpenShift CI registry:
- Final stage (this is what makes it OKD): replace
registry.ci.openshift.org/ocp/5.0:base-rhel9with the SCOS base image,registry.ci.openshift.org/origin/scos-5.0:base-stream9. This image is publicly pullable. - Builder stage (only because the CI image is private): OKD CI keeps the same builder as OpenShift,
registry.ci.openshift.org/ocp/builder:rhel-9-golang-1.26-openshift-5.0, but pulling it requires CI credentials. Replace it with the publicgolangimage at the Go version ingo.mod, and install the libraries the build links against.
sed \
-e 's|^FROM registry.ci.openshift.org/ocp/builder:.* AS rhel9-builder|FROM docker.io/library/golang:1.26.10 AS rhel9-builder\nRUN apt-get update \&\& apt-get install -y libgpgme-dev|' \
-e 's|^FROM registry.ci.openshift.org/ocp/5.0:base-rhel9|FROM registry.ci.openshift.org/origin/scos-5.0:base-stream9|' \
Dockerfile.rhel7 > Dockerfile.okd
This uses GNU sed. On macOS, install gnu-sed and run gsed, or make the same edits by hand.
The top of each stage in Dockerfile.okd should now look like this:
FROM docker.io/library/golang:1.26.10 AS rhel9-builder
RUN apt-get update && apt-get install -y libgpgme-dev
...
FROM registry.ci.openshift.org/origin/scos-5.0:base-stream9
Then build with TAGS=scos and push:
podman build \
--build-arg TAGS=scos \
-f Dockerfile.okd \
-t quay.io/<your-user>/machine-config-operator:okd-5.0-mychange .
podman push quay.io/<your-user>/machine-config-operator:okd-5.0-mychange
Check that the image has the release-operator label, so oc adm release new will pick up its manifests:
oc image info quay.io/<your-user>/machine-config-operator:okd-5.0-mychange -o json | jq '.config.config.Labels'
# look for "io.openshift.release.operator": "true"
The Dockerfile name, build args and base image are specific to the MCO on release-5.0. To find the right values for
another component or branch, see Finding the build settings for any component.
Step 2: Create a new release with your operator
Take the existing release and override the MCO with your image:
export TO_RELEASE=quay.io/<your-user>/okd-release:5.0.0-okd-scos.1-mychange
oc adm release new \
--from-release ${FROM_RELEASE} \
--to-image ${TO_RELEASE} \
--name 5.0.0-okd-scos.1-mychange \
machine-config-operator=quay.io/<your-user>/machine-config-operator:okd-5.0-mychange
If you're changing the cluster-version-operator itself, append these flags to the oc adm release new command:
--keep-manifest-list \
--allow-missing-images
Verify that the new payload points at your image:
oc adm release info --image-for=machine-config-operator ${TO_RELEASE}
oc adm release info --pullspecs ${TO_RELEASE} | grep machine-config-operator
Step 3: Install the release and test
Option A: Install a fresh cluster
Extract an openshift-install binary from your custom release. It installs your payload by default:
oc adm release extract --command=openshift-install --to=./bin ${TO_RELEASE}
./bin/openshift-install version
Create an install-config.yaml as usual (see the installation docs). If all your
images are public, you can use a placeholder pull secret:
{ "auths": { "fake": { "auth": "aWQ6cGFzcwo=" } } }
Then install:
./bin/openshift-install create cluster --dir ./mycluster
Option B: Upgrade an existing cluster
If you already have a cluster on the release you built from, upgrade it to your custom payload by digest:
DIGEST=$(oc adm release info ${TO_RELEASE} -o jsonpath='{.digest}')
oc adm upgrade --allow-explicit-upgrade --force \
--to-image quay.io/<your-user>/okd-release@${DIGEST}
Only do this on clusters you're fine with breaking.
Test your change
Once the cluster is up, confirm that your image is running and the operator is healthy:
oc get clusterversion
oc get clusteroperators
oc get clusteroperator machine-config -o yaml
oc -n openshift-machine-config-operator get pods \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.containers[*].image}{"\n"}{end}'
Then exercise the behavior you changed. For broader coverage, run the conformance suites from openshift/origin against your cluster:
oc adm release extract --command=openshift-tests --to=./bin ${TO_RELEASE}
Faster iteration: skip the payload
Building a whole release for every one-line change is slow. Once you have a cluster, you can swap a single operator image in place:
-
Tell the CVO to stop managing the operator's Deployment by adding an
overridesentry toClusterVersionwithunmanaged: true, or scale the CVO down entirely for a throwaway cluster:oc scale --replicas=0 deployment/cluster-version-operator -n openshift-cluster-version -
Edit the operator's Deployment to point at your new image.
This is great for quick experiments, but remember to remove the overrides (or scale the CVO back up) afterwards. Overrides block minor and major version upgrades. Some operators also need extra steps beyond swapping the Deployment image. When you're happy with the change, build a proper release as described above to test it the way users would get it.
Under the hood
What is a release payload?
An OKD release image is a Cluster Version Operator (CVO) image with an extra layer containing release-manifests/:
an image-references file listing every component image by digest, plus the manifests for each operator. When you
install or upgrade a cluster, the CVO reads that payload and rolls out every component it references.
That means changing OKD comes down to two things: build a new image for the component you changed, then build a new payload that points at it. The Building the OKD payload post goes deeper into how CI does this at scale.
What makes an OKD build different
The same source code builds both OpenShift and OKD images. Two things are different for OKD:
- The
TAGS=scosbuild argument. Most payload Dockerfiles declareARG TAGS=""and pass it through togo build -tags. Withscosset, OKD-specific code is compiled in. These are files guarded by//go:build scos. For example, the MCO setsversion.SCOS = trueand rewritesrhel-coreosimage references tostream-coreos. The installer switches to SCOS boot images, and the console-operator switches to OKD branding. - An SCOS base image. OpenShift images are layered on a RHEL base. OKD images are layered on the
CentOS Stream-based
origin/scos-X.Ybase image.
Finding the build settings for any component
You don't need to guess the build settings. Every component that ships in OKD has an OKD variant CI configuration in
openshift/release, named
openshift-<repo>-<branch>__okd-scos.yaml. For the MCO on release-5.0, that is
openshift-machine-config-operator-release-5.0__okd-scos.yaml.
The parts you care about look like this:
base_images:
origin_scos-5.0_base-stream9:
name: scos-5.0
namespace: origin
tag: base-stream9
images:
items:
- build_args:
- name: TAGS
value: scos
dockerfile_path: Dockerfile.rhel7
from: origin_scos-5.0_base-stream9
to: machine-config-operator
promotion:
to:
- name: scos-5.0
namespace: origin
This tells you:
- Which Dockerfile CI builds (
Dockerfile.rhel7here; other repos useDockerfile.rhel,Dockerfile.ocp, and so on) - Which build args to pass (
TAGS=scos) - Which base image replaces the final
FROM(registry.ci.openshift.org/origin/scos-5.0:base-stream9) - The image name in the payload (
to: machine-config-operator), which is theNAMEyou override inoc adm release new
In CI, from: replaces the image in the Dockerfile's final FROM line. That is what the second sed expression in
Step 1 does locally: the MCO's Dockerfile.rhel7 ends with FROM registry.ci.openshift.org/ocp/5.0:base-rhel9, and we
swap it for the SCOS base. Despite the base-stream9 tag, the OKD 5.0 base image is CentOS Stream 10.
The main branch configs target the next release (for example scos-5.1:base-stream10), so always read the config
for the branch you're building.
Build notes
- Builder images. OKD only swaps the runtime base image. CI builds every component with the same private
registry.ci.openshift.org/ocp/builderimages as OpenShift, and there is no public equivalent. If you have access to the CI cluster, you can log in withoc registry loginand keep the original builder. Otherwise, use the officialgolangimage at the Go version fromgo.mod. It ships Go,makeandgit, but not the C libraries the private builder includes. Add whatever your component links against with cgo. For the MCO that'slibgpgme-dev, which the vendoredproglottis/gpgmepackage needs. If the build fails withPackage <name> was not found in the pkg-config search path, install the matchinglib<name>-devpackage in the builder stage. - Repo helpers. Some repositories ship a helper. For example, the MCO's
TAGS=scos make imagepasses the build tag topodman build. It does not swap the base image, so you still need theFROMchange.
Components that need extra work
Most payload components follow the same pattern as the MCO, so Step 1 works for them unchanged. We checked every image
in 5.0.0-okd-scos.1 against its release-5.0 Dockerfile, building without any Red Hat credentials. The images below
are the exceptions.
These components can't be rebuilt outside of OpenShift CI today. Most of them install RPMs that are only available from
an internal OpenShift repository, so dnf fails with No match for argument: <package>. The rest build with a Node.js
image that isn't publicly available.
| Image | Why it can't be rebuilt |
|---|---|
agent-installer-api-server | Installs the openshift-clients RPM |
agent-installer-csr-approver | Installs the openshift-clients RPM |
agent-installer-node-agent | Installs the dhclient RPM, which CentOS Stream 10 no longer ships |
agent-installer-ui | Builds with the private Node.js builder |
baremetal-installer | Installs the openshift-clients RPM |
console | Builds with the private Node.js builder |
haproxy-router | Installs the haproxy32 RPM |
haproxy-router-haproxy28 | Installs the haproxy28 RPM |
haproxy-router-haproxy32 | Installs the haproxy32 RPM |
insights-runtime-extractor | Installs the cri-tools RPM |
metallb-frr | Installs the frr10 RPM |
monitoring-plugin | Builds with a Node.js image from registry.redhat.io, which requires a login |
networking-console-plugin | Builds with the private Node.js builder |
ovn-kubernetes | Installs the openshift-clients RPM |
These components build, but need more changes than the two FROM lines in Step 1. "The release's <name> image" means
the pullspec from oc adm release info --image-for=<name> ${FROM_RELEASE}.
| Image | What to change |
|---|---|
agent-installer-utils | Its builder stage runs dnf, which the golang image doesn't have. Use a CentOS Stream 10 builder with Go installed |
azure-file-csi-driver | Replace ocp/5.0:azure-storage-azcopy-base with origin/scos-5.0:azure-storage-azcopy-base |
cli | Add libkrb5-dev to the builder stage |
cli-artifacts | Replace ocp/5.0:cli with the release's cli image, and add libkrb5-dev to the builder stage |
cluster-node-tuning-operator | Replace the final quay.io/centos/centos:stream9 with the SCOS base image |
deployer | Replace ocp/5.0:cli with the release's cli image, and add libkrb5-dev to the builder stage |
installer | Replace ocp/5.0:installer-etcd-artifacts and ocp/5.0:installer-kube-apiserver-artifacts with the origin/scos-5.0 images of the same name |
installer-artifacts | Replace ocp/5.0:installer-etcd-artifacts and ocp/5.0:installer-kube-apiserver-artifacts with the origin/scos-5.0 images of the same name |
ironic | Replace the final quay.io/centos/centos:stream9 with the SCOS base image |
ironic-agent | Replace the final quay.io/centos/centos:stream9 with the SCOS base image |
machine-os-images | Replace ocp/5.0:cli and ocp/5.0:installer with the release's cli image and installer images |
must-gather | Replace ocp/5.0:cli with the release's cli image |
network-tools | Replace ocp/5.0:ovn-kubernetes with origin/scos-5.0:ovn-kubernetes, and ocp/5.0:tools with the release's tools image |
operator-marketplace | Its Dockerfile.okd ends with FROM origin/4.23:base, which isn't published. Replace it with the SCOS base image |
ovn-kubernetes-microshift | Replace ocp/5.0:ovn-kubernetes-base with origin/scos-5.0:ovn-kubernetes-base |
pod | Its builder stage runs dnf, which the golang image doesn't have. Use a CentOS Stream 10 builder with Go installed |
tests | Replace ocp/5.0:tools with the release's tools image |
tools | Add libkrb5-dev to the builder stage |
The stream-coreos and stream-coreos-extensions images are the node operating system, and aren't built this way.
Release notes
- Overriding several components. You can pass several
NAME=PULLSPECarguments tooc adm release new. This is handy when a change spans repositories, such as an API change and the operator that consumes it. Every other component still points at the original OKD images, so you only build what you changed. - Multi-arch releases. If the source release is a multi-arch manifest list, add
--keep-manifest-list, or point--from-releaseat the single-architecture digest for the platform you're testing on.
Install and upgrade notes
- The extracted installer.
oc adm release extract --command=openshift-installembeds the release pullspec, pinned by digest, into the binary. That's why it installs your payload without any extra flags. - Pull secrets. OKD doesn't need a Red Hat pull secret, which is why the placeholder works. If your registry repositories are private, put real credentials for them in the pull secret instead.
OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE. You can point an existing installer at a different payload with this environment variable. The installer and payload versions should match, though. Extracting the installer from your release is the safer option.- Upgrade flags. Your release isn't signed and isn't in the update graph, so
oc adm upgradeneeds--allow-explicit-upgradeand--force. With--force, a by-tag pullspec is accepted, but a digest is safer.
Getting your change upstream
When your change works, open a pull request against the component repository. Repositories that ship in OKD run the
OKD (__okd-scos) variant jobs on PRs too, so your change is built with TAGS=scos on the SCOS base and tested in CI.
Once it merges, the post-submit job promotes the image into the origin/scos-X.Y image stream, and it shows up in the
next OKD build on the release controller.
If something is OKD-specific, such as a CentOS package name, a base image, or a build argument, the fix may belong in the OKD variant config in openshift/release rather than the component itself.
If you get stuck, come find us in the
#okd-dev channel in the Kubernetes Slack. Happy hacking!
