State of affairs in OKD CI/CD

OKD is a community distribution of Kubernetes which is built from Red Hat OpenShift components on top of Fedora CoreOS (FCOS) and recently also CentOS Stream CoreOS (SCOS). The OKD variant based on Fedora CoreOS is called OKD or OKD/FCOS. The SCOS variant is often referred to as OKD/SCOS.

The previous blog posts introduced OKD Streams and its new Tekton pipelines for building OKD/FCOS and OKD/SCOS releases. This blog post gives an overview of the current build and release processes for FCOS, SCOS and OKD. It outlines OKD's dependency on OpenShift, an remnant from the past when its Origin predecessor was a downstream rebuild of OpenShift 3, and concludes with an outlook on how OKD Streams will help users, developers and partners to experiment with future OpenShift.

Fedora CoreOS and CentOS Stream CoreOS

Fedora CoreOS is built with a Jenkins pipeline running in Fedora's infrastructure and is being maintained by the Fedora CoreOS team.

CentOS Stream CoreOS is built with a Tekton pipeline running in a OpenShift cluster on MOC's infrastructure and pushed to quay.io/okd/centos-stream-coreos-9. The SCOS build pipeline is owned and maintained by the OpenShift OKD Streams team and SCOS builds are being imported from quay.io into OpenShift CI as ImageStreams.

OpenShift payload components

At the time of writing, most payload components for OKD/FCOS and OKD/SCOS get mirrored from OCP CI releases. OpenShift CI (Prow and ci-operator) periodically builds OCP images, e.g. for OVN-Kubernetes. OpenShift's release-controller detects changes to image streams, caused by recently built images, then builds and tests a OCP release image. When such an release image passes all non-optional tests (also see release gating docs), the release image and other payload components are mirrored to origin namespaces on quay.io (release gating is subject to change). For example, at most every 3 hours a OCP 4.14 release image will be deployed (and upgraded) on AWS and GCP and afterwards tested with OpenShift's conformance test suite. When it passes the non-optional tests the release image and its dependencies will be mirrored to quay.io/origin (except for rhel-coreos*, *-installer and some other images). These OCP CI releases are listed with a ci tag at amd64.ocp.releases.ci.openshift.org. Builds and promotions of nightly and stable OCP releases are handled differently (i.e. outside of Prow) by the Automated Release Tooling (ART) team.

OKD payload components

A few payload components are built specifically for OKD though, for example OKD/FCOS' okd-machine-os. Unlike RHCOS and SCOS, okd-machine-os, the operating system running on OKD/FCOS nodes, is layered on top of FCOS (also see CoreOS Layering, OpenShift Layered CoreOS).

Note, some payload components have OKD specific configuration in OpenShift CI although the resulting images are not incorporated into OKD release images. For example, OVN-Kubernetes images are built and tested in OpenShift CI to ensure OVN changes do not break OKD.

OKD releases

When OpenShift's release-controller detects changes to OKD related image streams, either due to updates of FCOS/SCOS, an OKD payload component or due to OCP payload components being mirrored after an OCP CI release promotion, it builds and tests a new OKD release image. When such an OKD release image passes all non-optional tests, the image is tagged as registry.ci.openshift.org/origin/release:4.14 etc. This CI release process is similar for OKD/FCOS and OKD/SCOS, e.g. compare these examples for OKD/FCOS 4.14 and with OKD/SCOS 4.14. OKD/FCOS's and OKD/SCOS's CI releases are listed at amd64.origin.releases.ci.openshift.org.

Promotions for OKD/FCOS to quay.io/openshift/okd (published at github.com/okd-project/okd) and for OKD/SCOS to quay.io/okd/scos-release (published at github.com/okd-project/okd-scos) are done roughly every 2 to 3 weeks. For OKD/SCOS, OKD's release pipeline is triggered manually once a sprint to promote CI releases to 4-scos-{next,stable}.

OKD Streams and customizable Tekton pipelines

However, the OKD project is currently shifting its focus from doing downstream rebuilds of OCP to OKD Streams. As part of this strategic repositioning, OKD offers Argo CD workflows and Tekton pipelines to build CentOS Stream CoreOS (SCOS) (with okd-coreos-pipeline), to build OKD/SCOS (with okd-payload-pipeline) and to build operators (with okd-operator-pipeline). The OKD Streams pipelines have been created to improve the RHEL9 readiness signal for Red Hat OpenShift. It allows developers to build and compose different tasks and pipelines to easily experiment with OpenShift and related technologies. Both okd-coreos-pipeline and okd-operator-pipeline are already used in OKD's CI/CD and in the future okd-payload-pipeline might supersede OCP CI for building OKD payload components and mirroring OCP payload components.