Workflow: Promotion pipeline
Promotion pipelines are composed by a set of related CI jobs that are executed for each supported OpenStack release to test the content of a specific RDO repository. There are two sets of promotion pipelines for each OpenStack release:
- TripleO promotion pipelines - to validate RDO packages with TripleO and promote the content used by TripleO upstream check and gate jobs
- Puppet promotion pipelines - to validate RDO packages with OpenStack Puppet Modules and to promote the RDO packages used in upstream OpenStack Puppet modules check and gate jobs
Both these pipelines run in parallel and are composed of different jobs.
The promotion workflow can be described with following schema:
Define the repository to be tested. RDO Trunk repositories are identified by a hash based on the upstream commit of the last built package. The content of these repos doesn't change over time. When a promotion pipeline is launched, it grabs the latest consistent hash repo and sets it to be tested in the following phases.
Build TripleO images. TripleO is the recommended deployment tool for production usage in RDO and as such, is tested in RDO CI jobs. Before actually deploying OpenStack using TripleO, the required images are built.
- Deploy and test RDO. We run a set of jobs which deploy and test OpenStack
using different installers and scenarios to ensure they behave as expected. Currently,
there are following main promotion testing methods:
- TripleO deployments - this kind of deployment (with many kinds of config) is managed by TripleO promotion pipelines. Documentation of these pipelines and whole process description can be found here.
- OpenStack Puppet scenarios. Project puppet-openstack-integration (a.k.a. p-o-i) maintains a set of Puppet manifests to deploy different OpenStack services combinations and configurations (scenarios) in a single server using OpenStack Puppet Modules, and run tempest smoke tests for the deployed services. The tested services on each scenario can be found in the README for p-o-i. These scenarios, together with Packstack deployment ones, are executed in puppet promotion pipelines to test new packages build in RDO Repos. Part of scenarios are currently tested in RDO CI as Packstack deployment.
- Repository and images promotion. When all jobs in the previous phase succeed,
the tested repository is considered good and it is promoted so that users can use these
- The repo is published using RDO Trunk servers in https://trunk.rdoproject.org/centos9-<openstack_version>/current-tripleo/delorean.repo
- The images are copied to http://images.rdoproject.org/centos8/ and http://images.rdoproject.org/centos9/. to be used by TripleO users.
Tools used in RDO CI
- Jobs definitions are managed using Zuul, via a gerrit review workflow in review.rdoproject.org
- WeIRDO is the tool we use to run p-o-i and Packstack testing scenarios defined upstream inside RDO CI. It is composed of a set of ansible roles and playbooks that prepare the environment and deploy and test the installers using the testing scripts provided by the projects.
- TripleO Quickstart provides a set of scripts, ansible roles and pre-defined configurations to deploy an OpenStack cloud using TripleO in a simple and fully automated way.
- ARA is used to store and visualize the results of ansible playbook runs, making it easier to analize and troubleshoot them.
RDO runs the promotion pipelines in the CI infrastructure managed by Software Factory.
Handling issues in RDO CI
An important aspect of running RDO CI is properly managing the errors found in the jobs included in the promotion pipelines. The root cause of these issues sometimes is in the OpenStack upstream projects:
- Some problems are not caught in devstack-based jobs running in upstream gates.
- In some cases, new versions of OpenStack services require changes in the deployment tools (puppet modules, TripleO, etc…).
One of the contributions of RDO to upstream projects is to increase test coverage of the projects and help to identify the problems as soon as possible. When we find them, we report it upstream as Launchpad bugs and propose fixes when possible.
TripleO uses launchpad to manage bugs, so any issues should be reported there. Bugs unrelated to TripleO should be also reported in Bugzilla which specified component (if not known, should be set to "distribution").
Overall TripleO approach to CI is described in this document.
Status of promotion pipelines
If you are interested in the status of the promotion pipelines in RDO CI you can check:
Zuul pipelines status view can be used to see the result and status of each kind of puppet promotion pipeline and OpenStack release.
RDO Dashboard shows the overall status of RDO promotion pipelines.
- RDO FTBFS Dashboard shows status of currently failed package build attempts.
- TripleO dashboard shows many different information and stats, about promotion pipelines and others
- TripleO quickstart demostration by trown
- Weirdo: A talk about CI in OpenStack and RDO by dmsimard.
- ARA blog posts - from dmsimard blog
- Ci in RDO: What do we test? - presentation in RDO and Ceph Meetup BCN.
- RDO dashboards repo
← Trunk repos | → Adding packages | ↑ TOC