Requirements management in RDO

Requirements management in RDO

Introduction

OpenStack services usually need some pieces of software which are not developed as part of the project. They are are general purpose libraries (typically python modules) or services used in some way to run or build OpenStack packages as databases, messaging brokers, etc…

OpenStack requirements project defines the policies and processes to manage requirements in upstream projects from a global perspective.

Managing OpenStack requirements in RDO

RDO provides all requirements for packaged services in RPM format from their own repos, so that no software should be installed from external repositories. This packages can be provided by:

CentOS base repositories (base, updates and extras). This is the preferred source of packages whenever possible.
Other CentOS SIG repositories (Virtualization, Storage, etc…). When a required package is being maintained by other CentOS SIG, it will be reused for RDO repos.
RDO CloudSIG repositories. When a package is not available from previous repos, it will be provided in RDO repositores. Note that it's required that these packages exist previously in Fedora so that they can be rebuilt with minimal changes (if any).

If you have questions or special requests, don't hesitate in contacting RDO using our mailing lists or #rdo channel in freenode.

Adding a new requirement to RDO

When a new requirement is needed for an OpenStack project included in RDO, package maintainers must follow this workflow:

RDO dependencies

Note that, typically new requirements are added only for the release of OpenStack under development, not for stable releases, although they may be accepted in previous releases if it's properly justified.

If the project follows global-requirements processes, make sure that the requirement has been added to global-requirements.txt and upper-constraints.txt files as described in the upstream documentation

Check if the new requirement is present in CentOS base channels. The easiest way to do this is using 'repoquery' command from a system running RPM based OS(CentOS/RHEL/Fedora):

 repoquery --repofrompath=tmp,http://mirror.centos.org/centos/7/os/x86_64/ \
 --repofrompath=tmp2,http://mirror.centos.org/centos/7/updates/x86_64/ \
 --repofrompath=tmp3,http://mirror.centos.org/centos/7/extras/x86_64/ \
 --disablerepo=* --enablerepo=tmp* "*<dependency>"

If it's present, the desired package is already available to RDO users.

If the package is not in CentOS base repos, you can check if it has been already built by the CloudSIG using rdopkg:
```
 rdopkg info <package name>
```
as, for example:
```
 $ rdopkg info python-eventlet
 1 packages found:

 name: python-eventlet
 project: python-eventlet
 conf: unmanaged-dependency
 patches: None
 distgit: https://github.com/rdo-common/python-eventlet.git
 buildsys-tags:
   cloud7-openstack-common-release: python-eventlet-0.17.4-4.el7
   cloud7-openstack-common-testing: python-eventlet-0.17.4-4.el7
   cloud7-openstack-ocata-release: python-eventlet-0.18.4-2.el7
   cloud7-openstack-ocata-testing: python-eventlet-0.18.4-2.el7
   cloud7-openstack-pike-release: python-eventlet-0.20.1-2.el7
   cloud7-openstack-pike-testing: python-eventlet-0.20.1-2.el7
   cloud7-openstack-queens-release: python-eventlet-0.20.1-5.el7
   cloud7-openstack-queens-testing: python-eventlet-0.20.1-5.el7
   cloud7-openstack-rocky-release: python-eventlet-0.20.1-5.el7
   cloud7-openstack-rocky-testing: python-eventlet-0.20.1-5.el7
   cloud7-openstack-stein-testing: python-eventlet-0.24.1-3.el7
 master-distgit: https://github.com/rdo-common/python-eventlet.git
 review-origin: null
 review-patches: null
 tags:
   dependency: null
 maintainers:
 - apevec@redhat.com
 - hguemar@fedoraproject.org
```
Note that the version of the package included in repositories is given by the CBS tags applied to each package (shown under buildsys-tags section for each package). Tags have a format cloud7-openstack-<release>-<phase> where:
- release: is a the OpenStack release name, as pike, queens, rocky or stein.
- phase:
  - candidate phase is assigned to packages to be rebuilt in CBS but not pushed to any RDO repository.
  - el7-build (only available for Rocky and newer releases) is assigned to packages that only required to build other packages but are not a runtime requirement for any other package.
  - testing phase means that the package is used in deployments using RDO Trunk repo and published in a testing repo, but not official CloudSIG repository.
  - release phase means that is published in the official CloudSIG repository. This phase is only available after a RDO version has been officially released not for the one currently under development.
For example, the package included in cloud7-openstack-rocky-release will published in the CloudSIG repo for rocky. The CBS tags flow will be:
- Runtime requirements: candidate -> testing -> release
- Build requirements: candidate -> el7-build
Note that, for the release currently under development (stein right now), testing and el7-build phase will be available. The package included in cloud7-openstack-stein-testing will be the one used to deploy from RDO Trunk Master repositories and it will be automatically pushed to cloud7-openstack-stein-release at RDO Stein is officially released and published.

If the package is found for the required CBS tag, it's already in RDO repositories and no more actions are needed to add it to the repos.
In case that the dependency is not in CentOS base or CloudSIG repo, you can check if it has been built by other SIGs in CBS web interface. You can use wildcards in the packages search expression. If you find the desired dependency, you can open a bug in Red Hat Bugzilla for RDO product requesting the inclussion of the package in RDO repos. RDO Core members will handle the request.
If the new package is not in CBS, you must check if it's packaged in Fedora using the Koji Web Interface. If the package exists, you need to open a review to rdoinfo project in RDO gerrit instance adding the new dependency to deps.yml and buildsys-tag in buildsys-tags/cloud7-openstack-stein-candidate.yml file as in this example:
```
 # in deps.yml
 - project: python-openshift
   name: python-openshift
   conf: fedora-dependency

 # in buildsys-tags/cloud7-openstack-stein-candidate.yml
 - project: python-openshift
   buildsys-tags:
     cloud7-openstack-stein-candidate: python-openshift-0.8.4-2.el7
```
Where:
- project and name must be the name of the main package (the same as in fedora).
- conf must be fedora-dependency.
- In buildsys-tags section a new line for the candidate tag in the OpenStack release in development (cloud7-openstack-stein-candidate) with the required NVR (name-version-release) required, which must be the same one found in Fedora replacing fcXX part in release by el7. For example, for python-openshift the build is python-openshift-0.8.4-2.fc30, so in buildsys-tags/cloud7-openstack-stein-candidate.yml, cloud7-openstack-stein-candidate must point to python-openshift-0.8.4-2.el7.
This review will rebuild the Fedora package in the CentOS Build System and make it available to be pushed to the next CBS phase.
When the packages doesn't exist even in Fedora you need to add the package following the New package process. Note that a Fedora packager needs to participate in this process. While RDO core members may maintain the new package for common requirements used by different projects, dependencies for specific project must be maintained in Fedora by the project team. Once the package is included in Fedora repos you can create a gerrit review as explained in step 5.
Once the package is rebuilt in CBS (review in step 5 is merged) you can push it to the next phase, this means testing (for runtime dependencies) or el7-build (for build requirements in rocky or newer releases). This is done by sending a new review to rdoinfo project adding a new line under buildsys-tags to buildsys-tags/cloud7-openstack-stein-candidate.yml file for the new tag as in this example:
```
 buildsys-tags:
  cloud7-openstack-stein-testing: python-openshift-0.8.4-2.el7
```
Once this review is merged, the tag will be applied to this build and the package will added to the testing repo for stein (note that some delay, up to 30 minutes is expected).
After the package is available in the repos, you can add it to the list of Requires or BuildRequires in your package spec file. Note that optional dependencies not used in default or common configurations should not be added as Requires but installed only when needed.

Updating a requirement in RDO CloudSIG repositories

There are some rules to follow when a requirement update is needed by a OpenStack project:

If the dependency is included in upstream requirements project, the required version must be equal to the version in upper-constraints file.
For packages installed from CentOS base repos, the package should be updated in CentOS/RHEL repos. This can be requested opening a bug in bugzilla for RHEL product. This bug will be evaluated following the RHEL process.
For packages installed from RDO CloudSIG repos, the package must be updated in Fedora first to the required version. If it has not been updated first you can contact Fedora package maintainer or search for open bug against component, example component:python-migrate, if bug not exist,open a bug for Fedora product.
Once the package is build in Fedora, you can update the requirement in RDO by sending a review to rdoinfo project in RDO gerrit instance like example.
Once the patch is merged, the package gets rebuild in CBS. Next step is to include the package in RDO Testing repo, this can be done by sending a patch to rdoinfo project like example
Once the package exist in testing repository it's ready to be used in jobs or update in rpm spec.

Contact us

If you have questions or special requests about requirements, don't hesitate to contact RDO community members using our mailing lists or #rdo channel in freenode.