RDO Community News

See also blogs.rdoproject.org

RDO Ocata Release Behind The Scenes

I have been involved in 6 GA releases of RDO (From Juno to Ocata), and I wanted to share a glimpse of the preparation work. Since Juno, our process has tremendously evolved: we refocused RDO on EL7, joined the CentOS Cloud SIG, moved to Software Factory.

Our release process does not start when upstream announces GA or even a milestone, no it starts from the very beginning of upstream cycle.

Trunk chasing

We have been using DLRN to track upstream changes and build continuously OpenStack as a RPM distribution. Then our CI hosted on CentOS community CI runs multiple jobs on DLRN snapshots. We use the WeIRDO framework to run the same jobs as upstream CI on our packages. This allows us to detect early integration issues and get either our packaging or upstream projects fixed. This also includes installers such as OPM, TripleO or PackStack.

We also create Ocata tags in CentOS Community Build System (CBS) in order to build dependencies that are incompatible with currently supported releases.


We start branching RDO stable release around milestone 3, and have stable builds getting bootstrapped. This includes:

  • registering packages in CBS, I scripted this part for Ocata using rdoinfo database.
  • syncing requirements in packages.
  • branching distgit repositories.
  • building upstream releases in CBS, this part used to be semi-automated using rdopkg tool, Alfredo is consolidating that into a cron job creating reviews.
  • tag builds in -testing repositories, some automation is in preparation.

Trunk chasing continues, but we pay attention in keeping promotions happening more frequently to avoid a gap between tested upstream commits and releases.

GA publication

Since OpenStack does releases slightly ahead of time, we have most of GA releases built in CBS, but some of them comes late. We also trim final GA repositories, use repoclosure utility to check if there's no missing dependencies. Before mass-tagging builds in -release we launch stable promotion CI jobs and if they're green, we publish them.

At this stage, CentOS Core team, creates final GA repositories and sign packages.

For Newton, it took 10 hours between upstream GA announcement and repositories publication, 4 hours up to stable tagging. As for Ocata, all stable builds + CI jobs were finished within 2 hours.

Fun fact, Alan and I were doing the last bits of the Ocata release in the Atlanta PTG hallway and even get to see Doug Hellmann to send the GA announcement live (which started the chronometer for us). So we sprinted to have RDO Ocata GA ready as soon as possible (CI included!). We still have room for improvement but we were the first binary OpenStack distro available!


As of Ocata, there are still areas of improvement:

  • documenting releases process: many steps are still manual or require specific knowledge. During Newton/Ocata releases, we enabled Alfredo to do large chunks of the release preparation work. With post-mortems, this helped us clarifying the process, and prepare to allow more people helping in the release process.
  • dependencies CI: dependencies are a critical factor to release RDO in time. We need to test dependencies against RDO releases, RDO against CentOS updates and ensure that nothing is broken. That's one of our goals for Pike.
  • tag management: tags are used in CBS to determine where builds are to be published. Unlike Fedora, CBS has no automated pipeline to manage updates, so we have to manually tag builds. I'm currently working on having a gerrit-based process to manage tags. The tricky part is how to avoid inconsistencies in repositories (e.g avoid breaking dependencies, accidental untagging etc.)
  • dependencies updates: we want dependencies to remain compatible with Fedora packages, as Fedora is the foundation of next RHEL/CentOS, some of them are maintained in Fedora, others in RDO common, some with basic patches to fix EL7 build issues (not acceptable in Fedora), the rest being forks that we effectively maintain (e.g MariaDB). As a first step, we want to have the last set of packages to be maintained in our gerrit instances to allow maintainers doing builds without any releng support.
  • more contributions! Our effort into automating the release pipeline also serves the goal of empowering more contributors into the release work, so if you're interested, just come and tell us. ;-)

I hope this gave you an overview of how RDO is released and what are our next steps for Pike release.

View article »

RDO Ocata released

The RDO community is pleased to announce the general availability of the RDO build for OpenStack Ocata for RPM-based distributions, CentOS Linux 7 and Red Hat Enterprise Linux. RDO is suitable for building private, public, and hybrid clouds. Ocata is the 15th release from the OpenStack project, which is the work of more than 2500 contributors from around the world (source).

The RDO community project curates, packages, builds, tests and maintains a complete OpenStack component set for RHEL and CentOS Linux and is a member of the CentOS Cloud Infrastructure SIG. The Cloud Infrastructure SIG focuses on delivering a great user experience for CentOS Linux users looking to build and maintain their own on-premise, public or hybrid clouds.

All work on RDO, and on the downstream release, Red Hat OpenStack Platform, is 100% open source, with all code changes going upstream first.

Interesting things in the Ocata release include:

For cloud operators, RDO now provides packages for some new OpenStack Services:

  • Tacker: an ETSI MANO NFV Orchestrator and VNF Manager
  • Congress: an open policy framework for the cloud
  • Vitrage: the OpenStack RCA (Root Cause Analysis) Service
  • Kolla: The Kolla project provides tooling to build production-ready container images for deploying OpenStack clouds

Some other notable additions:

  • novajoin: a dynamic vendordata plugin for the OpenStack nova metadata service to manage automatic host instantiation in an IPA server
  • ironic-ui: a new Horizon plugin to view and manage baremetal servers
  • python-virtualbmc VirtualBMC is a proxy that translates IPMI commands to libvirt calls. This allows projects such as OpenStack Ironic to test IPMI drivers using VMs.
  • python-muranoclient: a client for the Application Catalog service.
  • python-monascaclient: a client for the Monasca monitoring-as-a-service solution.
  • Shaker: the distributed data-plane testing tool built for OpenStack
  • Multi-architecture support: aarch64 builds are now provided through an experimental repository - enable the RDO 'testing' repositories to get started

From a networking perspective, we have added some new Neutron plugins that can help Cloud users and operators to address new use cases and scenarios:

  • networking-bagpipe: a mechanism driver for Neutron ML2 plugin using BGP E-VPNs/IP VPNs as a backend
  • networking-bgpvpn: an API and framework to interconnect BGP/MPLS VPNs to Openstack Neutron networks
  • networking-fujitsu: FUJITSU ML2 plugins/drivers for OpenStack Neutron
  • networking-l2gw: APIs and implementations to support L2 Gateways in Neutron
  • networking-sfc: APIs and implementations to support Service Function Chaining in Neutron

From the Packstack side, we have several improvements:

  • We have added support to install Panko and Magnum
  • Puppet 4 is now supported, and we have updated our manifests to cover the latest changes in the supported projects

Getting Started

There are three ways to get started with RDO.

  • To spin up a proof of concept cloud, quickly, and on limited hardware, try the All-In-One Quickstart. You can run RDO on a single node to get a feel for how it works.
  • For a production deployment of RDO, use the TripleO Quickstart and you'll be running a production cloud in short order.
  • Finally, if you want to try out OpenStack, but don't have the time or hardware to run it yourself, visit TryStack, where you can use a free public OpenStack instance, running RDO packages, to experiment with the OpenStack management interface and API, launch instances, configure networks, and generally familiarize yourself with OpenStack. (TryStack is not, at this time, running Ocata, although it is running RDO.)

Getting Help

The RDO Project participates in a Q&A service at ask.openstack.org, for more developer-oriented content we recommend joining the rdo-list mailing list. Remember to post a brief introduction about yourself and your RDO story. You can also find extensive documentation on the RDO docs site.

The #rdo channel on Freenode IRC is also an excellent place to find help and give help.

We also welcome comments and requests on the CentOS mailing lists and the CentOS and TripleO IRC channels (#centos, #centos-devel, and #tripleo on irc.freenode.net), however we have a more focused audience in the RDO venues.

Getting Involved

To get involved in the OpenStack RPM packaging effort, see the RDO community pages and the CentOS Cloud SIG page. See also the RDO packaging documentation.

Join us in #rdo on the Freenode IRC network, and follow us at @RDOCommunity on Twitter. If you prefer Facebook, we're there too, and also Google+.

View article »

OpenStack Project Team Gathering, Atlanta, 2017

Over the last several years, OpenStack has conducted OpenStack Summit twice a year. One of these occurs in North America, and the other one alternates between Europe and Asia/Pacific.

This year, OpenStack Summit in North America is in Boston , and the other one will be in Sydney.

This year, though, the OpenStack Foundation is trying something a little different. Wheras in previous years, a portion of OpenStack Summit was the developers summit, where the next version of OpenStack was planned, this year that's been split off into its own separate event called the PTG - the Project Teams Gathering. That's going to be happening next week in Atlanta.

Throughout the week, I'm going to be interviewing engineers who work on OpenStack. Most of these will be people from Red Hat, but I will also be interviewing people from some other organizations, and posting their thoughts about the Ocata release - what they've been working on, and what they'll be working on in the upcoming Pike release, based on their conversations in the coming week at the PTG.

So, follow this channel over the next couple weeks as I start posting those interviews. It's going to take me a while to edit them after next week, of course. But you'll start seeing some of these appear in my YouTube channel over the coming few days.

Thanks, and I look forward to filling you in on what's happening in upstream OpenStack.

View article »

RDO blogs, week of Feb 13

Here's what RDO enthusiasts have been blogging about in the last few weeks. If you blog about RDO, please let me know (rbowen@redhat.com) so I can add you to my list.

TripleO: Debugging Overcloud Deployment Failure by bregman

You run ‘openstack overcloud deploy’ and after a couple of minutes you find out it failed and if that’s not enough, then you open the deployment log just to find a very (very!) long output that doesn’t give you an clue as to why the deployment failed. In the following sections we’ll see how can […]

Read more at http://tm3.org/dv

RDO @ DevConf by Rich Bowen

It's been a very busy few weeks in the RDO travel schedule, and we wanted to share some photos with you from RDO's booth at DevConf.cz.

Read more at http://tm3.org/dw

The surprisingly complicated world of disk image sizes by Daniel Berrange

When managing virtual machines one of the key tasks is to understand the utilization of resources being consumed, whether RAM, CPU, network or storage. This post will examine different aspects of managing storage when using file based disk images, as opposed to block storage. When provisioning a virtual machine the tenant user will have an idea of the amount of storage they wish the guest operating system to see for their virtual disks. This is the easy part. It is simply a matter of telling ‘qemu-img’ (or a similar tool) ’40GB’ and it will create a virtual disk image that is visible to the guest OS as a 40GB volume. The virtualization host administrator, however, doesn’t particularly care about what size the guest OS sees. They are instead interested in how much space is (or will be) consumed in the host filesystem storing the image. With this in mind, there are four key figures to consider when managing storage:

Read more at http://tm3.org/dx

Project Leader by rbowen

I was recently asked to write something about the project that I work on – RDO – and one of the questions that was asked was:

Read more at http://tm3.org/dy

os_type property for Windows images on KVM by Tim Bell

The OpenStack images have a long list of properties which can set to describe the image meta data. The full list is described in the documentation. This blog reviews some of these settings for Windows guests running on KVM, in particular for Windows 7 and Windows 2008R2.

Read more at http://tm3.org/dz

Commenting out XML snippets in libvirt guest config by stashing it as metadata by Daniel Berrange

Libvirt uses XML as the format for configuring objects it manages, including virtual machines. Sometimes when debugging / developing it is desirable to comment out sections of the virtual machine configuration to test some idea. For example, one might want to temporarily remove a secondary disk. It is not always desirable to just delete the configuration entirely, as it may need to be re-added immediately after. XML has support for comments which one might try to use to achieve this. Using comments in XML fed into libvirt, however, will result in an unwelcome suprise – the commented out text is thrown into /dev/null by libvirt.

Read more at http://tm3.org/d-

Videos from the CentOS Dojo, Brussels, 2017 by Rich Bowen

Last Friday in Brussels, CentOS enthusiasts gathered for the annual CentOS Dojo, right before FOSDEM.

Read more at http://tm3.org/dp

FOSDEM Day 0 - CentOS Dojo by Rich Bowen

FOSDEM starts tomorrow in Brussels, but there's always a number of events the day before.

Read more at http://tm3.org/dq

Gnocchi 3.1 unleashed by Julien Danjou

It's always difficult to know when to release, and we really wanted to do it earlier. But it seems that each week more awesome work was being done in Gnocchi, so we kept delaying it while having no pressure to push it out.

Read more at http://tm3.org/dr

Testing RDO with Tempest: new features in Ocata by ltoscano

The release of Ocata, with its shorter release cycle, is close and it is time to start a broader testing (even if one could argue that it is always time for testing!).

Read more at http://tm3.org/ds

Barely Functional Keystone Deployment with Docker by Adam Young

My eventual goal is to deploy Keystone using Kubernetes. However, I want to understand things from the lowest level on up. Since Kubernetes will be driving Docker for my deployment, I wanted to get things working for a single node Docker deployment before I move on to Kubernetes. As such, you’ll notice I took a few short cuts. Mostly, these involve configuration changes. Since I will need to use Kubernetes for deployment and configuration, I’ll postpone doing it right until I get to that layer. With that caveat, let’s begin.

Read more at http://tm3.org/dt

View article »

RDO @ DevConf

It's been a very busy few weeks in the RDO travel schedule, and we wanted to share some photos with you from RDO's booth at DevConf.cz.

RDO @ DevConf

Led by Eliska Malikova, and supported by our team of RDO engineers, we provided information about RDO and OpenStack, as well as a few impromptu musical performances.

RDO @ DevConf

RDO engineers spun up a small RDO cloud, and later in the day, the people from the Manage IQ booth next door set up an instance of their software to manage that cloud, showing that RDO and Manage IQ are better together.

You can see the full album of photos on Flickr.

If you have photos or stories from DevConf, please share them with us on rdo-list. Thanks!

View article »

Videos from the CentOS Dojo, Brussels, 2017

Last Friday in Brussels, CentOS enthusiasts gathered for the annual CentOS Dojo, right before FOSDEM.

While there was no official videographer for the event, I set up my video camera in the talks that I attended, and so have video of five of the sessions.

First, I attended the session covering RDO CI in the CentOS build management system. I was a little late to this talk, so it is missing the first few minutes.

Next, I attended an introduction to Foreman, by Ewoud Kohl van Wijngaarden

Spiros Trigazis spoke about CERN's OpenStack cloud. Unfortunately, the audio is not great in this one.

Nicolas Planel, Sylvain Afchain and Sylvain Baubeau spoke about the Skydive network analyzer tool.

Finally, there was a demo of Cockpit - the Linux management console by Stef Walter. The lighting is a little weird in here, but you can see the screen even when you can't see Stef.

View article »

FOSDEM Day 0 - CentOS Dojo

FOSDEM starts tomorrow in Brussels, but there's always a number of events the day before.

This year, several speakers from the RDO community are participating in the CentOS Dojo ahead of the main event tomorrow.

  • Haïkel Guémar, Matthieu Huin - CI in the cloud - How RDO uses OpenStack infra tools for packaging
  • Spyros Trigazis - OpenStack @ CERN: Status update
  • Nicolas Planel, Sylvain Afchain, Sylvain Baubeau - Skydive - a real time network analyzer

Of course, since RDO works so closely with the CentOS CI infrastructure, all of the other content is relevant too. We're looking forward to learning about the various aspects of the CentOS project, and strengthening our bonds between the two communities, today, and in the coming years.

Here's KB addressing the opening session (Video).

View article »

Testing RDO with Tempest: new features in Ocata

The release of Ocata, with its shorter release cycle, is close and it is time to start a broader testing (even if one could argue that it is always time for testing!).

One of the core pieces for testing the cloud is the Tempest.

Tempest in RDO

Current status

The status up to the Newton is well described in few blog posts, either from packages or from git. In short, RDO used a forked repository, which regularly received all the changes from the official Tempest. The main reason for this was a configuration script which auto-discovers the features of the cloud (with some hints based on the version) and creates a valid tempest.conf.

Changes in Ocata

The auto-configuration script was decoupled from the internal Tempest and moved to the new python-tempestconf repository, thanks to the work of Martin Kopec, Chandan Kumar and Daniel Mellado.

This means less burden work required (no need to keep the fork) but also simplifies a bit the steps required to initialize and run Tempest tests, close to the process documented by Tempest upstream.

New configuration steps

Configure the RDO repositories, then install the required packages:

$ yum -y install openstack-tempest

python-tempestconf will be installed as well, as new dependency of openstack-tempest.

Now source the admin credentials, initialize tempest and run the discovery tool:

$ source </path/to/>keystonerc_admin
$ tempest init testingdir
$ cd testingdir
$ discover-tempest-config --debug identity.uri $OS_AUTH_URL \
      identity.admin_password  $OS_PASSWORD --create

discover-tempest-config is the new name of the old config_tempest.py script and it accepts the same parameters.

And that's it! Now it is possible to lists and run the tests a usual:

$ tempest run

For more details, see the the upstream documentation of tempest run. As this is a wrapper to testr, ostestr (and direct calls to testr) works as before, even if the usage of tempest run and its filtering features is highly recommended.

Tempest plugins

Current status

Tempest is really composed by two big parts: a library (with an increasing number of stable APIs) and a set of tests. With the introduction of Tempest Plugins the scope of the tests included in tempest.git was limited to the core components (right now Keystone, Nova, Neutron, Cinder, Glance and Swift).

The tests for other projects, as well as the advanced tests for the core components, have been moved to separate repositories. In most of the cases they have been added to the same repository of the main project. This introduces a complication when the tests are split in a separate RPM subpackage, as it is the case in RDO: the entry point for the tests for a component foo is always installed as part of the base subpackage for foo (usually python-foo), but the corresponding code is not (the python-foo-tests is not required). Running any tempest command, the entry points are found but the code could not be there, leading to errors.

The script install_test_packages.py provided in the openstack-tempest RPM could discover the missing entry points and install the required package, but that was clearly a workaround with a maintainance burden.

Changes in Ocata

Thanks again to the work by Chandan Kumar, the packaging was fixed to prevent this problem by automagically tuning the entry points in the generated packages. The interesting technical details are described in a past blog post.

So no more obscure errors due to missing packages while using Tempest, but you may want to check which packages containing tests are really installed if you really want to maximize the testing against your cloud.

View article »

RDO at FOSDEM this week

This week, RDO will have a presence at the FOSDEM conference, in Brussels, Belgium.

First, on Friday, there will be a number of RDO-related presentations at the CentOS Dojo which will be held at the Marriott Grand Place. Registration is free but there are a very limited number of spaces available, so register soon to secure your spot.

FOSDEM_Exhibits_day2 (17 of 38).jpg

On Saturday and Sunday, RDO will be sharing space with CentOS in the stands area, along with dozens of other Free/Open Source organizations, projects, and foundations. We'll also be helping to staff the OpenStack stand.

If you're going to be at FOSDEM, and want to help us staff the OpenStack stand, have a look at the schedule and sign up.

View article »

Last week's blog posts

Here's what RDO enthusiasts have been blogging about in the past few weeks. If you blog about RDO and you're not on this list, please let me know so I can add you to the list.

Standalone Cinder: The definitive SDS by geguileo

Are you looking for the best Software Defined Storage in the market? Look no further, Standalone Cinder is here! Let’s have an overview of the Standalone Cinder service, see some specific configurations, and find out how to make requests with no other OpenStack service is deployed. Cinder Until not so long ago Cinder was always […]

Read more at http://tm3.org/dh

ANNOUNCE: new libvirt console proxy project by Daniel Berrange

This post is to announce the existence of a new small project to provide a websockets proxy explicitly targeting virtual machines serial consoles and SPICE/VNC graphical displays.

Read more at http://tm3.org/di

OpenStack and services for BigData applications by Carlos Camacho

Yesterday I had the opportunity of presenting together with Daniel Mellado a brief talk about OpenStack and it’s integration with services to support Big Data tools (OpenStack Sahara).

Read more at http://tm3.org/dj

OpenStack Use Cases – New Analyst Papers and Webinar Now Available by Peter Pawelski, Product Marketing Manager, Red Hat OpenStack Platform

As the OpenStack market continues to mature, some organizations have made the move and put OpenStack projects into production. They have done this in a variety of ways for a variety of reasons. However, other organizations have waited to see what these first-movers are doing with it and whether or not they are successful before exploring for themselves.  

Read more at http://tm3.org/dk

We'll be at DevConf this weekend! by Rich Bowen

Come see us this weekend at DevConf.cz,in Brno, Czechia. Like last year, we'll be just inside the entrance, with stickers, command-line cheat sheets, RDO tshirts, and people to answer all of your RDO and OpenStack questions.

Read more at http://tm3.org/dl

What are Clouds? by Zane Bitter

Like many in the community, I am often called upon to explain what OpenStack is to somebody completely unfamiliar with it. Usually this goes one of two ways: they turn out to be familiar enough with cloud computing to quickly grasp it by analogy, or their eyes glaze over at the mention of the words ‘cloud computing’ and no further explanation is sought or offered. When faced with someone who is persistently curious but not an industry insider, you immediately know you’re in trouble.

Read more at http://tm3.org/dm

View article »