RDO Community News

See also blogs.rdoproject.org

Blog posts last week

Here's what RDO enthusiasts have been posting about in the last week:

Digital Foundations – Challenges CIOs Must Embrace by Eric D. Schabell

When building anything substantial, such as a house or bridge, you start by laying down a solid foundation. Nothing changes this aspect of building brick by brick when you move from traditional constructions to application development and architecting your supporting infrastructure. Throw in Cloud terminology and you might think that the principles of a solid foundation are a bit flighty, but nothing is further from the truth.

Read more at http://tm3.org/cv

Is OpenStack Neutron ML2/OVS Production Ready for Large Scale Deployments? by assafmuller

One of my personal highlights of the recent Barcelona Summit was a session by Mirantis engineers Elena and Oleg titled “Is OpenStack Neutron Production Ready for Large Scale Deployments?”. In the session they outline a comprehensive control and data plane testing effort, run on two labs, one with 200 nodes and run of the mill hardware, and the other with 378 and top of the line hardware, all running the Mirantis distribution based off Mitaka with standard ML2/OVS, DVR, L2POP and VXLAN. In the session they show near line-rate speed for east/west and north/south routing with jumbo frames and VXLAN offload enabled. They were also able to spawn 24,500 VMs across 125 networks without errors and low CPU consumption.

Read more at http://tm3.org/cw

Keystone Development Bootstrap with Service Catalog by Adam Young

My Last post showed how to get a working Keystone server. Or did it.

Read more at http://tm3.org/cx

QEMU Advent Calendar 2016 by kashyapc

The QEMU Advent Calendar website 2016 features a QEMU disk image each day from 01-DEC to 24 DEC. Each day a new package becomes available for download (of format tar.xz) which contains a file describing the image (readme.txt or similar), and a little run shell script that starts QEMU with the recommended command-line parameters for the disk image.

Read more at http://tm3.org/cy

Keystone development server on Fedora 25 by Adam Young

While Unit tests are essential to development, I often want to check the complete flow of a feature against a running keystone server as well. I recently upgraded to Fedora 25, and had to reset my environment. Here is how I set up for development.

Read more at http://tm3.org/cz

6 videos on how to install Red Hat OpenStack Platform and CloudForms by Marcos Garcia - Principal Technical Marketing Manager

Our excellent Training & Certification team has posted some videos in our RedHatCloud youtube channel that quickly go over the installation procedure of Red Hat OpenStack Platform 8, and how to boot a CloudForms instance to perform basic management functions. Kudos to our awesome video team (Jim Meegan and Ben Oliver) and to our curriculum architect (Forrest Taylor).

Read more at http://tm3.org/c-

View article »

Blog posts last week

Here's what RDO enthusiasts have been blogging about in the past week.

ARA 0.10, the biggest release yet, is out ! by dmsimard

19 commits, 59 changed files, 2,404 additions and 588 deletions… and more than a month’s on and off work.

Read more at http://tm3.org/cp

Upstream Contribution – Give Up or Double Down? by assafmuller

Ever since I’ve been involved with OpenStack people have been complaining that upstream is hard. The number one complaint is that it takes forever to get your patches merged. I thought I’d take a look at some data and attempt to visualize it. I wrote some code that accepts an OpenStack project and a list of contributors and spits out a bunch of graphs. For example:

Read more at http://tm3.org/cq

Recorded presentations from OpenStack Canada Day by dmsimard

OpenStack Days are sort of local mini one-day OpenStack summits.

Read more at http://tm3.org/cr

How to Install and Run Tempest by mkopec

Tempest is a set of integration tests to run against an OpenStack cluster. In this blog I'm going to show you, how to install tempest from git repository, how to install all requirements and run tests against an OpenStack cluster.

Read more at http://tm3.org/cs

On communities: Empower humans to be amazing by Flavio Percoco

When it comes to communities, a system is the set of processes you put in place

Read more at http://tm3.org/ct

How are you using RDO? (Survey results) by Rich Bowen

Over the last few weeks, we've been conducting a survey of RDO users, to get an idea of who is using RDO, how, and for what.

Read more at http://tm3.org/ck

Testing composable upgrades by Carlos Camacho

This is a brief recipe about how I’m testing composable upgrades N->O.

Read more at http://tm3.org/cl

TripleO cheatsheet by Carlos Camacho

This is a cheatsheet some of my regularly used commands to test, develop or debug TripleO deployments.

Read more at http://tm3.org/cm

JSON Home Tests and Keystone API changes by Adam Young

If you change the public signature of an API, or add a new API in Keystone, there is a good chance the Tests that confirm JSON home layout will break.  And that test is fairly unfriendly:  It compares a JSON doc with another JSON doc, and spews out the entirety of both JSON docs, without telling you which section breaks.  Here is how I deal with it:

Read more at http://tm3.org/cn

View article »

How are you using RDO? (Survey results)

Over the last few weeks, we've been conducting a survey of RDO users, to get an idea of who is using RDO, how, and for what.

While the sample size here is small, it's a start at understanding the makeup of our user community. And we'll conduct the survey again around the next release, so if you missed this one, stay tuned.

The Numbers

Let's start with the numbers.

First of all, there were only 39 responses to the survey. Hopefully next time we can do a better job of getting responses, but this is a good start for the first time.

Most of our users (ie, more than half) are running the Mitaka release, witht he next-largest number (46%) having already moved over to the Newton release.


Nearly half of our users are running RDO as their production cloud.


38% of users are deploying with Packstack, with just 20% using TripleO.


There's no clear leader in terms of what industry our users are in, however, Research, Service Providers and Telecom are the three at the top.


Finally, in the distribution of cloud size, over half of respondants were in the 1-10 nodes range, with the rest spread everywhere from there to more than 7500 nodes.


There were some additional questions that will be summarized to the rdo-list over the coming days, regarding how people want to get more involved in the project, and what things they feel are missing.

Next Time

Doing surveys is hard, and invaribly as soon as you send a survey out into the wild, you realize some things that you wish you'd done differently. This time, one thing we did was make the "industry" and "size" fields free-form entry, rather than providing options. This made it a lot more work to tally the results.

Beyond that, if there's some things that you feel that we should have done differently in the survey, please speak up.

View article »

Blog posts in the past week

Here's what RDO enthusiasts have been blogging about over the last week or so:

Chasing the trunk, but not too fast by amoralej

As explained in a previous post, in RDO Trunk repositories we try to provide packages for new commits in OpenStack projects as soon as possible after they are merged upstream. This has a number of advantages:

Read more at http://tm3.org/c9

Enabling nested KVM support for a instack-virt-setup deployment. by Carlos Camacho

The following bash snippet will enable nested KVM support in the host when deploying TripleO using instack-virt-setup.

Read more at http://tm3.org/ca

Ocata OpenStack summit 2016 - Barcelona by Carlos Camacho

A few weeks ago I had the opportunity to attend to the Barcelona OpenStack summit ‘Ocata design session’ and this post is related to collect some overall information about it. In order to achieve this, I’m crawling into my paper notes to highlight the aspects IMHO are relevant.

Read more at http://tm3.org/cb

On communities: Emotions matter by Flavio Percoco

Technology is social before it's technical - Gilles Deleuze

Read more at http://tm3.org/cc

New TLS algorithm priority config for libvirt with gnutls on Fedora >= 25 by Daniel Berrange

Libvirt has long supported use of TLS for its remote API service, using the gnutls library as its backend. When negotiating a TLS session, there are a huge number of possible algorithms that could be used and the client & server need to decide on the best one, where “best” is commonly some notion of “most secure”. The preference for negotiation is expressed by simply having an list of possible algorithms, sorted best to worst, and the client & server choose the first matching entry in their respective lists. Historically libvirt has not expressed any interest in the handshake priority configuration, simply delegating the decision to the gnutls library on that basis that its developers knew better than libvirt developers which are best. In gnutls terminology, this means that libvirt has historically used the “DEFAULT” priority string.

Read more at http://tm3.org/cd

New libvirt website design by Daniel Berrange

The current previous libvirt website design dated from circa 2008 just a few years after the libvirt project started. We have grown alot of content since that time, but the overall styling and layout of the libvirt website has not substantially changed. Compared to websites for more recently launched projects, libvirt was starting to look rather outdated. So I spent a little time to come up with a new design for the libvirt website to bring it into the modern era. There were two core aspects to the new design, simplify the layout and navigation, and implement new branding.

Read more at http://tm3.org/ce

Quick Guide: How to Plan Your Red Hat Virtualization 4.0 Deployment by Eric D. Schabell

On August 24th of this year Red Hat announced the newest release of Red Hat Virtualization (RHV) 4.0.

Read more at http://tm3.org/cf

Visualizing Kolla’s Ansible playbooks with ARA by dmsimard

Kolla is an OpenStack deployment tool that’s growing in popularity right now.

Read more at http://tm3.org/cg

Recapping OpenStack Summit Barcelona by Peter Pawelski, Product Marketing Manager, Red Hat OpenStack Platform

More than 5,200 OpenStack professionals and enthusiasts gathered in Barcelona, Spain to attend the 2016 OpenStack Summit. From the keynotes to the break-out sessions to the marketplace to the evening events and the project work sessions on Friday, there was plenty to keep attendees busy throughout the week. In fact, if you were one of the lucky ones who attended OpenStack Summit, there was probably many sessions and activities you wanted to make it to but couldn’t.

Read more at http://tm3.org/ch

View article »

How to Install and Run Tempest

UPDATES: RDO Ocata introduces few changes that partially obsoletes the content of this article; check the updated article for details.

Tempest is a set of integration tests to run against an OpenStack cluster. In this blog I'm going to show you, how to install tempest from git repository, how to install all requirements and run tests against an OpenStack cluster.

I'm going to use a fresh installation of Centos7 and OpenStack cluster provided by packstack. If you've done that, follow the instructions below.

Tempest Installation

You have two options how to install tempest. You can install it through RPM or you can clone tempest from GitHub repository. If you choose installation through RPM, follow this link.

Installation from GitHub repository

Now you can clone upstream tempest or you can clone RedHat's fork of upstream tempest. The RedHat's fork provides config_tempest.py, which is a configuration tool. It will generate tempest.conf for you, what can be handy.

[1.] Install dependencies:

$ sudo yum install -y gcc python-devel libffi-devel openssl-devel

[2.] Clone tempest:

$ git clone https://github.com/openstack/tempest.git

Or (RedHat's fork):

$ git clone https://github.com/redhat-openstack/tempest.git

[3.] Install pip, for example:

$ curl "https://bootstrap.pypa.io/get-pip.py" -o "get-pip.py"
$ sudo python get-pip.py

[4.] Install tempest globally in the system. If you don't want to do that, skip this step and continue reading.

$ sudo pip install tempest/

Install tempest in a virtual environment

Sometimes you don't want to install things globally in the system. For this reason you may want to use a virtual environment. I'm going to explain installation through virtualenv and tox.

Setting up Tempest using virtualenv

[1.] Install virtualenv:

$ easy_install virtualenv

Or through pip:

$ pip install virtualenv

[2.] Enter tempest directory you've cloned before:

$ cd tempest/

[3.] Create a virtual environment and let's name it .venv:

$ virtualenv .venv
$ source ./venv/bin/activate

[4.] Install requirements:

(.venv) $ pip install -r requirements.txt
(.venv) $ pip install -r test-requirements.txt

NOTE: If problems occur during requirements installation, it may be due to an old version of pip, upgrade may help:

(.venv) $ pip install pip --upgrade

[5.] After dependencies are installed, run following commands, which install tempest within the virtual environment:

(.venv) $ cd ../
(.venv) $ pip install tempest/

Or this command does the same without using pip:

$ python setup.py install If you need to trigger installation to developer mode run:

(.venv) $ python setup.py develop `setup.py develop` comes from limitations on [pbr](http://docs.openstack.org/developer/pbr/). If you are interested, [here is](https://setuptools.readthedocs.io/en/latest/setuptools.html#development-mode) an explanation of difference between `install` and `develop`.

Setting up Tempest using TOX

[1.] Install tox:

$ easy_install tox

Or if you want to use pip:

$ pip install tox

[2.] Install tempest:

$ tox -epy27 --notest
$ source .tox/py27/bin/activate This will create a virtual environment named `.tox`, install all dependencies (*requirements.txt* and *test-requirements.txt*) and tempest within it. If you check `tox.ini` file, you'll see tox actually run tempest installation in develop mode you could run manually as it was explained above.


[3.] If you want to expose system-site packages, tox will do it for you. Deactivate environment, you are currently in (if you followed the step before) and create another environment:

(py27) $ deactivate
$ tox -eall-plugin --notest
$ source .tox/all-plugin/bin/activate

[4.] Then if you want to install plugins test packages based on the OpenStack Components installed, let this script to do it:

(all-plugin) $ sudo python tools/install_test_packages.py
(all-plugin) $ python setup.py develop

Generate tempest.conf

About tempest.conf and what it is used for you can read in this documentation. If you want to create tempest.conf let config_tempest.py do it for you. The tool is part of RPM tempest (check this documentation) or if you don’t want to install tempest globally, you can clone RedHat's tempest fork and install it within a virtual environment as it was explained above.

RedHat's tempest fork

Create a virtual environment as I already mentioned and source credentials (if you installed OpenStack cluster by packstack credentials are saved in /root/):

(.venv) $ source /root/keystone_admin

And run config tool:

(.venv) $ python tools/config_tempest.py --debug identity.uri $OS_AUTH_URL \
         identity.admin_password  $OS_PASSWORD --create  After this, `./etc/tempest.conf` is generated.

NOTE: If you running OSP, it’s needed to add a new argument to config_tempest tool:

(.venv) $ ./tools/config_tempest.py object-storage.operator_role swiftoperator

It's because OSP uses lowercase operator for the swift operator_role, however, tempest default value is "SwiftOperator". To override the default value run config_tool like this:

$ python tools/config_tempest.py --debug identity.uri $OS_AUTH_URL \
  identity.admin_password  $OS_PASSWORD \
  object-storage.operator_role swiftoperator --create

Running tests

If you've installed tempest and have tempest.conf, you cant start testing. To run tests you can use testr or ostestr. If you want to run tempest unit tests, check this out.

Note: following commands run withing the virtual environment you've created before. To run specific tests run for example:

$ python -m testtools.run tempest.api.volume.v2.test_volumes_list OR

$ ostestr --regex tempest.api.volume.v2.test_volumes_list

Alternatively you can use tox, for example:

$ tox -efull Run only tests tagged as smoke:

$ tox -esmoke
View article »

Chasing the trunk, but not too fast

As explained in a previous post, in RDO Trunk repositories we try to provide packages for new commits in OpenStack projects as soon as possible after they are merged upstream. This has a number of advantages:

  • It allows packagers to identify packaging issues just after introduced.
  • For project developers, their changes are tested in non-devstack environments, so test coverage is extended.
  • For deployment tools projects, they can use these repos to identify problems with new versions of packages and to start integrating any enhancement added to projects as soon as it's merged.
  • For operators, they can use these packages as hot-fixes to install in their RDO clouds before patches are included in official packages.

This means that, for every merged commit a new package is created and a yum repository is published in RDO trunk server. This repo includes the just built package and the latest builds for the rest of packages in the same release.

Initially, we applied this approach to every package included in RDO. However, while testing these repos during the Newton cycle we observed that jobs failed with errors that didn't affect OpenStack upstream gates. The reason behind this is that commits in OpenStack gate jobs are tested with versions of libraries and clients defined in upper-constraints.txt files in requirements project for the branch where the change is proposed. Typically these are the last tag-released versions. As RDO was testing using libraries from last commit instead of last release, we were effectively ahead of upstream tests, running too fast.

While this provided some interesting information and we could identify issues very early, it made very difficult to get stable repositories that could be promoted and used. After some discussions in RDO weekly meeting, it was decided to apply some changes in the way libraries are managed to leverage the work done in upstream gates but try to keep catching issues as soon as possible:

  • For master branch, it was decided to pin libraries and clients to the versions included in upper-constraints for repositories served in http://trunk.rdoproject.org/centos7. This repositories are used by RDO CI promotion jobs and marked as current-passed-ci when all tests succeed.
  • Additionally, a new builder was created that chases master in all packages, including libraries, clients, etc… This builder is able to catch issues based on unit tests executed when packages are created. The produced repos are available in http://trunk.rdoproject.org/centos7-master-head but promotion jobs are not executed using them.

The differences between master and master-head are shown in following diagram:

RDO master pins

  • For releases in maintenance phase we pin libraries to what's found in upper-constraints.txt file in the corresponding branch.


In order to manage the libraries versions properly, RDO is using a peer-reviewed workflow of gerrit reviews proposed to the rdoinfo project in http://review.rdoproject.org. You can see an example here.

A job is executed periodically that automatically creates gerrit reviews when versions are updated in upper-constraints files. Manual approval is needed to get the changes merged and the new versions built by DLRN builders.

View article »

Blog posts last week

We've had more followup blog posts from OpenStack Summit, along with some more from the RDO community.

Querying haproxy data using socat from CLI by Carlos Camacho

Currently (most users) I don’t have any way to check the haproxy status in a TripleO virtual deployment (via web-browser) if not previously created some tunnels enabled for that purpose.

Read more at http://tm3.org/c3

Keystone Domains are Projects by Adam Young

Yesterday, someone asked me about inherited role assignments in Keystone projects. Here is what we worked out.

Read more at http://tm3.org/c4

OpenStack Summit: An evening with Ceph and RDO by Rich Bowen

Last Tuesday in Barcelona, we gathered with the Ceph community for an evening of food, drinks, and technical sessions.

Read more at http://tm3.org/c5

OpenStack Summit Barcelona, 3 of N by rbowen

Continuing the saga of OpenStack Summit Barcelona …

Read more at http://tm3.org/c6

Red Hat Virtualization: Bridging the Gap with the Cloud and Hyperconverged Infrastructure by Ted Brunell

Red Hat Virtualization offers a flexible technology for high-intensive performance and secure workloads. Red Hat Virtualization 4.0 introduced new features that enable customers to further extend the use case of traditional virtualization in hybrid cloud environments. The platform now easily incorporates third party network providers into the existing environment along with other technologies found in next generation cloud platforms such as Red Hat OpenStack Platform and Red Hat Enterprise Linux Atomic Host. Additionally, new infrastructure models are now supported including selected support for hyperconverged infrastructure; the native integration of compute and storage across a cluster of hosts in a Red Hat Virtualization environment.

Read more at http://tm3.org/c7

Running Tempest on RDO OpenStack Newton by chandankumar

Tempest is a set of integration tests to run against an OpenStack cluster.

Read more at http://tm3.org/bk

View article »

OpenStack Summit: An evening with Ceph and RDO

Last Tuesday in Barcelona, we gathered with the Ceph community for an evening of food, drinks, and technical sessions.

There were 215 in attendance at last count, and we had 12 presentations, from members of both communities.


A huge thank you to all of the speakers, and to all of the people who turned out for this great evening.


More photos HERE.

Some of the presentations from the event are available HERE

View article »

Blog posts last week

With OpenStack Summit last week, we have a lot of summit-focused blog posts today, and expect more to come in the next few days.

Attending OpenStack Summit Ocata by Julien Danjou

For the last time in 2016, I flew out to the OpenStack Summit in Barcelona, where I had the chance to meet (again) a lot of my fellow OpenStack contributors there.

Read more at http://tm3.org/bu

OpenStack Summit, Barcelona, 2 of n by rbowen

Tuesday, the first day of the main event, was, as always, very busy. I spent most of the day working the Red Hat booth. We started at 10 setting up, and the mob came in around 10:45.

Read more at http://tm3.org/bx

OpenStack Summit, Barcelona, 1 of n by rbowen

I have the best intentions of blogging every day of an event. But every day is always so full, from morning until the time I drop into bed exhausted.

Read more at http://tm3.org/by

TripleO composable/custom roles by Steve Hardy

This is a follow-up to my previous post outlining the new composable services interfaces , which covered the basics of the new for Newton composable services model.

Read more at http://tm3.org/bo

Integrating Red Hat OpenStack 9 Cinder Service With Multiple External Red Hat Ceph Storage Clusters by Keith Schincke

This post describes how to manually integrate Red Hat OpenStack 9 (RHOSP9) Cinder service with multiple pre-existing external Red Hat Ceph Storage 2 (RHCS2) clusters. The final configuration goals are to have Cinder configuration with multiple storage backends and support …

Read more at http://tm3.org/bz

On communities: Sometimes it's better to over-communicate by Flavio Percoco

Communities, regardless of their size, rely mainly on the communication there is between their members to operate. The existing processes, the current discussions, and the future growth depend heavily on how well the communication throughout the community has been established. The channels used for these conversations play a critical role in the health of the communication (and the community) as well.

Read more at http://tm3.org/c0

Full Stack Automation with Ansible and OpenStack by Marcos Garcia - Principal Technical Marketing Manager

Ansible offers great flexibility. Because of this the community has figured out many useful ways to leverage Ansible modules and playbook structures to automate frequent operations on multiple layers, including using it with OpenStack.

Read more at http://tm3.org/bs

View article »

Next week in Barcelona

Join us next week in Barcelona for OpenStack Summit. We'll be gathering from around the world to celebrate the Newton release, and plan for the Ocata cycle.

RDO will have a table in the Red Hat booth, where we'll be answering your questions about RDO. And we'll have ducks, as usual.


On Tuesday evening, join us for an evening with RDO and Ceph, with technical presentations about both projects, as well as drinks and light snacks.

And, throughout the week, RDO enthusiasts are giving a wide variety of talks about all things OpenStack.

If you're using RDO, please stop by and tell us about it. We'd love to meet you, and find out what we, as a project, can do better for you and your organization.

See you in Barcelona!

View article »