OpenStack Swift mid-cycle hackathon summary

Last week more than 30 people from all over the world met at the Rackspace office in San Antonio, TX for the Swift mid-cycle hackathon. All major companies contributing to Swift sent people, including Fujitsu, HPE, IBM, Intel, NTT, Rackspace, Red Hat, and Swiftstack. As always it was a packed week with a lot of deep technical discussions around current and future changes within Swift.

There are always way more topics to discuss than time, therefore we collected topics first and everyone voted afterwards. We came up with the following major discussions that are currently most interesting within our community:

  • Hummingbird replication
  • Crypto - what's next
  • Partition power increase
  • High-latency media
  • Container sharding
  • Golang - how to get it accepted in master
  • Policy migration

There were a lot more topics, and I like to highlight a few of them.

H9D aka Hummingbird / Golang

This was a big topic - as expected. It has been shown by Rackspace already that H9D improves the performance of the object servers and replication significantly compared to the current Python implementation. There were also some investigations if it would be possible to improve the speed using PyPy and other improvements; however the major problem is that Python blocks processes on file I/O, no matter if it is async IO or not. Sam wrote a very nice summary about this earlier on [1].

NTT also benchmarked H9D, and showed some impressive numbers as well. Shortly summarized, throughput increased 5-10x depending on parameters like object size and the like. It seems disks are no longer the bottleneck - now the proxy CPU is the new bottleneck. That said, inode cache memory seems to be even more important because with H9D one can do many more disk requests.

Of course there were also discussions about another proposal to accept golang within OpenStack and discussions will continue [2]. My personal view is that the H9D implementation has some major advantages and hopefully (a refactored subset) will be accepted to be merged to master.

Crypto retro & what's next

Swift 2.9.0 has been released the past week and includes the merged crypto branch [3]. Kudos to everyone involved, especially Janie and Alistair! This middleware make it possible for operators to fully encrypt object data on disk.

We did a retro on the work done so far; it has been the third time that we used a feature branch and a final soft-freeze to land a major change within Swift. There are pros and cons for this, but overall it worked pretty well again. It also made sense that reviewers stepped in late in the process, because this added new sights onto the whole work. Soft freezes also enforce more reviewers to contribute to it and get it merged finally.

Swiftstack benchmarked the crypto branch; as expected the throughput decreases somewhat with crypto enabled (especially with small objects), while proxy CPU usage increases. There were some discussions about improving the performance, and it seems the impact from checksumming is significant here.

Next steps to improve the crypto middleware is to work on some external key master implementations (for example using Barbican) as well as key rotation.

Partition power increase

Finally there is a patch ready for review now, that will allow an operator to increase the partition power without downtime for end users [4].

I gave an overview about the implementation, and also showcased a demo how this works. Based on discussions during the last week I spotted some minor eventualities that have been fixed meanwhile, and I hope to get this merged before Barcelona. We somewhat dreamed about a future Swift that might be usable with automatic partition power increase, where an operator needs to think about this much less than today.

Various middlewares

There are some proposed middlewares that are important to their authors, and we discussed quite a few of them. This includes:

  • High-latency media (aka archiving)
  • symlinks
  • notifications
  • versioning

The idea to support high-latency media is to use cold storage (like tape or other public cloud object storage with a possible multi-hour latency) for less frequently accessed data and especially to offer a low-cost long-term archival solution based on Swift [5]. This is somewhat challenging for the upstream community, because most contributors don't have access to large enterprise tape libraries for testing. In the end this middleware needs to be supported by the community, and a stand-alone repository outside of Swift itself might make most sense therefore (similar to the swift3 middleware [6]).

A new proposal to implement true history-based versioning has been proposed earlier on, and some open questions have been talked about. This should land hopefully soon, adding an improved way to versioning compared to today's stack-based versioning [7].

Sending out notifications based on writes to Swift have been discussed earlier on, and thankfully Zaqar now supports temporary signed urls, solving some of the issues we faced earlier on. I'll update my patch shortly [8]. There is also another option to use oslo.messaging. All in all, the whole idea will be to use a best-effort approach - it's simply not possible to guarantee a notification has been delivered successfully without blocking requests.

Container sharding

As of today it's a good idea to avoid billions of objects in a single container in Swift, because writes to that container can get slow then. Matt started working on container sharding sometime ago [9], and iterated once again because he faced new problems with the previous ideas. My impression is that the new idea is getting much closer to something that will eventually be merged, thanks to Matt's persistence on this topic.

Summary

There were a lot more (smaller) topics that have been discussed, but this should give you an overview of the current work going on in the Swift community and the interesting new features that we'll see hopefully soon in Swift itself. Thanks everyone who contributed and participated and special thanks to Richard for organizing the hackathon - it was a great week and I'm looking forward to the next months!