OpenStack Summit May 2015 Vancouver

For the last 4 cycles, Rackspace has shared in true transparent style the experience gained while learning to scale OpenStack. From poorly optimized SQL queries that threatened to take down entire data centers in Grizzle to having to rewrite the deploy orchestration not once, not twice, but three times to deal with an ever-growing number of nodes, Rackspace engineers working on creating the Build Release Deploy Systems have truly seen it all. Rackspace invites the OpenStack community to hear the story of deployments at scale during the Kilo cycle and share in the latest challenges, triumphs, and lessons learned from regularly upgrading the control plane of the largest production OpenStack public cloud. 

 

While Juno was a challenge of deployments at scale (approximately 45,000   nodes across 6 data centers), Kilo has introduced new complexity in the   form of a service oriented deployment process. In this talk, Jason   Meridth, Rainya Mosher, and Philip Schwartz -- all materially involved   in the build release and deploy systems used by the public cloud --  will  take you deep into the transition from a monolithic packaging for  all OpenStack projects to separate packages and orchestration for  Neutron, Glance, Nova, and Ironic. We will also discuss how Rackspace  consumes  upstream OpenStack from master, upgrading the control plane  many times  during any single OpenStack cycle. Through this practice, we  find  challenging issues for the Operator Community and work with  developers and operators alike to resolve them and strengthen OpenStack  as a whole.

 

Talk Outline 

 

* Config Management: How did we move from Puppet manifests to ansible without gouging  our eyes out?

* Packaging: How did we go about packaging each project separately while still shipping releases to production? 

* Orchestration: How do we deploy to 5,000+ nodes in a single data  center  in about 45 minutes (with 7 minutes or less of customer impact)  and how do we plan to go even faster?

* Ownership: How did we empower development teams to own their product from code to production?

* Complexity: How are we managing the increase in complexity introduced by multi-service deployments? 

* Tooling: How are we using Docker, Ansible, and Jenkins to make all this possible?

 

Talk Takeaways

    

* Viable ways to manage configurations, orchestration, and packaging of  separate services across a large-scale cloud installation

* Best practices to introduce release engineering and system reliability engineering into your operational environment

* The tools being used inside Rackspace Engineering to survive at scale in the constant change of OpenStack