OpenStack Summit May 2015 Vancouver

The CERN IT Department operates a large distributed OpenStack infrastructure with around 10,000 VMs and 100,000 CPU cores. This cloud is predominantly used for physics data analysis, but there remains a long tail of conventional IT services and user-managed application VMs. These varied use-cases created a new requirement in our department, one to build a scalable, reliable, performant and future-proof block storage backend. To solve this problem, in mid-2013 we deployed 3 petabytes of Ceph software-defined storage.

This talk will present a year in the life of our Ceph cluster, focusing on the operations issues related to the RBD use-cases. Based on our experience hosting around 1000 Cinder volumes and 1500 Glance images, we’ll describe how the cluster is used, its access patterns and growth, mistakes we've made, problems overcome, upgrades successfully performed, and highlight some tips and tricks we’ve learned to tune both the servers (Ceph) and clients (qemu-kvm). Lastly, we’ll conclude by outlining our future plans for cloud-based block storage across our multiple datacentres and describe where we believe cloud storage may have further potential at CERN.