Problem:
Any Openstack operator running a large cloud will eventually find a need to move tenants between zones quickly for:
- Newly added capacity in a new zone
- OpenStack version upgrade
- Cross datacenter migration / cloning
Naïve solution:A simple way to migrate tenants across zones without Flyway is to use Nova and Glance to create a snapshot, download and upload it to the other zone and nova-boot the new VM instance. The downside to this is that glance needs to store and move a lot of redundant data as this the VM's base image that is logically copied over for every snapshot taken. Plus destination not an exact copy of the source, compute and storage wise.
Our Solution: At EBay/PayPal we have developed
Flyway, a tool which enables seamless migration of a tenant from one datacenter to another.
- Tenant view - Can move the array of tenant specific resources (users, roles, key pairs, quotas etc).
- VM, cinder instances, volume data and images from source to target.
- Hypervisors connects to Celery task queue and a message bus enables the transport of VM’s sparse files directly from their source to target hypervisors efficiently. Thus avoiding the redundant data migration as explained above.
To monitor and visualize the progress of Flyway as it runs, we also developed the
Flyview utility. This tool gives a bird’s eye view of a zone and helps us see patterns of usage and quickly identify unusual tenants. It generates SVG images as well as animated GIFs of the zones over their lifetime providing a very insightful visualization.
