Our two storage clusters in Zurich and Lausanne are starting to get well filled. Some of the disks are reaching a fill state that could turn critical soon. Should that happen, then the whole cluster basically stops working - not something we are too keen on having happen to us.
We therefore will configure both clusters with more disks. Zurich is going from 128 to 160 disks, Lausanne from 48 to 96. The upgrade of storage capacity has the effect that our storage system will re-shuffle a lot of data to optimally use the available disk space.
Now comes the unfortunate part. During the re-shuffle (which will probably move around around 1 PB of data), the performance of the storage cluster will be degraded. That means that read/write requests from the VMs to their disks can hang for large amount of times. The effect of that is, that the virtual machine will seem to be frozen.
We expect performance degradations for up to 24 hours after the expansion has started (the worst part is the first 8 hours). We will start the expansion at midnight, Friday 11.12.2015 and monitor the situation as long as the cluster is re-balancing.
The net, positive effects of this upgrade are: More storage capacity and increased performance.