The problem: Manual and inefficient failover processes
A major multinational investment company had a highly manual and inefficient failover process for both tests and real disaster recovery failovers. The steps to perform a failover were documented in a static log and users had to manually type in scripts to execute the jobs they needed to run through Ansible. This involved a lot of people and was difficult to coordinate and manage. Handoffs were managed over bridge lines and messaging tools and often became messy and disorganized.
The solution: Automated failover runbooks
The team moved their failover process out of static documentation and into a Cutover automated runbook template. The new process only requires one person to kick off the failover recovery or test with virtually one click, which automatically creates individual runbooks for each service being failed over. These runbooks integrate directly with Ansible, and complete tasks automatically without the need for human intervention or manual inputs.
Cutover provides a custom dashboard so it is easy to see how many runbooks are running and the average duration of each runbook, with alerts for any that fall behind schedule. Plus, auto-generated audit logs made continuous improvement and regulatory reporting simple.
The outcome: 53% more efficient failover!
As a result of this new automated process with Cutover, both manual effort and the number of people needed to execute failovers are reduced. Cutover is 53% more efficient than the previous process and the average recovery time for failing over each application is now 38 minutes, when previously it could take over four hours to recover a scope of applications. The team is now confident that they can continue to increase efficiency with Cutover and reduce that time even further, as well as scaling up to other use cases.
“Cutover beat every expectation that we had” - participating App Team member
