The problem: Reducing manual burden during data center failover
Merchant banking group Close Brothers wanted a better, less manual way to manage their data center failover process.
The previous solution used for this was spreadsheets but they wanted something more automated as the process with spreadsheets was open to error and had no audit or control features. It took a long time to tick off every task on the 200-line spreadsheet and record the time it was done, requiring four people to take turns updating the spreadsheet. This often led to delays of at least an hour to catch up, with those running the failover having to look into chats manually to check if and when tasks had been started. The failover was supposed to be completed within 24 hours but could be delayed by an hour or two due to these issues.
The solution: Cutover runbooks provided efficiency, control and visibility throughout the failover process
Instead of this previously manual and inefficient method, the team at Close Brothers decided to use Cutover. This involved around 200 people operating within the platform and another 30 stakeholders who needed to be updated on progress.
Pre-failover
Cutover’s node map made it easy to see the dependencies between tasks and the critical path for the data center failover as a whole. This enabled the team to identify risks immediately and ensure the plan flowed correctly. There was a mix of new and old systems being failed over, with key legacy platforms that had dependencies with multiple other systems. Carrying out tasks in the right order in light of these dependencies was very important, and the node map allowed them to look at dependencies when running through runbooks to see if they had the right flow.
Having the plan in Cutover made it much easier to filter and identify where there were any gaps in information in the plan during the pre-failover checks. The central teams function also made it easy to add or remove people to the runbook, helping admins save time as they didn’t have to amend the runbook every single time there was a change to the plan.
During failover
The failover involved around 200 people using the platform to execute 1400 tasks plus around 30 stakeholders that needed to be kept updated on progress. For individual members of the team, being able to filter and view the tasks assigned to them and see which ones were startable or the progress of previous tasks was convenient and saved time.
For stakeholder comms, the team was able to automate the sending of the Cutover dashboard to stakeholders at regular intervals, rather than having to manually type updates. Any comments against tasks in Cutover could also be highlighted to include in these reports for added context. Cutover also provided a RAG status so users and stakeholders alike could see if the failover was on target.
Admins also set up filters for each business unit so they could quickly check the progress of each individually at the tap of a button as well as see the progress of the failover as a whole.
Post-failover
Cutover’s automated audit trail enabled admins to view exactly which tasks had been executed, when, and by whom, without having to manually search for the information in multiple sources.
The outcome: Confidence in data center failover
Overall, the team was able to successfully complete the data center failover. They had more confidence during the planning stages and saved significant time and manual effort, particularly on sharing stakeholder comms.
