The problem
A British multinational bank needed to implement a comprehensive resilience solution. They were under regulatory scrutiny and needed the capability to pull together thousands of standardized Service Recovery plans into test scenarios in minutes rather than months. The operational resilience testing function involved thousands of people globally with resilience events every weekend involving hundreds and at times thousands of applications.
The existing home-grown system for this process was insufficient and did not provide the level of planning, visibility or communication that was needed to ensure success. It did not provide the required level of resilience assurance they needed to meet regulatory requirements. They needed to find a better way than spreadsheets, word documents and wiki pages to store and execute the SRPs for their 5,000 services.
How we helped
Cutover provides a comprehensive operational resilience platform that hosts thousands of SRPs that can be configured into various test scenarios in minutes. Now that the bank is using Cutover, a single DCR test that can encompass more than 300 SRPs can be prepared and managed independently and then merged into Cutover for orchestration and enterprise observability. Having SRPs on Cutover also allows the bank to standardize them using a template, so that there is minimal work to finalize DCR test Runbooks. Cutover provides users with the status information and updates they need during the test itself to ensure success without manual effort, such as being able to visualize the critical path which is highlighted in Cutover. As a result of Cutover’s built-in automated communications, the bank has eliminated conference calls saving time across the teams.
To increase efficiency and resilience, the bank also created a library of SRPs in Cutover, replacing the existing spreadsheet versions held on shared drives. This made it quicker and easier for users to find, review, edit and execute on SRPs and build new ones based on templates and to collect data on executed SRPs.
Results
Through their partnership with Cutover, the bank reduced their time to test from 12 weeks down to two, saving significant time and resources. The team running the event are now better informed during data center tests, helping to improve decision making. They were also better able to
collaborate with the auditor and meet audit points as a record of all activity was automatically provided by Cutover.
Due to the success of past and current resilience activities, Cutover is also being used for other resilience events, such as a Mainframe Crossover, where a substantial portion of the banks’ infrastructure, based around their primary mainframe, is failed over to an alternate site. Cutover is also being used to support utility testing, such as testing the resilience of a specific infrastructure service.
Cutover helps us go faster on change and safeguard customer-facing operations in a range of use cases, from supporting our cloud migration to enabling an effective service recovery capability.
- Group CTO & Chief Innovation Officer
