So the question is, how confident are you that your organization could show regulators consistent evidence that it's Doctor plan works under real pressure across the appropriate jurisdictions that you operate in. And the options are A, very confident we test regularly and can show the evidence that the sort of the plan is being used for that test. Two, B somewhat confident we have a plan, but evidence varies by region or hasn't been fully stress tested. C not very confident, consistent proof is the gap for us right now or D not surenot my area. So again, those options are A, folks are very confident they test regularly and can show the evidence particularly that the plan was executed and not just the evidence of the actual failover or be somewhat confident that we have a plan, but evidence varies by region or hasn't been fully stress tested. C not very confident. Consistent proof is the gap for us right now or D not sure this is not my area. Okay, so let's look at where the pool landed. So I think it's pretty statistically significant that folks felt that they're somewhat confident and the evidence is varying and maybe that's a good perspective for us having the conversation for where the guests of the webinar are right now. And a close, well, the second by some gap actually is not sure, not my area before we get too very confident. So yeah, I think that shows a really interesting perspective. Okay, so to start the discussion panel question number one is what are recent developments you have seen in terms of using AI agents in operational resilience and disaster recovery? And I'll start by asking Bill about this and come to Tania. Yeah, sure. So what we're seeing is the move to more active participation by an AI agent, for example. So rather than just simply providing reporting and insights. I think that it is accelerating incident detection if you set it up to do that. Traditionally, teams could spend hours trying to correlate your infrastructure, your application failures, your impacts to customers and third parties. It can... You know, an AI agent, deployed correctly can really cut down on that information gathering. That... That's what it does really well if if you can figure it. It can analyze your your your infrastructure. So if you can connect it to ServiceNow, Splunk, any other environment so that it can be constantly reading that. I think that during a recovery event, you can see maybe there's predefined recovery playbooks that I can execute against. I I I think that I was at a a conference recently where, University of Chicago had already set up agents where their their BCP plans, RTO and RPOs were already connected to various systems so that it would automatically email the owners of those systems that they're... That they've breached. Right? So there there are some pretty good use cases that run twenty four seven without a human having to be there to know that something's going down or see an IT ticket. You know, I think scenario testing is a really good area to use it. I recently did an executive desktop, and in in in putting this this desktop together, we really pushed the limit on, you know, what we've done in the past versus what we're doing. And the scenario that I built, a lot of the the core ideas did come from using some AI input. Right? It it would... It it can really give you insights on, you know, what the industry is seeing as opposed to just what your organization's seeing and and building that in. So... And then finally, I think that, you know, it can help you map your important business services. Important business services are going to be kinda, I think, the linchpin to building a good resiliency program. And so the faster you can do that... I did not have the opportunity to use AI to help build my IBS. I wouldn't recommend you jump right in and and do it with an AI agents, you know, because you should know what you're doing first before you deploy a tool. But, so, yeah, I think that, you know, overall, you know, the organizations that are gonna get the most value aren't replacing humans. They're using this to compress detection time and to and to make, you know, resilience an advantage, right, using an AI tool or agent to help you along. So, yeah, I think that's that's kinda where I'm where I'm sitting at. That's really insightful, Bill. And and maybe I'd prompt a few thoughts from myself on that, but I'll come to Tanya first to to share yours. But thank you for... Thanks, Bill, for that. Yeah. No, that's great. And I think building on that, I think what stands out for me from a governance perspective, a second line, a risk and compliance perspective is just how quickly the governance framework is becoming, the most material or more material than the actual initial use case. And I think to Bill's point, you better understand what you're doing before you deploy the agent. I feel like that's gonna be the theme throughout this entire discussion, at least one of the key themes, because in fact, that's true. You know, the concern always on the second line is the view and the caution is to ensure that the capability is moving in line with, or at least the assurance is moving in line with the capability. So organizations may be experimenting with agents before they have a complete inventory even of where they're used and what what data and services they depend on and and what permissions they even hold. So if you think about in the fintech environment, if we know you deploy an agent, maybe to simplify and help correlate an incident after, let's say, a customer facing payment service goes down. What information can that agent rely on? Can it only recommend? Can it initiate any actions, and ultimately who remains accountable. So the shift really that we're seeing is from, in the governance side anyhow, is from, an AI as an analytical assistant to an agent that can actually influence decision set that's actually bringing in, some insights and influencing decisions and in some settings initiate, actions. So the opportunity is real, but the capability moving faster than the assurance is the real concern to the risk and compliance professional. I think that great points. Think just reflecting first on Bills then Tania is there. I think the, as you say, the sort of involvement we see of agents early in major incidents where correlation is done super fast. So humans are coming into a very well briefed major incident where otherwise that would have been the first X minutes of any major incident would have been scrabbling around the logs and checking whether there's any change tickets recently for that system. Having that all populated and predefined pretty early on, I think it's been a huge help. I think the, and like you say, Bill of sort of making sure you learn what you do before you try and develop agents. I think it was this the same framework we wanted to use for automation from back in the day. It makes it absolutely vital. But I think that there can be an amount of legwork that can be, we've seen folks get, make advances on with developing the important business services framework and the dependencies and making sure they know sort of all the applications that support an IBS, the sort of the how to build the sort of build back in alignment, the sort of not the spotting the weakest links. I've seen some good advances on that with using capability. And then just by me, but I think Tanya, your stuff was spot on. I felt I think on the governance framework because I think that there's a of concerns in the industry around that sort of insurance being aligned with capability. You see great strides being made in terms of agentic coding and but often that's generating millions of lines of code that maybe folks haven't spent much time on and that could behave in a very different way under certain stress. So how you sort of keep that in line, think is a super critical. I think it moving like what Bill said earlier, sort of the using the agentic capability at the moment, which I see across a number of clients where it really is that analytical assistant still versus truly influencing initiating actions because the human in loop really is required still because of the non deterministic nature of some of those actions. But I think that what you both highlight in there is a very useful effort for guests. Okay, so moving on to panel question two, which is what is the practical starting point for using AI agents, in resilience? And on this one, maybe I'll start again, with yourself, Bill, and come to Tania. Yeah. Sure. So, you know, I I guess I'll put it in context of I've implemented a few g r c tools, you know, when those were hot. Right? And so I think the biggest mistake there that people made were getting ahead of themselves. The biggest mistake you can make with AI is trying to automate before your Doctor or BCP or whatever you you want resiliency before you really understand your operational resilience landscape. Right? You need to know what you're working with. You just can't throw a tool at it. So you have to do the hard work of mapping your IBSs or your critical business services or whatever you're calling it. But you have to know that you have to map it to your apps. You have to map your apps to all your IT service chains. Know exactly what if something goes down, what applications and what business functions are going to be impacted. And before you do all that, an AI agent, once you have that baseline, an AI agent can actually help you react to an incident faster. But the starting point is doing your homework. Right? Doing the hard work. It it it... There's no shortcuts to to get this thing going. And it's... I call it the slow crawl your use case. Start with just informing, right, unlikely disaster recovery business continuity, items before before you actually go live. Right? And know your root causes or your failure point reporting. Ensure that humans in the loop to validate all decisions. Like, don't let... Don't go out of the gate and let your AI agent make decisions. Right? Know where your kill switch is. Know where, you know, it's going to come to you for a decision or recommendation, and know know know that you've got a harness on there to stop it. Right? Because there are cases. And I was at a conference back in, I think, August, the NYSACA conference, where ASISO was speaking about two twin agents that were doing some operational payment processing. And one agent said to the other, you know, blank, I found a way out. Right? So you don't want that situation. So make sure that you put the right harnesses on there and, always keep a human in the loop. In fact, I think from the financial services conference I went to at Loyola last May, at Loyola, AI and financial services, the head of AI for Northern Trust actually used the term that was new to me, a human outside the loop. Right? Make sure you have somebody outside the loop looking at this as well, to make sure that, you know, you... You're not too desensitized to what you're building. So I think that this... You know, going back to where I started, the practical starting point, just know know what you want to address and do the do the hard work to get there. Thanks Bill, great insights. And Tania, if wanna add? Yeah, for sure. So, and I think, the key there is, and again, it seems like another theme, but, we don't need to automate everything, to get meaningful value from AI. And so crawl before you walk, or you've got to slow down to speed up, there's all kinds of adages that we use. But sometimes the greatest initial benefit is, giving people just the better information, faster so we can make better decisions. And again, maybe because my bias is that as a risk officer, I start with the service and the risk. So understand what service we're trying to protect, the importance of that service, so the risk assessment of that service, its impact, its dependencies of the service, its recovery objective, impact if it's disrupted. So similar to what Bill was saying, that disaster recovery and the business continuity plans, those things have to be discussed and in place before you throw a tool or an agent on it, or in parallel tandem as quickly as possible, it shouldn't come after. Secondly, inventory, all your AI agents that are already in use. So any embedded capabilities in your third party platforms, and know what capabilities, what their purpose is, who their owners are, which data, access, which data can they access within your environment, which systems are connected because that's going to inform, the context for the outputs, like for the decisions or for the recommendations and actions of the agent. And then similar to what Bill said, start with a really bounded, low consequence use cases and builds from there. So for example, in a payments environment, an AI agent could help consolidate incident information, across the technology, across operations because it has access to all of those data points and systems, and you would know that because you've inventoried and you've classified all of that. And then it can identify potentially, dependencies, prepare your incident briefing for your response team, and give a lot of value to those people that are involved. And then once you're testing that and the reliability of that, you can start giving it a little bit more power or authority to take actions. But the important thing is that autonomy should be given, based on the evidence and based on your testing, and that feedback loop regularly to mature your process, but also to mature your use of the tool that you're using. Yes, Diane, I think that's spot on as well. And I think from both of your pieces of feedback, I think it is absolutely right that the sort of the core framework of preparedness of the sort of systems, tools, processes, having that all sort of set out and have a better maturity framework there allows you to take the right starting point in AI agents, so to speak. So there's a bit of pre work I think. And I'd agree with that in terms of being able then to take the next step in terms of AI. And it reminds me of a point my CTO made, which is often you get a wave, a new wave of technology and that wave makes people address the prior wave properly. And I think it's often I've seen clients that then have gone back and kind of sorted out sort of having very modular controlled infrastructures code scripts, all of their automation set out that then and the sort of the auth and permissions that things can have in different domains and it not being sort of as widespread. I think that those pieces and frameworks then allow the next wave of AI agents then to be used in a much better way because you still have that sort of a worrying framework of where it might forget something and then you remind it and it goes, hey, absolutely right. Why didn't I think of that? But I feel like humans always did that anyway. And we always had capability in our systems not to sort of trust humans enormously out of the gate. We had checks and balances and I think you're both talking about those right checks and balances to sort of move forward with AI agents and resilience. So it's really appreciated both of those points. Now we have another poll. So this poll again, I'll read out twice give folks in the audience time to respond. So the question is, where is your organization today with Agendaq AI in disaster recovery execution and how is it governed? So the question, where is your organization today with Agendaq AI in disaster recovery execution and how is it governed? And the options are a, not yet exploring it, b, exploring with no defined governance yet, C, testing the agent acting on recovery steps with oversight and decision rights being defined or D, using it in live recovery workflows with defined controls and audit drills. So just going over those options again. A, not yet exploring it. B, exploring, but with no defined governance yet. C, testing the agent acting on recovery steps with oversight and decision rights being defined. And D, using it in live recovery workflows with defined controls and audit trails. Let's look at the response. So here I think in line with I think the conversation so far in the webinar folks in terms of actual execution or the majority are not yet exploring it for all the reasons. I think that our folks, our guests have mentioned a 55% there and the next best exploring it with no defined governance yet and then to get to it. And then we have 605% on folks are testing it or using it in live recovery respectively. Super interesting data and really appreciate folks sharing that. So final question three is how can financial institutions ensure there's adequate human oversight with this technology? And for this one, I'll come to Tania first and then Bill. Great. And this is an exciting question. We've been, in with my industry peers, been talking about this for a while, ever since we, you know, with the, speed at which AI and AgendaK AI and all these tools are rolling out, you start to hear, oh, we have more capacity, where, you know, just just throw... Just make sure you throw a human in the loop. So the human in the loop has become the control description, but, now we're really seeing that the human in the loop only counts if the human has information, if we have the actual authority, if we have the time to actually understand the recommendation being given to us, and that we can demonstrate, that the effective challenge, of that human in the loop is real, was real and was meaningful. So the human in the loop cannot be just a rubber stamp. And I think, the most important thing, again, back to that risk proportionality, is that the oversight should be proportionate to the consequences of the action, to the importance of the function that you're trying to protect, to the activity you're trying to protect, the service you're trying to protect, and the consequence of the action. And then in terms of a graduated authority model, you know, obviously the first one that we've been talking about is really basic observe and summarize kind of capabilities, a diagnosis maybe capabilities, lots of really cool stuff where the AI can say, you know, you... You've, in a previous incident, this has happened and you've acted in this way. You may wanna consider what happened here. And so that's really great insight on the spot in a timely manner, and give some recommendations on where to look because it is so connected, across your organization. But then the more advanced, capabilities like the executing, maybe executing only with approval and in really carefully defined circumstances executing within pre approved guardrails. So proportioning the capabilities and the oversight, is useful to be able to reap the rewards of the efficiencies that your AI... You don't want to over architecture your oversight because then you're kind of, not maximizing the use of this... Power of Vagintic AI, so and AI in general. I I think the... We, in terms of the the rubber stamping, because that's a really important a really important question and assessment that regulators are paying attention to now, is that is there evidence that the person approving or that that human in the loop had relevant context, had a clear explanation? Did they... Of the proposed action, did they have enough time in a practical way to reject or override it? Are we monitoring approval times to see how quickly it was done and was... Is it possible that in thirty seconds you were able to review the context? Are we looking for other indicators of meaningful approval? So that regulators in terms of a testing environment are able to assess the meaningfulness. Is it realistic that your human in the loop had time to assess this because your timestamp here says that they made this decision in thirty seconds. So so... And so that kind of stuff we never thought about before, those new metrics that, from a risk perspective, maybe maybe now on the the the regulators radar and also on your, you know, your third your your third line in your auditor's radar. So I think the test is not whether a human is technically in the loop, it's whether the human is genuinely in control of the use of the tool. Thanks, Tanya. Super insights there. Lot lot of provocative thought. And Bill, your thoughts on that? Yeah. And I think, Tanya, you're absolutely right on the governance. That that that piece of of making the human oversight has to be there. There's no question that the regulators... That's where they're going to kinda really start to put the microscope. But kind of on a more tactical level, I think you need to treat your AI tool like a privileged access account. Right? You need to show authentication. You need to show the logging and monitoring. You make make it auditable. The whole chain of custody of identification to recommendation and action taken has to be captured and stored. And I would say it should be stored with immutabilities. Maybe think about storing it in worm format, right, so that you can't override it eventually. That that ensures you're staying true to the to the legacy, you know, confidentiality, integrity, and availability principle. So the... Some of these principles that are... Have been out there for years apply to this framework still. I I think that you have to look at your critical services because these are areas where you absolutely have to have a human intervention. And these are areas where you don't want the AI to make decisions without a human there to do that. And I mentioned the human outside loop. This is where you wanna put that. Like, your critical services or or your I b s's, if you're going to put an agent on top of it, you need to have somebody besides the person who is owning the the tool validate the decisions are right. Now, whether it's real time or not, I think you have to make that your own decision point. The real time decision belongs to the owner, the first line. The second line is probably somebody who comes in and overlooks it. But you just have to make sure you show that monitoring. I think you have to work with your teams like Tanya on complaint management. Right? If AI service customer service agents are out there, you need to look at your customer service complaints to make sure that you're not seeing a trend. Right? I mean, you should be doing that anyways in a financial institution, but with with AI taking over a lot of your customer service points, this is an important piece to have that human oversight. And then finally, you you need to really understand your harnesses. Work with your information security team or whoever is putting these harnesses in place to make sure that AI is not trying to find a way to bypass controls and start making decisions a human needs to make. So I think that as long as you kinda start to take these principles, you know, whether it's NIST or whatever whatever principles that FFIEC and apply them into your framework and make sure that you go to governance with this. You know, AI is not its own govern... It it kind of is its own governance stack, so to speak, but it it feeds into your enterprise risk management oversight. And this is another element to human oversight that you're giving transparency up to your e f... Enterprise risk committees and to the board, what you're doing, how you're doing it and how effective it is. Great insights. I loved your point earlier, Tanya, around the human loop. It's not a rubber stamp. You can't just say, hey, there's a human loop, we're all okay. I think you've got that sort of sense of what is that sort of human, backstopping, there. You know, we a lot of our systems previously, we'd have four eyes checks and wider when it was pre AI, we wouldn't often just rely on the being a sort of single human, because of the, just as you say, the criticality of the decision, in that point of the process, that was super important. So you need more than that. So I think be not just being able to say, hey, tick human in the loop here, it'll all be okay. We have to sort of make sure those points have the appropriate capacity and capability behind them. Because I think it's often a case now when these processes are seen to move very fast and folks have been approving things by the click, you get a bit of click momentum when maybe that this point needed a bit more review and you can easily make a mistake. So I think that that point very useful insight for our guests. And I think also in terms of what sort these agents are doing, it's important also that you're not getting a perspective and putting more and more into a sort of a dark agent that's taken all your data and generalizing and training on it. I think how to think about that within the control framework is important as well for what answers you get back on it. I thought the rubber stamping point was very useful. And I think Bill, your point on this or human outside the loop, think comes back in it, do it again and sort of chimes in around that component. I think that again about the speed, as you say, when these processes are operating super fast, you've still got to have the ability for that control and people not to feel like they can't put their hand up to slow something down and I won't stand with the regulator when you've got to spend a bit more time at this point and it get down to human speed instead of machine speed considering that risk. So yeah, great points. Thank you. Thank you both for sharing. And this is a this is a very, very interesting stuff. So I think we come to question four here, which is about how are evolving global regulations changing expectations on the governance of AI, within resilience and Doctor. And here, I wouldn't mind coming, like last time to, Tanya, you first, then Bill in terms of sharing insights. Okay, great. Yeah, and in this area, it's interesting, because the current landscape is not one of harmonized rules for AI agents. There's a direction of existing expectations across operational resilience in general. So if you think about the, you know, maybe a decade back when operational resilience, made its more, main stage entrance, you know, in The UK and then in Canada under OSFI and now under different regimes in in... With the business... With just banking as a service and more open open, ecosystems. There's already expectations around what operational resiliency is, what technology and cyber risk and mitigation strategies and third party risk management models, accountabilities. And that's all still relevant, and even more so now when AI influences critical services. So the the practical expectation and approach from regulate... Regulators, especially in Canada and and, the EU, UK, EU with DORA, and some additional guidelines alongside DORA has... Is is converging on outcomes. So what are the outcomes that we wanna achieve and not so much, having a standalone or separate AI rule book. It's more about embedding AI into these new, updated stress case scenarios, use cases in your stress testing, and continuing to understand where AI is being used, the dependencies and the risks, ownership of that and establishing the controls that are proportionate and putting that into your operational resiliency and decisioning. In Canada, have, so the banks, for example, which is a good best practice since the banks are the prudential regulator, OSFI is a prudential regulator for banking in Canada. They put out a new updated operational risk management guideline, an updated guideline, not a new separate AI guideline, but an updated guideline that embeds now the expectation to update your use cases and your scenario testing to incorporate these things. The EU also issued some guidance earlier this year in the summer, but again, to go alongside their DORA framework, so it's their Op Resiliency framework. In The United States, there's some more fragmented, and Bill can probably speak better to that. But again, it's, focused on keeping the broader risk management and governance practices. So I think the regulatory message is less about let's wait and see what AI agent rule books may come. I don't think you should be waiting and seeing, what regulation is gonna say. I think it applies right now. We have established risk and resilience, disciplines and frameworks, and now we should be embedding, these new tools, these new processes, these new controls into those existing frameworks to create new scenarios to test those things to stress that and to take action accordingly. Fantastic stuff, Tim. Yeah, a lot of food for thought, again. And I wanted to, Bill, if you want to add to that. Yeah, sure. Kind of building off what you're saying in The US, like, you know, if you look at the CFPB, for example, I don't see them coming out with an AI, you know, review. What I do see them doing is embedding AI into a UDAP review or or a, you know, HMDA review with the OCC. Wherever you're going to have this AI tool deployed, they're going to embed that into their existing review process. And so, you know, with with the the changing of the guard, so to speak, in the house, in the senate, I do expect to see more enforcement through the CFPB again. You know, they were kinda cut off at the knees, but I do expect that that's going to come back. And any bank, I think, that, you know, has been on what I would call a little bit of a reprieve for the last few years knew that this would happen. And so if, you know, I don't think any compliance department has taken their foot off the pedal, so to speak, but we did get a little bit of reprieve. So I would expect that to kind of come back. But the regulatory direction is pretty clear. It's not just another technology. If it's going to influence your Doctor plan or your critical service restoration, they expect it to be governed. And it's part of your operas framework. They expect in some cases, and and I think this happened with, The UK and, about ten years ago with their data privacy laws that came out, they expected a named executive. Right? I think that the the chief data officer at HSBC had to be named and given off to the PRA. And so, you know, that was... That's... That becomes personal accountability. Right? Because they are personally accountable. And I think in The UK, at least at the time, there was personal accountability that you could get fined or even worse. So I think that, you know, that could be the case more in The EU than The US. But nonetheless, I do feel like that, you know, they're they're going to start tightening the reins a little bit around AI from a regulatory perspective. You know, I think a lot of people have some examples that they can just leverage in their organization. You know, if you have a model risk governance process, you have a platform you can build off of. The principles are pretty close to being the same. You have to know what the data is being consumed, you know, how it's being protected. You have to understand the chain of custody, how it will be used, who's responsible for it, who's monitoring it, who's logging it, are you... Is it subject to periodic reviews? These are all things that you have to do in the model risk space. So if you're not sure where to start, take a look at your model risk process. It would probably largely cover a lot of what you need to cover. And and it and it is, you know, a somewhat repeatable process that you can leverage. But you are going to need some sort of AI governance as well. Right? So something where your AI tools go up and then they they they feed into your enterprise risk management process. This is where all the use cases go. Right? And and and and all the, you know, information security, op risk, compliance, all the risk stacks have an opportunity there to take an opinion on what what this tool is going to do. And that's gonna round out your ability to put out an AI tool that you don't have to pull out dethread, so to speak, down the road. No. Thanks, Bill, for that. And I think, you know, I'd have conversations with execs in various other verticals and over the last sort of twelve to fifteen years. And there'd often be a thought that, hey, you know, financial services has got this sort of crazy regulatory framework for ensuring technology resilience, operational resilience. I don't have to sort of deal with that. I'm not as important as the sort of economy and payments, but I think increasingly now with the sort of how things are playing out in the wider environment, particularly relating to cyber risk. I think folks look to the regulation more and more of financial services and think, hey, that'd be a pretty useful framework that I'd want to adopt and thinking that the regulations make sense and that level of maturity is something to aspire to. I've certainly seen that in like retail and other functions coming forward. So I think the sort of the awareness and respect for those core capabilities is increasing more and more. And I think what you've both set out there for how AI is being considered within that is and you look at sort of, I think the classic speech or talk that Pat Obert gave at RSA on meantime to adapt of how sort of AI is super important for how we might use it in resilience because it's the primary weapon being used by the the sort of attackers. Just brings the sort of your your conversations that you both shared there and insights on the ability to use AI and resilience in Doctor right to the forefront. I think of many conversations that I'm seeing happening in the marketplace today. So thanks for sharing on both of you on that particular question, very pertinent. So question five is how can risk compliance technology and resilience work together to challenge, test and assure AI enabled, resilience strategies. And for this one, I wondered if I would come to you first Bill and then Tanya. Sure. Sorry, I just lost my screen here. No worries. So I think that... Yeah. Go ahead. No. I think the one such thing is don't don't create silos, in your in your organization. I think it's important that you have a resilience program that's governed through multiple multidisciplinary assurance models, part of your overarching enterprise risk management solution. You know, the example I got is I recall the push for digital banking years ago. Right? And digital risk was its own risk stack, and we're going to create its own risk stack. Well, what you really did is created a shadow op risk governance process around it. If... You know, what we found out is we needed to merge it into the overall operational risk framework. And and that's important in order to keep your, I guess, you know, strategies and and and your governance aligned. You know, you need to be able to test your AI. And and how you do that is up to you, but, you know, you could have internal audit do it. You could have an insurance function do it. But mostly, it's probably an IT audit where you test technical failure points. You test access control, kill switch tests. Know, resiliency can run run their scenarios and and make sure that they can withstand their impact tolerances. You need the independent challenge. You need your second line partner there. I think compliance in Tanya's world, you know, they confirm that our decisions, approvals, and the agent actions are traceable so that they're auditable. You need to monitor all your decisions and especially as I mentioned earlier in the complaint space. And then you also need to kind of probably include two to three, maybe four operating models that are are in place and, you know, test a normal operation, test it if it's degraded, and then test if it's... You just totally lose your AI capability. And and that last test is gonna be able to tell you if you can recover if it's unavailable. What happens if your agents go down? Who's going to pick up that work? Right? Do you have a BCP plan to pick up in a BIA that says, okay, in the event that this customer service agent goes down, which previously took, you know, maybe 10 people or half of a department, how are you going to kind of reroute that? And so these are all the things that came to mind for me. And you want to make sure that you are building a resilient AI process, I guess you could say. So, yeah, I I I think I put a lot on on the strategy, on resilience to be able to withstand its own kind of loss of functionality with an AI agent. Very useful, Bill. Thank you. I've got a few good few thoughts on that. Tanya? Yeah. I think consistent with Bill very much because this is how the kind of the industry and disciplines, the risk management layer has evolved over time. We always have a tendency, any new technology or new innovation, develop a separate stack or a risk discipline, a separate guideline, and then it becomes siloed and then the pendulum swings again and we say, oh, we're all stepping on each other's toes. You know, it was the three lines of defense model and everyone's like, well, what are these guys doing if we're doing that and why are we all doing the same thing? And then of course there's... So the pendulum is always swinging for sure. I think it's definitely now converging again back to business outcomes. Business outcomes, control outcomes, is always what we come back to. What are we trying to achieve? Who are the stakeholders or players in that, decisioning? And and how do we share, inputs and insights across these different areas? So across the risk team, the compliance team, technology resilience. Ultimately, the business and technology still are the owners and remain responsible for the services, the technology that's rolling, the agents and the controls, the day to day operations. So in the day to day operations, ultimately, they're still responsible for it. So they've got, you know, technology is monitoring it, technology is raising any issues, any issues with recommendations because they're the SMEs to that second line for assessment. Compliance is assessing potentially any regulatory breaches, anything that's coming outside of the perimeter, the regulatory perimeter. But risk is really putting it into, perspective and saying, well, here's our heat map and let's put this into perspective and technology is helping to find solutions to bring that risk back down into kind of an area, into some space that the business is comfortable with. Again, going back to business critical functions, so important business services. Like we don't want some, we want to maintain CIA or the confidentiality, integrity, authenticity of accessibility of our customers' data and information and assets. So it all has to tie together and the less duplication and GRCs are great for that. Technology and AI is really where it's really pushed, our practices into a more efficient streamlined approach where you can identify all your stakeholders, their roles and responsibilities, they can be tagged in decisioning, you can document that rationale. And then in the testing of that, we can then go and into the systems and into the evidence of the artifacts to test, what was used, to test the assumptions that were made in all of the use case scenarios or the use cases that Bill just mentioned in normal operations, in slightly degraded operations or in full, systems down. So that AI is really bringing us the opportunity to streamline that process, but ultimately it comes back to identifying the roles and responsibilities of each of these stakeholders in each of these areas. What is their function? Tagging them in the proper flow, having information sharing openly. So people that are marked with or tagged with making a decision or have some responsibility have the right access in a timely manner, and we can later go back and test that. And you can test that in tabletop exercises or internal audit, etcetera, going back to the data that was there when they made the decision. I think one of the, things that we always find is when we're if we're in an incident and we discover something new, like an agent or something new or AI usage, and we didn't know that that even existed as a capability, that's probably too late. We should have... Compliance or risk should probably have known about this, earlier, because now we're dealing with two things. We're trying to, resolve an incident, and we're trying to figure out what the heck is this new... Like, what is this giving us? What is that? It's too late. And so let's... So I think, early early integration is best, and integrating all of these different teams is best. Thanks for that, Tanya. Think that, so three bits that it's in chain with me. Think Bill, when you said the sort of the, what if you lose the AI capability? I can remember working with a significant G SIB on the not only did it lose the AI capability, but in the perspective of AI in this sort of framework, what if they lost the underlying automation because a number of cases you've moved forward with an operating model where maybe you could not have enough hands on keyboards to sort of do it back the old way. So looking at scenarios like that, I think is super important. And I think as Tania you said, the sort of real testing in scenarios is really important. It isn't just a, hey, we tested it in a fully supported situation for like one scenario of a full data center or cloud region fail... Failure. You've got some real capability to use these things. And maybe that also sort of pushes out the ability to sort of make sure you aren't finding out about the new things at the last minute in the incident, which is kind of way too late. So yeah, thanks for the good insights on that question. So if we come to question six now, which is last prepared question, I think we don't have a lot of time left, but I don't want to shorten answers. This question is very important. We might have time for one or two submitted Q and A questions from the audience. But for this question is how can resilience professionals ensure there are the correct audit trails to make sure that AI agents are working within risk parameters? And maybe if I come to you first, Tanja, then Bill on this. Sure. Yeah, I think this is a nice way to end it because it summarizes everything that we've been talking about throughout the questions. But if we think about an outage affecting, and in my space right now, digital assets or fintech, fintech, we should be able to produce a reliable timeline showing, what the agent observed, what it recommended, whether a person approved or overwrote. So we we should be able to reconstruct that timeline. That's really the principle of the spirit at which we think about risk parameters. In this case, it's not just logging it. It's not just the volume of the logs, but it's the reconstructability and the evidence of it. Things like some of the things that really we may not think about in the moment, but that have become very material. At a minimum, capturing the agent and the model version is important. What... Because we know what goes into the agent. We know what goes into the box. We know what comes out of the box. Sometimes we don't know what's happening in the box. So to the extent that we know which version of this tool or of this, decisioning we used, will be helpful in that in that validation. The timestamp, the session identifier, the relevant prompts or instructions, because that gives us the context, like what did you put in? What question did you put in when you, you know, to get this output? Because that helps with understanding, the decisioning and the recommendations, data sources, the rationales, etcetera. So being able to construct that timeline, being able to evidence those kinds of pieces, and there's a long list of other items that, would probably need a separate session to go on the practical building of, you know, the risk parameters and which maybe we could do in a separate session. But all of those things should be built out so we understand what are the critical... Here here are all the things that our tools will record, so that we can reconstruct. But the spirit of it is how do we reconstruct this and and have the artifact to show and to and to do this again. Thanks, Tanya. Super useful. And, Bill, your thoughts? Yeah. You know, what Tanya said, you have to make sure that audit trail captures more than what just the AI agent did. It it has to allow you to reconstruct, has to allow you to... You have to be able to rely on the information that's using, whether where the human intervene. But, yeah, as you really do need to know what the agent's approved operating envelope is. Right? And so you have to define the systems, the data, the transaction limits, let's say, what should be monitored and logged. You you want the full chain of of of custody captured. And this is where I said maybe you wanna consider worm storage because if the AI AI agent can go rogue, let's say, and start changing the the log files, then you've lost all the integrity that, that you need, to to audit and to prove that, know, what the AI was supposed to do, they did. And so, you know, I, you know, I wrote down a number of things that should be, captured, and I think, Tanya, you mentioned a few of these, the agent model version, the date and time of the session, all the prompt instructions that were used, systems and tools that it used to access and make decisions with, if it's any recommendations, what his confidence level was, any action attempted in the actual result, if there was a policy or recovery playbook applied and what that was and what it pulled from it, where the human approver override or rejection happened, and then any exceptions and escalated activity that maybe it needed a human to come in and approve. That log integrity is important. If you don't have it, you're you're... You won't be able to use that agent once internal audit does the review. They they would probably put a pause on it. There should be some real time monitoring depending on your use case. If it's a critical, application or IBS, use of unapproved data is something that you want to know that the agent is using. You want to know if there was an attempted to bypass a control that should be logged. Any actions outside of a permitted time window, any excessive attempts, just like you're trying to log, we have controls when people try to log in the third or fourth, fifth attempt, right? They say, no, it's time. You know, you have to wait twenty four hours before you can try to sign in again. If an agent is trying too many excessive attempts to get something and it might tell you it's not authorized. So, kind of a summary, you need to know what the agent did, what did it know when it made the decision or acted, who authorized it or how it was authorized, what was the result of the action, was it expected or not expected? And being able to reconstruct those answers, especially after an incident, if you can't do that, then you don't have an effective oversight. And we have to be able to know how much logging and and that we collect as well because not everything in the log, I would say, will give you what you need to know. So instead of writing everything out to a log, just decide what you really need so that you can make better decisions faster with the information that it presents. Because, you know, just the opposite of too little information, too much information, you won't be able to make a, you know, a timely decision.