Kristy Hornland: [00:00:00] I've definitely talked to organizations that have made it where if something gets open, then everything shuts down.

Vijay Bolina: We're at an inflection point where we're going to see more and more code being written by models. The way that we think about this at Google is that okay, look, if we have these systems that we expect to write more code for us, how do we ensure that they're writing safe and secure code?

Jason Clinton: I think one of the lessons that I've learned the hard way is making sure for every vendor you have that you've subscribed to the notification. That the terms of service have been updated as you just sometimes get surprised.

Ashish Rajan: If you are someone who's looking into the AI cybersecurity space and wondering where is everything at the moment in terms of what does the next three years look like from an AI perspective, is AI going to talk to itself?

It's really true and spoiler alert it is not true. The other thing you would also might be wondering in terms of building a cybersecurity program, what are some of the things you should be starting off at for this conversation? We had Jason Clinton, who is a CISO for Anthropic, which is the company behind Claude and some of the other projects that they've been running.

We also had Vijay Bolina, who [00:01:00] is a CISO for Google DeepMind. We had Kristy Hornland who is a cybersecurity director for K-P-M-G-U-S-A, and of course you favorite two hosts Caleb Sima and Ashish Rajan. We wanted to have this conversation from a lens of, where is this going with AI in 2024? As you sit today here in Black Hat 2024 in the USA and talking to all these people at the CISO summit , at AI Summit, what is the big takeaway for what is the state of cybersecurity?

And this episode is jam packed with where is it going at the moment? What you should be looking out for, and it's truly real. And anything else you might wanna consider for putting into your plan for the next 3-4 years as you start replacing things with AI slowly. Yes, it is going to happen that some functionalities would start evolving and start being replaced by AI.

What is going to be and how much it would impact your job, you probably want to listen to this episode. If you know someone who's probably building a cybersecurity program with an AI mindset or trying to see how AI can be used for cybersecurity with cybersecurity, this is the episode. Please do share this episode [00:02:00] with your friend or colleague who's also researching or looking into this area.

And if you're here for a second or third time, and you've been finding AI cybersecurity content valuable, I would really appreciate if you can drop us a follow or subscribe if you're watching this on a video platform like YouTube If you're listening to this on an audio platform like Spotify or iTunes, definitely drop us a review or rating.

It definitely helps more people find out about us. Welcome to AI Cybersecurity Podcast.. This is the State of AI Security episode. We have a lot of people around us. I'm going to start with the intros of the non host people first, Kristy, just want to give an intro about yourself.

Kristy Hornland: Yeah, thank you for having me. So my name is Kristy Hornland

I am a director at KPMG in the U. S. and I'm also the lead for the AI Security Working Group for the Global Resilience Federation. As well as working with clients today to really focus on how do you stand up responsible and secure AI within your organization and all the implications. So excited to join the conversation.

Jason Clinton: I'm Jason Clinton. I'm the CISO at Anthropic. I've been there for a little over a year and 12 years at Google before that, and four startups before that. It came from a background of defending against [00:03:00] nation States. And at Anthropic, I'm thinking a lot about how do we protect the model weights and how do we lock down the security environments, so the customer data is protected.

Vijay Bolina: Hey, I'm Vijay. I'm the CISO at Google DeepMind. I've been there for about four years. I lead a research and engineering group that thinks about the emerging risks around the frontier systems that we're building both on the tech and defense side of things.

But we are also involved and partner greatly with the broader Google security team when we think about protecting our intellectual property and some of the things that we're working on.

Caleb Sima: I'm assuming I don't have to do an intro myself, since my name is on the podcast, but, but I could do one. Everyone knows Caleb anyway, so I think but I maybe my name is on the podcast. ,

Ashish Rajan: I think he called our frontier model. Maybe a good place to start talking about the state of AI security is by defining a few fundamental things so we don't lose the audience. Yeah, most of the conversation that some of us had at AI summit and CISO summit, seems like there's a lot of opportunity for people to still learn about AI, [00:04:00] even though we know it's not just ChatGPT or Claude and there's more than that.

Caleb Sima: Some people call it frontier model

I've also heard foundational model. Yeah. What's the difference between the two? Are they the same? And they're just different, like also help go into that. Those two,

Jason Clinton: I think the foundation and frontier did interchanged a lot. And at this point from a nomenclature perspective, there's probably no difference, I think from a empirical perspective, the EU AI act defined models of interest of as 10 to the 25th.

For loading point operations that have gone into training that model and the White House Executive Order defined it as 10 to the 26th. So a much bigger 10 times larger. So when we talk about the leading edge, the frontier, the foundation, we're talking about models that are in that class and the reason we use the amount of FLOP that have gone into the model as a benchmark is we in the industry assume that the larger the model, the more compute that's gone into it.

The more intelligent it's going to be. So this is called the scaling laws hypothesis. And it's the foundation of the entire [00:05:00] industry's investment in this area. And if you just assume that as more compute goes in, it gets more intelligent. Then you just add more compute, you add more data, you add more data centers, you get more chips and you do all of these things and you will get a more intelligent model.

Now, some people disagree with the scaling loss hypothesis, but so far the empirical data is that more compute does equal more intelligence. equal more intelligence.

Caleb Sima: So anything above 10 to the 26, yeah, is now called a frontier and or foundational model. Yes. And who's on that list?

Jason Clinton: That, that is a closely held secret, I think right now. And so I don't know if there's a complete and comprehensive list of who's in that class, but you can look around and guess based on who's making headlines. I would think that

Caleb Sima: out of the normal people It's Open AI, it's Anthropic it's DeepMind these are all would be considered.

Jason Clinton: Yeah. I think just like working backwards from the word foundation and frontier, I would say that definitely those more powerful so

Ashish Rajan: would you describe them as the same thing? Because a lot of people make, say, Hugging Face, a lot of open weights model, open weight model as well. Because I guess, can I use that same [00:06:00] analogy over there as well, or are they,

Jason Clinton: I would frankly say that Lama 3 405 billion is a very impressive model and it's definitely at the leading edge and so they've done amazing work there and very congrats to the team for the work that they've done.

So absolutely, I would include them.

Ashish Rajan: Awesome. If you guys want to come on the podcast, definitely. I'm sure Mark Zuckerberg would come in any time to talk about Llama and how amazing it is. Because I think the list is like wishlist. The reason I asked about that was because I touched on the whole open and closed model as well.

I've mentioned Hugging Face. People may not even know what Hugging Face is. Is it worthwhile putting in the whole open versus closed model as well for you guys? Do you guys actually see that as a part of the ecosystem of AI that we are building and growing with?

Vijay Bolina: Yeah, absolutely. Google, obviously, we have our open source model as well, Gemma which is quite performant, even at smaller sizes.

And so we think it's part of the ecosystem and where and how it fits is still something that we're all exploring as an industry, right? I think people are finding quite interesting use cases [00:07:00] when it comes to smaller, more tractable models that can be deployed in more everyday settings. And those things are relevant and important.

And then on the frontier side, I think people are still pushing the limits on those as well too. We're seeing larger and more capable frontier models that are going to be open models as well too, right? And frontier, Jason did a great job explaining frontier models. I think one thing I would maybe add is also capabilities.

So oftentimes frontier models are also introducing new capabilities that are emerging and or consider not maybe standardized or part of, existing models that are already out there. There's some speculation of what could come out in the next few months. So I won't get into the technical details, of what we think maybe on the frontier on the capability side.

But you often hear about some of these things in conversations as well. So it's not just pushing larger models that are, bound to some FLOP compute metric, but also capabilities.

Jason Clinton: By the way, Gemini's ability to, for example, to like process very large video and it's to a million context window is an [00:08:00] impressive feature as well.

So it's a good example of a capability.

Caleb Sima: Yeah. Is there a difference? So now when we say the definition of this is based off of compute and FLOP. Does that make a difference on modality?

Jason Clinton: It does. So one thing that you can think about when you're talking about multi modality is that during the training run, you have to split up the kinds of input data that are being pushed into the neural network.

And in a multimodal model, if you're just doing text as an example, then you have more opportunity to train that text modality to be more intelligent. Now some have argued and there's been some empirical evidence that if you train across modalities that you're actually bootstrapping a wider breadth of understanding of the real world, but there's also a failure mode or maybe a specialization where the network splits into modalities and the modalities intelligence doesn't cross connect.

So it might be really good at image processing. But none of the intelligence from that network also comes over to the text processing network. That's one thing that you have to watch out for in the training process and is definitely an area of academic interest for sure.

Ashish Rajan: Oh, so [00:09:00] people who are trying to use this in their organization, probably worthwhile calling out a lot of people would hear multi modal and go, Oh, I have to now protect myself or my organization against text, against video, against audio.

Yeah. It also depends on the model that they're using behind the scene, because that may just not be the best model to work with.

Jason Clinton: Absolutely. Yeah. You should, in every case of whatever you're doing, whether it's text or multimodal input, do an evaluation for your specific use case and find out like, does this model perform very well on the thing that I'm trying to do and some models are stronger at certain kinds of tasks than others.

Caleb Sima: Which by the way, I would love to segway that into when we talk about the state of security in AI today, everybody is thinking about LLMs and text. And when you go look at the vendors that are providing security solutions or all the people who are saying they do security, they're all looking at models in its current form of text in and text out. And then when you start even talking to some of these things about Hey, in two years, won't we just be talking to these people like avatars on video stream?

And, [00:10:00] or will these models communicate with each other in their own language versus even English or text as their own protocol that they determine is most efficient?

How do you deal? Then with the security aspects of that coming in the future have anybody or you guys ever heard anything about that? Because basically when I ask those questions, nobody has any answers

Kristy Hornland: From a question perspective a lot of organizations aren't asking the question so much into the lens of multimodal.

It's really from the perspective of hey, we are just looking from our first kind of line of sight into that text to text use case. But really from when they ask that question, my question back is, do you have any idea of the lineage? So I know that we talk about like the principles of transparency in a lot of this, and that's such a complicated topic in itself that people are not necessarily, I wouldn't say that we're at a standardized point where you already have an answer there.

So you introduce more complexity, which is great for innovation, but from the security standpoint, I think towards our organizations [00:11:00] thinking about their resiliency and some of the frustrations that may come from introducing more elements to this without answering some of that, more of the concepts of AI bill of materials to really understand what are some scenarios where even this, like simpler at the forefront, simpler, I say, but like text to text, we don't necessarily have that understanding of if this fails, what do we do to bring back.

What do we do if we're really

Caleb Sima: We're just failing even at the first basic. I can't even wrap my head around multimodal scenarios now,

Kristy Hornland: It's not that it's not worth exploring It's just that when people are asking that question my question back is really like foundationally Do you have visibility into that chain?

Because most organizations, when they come to talk, it doesn't really sound like we're seeing the same level of like resiliency planning that you would see for traditional application security.

Caleb Sima: Of course not, because we just need to push it out as fast as possible in order to get. Because I don't push AI now, my competitor will push AI, but we both can't figure out what to [00:12:00] do with it right now, but we're going to say we do.

We're going to try to push it out the door.

Ashish Rajan: Actually, I do have a question. What do you see customers talk about? Like we've been having a lot of conversations, on the AI cybersecurity podcast about what are the use cases that people are using it for. And somehow 90 percent of the time we've come up with the fact that a, it's either a chatbot or people are trying to figure out in dev environment what they are going to do.

Have there been any use cases that come out in the conversations you've been having in terms of what are people doing with AI? And I guess maybe. What kind of AI are they using? Are they multi modal to what you were saying?

Kristy Hornland: Even to the point that was raised earlier about which model are you using?

We've seen organizations step through that are, to your point, they're going to the lowest risk per se use case. So it is, we start with internal, we're looking at policy documents. It only is really surfaced around our employees. And that is not a, say particular decision point that would impact our business considerably, or if it failed, okay, worst case scenario, you now manually do this.

Caleb Sima: It helped me [00:13:00] write reports. Yeah. And that's what I'm using it for, which is fantastic.

Kristy Hornland: But at the end of the day, when they look at what are we using it for? It's Some of those scenarios they can then see, hey, maybe we look towards a model that wasn't exactly the right, was maybe us needing something actually much smaller.

And so even that in terms of now we move to retraining on a smaller model for us to achieve the same thing. That's been interesting and even there you start to think of, do we have any logging, do we have any archive of this now shifted here. We now, retired this.

Ashish Rajan: We were talking about this earlier, a lot of people are probably not even aware of how AI works or what they use.

Multimodal would be a, oh my God, did someone say multimodal? They were just googling over there, multimodal. When you talk to people who are trying to do this, and I'm curious across the board as well, are there any blind spots that you see people? Obviously, because we spoke about just now, you spoke about logging, resilience and all.

Is that even a question that people are thinking or it's more like, how do I block Open AI completely?

Kristy Hornland: I've definitely talked [00:14:00] to organizations that have made it where if something gets open, then everything shuts down, right? And they're like trying to teach a lesson. But I've also spoken to most organizations which are at a point where they're trying to do joint the efforts of grassroots to really say bring forward your ideas and how do you compliment that in a way that also regards responsible practice, which is now we go through the intake process, which is not fun for any, like it's all governance, but what you start to see is like maybe 80 percent of the use cases that people are proposing are actually just like RPA, like it's process automation, but why would you need a model to do what you're talking about?

Which isn't a bad thing. Now you've got all these enhancements to processes that you have. But whether or not they actually fit through as good use cases for AI. That's the,

Jason Clinton: I guess to address the question of blind spots, I would say the big one that comes to mind is the trust and safety aspect of model deployment and every place where you can deploy models today offers the ability to put guardrails on the way that the model is deployed so that you have that confidence that you can hand to [00:15:00] your GRC function.

Then that will circumscribe the failure mode to the model. So it's not going to engage in harmful behaviors in a way that's like at risk in your enterprise environment. So trust and safety is definitely one of those cases, but, to go back to the broader, like deployments, I think we have a range of listeners in the world to to a podcast like this, or some people who are just getting started and are looking at oh, a chatbot or a report writers where I'm getting started, I think like my perspective is that if you're thinking about that right now, the world is going to change pretty dramatically over the next three years with regard to AI. And if you want to skate to where the puck is going to be, you need to anticipate what is the world going to look like three years from now.

Just to preview some of the ways that, we know that models are going to be adopted inside enterprises and SOC automation is an obvious one. So if you're a CISO listening to this podcast and you want to get alerts, test a hypothesis that this is a breach. Use agents to suss out the evidence and go through logs and

Caleb Sima: then write a report

Jason Clinton: and hand it to a tier two.

Immediately, that's the obvious case. If you're doing third party risk management in an organization, you can scan all of [00:16:00] your dependencies, go through the GitHub repository history, look at the maintainer's rate of risk or associated with a certain dependency. You can integrate AI right now into your CI and CD pipeline and look for defects.

Google's done amazing work on this internally. You can look for vulnerabilities and I think I'll leave that to you, Vijay, to cover it on Project Naptime. I think that'd be really interesting to cover as well. If you're in compliance and you're writing questionnaires, or if you're reading questionnaire responses, And like looking for those little red flags that you want to look for in third party risk management.

All of that is already automatable. We're using that inside Anthropic to some extent. And then the obvious case and the most exciting one that's gone viral most recently on socials is making engineers incredibly more productive. One of my co workers broke his hand a couple months ago in a biking accident.

So he's a programmer. And he was able to switch to a voice input mode where he was writing 3, 000 lines of code per week without the ability to use his hand. He just, points the [00:17:00] mouse cursor at the section of the code that he wants to modify and says, Claude, make the change. And it wrote all of this code in a week, 3, 000 lines of code.

So this is happening right now. You're scared. If your enterprise has software engineers and you're not thinking about that right now, you need to be thinking about that.

Vijay Bolina: you want to make use of these models internally in a hyper specific way that is unique to your organization and your organization's enterprise data.

Oftentimes that requires you to then fine tune or domain adapt that model into your organization's specific use case. But one of the things that you lose is, authorization scopes of what users may actually have access to within your enterprise. And so you end up in a situation where you have this pretty useful model that's been trained on all the data within your organization, but you may now have a situation where that data is readily accessible to anyone that prompts a model.

And so you have to think about then is that what I really want? Is there some sensitive information that is, in fact, need to know, maybe? I think sometimes there's a [00:18:00] desire to explore and that's fine, but I think sometimes it's also important not sometimes it's also important to just better understand what the actual use case is that you're trying to optimize for before you slam in your entire enterprise's data lake into your fine tuning run.

Caleb Sima: Least privilege is a very hairy subject in general. And then when you apply that to AI and then when you apply that to the fact that, oh people are shoving all these things in these RAG. And then using it in this format and people don't understand and actually prompt injection still is a real problem.

That's right. In these scenarios. Yeah. I would also say another blind spot I think is about where you're using AI. I think there is a, Oh, is it an ethical or maybe best reasoning capabilities where if you're, you've heard the thing of, Oh, where AI is using it to look at resumes and identify the Yeah. ones that are most fitting to the job when they may have a lot of areas and prejudice that may not necessarily exist without, right? Yeah.

Vijay Bolina: Distributional bias is definitely a big area of concern as well too, right? If you're only looking at [00:19:00] resumes in the English language that have been optimized for a very specific region and that's what your model is trained to recognize and filter for that definitely and obviously is, swaying the bias towards that population or demograph then at that point. So you have to think about those nuances as well too, when you're adapting these models for your

Caleb Sima: Where is AI being used and what output value is it doing? And then Yeah. If it's making decisions and or filtering things for you, the bias is very, you have to look.

Yeah.

Vijay Bolina: And is the data well representative of the total population of users and or, the set of individuals that may be engaging with the system as well.

Caleb Sima: Yeah. And I'm going to get a little future. I'll even apply something to security. And this goes back to what actually you can talk more on. AI doing vulnerability assessments, right?

So there's been a lot of buzz around, you're, there's reports on Twitter. Oh, AI is almost better than humans now. Look at all the things, but actually it's very biased in the way that it's doing this. And so if you are now saying, [00:20:00] Okay, I can replace my pentest using AI actually it's not even close to reality, but that people are believing this because they are comparing it against a list of tests that are on here saying, look at all of the things.

Now, AI is checking that all that on the human test. It's reaching 100 percent and humans are reaching 95%. Clearly, AI must be better now. And so this is also, I think, a big blind spot where there's bias on, no, this is not true, right? This is the same way at which AI today is taking tests on models on, oh, I can pass an academic test.

Just because you can pass that doesn't mean you can reason and think Same way as this is not going to automate pentesters.

Vijay Bolina: The code safety space right now generally is pretty exciting when it comes to folks that are thinking about real world problems that AI can help solve. I think there's maybe two points that I'd like to talk a little bit about that aren't often spoken a lot about.

And the first point is this. Is that we're at a bit of an inflection point. And I was talking a little bit about this yesterday. We're at an inflection point where we're [00:21:00] gonna see more and more code being written by models in a, either assistive way or in a fully autonomous way because you've, broke your hand or something.

I think that's great. I think that will unlock a lot of uplift in the day to day ways that we think about productivity from an engineering standpoint. The way that we think about this at Google is that okay, look, if we have these systems, that we expect to write more code for us, how do we ensure that they're writing safe and secure code?

And so a lot of our research is going into what are the things that we need to do to unlock to make sure that at inference time the code that is being generated is not just good quality code, but it's also safe and secure code. So a lot of the research that we talk about, whether it's Naptime or some of the other things that you may have heard from other groups like GOSST or Google Open Source Security Team, the foundations for that is to ensure that, the models are generating safe and secure code from the start.

And so we have to work backwards when we think about, so how do we teach our models to identify vulnerable code? How do we instrument our models [00:22:00] with appropriate tools? and or techniques because we have some of the world's best experts in this space to reason about the best approach to find a vulnerability in code.

And then how do we distill that back into the model at train time such that when the model is being used to generate code that it's also thinking about

Caleb Sima: self auditing itself,

Vijay Bolina: not self play, but there are some aspects of that when we're in some of the research that we're doing. So that's. That's one thing.

And I think the other thing that makes this space generally exciting is that we spent what the last couple of decades trying to, and maybe to some degree, quite successfully shifting left when it comes to software security in general, and ensuring that the software that we're generating throughout our development lifecycle is being checked and audited and, instilled with best practices when it comes to writing software, whether it's static analysis tools or dynamic analysis approaches.

And the nicety of all of what we have been doing over the past couple of decades is that we have the frameworks, we have the tools, we have the pipelines, if you will, that [00:23:00] allow us to introduce new technology. Such that if we found a, a really strong model that I was able to find vulnerabilities, it's quite easy for the good guys to instrument our existing infrastructure and frameworks to be able to then, write better code or ensure that the code that we're developing, deploying is actually addressing some of those things that we continue to innovate in, but, be able to think about what it means to shift left as well.

Caleb Sima: Yeah. Because I dig in a little bit into that. Yeah. Obviously to your point, the more the AI writes code, we're going to have way more code. Your buddy with a broken hand is now writing like 30, 000 lines of code a day. With a broken hand. Yeah. Yeah. I was thinking about the guys like we're now generating hundreds of thousands.

So the attack surface in code is far larger now with AI generating code. However, to your point AI, I think by default, at least in the models, because they're trained on so much, they are, Because the default code they're trained on is usually insecure, they will produce insecure code, but it is also very viable to have it passed and audited by another model with the [00:24:00] mindset of the known issues that we know of today, right?

It is a much better static analysis tool in the fact that I can look at the code being generated, identify the standard flaws that we know of, right? That's right. Whether AI is good at finding unknown is an entirely different question, but it should be able to audit that, generate the replacement solution code automatically, then it now is created in its final form.

Is it safe? That's right.

Jason Clinton: Yeah, actually, and this is getting, this is illustrative of a broader point about agents. So I know there's somebody who's listening to this and who's like pumping their fist at the air because they say static analysis is, that already exists. You have the covarieties of the world who have been out there for a long time doing static analysis.

Caleb Sima: Yes, but no one's believing the but, but yeah, yes, we all know this, but yeah, this does not work.

Jason Clinton: So much of the work that Google's done on Naptime and some of the stuff that went viral on socials this week around vulnerability discoveries is actually an agent architecture, and it really speaks to being able to integrate lots of different signals [00:25:00] into making a decision so you can use a traditional static analysis tool in your shift left strategy and have an AI pulling that result in and proposing the alternative version of the code that doesn't have the vulnerability in it, as a signal that's an obvious, like a simple tool, you don't need massive amounts of intelligence to pull that off. There's other examples in this too, that are just like, very illustrative of from the Naptime example, Naptime's project is a giant agent architecture where there's no like super intelligent AI, just looking at code and saying, Oh, there's a vulnerability.

No, it's just taking all the stuff that we've always done as security professionals looking for vulnerabilities and breaking it down into little tasks and then having an AI take a piece of each of that puzzle. And that is true of everything that we're going to be doing in enterprises this way forward.

That's how you get the observability. That's how you get the level of integrity and sense of confidence around deployments is because you have at each step along the way in the enterprise process or the business process, an opportunity to inspect that [00:26:00] models input and output on that one little fragment.

And yes, it's part of a bigger decision at the business level, but if you're trying to project three years out, maybe we'll have AIs that are doing the entire big picture. But even if you can just break it down into little steps, that's what we're talking

Vijay Bolina: to each other. Yeah,

Caleb Sima: Here's the thing is to me, it's always been AI is all the definition for me and AI has always been if I had a thousand smart junior engineers, what would I make them do?

Jason Clinton: Yeah,

Caleb Sima: And going back to your, you've made this comment was about RPA, right? And this is exactly okay, I could take one junior engineer, focus them on each individual step in an RPA. Like AI can be done to use this. Take each discrete steps. Yep. Just like that. Use tools, but you're essentially doing the same thing a human would do.

Kristy Hornland: But that's like the thing too, that like you're answering and both of everyone's perspective really is the biggest like struggle with it today is the fact that one engineer is going through and using it, then you have to have a review come through and folks are saying, Hey, suddenly my super productive analyst that's on my team that's running [00:27:00] through is producing so much and I am already over inundated with tasks to do.

So what the solution is that you're talking through is basically how do we provide

Caleb Sima: More AI.

Kristy Hornland: More AI. It's more AI. How do we more?

Caleb Sima: Yeah, it really the AI produces so much. The only way to consume it is through AI. Yeah.

Ashish Rajan: Agent based architecture. But we should define RPA because we just blurred through the acronym.

The security loves acronym. Others, we just threw another one at people. What is RPA?

Kristy Hornland: Robotic process automation.

Ashish Rajan: Cool. What is it? For people who don't know what RPA is?

Kristy Hornland: That, that's like the deterministic side. So that is saying on any of the times that you are going through and you're like, I'm going to do a workflow that's just this task.

It's not like probabilistic or weighted. So that's what's separating it. I don't know if there's anything else that folks want to add, but I, that's usually how I delineate.

Caleb Sima: And so the other thing is is, and if they're going to talk to each other, English is not the most efficient way to talk to each other.

Jason Clinton: Yeah. That's an active area of research. So they are talking to each other in the sense that when you think about an AI calling an AI, I'm just going to go back to basics here. Every [00:28:00] case that I'm aware of that's been deployed out there in the world is they're using an API call to do it. So the AI is calling an API as though it's a tool and it just so happens there's another AI behind the API, and that could mean a multimodal, it can mean a completely different model from a completely different company.

And they're using English as the intermediary communication framework in some cases, they'll use like structured JSON output

Caleb Sima: Correct.

Jason Clinton: XML types.

Caleb Sima: But these are all protocols that we know of, and love, and try to today. Yeah. A proto buff any of these types of things, but English is easy to debug, right?

And so it's easy to see, but if you really, I feel in three years, if agents are going to be all talking to each other that is probably a very inefficient way of doing it.

Jason Clinton: And I think we can just look at human brains. We have a corpus callosum, which is a tiny bundle of nerve fibers that connect the two hemispheres of our minds.

And that is a very efficient way for organizing the way that we think. And if we thinking about models developing in the future, as Caleb's alluding to. Perhaps there will be something very akin to that in the future,

Vijay Bolina: but if we [00:29:00] still have to do vendor risk assessment, One GRC agent talking to another GRC agent, I think that would be the killer app.

You laugh,

Caleb Sima: but it's a good idea. They'll get in a fight and just quit.

Ashish Rajan: I guess that's a good well raised point that may, they may be talking not in English, but in JSON or YAML or whatever. Yeah. But what is the current way people are using it? I think a lot of people, and a conversation that I had was, doesn't really sometimes go beyond a Claude.

Or chatbots. Writing reports and translating. Is that all?

Vijay Bolina: I think it's like assistive tasks. Then there's like knowledge democratization tasks where you're taking a corpus of information that is maybe unique to a specific domain and then democratizing it to a population of users, maybe internally within your org.

And then there's very specific, domain specific stuff like code and, or maybe like generative media, not text.

Caleb Sima: Yeah. That's true. We haven't talked about any of that. [00:30:00] Yeah. Movie music industry, all that is amazing.

Ashish Rajan: Yeah. That's where I was coming from cause it's not just that, I guess we obviously come from a technical background, so we're primarily looking at narrowing enterprises.

Yeah, that's right. Very much on the enterprise space, but the AI space has gotten a lot more broader. They're like video generation. Video production, media networks are already looking into it as well. This obviously non enterprise, non tech companies have had to go down that path. There are security people in those organizations as well who have to figure out a, like the whole conversation of shadow AI people have been talking about.

Or if I just, Stop OpenAI it's all, but no one looks into browser plugins. There's a whole another layer there. There's wrappers around this as well. So you may block.

Caleb Sima: Can we talk about a little bit about that too? Actually I, you guys did a panel yesterday at the CISO summit. And actually part of the original intent was some of the panel was going to go off this podium I'm about to stand on, which is, I feel like there's so much fear around AI and people are like, CISOs are [00:31:00] immediately the two things. I either I'm blocking it outright for my employees because I'm afraid of my employees uploading confidential data or two, I need to see visibility. So I basically need visibility of the data going into these models because I'm afraid of what my employees are uploading into these.

But which is basically like, you can get this from CASB, DLP, but everyone is saying they need a different solution to go and do this and where I feel like security and CISOs have had a real struggle that I have continued to push is the threat that you're afraid of the model and the technology itself, right?

Where I upload PII into the model and the model will get trained on it. And then an attacker will extract that data. That is the thing there. Yes, absolutely. I'm afraid of that. Or is this a more of a third party risk problem to me. Because then I asked the question Hey, everyone's going to be using this.

What do you do it's Anthropic, Google, Open AI that you upload data into Salesforce, don't you? [00:32:00] Yes, we do. Why do you feel comfortable and doing that? What happens if Salesforce is using models behind the scenes? Cause you know that and it was like, Oh so then it starts becoming into, you go down the logic path and then it really becomes a third party risk problem.

Actually, the problem that CISOs have is not, Okay. When you really talk logically, okay. Maybe it's not the model, the tech itself, but it actually is the companies themselves. And what you and you guys represent some of those companies. Is that what you're saying that these are third party risk problems and people just aren't really getting to that?

Vijay Bolina: Absolutely. Absolutely. It is a third party risk. And I was laughing a little bit because we're all in the same chats. And I remember about a year plus ago, we, there was a long thread about whether we should just block all access to these third party large language model providers as their use appeared to exponentially increase.

And there's a genuine concern about what are people actually sending over the line, across our boundaries into this third party. And what guarantees, and this was a year plus ago, right? What [00:33:00] guarantees do we have about those model providers or those CSPs not training on our data, how they're protecting even at inference time, right?

So even if they say, all right we're not training on your data, you could tick this box and we won't do it. Okay. You're still sending the data to the third party's environment. It still needs to be served to the model. Inference still needs to happen. And so if you were an adversary, one of the most opportunistic plays you would have is sitting at that inflection point on the edge where that inference is happening, and you'd probably be able to observe a lot of really interesting interactions, things I'm worried about.

Caleb Sima: It's easier for an attacker to go and get the data by hacking you guys. And even getting into your proxy and getting the chats coming in, then it is trying to extract it directly out of a model that has been trained on that data.

Vijay Bolina: Yes. Yeah, I would say so. And I think this is what, I think most adopters of the technology or the folks that, see this entire space as a risk when it comes to third party model providers.

Is okay, [00:34:00] like how comfortable are we sending certain aspects of our internal non public data to this third party? And to your point, look like we do this already with the proliferation of SaaS software. We've been doing it for the last 10 years.

Caleb Sima: I just want to also emphasize. When I talk about management of risk, all these same companies all allow all their employees to use third party email apps, like Spark, like everything where all your confidential documents, emails are all going into these SaaS providers that are startups that have no security.

And you think the biggest threat is someone extracting it out of a model versus hacking this third party email provider and grabbing it. Come on guys we'll look at the risk.

Vijay Bolina: It is a risk exercise, I think, especially, right? I would say it's no different to some of the more traditional applications that we have when it comes to data flow, right?

And what your comfort levels are when it comes to where that data is going. Now, I guess if you're genuinely concerned about your data being used to then optimize that third party's model, then that kind of [00:35:00] goes into maybe things that are unclear and, or maybe tangential to some existing, service providers.

Or SaaS software vendors, but I don't know, I, I think it is a risk exercise. I think there's genuine concern that most people have when it comes to this space, when it comes to legitimate use case and, where you want to draw the line of where are you going to, want to adopt the technology or not.

And I think if the risk is so high, people are just going to go and opt to just deploy internally, right? Using an open model of some sort.

Jason Clinton: Yeah, I think if you're a CISO looking at it from the perspective of it's a data governance issue, just like anything else. And looking at each vendor individually is important.

So many of the conversations that I have with folks who are my counterparts at other organizations starts with the question that you raised, is my data going to end up being used to train a model? And to just address that directly, there's like a misconception first of all, that an inference model can actually do training.

Learn in real time. Yeah. So just to talk through that, it takes thousands of GPUs, all working together to train one of these very [00:36:00] powerful models. And it's not the case that you're just going to accidentally, oops, I entered training mode. Although that should

Caleb Sima: come

Jason Clinton: at some point.

Caleb Sima: Three years down the line.

Three years down the line. Like three years, guys, you have clock is ticking. Three years down the line. Real time training and learning.

Jason Clinton: We'll see, The energy requirements right now are just astronomical for training mode. And so if you're doing inference, you're just talking to a few GPUs.

Those GPUs are going to be responding just to your request. And then it is still the case today that with every single one of the model providers or an open provider, if you're getting it from managed service provider, who's running the open model for you, you can just ask them to do no retention, just like any other day of governance that you've ever done before, right?

Yeah. So it's up to you to just do that. The other thing I'll say, and I think this is really important, most people listening to this are not aware. That every one of the model providers, and this is Google included, have the ability to run the model and inside your VPC. So if you're doing direct integration with a model, you [00:37:00] can just get this environment that just feels very safe from you from a data governance perspective because it never leaves your network.

And that's an option on Amazon on Microsoft and on Google's clouds and then that relationship between you and the cloud provider, you don't even have to have a conversation with.

Caleb Sima: And I think a lot of people also there, a lot of enterprise get it. Okay. Yes. If I'm using an enterprise model, I can set up my enterprise model in the configuration that you said I could do it.

Then I have to figure out what about my employees sending it to, they're using as a consumer. Not through my enterprise, so I either have to force proxy all my things into my own enterprise version of that model, but these guys are still using it over here, and then they have that fear where you do train off of the consumer, right?

Jason Clinton: One of the things that's been most surprising to me, and I think just people, it really underscores how valuable the technology is. Every single person listening to this has been in a zoom. Or a Google Meet call that's been transcribed. And then it automatically wrote the meeting notes at the end.

And people say the [00:38:00] craziest stuff in meetings sometimes. And yet, every single organization that I've interacted with has adopted this in some way, shape, or form. Maybe they don't use it for every call, but it's out there and it's happening. And it just speaks to the value of the technology.

It's obviously reducing toil and people are excited to adopt it,

Ashish Rajan: if someone does use ChatGPT or any other tool, Just by typing in a sensitive information does not really mean you've trained the model on the other side to now suddenly know your confidential information. And to what you also said, you can even have it as an enterprise level on your side.

So he doesn't ever even leave the environment. Yeah.

Jason Clinton: So at Anthropic, we've never trained on user input. And with enterprises, we'll have a conversation around what their security requirements are. And there is a bit to flip there that we can have a conversation around.

Caleb Sima: Absolutely. Have you heard, what have you heard these enterprises say about this? Does it mimic me or is there a different

Kristy Hornland: I was just laughing when you were like, when you're in like a zoom meeting and it just starts transcribing and , I've been in meetings with clients where they've been like, I was just on this call and I just started transcribing, but nobody asked [00:39:00] for my consent.

Nobody asked for permission. And we have a, everything needs to be approved, is what we put in our acceptable usage policy, right? That's a paper document. So nobody is looking at that and going yeah, turn off that feature. So what's been so interesting is like the reality for most organizations is that, even to your point about you're seeing these tools get used outside.

What if they are just starting up on people's laptops because they had AI integrated. They were already a pre existing vendor. That's a reality. And who's going to go into for example, there are some where it was, to assist with the your spelling, your grammar. And who is going to flip into every time that they're opening up a word doc and go, I'm going to turn this off because I'm a good employee.

I don't know. That's just not, bless you if that's the case, but that's not really reality. So like for most organizations, they have to go through and really set up, how can we make this appealing to work within our boundaries? But I also was laughing about, okay you rely on your data governance.

Data governance and enterprise risk management, both of those things, most organizations [00:40:00] have quite a bit of work to still do.

Caleb Sima: Okay, guys, challenge me on this. This is another thing I've heard CISOs talk about. I need all visibility of where my AI usage is and where it's going. And when I say a lot of CISOs, I'm talking majority are all scrambling to find products that show them their AI usage in the enterprise.

The fear is again, people are using AI. I don't know where they're using AI and what they're using AI for. Therefore, I feel at loss of both control, visibility. And I have fear that it's being used, or all my data is going to these things. And I don't know what it is.

Jason Clinton: But there's two sides to that concern.

One is the risk management aspect. And maybe I'll leave Kristy to talk to that one. And then I'll just address the incident response and the sort of the SIEM, if I can liberally interpret that term. I think that there probably will be a marketplace for SIEM for AI. I don't know what it will look like.

I don't know what products will come out. That was support that, but [00:41:00] certainly it may be the case that, and especially when we're talking about agents in the future, you will want to be able to, in the context of a response to an incident, a leak some privacy incident, whatever it is. be able to go back and investigate what did the employees do?

What did the systems do in this context? And, can I see just what happened?

Caleb Sima: Like the conversations that occurred.

Jason Clinton: Absolutely. Absolutely. And that's not something that's quote required by any risk management framework, but I could see how you could just extrapolate from existing security context on SIEM and imagine what that looks like in the future.

I think the other half of it though, is If you want metrics on who's using what where and when and how and you can use that part as part of your GRC, that could be very valuable.

Kristy Hornland: Yeah, I think to that point too, it's like maybe also looking at from the perspective of what are the crown jewels of your organization feeds back to that point and maybe that's where you steer the conversation more from the lens of, if this is something that is critical to our business and I don't know that it's here, how do I answer some of the questions that may come [00:42:00] both from a regulatory perspective and future talking about how did you actually go through the steps to secure and even things that are traditional.

Caleb Sima: You have to answer those, yeah.

Kristy Hornland: You have to answer those. So if I were to look at it I think that there's an immediate need to like, see everything is what We're pointing to, and they're like I want to look for a shadow AI, which is both my third parties, it's also internal, I want to know what marketing's doing, that could have impact on our business.

But all of that to really say there's a prioritization in the background cause right now trying to get visibility into everything is just, it's not standardized.

Vijay Bolina: Yeah. I want to steel man this a little bit. So it sounds like there's genuine concern about being able to observe and, or assess.

There's any risk associated with internal users engaging with external third party model providers in some way, shape, or form.

Caleb Sima: Yeah, even internal, yeah. Yeah. I would count internal as anything. That's, my, again, it's not well defined. Yeah. These are like, Oh, AI usage visibility.

I need to know what my.

Kristy Hornland: Like what if you have like a research and [00:43:00] development department that for a long time has been deploying models because it's not new and they've been really creating products or services that are core to your business and you've got a CISO going I'm not integrated quite enough in that business to get that line of sight.

Like it's not again, just the third party. That's also, again, this is a piece of IP for our business,

Vijay Bolina: I guess I'm just thinking, do we worry about this as CISOs when it comes to employees, maybe emails and having email exchanges with external parties and then be concerned about what the content of that email be.

So I'm just thinking about if we want to have some level of observability, there's existing classes of solutions that we've developed over the past couple of decades now that service some of these concerns, DLP, CASB. Yes, I know these things are things that we know. So why aren't they being used?

Why aren't they being used? They can be used when it comes to maybe some of that. Concern, I think. And by the way,

Jason Clinton: DLP has gotten so much more powerful with AI.

Vijay Bolina: It has, yeah. Much more [00:44:00] accurate, much better than Regex. Gosh,

Caleb Sima: I feel like that's a session, by the way, all on its own, is the way DLP is changing because of AI.

Yeah. It's really doing amazing things. Sounds like a part two. It is.

Vijay Bolina: Models that are a size that they're quite performant when it comes to, the amount of transactions you're going to need to accurately, or to service it, a DLP use case.

Caleb Sima: Now, my question was, and also going a little bit when I, when people tell me that, I also think about Kristy, where you were saying, which is okay, but if AI is embedded in everything, if I open up a word document, AI is going to be a Salesforce, word, Google, you name it.

AI is, if you think back on cloud days when people were like I'm very, I need to know anything is going to cloud. What's going to come. I'm very afraid of cloud, but then now cloud is everything. It is ingrained into what you do. AI is just very similar. So they're, what is the point, is more where I've been coming from.

How do you even get that visibility? It's like trying to say we're on what runs on a CPU.

Jason Clinton: So TPRM is in my function at Anthropic. And so I have the same problem that anybody listening to this has of we're trying to run a company and we have the SaaS providers [00:45:00] and I'm trying to make decisions about which SaaS providers doing what with which AI and all of those challenges are quite interesting.

And so in the questionnaire and the third party risk assessment, I'm asking the questions that I care about. For example who is the actual service provider for the model?

And so I'm looking at the risk and just making a decision based on the risk and deciding in this particular deployment of AI, I'm not comfortable. And this one, they've checked all the boxes and I'm really comfortable.

And even if it's not an Anthropic model, it's perfectly fine. And I think everybody just needs to figure out what those check boxes are for them. And the ISO42001's of the world, the CSAs. Yeah. Of the world are starting to write standards for what it means to deploy AI securely. Yeah.

Vijay Bolina: EU AI acts going into it.

Yeah. Already active critical applications. Critical success.

Jason Clinton: And then CoSAI which you know, Google and OpenAI and Anthropic are members of, are also starting to work on this technical stuff too. Really looking forward to all these technical standards. CSA has probably got one out pretty soon.

I think this is going to help guide people in thinking about what are the checkboxes that are most important.

Ashish Rajan: It also goes back to what you were saying [00:46:00] initially, when I think we mentioned trust part as well, as a third party being used in an organization. And yes, you answer a question and saying, yes, I'm not using any AI now, as at the time of answering this question, I may tomorrow, but hey, that the question didn't ask for tomorrow, it just right.

Where do we find that? Is there a place for us as a community, as a people to start raising that standard. Yes, 42001 is great. I think there's four levels to it as well, consumer and everything. Is the conversation of trust and safety coming up even more when you talk to people?

Is that like becoming, is that, or is that just basically?

Kristy Hornland: That's the thing is like looking backwards into your third party. It's not exactly like the easiest process. You have so many vendors. And when you're looking at the like broader supply chain, you're going. Okay, how do I prioritize even doing that reach back?

Because chances are that your vendor, they might proactively reach out to you and say, we have integrated AI into some of the core product services that we offer you, but that's not everyone. And there's also not like a [00:47:00] SOC 2 report that they're going to be like, here you go. Like we're so good that we have brought this to you proactively here.

So in the case of majority of vendors, it's Okay. You're working with a startup, maybe that they're not going to have a governance COE for you to look towards. So instead you're having to probably, again, prioritize based on which of these processes, which of these third party vendors are absolutely critical.

What are they doing? That's mission critical to our business. And that's just the reality of it. Cause you're working with folks that have a 7, 000 vendor pool, like how do you, when you have two people working on it,

Caleb Sima: I think they all have, again, all have AI in it. And what are you really concerned with on AI is a good question, right?

What do you ask for? You ask for the provider. What else do you ask for?

Vijay Bolina: Yeah. And I think, what we're also going to see when it comes to TPRM is that vendors are going to update their terms and service to add a couple of lines and say that they may start to incorporate AI in some of their services.

And so what I know about TPRM approaches in most organizations is [00:48:00] going back and re reviewing your relationship with this vendor and ensuring that the terms of service that are in place are in fact what you signed up for originally. If there's any questions, additional risks that may have been introduced into the relationship that you have with your vendor or all kind of nuances that often get mislooked when it comes to, large scale TPRM programs and organizations.

And so I think it's going to be an interesting time because you're going to see vendors that start to incorporate this technology into a lot of their space. And then I'm telling you, I think we need to, we need a GRC agent.

Jason Clinton: It's a really good call out. And I think one of the lessons that I've learned the hard way is making sure for every vendor you have that you've subscribed to the notification.

That the terms of service have been updated as you just sometimes get surprised that magically an AI product has shown up in the user interface and you had no idea that it was going to happen because you missed the,

Caleb Sima: and here's a bigger question. Let's say this does happen. Are you going to turn it off?

No, how's that going to work?

Ashish Rajan: No, do you have a contact for your third party?

Caleb Sima: Obviously we know in [00:49:00] our heads that this is an impossible task almost in multiple ways, right? But it's just okay, great. You've now added AI. We don't want that. Now my user, my employees are going to scream at me. And by the way, the vendor themselves be like, Oh, we don't have a way to turn that off.

Ashish Rajan: But maybe it goes back to what you were saying about the visibility, the reason why there is some nervousness around visibility is because of this, because of the fact that a yes, at the moment, we can trust the fact that third parties are answering the dr the third party questionnaire. Honestly, they're not gonna use it.

They use it.

Caleb Sima: But to what end?

Again, I'm gonna push this. Yeah. And this is what this co like here's the thing. It's okay, I know I already have a contract with you as a third party. Yeah. And in some sense. My employees are already uploading confidential data into you as a third party.

I'm making this assumption, if that's the negotiation we have. So at this stage legally we need to make our cover. That's the point of the documents you are going to do in your best way to secure and protect my data and whatever [00:50:00] way that is. I have now established a trust relationship between you and I, you decide I'm going to implement AI in the back end of this.

I am still uploading my private data, you've done this. Oh no, you can't do that. That is bad, but let's just say it's not AI. Let's say in the background, they've decided, Hey, we're going to shove all of our stuff in this open Redis instance on the internet and all your data is going there, but you're not going to know that it's happening.

You've established a level of trust with this company that says, I know I'm uploading my data. So what is the risk? The risk here is that trust, they're not obeying that trust, which, it's a hard job and they are uploading these things to some open Redis instance, but what are you going to do about it?

But then they say we're using AI and then all of a sudden the world is broken. Why? I

Jason Clinton: think it's a good point. It's just the same stuff that we've been dealing with for the last few decades. Yeah. And it's just a new

Caleb Sima: And they're using third party providers behind the scenes too. Oh yeah. Of course.

They have third party as well. Yeah, so they have their own agreements. Yeah. And but do I [00:51:00] know about their sub agreements? How deep do you go in that chain, right?

Ashish Rajan: Maybe to set some more context and probably some frameworks as well. It's interesting you call out the 42001 yeah. 42001 I'm like, which is the acronym. The 42001 coming up, it's not mandated on anyone. The people can choose to go down the path of being certified for it. It's not mandated. Yeah. There's obviously a lot of work still to be done from a policy perspective. And to your point about building trust perspective, that am I supposed to go into the supply chain and how long is the chain of trust?

I just feel like there's a lot of FUD. That's just, but I am in agreeance with that.

Jason Clinton: There's an opportunity for the industry to provide a lot of clarity and guidance and confidence across the board. So I already talked about CSA, CoSAI, we've got ISO42001 when these things come out. It'll be like the adoption of any other security standard has been in the past, right?

We will decide as organizations, the kinds of standards that we want to hold other organizations to and certifications that we're looking for. And then in each of the frontier model providers, we have [00:52:00] all done something called like a responsible scaling policy. Everybody has a slightly different name for this, but Anthropic, it's a list of our commitments to our customers and the public about the way that we will deploy models responsibly.

And, we think we're at ASL 2 right now, we might be at ASL 3, soon, and then ASL 4 is a little bit further down the line, we'd using the biosafety levels as a sort of benchmark. That's an example of that industry led opportunity to create confidence and help the public understand the risks and rewards that come from these models.

And I think that there's just an opportunity for everybody in the, playing in this space to, to act in that way of trying to help understand exactly what the risks are and play above board and be transparent. And that's going to help a lot with the adoption and concerns.

Ashish Rajan: And maybe get the FUD out of the way for some.

But maybe and where I was going with this is also because we are talking about the current state of AI security as well. And a lot of people would hear all this conversation and starting off with AIs talking to AI.. They would have their own language. This is JSON,

Caleb Sima: YAML. Yeah, this is the state of three years from now.

It's entirely different.

Ashish Rajan: So just to set the scene for [00:53:00] people as well then, at the moment, a lot of people are thinking about so many conversations that I had, or where people's thinking about, Oh, what am I doing with data classification, blah, blah, there's a set of ideas people have for how do I even make myself ready for AI because I may not be there yet right now.

So people who are building a cyber security program today, as in like when I was at the point is recorded 2024, what do we feel are probably the good starting point or building blocks they can work with? Are there any easy tactical things they can start with in their programs to or include in their programs?

Kristy Hornland: If somebody is coming into an organization that I'm assuming has already been established, but they're still trying to stand up their program, per se, I think that what we've seen be most successful is one, understanding who else are the major decision makers in that particular strategy. Cyber alone doesn't set an AI strategy for the organization.

And so that means that you have to find your connections across to legal. You have to find your connections across to privacy your finance, your honestly if you've got HR should be [00:54:00] part of the conversation too. You have so many different groups based on the types of risks that you see from a broader lens of AI. So that's everything from like degradation of skill or potential exposure to say misinformation or some of these bias concerns, all of those like varying pieces, you should have an overview of who are the people that I need to have this conversation with. And then when I'm talking about, standing up my responsibilities, I should be looking towards the NIST AI Risk Management Framework, right?

That should be a starting point because it is looking at overarching governance and all those pieces. If I want to address, very specifically, what controls can I put over top of, say, an existing application security structure, then I might look towards something like 800 the NIST special publication 800 218a.

It's that is something that could give you an idea about what are things that I could implement across the life cycle. Chances are you can't do everything in there off the start, but it gives you ideas of what would this look like in terms of [00:55:00] applicability and also starting to look at how do I look at the use cases

Jason Clinton: just to be very tactical?

I just went through this, right? As recently as just 18 months ago was it was pretty small and we were just getting started. The very first thing to do here is to hire a compliance function and a security engineering function. The compliance function will eventually become a GRC function.

So just getting that in your mind as the path forward and when GRC stood up, which is, maybe something you do at 250 or 200, that's where you bring those stakeholders that Kristy was talking about into the conversation where you have an executive risk council and they're reviewing the risks of particular decisions that you're making in your organization.

And that can be products we're launching, it could be the data governance framework that we're putting in place in an organization for the first time. So we can say, you can use AI with all of the stuff that's not PII or all of the stuff that's not very confidential information and whatever

Caleb Sima: you're in critical systems or non critical systems.

Jason Clinton: Oh yeah. Yeah. All of those are very important things to be thinking about very early on, because if you're that early in a company, you have an opportunity to set the trajectory of the way that your entire organization for the [00:56:00] entire life, Is thinking about data management, data governance, and risk management, and so much of the default for a young company is to hire technically very knowledgeable folks into this role, and you just immediately go to like, how can I solve this with software?

You have to also manage the risks and the governance and the privacy aspects as well. And that's very important to get a handle on immediately.

Vijay Bolina: There are nuances around how a machine learning developer or a researcher within your organization or a data scientist within your organization may interact with the tooling the data that they were pulling down, the data that they're curating, the way that they're interfacing with the models the Jupyter notebooks that they may be turning up, there's sharp edges with all of that. The tools that you use to develop and deploy machine learning systems are different.

And so having a deep understanding of where your organization is on that journey when it comes to building and deploying machine learning applications and or applications that enable machine learning models underneath the hood. [00:57:00] I think it's something that you're going to need some expertise on, and hopefully there's some guidance through organizations like CoSAI, CSA, NIST are all good organizations that are putting out standards and frameworks that are going to be useful when it comes to the technical controls that you're going to want to consider when it comes to some of these things. CoSAI actually published a risk taxonomy recently that highlights All the different sharp edges when it comes to developing and deploying machine learning systems.

So that's a good resource for folks to look at when it comes to things that could go wrong when you're developing machine learning systems.

Caleb Sima: I feel like there's a mid level of, when I think about it, I think of it as there's going to be models or AI we consume as employees.

So there's like corporate enterprise usage of AI, right? This is anything that employees want to use. And they're either going to be and those will have their own segments, right? Am I using Open AI or Anthropic or Google and how am I using it? And for what features and benefits in my employee usage day to day.

Yeah. And then I think there's the models of which the enterprise is building. I am [00:58:00] producing these things. It's either being embedded in my product that I offer or in services that I, as part of my feature set. That I'm selling or I'm building it for my internal employees to use. And that is where I think you get more into SDLC of model delivery, pipeline, a lot of these things.

It now goes into pretty much, it is a software development lifecycle, but just for models and you've got to do the same thing. Yes, they are somewhat different tools, but in the same sense, I don't think anyone was making a big deal out of machine learning. data pipelines three years ago on security, but yet it's the same thing, right?

When people were doing machine learning, it's the same process. There's really no difference except bigger compute, but basically the same kind of things to some degree. So to me, all of those are all things that apply and I would look at it. So there's going to be Policy, GRC, and guidelines for my enterprise corporate usage of AI, and then there's going to be my standards and [00:59:00] things I want to apply to my creation or usage of producing AI.

Jason Clinton: One thing I'll add to that, that maybe most people listening to this podcast haven't internalized is going to be happening in the next few years. is there will be people in your organization who don't think are developers, who are suddenly now developers. You're going to go from a world where a program manager who's never written a line of code in their life and never planned to write a line of code, Gets to a world where they're just like I wanted to do a little data analysis, and it was too much work to put it in an Excel spreadsheet, so I just asked a model to write me a Python script, and lo and behold, you now have, a data science platform responding from a section of your organization that you had no idea was ever going to be thinking about coding.

Software is eating the world still. That was true a decade ago. And with AI writing software and making it so accessible to just anyone to write, you need to be thinking, even as an organization that's not technical, what does it mean to have software development emerging organically from every facet of my organization?[01:00:00]

Ashish Rajan: Wow. So it's like turning from a three course meal into a buffet for everyone. Yes, exactly. Maybe if you flip the table to more enterprise then, what would be different? Because established enterprise have been doing. tech for a long time. They have policies, governance, a lot of things established already.

What would be good from an enterprise perspective? Clearly, they're also facing the same problem as well. What would be a good starting point for them?

Kristy Hornland: I think that's unfortunately where I was coming from too, which is just more so there are established decision makers at the table that they need to get in and connected with because if they go about trying to institute a policy, but there are no champions in the room for it.

There's no agreement as to their priorities. We talked about, you mentioned like your commitments, right? Organizations, especially large enterprises, also have their values commitments that they've made publicly. We see them standing up basically their own model around this. These are the principles that we're going to adhere to that help steer because there are not explicit dimensions for every organization about this is banned, this is [01:01:00] acceptable, et cetera. There's a lot of gray area. So by building in principles that they've agreed upon with those different stakeholders at the table, then they're able to more easily understand these are the types of use cases we want to pursue and then start to look at it and how does this go in joint with our AI strategy, which is collectively, we want to use it.

Like most organizations want to use it. And if it's not at the top, it's getting pushed from the bottom saying, we need this also just to, we'll look for it anyways, to the point of, kind of forcing a hand. So from the perspective of you can set up policy, but it really comes down to those relationships agreement on principles, really aligning to obviously those frameworks that were mentioned, but also looking towards how does this accelerate our business into the stamp of innovation.

Ashish Rajan: Would you say it's the same as people went through digital transformation, but it's like V2 of digital transformation with a higher benchmark.

Kristy Hornland: It's just the accessibility piece. I think what like really unravel is like people were comparing it to cloud and it's was every individual in your [01:02:00] organization, like I'm going to stand up a cloud environment.

No, like this is everybody is trying to touch AI somehow. It's a buffet. So yeah, the buffet.

Caleb Sima: Yeah. I'd love to just know around the table what you guys have found amazing about just you personally using AI yeah, like just example. Like I was giving an example about how just recently I was doing something and I was using Claude and it came up with this thing that was like, I read it and I was like, this is 50 times better than what I would have ever put together, and it blew my mind a little bit on some of these things.

What do you guys just normally, I would love to see, what's your normal day to day personal usage?

Vijay Bolina: I internally I use this as example a lot. At Google, we have internal models that have been trained on the corpus of information and documentation around how complex our corporate and our production environments are we can, which is

Caleb Sima: quite true.

Vijay Bolina: Yeah. and we call it internally Broccoli. Broccoli because people hate it. But you have to eat it 'cause it's good for you. .

Exactly. . [01:03:00] Yeah. That's awesome. So we have these bottles

that are trained on Google broccoli knowhow and knowledge. And I've been at Google for about four years, so I have a pretty good firm understanding of how a lot of it works these models are trained on a lot of the very technical details that are Google infrastructure and are quite verbose and helping you reason around the things that you're trying to fumble around and get to work. And so I'm pretty impressed about the capabilities that the models have to be able to really understand what you're asking and then present you with some information, sometimes comically wrong which I'm able to pick out.

And I think that maybe that's a surprising thing that. In some instances, the models aren't superhuman or way better than me in any one thing, but I could be able human, they make mistakes. They're human, they make mistakes, maybe. And yeah, and but I use this example often because some days I find myself amazed about how capable this model is and able to generate exactly what I need. Yeah. Or tell me the very specific detail that I'm actually reasoning or thinking about. And [01:04:00] then some days it's shit, that was wrong. That thing got deprecated like five months ago. Like the other Google meme that we.

Caleb Sima: But it's probably like a, it is like a true copilot. It's like having a 20 year Google veteran sitting next to you that you can just ask. Yeah. I'm trying to figure that out. It's super powerful because

Vijay Bolina: it's Nooglers new Googlers that come to Google. Expend Jason, maybe could attest to this quite some time to get up to speed on just how to do the most basic

Jason Clinton: three to six months.

Vijay Bolina: Oh, yeah, and that's how long it takes on average of Google engineer to be productive. It's quite daunting, right? And so now we have systems And the technology to really make people quite productive.

Caleb Sima: Yeah, so the way it used to work is I'd have to know somebody, I'd have to buddy that it gets assigned to me.

Or I'd be like, wait, what's the new HR thing? You got to go find the right person that can give you the answer or read a Wikipedia or a Confluence page that's out of date. And enterprises, but now these things can are fairly up to date.

Jason Clinton: For everyone who's a security professional who's listening to this, All of us have written threat models. All of us have written security strategies. We've written [01:05:00] OKRs or KPIs. One of the things that I do on my team is I use the very large context window and I've gotten pretty good at prompt engineering. I'm not the expert, but I write a really good prompt that says, here's the context of our team for the last couple years.

Here's what we've worked on already. Here's the threats that we're thinking about and what we've already addressed. And then I put that in the context window, and I say, you're an expert executive decision maker in a company. Yes. Write me the next strategy refresh, the next threat model, the next the iteration of the risk framework.

And it requires a little bit of back and forth. You have to say, oh, you, you got something a little wrong here. You need to address this. This seems a little weird, and you've structured that a little bit weird. But a couple rounds, maybe five. Back and forth and you have 95 percent complete document at that point, which is if you're replacing you, yeah, it's a lot of time.

Makes time for podcasts.

Caleb Sima: What's funny is what's funny is executives are all aboard AI because their belief is that this is going to help eradicate the lower tier, by automation. But what they don't [01:06:00] realize it's actually going to replace the executives at the top, not the people at the bottom.

And it's going to be more like, Oh, you're an executive. That's easy to replace. And it's the, and I actually commandeer and control the people on the lower. So it's the executives that are going to get wiped through AI. They aren't ready for it.

Kristy Hornland: I guess from my standpoint, like some of the stuff that we're looking at, especially within AI security, a lot of it is like just out there.

It's a bunch of documentation. And so instead of having like our team go through and individually like grab stuff, we were basically able to pull together like a threat matrix about the different stages of the lifecycle. Obviously, this is now published, but this was maybe like 10 months ago, we were going through that.

Yeah. Yeah. building out a table on this against the different life stages instead of going through and being like, Hey, start from zero, start from nothing. And I think that's where the biggest value is it's easier to review than to go and stare at a blank sheet and go, I got to start somewhere. And that procrastination is one thing of it.

And the other bit is just, how do I even [01:07:00] structure this? Oh,

Ashish Rajan: Having a starting point for anything that you want to go, or even it could be a law or a policy or whatever. Yeah. At least having something, which is a good starting point. Cause I think If you have a starting point, it's easier for you to write the v2 of it, v3 of it, because you can improve it.

But sometimes starting is the hardest part as well.

Kristy Hornland: Yeah. And also just thinking about like your leaders are stretched on time already. Yeah. Having them sit in a workshop with you, what if you sat with this and really were able to have that conversation first off, that is such an advantage for a lot of our team members to come to the table and go, I had something to riff ideas off of, I had a structure to start from, and now I'm bringing this to you.

That is the level of this is where we see all the resources is coming together for individuals.

Jason Clinton: I love that you brought that up. And actually this ties a bunch of things together. One of the things that's merging right now is the ability for a small team to decide what the context window that they're going to share with each other is going to be.

So you have in this user interface where you go in and do a project and a project has the context window fully loaded with the Google case, the small team [01:08:00] case, and you're all working from the same knowledge base. And you start with, you can even in some cases put your own system prompt in there.

Yeah. So it's tailored to your specific enterprise and that's an opportunity for everyone on your team to get from the same baseline, a place where they can ideate and then get to decisions faster.

Ashish Rajan: My example is going to be I'd use ChatGPT to come up with questions for this, but I'm like, that was my example.

Like you did a good job.

Caleb Sima: I'm going to give you the exact thing that blew my mind.

Ashish Rajan: Okay. Okay.

Caleb Sima: So tomorrow I have to go to Travis is doing this fireside camp story. So I'm going to do this 30 minute talk. And the topic that I'm talking on is why is the cybersecurity industry not working?

That's the topic. So I wrote just a little, okay. I just basically blobbed my thoughts together. Super generic. Thoughts like, Hey, there, I don't think we understand how to really evaluate risk. If I went to a CISO and said, Hey, by the way, all of your employees are sending all of their email, all of the documents and attachments to this unknown third [01:09:00] party that you don't know anything about, that's a small startup.

What would you say? They would be like, that's impossible. That would, but we don't understand it that way. Anyways, I blobbed a bunch of them. I said, Hey, Claude, give me a 10 minute talk track on this thing. I want to read to you. Just some of the things that it came out with. Okay. It says, number one, misunderstanding risk evaluation.

We're drowning in a sea of alerts, treating every ripple like a tsunami. In 2022, a major financial institution reported that of 10 billion, yes, billion daily security events, only 25 were significant threats. That's a 0. 000025%. But here's your counter argument. But what if we miss something critical?

By focusing on everything, we risk missing what's truly important. We need to shift from quantity to quality in risk assessment.

I'm like. What the? Yeah. It's quite believable. Ready? Investing in visibility, not prevention. Because one of my other things is I think we as CISOs are influencers.

And all we can do is invest in visibility. We're building watchtowers when we should be building walls. 80 [01:10:00] percent of security budgets go to detection and response, leaving only 24 percent for prevention. Gartner 2023. Wow. Imagine a homeowner who installs cameras in every room, but leaves the front door unlocked. That's our industry right now.

Ashish Rajan: So Atnropic knows our industry better than us.

Caleb Sima: I'm telling you, I was, you're reading this. You're like, okay, like this is doing way better than I would have ever done. The research is just amazing. It's amazing. Keep going though. Another one of my thing is Hey, actually, I think we invest too much in people, not enough in products.

So we're creating jobs, not solutions. The cybersecurity skills gap reached 3. 4 million unfilled positions globally in 2023, but isn't cybersecurity a human problem? Of course it is, but we need to empower humans with better tools. Not just more tools, right? Like it's just, okay. Like this is good, [01:11:00] right?

Jason Clinton: We'll combine this with a deep fakes and then I'm done.

Caleb Sima: It's already writing your security strategy already. Yeah, then you get the answer and it clearly automatically write the board pitch. on any of this stuff,

Vijay Bolina: Jason Haddix had that board bot or something he was talking about yesterday on the panel.

Oh, yeah. It was kinda good.

Caleb Sima: Look at this is, by the way, I didn't put any of this stuff in my first blurb. Oh, this is all stuff that it just, you know. I put the little nuggets oh, I think, people and yeah. And it's random. Yeah. Ramblings, basically, but then it produced this thing with links and references as well.

And I I had to go double check the reference, the actual statistics I had to go and double check right on a lot of these, but I would say about 70 percent of the statistics were accurate. Yeah. So executives have been warned. Oh my God. It takes my job. If you really think about it, like this is doing it way better.

It takes everyone's jobs to be precise. Yeah.

Ashish Rajan: But in the last couple of minutes we have. I obviously wanted to give everyone opportunity where can [01:12:00] people reach out to you guys or for, I would love to continue this conversation for we feel like I have another half an hour on this, but because we are short on time I would love to at least close on one final thought on AI.

Jason Clinton: It's very much the case that if you look at the data, On the amount of training that's gone into models and there's a great website for this our world and data has a has plots you can go back 70 years the perceptron and just draw a line to the total amount of compute that's gone into AI models and it's 4x year over year increase every single year.

Going back 70 years. No, I'm not a betting person, but I'm guessing that line is going to hold for the next at least three years based on everything that I'm seeing.

Caleb Sima: Isn't that why we're putting 30 billion into data centers?

Jason Clinton: And so I really think that as a decision maker in an organization, if you're not looking at that line and skating to where that line is going to be when your policy or your product launches, you're going to be a little bit behind the curve.

And so it's important that on an exponential curve like this, that we think about the future and not what's today. [01:13:00] Make sure that as you're thinking about your internal strategy, you're thinking about a model that is 4X, 16X. More compute has gone into it more capable. And what does that mean for your enterprise strategy?

And what does that mean for your product?

Caleb Sima: I would love to do just a session on three years from now. Oh, yeah, I can already see that's all it is. It's only let's talk about where this is in three years and what we think it's going to

Ashish Rajan: be Sounds like a , part three now. Like you have part two in the middle as well.

Yeah. And you can reach me on LinkedIn. I'll put the link in the show notes as well.

Kristy Hornland: From my perspective, a lot of us came up and the idea of, Hey, you're at a place where you're a knowledge gatherer. You're not the one building all of these things initially. I think there's going to be a shift into where really solution, say, more of a solution, say, architect where we are bringing solutions and fitting them into the right places rather than being at the knowledge gatherer step.

And so like switching your mindset around the [01:14:00] resources that you have today when people think about what's my role? It's like your role is actually going to be taking finally, you've got into solutions that are coming and now where is the right place to slot these things in. And that's a, I think, major shift that we've just seen slowly evolving.

Vijay Bolina: Thanks for having us. This has been fun. You guys could just ping me on LinkedIn if you want to chat. I'm also on X, I think going forward, there's going to be a lot of excitement because there's going to be new capabilities. Models are going to continue to improve as Jason captured quite nicely.

I think the ways that we experience this technology is going to vastly change as well. Right now we have text interfaces and those are cool. We have APIs and those are cool. And there's a lot of application for that will continue to unlock over the next few years, especially even with what we see right now.

But I do think that there's going to be other modalities, other types of deployment settings that I think are going to be interesting. We are starting to see a little bit more in the robotics space. Google I O announced Astra. which is an intelligent assistant, and we've been exploring what that could look like.

What we've [01:15:00] concluded, at least for now, is that this class of technology is going to be quite useful for us in an individual setting. Having more contextual understanding and personalization is going to be extremely interesting, and there's going to be some novel ideas in that space. Memory personalization are going to be extremely exciting

Caleb Sima: The ability to understand runtime.

Yeah, like real time data like that would be . Training. Yes.

Vijay Bolina: Yeah, and you'll we'll have in an enterprise setting I think we're going to see a lot of exciting applications in that sense where you're going to have these systems that are able to reason around what's happening within your organization much deeper than a pretty good integrated ERP system.

And, there's some vendors out here that have been thinking a lot about this space, generally, but on the personal front, I think how we experience this technology in a personal setting is going to be quite interesting over the next few months and maybe years to come.

Ashish Rajan: Thank you so much for listening to that episode of AI Cybersecurity podcast.

If you are wondering why aren't we covering all topics, because maybe the field is evolving too much. [01:16:00] too quickly. So we may not even know some of the topics we have not covered. If you know of a topic that we should cover on AI Cybersecurity Podcast or someone we should bring as a guest, definitely email us on info at cloudsecuritypodcast. tv, which reminds me, we have a sister podcast called Cloud Security Podcast, where we talk about everything cloud security with leaders, similar to the AI cybersecurity conversation. We focus on cloud security specifically in the public cloud environment at cloudsecuritypodcast.tv. Which if you find helpful, definitely check out www. cloudsecuritypodcast. tv. Otherwise, I will look forward to seeing you on the next episode of AI Cybersecurity podcast. Have a great one. Peace.

‍