Episode #133: Moving to Serverless Safely with Jeff Williams
April 18, 2022 • 50 minutes
On this episode, Jeremy and Rebecca chat with Jeff Williams from Contrast Security about the creation of the OWASP Top 10 and how it applies to serverless applications, the shared responsibility model and how serverless goes even further, the current security risks and vulnerabilities, and so much more.
About Jeff Williams
Jeff brings more than 20 years of security leadership experience as Co-Founder and Chief Technology Officer of Contrast. Previously, Jeff was Co-Founder and Chief Executive Officer of Aspect Security, a successful and innovative application security consulting company acquired by Ernst & Young. Jeff is also a founder and major contributor to OWASP, where he served as Global Chairman for eight years and created the OWASP Top 10, OWASP Enterprise Security API, OWASP Application Security Verification Standard, XSS Prevention Cheat Sheet, and many other widely adopted free and open projects. Jeff has a BA from the University of Virginia, an MA from George Mason, and a JD from Georgetown.
Transcript
Jeremy: Hi everyone. I'm Jeremy Daly.
Rebecca: And I'm Rebecca Marshburn.
Jeremy: And this is Serverless Chats. Hey Rebecca, how are you doing today?
Rebecca: You know what, Jeremy? I'm doing really well. You and I were both up very late and I noticed this because I went on Twitter last night or this morning let's say, and it was 2:00 AM and you had just tweeted. I know that feeling of preparing for launches, of doing launch things, of getting really excited. Common Room is moving in that direction too but yours just happened. Launch, launch, launch, launch, launch, launch, launch. Tell us about it.
Jeremy: Well, we just launched Serverless Cloud on Product Hunt today. When this episode comes out, it'll have been a few weeks ago at this point but yeah, it was super exciting to get that up and running. Of course, launches are crazy. I'll tell you one thing though, one thing that we think about with Serverless Cloud, and this is something you should be thinking about everywhere is security. We don't talk about Serverless security very often. I used to have a section on my newsletter called Serverless security, especially when there was some early companies like Protego and Ory Segal's company there, PureSec. There was a lot of articles about server security back then, and then those companies got acquired and those articles all went away. It seems like there's not a lot on server security anymore. Maybe that's a good thing because it's already built in. Maybe it's not a good thing because we're not talking about it and we should be. Anyways, our guest today is going to talk about serverless security. Rebecca, could you introduce him for us?
Rebecca: Absolutely. Our guest today is co-founder and chief technology officer at Contrast Security, Jeff Williams. He's here to talk about what I think sometimes a lot of people... It's like those unsexy parts, they're like, "Oh, it's like the mail system." They're just like, "Oh, it just works." You're like, "Well, when it doesn't work, you notice and it's actually really bad." It's so important. I'm so excited to have someone who's focused on these parts that I think a lot of people when it works, people don't think about it, but it's so important that it works. Hey Jeff, thank you so much for joining us.
Jeff: Thanks, Rebecca. It's great to be here. I got to say I think that security, when it works right, is super sexy. Let's just start there.
Rebecca: Yes. That's going to be the name of this episode, security when it works right... It's just security is super sexy, period. Jeff, why don't you tell the audience a bit about yourself and what Contrast Security does?
Jeff: Yeah, sure. Again, my name's Jeff Williams. I'm CTO and co-founder at Contrast, and I've been in application security since there was application security. Actually, I started one of the world's first application security teams at Exodus Communications, which was a big data center company in the dot com boom era. It totally went bankrupt later. We built a great practice doing security on all the dot com kinds of major websites and helping them secure their code. Actually, quick side story. It was GE came to us and said, "Hey, we love your data centers but we want to make sure every line of code is secure before it goes on the internet." My management fell all over them. They're like, "Yep, we can do that. No problem." They were like, "Jeff, you got to go figure it out." We built a fantastic team and actually when Exodus went bankrupt, we spun out a consulting company and we kept going for about 12 more years selling security services, doing code review and pen testing and training and AppSec programs and just working with development teams on security.
Jeff: That's how we got there. Then we had an idea... I got involved with OWASP along the way and that was really good for Aspect, my consulting company. We participated a lot there. Instead of doing traditional marketing, you know how security vendors are annoying with their marketing? Instead of doing that, what we did was we actually said, "Let's spend that money on doing opensource projects and contributing something back to the community." I wrote the OWASP Top 10, we built WebGoats, we built ESAPI and the ASVS, all these things that are now part of the AppSec world, we built those and contributed them for free, but it opened the door for us to talk to big companies and help them with their AppSec programs.
Jeff: I really love that way of doing business. Along the way, we invented a new way of doing application security analysis from the inside out, more like a profiler or a debugger than a traditional outside-in scanner. It turns out when you do security from inside the running application, you get way more context and you can be way more accurate and way faster. That's what developers need. That's what Contrast does. Your whole platform to support all the things that you need to do to make sure that your code comes out secure.
Rebecca: Before Jeremy asked you anything, I'm going to say, that does sound pretty sexy.
Jeff: It is really exciting when it works. People have called it magic. I love that too.
Jeremy: All right. We might be getting off topic with the sexy talk here, but let's go back to OWASP for a second here. OWASP is something that... It's really funny. I've worked with so many developers over my career and just amazing the basic security things that people don't know. The OWASP Top 10 is such a great resource because again, it just lists the top 10 risks or vulnerabilities that most applications on the web will face. Let's start there just in case users don't know because AppSec is a hugely important... Werner says all the time, security from day one or you should think security first or whatever. I just don't think a lot of people do. Let's level-set the audience here. If you haven't heard of the OWASP Top 10, why don't you just quickly explain... You don't maybe have to give all of them but just maybe hit the big points of what that's all about.
Jeff: Yeah, sure. OWASP is interesting because its whole mission is to make applications security accessible to everyone as opposed to... When we started OWASP, all that knowledge was locked up in the heads of a few consultants that did this work. We said, "No, we got to make all of that public. We got to make it known or we're never going to make any progress here. The OWASP Top 10 was very instrumental in doing that. Basically, me and a few of the folks at Aspect, we said, "Let's just write down the biggest risks." In the early days, we did it all just from our knowledge of the industry. At that time, it included things like authentication, access control, input validation, that covers a bunch of injection kinds of flaws, some things about cookies and a few other risks, some encryption things, but pretty straightforward stuff, basic security kinds of vulnerabilities.
Jeff: Here's the weird thing. It's been 20 years now. We did the first one in 2002. Yeah, 20 years. It's unfortunately still basically the same stuff in the OWASP Top 10. I had this idea when we started it like, "Hey, let's put these 10 out. We'll probably knock off a few of these and then next year or a few years from now, we'll move the bar up and we'll bring the whole industry along with us." But sometimes when you set out to build a floor, you actually end up building a ceiling. It's unfortunate, but we should be way farther down the road than we are in AppSec. I still feel like we're struggling with basic things. Why is SQL injection still a problem? It's still a problem in serverless. We can talk about that. But it's absolutely crazy.
Jeremy: Right. Well, I think the other thing too and this is interesting as... I want to talk a little bit about where developers fit in now versus maybe the operations team, the people that were doing security before because with more traditional systems, you talked about these scanners that were helping to mitigate... Even WAFs and things like that will help mitigate some of these SQL injection attacks and some of the other things that happen. It's always these new batch of developers that keep coming in. You always have fresh eyes or new people that don't know some of these specific things. Maybe thinking about moving from more traditional of WAFs and other things like that were in place before that would help mitigate some of that. What are some of the new vulnerabilities moving to the cloud where some of that stuff has shifted away from that, especially maybe in serverless where you might not have the benefit of having all of those network scanners and some of those attacks can make it to your code?
Jeff: We are exactly right. What I call them is traditional or legacy kinds of tools that work from the outside in. Those tools aren't fantastic because they don't have enough context to make smart decisions. A WAF, if you think about it, it's sitting at the network perimeter and it's trying to look at HTTP traffic and decide whether there's attacks in there. Well, I don't know if you've looked at any HTTP traffic recently but it ain't simple anymore. It's not name value pairs, it's JSON and XML and serialized objects and web socket and all kinds of stuff. You can't just look at it and go, "Oh, that's an attack." You don't have enough context. You don't know what that data is used for. It's the same thing with vulnerability detection tools like scanners that you might run on your traditional monolithic web app.
Jeff: That's one thing, but try and run it on a set of APIs or a set of serverless functions and it's a whole different ballgame. You don't have enough context to invoke the function right and understand the response. These outside-in approaches don't work very well. Even static analysis doesn't work very well on serverless because you don't have enough context because instead of having one big thing that you're analyzing, now, you've got a thousand serverless functions sprawled all over the place. The static tool can't assemble those and figure out what it looks like at runtime. You can't analyze it. That's the problem. Now, the solution going forward is what I tried to introduce before is you got to get more context.
Jeff: You got to get inside the environment. For serverless, it's you got to get inside the serverless environment, understand all the pieces that are there, the S3 buckets and the functions and the data stores and so on. You got to see how they interact and you've got to build a picture that then you can analyze for security flaws.
Rebecca: I want to get your take on... We love to get our guest's vision of the future of serverless applications and usually we maybe save that question to the end, we talk about it and then we're like, "Okay. What's your vision for the future?" But we actually thought it'd be a really great way to work backwards from what your vision of the future of serverless applications is and then work backwards from what we need to do in order to achieve and get to that vision. Maybe you could describe a little bit about what your vision for that future is and then we can get into the current security risk and how we build toward that future.
Jeff: That's a fantastic question. There's a huge amount of interest in serverless. Some of our customers are all in. We've got customers with a million serverless functions. We've also got a lot of customers that are dipping their toe into it, they're building a few pieces in serverless environments. But I'm almost 100% sure that the future is really complicated. It's applications that are split up into lots of different pieces, some monolith, some traditional APIs, some serverless, multi-cloud. We're going across different environments, containers.
Jeremy: Edge functions.
Jeff: Exactly. It's really going to be a complex environment. Like I said, to do security analysis, you're going to have to understand all the pieces and how they fit together. That also includes opensource libraries and frameworks. As part of those pieces, each one has opensource in it. You got to understand that. In terms of how we do security in that environment, we've got to have sensors all over the place. We've got to have sensors that are reporting back, "Hey, here's what pieces are in this environment. Here's how they connect together. Here's data about what data they're processing and what they do with it." Show me all the backend connections. You're going to have to build that visibility in order to do the security analysis.
Jeff: How do we get there? Well, it's going to be piecemeal... If I had to put a name on it, I think ultimately, it's transparency or observability but for security purposes across a complex environment. We're in the early days of this. There's very little real observability in the pieces of our software ecosystem today. Contrast, we're not all the way there. We're building out pieces of that visibility. I think we're way ahead of the curve here but it's still got a ways to go.
Jeremy: I love that idea of the inside out because I think there are too many traditional vendors now that are looking at this of saying like, "We can just scan packets and all that kind of stuff." But if you think about it, with a JSON web token or whatever, it's encrypted. Then you decode it in your application code. You need the keys do that. Where does that get decoded? Where do you look at that? You said you got people uploading to buckets. I did a proof of concept where you upload to an S3 bucket, it fires off a lambda function and then tries to insert the name of the file into a SQL database or to MySQL.
Jeremy: When it does that, if you don't sanitize the input, you can name the file name, anything you want. You could do semicolon delete all or whatever [crosstalk 00:14:02] recommend is right. Again, it's unlikely that some of that stuff will happen, but really there should be a focus on making sure that these get mitigated at the application level as opposed to the network level. To dive in a little bit more, because this is something that has always fascinated me is you think about the shared responsibility model of cloud, and there's already a bunch of things that cloud does for you that you don't have to worry about. I don't have to worry about network protection. Somebody's doing that for me.
Jeremy: As we move closer to serverless or we get to things like managed services or functions as a service, now the run times are managed for me. In some cases, everything's managed for me. Really, it just comes down to the application itself, which I think is where Contrast is really focusing. But just maybe as a general... Again, another level set here is what are some of those serverless security benefits that you just get out of the box?
Jeff: Yeah, that's a good question. I think there's three key areas in serverless security that we got to focus on. The first is the security of your code and the service environment doesn't really help you with that. You can introduce an XXZ or unsafe deserialization flaw in your serverless code. You got to deal with that. There's also the opensource piece. You include a bunch of opensource functions, you got to deal with that. The platform's not going to do that for you. One of the key differences between traditional monolith web environments and serverless is that you can control the identity and the access management permissions of each of your functions. That's actually really powerful. It limits what that code can do. It's a powerful mechanism that's present in serverless environments that isn't in traditional environments.
Jeff: There's no way to do that in a traditional application. Your whole app gets basically root privilege, but with serverless functions, you can narrow it down really carefully. Now, below that, everything comes with your serverless environment like securing the operating system level, the network level. All that happens automatically. There's a million things you don't have to do. I think that's why we're seeing such rapid adoption of serverless because there's a lot of stuff that just comes for free. You want it to be elastic, great. Happens automatically, ultimate scale. I think it's interesting how big bets people are placing on serverless. I heard recently that AWS spends half of their marketing budget on serverless, which is obviously massive. I think that's a massive bet and you're probably not going to do well betting against AWS in this market.
Jeremy: You're 100% right about the IAM stuff, because that is just one of the things where I try to tell that there's so many benefits to isolating a little snippet of code into its own function. One of the things again with SQL injection, let's just take that for example, where that becomes such a problem is because you have a bunch of functions that might be signing up new users. They do have to do an update or an insert. Then you have other functions that might just be reading things. You're just pulling back products' data or whatever that is. Then you have a few of those functions that are like, "Well, we do need to delete a record here or we need to delete this user or we need to manipulate it some way."
Jeremy: The problem is that when you give a container or a VM permission to use that, it has full permission to do whatever you allowed it to do within that database and they can do anything, whereas with serverless... This is a real risk. I want to talk about these risks in a minute. I think Rebecca's going to ask you a question about that but the there's a real risk of using opensource packages and having somebody manipulate that, especially when you have thousands of functions and all of these automated build services that are constantly downloading the latest versions if you're not locking package versions and things like that. There's all kinds of issues there. If you download colors or something like that that somebody decided to not even exploit, it wasn't exploited.
Jeremy: The developer did it himself. Something like that that could access your full database, one included package that maybe helps add a shim or a padding to a text string could go in and delete everything in your database. It's just crazy to think about the amount of risk that you can open yourself up to when you're using potentially untrusted third-party packages.
Jeff: It's complicated because opensource software is in general, in my opinion, really pretty good. I've looked at a lot of code over the years. I did tons of manual code review. Enterprise application code that companies have written for themselves is often not as good as what you see in opensource packages. There's some advantages in working in public where you don't want to embarrass yourself with crummy code. I don't think that anyone should take away the message like we got to stop using opensource code. That would be-
Jeremy: Right. Oh, definitely not. Yeah.
Jeff: But you got to be careful and smart about how you use opensource. It's not free. When you use an opensource library, you're saving a bunch of work up front but you're taking on an obligation to keep that opensource up to date if there's a known vulnerably gets discovered, you need to be able to replace it quickly and get to a safe version or use a technology like RASP, Runtime Application Self Protection that will prevent those vulnerabilities from being exploited, ensure that inside-out approach. But you have to have a strategy for dealing with source because guaranteed, there's going to be another Log4Shell. It could be today. You never know. You got to have that infrastructure in place so that you can very quickly respond.
Rebecca: That's such a humbling reminder, just like PSA. Opensource is not free. That's such a true statement or else it would be just called free software. It's not. It's not just free code. It's opensource and there's a distinction between the names for a very real reason. I think this underscores... Like you said, the future is really complicated when you're talking about the vision of serverless. This idea of complications has come up a few different times. Then you talked about three key components of what needs to be secure is security of your code, security of opensource projects, and then at that function level, those IAM permissions. How would you describe at the most basic level, the current security risks and vulnerabilities in serverless itself across those three components?
Jeff: Let's just go through those three categories. It's not too hard, actually. The first category is about your code and you need to think about... Really, it's the same stuff as in the OWASP Top 10 for the most part. It's things like authentication, access control, encryption, injection. There's a whole range of injection problems that you really need to be aware of. It's not just the traditional SQL injection and command injection. There's also more complicated things. Like SSRF is really interesting in a server environment that's tricking a serverless function into issuing an arbitrary HTTP request on your behalf so that you can... Using that kind of attack, you can drill through the layers of a serverless environment.
Jeff: Then there's problems handling things like XML and JSON. Maybe you've got an unsafe deserialization flaw. There's the same kinds of flaws that we see in every environment since back before 2002, they show up in serverless environments. They're always translated a little bit. I've seen a few generations of this now is we've figured out how to build monolith security after 10 years of doing it, we got okay at that. Then it's all APIs. Now, we don't [crosstalk 00:22:11].
Rebecca: Surprise.
Jeff: Right. I mean, I feel like everyone's like, "Yeah, new environment. We don't have to worry about all the security crap." Then it's 10 years before we figure out how to secure APIs. Now we've got serverless and I don't want that YOLO to happen again. It's still important to focus on security. Frankly, we should be better about seeing these risks before we add them to our applications but we're not. We're not very good at seeing things in advance, even though there's plenty of people going, "Hey, that could be insecure. Hey, that could be insecure." We don't do anything about it as a culture, as a world until there's a big breach. That's just dumb. That's like, "Well, I'm not going to wear seat belts until I crash my car." We're not doing that. Although, I didn't wear a helmet until I really had a bad wreck. I guess there's that.
Jeremy: Well, I actually started wearing a ski helmet. I was very early wearing a ski helmet. It was probably one in 50 people would wear ski helmets when I first started wearing one maybe 15 something years ago. Now, everybody wears them and I'm like, "Well, I was ahead of the curve there."
Rebecca: Turns out they're a great idea.
Jeremy: That's the thing is I care about security and safety but so I'm actually curious about this question because I'm almost all always in the AWS ecosystem, but you've got CloudFlare and you've got Fastly and we've got Google Cloud and Azure and every other cloud that's popping up. I'm really curious about, the shared responsibility model is going to be different in each one of these slightly. For the most part, there'll be a lot of commonalities there, but I'm curious if security, especially when it comes to serverless is going to, I don't know, maybe vary across those different clouds and different implementations.
Jeremy: I guess the question I have is more about something like maybe Kubernetes, for example. If you have a team running Kubernetes and you are managing and patching all of the 50 different processes that you have to run to make that work to give your developers a serverless experience, how much more responsibility does that add on to your team, whereas relying on the cloud in general or relying on the managed services from Azure or from Google or from AWS, is there a big shift in responsibility there or do you still see that as being, I guess, a good level that you can provide at least for the developer?
Jeff: It's a little complicated question because everybody builds software differently. I make this analogy a lot. I say, look, imagine you're thinking about the different ways you might have a cake company. They make cakes, you might have a small mom-and-pop shop with some specialized equipment that makes wedding cakes. At the other end of the scale, you might have tasty cake factory, which is super automated and just pumping out plastic packages, very different businesses, both are fine. They're both profitable healthy companies but they do things incredibly differently. I don't think there's a right answer for any particular company on how they should build software. As a security person, it's sometimes really easy to go in and say, "Well, you should do this." Well, you probably really need to understand who you're talking to and how they build software.
Jeff: There's a lot of talk about, "Well, you need to shift left." Well, I don't know. Maybe in your software company, it makes sense to do security... Maybe you should shift right. There's a lot of good reasons for doing that at least if you're talking about protecting production assets. Maybe you want to do more in QA because doing it shifting left, there's too much noise doing it early in the process when you don't have a running system. Maybe you need to actually put it all together and see the serverless functions and how they interact. That's, I think, important. To answer your question, I think the north star is automate security. As you build your pipelines, as you build your 50 processes to deploy your Kubernetes environment, as you build your process to deploy serverless, think about how much can you automate.
Jeff: The answer is, what you want at the end is you want confidence, continuous confidence that all the code you've deployed is secure enough for what you're doing. You set the bar on what secure enough means because only you understand your risk. If you're building the lunchtime cafeteria menu, yeah, who cares? It's not super attackable. You're building the next swift processing system for financial transactions, you've got to be really sure. I think the key is to get security 100% automated so you can make a code change, it can go through a pipeline of some sort and you end up with that code running in production and you end up with the evidence that says, "Hey, we tested this. We're sure that we got authentication right. We tested it for injection flaws. We tested it for a variety of different things that could go wrong."
Jeff: Until you do that, you're never going to be able to have really DevSecOps because if you have to have people in the critical path, then you're going to wreck your DevSecOps. Because it takes people a lot of time to do things. It might take a week for them to do a pen test or to review the output of a static analysis tool that generated a thousand false positives, or whatever. But that's going to prevent you from pushing that code into production. Ultimately, dev teams, they just want to go fast. That's how they get maximum innovation for their companies. That's what makes companies winners in their sector is being the best at software.
Jeremy: Right. I think that security needs to introduce just the right amount of friction to make it hard to do the bad things but easy to do the easy things. Also, just it's almost lunchtime where I am. You mentioned cake, you mentioned cafeteria and it just got me thinking about that. But Rebecca, I don't think we've ever talked about cake on this show, have we?
Rebecca: No, but I was also delighted. It's morning time here and I'm still like, "Ooh. All right. Taste a cake."
Jeremy: I don't know.
Jeff: Time for breakfast. It's fine.
Jeremy: Do you have a favorite cake, Jeff? What's your favorite type of cake?
Jeff: It's all chocolate for me.
Jeremy: Chocolate, right.
Jeff: Chocolate, 100%.
Jeremy: I'm a big carrot cake fan.
Jeff: Chocolate with [inaudible 00:28:21].
Jeremy: Ooh, chocolate with [inaudible 00:28:23].
Jeff: It's pretty good. What about red velvet?
Jeremy: I can do red velvet. What about you, Rebecca?
Rebecca: I'm with Jeff here on the chocolate. Also, could get down with carrot cake. Red velvet, I always leave a little bit... I have a little bit more to be desired every time. I have an expectation of red velvet and it's usually right below the bar for me.
Jeremy: Interesting.
Jeff: Yeah, thank you for that.
Jeremy: Well, hopefully we didn't lose a bunch of listeners with that, but anyways, Rebecca, would you like to continue?
Rebecca: The idea of speed is simple but to maybe continue on this theme, going fast is complicated. You were saying how you want developers to go fast but enabling developers to go fast, there's actually all these things that go into it. I'm wondering how much of your security posture ends up becoming part of the developer's responsibility, security at the application level. How do you think about enabling developers to go fast but there are these new moments and new paradigms and ways of having to shift your thinking in terms of security at the application level, and then how you mitigate that balance between developer overhead and that ability to go fast?
Jeff: A fantastic question, as usual. Here's the way I think about it is if you go into a company that hasn't done a lot of AppSec work in the past and you add a security tool to start checking for vulnerabilities, you'll probably find some and you create a backlog. I want to get past the backlog. You've got to work through that but what you should be optimizing for is the future situation when you've eliminated the backlog and now you're just talking about the new vulnerabilities that get introduced with each change. That's what we want to optimize for because that is going to drive the future of our pipeline. One thing you can do is you can try to accelerate the mean time to remediate. If you reduce that time, it means you're getting vulnerabilities to the people that need them faster with the context that they need in order to fix them.
Jeff: You can enable developers to do their own fixing. We've been able to push this to really fast. You can get feedback to developers within seconds and all the information they need. They say, "Oh, that query I just created, I forgot to parameterize the parameters. Let's make that change, check in clean code." Then you go clean through the pipeline. But you can even do better than that. What you really want to do is... I call it optimizing for learning, is ultimately finding and fixing stuff is great but if you're finding and fixing a lot of stuff, it still takes time. What you want to do is reduce the number of things that you have to find and fix. That means learning has to happen.
Jeff: What you're really trying to do is teach developers not to make that SQL injection mistake in the first place or use the IAM permissions the right way in the first place or not use insecure libraries in the first place. The right way to do that is to give them really great feedback fast so that they have all that information. Developers learn by doing. If you give them feedback months later... Side note, the average time to remediate vulnerabilities if you're using a static analysis tool is 290 days. That's from one of our competitors, but that's way too long.
Rebecca: Wow. That is shocking.
Jeff: We've reduced that to about a week across all of our customers, across many tens of thousands of applications. That's exciting. I think we can push it even more. I think we can get vulnerabilities remediated within a day. I think it's reasonable. It's a stretch goal, but clearly 290 days is way too long. There is zero learning going on at that rate. If you can get it down to a day, developers are going to learn, they're going to stop making those mistakes and maybe we can stamp out some of these damn vulnerabilities in the OWASP Top 10 for 20 years. I think optimizing for learning is the key.
Jeremy: I can't remember what I coded yesterday. If it's something that's 290 days old, I would have no idea from it. This whole optimized learning thing, this is super important because we talk about learning and education with serverless all the time. It's a new paradigm. It's a different way to think about stuff. I get it that you got to go out there, you got to find resources. You got to go through the process, you got to build things yourself. You got to learn how to do this stuff. Security has been around for so long. Like you said with the Top 10, this hasn't changed. It's the same stuff over and over and over again but people still aren't learning and that's potentially because we haven't asked people to directly, especially with ops teams and we talked a little bit about that.
Jeremy: But I'm curious, besides this idea of more instant feedback, which is great by the way, if you write some code and the next thing you know, you start getting alerts saying, "Oh, by the way, there might be a SQL injection attack or a SQL injection vulnerability here or maybe remote execution or this is over-permissioned or things like that." That's all super helpful but what other paths can developers take to learn more about AppSec beyond becoming a security expert? I hate to think about shortcuts, but besides reading the Top 10 and saying, "Okay, I get it," what are some of the paths maybe that devs and anybody listening right now who are building applications, you need to be thinking about security, how do you learn more about it?
Jeff: A lot of people will say, "Well, you'd get training and trainings. We have to train..." We're not going to train our way out of this. Mea culpa here, I did instructor-led training for many years, probably 5,000 developers. I've created an e-learning course that was used by lots more people. The problem is they're always a little bit generic because they don't know exactly what environment I'm using, exactly what libraries I'm using, exactly what IDE I'm using. It's all custom to me. Then I have to do this translation from this generic training that says, "Don't do SQL injection. Use parameterized queries," to something that works in my environment and my RM.
Jeff: I think it's actually better to get a great tool that's compatible with your framework and your libraries and your environment that gives you that feedback on exactly the code that you're working, call it just in time training. You get feedback on the code you're working on and that is the best way to learn because we can't teach 30 million developers in the world everything about security. There's just no way. OAS Pata, their big hairy audacious goal for last year was to train 2,000 developers. I'm like, "Who cares?"
Jeremy: It's a drop in the bucket.
Jeff: It's 2,000 out of 30 million. It's not even a rounding area. There's no way we're going to get there that way. We've got to be able to give developers feedback in a way that they are willing to consume it.
Jeremy: I'm actually curious. If companies are moving to serverless, and we talk about this all the time where it's like, what do you have to do? Of course, you look at Liberty Mutual and you look at Lego engineering and some of these other companies that create serverless developer enablement teams where they basically they have to put in best practices, they have to bake insecurity, they have to create patterns and do all these things to make sure that they're following all the compliance and things like that. That's a huge step, especially for smaller companies or startups or things like that. Maybe what some advice that you can give for companies that actually want to start using serverless, but also maybe aren't as good with AppSec. How do companies go from... Or how do they adopt serverless and do it safely?
Jeff: I think the big picture goal is to turn security into code. If you can make it something that runs as part of your pipeline and checks your serverless functions, it has to be automated. You've got to turn it into code somehow. You're right. They make their own policies. They say, "Hey, here's how we're going to use this framework. Here's how we're going to use this library." You want to check those things but you got to automate it somehow. Our approach is to provide a platform that allows you to express those rules in an automatable format. I think that's the goal is if the bottom-up is how good is your platform for turning security into code and then the top-down is well, what are the things that we need to figure out to turn into code and make part of the automated pipeline?
Jeff: When you bring those together, then you can have a pipeline that goes really fast into production. But it's got to be accurate. Normally, my rant is about accuracy. Accuracy and security tools is the key. If they're not accurate, you have to have people involved. That's the only way to figure out if it's a false positive. That's the only way to figure out if you missed something and it's a false negative. You have to have accuracy because then if it's accurate, you can give it right to developers with confidence that they're not going to quit, right or spend three days figuring out whether it's a false positive only to walk away because it's annoying. It's true of code checking tools. It's true of library checking tools. They call them SCA tools or Software Composition Analysis tools to check your libraries.
Jeff: It's true of the permissions thing. We did a cool thing with our IAM permissions checking. We figure out what permissions each function needs by watching it run. We watch what it asks for and then we look at the policy that's defined and said, "Wow, you've massively over-defined the policy. You've got a wild card in here if you've got a dozen permissions and you only needed this one." Then we just rewrite it for you and give you the new policy to just paste in on your share less functions. It's a really fast way of giving perfect feedback to developers. It's got accuracy, it's got speed, and it's got automatability, that's the recipe for success here.
Jeremy: Never use star permissions. The one that always gets me is when people use DynamoDB, colon, star. You're like, "You do realize that you could delete the table with that? Or you can change all kinds of things about it." Yeah, star permissions, very, very bad.
Jeff: It's just really hard to figure out what the permissions are supposed to be because you don't know what thing... What is your library using? It's hard to go figure out. We actually watch the function run and calculate the appropriate permissions. We think it's really exciting.
Rebecca: Security is not the place to YOLO. There was a moment in time, actually, a lot of us are still living in this moment where infrastructure was not as code, but now is this idea of infrastructure as code. I think it makes a ton of sense that security as code. That tracks, that definitely tracks. Jeff, you keep saying, we, but by we, you mean Contrast and I want to allow you to talk a little bit more about Contrast. I think a fun way to do this might be spoiler alert for our listeners, we got to talk to Jeff a couple of minutes before we actually started recording and you were like, "Security belongs everywhere, not just in the software that we imagine it to be in, not just in the applications that we associate with digital technology." Maybe you could share that fun, surprising story about one of your customers about this idea of security belonging everywhere and how Contrast brings that to fruition.
Jeff: Yeah. Thanks. That's a great opportunity. I appreciate it. One of our customers is a company that you wouldn't think would have a lot of security needs. They build power tools. They're one of the larger power tool companies. When I first started, we were talking to them, I was like, "Where are we going to secure? Their website or something?" Actually, it turns out that modern power tools are IOT devices. They talk to serverless functions. They've got dashboards. They've got mobile apps. They've got all kinds of pieces of technology. We went in and started talking to them about our platform and really our platform does three major things. We help on the custom code problem. We've got several different technologies. We've got a technology you'd use on typical web apps.
Jeff: We've got one on serverless. We've got a static tool so you can cover your whole portfolio of applications. We work across Java, .Net, .Net Core, Node, Ruby, Python, Go, Scala, Kotlin, and PHP. We've probably got you covered in terms of languages. Then we've got our library analysis piece, our software composition analysis piece. We actually include that in the first piece. I said it differently because everyone else sells it separately but it's weird because it's one problem. You're analyzing one application that's got some custom code and some libraries all mixed up together. Like I've been talking about the whole time, you have to analyze the whole thing. We analyze libraries. At the same time, we analyze the custom code and it allows us to do things other companies can't. We can tell you which of your opensource libraries are actually used by the application.
Jeff: It turns out, more than half are never used, never invoked at all. They're just extra baggage in the trunk. We want to focus you on the libraries that matter, the ones that you really need to update. We generate S bombs and all that stuff. Then the last piece is a runtime protection piece. This is a technology you can add to web apps or web APIs. We don't have it for serverless yet, but it prevents vulnerabilities in those applications from being exploited, whether they're in custom code or in library code. For instance, when Log4Shell happened in December, we quickly checked and were like, "Yep. Our runtime protection product, it's called Contrast Protect, it prevented that vulnerability from being exploited because we stopped the underlying exploit. It's usually expression language injection or unsafe deserialization that was getting exploited.
Jeff: We prevent those vulnerabilities from being exploited. We got some great quotes from our customers. They were like, "Well, the teams that were using Contrast, they can go home for the weekend. Everybody else has got to stay and has got to keep searching for Log4j and trying to find it. That event really showed off all the advantages of our product. We were right there helping them find the library, helping them replace it, help them make sure that they're protected in production so they didn't have a fire drill and they could take care of it with their normal release cycle. Sorry for the dump, that's essentially what we do and how we help companies like this tool company. We come in with a platform that allows them to build their AppSec program in an automated way.
Rebecca: Don't be sorry. I asked. Thank you.
Jeremy: The world needs more serverless toasters and power tools, I think. Anyways, that's awesome. I think that, again, security is one of those things that just people don't want to talk about. They don't want to think about it. It's such an important piece to building modern applications in the cloud. Awesome work that you're doing over there at Contrast. We need to educate more people, read the OWASP Top 10, learn about those, dig deeper and check out some of the tools that are out there. Jeff, thank you so much for being on the show. It was great having you. I hope our listeners learned a lot. If people want to find out more about you or more about contrast and what you're doing, what's the best way for them to do that?
Jeff: Best way to contact me is probably LinkedIn. Connect with me. I post a lot about application security and serverless and so on. That would be a great way. I'm totally open to answering questions from folks about AppSec. Probably the best way to find out about Contrast is go to our website, we've got a bunch of resources there. We're happy to talk you about your AppSec program and helping you figure out how to get more efficient at it. I just realized we made it through this whole conversation without talking about sprawl at all which is crazy because with serverless functions, that's the big deal. I don't know what I'm thinking about but instead of having one big app, now you've got a thousand serverless functions all interacting in different ways. Maybe we can take it up on the next podcast but I just wanted to mention that you need to get some inventory system in place so that you know where all your serverless code is and what it's doing and what it's connecting to and all that.
Rebecca: Well, a little note for listeners as well, if you are fans of the artist, Keith Haring, Jeff often uses Keith Haring art in his posts that he writes on LinkedIn. I really admired that. I loved that. He also asked for feedback on his post. Jeff, I just wanted to note that because I think it's so neat. You talked about feedback a few different times. I think in the opensource world, it's so important. I love how at the top of your post, you're like, "Hey, I consider this a draft and if you have feedback, I'm going to edit it, I'm going to update it and I'm going to attribute it." Connect with Jeff on LinkedIn, read his posts, give him feedback and watch those drafts change.
Jeff: That's so fun. That's something I learned at OWASP is you make a product, you make some project and you put something out there and there's going to be a lot of haters that come back and they're like, "Oh, you blew this or you messed this up." My reaction initially was like, "Oh, that hurts my feelings." Now, it's just like, "Oh, well, fantastic. I guess you want to help? Why don't you pitch in and help me make this better?" It would be great if the whole internet was like that.
Rebecca: That is my vision for the future, serverless and internet future in general. We're all in it together moving toward not just productive for production's sake but constructive, I should say and helpful relationships.
Jeff: Have you been on Twitter lately?
Rebecca: I only follow Jeremy on Twitter.
Jeff: Good move.
Rebecca: Well, if we lived in a world where PRs were always welcome, that'd be pretty good. Anyways, Jeff, thanks again. We'll make sure we get all this contact information into the show notes. This was great.
Jeff: Thanks everybody.
