Episode #125: Configuration over Code with Eric Johnson
February 21, 2022 • 57 minutes
On this episode, Jeremy and Rebecca chat with Eric Johnson about the emergence of Step Functions within our serverless applications, bringing the developer to the cloud versus the cloud to the developer, the importance of serverless advocacy and teaching, and so much more.
Eric Johnson is a Principal Developer Advocate for Serverless Applications at Amazon Web Services and is based in Northern Colorado. Eric is a fanatic about serverless and enjoys helping developers understand how serverless technologies introduces a major paradigm shift in how they approach building and running applications at massive scale with minimal administration overhead. Prior to this, Eric has worked as a developer, solutions architect and AWS Evangelist for an AWS partner company.
Transcript
Jeremy: Hi everyone, I'm Jeremy Daly.
Rebecca: And I'm Rebecca Marshburn.
Jeremy: And this is Serverless Chat. Hey, Rebecca.
Rebecca: Hey, Jeremy. Tell me about the latest games you've been playing on your computer. I heard a little bit about Mine Sweeper, maybe some SIMS. Is this true?
Jeremy: Right. I think I was playing the original version of Chess Master. No, we were talking about ... We were trying to come up with some jokes around games and I couldn't think of any modern ones. I think the one I came up with finally was Fortnite, but I also feel like that's kind of old at this point. But that's how I feel, I just feel old. So you know what? I'm going to own it. I'm going to play some Q*bert, and maybe some Asteroids. I don't know, we'll see what happens.
Rebecca: Well, you certainly weren't alone, our guest and myself also said things like Snake or like Tetris.
Jeremy: You play on your flip phone, right? Yeah, exactly. On the old Nokia 5180 or whatever it was.
Rebecca: Exactly. The perfect brick. Well, with that in mind, do you want to introduce who our guest is today?
Jeremy: I would love to introduce our guest today. Now I have to be honest, when I heard the name of who was coming on. I'm like I love Cliffs of Dover, one of my favorite guitar songs of all time. Amazing. Then I realized this guy was from AWS and he plays the drums. But anyways, our guest today, principle developer advocate for AWS serverless applications, Eric Johnson. Great to have you here.
Eric: Ah, thank you. Good to be here. And I have to call out, you're a musician when you can call it Eric Johnson, the guitar player. Yeah, that's reaching back. Some of my favorite music ... Just today. I was actually looking for a YouTube video of mine so I typed YouTube into Eric Johnson. I don't even ... I'm like nine pages in, still Eric Johnson, the guitar player, so yeah.
Rebecca: I mean, page nine, you should feel pretty proud. That's pretty good.
Jeremy: Yeah, that's a pretty good ranking. I mean, some people get ...
Eric: Nine, I still hadn't found myself. Still haven't found it yet.
Jeremy: Right. Well, I will tell you something though, Eric Johnson, the guitar player has nothing on you when it comes to serverless.
Eric: Oh, thank you.
Jeremy: So tell us a little bit about just ... I guess, maybe be your journey into Serverless, like kind of when you arrived at AWS, what was it three and a half years ago? Four years ago now? Something like that.
Eric: Three and a half, yeah.
Jeremy: Yeah, I mean, it was just starting to grow, right? It was just ... I mean, back then, it's kind of crazy to think about it, but it's like even just three years ago, we weren't anywhere near where we are right now, so just tell us a little bit about the landscape. What excited you about serverless and sort of what it's been like over the last three and a half years there.
Eric: Sure, I'd be glad to. At the time, serverless was announced when they came with the AWS Lambda but they didn't call it serverless at the time, as you well know. It was AWS Lambda, right? And I remember I was working for a partner company, I was a solutions architect, and I was doing the whole breadth and width of AWS, which you know is no small feat. And so ... But I remember working and I remember seeing this announcement, right? About the reinvent, and I saw this announcement and just being befuddled. It was like it clicked into me almost immediately why would you do it any other way? If I can get rid of servers, if I get ... I'm just going to say it, if I can get rid of VPCs, if I can get rid of security groups, and I'll probably take some heat for this one later, and just deal with code, man, why wouldn't I?
Eric: And so at the partner company I was worth working with, I started pushing serverless. Now you have to remember the partner companies, at this particular one, we were doing infrastructure maintenance, that's where our that's where our bread and butter was. We'd come in and serverless makes that a little hard, and so we didn't do a lot of serverless. It's kind of hard for ... Well, we won't get into all that, but we were doing mainly infrastructure and management but things just made sense with serverless. So I became known as the serverless guy at this partner. You can't see it, and I don't know if there's still a video, but I'm doing air quotes for both of you. Air apostrophes.
Eric: And so I really dug into serverless, and so I started before, and Jeremy, you obviously well know serverless.com, before the frameworks came out, and that was one of the first, I was writing my own scripts for deployment, I was doing all those things. I was trying to figure out how do I zip up locally and I was automating, and so I've always loved automation. I was automating my serverless journey and how to make that work. But at that time, it really was Lambda and API gateway and S3, that's kind of what it was. I know we had SQS and those other things, but those are kind of the ones that caught my attention.
Eric: So throughout that journey at partner, I saw this position come open and I love to speak, that's kind of my ... Speaking is my thing. You wouldn't know it all the time, but listening to me, but I do love to speak. You would know it by reading my blogs. He should speak, not write. But I really fell in love with the whole idea of serverless and the logical move for me was to come to AWS. So when Chris Munns opened the position, I immediately applied because I that's all I wanted to talk about. I didn't want to talk about all the other stuff. All great services, but I'm not a huge fan of doing instances, or dare I say, even containers, I'm a fan of going straight to serverless, and so that kind of brought me to my journey here. That's the short version.
Rebecca: So I want to talk about that a little bit more. To go back in the way back machine of 3.5, three years and I believe five months ago, if I did my creeping on you just right.
Eric: That is creepy.
Rebecca: I know, right? It's one of my super powers. So as you mentioned, right? It was Lambda, API gateway, some S3, maybe and SQS, step functions was very young when you joined, [inaudible 00:05:33] was even younger, and then EventBridge did not even exist. And so how would you describe the evolution of serverless and that was then, right? In that Lambda, API gateway, S3, SQS thing, and now these different services have come up and enabled devs to perhaps do a lot more. I don't mean to ask that as a leading question, how would you describe what serverless enables developers to do now in just this three year and five month span?
Eric: So let me give you a little background real quick on step functions and Eric Johnson, right? So when I first started with AWS, it was right before reinvent. If you come to work for AWS, I really encourage you to do it right in front of a reinvent. That's no pressure there at all. And so we had this serverless, and Jeremy, you were there because I remember going watch Jeremy Daly, and others. And I remember being in the serverless after hours party, whatever it was, and I'm four or five Dr. Peppers in, so I'm loud. And so I remember standing there talking with Ben Kehoe and I was a little starstruck, not going to lie. I'm like man, those are cool suspenders.
Eric: And so we're talking and he starts asking me questions about step functions. I didn't have a clue. I knew step functions existed, I knew the concept of it, but I couldn't answer any of the questions and things like that, and I had kind of avoided it. It was too hard for me. I didn't know the state's language I did, and so I kind of avoided it. I was just do it all in Lambda, and you know where I've come to, Jeremy, so you see this kind of change out. But it was like, "Look, just use a Lambda function. It can do everything. It is the Swiss army knife of AWS." So I remember again, with Ben being like, "I don't know, let me go get someone."
Eric: So let me roll that forward three and a half years, and where I've come to you was ... And I had this debate with Rob Sutter, and he told me, "Eric, every serverless application should be step functions." And I was like, "No, that's a little drastic, Rob. That's a little drastic." He said, "No, no. Nope. I'm telling you, everything should." I tell you, he almost has me one over now. And here's the things that's changed, and then I'll get to your question. I like to answer questions as long as I can, and so ...
Rebecca: Tell us a story. Get in.
Eric: Yeah, yeah. I'm a storyteller. And so I remember when synchronous execution came out, that was a huge change for step functions because now I'm not having to do everything asynchronous. Now asynchronous is good and I'll always lean towards asynchronous, but not every application could be asynchronous. Sometimes you need that instant response and you couldn't do that with step functions. So therefore, no, Rob Sutter, not every serverless application should be in step functions. But when they release the synchronous execution, that is a game changer because you're able to do a lot of things and get an answer back. So you get a mix of synchronous and asynchronous.
Eric: So let's step forward a little bit. I'm so excited, I'm running out of breath, or I'm just fat, one of the two, so I'm like ah. So come forward a little bit, then we introduce things like the workflow studio, and so the Workflow Studio ... And that answers that for me, I'm going to be really honest, with the Amazon states language, I had a hard time figuring out is it one dollar sign? Is it two dollar signs? And how do I line those up? But the Workflow Studio helps do that. Honestly, I have to say, it's probably one of the best UIs I've ever seen anywhere. If you're not familiar with one I'm talking about, this is where you go into step functions on the console, and I'm always a fan of IAC, but this is one of those few places where I say start in the console.
Eric: So you come to there and you can design your whole step functions, set up everything you need. And then finally, at Christmas time, at reinvent time, which was Christmas, we announced the SDK integration. And so what this does is instead of just having the custom connectors that are in step functions to some services, and then for everything else, you would use a Lambda to kind of transport that data back and forth. We now add the ability to connect to almost any service in AWS if it has an API or an SDK integration. And there's some caveats on those things, like we were looking at one where streaming is involved and step functions doesn't handle streaming responses and things like that. But for the most part, it handles most of the SDK.
Eric: So now let me get to the answer to your question, Rebecca. So with all that under the hood and that kind of power, what happens is we're able to take a lot of what we were doing in Lambda functions, SDK calls. I mean, if you look at Lambda code, how much of it is business logic and how much of it is just calling SDKs and moving data around and how much of it is just transporting. And if you're looking at a lot of transporting and calling SDKs, then get out of it. Go over to step functions because then we get into, Jeremy, you've heard me say a lot, either storage first or more to the point, configuration over code.
Eric: You configure that and then you're done. And now, obviously you got to get it right, but that's like your code. But the reality of me getting a configuration in step functions right before I get it working in code is very high, right? So I've always said, and you've heard this joke, but the most brittle part of an application is Eric Johnson's code. So anytime I don't have to code, I don't code. So we've brought a lot of power to the developer before they actually have to develop.
Jeremy: Love it. So first of all, we are out of time. That was a great story.
Eric: Okay.
Jeremy: Yeah.
Eric: Yeah, sorry.
Jeremy: It's funny actually, I was joking with Rebecca when we were planning ...
Eric: I took two breaths.
Jeremy: When we were planning this episode, I said, "With Eric, we might be able to just do free association or word association, and then just let him go. And then we don't even have to do any work. It'll be super easy."
Eric: Yeah.
Jeremy: Anyways, so speaking about configuration over code, so this is something you've talked about a lot and you and I actually have had a number of conversations about this because it is an interesting sort of movement I think that is I don't want to say it's unique to AWS, but it certainly is something where AWS is fully embraced, especially with just the integration of all the different functions and the tools and the SDKs and everything that they're going there. And you had a tweet, I think either yesterday or today, or whenever it was, quick serverless architecture evaluation, are your Lambda functions doing a lot of routing or SDK calls to AWS services? It might be time to look at step functions and SDK integrations, take advantage of error, and re-try handling configurable in step functions #configurationovercode.
Jeremy: And I think this is probably where you go to the next level and you maybe lose people even more, right? So asynchronous code is one thing, and that's something that ... I mean, almost every serverless application I have written, I would say 95% of the code that runs, or the processes that run, are asynchronous, because they don't have to happen in real time, right? And you just want to hand that off the back. I mean, again, there's all kinds of, I don't want to say issues, but between cold starts and especially databases, having to call ... Speaking of VPCs, right? If you want to call RDS, you want to do some of these things, right? You got to be in a VPC that speed problems are kind of taken care of, but you then have the [inaudible 00:12:59] gateway and you can ask Cory Quinn about how much he loves those. So there's all kinds of other things that just become more complex.
Jeremy: But now we're getting to this point where you're saying all right, so now you're not even writing code anymore, right? Now we're just writing configuration, and a lot of this stuff, I mean, I think I could probably refactor 80% of most applications I wrote to just use step functions now and the SDK integrations and things like that, which will probably be a good thing. But you're also talking about writing a lot of configuration, right? So that's another thing, and all code is a liability. So even if it's IAC, it's still a liability.
Eric: It is. Agreed.
Jeremy: I guess the question here, and again, I joked with Rebecca before, it usually takes me a good 30 seconds of just talking before I even come up with a question and I'm not a hundred percent sure I have one here, but it's a lot, right? So how do we simplify that? You talked about Workflow Studio, but where's the mental model? Where's the mental model going here?
Rebecca: Just to jump in real quick, well, gents, it's been great. We're at time.
Jeremy: I used up the rest of the time with my wandering question about ...
Eric: Yeah, really honestly.
Rebecca: Okay.
Eric: Looking in a mirror, baby.
Jeremy: Okay, well let's do the free word association. Butterfly. No.
Eric: Yeah.
Jeremy: But seriously, the mental model, like what's the new mental model for serverless because it used to be write these asynchronous functions or try to do as much asynchronously as you can, hand off that work, let a Lambda function run that code. Now we're moving away from that even. I mean, not so much the asynchronous piece, but moving away from even writing your own code.
Eric: Yeah. Yeah. So I don't know what question you're asking, but I'll answer the one that I want to.
Jeremy: I don't either. Yeah.
Eric: No, just kidding. There is some complexity there, so let's take step functions and this isn't alone to step functions, but there's some complexity in a lot of what you're doing, you really got to be in the cloud to test that, right? And so if we look at our old model, it was emulate everything locally, get it working, off it goes, that's good. Whereas now that gets ... Especially with asynchronous, right? That gets a little complex when those asynchronous services or those asynchronous responses are based on different cloud services that you can't emulate. You're not going to emulate poly locally, not with the machines we have. You know what I mean? Or you're not going to emulate transcribe and things like that.
Eric: And so there is this ... We're kind of at this ... We're wrapping our heads around it still. I don't think we've got it down perfect yet, and forgive me for all RSDs and stuff they're working on. I think we're headed in that direction. And again, we'll go to step functions as an example, and SAM, I don't know if this is where you leave, but I know we can talk about SAM Accelerate in a minute, but the idea of kind of where we were to where we are is now we can do some local sanity checks but we need to get to cloud as fast as we can, right? Dor development. So one thing we just did, and I'm going to talk ... I'm going to back pedal for a minute. I don't know if you saw in step functions, we released the local testing.
Jeremy: Yes.
Eric: The local mock testing. That's huge. That's a really big thing because yes, while you'll hear me constantly say, "Get to the cloud as soon as you can." Sometimes you need that sanity to check local, and it's just a quick sanity check real quick, and that's what that mocking tool allows because at step functions, we know that it gets complex to say, "I'm going to integrate with 17 different services. How do I test that, number one, at that unit level? How do I test it at that end to end level?" And so the way we talk about this is really small functional tests locally, sanity checks, things like that. Same with SAM. When you're working with the other services, and things like that, small functional tests locally, if you can. And usually, that's going to be step function and Lambda because those are ... Lambda, obviously, is business logic, step functions is an orchestration of business logic, right?
Eric: So those are things you're checking your code, Eric, Johnson's code is going to fail seven out of eight times, right? And so ... Pr I could do one out of two so I can do the cool finger thing. But you get the idea. So then what we do is we say then get to the cloud as fast as possible, and that's where SAM Accelerate comes in. Does that kind of make sense? We can go further than SAM Accelerate, but is that kind of where you're driving with your question?
Jeremy: Well, yeah. I mean, I think that this is part of the ... This goes back to, I think the wiring that AWS is bringing that I think is a bit unique to the cloud space, right? And everyone is trying to ... What we're trying to do with serverless cloud and all these other things, we're trying to basically say, "There's just so much happening here that you have to be ..." If it took three and a half years to convince Eric Johnson that step functions were the way to go, right? And you're in this stuff every day.
Eric: Yeah, yeah, yeah.
Jeremy: I mean, there's just a lot to think about here. So let's talk about SAM Accelerate for a second, because again, that's one of those things where what you did there was you said, "Look, you can run these things locally with like ..." I think, is it SAM, not Invoke? Is it SAM ... What's the?
Eric: SAM Local Invoke.
Jeremy: SAM Local, right. Yeah, sorry. So you have SAM Local that it allows you to run that little HTTP server locally and you can hit up against your Lambda function, whatever. But then as soon as you have to call DynamoDB or again, integrate with Poly or Translate or any of the other things, unless you're routing that to the cloud to use those services in the cloud, that is not actually that difficult to do.
Eric: Right, right.
Jeremy: But as soon as you're like, "Oh, I want to put it in a SQS queue, and I want to read off an SQS queue, I want to send an event through EventBridge, or I want to use step functions or something like that, it becomes harder and harder to do those things locally. So it makes sense that it's like you can't bring the cloud to ... It's too hard to run locally. We know that. And again, LocalStack, by the way, does a great job.
Eric: They do. They do.
Jeremy: Really trying to make that work, and it works really, really well for most things or for a lot of things. And then we always talk about Brian LaRue and Architect and what Ben's doing, and with the services that they have, they do a really, really great job of letting you run that local, which is amazing. But as these things get more complex, the only place for you to fully test this is in the cloud and the sooner you get there, the better you are. And so the thing is, if you have to do all these magic tricks to make it run locally, then that's just brittle, right?
Jeremy: In the whole, it works on my machine thing, that is that's crazy, but it's true, right? So that's my point. I actually don't even know what my point here is other than the fact to say how do you ... I guess let's talk about SAM Accelerate and the idea of bringing everything into the cloud. And I know that AWS tries to meet customers where they are, so they're like well, we can also do it locally, but really is that what we want? Or do we really want to just be moving into the cloud?
Eric: So let me back up real quick on the LocalStack. I agree with you, LocalStack does on a phenomenal job. Brian, his team is doing a great job. Where LocalStack falls down is not LocalStack's fault, it's Eric Johnson's fault. And Eric Johnson is a great example of a standard developer. Eric Johnson can't figure out docker network to save his life. I cannot get my dockers to talk to each other, things like that. So it's not always the emulator. I applaud the emulators, I think they're great. But what happens is we spend so much time trying to get everything working, we have less time developing, changing the world with our code, right?
Jeremy: Right.
Eric: Or our business logic. So what we did is we said, "Okay, we're going to leave SAM Local, and we're going to bring in some tools like the local mocking test for step functions." But the ultimate goal is to get to the cloud as fast as you can, and the reason for that is, and you were saying it earlier, I don't think you used the exact words, but synchronous is easy to test. Hey, I'm going to invoke a Lambda to get an instant response. And from a Lambda, I could call DynamoDB, I can call some different things. When we go to asynchronous, which is what we're driving for, right? Like you said earlier, we want to get to asynchronous as fast as we can because that keeps your client not waiting, that allows you to all the retry, and all that kind of stuff. We can talk about benefits of that in a minute, but that's hard to test because it's a fire and forget, right? I've triggered the event, now I don't know what's going on past that.
Eric: So with SAM Accelerate, we designed it so that every change you make, and we looked at several different things. First of all, we differentiate between configuration and code, and we say, if you're anything ... You're configuring a Lambda, you're changing the time out, you're changing the run time, things like that, that's all configuration. That requires a deploy. If you're changing anything like that. But if you're changing code and we look at code is obviously, your Lambda code, layer code, states language as code, and we look at open API as code. If any of those are being changed, we'll do a really quick sync, and we're talking a matter of seconds, of your change to the cloud. So that allows you to say, "All right, I've changed my Lambda, now I'm going to test." I just said Lambda, Lambda function. Chris Munns, I apologize. It was a lower L too.
Jeremy: He's not your boss anymore though, right?
Eric: Well, he isn't, but he's still being great.
Rebecca: Somewhere he's like ...
Jeremy: So Lambdas, Lambdas, Lambdas. Let's just say it, right?
Eric: Yeah, let's do it. Lambdas. No, I can't do it. But yeah, we get that out there and we contest that immediately, hand it through an API call, things like that. But you still have the asynchronous, so those are still synchronous tests, right? So for asynchronous, we said ... I mean, who doesn't debug with logging? I mean, that's just ... I mean, we'd love to do the stepping through and you can do that locally, but the reality is testing the cloud. That's a tough thing to do. And I know there's some folks who do it, but it's tough, right? And so we end up saying this code was hit. I can't tell you how many times in my production code it says dysfunction was hit in the logging, you know what I mean?
Eric: And so what we were able to do is we would say, "Okay, you could put logs anywhere in my Lambda function, I'll collect step functions, logs, I'll collect API gateway logs, any of those serverless things. And we'll aggregate those through the same logs part of it. So it's a combination. And when we say SAM Accelerator, people go ,"What is SAM Accelerator?" It's an actual function called SAM Accelerator. It's a collection of tools. One is the same sync that we just talked about, where you sync the code up, two is SAM logs, where we actually aggregate your logs from your Lambda, your API gateway, your step functions, things like that.
Eric: And you can see it all. And we also tie in tracing there. And there's some other things we do to make that sync happen very fast that I'm glad to go into it another time. But the idea there is now I can test my asynchronous processes. I can say, "I expect this Lambda function to do A, B, and C, or I expect once it hits a queue and then it goes through ..." Or let's say, a topic, then it goes through a queue or whatever, I expect these things to happen and I can get those logs and I can look at the tracing, but it's all done in the cloud. Get to the cloud as fast as you can.
Rebecca: So I think something that you're getting at here, especially if we're going to take this from a philosophical level, right? Is bringing the developer to the cloud, not the cloud to the developer, is essentially allowing that developer to be a human rather than a robot, because a robot would perhaps never get it wrong, but as a human you make brittle code.
Eric: Right.
Rebecca: And the only brittle that we want is peanut brittle, we don't want brittle code.
Eric: Right.
Rebecca: We don't want brittle applications. And so I think there's ...
Eric: Brownie brittle.
Rebecca: A bridge ... Ooh, brownie brittle. I'm into that too.
Eric: Brownie brittle. Sorry to interrupt. I can't leave that you.
Rebecca: No worries at all. We should represent all the brittles here except for the brittle code.
Eric: Yes, all brittles represented. That's right.
Jeremy: We'll get tweets if we don't.
Rebecca: Yeah, exactly, #browniebrittle. So there's something that you, I think are really passionate about, Eric, which is allowing for humanity in tech, and you're actually going to be talking about rediscovering humanity in tech in some upcoming talks, and so I wanted to make that bridge here because I think ... I mean, we can joke about it. I know everyone knows that developers are people too, but a lot of times it's like we expect a developer to act like a robot and therefore, oh, just make them do all this stuff and not get it wrong, and then they end up never coding because there's all these things where it's like oh I'm human too. So maybe you could talk to us a little bit about how those two ideas and philosophies sort of intermesh around how you create the constraints or allow developers to also be humans while to doing their work and what that means in tech and how we're bringing ... How you're helping people rediscover that way you want to talk about.
Eric: Yeah. Yeah, and where this comes from is I'm actually doing a talk where our entire team is doing go to, all the go to development events this year. And so we're doing a keynote and that's exactly what it's called, it's called Rediscovering Humanity in Tech. And this came from not what you were ... Not exactly what you were talking about, but ties into it, so I'll kind of bring it back a little bit. This came from just I feel like we beat each other up sometimes. I feel like the expectation is perfection. If you work for AWS, and this is not tied to AWS and this is one of those things, this is Eric's opinion and not AWS's. But we tend to beat each other up.
Eric: Then what we do is we get on Twitter, we yell at each other. We have opinions and this kind of ... I mean, I'll be very careful here, but it can apply to a ton of topics. And we've lost the art of innuendo. We've lost the art of friendly debate somewhat because we do it all through Twitter. We do it all through social media. Boy, you're probably getting more than you want. You can cut all this if you want, but we kind of expect perfection and we kind of beat each other up when it doesn't happen. And there's one of my favorite movies, it's not a great movie, but there's an old 80s movie. Jeremy, you'll know it. Rebecca, you'll have no idea. No, you may. It's called Roadhouse.
Jeremy: I love Roadhouse.
Eric: Yeah. One of the lectures ...
Jeremy: We talked about Roadhouse last week, didn't we?
Eric: Did you really?
Jeremy: We might have mentioned Roadhouse for some reason. I don't know why, because we were talking about Patrick Swayze. I don't remember. Anyways, sorry, go ahead.
Eric: Yes. In this scene and I'm not going to do it, but Patrick Swayze and I are basically mirror images, but I'm not going to do his voice. That's our only difference. But he says ... He's telling his people how to handle conflict in the Roadhouse, and he says, "If they get in your face, be nice. If they hit you, be nice. If they whatever." And I feel like we've lost this art of being nice to each other. Now that does not mean except everything that's said and it's okay to ... Oh, that's a mistake. We still ... We do love you. We do things like that. But I'd like to see people ... I'd like to see grace, you know what I mean? Grace and mercy is a term that I use with my kids all the time, and when I make a mistake, show me, point it out, help me reach that. But finding that humanity of dealing with each other.
Eric: And what I found is when we go to social media and things like that, we don't go to social media to hear opinions, we go to social media to espouse opinions, right? And I feel like that's ... If you want my opinion, let's sit down, let's talk, call me or email me, things like that, but let's be humane to each other, right? And so going back to this code thing. Yeah, yeah. I think there's a ... And that's where it really kind of hit me is you see things go out, and I have some stuff I've worked on that I've just poured literally this will help people, and I'm going to build this and this is going to be great.
Eric: And then it gets out there and someone goes, "Yeah, you misspelled this." And they only see that. And again, this is not just be nice and never disagree, but it's disagree with edification or disagree with an idea of hey, I'm going to help you, be nice. I don't know if that makes sense to you, but that's a small precursor of where we're going on the bigger scale of understanding. How do you help each other? How do you work with each other?
Jeremy: Yeah. I mean, I don't know why people try to be so controversial. So with that, what's your favorite programming language? Or sorry, what's the best programming language?
Eric: The absolute best programming language is configuration.
Jeremy: No, I think you're right. I mean, honestly, that's the ... I put a tweet out there the other day that actually was making a joke about how bad this article was that was talking about TikTok or something like that.
Eric: Yeah, yeah.
Jeremy: And it was clearly a joke to me and people were like, "There's no research to back this." And I'm like, "Yeah, that's exactly why I posted it because there's no research. There's no evidence this is even true."
Eric: Show me the data.
Jeremy: "That's why I posted it."
Eric: Yeah.
Jeremy: But I mean, it just happens. But no, I get it, but anyways, all right. So let's ... That sounds awesome. We've got a whole bunch more we want to talk about and we are actually starting to run out of time, but so ...
Eric: You're welcome.
Jeremy: You are now ... Well, right. We just have you ... I mean, actually we could do 15 episodes just with Eric and then our job would be great.
Eric: No, I'm just kidding. I'm playing.
Jeremy: No, so you have been at AWS for a while now. You're now a principal developer advocate. You were formerly a vice principal developer advocate, I believe, right?
Eric: Yeah.
Jeremy: So curious though, going back to everything you're talking about too with just the humanity aspect of it, being nice and so forth.
Eric: Right.
Jeremy: I bet that experience you've had, talking to all these people, which is amazing by the way, this is one of my favorite parts of what I do is actually getting to meet and talk to people.
Eric: Yes. Yes.
Jeremy: So just tell us a story, because apparently you're good at that, just reflecting over those last couple of years, what have you learned? Besides now moving into thinking step functions are the greatest thing, but other than that, what have you learned? What has stuck with you over the last couple of years? Or sort of takeaways from this?
Eric: Well, I've learned they'll just let anybody be a principal at AWS. I have to tell you, when I got the call from Munns, he called and said, "Hey, you are a principal now." And I was like, "All right." So we went out to dinner to celebrate that night and the girls kept telling the waitresses, "My dad is a principal of AWS." And then sometimes it was, "My dad is a president of AWS." It's like, "No, no, misunderstanding."
Eric: So what have I learned? Part of it is that I think I've learned that we kind of get this idea in our head that everybody knows if you work at AWS or any other company that's doing a lot of things, or any other company in general, if you're higher up, it's they know, they have all their stuff together. They have got it locked in. And then you start talking, it's like you don't know either, you know what I mean? Now don't get me wrong, I have worked with some really, really smart people, and when I started here, I was terrified to ask the questions because I work for AWS. I'm a senior ... At the time, I was a senior developer advocate, I should know everything about serverless, but the reality is I didn't. I knew some things. There was some things, the API gateway, that was my thing. That's my jam. I can dive deep on that. Step functions, not so much. Now I'm much better at it.
Eric: And so there's no way you know everything, and so I would sit in meetings and just quietly, the QRSW, niner niner, 42, whatever, and you know, I've been in those meetings and it's like, I don't know what that means. Finally, just one day I literally said, "You know what? I'm asking the question. I don't know what that means." I raised my hand virtually, I don't know what that means, and it was really funny because over 50% of the room goes, "Yeah, I don't know what that means either. I'm so glad you asked. I don't know." So I think the first thing I've learned is you don't have to know everything and you'd be surprised, somebody, if they're talking about it, somebody knows, but you don't get better if you don't ask the questions. I think it's really critical that if you don't know, ask. If you get fired for that ...
Jeremy: There's something wrong with the culture.
Eric: That's not your fault, that's somebody else's fault. Yeah, there's a problem there because the only way we learn is by asking. My dad was a teacher, he was a high school teacher, and here's his thing was always ... And we always say this, there's no question too dumb. But he really meant that. If you don't know, the only way to learn is to ask the question. I've said this before and I'll say it again, if you're the smartest person in the room, you need to change rooms. I want to be in the room where I know the least. Case in point, and here's a story, Jeremy. I'm a drummer, we talked about, it turns out. I was playing in community college. I was the best drummer they had and I really thought that was something. And then I transferred over to a school of music in Phoenix, Arizona. It turns out I'm not that good. I'm really good for one finger, I'm really average in life. You know?
Eric: And so the worst drummer there ate my lunch, but I learned more in my time there with those drummers being better than me than I ever did being the best because if you're always teaching, you're never learning, right? And so ... I'm just full of cliches, aren't I? How many cliches can I fit it? But that's one of the things I learned, you have to ask the questions, don't be embarrassed about it. And what I've found is at least in my experience, and you hear people who, "Oh, AWS is this, or Amazon is this, or Oracle is this." Or just whatever. They classify companies based on people. I guess this would be a second lesson, based on experiences with people. But the reality of a company is it's a group of people. And yes, while we have policies in place and things like that, some people are better managers than others. And so they base all story off a bad manager, all story off a good manager.
Eric: My experience has been very gracious. I mean, there's an expectation. You work hard. We work hard, play hard. But the there's a gracious allowance of oh, you don't know? Let's teach you. Now if you keep saying I don't know what that means to the same thing every time, probably the problem might be you and not them. But yeah, I mean, those are some of the things that I've learned. I'll stop there unless you have other questions, but those are some key things.
Rebecca: I want to follow up on that because ...
Eric: Okay.
Rebecca: So in my time at AWS, that was also one of my key learnings. If you don't know, don't be afraid to ask because someone says, what do you say, QR QR 21, 21, 9, and all of a sudden someone goes long. They're like, "Blue 42. I thought you told me a button hook." And you're very confused.
Eric: Was there a niner in there?
Rebecca: Yeah, niner niner. Yeah. So when I first entered AWS, I mean, everyone has different levels and stuff like that, but L5, so lower quote, air quote, lower level in terms of the hierarchy of an organizational structure.
Eric: Sure.
Rebecca: And I would be in meetings with VPs of people who were making very large organizational business decisions and something that at one point, I was reading a doc and then I had all my red pen circles, don't know what this means, not quite sure how this fits together. And finally, I got to the same point you did where you're like I'm just going to ask. And something that was interesting is when I asked, afterwards the VP stayed and said, "I also didn't know what that meant."
Eric: And I was talking about it.
Rebecca: Yeah, and I was like, "I was a little worried to ask that question." And he said the same thing, right? Don't be afraid to ask. And then we talked about it and I was like I think something that happens, right? Is someone who is, let's say, quote, air quote, lower on the level hierarchy is more afraid to ask because they feel like they need to rise up since everyone around ... They don't want to be that person that's like, "Oh, I'm at a low level and clearly, I'm at a low level because clearly, I deserve to be at the low level because I don't understand." And so we talked about what does that mean for that person at the higher level to ... How do you create? And now that you're a principal and having a principal developer advocate, are there any tactics or ways where you're like, "Hey, how do we create this space so that people ..." It's seen as humble when someone at a high level asks what could be a scary you should know this question, right?
Eric: Right.
Rebecca: Because they're like, "Hey, clearly, I'm at a high level, but this doesn't make sense to me. Can you explain it?" Versus it feeling like a different burden of proof on someone who is at a different level. So do you have any ways where you're like hey, when I'm in a room, even if I know the answer, I ask the question to open up that safe space or any other tactics or ways that you think that space can be created so that people can feel like they can ask those questions?
Eric: Yeah, and I think you just hit on it. We do service office hours. We do service office hours on Tuesdays, and one of my jobs, I feel, as the host, but it's the same in rooms and private rooms as well, is I ask the question. I know the answer, sometimes, not always, but I ask the question that I don't think has stuck. You can kind of see the eyes and stuff like that, and that's one of the things I really try hard to do is pay attention to the room because you can often tell when people are off the tracks, right? They're just in there kind of doe eyed, shaking their heads, yes, yes, yes, I get it. I understand. But do you? And so I really try to open that up and ask the questions.
Eric: The other thing, and this was interesting, Chris Munns and I had this conversation when I started there, he said, "What are some things you would change?" And one of my frustrations, which was documentation, and AWS is not alone in that, documentation is hard. But a lot of times we make assumptions and this is a safe space internally. We make assumptions, we go, "I'm sure you know this so I'm just going to skip right over it and tell you to do this with this, with this, and the documentation." And I spent half my life going, "I don't know that. I don't understand how we made the leap from here to here."
Eric: And so when I write docs or when I do explanations, either I'm speaking publicly and internally, is I try to connect every dot and sometimes it's too much. I'm a talker. Sometimes it's overkill. And I'll say, "Look, if I'm telling you something you already know, go ahead and interrupt me, that's fine. But I'm going to say this and then you can ... You might correct me." But those kinds of things to not make the assumption. I think a lot of times we assume we're all in the same room, we're all at the same level, and that's not the case. And so those are some of the steps that I take.
Eric: I will tell you another thing, there's a guy named Chris Lewis. He works with the SAM team. He's a PM. I really like him, he's really awesome. But I bring things back to the SAM team a lot and I'll say, "Hey, I didn't know this was happening or what about this?" That's my job. I interact with SAM all the time. And his first response almost every time is, "Okay, Eric is asking that question. That means probably other developers are asking that question. How do we make that clear? How can we fix our docs? How can we fix our whatever to make that clear?" And I think that's another thing is just that key awareness that it might not be that Eric's dumb, there's a good chance it is, but it might not be this is an Eric thing, it could be an us thing not explaining it. And so I think there's that burden of sharing that load of getting ... Of communication.
Jeremy: Yeah, so there's so much there, which is awesome advice and I totally agree with you, you have to ask questions. It's one of those things where ... And you know what actually kills me the most is acronyms. People love to use acronyms and I'm like acronyms do not mean what everybody thinks they mean, right?
Eric: That's right.
Jeremy: So I was in a meeting, I don't know, probably a year ago or something like that, and sales and marketing were talking about SQL with a bunch of developers and we're like SQL, and then referring to it in this weird thing. I'm like what are they talking about? Sales qualified lead, not structured query language, which is a very different context to be in.
Eric: Yes.
Jeremy: And when you have these different things, it gets very, very confusing for people. But I think that what I took away from all of that was really nobody knows what they're talking about most of the time, which is actually quite true.
Eric: Or Eric doesn't. Yeah.
Jeremy: Well, no, actually I think it's one of those things where you can't know everything.
Eric: Right.
Jeremy: I mean, it's just ... I don't know. I mean, it's just that even if you know things well, you just have to sometimes be aware of it and then stack overflow and Google and the docs and all those things are your friends, because those are really what I think dig you into it. And I want to talk about education in a minute, but I know that Rebecca has a follow up question so I'll let her go, and then we'll move into some more education stuff.
Eric: Okay.
Rebecca: Yeah, jeremy is excited to ask you about university and the youth, but before we get there, a couple weeks ago now perhaps, we were recording with Shawn Wang, the head of developer experience at temporal.
Eric: Okay.
Rebecca: And what's super interesting I think is I want to ask you ... There's like developer experience, right? And developer advocacy, and one way that we were sort of talking about this was developer experience gets involved at the beginning when you're designing the product to say if we are sitting in the developer seat this whole time, what is the best way to design this product so that it feels the most intuitive for a developer by the end state? And I think a goal of developer advocacy is also similar, right? You want to have the inputs and get the feedback from customers that it can become the next input to the product, but a lot of times developer advocates are almost somehow now at the end where it's like this product or service exists and now I need to educate you about the gotchas and educate you around even not just the gotchas, like the benefits and how it all works.
Eric: Right.
Rebecca: And I'm wondering if you, or your team, if you're able to start to integrate into the beginning of experiences or how you might advocate for that even internally to say, "Hey, love to educate about this, but how do we get involved at step zero rather than at step 17?"
Eric: Yeah. That's really good. I think part of that in this ... Any company deals with this, "Hey, we just want to get stuff out." And two pizzas teams are ... While they're great. Let me try that again, two pizza teams without pineapple. While they're very fast and while they're can move things, sometimes that becomes an issue, right? And so one of the things that we've done is we insert ourselves in these teams, right? So I sit with the same team once a week and they tell me here's what's coming. I meet with [inaudible 00:42:40], who's the PM and she says here's what's coming. Same with API gateway. And so we know and we kind of each ... Ben works with our step function. We all work with all the teams but we kind of have our focus areas.
Eric: And one of the things we've been asking and they're very responsive is how early can I get a look at this? Give me a working copy, let me play with it. We try not to get too much, and some would say, "Oh, it's because you're lazy, Eric." But there is a method to my madness. I don't necessarily read all the docs. Now I'll read the first couple. If you know anything about Amazon, we're a doc company, right? And so if we're going to come out with a feature, there's a doc somewhere. If we're making a decision, there's a doc. So I'll read those but I don't necessarily get into all the UX, other than I will give some guidance up front. "Hey, I think this, this, and this." But I will play with what they've done after the fact.
Eric: So I'm a little at the beginning, much heavier in the middle, and towards the end where I can say, "Hey, this doesn't make sense. Or this works or ..." and I try to come at it. And the reason I stay a little bit out of it is I try to come at it from a regular developer because I am, I mean, I develop every day. I'm working. It is part of us, and I'm not ... All joking inside, I'm not the strongest developer. We have some really strong developers on our team. My background is theology. I mean, that's my degree, right? So it's like whoa, all the development I learned was at Google in the middle of the night, or in Googling, not at Google. You know what I mean, and so to clarify.
Eric: So we try to do that approach, but we have another thing at Amazon that's very helpful is we have this thing called the end of the quarter. And Rick, I think you know what I'm talking about. This may be a secret, maybe I'm giving you a company secret. I don't know, we'll see. But it allows you to go whoa, this is detrimental. This isn't going to work. And we had one last year where I was looking at it and said, "Mm mm, this doesn't make sense. This is going to cause some confusion." And I document it, here's where it all is. And the team went, "Okay, whoa. Let's stop." and they said, "We want to get this out as soon as possible, but we want to get the right thing out and so let's step back."
Eric: Because we all know, it's better to wait and put out something great than to put out something we already know is bad and then try to fix it one way. There's a balance there. And so that ability is very powerful. We don't do ... Try not to do it every time or they won't talk to you anymore, but it is the ability to say, "We need to fix this." So we use that. I've only used it one time and the team was incredibly responsive, and so those are some of the tools that we use for that. If we were to take this on like, "Hey, I'm a new DA, I work for this group." My advice would be get into that group as much as you can. Sit in their meetings, get to know the PMs, get to know the GMs. Don't just wait for them to come to you, go to them and insert yourself. Make them know who you are. So like it or not, that's what I've done with our teams.
Jeremy: Yeah. Well, by the way, while you were talking, I don't know why Alex [inaudible 00:45:43] sent me something about pineapple pizza, so don't even know how he knew we were talking about.
Eric: I know. I think you're lying or it was detrimental because I know Alex [inaudible 00:45:52] would not put a pineapple on pizza.
Jeremy: No, no. He basically was saying not to put pineapple on a pizza.
Eric: Okay, good. Okay.
Jeremy: He was calling you out basically.
Eric: He was feeling it, yes.
Jeremy: Right. Yes, exactly. So all right, well anyways, we should definitely move on to ...
Eric: Okay.
Jeremy: A little bit more about education, and I know you're out there teaching a lot of people learning serverless, a lot of developers, companies, they read your blog posts, you're out there speaking, you're doing a lot of this stuff. Now it's a little scary that you're thinking about educating or shaping younger minds around this.
Eric: Yes, very scary.
Jeremy: But you'll be teaching them serverless, so tell us about this thing that you're going to be doing pretty soon.
Eric: Sure.
Jeremy: So you're going to be going into universities and you're going to be actually teaching kids in college serverless or about serverless because I remember a couple years ago now, we were at a dev conference or whatever it was, I forget, what was it like the developer ...?
Eric: [inaudible 00:46:47] together, yeah.
Jeremy: Whatever we were, we were out in Seattle doing something.
Rebecca: Developer Influencer Summit.
Jeremy: Yes, that's what it was.
Eric: There it is.
Rebecca: Gotcha.
Jeremy: You were trying to influence at AWS.
Eric: Yes.
Jeremy: And one of the things we told them was why are we not teaching serverless in college? And how do we do that? And so a couple years later, it sounds like we're getting there, so tell us about this.
Eric: So here's where it started, last year I was approached by somebody, kind of a third party, and he was working on teaching university. He was basically tapping into the university students as developers for his team and he would ... And he had actually partnered with a lot of the professors, several universities across U.S. and Canada, he had partnered with some professors saying, "I want them to build the product that I'm trying to get out but could it be their final project?" So these CS majors, master final project, things like that. And so he really wants to use serverless. And so he got ahold of me, found out ... Actually, he was working with ASU and my son just started at ASU. That's a whole other story. But he's going to ASU, and so he reached out to me saying, "Hey, I hear you have a son at ASU, I'd love to talk to you about projects." So I told him, "Look, I'll just kind of on the side, I'll take on and I'll talk to your students."
Eric: So I was doing some office hours and not just ASU students, students all across, like I said, the nation, and I was shocked. He wants them to build serverless, but no clue. No clue what serverless is and more event driven architecture, so that's really what we're talking. We're starting to lean more towards event driven architecture on top of service, right? And it just ... I mean, deer in the headlights. And I was like, "Okay, okay." And then as I started to dig in, this was ... And I really don't want to do this on a hey, I'm condemning this university, nothing like that. But I was surprised to hear not a lot of cloud education in general.
Jeremy: Right.
Eric: Yeah, it was shocking. And so I said, "Okay. Well, what are you studying?" Well, one student said, "Well, I do C++." "Okay, that's great. That's great. Where are you using that?" And I'm not saying you don't use C++, okay? I can't spell C++, so that's great on you. But it was shocking to me, so I decided what if ... And I'm still talking to the same person. What if I could create a serverless immersion data? We do these in companies and things like that. We've done a road show before but we targeted at university students, would you be interested? I would be totally interested. I'd love that. So we've kind of set that up.
Eric: And so I'm doing two kind of prototypes of this. One, I love this too, one is at Notre Dame and one is at Harvard in April, so you can be sure my LinkedIn is going to say, "Guest lecture at Harvard and Notre Dame." Probably before I actually do it. But one of them is going to be with the CS 50 class at Harvard, and so very excited about that. And so what I want to do is I want to go in, and you can't teach all serverless in a day, right? But you could teach the concept, the idea.
Eric: Hey, there's something else than spinning up machines, spinning up containers, all that. And we teach the idea and get it in their head, event driven architecture. So if we teach them enough where they go, "Okay, I'm going to pursue that and look after, and see what's out there." And at that point, and this is honestly, yes, of course, we want them to come to AWS cloud, but serverless, serverless for everyone, kind back to that mantra that I used to do years ago is let's get you serverless and then we'll talk [inaudible 00:50:06].
Rebecca: Yeah, I love that you're saying ... You're basically like we need to plant the seed of the paradigm shift.
Eric: Yes.
Rebecca: So you don't get to the end, you don't actually leave college, and then you're like, "Oh, I guess I'll ... I can code." And then you end up trying to use the cloud, and now you're trying to flounder with this paradigm shift that could have been planted many years before. So thank you, and the future is bright.
Eric: Yes.
Rebecca: And I'm excited for all the students you are going to reach as a guest lecturer at Harvard.
Eric: At Harvard.
Rebecca: And Notre Dame.
Eric: Yes, I'm going to use that voice. Pretty excited about it. Yes.
Jeremy: I wouldn't use that voice there. I'm from the area.
Rebecca: Yeah.
Eric: I walk in wearing that ...
Jeremy: Whenever we see Alec Baldwin giving a terrible Boston accent, you know it's ...
Eric: I'm going, "Hey, after this we're going to play stickball bowling."
Rebecca: Yeah.
Jeremy: Candle pin bowling.
Rebecca: Candle pin.
Eric: Oh, I'm sorry.
Jeremy: Candle pin bowling.
Rebecca: Candlepin.
Eric: I'm sorry. Candle ...
Rebecca: Candle pin stickball. You and The Rock.
Eric: I don't know.
Rebecca: Think you're going for Midwest right there.
Eric: Well, the idea behind it just a little more is if you remember way back when I was a kid, our school had Apple computers everywhere. Apple pushed their ... They got their computers in, and that's what I knew, that's what I learned. And so it's that kind of idea because what happens when they come out of university, they've learned, let's say, they learn containers or they learn VM, that's what they know. And so it's very hard to go yeah, that may be better, but I don't know it and I've got to build something. So while they have time to go, let me decide what I like better. They need to have this option as well.
Rebecca: Yeah, absolutely. And with that, I lean back in my chair and I say, "Eric Johnson, thank you for sharing your knowledge, your stories with us, and with the future of coders, developers, and cloud practitioners. How can our listeners find out more about you?
Eric: Well, you can look me up on Twitter. I'm @edjgeek on Twitter. That's probably the best place. Also, serverlessland.com. I have my own profile there. They let me be on it, and so ...
Rebecca: Kind of a big deal.
Eric: Those two places. Yes, kind of a big deal.
Jeremy: People swoon over you now.
Eric: Just say it. But you see ... Yeah, they're swooning. Here, I will tell you a real quick story, we were in Saudi Arabia a couple of weeks ago. I was literally called point blank "the obnoxious American". We love obnoxious Americans, you're so loud. So there you go. It was meant in the kindest words, I'm pretty sure, but yes. But yeah, those are some ways to get ahold of me. I really appreciate that y'all do this. I love this show and thank you very much for having me.
Rebecca: Well, we love in our affection by ... Affectioned by. Have a lot of affection for you as well.
Eric: You have to promise not to edit that out.
Jeremy: When you are at Harvard in April, reach out, maybe we'll meet up for a Dr. Diet Pepper.
Eric: Let's do it. And if you get the name right, [inaudible 00:52:50]
Rebecca: Dr. Diet Pepper.
Eric: Yeah.
Jeremy: Dr. Diet ... Diet Dr. Pepper.
Eric: Diet Dr. Pepper.
Jeremy: I said it wrong, oh God. I haven't ... Maybe I haven't slept. Let me say that ...
Eric: I'd say you've had too many Diet Dr. Peppers.
Jeremy: Yeah, I've had too many Diet Dr. Peppers. Diet Dr. Pepper. Okay.
Eric: Yeah, let's do it.
Rebecca: Nice. It's like, "Oh, I'm here for my check up with Mr. Dr. Diet Pepper." Paging Dr. Diet Pepper.
Eric: Paging Dr. Diet Pepper.
Jeremy: Whatever, it's close enough. You know what? This is ...
Eric: That's funny.
Jeremy: It's a lot of mental strain to do this show.
Eric: It's a lot to remember.
Jeremy: I mean, it's just ... It's so hard.
Eric: That's right.
Rebecca: I hope people have stopped listening by now. They're like, you know what? We don't need the outro music.
Jeremy: They've already unsubscribed. They're done. They're done. They don't even care anymore. Anyways, Eric ...
Eric: I wish they had a point,
Jeremy: Eric, thanks again. It was great having you.
Eric: Yeah, thank you. Thanks, sir.
