Episode #117: Serverless Cloud with Doug Moscrop, Eslam Hefnawy, and Ben Miner
November 1, 2021 • 58 minutes
On this episode, Jeremy and Rebecca chat with Doug Moscrop, Eslam Hefnawy, and Ben Miner from Serverless, Inc. about the launch of Serverless Cloud, how serverless shifts developer responsibility, the limitations and advantages of abstractions, the importance of developer experience, and so much more.
Doug Moscrop is the Lead Software Engineer at Serverless, Inc. working on Serverless Cloud. He is a pragmatic programmer with a strong engineering discipline, a thirst for knowledge and a desire for accuracy. He's the author of several Serverless Framework plugins and the creator of serverless-http.
Eslam Hefnawy is a Principal Software Engineer at Serverless Inc. working on Serverless Cloud. He's been writing software since the age of 14 and is passionate about open source, dev tools, and serverless technologies. In 2015, he joined Serverless, Inc as their first hire and co-created the Serverless Framework with founder, Austen Collins. In 2018, he lead the design and development of Serverless Components, a next-generation Serverless Framework.
Ben Miner is a Software Engineer at Serverless, Inc. working on Serverless Cloud and has experience all over the spectrum, including DevOps, backend, and frontend technologies. He's also a pursuer of SAAS architectures, functional programming, and great end user experiences.
Transcript
Jeremy: Hi, everyone. I'm Jeremy Daly.
Rebecca: And I'm Rebecca Marshburn.
Jeremy: And you're listening to Serverless Chats. Hey, Rebecca.
Rebecca: Hey, Jeremy. How you doing today?
Jeremy: I am super excited today, because on the show, I have some of my favorite people right now from the serverless industry, because I work with them.
Rebecca: Well, I've heard that you have actually two of your favorite people, but we have three guests, so I'll just let them decide which of those two it is.
Jeremy: No. There you are, causing trouble. Dissension in the team. With us today, no, these are seriously some of my favorite people in the serverless industry, some old, some new, but people that are making, I think, a ton of progress, and I am super happy to be working with these people. I've got a few members of the Serverless Cloud team from Serverless Inc. we recently launched Serverless Cloud into public preview, and I think we're all super excited about it. But I've got some really cool voices. Would you like me to introduce them, Rebecca?
Rebecca: Oh my gosh, please do, and when you introduced them to me, told me about them, I was super excited to be able to be on the show with you all.
Jeremy: Awesome. All right, so-
Rebecca: Go for it. Floor's yours.
Jeremy: Floor is mine. All right, so first up, we have Eslam Hefnawy. Hey, Eslam.
Eslam: Hey, Jeremy. Hey, Rebecca.
Rebecca: Hey.
Jeremy: We have Ben Miner. Hey, Ben.
Ben: Hello. How you doing?
Jeremy: And last, but certainly not least, Doug Moscrop. Hey, Doug. How you doing?
Doug: I'm doing great. Glad to be here.
Jeremy: Well, I am excited that you are all here. Full disclosure, I am the GM of Serverless Cloud, so these people are all the amazing people that work on my team, or I should say some of the amazing people that work on my team, but I don't want this episode to be about me. I want it to be about them, because they all bring a really unique perspective to the serverless community, and they all bring a really unique perspective, I think, and a lot of really great traits and skills to what we built at Serverless, excuse me, at Serverless Cloud. Rebecca, I don't know, do you want to maybe start, and ask them some questions, and see if we can figure out what Serverless Cloud is?
Rebecca: Yeah. Well, first, I actually really want to know. I know that each of you sort of bring a really special role or kind of a superpower, as Jeremy has told me, to the team, so I'd love to start with you, Eslam, and sort of get a quick overview of what it is that you do on the team, and then let's roll into talking about Serverless Cloud.
Eslam: Sure. Yeah, that makes sense. Yeah, Serverless Cloud's... We all do a bit of everything, honestly, but now that the team has grown, there are some roles, although we try to fill each other's roles, but for the most part, I focus on developer experience. I take care of the CLI, and a little bit of the API, but yeah, other than that, I can dig in a little bit here and there in the backend as well, but that's my main focus at Serverless Cloud right now.
Jeremy: And developer experience, by the way, is one of the most important things that we're striving for here.
Eslam: Yeah, that's right.
Rebecca: What about you, Ben?
Ben: I think I'm still kind of wandering, to find my true niche on the cloud team, but so far, I've definitely spent a lot of time just modernizing the codebase and cleaning it up, especially migration to TypeScript, migrating to Jest, you know? A lot of stuff like that, using esbuild, to just make deployments faster and the bundle size smaller, so I'm still kind of trying to find where I'm going to sit, but it'll come.
Eslam: [inaudible 00:03:32]
Jeremy: Well, I was just going to say that Ben is definitely, has pushed the team to move to TypeScript. I mean, one of the things that we did was this was a POC originally, right? This was something that we were trying to figure out, can we do these things, can we create these isolated environments and move them through this developer experience, and through these workflows, and make that fast and reliable? So again, a lot of MVP work that was done, and now that we've gotten to this point of a public preview, Ben has actually been very, very instrumental in helping us, like he said, modernize that codebase, get a lot of those tests in there, get a lot of that automation set up for us and help us with the TypeScript migration. So, don't undersell yourself Ben. You've been a very, very valuable member of this team.
Rebecca: I was going to say Ben being all humble, but Doug, last but not least, Jeremy actually used the word "journeyman" about you, so if you could elaborate a little bit on that as you intro yourself too.
Doug: Yeah, absolutely. My background is I've always loved programming since I was young, and I tend to be a little bit scatterbrained and sort of every man, like there's nothing that I can't get excited over, so my superpower, I think, is I just sort of bring this really big picture, pushing the boundaries of what's possible, just the crazy idea guy, plus I tend to be invested or involved heavily in runtime-type things, so sort of the inner workings, the gory details. Yeah, I think that's my superpower, is just sort of never being satisfied, and always having a new idea.
Jeremy: Yeah, and let am add to that, Doug, because one of the things that you did that was huge, I mean talking about pushing the boundaries, is the ability to sync code from your local IDE to the cloud, and actually have that in production, or available being your cloud, your personal developer instance as we call them, but have that available in less than a second. I mean, that is something that AWS can't do, right? Because AWS has to do all this whole deployment thing, and these other things, and within this constrained environment that we've created, we're able to do that very, very quickly, and that is something that I know you've been working on for a long time, and pushing and pushing. Those are the types of things that it's great, because we just know the pains, and we're like, "Hey, what can we do to make that better?"
Rebecca: So pretty excited to get into these "gory details," as you called it. Can we talk a little bit about what is the launch of Serverless Cloud? And then let's get into the gory details.
Jeremy: Yeah, so who wants to start. Maybe Doug, you want to give an overview of what cloud is?
Doug: Sure, yeah. Okay. Serverless Cloud, for me, was born out of a lot of my experience bringing serverless to a place where I was working, and just, I don't think I was as early to the game as Eslam was, but I was pretty early on the train in serverless, and so feeling a lot of those pain points, especially around the local versus cloud impedance problem, and just the speed of deployments, and all that stuff. And even beyond that, it was just a general sense of like I got sick of having to write so much code, and I had a feeling that I was duplicating myself a lot.
Doug: Like, I would write all my code, and then I would write all my configuration or my YAML, and I felt like there was a lot of times where they were really just mirroring each other, or it felt like one should be able to be sort of inferred from the other, and it was actually the product of many years of feeling this and trying to iterate on it, and try to move the needle where I started coming up with this notion of infrastructure from code rather than infrastructure as code. That's sort of my pet term that I've been holding onto and trying to personalize.
Doug: But yeah, it really was born out of just something I always ask, is, "Can this be better? Can I write less code?" It's not about not writing any code. It's about writing as little code as is necessary to communicate what you're trying to do, and avoid any kind of quirks that come up when you're forced to repeat things, and then one of them changes and the other one doesn't change to match it. So for me, Serverless Cloud is sort of like trying to find that perfect sweet spot in this sort of low code/no code movement that's happening, but without taking any power away, just finding those common patterns that you're going to inevitably write anyway, the boilerplate, the undifferentiated heavy lifting as it's been called, and just making that one less thing to worry about, which is really what I think the serverless movement is.
Doug: It's not about no servers, as everyone likes to rib. It's really about less to worry about, less things for you to manage, less things for you to provision. So Serverless Cloud is just, to me, that next iteration. You know, and I think it's revolutionary. Calling it an iteration is probably a bit of an understatement, but it's about finding the next set of things that you don't have to worry about, you know? And some of that comes in the form of our abstractions, where we're just letting you express what you're trying to do cleanly and succinctly, and in other cases, it's about the infrastructure and runtime, where the less stuff you have to be specific about, the more things we can tune or optimize under the hood, in order to give you the best developer experience, as well as your application the best operational behaviors.
Jeremy: Yeah, and I'm actually curious to follow up on that with you, Eslam, because you've been working on the Serverless Framework, the open source project that has been around now, like what for five years or something like that. You've been working on that for quite some time, and I'm curious, you know, Doug sort of mentions, and he didn't maybe use this exactly term, but this sort of impedance mismatch of when you're trying to build a serverless application, there's been this promise of just write code, and then it all turned into configuration, so now it's like just write code, and a whole bunch of configuration. And that's sort of the path that the Serverless Framework took. That's the path that obviously CloudFormation has, and a lot of these other ones, even things like AWS SDK, I'm sorry, AWS CDK, still requiring you to write a lot of code in order to tell it how to configure the service.
Jeremy: So I'm curious from your perspective, where you started with the Serverless Framework, and what was available there, and what were the challenges there, and then moving onto something called Serverless Components, which I'm not sure everyone's familiar with, but was a way that you were trying to do, in a similar way, what we're doing with Cloud, to kind of encapsulate these use cases, and then maybe where that's gone with Cloud. So just from your perspective, that developer requirements, how have you seen that sort of grow, what the developer has to do, and your perspective on maybe reducing that?
Eslam: Yeah. So, the Framework, it started a long time ago, where it's like, this was 2015, I think at the time, serverless was really immature, and Lambda just got out, you know? I don't even remember there were Dynamo back then probably, but there were very few services that are truly serverless, and the Framework came to sort of help all of these together, tie them together. So at the time, there wasn't a lot of configuration. It was great, actually.
Jeremy: Right.
Eslam: There were less dependency on CloudFormation also, so you just have to worry about the Lambda functions, API gateway, and maybe like an S3 pocket, and a table, whatnot. So it was really simple, and even the Lambda configuration itself wasn't that much. You know? They didn't support everything. They didn't support all languages and all the runtimes that they have now, but it was a lot simpler at the time, and no one ever really worried about the configuration.
Eslam: But fast-forward a few years, you see how YAML files look like now. It's still a lot less than CloudFormation, and it still helps you a lot when you're working with a lot of Lambdas, but at the end of the day, you're still going to have to know CloudFormation to build anything, even a little bit of CloudFormation will be required, at least for the tables, for the buckets, and last of all, the provision stuff.
Eslam: So getting started with serverless with the Framework was easy, but building that and taking that to production required knowledge of AWS, like real knowledge. Framework abstracts a lot of stuff for you, but at the end of the day, you're working with AWS, your own account, and the Framework just lays the framework for you, you know? Like, lays the road ahead, just to figure out how to do this, but you still needed to learn a lot about serverless, and to put things together.
Eslam: So what we noticed, after a while, is that people were just starting to build use cases, and they didn't really care about what kind of infrastructure they're using. A lot of them were just using examples. We had a lot of examples for the Framework. So someone wanted like a user service, you know? They didn't care that much that it must use DynamoDB, but they know that this is a serverless solution for it, so they just tried to figure out how to use it, and put together the right resources, and a YAML file, and they wanted to have like an image resizer, they needed an [inaudible 00:12:21] bucket or whatnot, so this made us realize that we want use cases, and this is where we started to evolve, because this is how I think about Components and Data Cloud, just like we're evolving to the next abstraction, basically.
Eslam: So with Components, it just offers a way for you to abstract use cases rather than worry about infrastructure. When we came up with Components, one of the most popular use cases is a website, so it lets you deploy an entire website on AWS, your own account still, so you still have to figure out at least the provision, you know? At least data credentials. You need to create an AWS account or whatnot, but you don't have to worry about the resources as much, so the config shrinks. It really shrinks, so you just had to provide a couple of inputs, you put it in the Component input, and then everything gets deployed for you. For these particular use cases, it was great, and it worked really well. It got a little bit harder when you tried to do something a little different, or edge cases for example, because you can only do what you can configure. Other than that, you're going to have to create your own Component. I'm not going to get into how to do that right now, but yeah.
Jeremy: Well, we've moved past Components. I mean, I know Components-
Eslam: That's the thing, yeah.
Jeremy: ... are still out there, but I think that's right. I think the point you made about Components was you try to encapsulate a use case and make it super easy, which is sort of what CDK constructs are doing, or I forget what they're called in Pulumi, but they have a similar construct-type thing, where you can kind of encapsulate a bunch of functionality, and sort of bake that all in and make it easy for you just to put a few inputs in. But I think as we transfer to Cloud, I think, at least for me, my biggest sort of complaint about Components was a little bit of a lack of control, where you couldn't do some of that extra configuration. So from a Cloud perspective, I'll let you explain this, but what was that evolution now, where it's like this balance, right, between control as ease of use.
Eslam: Yeah, so with Cloud, I'm going to start with ease of use, because one of the biggest issues we had with Components is that now we have very simple use cases, but people still needed their own accounts, their own AWS accounts, and they needed to figure out their credentials and whatnot, so Cloud helped that a lot, because now you don't need to worry about AWS at all. You don't need to worry about credentials, sign up and all of that. With Cloud, you can just start the CLI, log into the Serverless, and that's it. You're good to go. You'll be able to deploy your own application right away, so it's super, super easy, and we've taken a lot of what we've done with Components and put it into Cloud, but even better and faster, because you control the environment. That way, you can make better use cases. It became a lot simpler for you to run an application, and we offered more use cases that gives you more control on how you can configure it through the runtime.
Eslam: At the time, we didn't really have a runtime back with Components. It was just a set of components basically, but the runtime gives you a lot of things out of the box without having to deploy them. You have data, serverless data. You have the [inaudible 00:15:39] and the testing. All of these things, we didn't have in Components, and things that users actually needed, so in that sense, you could do a lot more than you could have done with Components, and it made things easier, a lot easier I think.
Doug: If I'm going to just add that as an early adopter of the Framework, the plugins were the really attractive thing to me, and I always explained it to my colleagues, and we did a lot of workshops at my previous job, on the Serverless Framework, and I was always like, "You know, it's ultimately a CloudFormation generator, but the ability to write plugins gives you this programmatic power that you just can't get from raw declarative templating," right? And then of course, things like CDK have come along, but even in all that space, it feels very sort of outside-in, and one of the main motivators for Cloud for me was more of an inside-out.
Doug: Like, I always felt that a lot of the elements of even just a language, the runtime itself, whether it's Node, or in this case, it's heavily Node based right now, but it felt like a lot of the things you would want to do, you could actually express just as like vanilla language constructs or vanilla runtime constructs, and we ought to be able to infer the infrastructure from them, so it felt like it was really kind of flipping the perspective of how you're looking at what's necessary or how you're creating your applications, and that's what I think is one of the things that differentiates Cloud from other sort of programmatic automation deployment platforms.
Rebecca: Yeah, I'm curious. Eslam, you talked a little bit about like when serverless was really nascent and young, right? It was actually really simple because there wasn't a lot to configure, because there was really just a few truly serverless services, and then with some insider intel that I think I know, Ben, you actually entered the serverless space when there were probably way more tools, it was actually way more overloaded, there were way more choices, so I'm curious, Ben, if you can talk a little bit about... It's like Eslam went from simple, to overloaded and complicated, to making it simpler, but you entered, Ben, I think basically when it was... Overloaded is not necessarily the right term. It was much more complex in terms of the amount of choices that there were, so what that experience has been for you, starting out where it was actually you entered into it when there were so many choices, and now you're watching it simplify through this Serverless Cloud launch and moving in that direction.
Ben: Yeah, totally. At the end of my college career, I had the great opportunity to intern at this kind of national startup called [Loompa 00:18:02]. They specialize in music technology, and the CTO there, who I was kind of working under, was very much like very about the next big thing, and using the latest and greatest, so when I joined the full-stack web team, he was all serverless. He was kind of showing me serverless. He had just discovered it, and he was like, "I've wanted a way..." Because they were running .NET servers on like, I think it was like... I can't remember the hosting service.
Ben: It wasn't AWS. It was like huge .NET monoliths, and he was like, "But there has got to be a better way, and they're trying to save..." These are multi-tenant applications for multiple, multiple festivals at the same time, but running the same code, so they needed a way to containerize and scale automatically, because that was a big thing, was scale, because these festivals would have 10, to 20, to 50,000 people getting onto the app at the same time on a weekend, and scale became a big issue, because they were like, "Well, now we have to have some, but we have to have a devops guy, or girl, you know? Like watching when the festival starts, and scaling up our servers."
Ben: So we adopted the serverless thing and kind of became obsessed with it, to be frank. We started building everything in it, which kind of came to a fault. We weren't maintaining the same patterns, so every new product was slightly different, but by the time I entered the space, it was definitely a mature framework. There was tons of plugins. It was kind of funny. I was telling Doug when I first joined, we had to use his split stacks plugin, because we maxed our CloudFormation stacks, and I was like, "I know this name. Oh, that's why I know this name, because I had to..." And he just replied, "I'm sorry you had to use that."
Ben: But yeah, so it was a really cool way to jump into AWS, and just like distributed computing. I was fresh out of college. I had no idea what I was doing, so I thought this was like the standard kind of thing, because it was my first time... It was my first job, so I thought this was what everyone was doing, and then I went to my second job and realized no, we were way ahead, and I was kind of just confused a little bit, why not everyone is doing it this way. And it makes sense, you know? Some people have their teams used to what they're doing, and CloudFormation is a huge learning curve, even with a framework as good as Serverless, so it's like, a lot of people are scared to jump in, but once you do, it makes a lot of sense.
Jeremy: Yeah, well I think it's Corey Quinn, "A legacy app is just another term for apps that make money," and that's part of the problem, is that you have so much legacy infrastructure, because you know, we are ahead. I mean, I think Serverless, and even though it's been around for five, six years now, the tooling is just starting to mature. We're just starting to find these things where it's easier for people to build.
Jeremy: You know, we talk to customers all the time at Serverless Inc. and one of our recent customers basically said, "We love the idea of Cloud, because we have a small team. We need to move fast. We've got a lot of big ideas, but we don't have the time to learn the nuances of API Gateway, and Lambda, and Fargate, and DynamoDB, and the permissions model, and all these other things, and cross accounts, and all the best practices," and that's not to say that these things are not valuable to learn, because I think they really are, but I'm curious, just to go back to you, Ben, when you encountered this serverless paradigm, and cloud, I mean just cloud in general, forget about serverless, just cloud in general.
Jeremy: I mean, did you feel like you were kind of hit with this wall of like there is just so much to learn that they didn't teach me in college? Because I think that, to me, when I first encountered this, cloud wasn't... I started working in cloud in 2009, so for me, it was a slow evolution, and I learned as new things came out. I can't imagine hitting this wall right now, and just feeling overwhelmed. I don't know if you had that experience.
Ben: Definitely. It's always a tip of the iceberg. You would figure out one service, or like how this one thing works, and then you'd need to use one of their other services, like insert Amazon name here, and it was like, "Oh, I have to go learn this," and how do I do this with the Framework. That was a big thing too. Like, we would have this new service. We tried out AppSync for one product. Didn't have a good experience, but the whole thing was like, "How do we get this to use... How do we deploy this with Serverless," you know?
Ben: It was kind of strange. I found how valuable AWS knowledge is pretty early on, because I was kind of forced into it. I had to understand and learn how AWS worked, and then I realized, that was a huge selling point going forward, because not a lot of... You know, it's usually separated, right? You have your devops people, who understand AWS very well, and then you have your engineers and developers, who just build on the things that the devops people put together, and I think at my first company, I was really lucky to be at a company that was kind of joining it, like most are doing now, to where I had this really broad understanding of distributed computing, and AWS specifically, going forward. And while overwhelming, and I still haven't understood everything, it definitely has helped building good software.
Jeremy: Yeah, so you mentioned devops, right? And I guess this classic ops versus dev mentality, and now we're kind of in the devops thing. I gave a talk recently, about this contention, where the serverless spectrum of less ops, or more ops means less developer responsibility, whereas less ops now means more developer responsibility, right? Sort of as you go through that continuum, it's like, "Hey, the ops people have to do less, but guess what. Now the developer's more responsible for that configuration," and so forth. So I'm just curious, and maybe, I don't know, Doug, if you want to answer this, what are your thoughts about that responsibility shift to the developer, and where... Is serverless... I mean, there's a huge benefit to serverless. I mean, articulating all of them is probably hard right now, but there's so many benefits to serverless, but that does put a lot of responsibility now on the developer.
Doug: It sure does, and I mean, I think it cuts both ways. It's a blessing and a curse, because you put more responsibility on the developer, which means the developer starts making code changes or decisions. I think we've all experienced or at least heard of the notion of the attitude of just throwing something over the wall and then it's someone else's problem to run it, or you write whatever query you want and it's up to the DBA to figure out how to make it run performantly or to come back and yell at you once it takes down production. Anything that can help alleviate that is, in my opinion, a good thing, but it also means that it does put more cognitive load on you, as someone with ADHD, like interruptions are just brutal for me, so you know, seeing notifications, alerts, all those kind of things do weigh heavily, so it can be overwhelming at times, or it can be something that's not necessarily just a purely... It's net positive, but it's not all just obviously better, you know?
Doug: But when you look at it with a serverless lens, at least, I'm going to reiterate or repeat myself here, it's less for you to worry about. If I was devops on something where it was completely unmanaged, I think that would be just so much worse, so at least you've got that sort of new attitude or that new paradigm of there's a lot less for you to worry about, so I like it because it forces me to think about performance. It forces me to think about cost effectiveness. It really does try to... And it's faster, right? The connection between the changes you're making in a pull request and the effect it's going to have on the system just feels like it's a much shorter loop for feedback, so I really do think it's a net positive, but I think it's also unfair to just put it in a light that ignores the fact that it does create additional cognitive load on developers.
Eslam: I think it's a lot like... Sorry to interrupt there, but I just want to say that I think it's a lot like testing. You know, whenever I do some integration testing or unit testing, it's your responsibility as the developer to do that, not some tester, right? But with devops, it's also now your responsibility to make sure that all of these operations should be... like, you should set it up easily, you should be responsible to maintaining it, and not just leave it for someone else. So I do agree with Doug. It's about net positive, I think, but you don't need to worry about as much as you used to, so I think it's good.
Doug: Yeah, and I think that there's a right and a wrong way to go about it, right? If an organization thinks that devops is a pixie dust you sprinkle, and then magically you get more with the same input, no, right? But just like testing, you're right, just like accessibility, all of those things, they're things that you have to have a perspective of, right, in totality. But the organization also has to understand that you're asking people to do more, and adjust their expectations accordingly.
Rebecca: Yeah, so it sounds like, I mean, both of you are addressing like a friction or a balance, and then it is trying to figure out how to balance those things. I'm wondering, because we have kind of three different folks here, that are at different stages of their career, or came into even cloud or serverless at different times. If you see benefits in, let's say air quotes, "The struggle," right? Having struggled before, let's say making a shift in mindset, or a shift in dev and ops, or whether or not it's a shift of going from containers, applications, to serverless, whatever that is, if that struggle actually ends up being something that you're like, "I'm actually really glad that was really hard, because now I think of this in a different way"? I see, as I'm thinking, he's like, "Was the struggle ever work it?"
Eslam: No, actually, I do agree. When I think a lot, it was a struggle, like to deploy and whatnot, one of the things that I loved about the Framework is this live URL that gets created at the end whenever you deploy, and thinking that it's skeletal, like it's production ready, before you had to set up your stages to be the same exact environment and whatnot, like the Framework, you didn't have to do all this. With Framework, you don't need to worry about servers of course. I felt that it's always ready for production, from day one, and it was a struggle, I think. Like, before, I used to always think like, "I've developed the application. Now, how do I take it to broad and make sure it scales?" I don't need to worry as much right now, you know? So yeah, no, no. I was thinking about the struggle.
Doug: Yeah, for my perspective, the struggle is a different form of, I guess Cunningham's law or something, right? Where the surest way to get an answer is to post the wrong one, and then someone corrects you. With developers, like the surest way to improve things is to make them feel pain, right? And then they respond, right? Whereas, I mean, there's no shortage of detractors. I can still find people today who are like, "The cloud is a fad," right? People still say that to this day, so it's not... The struggle is manyfold, and I think the reality is just that these pain points are just another way that you inspire people to improve the situation. If things were comfortable, I think we'd be at risk of not innovating enough.
Jeremy: And I'm curious, Ben, your perspective, because I think this is just the way for you now, right? If anything, you experienced it backwards. You're like, "Oh, this complex cloud serverless thing that we got to learn," then you go back and be like, "Oh, this old way is... Wow, that's kind of crazy."
Rebecca: Benjamin Button over here, going backwards.
Ben: I know. It's interesting. Yeah, it happened to me with like frontend technology too. Like, I started with... I think I started with React Native, to iOS, then to web development. Yeah, I went backwards. It was very strange. I had a weird start to my career. But yeah, I was going to say, it's definitely... The learning curves that were introduced, I think made me appreciate what cloud is now, but someone 10 years ago would be laughing at me like, "Oh, you haven't had to worry about an SSL token on your server before," and you know, those things still exist, but I guess like my point is wasn't there a tweet or something like, "The next AWS will be built on AWS"?
Ben: So people five years, or kids graduating from college five to seven years from now will be using something, hopefully cloud, but something very much like cloud, and they'll have no idea what goes behind it, and it'll just be further and further learning... They'll get frustrated with something in there, and then something else better will come out, and it's just going to be kind of like... Well, it's just the natural advancement, I think. I don't know if that was a great answer. I was kind of scrambling there.
Jeremy: That was great. And you know, again, not to make fun of you because you don't understand how it used to be, but we did have to walk uphill both ways in the snow with servers on our backs to the co-location facility to run applications in the past. I posted that on Twitter, so I'm reusing a joke, but anyways, so I do want to go into something, though, that Eslam keeps alluding to, and this is the idea of functions was the original sort of intent with what was happening with the Serverless Framework, and that was to say that functions, the execution itself, the actual execution environment sort of was the target, and then the Serverless Framework did this sort of brilliant thing, which I'm pretty sure was sort of invented by them or at least sort of codified by them and everyone else has adopted, is this idea of the event sourcing or the event model, where you have these event mappings that go to these individual services, or sorry, that go to the Lambda functions and execute these environments.
Jeremy: So for the most part, that's what the Framework has focused on, was this idea of you create some sort of mapping, and sometimes that mapping might generate resources like an API gateway, or an SQS queue, or something like that, that goes to the function. But if you want to build DynamoDB, you want EventBridge, you want some of these other services, you still have to write those as CloudFormation, and those to me are the primitives. Those are the things where it's like, now I need to start understanding not only what I want to do from an execution standpoint, but I have to understand what other primitives or what other resources I need to create. What are the limits of those resources? What are the nuances of those resources? How do they connect together? What are the permission models, which is, again, I think takes a PhD in many cases, for you to figure out how to do the permissions correctly.
Jeremy: So I'm curious, and maybe just ask this as a more general question, whoever wants to jump in, but this idea of abstractions, and the function aspect of what the Framework did was a bit of an abstraction, to try to abstract away the event mapping, but then a lot of these other things, like interacting with Dynamo and so forth, are all things that you have to explicitly call out. So I'm wondering if, from any of your standpoints, what's that right level of abstraction? And is that something that you believe... I know we've tried to get rid of that in a way in the Cloud, but is that something you believe, that all of those primitives become too burdensome, or maybe they're too much of a distraction, and we need a better abstraction on top of that?
Doug: I'll jump in first, because this is something that's very near and dear to me. I mean, first of all, all abstractions fail at some point, right? So I don't know that I believe that there's a singular abstraction that can work for everybody, but my sort of vector on it is the ideal abstraction to me is a programming language, right? I don't really want to think about anything else. Anything after a program that just looks like a computer that has thousands of cores, to me is an opportunity to try to find a better abstraction for that runtime environment. I don't know if that answers your question, but that's sort of the holy grail to me, is why do we have to think about multiple computers? Why do we even have to think about queues, or message passing, or anything like that? All of those things-
Eslam: [crosstalk 00:34:08]
Doug: ... to me, should be emergent properties. Yeah, just write code is really what I want. And I think there's a lot of stuff that can be gleaned from... I love Erlang as a source of inspiration for things, and there's lots of other interesting projects, databases, lots of forgotten technology to be rediscovered, not necessarily to like reuse it, but even just inspirationally, right? But in some cases, yeah, just use the thing that's been built and run for a long time, but yeah, to me, it's about looking at what's been done, not throwing away the past, and keeping the end goal in mind, which is just write code.
Jeremy: And it's funny that you mentioned that, because we were just talking to the step functions team at AWS in our last conversation, and we were talking just about this idea of microservices, and high cohesion, or tight coupling, and loose coupling, and all these concepts around microservices, and how all of this stuff is coming back up again because the technology is so accessible, through step functions, and EventBridge, and those sort of things. I think the quote was, you know, "Everything that was old is new again," because that's what we're starting to see, is a lot of these old patterns that were developed in the '60s and the '70s, that are coming back to life now, and some of these books are out of print. You can't even get them anymore, but the concepts are all there.
Jeremy: So I think that's really interesting, is this idea is to say that there are patterns that we've used in the past, that worked really, really well, and we can apply those patterns to a distributed infrastructure, but understanding where that step is. Where is that abstraction that says I can go from this pattern and apply it to an infrastructure? Is really hard.
Eslam: Yeah. I can actually give an example of that, because I was going to say the same thing that Doug said, that like, and no abstraction is perfect, you know? Because what we had with Components, we had the [inaudible 00:36:01] component, I think it was, because we didn't want this component to be just focused on our website, and we wanted users to have the ability to also set up like Cloudfront, because it's based on Cloudfront [inaudible 00:36:12] I'm not going to get to the details, but we wanted to be flexible. We wanted to cover all the cases, and what we ended up doing, we created a component for S3, a component for Cloudfront, and I think a component for IM. So these were the primitives, I think what you referred to Jeremy.
Eslam: Then on top of that, to use these components and create [inaudible 00:36:34] component, [inaudible 00:36:34] component itself is a very thin layer, and it worked for a while, but then all of these dependencies made it very hard to update these components, pull out breaking changes, and we have to think about the versioning and whatnot. So this is when we realized, okay, we need to create a completely separated abstraction that is [inaudible 00:36:56] that doesn't depend on these primitives. When we did that, we suffered from the flexibility that we had, but we made it simpler. So it's really hard to have this balance, or it's really very hard to answer this question, but we strive to it, and based on feedback, I think we just keep adding as much config or not config, like features to [inaudible 00:37:21] but I'm sure there will always be edge cases, you know? For those probably different abstractions. It's hard.
Jeremy: Many years ago, I stumbled across a blog post, and it was, "What happens when I enter a URL in my address bar?" Right? I thought to myself, "Okay, this is a question that I might ask someone when I'm interviewing them. It's going to feel really self-assuring to just go through and read a bunch of stuff I already know, right?" So I'm like, "Ah, it's going to be about HTTP, and I know everything there is to know, or I think I know everything about HTTP," but it's like, I went from I hit the letter H, and it talked about the interrupt control in the keyboard, and the LCD rendering thing, and it was just... It just totally knocked me off my high horse, right? And-
Eslam: [crosstalk 00:38:03]
Jeremy: Right? And I was like, "I know nothing about how any of this works."
Doug: [crosstalk 00:38:07]
Jeremy: Right? And I love those experiences, because they're humbling, and they teach you a lot, and it also really highlights to me that like, in your abstraction, detail is actually somebody else's abstraction, and it's just turtles all the way down. It's abstractions all the way down. You can just keep going until you get to the silicon that we've tricked into thinking, right?
Rebecca: Yeah, I want to take that-
Doug: [crosstalk 00:38:27]
Rebecca: Well, first Eslam, I just want to say you said, "It's hard." That's how you ended your answer, and I was just like, "You know what, actually? That's just what needs to be put on the shirt, like 'It's hard.'" Simply put, it's all pretty hard, or just complex. But when you're talking about these things, right, where you realize it's abstractions on abstractions, to me, that's even this idea of what let's say Serverless Framework or Serverless Cloud is trying to do, in terms of saying like, "We want to create a better developer experience overall when you're trying to use something like Serverless."
Rebecca: There's something to that, where what did you say, "One person's abstraction, another person's detail"? I think when we talk about developer experience, so often we're like, "Oh, that's a bad developer experience or this is a good one," but what that doesn't do is identify that each developer is actually like a unique functioning human being, that has their own habits, and their own ways of doing things, and their own ways that their brain works, that they prefer to work. So I'm curious that when you're developing something like Serverless Cloud, all being three unique individuals as different developers, did you ever come to a head, where you were like, "That's not how we should make this experience, because we should actually do it this way, and that would make it better"? And were there any moments where you almost had that moment where you said, "Oh, this actually... This either all funnels together, and we know what that experience should be, and it would make it better for everyone" or "We're running into this fork, where I think the experience would be better this way, and you think it would be better this way," and then how do you end up kind of bringing those things together to create one individual improved experience or product?
Eslam: One, [inaudible 00:40:05] that's a hard part, but yeah, I think developer experience maybe depends on your target audience, and you cannot just have a good developer experience overall. It has to be like the developer experience for customer X or developer X, who has this requirement, so that's what they care about. So for like if we go to the example of Components, I think it's a very good example, because it sits in the middle, you know? It's not very complicated, but it's not very simple, like Cloud. We had two personas.
Eslam: There are people who are new to AWS. These build the one things that are zero config, you know? They don't know anything about AWS, they don't want to learn. They want to learn, but they don't have the time to learn, so they just want something that will just work, you know? Just a single command, whatnot. So the developer experience for those just is like a single command, zero config, just get your website out there, just provide your code, and they don't have a lot of edge cases, you know? They just want simple website.
Eslam: But then you have the other persona, like people who are experienced with AWS, and these people, they want flexibility. They want as much configuration as they could. They have requirements, you know? They know how it works, and even if you try to make it easy for them, they will ask about a missing feature. So for them, I think the best developer experience to give them all configurations that they need, and I would even go as far as say you don't even abstract it, because it could distract them, because they're used to sort of documentation, they're used to AWS CloudFormation and whatnot, but they might not like if you change the property name, for example, you know?
Eslam: So again, it depends on your target audience, and if you try to make a tool for both, it's hard to have a balance, and you know, this is [inaudible 00:42:00] what makes it hard, you know? Is that you need to find this centerpiece, you know? Like the magic line, you know? Between not too simple, not too flexible, and just in the middle, to help them both. But yeah, if they're too far ahead, we're going to need two different products or two different tools, but [inaudible 00:42:24] you could just focus on what they both need.
Jeremy: I'd love to get Doug's perspective on this, but before I get there, I want to talk to you, Ben. As someone who came in... Again, honestly, I wish this was sort of maybe my experience too, is you come into a world where you've got VSCode. You've got autocompletion in your IDE. You've got plugins that allow you to do all these kind of crazy things. When I first started writing HTML, and CSS, and Perl scripts way back in the day, I was just using a text editor, because that's what there was, right? And the developer experience have gotten better. Some of them have gotten convoluted. I don't want to dismiss that. But I'm just curious, from your perspective as someone who comes in with a whole bunch of tools, and some really good developer experience, I think GitHub is a great developer experience, again VSCode is great, there's a whole bunch of other tools that are out there. I mean, what is the expectation, maybe, for people new to the industry, or relatively new to the industry, what's the expectation of a developer experience, of the developer experience?
Ben: Definitely the... I've noticed... I mean, I used to do this too, when I was learning how to code, was relying on the IDE to pre-fill what you can call, right? So you would put a dot on some class, and then you'd see all these options, like, "Oh, I want this one," and obviously, as you grow and get more knowledge, you rely way less on that, but just starting to learn how to code with that tooling, and inspection tools, kind of I think that becomes the standard, right? And then someone perhaps switching from a Java environment, where they learned to more JavaScript-y, or even Python, where there's more interpreted, less structure to it, for better or for worse.
Ben: It does become a little more daunting, I think, even with all the tools you can get, because you are used to this kind of like, "Oh, I have everything right here. I can put a break point," because that's how everyone learns in college, is, "Oh, I'll put a break point and step through this function," and then you kind of learn the hard way that if you want to do break points in JavaScript, you have to put some more effort in.
Ben: I would say it hasn't really changed that much though. I mean, obviously no one's... Everyone's using VSCode, everyone's using plugins, and everyone's using whatever they can get really. I wouldn't really say there's one specific feature that newer developers are expecting by default, or I'm naïve to it, you know? Or I've just had it this whole time, that I didn't know it was something that was that crucial, that everyone just uses today.
Jeremy: Well, maybe I can add some perspective, and Doug, you can maybe comment on this as well. Remember when browsers didn't have an inspect panel? It was view source. That was basically the extent of your debugging, as well as like even when you were writing code, and you were testing that code, often single-threaded applications, a lot of log statements, lot of exit statements, trying to step through the code, trying to figure out where things break.
Eslam: [crosstalk 00:45:37]
Jeremy: Eslam agrees. I'm just curious, like I mean, this evolution of developer experiences, and look, not everybody nails them. I mean, I can name a couple that are pretty bad. I won't, because I don't want to call anybody out, but I mean, your perspective on developer experiences, I mean, how much does that, how much is that, or how important is that, in order to get people to learn what you're trying to do, get them to be more productive, get them to get something to production faster? I mean, how much weight would you put behind developer experience?
Doug: Well, I mean, hey, I think I remember when browsers didn't have buttons, so never mind inspect element, right? We are in the era of developer experience, right? There's so many options to choose from, if you're not prioritizing that, you're going to easily be passed by, because that's it. That's the name of the game right now, is DX.
Doug: I just wanted to say that in the topic of Serverless Cloud and abstractions, I think having the diverse set of experiences on the team that we've talked about here helps keep in mind that there's not probably one right way to do things, and I'm actually kind of impressed that we haven't butted heads so much. Rather, I think we've talked about what are the layers we're trying to offer, right? We've got a simple API helper to get people up and running, but we've also got a bring your own framework, because we know people are going to maybe want to not write things exactly that way, so I think we're actually striking a pretty good balance, in terms of offering an opinion but not locking or boxing you into that opinion as well. So it's batteries included, but you can take them out and replace... That's a terrible metaphor, but that's the thing we're going for right? Is trying to give people a great place to start, but not limit them on how far they can take it.
Doug: In that spirit, developer experience, as I said, is everything. I think that's our main motivator. Every discussion we have is about how it's affecting developer experience, or how someone might see it, and when there are times when we are looking at it from different perspectives, because we are at different stages in our careers or our brains just work differently, and we just have different comfort levels with how we look at things. We tend to try, then, to split it up into, well, what's the nice to have that we're offering and then what are some of the lower-level things that we might be able to offer, to give people the ability to sort of diverge somewhat from what we're offering?
Rebecca: So, Shawn Wang, who is known as Swyx, if you follow him on Twitter, he wrote an article about self-provisioning runtimes, and Doug, you even talked about this at the very beginning of this conversation, and I'm wondering how you see that, how you think about it. You could say cloud as a computer, if you will.
Doug: Yeah. That is, I think, exactly how I think of it. The cloud is the computer, and we're just talking about the next generation of complex instruction sets, where running an entire sort of program is just a thing that you can ask it to do, and it's got thousands of cores, and many, many terabytes of memory, or whatever you want to imagine to your heart's content. When I think of... I really like the term "self-provisioning runtime," and I've actually been like mulling over a follow-up article, where I'm trying to talk about a self-replicating runtime, because I think as we move from cloud into fog, there's going to be this era where you write code and you don't even know where it's going to run, and it might move, and get re-hosted, and things like that, right? So you're not just deploying to one fixed location.
Doug: That's, I think, why also abstractions are so important, because the runtime really needs to be the thing that decides what's best, and I believe in this with all of my heart. It's like the pinnacle of what I think my career has led to. It's just that as a developer, every time I make a decision, I'm probably wrong, you know? I might be right at the time, but then the environment changes, the real-world production data changes, and the last thing I want to do is spend an entire day hand-optimizing database indices, only to find out that that's not the access pattern that actually came up. And while you can absolutely try to gather data, and have OI, and all the things that you want, then your users change their mind, or something happens outside of your control. You can keep chasing that feedback loop or trying to make it possible, or you can just step back and say, "You know what? What if the runtime just figured out what was best for my application?"
Doug: That's really where, not to get too lofty, but that's where I see the future of cloud and fog computing, is just it's about all the decisions that you might have to make, not needing to make them, because making them is an expensive exercise, and is fixed at a point in time, and the more you can push into that runtime, where it decides, and in the concept of self-provisioning, I think it will eventually get to the point where the database itself is self-provisioning. And I don't just mean spinning up another node. I mean the database decides when to materialize data, when something should be replicated or not. Like, I really think... And if I'm not mistaken, I think that even Oracle Cloud has their autonomous database, that I think everyone just ignores Oracle because of the name.
Doug: But I think they've actually got some neat stuff there, and there are other legacy database systems that had these types of features too. So again, everything old is new. The self-provisioning runtime, to me, is just a very succinct way of explaining putting more things into machine control, and to me, that's the future. It's more automation, more self-automation, more program synthesis. I really think that that's the future we're looking at, because there's just too much code. There's too much code to write. There's too much stuff to keep in your head, and it's just getting worse, but I think it's going to get better as more of these aspects of self-provisioning runtimes come into play.
Rebecca: Almost sounds like you know when you've got to clean your house or your closet, it's going to get dirtier before it gets cleaner? Like, it has to get messier. Stuff has to come out and go... And then you look around. You're like, "I thought I was cleaning, but it's actually just messier." But that is. It's like getting worse to get better, right? you've got to move stuff around, to rearrange and figure out where it's going to go. Then afterwards, you're like, "Ah, this is where we were headed to."
Jeremy: Yeah, I wish I could say that about my kids cleaning their room, but somehow, after they clean their room, they're worse off than they were before. Anyways, we're running out of time, so we do need to wrap up, but while we had Doug here, Doug, you wrote an NPM package called serverless-http, which is hugely popular, has a ton of usage. People who aren't familiar with it, essentially allows you to use Express or any of the rest/req format HTTP frameworks, and bring that into a Lambda function. I just would love to get your perspective, like what was it that you were like, "I absolutely have to write this"? Then also, sort of what's the experience been with having such a popular open source package?
Doug: Yeah. Honestly, like I alluded to, we started using serverless at my other employer, D2L, pretty early in the game, so my motivation was honestly more... Not sure if Lambda's going to... Cold start times weren't as good as they were, you know? It was just much more... Things were a little bit rockier, so the motivation was to be able to write something, and if we had to, quickly throw it into a container instead, right? So it was meant to give you that flexibility to A, stick with what you already knew, right? A lot of middleware already out there, a lot of people already knew how they wanted to structure applications, but also the escape hatch to be like, "Okay, Lambda isn't handling our use cases very well. Let's throw it in Docker and run it on EC2 or ECS" at the time.
Jeremy: And what about the success of it? I think it has like 280,000 downloads a week or something crazy like that. I don't know. Maybe not that high.
Doug: Yeah, I mean, that's humbling and amazing, and I never thought that would be the case. It was something I wrote in an afternoon. It was just like, you know, you get that idea, the muse hits you, and you're just like, "Oh, what if it worked this way?" Honestly, it's just the internal node Reqres objects that I wrote a mocking layer around, to trick it into thinking that it had a real socket connection. You know, those moments when they happen, the creativity hits you and you do them, but I wish I could make it happen on demand. It doesn't work that way.
Eslam: It's much-
Jeremy: Well, I'm glad-
Eslam: It's much smaller than AWS's version of it, because when we start with the component, I think just the express component, we used AWS's package. I don't know what [inaudible 00:53:43] it was called, and it was [crosstalk 00:53:43]
Jeremy: I think it was called Serverless Express, right?
Eslam: I think so. Yeah, yeah, yeah. Yeah, and it was like huge. I don't know what they were doing over there, but it was out of date with the new Lambda payload, and it had a lot of issues. Then we migrated to serverless-http, and it just worked. Then I looked at the code. It's a lot simpler, as... He did a great job. Like, seriously.
Jeremy: No, it's amazing, and I am so glad that Doug is part of the team, and that we actually use it as part of Cloud, because it does allow us to do that bring your own framework, and be able to process a lot of that stuff. All right, unfortunately we're out of time. I mean, I will continue to talk to all of you about all these amazing topics, because I'm lucky enough to work with you every day. The rest of you, though, if you want to reach out to these fine serverless engineers and learn more about them, Twitter probably the best way. Eslam, what's your Twitter handle?
Eslam: Eahefnawy, if you can pronounce my name, but-
Jeremy: I will put it in the show notes, so we'll get it there. Ben?
Ben: I quickly made mine, because I realized everyone here has a Twitter, so it's Devvyben, with two Vs.
Rebecca: Nice.
Jeremy: All right. And then Doug, of course, because he was smart, has @DougMoscrop, his full name, without the hyphen unfortunately, like mine does. Hey, thank you all for being here. This was awesome. Thanks for taking the time.
Doug: Thank you for having us.
Rebecca: Thanks so much, gentlemen.
Ben: Yeah.
Eslam: Thank you guys.
Jeremy: Oh, and don't forget, serverless.com/cloud, and obviously check out Serverless Cloud.
Rebecca: Plus one.
