Episode #115: Serverless Complexity with Ant Stanley
October 18, 2021 • 68 minutes
On this episode, Jeremy and Rebecca chat with Ant Stanley about whether the early promises of serverless have lived up to expectations, what complexities might be preventing or slowing down adoption, how internal competition at AWS might produce better products, the value of transferrable skills, and much more.
Ant is a consultant, community organizer, and co-founder of Homeschool from Senzo. He also founded and currently runs the Serverless User Group in London, is part of the ServerlessDays London organizing team and the global ServerlessDays leadership team. Previously Ant was a co-founder of A Cloud Guru, and was responsible for organizing the first ServerlessConf event in New York in May 2016. Living in London since 2009, Ant's background before Serverless is primarily as a Solution Architect at various organisations, from managed service providers to Tier 1 telecommunications providers. He started his career in 1999 doing Y2K upgrades in his native South Africa, and then spent 5 years being paid to write VB6. His current focus is Serverless, GraphQL and Node.js.
Transcript
Jeremy: Hi, everyone. I'm Jeremy Daly.
Rebecca: And I'm Rebecca Marshburn.
Jeremy: And this is Serverless Chats. Good morning, Rebecca.
Rebecca: Hey. Good morning. What a morning it is. Top of the morning.
Jeremy: Yeah. So it's morning for us. Well, more morning for you, middle morning for me and afternoon for our next guest, who I'm very excited to introduce. So he is an AWS serverless hero, a co-founder of A Cloud Guru, a co-founder of Homeschool from Senzo, a local organizer of ServerlessDays London and a global organizer for ServerlessDays all around the world, and recently a new dad. Ant Stanley is with us today. Hey, Ant. Thanks for joining us.
Ant: Hey, Jeremy and Rebecca. Good to see you two. It's been far too long.
Jeremy: It has been. I haven't seen you in person... I don't even know when the last time was I saw you in person.
Ant: Cardiff 2020. I think ServerlessDays Cardiff.
Jeremy: Oh, right. Right before everything sort of hit the proverbial fan. So are you going to re:Invent this year, by the way?
Ant: No. I have not been on a plane since, I think, January 2020, and I think my next flight will be back to South Africa to see my family.
Jeremy: Oh. Yeah, I think I'm in that same sort of camp with you where last time I flew was back from Nashville in 2020, and I still... I got things booked for re:Invent, but I don't know, it might just be like a small gathering of about 100 people that actually end up showing up.
Ant: Yeah, I think re:Invent will be maybe like 2015 size or 2014 size, so 20, 30,000 people, something like that. It'll be significantly smaller than the last time we were all there.
Rebecca: My favorite thing about that was watching... Jeremy goes, "I don't remember the last time I saw you," and Ant's like check notes, he's like, "Cardiff 2020. I remember the same day. I remember the day, I remember what you said." It was like that, it was so fast.
Ant: I remember every time I see Jeremy.
Jeremy: Oh, yeah. They're always unforgettable moments. So anyway, so 2020 is gone. We're almost done with 2021. It's kind of crazy that things have moved so fast. ServerlessDays has been a big part of a... It's certainly my interaction with the serverless community early on really, other than the virtual events, the virtual events that you've done globally too have been great, but really, I mean, just some of these start in person again. Paris just happened, I think last week or a week ago, two weeks ago, whenever it was, which is kind of exciting. But so besides having a baby and not flying on planes, what have you been doing for the last 18 to 24 months?
Ant: A mix of things. Myself and a partner of mine, Hanna Cevik, we launched a business called Senzo, which we're going to do in-person-based training. We launched February 2020, and we were basically going to do kind of classroom-based training and enable instructors who've got good reputations to go and teach various tech courses, not just serverless, but we were starting with serverless around the world, and we ran one in-person class, and then the entire world shut down. And I don't think that business is going to be resurrected till 2022, maybe even later. But yeah, so we had to do a rapid pivot, and we launched Homeschool, which was an online-based version of that. Basically, taking a three, four day course and turning it into a four week workshop effectively that people would effectively have access to a trainer. That was our big USP is around having access to a trainer. And that business has kind of been ticking along, and I've done some consulting work in between as well with a few different businesses as well.
Ant: Yeah, so that's kind of kept me busy for the last 18 months. The ServerlessDays thing has... Obviously because you can't really be able to do in-person events anymore, obviously it's quieted down significantly, but it was great to see Paris open up and have their first event. There's a few more coming through ServerlessDays, the team in China have been doing really well. I think they've got three events lined up around China coming up, and then hopefully we'll hopefully see more the US and Europe in 2022 hopefully. Yeah, I'd love to see the UK events, I'd love to start planning London again, but yeah, we just hopefully see the numbers start to kind of get back to normal and we're not quite there. I think we're still a few months off that.
Rebecca: In your typical casual way, you're like, "Oh, we just had to do a pivot from a three day in-person class to a four week workshop." That is not a small pivot, and I'm wondering if you... That's an incredible pivot really, that's an incredible thing to be able to package up that's three days and turn it into four weeks. I'm wondering if there's any specific learnings around how you transfer this in-person learning into this four week workshop, which is a wildly spectacular depth of experience to give and have.
Ant: I think one of the key elements, we looked at what was happening in the online training space and basically, it exploded post-lockdown. As soon as the world went to lockdown, any kind of physical education, in-person education was basically not an option. So every person and their dog was suddenly launching an online course, doing digital training, and we looked at it and saying, "Well, how do we differentiate ourselves? We've got this relationship with a group of instructors already that we've been building up for the Senzo business, in-person training business." And the big differentiator for us was always going to be access to the instructor. If you do a lot of these online courses, the good ones have 20, 30, 40, 100,000, 200,000 people who signed up to it. If you get stuck or you have a question, that instructor is never going to see your question because you'll be one of thousands, hundreds of thousands, even millions potentially who's done that course.
Ant: So I said, "Well, actually access to the instructor's really important for a lot of these people because a lot of these courses have very low completion rates." People pay for the course, never do the course, and that's the overwhelming majority of folks who pay for courses and never complete them. So I think a lot of that's because people get stuck. So I said, "Okay, how can we solve this problem?" So the idea was, firstly, let's give access to instructors, and that was our kind of major USP is you'd have access to an instructor, we limited the size of classes, it was time-boxed. So probably spent more time figuring out the format and how to make that work rather than actually the technical work because our learning management system, which we built from scratch, isn't that complex. At one level, it's a glorified playlist, and then we've integrated it with a few other systems. So we said, very true serverless style, "Don't build what already exists." So we said, "Okay, how can we give access to the instructor?"
Ant: So the one way was, okay, chat. Well, we decided not to even build chat, we just used Discord. So we've got a private discord community, which as you sign up, you'll get added to. The other way was we wanted to do live video sessions, we wanted it embedded in the platform, we decided not to build anything, we just literally embedded Vimeo. So Vimeo's platform, they have live streaming capability that you can embed in a private platform, so we just did that. And if you actually look under the hood on our platform, it's like six or seven different pieces of tech tied together using AWS with a functional UI in front of it. So that was a key element to do that, and version one of that took two days to build, and then we just slowly iterated on up from that. But we had something that was good enough in two days of technical work to get something up, of which I say the majority of the work was actually frontend. The backend stuff is relatively simple because all we're really doing is orchestrating other platforms together.
Jeremy: So you mentioned everybody and their dog, and I do not recall seeing your dogs do a training video, which I think would have been kind of funny. It could have been a new thing, dogs teaching people how to do something, and I'm still waiting. I'm going to hold out hope on that one. But the other thing you said, right at the end, you said that the backend wasn't that complicated, that it was just a matter of sort of stringing a couple different services together, but then earlier, you did sort of mention complexity. I mean, that's one of those things where, for me, serverless... Serverless started for me as super simple, like just again, you put a function up there.
Jeremy: I use this analogy all the time, it was like uploading to a CGI bin, but it really felt that magical to me, and then you start adding more complexity to it because you have to, right? Because you need to do more complex things, and of course, the cloud and distributed systems are very complex, and you have to deal with all of the failover and have to deal with the resiliency, and you have to deal with all these different modes in there and all the connectivity that happens between them. And it does become complex. So I'm really interested, sort of this... I guess, what's the right word for it? But there's this tension between serverless being something that we're trying to sell as the next wave, and it's easy or it's very powerful, and it's an easier on-ramp, but then at the same time, it's loaded with complexity. How do you reconcile that in your brain?
Ant: Yeah. So the Homeschool thing is actually a good example where complexity wasn't particularly large. Because of the decisions we made about our business model, which is basically going to be a low number of students, we knew we didn't have to do a whole bunch of things. So we knew it was going to be a low number of students and we're going to offload as much functionality to third-party services as possible. So because we made those decisions, there wasn't a lot to do. One of the pieces of consulting I've done this year is working with a startup who's subsequently launched, and they've got hundreds of thousands of users that they've picked up within days from launch. They had a pretty good marketing campaign leading up to the launch, and post-launch, they had a bunch of pre-orders, and they're now in the hundreds of thousands of users.
Ant: That is significantly more complex, and I think I've been swearing multiple times at my computer screen, at AWS docs, at various other pieces of tooling multiple times because it's not quite working the way you want it to work, or there's something that's really complex that really shouldn't be complex, and I keep thinking like... I've been doing this for six years. 2015 was the first time I wrote a serverless function, and now in 2021, I'm going, "Why is this not easier?" I made a point of making Homeschool as simple as possible to do, but it's not particularly scalable. It's missing a whole bunch of things we wanted to do if we were going to go to thousands of users, but it's literally hundreds of users.
Ant: This other project, hundreds of thousands of users. You can't cut corners, you've got to do things properly from day one, and it's just some things haven't improved. Other things have. I think monitoring observability is 100 times better than it was in 2015, but some of the tooling is just still not there. Your development lifecycle is still quite painful. The easy part is sometimes not that easy because there's a lot of opinionated tooling that keeps you really tied to a single way of doing things, and if you go past that, you're a little bit stuck.
Ant: So like this particular project, we use Cognito. We couldn't use Cognito groups because we're going to have... Because of the way it works and the way permissions work on it, we were going to blow past the maximum number of groups you can have with Cognito. So we had to implement that all ourselves. I'm going, "Okay, this is a bit painful now doing all of this and then trying to implement it." Yeah, we basically have to re-implement a bunch of functionality that exists in AWS because we would have hit a limit on what AWS can do, and also just the whole lifecycle is really, really painful.
Rebecca: Yeah, I want to poke on that a little bit because is it a gap in the tooling, or a myth in the limitation setting, or is a little bit of both? And I'm curious if there are certain parts in the lifecycle where you're like, "I know what I would wish I would see here to help that lifecycle be, let's say, a more round, perfect circle," or is it more like the limitations set on these things aren't actually optimized for the scale that you are seeing people want today?
Ant: It's a bit of both, I think. Cognito, for example, is a great service in terms of cost of performance, an absolutely awful service in terms of user experience and developer experience, and it's not that flexible. If you're doing anything big, you want to use Cognito because it's going to be cheaper than Auth0 or anything else. The alternative is kind of build it yourself, but to actually use it effectively is really painful. So a lot of folks that are building them will then use Amplify with it because Amplify is simplified a lot of the pain of dealing with Cognito, but then you now have introduced the complexity of Amplify into your project, which we can't do everything with Amplify for this particular project. So we've ended up, that particular project, with this bastardize hybrid thing where 90% of the resources are defined in the Serverless Framework, and then all the auth stuffs defined by Amplify, and it's just a pain.
Ant: It's an absolute pain to deal with because of these limitations, tooling limitations, and also the limit of the services as well beyond that. I think part of it is just kind of the way these things are built. They are definitely built in silos, and there's no real... Definitely from the AWS world, there's no real thought of that full end-user experience. You see in the silos dealing with AWS a lot. So that's trade off, and this is where all the complexity comes in, and then it becomes like if you want to build anything real meaningful that's secure, scalable, and all the good things, you have to have fairly competent engineers doing that. You need to have your seniors and your principals leading the way there, which is a barrier effectively.
Rebecca: Do you ever take your real life learnings from your consulting work and then basically just turn it into a new course, like course for experts, course for... It feels like all of those things are so real that you're like, "Okay, this actually... All the struggle that I went through is a new course."
Ant: No, I don't. We work a lot with third-party instructors, and the answer is [inaudible 00:15:08] is someone we've worked with a lot. He does to an extent, but the time it takes to build a new course is quite extensive, and some of the issues I've got are relatively niche, but you find enough of these. It's like I'd rather this problem gets solved than me teach people how to work around it. I'd like to see an application deployment framework that works across all the serverless services, instead of SAM for this, and SAM for this bunch of stuff or Serverless Framework for this bunch of stuff, or Amplify for this bunch of stuff, and actually get them all working together. If you deploy apps, the AWS console is going to tell you to use Amplify all the time even though you don't actually have to use it. SAM doesn't really have kind of first class support for Amplify. Serverless Framework's got support via a very, very good plugin, but it's not native support, and then you've got those similar issues with Cognito.
Ant: And so as you build it out, you find you're just cobbling all these old things together, and it's just not a unified experience. And then you've got to find the edges of it all the time, and you just... One thankful thing I do find now is that if I have hit a problem, and I google, I find an answer 99% of the time, whereas 2015, you'd put a question on Stack Overflow and it was Tim Wagner, who was a GM for Serverless, answering the question at the time. That community aspect has grown significantly and there's significant help out there, and there's lots of Discords and Slacks and loads of help. You only just ask a question on Twitter and you'll get answers. That part's definitely improved, but why are we still in that space? Why haven't they improved? Why is the tooling not improved? Also, why is AWS potentially not taking a more holistic view of all of this, and take a view from a developer?
Jeremy: Right. You gave an interview recently with Corey Quinn, and I believe your direct quote was, reading it here, quote, "Serverless sucks," end quote. I think the idea behind that is essentially what you just explained of these complexities, and that you're finding your way around the edges, but at the same time, you get tired of trying to teach people workarounds because those workarounds eventually disappear. And so I'm really curious, because way, way back, five, six years ago, whatever it was at this point, you and James Thomas and Paul Johnson, you started the JeffConf, which eventually grew into the whole ServerlessDays things, and you were, all of you and I think most of you still are very much into serverless, and love the prospect of what serverless is going to be.
Jeremy: And I think very early on, and you and I have talked about this before, where you sort of recognized this paradigm that was probably going to take over how we do cloud computing and how we build applications, modern applications in the future anyways. So back then, five years ago, whatever it was, when you did the first JeffConf, you had a vision I think for serverless, and you saw it as a solution to a lot of problems that we were dealing with, a lot of complexity problem, so forth. So I'm curious what you were thinking back then, whether or not now, serverless has lived up to that vision that... Is on that pedestal, right? And listen, I agree with you, serverless sucks. I mean, there are a lot of things that are problems with serverless right now that make it hard to use, but I'm wondering how you balance those two things. Has serverless let you down, or are we just still so early?
Ant: No. Yeah, I think it's... There's two bits. The first thing is familiarity breeds contempt. Because you kind of live and breathe this stuff every day, you know what you had two years ago and you wanted to improve on. I know what I had five years ago, six years ago, and you want that to improve. So there's frustration that this stuff hasn't improved. But the other hand, I've never had to run a container in production. So that's a whole world of pain I do not know, and I never want to know. I remember way back with Cloud Guru, when we were setting that up, we wanted to set up a blog, and initially we set up a VM with Ghost, and then the whole time I was thinking like, "I got to manage this thing now." Just a simple we wanted to have a blog, Ghost, use Ghost, set up a VM.
Ant: I deployed it, ran it, everything, and we were even looking at using an open source LMS before we went through of building it serverlessly at the time, it was 2015. And it was just the thought then of like, "I do not want to manage this thing." Yeah, it's easy enough to set up and going, but actually the operational overhead and managing thing, we just don't want that. We want our blog post to go viral and we do not want to manage it. So at the time, we went to Medium because Medium store was offering free blogs under your own domain, and that worked for us. I was like, "Cool. That's awesome."
Ant: That was probably the first step of us going, "We don't want to manage anything." And I'm still very much in that. I don't want to manage anything, I do not want to run a container workload, I do not want to look after container, I do not want to look after a VM. Those will always be the last resort. So in that respect, what you can do with serverless is significantly more now than you can in 2015, 2016. There's so many more services out there, there's way more choice, but the trade off for all this choice and improvement you've seen in the operational capability has been, I'd actually say, probably a degradation of the developer experience.
Ant: Before, you could only do a very limited number of things, now you can do a huge number of things, and you can do the same thing 10 different ways. I saw lots of debates today about step functions because now you can integrate it with synchronous step functions, you can integrate with every single AWS service, and I saw I think it was Eric Johnson saying, "Hey. Actually, you should do step functions instead of lambda, synchronous step functions instead of lambda, invoked by API gateway," and then someone else saying, "No, you shouldn't. How do you test this? Yada, yada, yada."
Ant: It's a new way to do things. Is it the right way? I don't know. It's just going to add complexity. Everything is here's another decision to be made, here's another trade off that needs to be weighed up, and service should be something that you can get going really quickly and iterate on, and now, you're spending six months planning or four months planning what you want to do, and testing all your assumptions and making all these trade offs. Because do you want to start with step functions for your asynchronous workflows, and then realize, "Oh, hang on. I've hit a problem with it," and then rollback?
Ant: So that's just on the decision-making, and then it's like, "Okay, tooling. What do we do with the tooling? How do we deploy this thing? How do we test this thing? What does that development lifecycle look like?" And that side's gotten significantly worse. So the capabilities are improved, how developers interact with that capability is gone, and I think a lot of that's because the developer experience is a second class citizen with a lot of these platforms who are deploying it. It's not just AWS. I've had issues with Azure and Google as well. I think Azure released a new service, and the SDK didn't have documentation. I've seen Google released a Firestore with an SDK there was 60 megabytes, and things like that, and it's just with no proper documentation.
Ant: AWS are the biggest fish in the pond, but everyone kind of has that problem. CloudFlare less so, because CloudFlare are very limited in what they do. I like what CloudFlare is doing. It's a very, very limited offering, it would be a replacement for some things you can do, but it's very limited, which is why I like it, and I would love to see a simplification of serverless going, "Hey, if you want a relational database, this is it. If you want a NoSQL database systems, this is it. if you want to run synchronous workloads, this is where you do it. If you want asynchronous workloads, this is where you do it, and one option. And here's your development workflow, and one set of tooling to make that work," but I think we're way off that.
Jeremy: You're living in a dream world, Ant. No, I think you mentioned though, with AWS, this idea of you should be using step functions for this or there's a different way to do that, and I think that's one of those things where we go down this road of a paradigm shift. Now first of all, there's the first paradigm shift. If I'm a client server model, backend developer, I'm very familiar with making an HTTP call, having something like Ghost on a VM or whatever, respond to me and send me back data. I'm using I'm usually monolithic, even having the database on the same server. There's a big jump in the mental model when you go from something like that to even event-driven architectures, whether that's using containers or serverless, or whatever.
Jeremy: So you've got that big leap there, and of course, breaking things up into single purpose functions and all kinds of stuff like that, all kinds of complexity there. But AWS I think is... I don't want to say they're unique in this, but they certainly do it to a higher extent, is the next paradigm shift, that is to say, when you connect to services, let's say the Retry there, well guess what? You don't have to do that Retry in your code, we'll handle that. And it started with, "We'll handle it in our SDK." Great. Then all of a sudden, it's, "We'll actually handle it in the service connection there. So we'll do things like try it two times. If it doesn't make two times, we'll send it to a dead letter queue for you."
Jeremy: Now, this is all stuff that is now configuration. Eric Johnson calls this configuration over code, which I totally agree with. I think this is a really interesting move because it takes complexity out of the hands of the code you write, it makes your code less of a liability because you're not worrying about trying to cover all those failover cases, and of course, there's a million other little things that it can do, including step functions completely eliminating lambda functions in some cases now that it can communicate with 200 different services. So I'm curious from your perspective, everything moves so fast, we've got people who don't even know what the cloud is. I mean, we have people who are just barely starting to deploy serverless functions, or I mean... Again, I agree with you, love what CloudFlare is doing, so maybe writing a worker here and there just to change them header. So they're starting to get into this idea of being able to run these things either at the edge or even centrally in a region and doing some sort of manipulation.
Jeremy: So they're getting into this idea of moving into event-driven. They're getting into the idea of using these ephemeral stateless use cases or whatever, but then all of a sudden it's like, "Well, yeah. Now if you want to build a real application, you want to wire things together, it's not just writing code. It's writing code, it's writing configuration to deploy the infrastructure that, by the way, there's five different ways to do WebSockets, there's 19 different ways to deploy containers. There's a bunch of different ways to do these things," but now not only that, not only do you need the code, not only do you need the infrastructure and define the infrastructure, now you're writing configuration that actually is business logic inside the configuration of the infrastructure. And it just gets more and more complex, right? So I don't even know where I'm going with this other than to say, "What's your take on this?" I mean, I think it's a good thing, but at the same time, it's another mental shift, another mindset change that developers... Because they have to be responsible for the whole thing.
Ant: Yeah, it's a massive mindset change, and I think that's a major problem for a lot of devs. Most folks, most early stage developers come in from a frontend perspective, particularly if you're at school in some sort of coding club or you're doing stuff. You'll be taught how to build a web page or mobile app or something like that, and then you might go to university and get into backend development and that kind of thing, but a lot of these folks, whatever they're getting taught, is always a few years behind what is being used out there, what the kind of the latest techniques are. So we're still getting young developers coming through who just want to build client-server, two tier, three tier applications because that's what they know, and they don't understand all the trade offs, why you want to make workloads asynchronous and why you want to go event-driven. So there's still a huge amount of education that needs to happen, and I think a lot of that has to happen at the primary level.
Ant: I don't think these concepts are hard to understand, but when you talk one way, you got to say, "Okay, throw away everything, and now learn the new way, the better way to do it," and you also try and justify why, and it's so different to what they've been doing. Plus all the developer tooling isn't there, and it's not as smooth and experienced anymore. It's not just about a paradigm shift, like a mental shift, but it's a shift in everything, the way you work, what your feedback loops look like, and also managing configuration. Werner put out a blog post about two, three weeks ago, talking about app config and more specifically, talking about configuration management and talking about how AWS do continuous configuration integration, I think they were calling it, or continuous configuration, where you're version controlling your own configuration.
Ant: But that's not just CloudFormation, but it's actually your secrets, your private keys, the names of your databases, whatever your application configuration is actually controlling that. So if we're going to be putting more logic into our configuration, we're going to have to be doing some sort of change control of that and testing that properly because a lot of people don't test their code. How often do you test your CloudFormation code or your Serverless code, or your... I know some folks do, and I know there's tooling out there to do it, but most folks I know don't. At best, they'll run a linter on their configuration. That's about it.
Jeremy: That's not easy to test.
Ant: Yeah, it's not easy to test. And then yeah, as we get more and more managed services, these more black box services, we have to test more and more of that. Our lives are going to get easier with this, but I think there's still going to be this new pain coming forward as that happens, as people try and figure this out. I saw a good comment from Slobodan Stojanović the other day on Twitter and his thoughts on the whole thing was just like, "Just do end-to-end testing. Start with end-to-end testing and work your way backwards because short-term, that's almost all you can do when you've got a black box service, and then as the service provider exposes more APIs, exposes more of the lifecycle, you can test more and go further and further back." I think the entire testing paradigm is going to completely change because of that, you're going to write your end-to-end tests first, and then work backwards, and hope that catches everything, hope that catches configuration errors and logic errors and all that. It's great benefit one side, but a pain on the other side, which is also a barrier to adoption.
Rebecca: So is this like... In my head, I always hear it as the Snakes on a Plane problem, where I believe it's Samuel Jackson, he's like, "There are too many mother-effing snakes on this mother-effing plane," and so is it there's too many variable developer tools in this cloud provider? Is it the developer tools to blame? We keep making these tools to keep solving these problems, and those tools make it easier, but it actually makes it harder. Where's the complexity in that paradigm shift? Because of the challenges, basically, the cloud providers are building by way of not considering the developer experience, let's say, as a whole. Is it the snakes or the plane?
Ant: Yeah, both. Both. I think in some instances where the cloud provider is... Particularly the individual product teams are incentivized to launch products. If your job is to launch products, you're going to launch products, whether they're needed or not. So there's that incentive problem. So there are a bunch of things that are not needed out there. There's also the desire to not sunset products, but... So maybe we should be sunsetting a few products and deprecating products, and things like Simple Workflow seems to have slowly disappeared out of the world for example, and CloudSearch has slowly disappeared out of the world.
Ant: I saw good tweet from Steve Brian, who's a principal solution architect at AWS here in London, and he was saying the year he joined AWS, CloudSearch got launched and Simple Workflow Service got launch, and something else got launched, and I'm thinking, "Oh, wow. Those are three services I haven't heard of anyone using for years," and maybe we need to be using more and more of that with some of the services, and say, "Hey, let's look at simplifying choices and simplifying the tooling around that." If you have less choice, there's less tools to build, you make it a simpler decision.
Ant: I love the idea of EventBridge because it seems to bring in the best of many worlds. It's almost as if they said, "Hey, what are all of our users doing? Let's build a product that meets all these use cases," but they haven't necessary removed some of the other services they're doing in the use cases, like maybe push SMS back to being purely for notifications instead of this pub/sub thing, and limit some of the tuning there, and slowly start to kind of force users into a subset of tuning, and then just keep some that other stuff deprecated.
Ant: And then like I said, the other problem is, which I think is more temporary thing, the tooling problem will solve itself eventually, but it's a pain that we have to take now unfortunately, and I keep banging on about it because I wanted solve sooner rather than later, and if providers prioritize develop experience as part of the launch, I think it would be solved a lot quicker, instead of the developer experience is something we do after launch, which is endemic with all the major providers. The small providers are a lot better with developer experience.
Jeremy: I think you brought up a really good point around incentives, and what the incentive is for teams at AWS to build new services, and I don't think it's just AWS, but one thing that was really interesting, I just read this post today and I'm now forgetting exactly what they called it, but it's from Google Cloud and it basically is an app, a nebulous app, I think is what they call it, where they can run... I think it's a reference to cloud. But basically, they can run the same application on Google App Engine, on Google Cloud Functions, and on Google Cloud Run. So the same exact application, no changes to the actual application code itself, can run on these three different services. I'm not sure I know of three different AWS services that you can throw the same code at, and it would run the same.
Jeremy: I may be wrong about that, but I just thought that was interesting how they actually positioned that as a like, "We're building this thing for different use cases maybe, but we're also trying to simplify the fact that you don't have to write code for three different implementations of it and that you maybe can move or maybe graduate from different services as your needs change." But again, back to the incentive thing, I don't think teams at AWS... and this is nothing against AWS teams. I think everybody there is doing amazing work, but they're just not coordinating with one another, right? You get the RDS Proxy problem, where two separate teams were building the same exact service because they weren't talking to one another and they're trying to solve it from multiple teams. Rebecca's is pointing at herself. She apparently was part of that problem.
Jeremy: But that's the thing where I have a hard time reconciling this idea of saying, "We want to build, we want to innovate, we want to launch new products that solve new problems," and as you said, sometimes maybe they're solving problems that don't exist. But rather than looking at it as an entire ecosystem, and saying, "What are we actually trying to solve all the way? Not the micro problems, but the macro problem we're trying to solve?" So I'm just curious, expand on the incentives thing. Is that just something where a company the size of AWS can't look at the big picture anymore?
Ant: AWS isn't one company. They haven't really been for a little while now. When they were small and it was Jeff and Andy and Werner in a room... I mean, it was always a bit more than that, but they were one company, but they're kind of be split into separate businesses, and as they've grown, those businesses have gotten more autonomy, more kind of decision-making power, more budget. I always kind of joke that the container team was just a lot more than just containers. I tried to recreate AWS on top of containers. They released Proton, their own CI/CD tool, and you've got a whole developer tools team doing CI/CD tooling. I think that's endemic. That is by design, that is the Amazon organizational structure, huge amounts of autonomy, limited coordination because if you're not spending time coordinating, you're building, but the problem is now you're starting to get overlap, but there's a tax to that and they haven't had to pay that tax before, and now they're starting to pay it. I see lots of people saying, "You're hitting the limits of the two-pizza team."
Ant: If I were AWS, I would actually double down on that in the respect that I'd go, "Okay, let's break these teams out into completely separate businesses." They're doing a lot of vertical solutions now, so in healthcare and finance and look out for industrial. Make those separate businesses. Have Amazon medical sciences or something like that, that just does healthcare tech solutions, and have those completely separate business. No one wants to see... I never want to see a healthcare solution or blueprint in my AWS docs because it's never relevant to me. That's a very verticalized solution. Take all of that out, and then launch a proper brand new platform that's for one event-driven application that is its own thing, and that says, "Okay, here's your one database, one relational database, your one NoSQL database, your one way of sending messages around, your stateful and stateless compute options," and really, really limit it, and have those as separate businesses, and let them all build on top of AWS.
Ant: And I think they tried to do that with Amplify, but Amplify, it's a user experience, it's not an organizational change that really needs to happen because there's huge limits to what you can do in Amplify as well. It's really hard to break out of Amplify. The option is either start coordinating more, which is going to slow them down, or actually break those businesses out, and I think there's a point where break those businesses out, avoid the confusion going, "Okay. Here's this cloud-native event-driven modern developer platform, which there's no VMs, none of the legacy stuff. There's only very limited choices, but it allows you to go incredibly fast in a safe, secure way, and here's your legacy platform. And if they communicate, it's not within an account. They communicate over a shared integration identity layer, and that's it." That's maybe a better way to do things, but what they've got right now, they're hitting the limits. They have to rethink like, "What's the next evolution of Amazon architecture?" It's Conway's law is biting them really badly right now, and they've got to figure out how to get around that and what's the next step.
Jeremy: Yeah, and we actually had Ali Spittel on talking about Amplify, and what I find interesting about Amplify is I think it's a great product, I think it's a good on-ramp, and it's gotten more and more complex, I guess, is the word of the day, but what I find interesting about that is they had to build tooling on top of existing tooling to make it so that you could actually do this, right, and so while there's a cool frontend components to which I still think Amplify is more of a frontend framework than it is sort of I guess a full-stack framework. It does all the backend stuff for you as well, but again, there's another framework or another set of tools on top of those tools to make this tool a more friendly user experience.
Ant: Yeah, definitely. I think it is, and that's also why I've used Amplify for Cognito because I don't use Cognito natively, I use Amplify libraries because they make it a lot easier, but it's pain, and I think now's a good time to make those changes. We've got a new CEO in place, a bunch of old guard have left for various reasons, and now's the perfect opportunity. The new CEO is going to have a bit of time to make changes, and hopefully Andy Jassy gives him the autonomy to say, "Okay, take this baby of mine, and I trust you to make all the changes and bring the discipline and make it what it needs to be." I think now's a perfect time to do that, and I think if they don't, they are going to... What's happened with CloudFlare recently is just going to continue with CloudFlare announcing the new object-based storage, which overlaps with S3. S3 does a ton of things it doesn't do, but it overlaps, and that's just going to keep happening.
Ant: Other folks are going to go, "Hey, we can challenge AWS in these niches." And if they don't address it now, in three years' time, it's going to be a massive problem. Now it's a small problem. Now it's a problem of the early adopters and the fanboys and the folks who tend to go to the AWS console and check out a service two minutes after it's been announced. Now, it's a problem for kind of the community that we're kind of in, but to the broader community, two, three years it's going to be a massive problem, and they have to start addressing it now. You don't move the ship quickly. They have to start looking now like, "What is the future of AWS and what is the future of Amazon?"
Rebecca: I love the idea you brought up around almost... So right now, it's maybe like AWS is in like a gray... Sometimes I still say we, even though I don't work there. AWS is maybe in this gray space-
Jeremy: You can't get away. You will always be associated with them.
Rebecca: Yeah, a lot of things that I'm proud of about it, so I'm okay with it. There's this idea of, right, where it's maybe in this gray space, where it's all AWS, it's all one big team, but really, it's a bunch of organizations and two-pizza teams within them, and we all know they have that decentralized structure, but basically, it sounds like what you're saying, Ant, is like, "Hey, make those actual fully whole businesses, so when these businesses want to work together, they basically have to work like partners would work together, and you have to reach out across the aisle, you have to have better communication if you're going to partner with this team. You have to state your case and people have to come together assuming that neither party has context on the situation, and they both bring their context," versus I think there's often...
Rebecca: I'll speak for the whole organization when it's not fair to do in very broad terms and my niche experience, where it's like, "Oh, we probably have a similar context because we're both trying to build similar things to fix very different, but similar problems for a similar audience," which is all this broad thing. And to tie that back to the RDS Proxy experience, I was on the serverless.org, on the Serverless team. We saw that there was this issue with databases. Turns out the database organization, which is separate than the serverless organization also looked at RDS and was like, "Okay, there's a thing that we can do here that could solve a lot of things, especially for people trying to build in event-driven patterns."
Rebecca: And so both our team's not knowing it for a while. I won't say how long, but for a while, we're building essentially the same answer, and I'm still not sure even if we took that idea and turned those organizations into totally separate companies, let's say, or totally separate pieces, that that would still be solved. Not to say you have to have all the answers here, but I'm wondering how... What's going to happen is there still so many solutions that would serve two separate companies trying to solve the same type of thing, and I'm curious if there's a way that you've seen from the outside, being able to observe it for so long. It's like how do we siphon off like, "Yo, this is event-driven architecture, and anyone who's going to do anything around this is going to serve all of that, and then anyone who's going to do anything around databases is going to have to come to them first because they are the specialists in event-driven architecture."
Ant: Yeah, I don't think you even do that. Let them compete as companies. If you stop competition, folks are going to get lazy, right? You see it so often where I see consultants [inaudible 00:44:53] to where folks just go all in on AWS, not because AWS has the best service, it's just because they have the best portfolio, and some of the services in that are pretty awful and some are great, and on the balance, it's got the best portfolio. If you separated a lot of those into completely separate businesses, where they're not forced to work together, where they're all focused on meeting a customer need, so Cognito is a really good example. Awful user experience. They have tons of users because it's in AWS and it's cheap. It's the easy answer if you're using all of AWS.
Ant: If that was a separate company, that would be a significantly... I guarantee you, their user experience would be significantly improved, and they themselves would work to fix a lot of those problems. But today, the Amplify team have built a workaround that makes it workable and it's good enough, and they're doing okay because they have millions of users with them, probably hundreds of millions of users, maybe billions of users on it, but they haven't had to fix core problems with it to do that.
Ant: So separating a company's out. If maybe the Amplify team said, "Hey, we're going to build our own auth service." That'd be great because it'll give the Cognito team a kick up the butt, but if this is all one company, it then becomes really confusing, whereas Amplify is a separate business, completely separate business. So Amazon Web developer experience, or whatever it is, Amazon mobile developer experience business, where they are not compelled to use anything else and they can go out and evaluate Cognito, Magic Link, Auth0, or whatever else all build their own, and they decided to build their own, that's because they think that's the best option. And if Cognito loses that business, they should lose that business.
Ant: A good example is Amazon retail. Amazon retail use Fastly for their CDN, which we all found out when Fastly an outage, because Amazon retail was not compelled to use CloudFront, and same thing. It's consistent. When you have these large groups and organizations are compelled to use other bits within that organization. If you're winning business just for existing, you're not a competitive option, and Cognito's the really good example of that, it just wins business because it exists, not because of anything else. And there's a bunch of other services in AWS.
Ant: So yeah, you're going to get some duplication in that, but it will get some real incentive for those separate businesses to actually perform and get rid of the gray area where we're separate businesses, but we're all under AWS with a unified sales team, now it's like, "Hey, change the brand slightly, give them full autonomy, their own budget, they live and die by their own decision, and they don't want business because they exist within a portfolio," and let the third-parties compete. Let Auth0 make the marketplace more prominent and compare native offerings right up against marketplace offerings, like amazon.com does. Half of Amazon retail revenue is from marketplace. AWS should do the same thing. Let marketplace come in and compete with the latest services because long-term, that's the only way Amazon grows and actually provides world class services. Right now they've got a bunch of subpar services that exists because it's part of a strong portfolio.
Jeremy: Right, and that's actually really interesting that you mentioned the competition aspect of things because I always look at AWS, and you also mentioned breaking off a biomedical division, biomedical services division, or whatever it was you said, for AWS, and I think you make a really good point. I think AWS has a lot of really strong services, like bedrock services, Lambda, I mean, it's great. DynamoDB, amazing service, right? EventBridge, awesome service, but then you're right. I think they build a lot of subpar integrations or services that sort of fill the need because it's just good enough, but then they maybe spend too much time trying to force that solution on you, whereas if AWS focused more on building primitive services, like a GraphQL, AppSync, great service, right? Fine.
Jeremy: Lambda as a serverless compute or fast model, EventBridge as a corporate event boss or whatever, those make a lot of sense. But then rather than you trying to stuff everything together, have different organizations, like an Amplify team, that says, "Hey, we're going to build a set of tooling and a set of controls that connect the best services together to give you some sort of use case outcome." And rather than AWS sort of forcing people to try to integrate with all these different things to keep it within the ecosystem, they would probably be better off launching these separate segments that are like AWS for web developers, AWS for data engineers, whatever, and then having a way to organize those services around that because you're right, then you could plug in Fastly if you didn't think CloudFront was what you needed, or you could plug in CloudFlare Workers instead of CloudFront Functions, but instead they build CloudFront Functions because they're like, "We have to compete. We have to give users a way to do this, even if it's not the best service and certainly isn't best in class."
Ant: Yeah, definitely. And these changes take time. Amazon is famous for making decisions early. I saw a good interview with Jeff Bezos, he says his exec team at the time, all they think about is the future. They do not think about the now, and I think Andy Jassy's team needs to be doing that now, and I think Adam Selipsky's team needs to be doing that as well. They've got to think, "What's the future?" And they got to make those decisions now and spend the next two, three years kind of rolling it out because if it's a matter of maintaining the status quo, they're going to hit limits. You could see in three years that that growth that they've had down in the very low 10%, 11%, not the 30, 40, 50% they've had now, which for them, would not a disaster because 10% growth on a 550 billion a year business is still pretty good, but compared to what they've had, it's like, "Okay, this is the slowing down of their business," and that's when you start to see the competitors slowly catch up.
Ant: And by that, I'm thinking more than CloudFlares of the world, the more agile organizations. I think Microsoft have a bunch of other problems, but Google's also. Actually, I've been very surprised with Google's performance over the last few years. I expected them to fade away, and they haven't. So the more time they give their competitors to sort themselves out, which is exactly what they're doing now. If they maintain the status quo, they're just giving competitors time to fix themselves and become more competitive, the more the market they're going to lose. I enjoy what AWS do, I like the platform. I just think they are doing themselves a disservice at this point in time.
Rebecca: So I have two spirits I want to dance with, one in the spirit of life cycle. I want to bring this back to the beginning of the conversation, and two, in the spirit of Senzo training and the pivot that you had to make from in-person workshops or in-person classes to multi-four week workshops. I want to talk a little bit about training and about really to base that in transferable skills. Something that really strikes me about you, Ant, is that you today focus on all things serverless and you say cloud and GraphQL and Node.js, but then you also talk about even in your bios are around how many things you've done in tech. So your first job in tech was doing Y2K upgrades of workstations running OS/2 Warp in 1999, which I think is amazing, and then you wrote VB6 for a living, and then you were a MySQL database admin for a few years. So you've done all these things.
Rebecca: And something that I'm so passionate about, that I have been for a long time, but especially my new role is this idea of transferable skills and this idea of how do more people get into tech, and a lot of ways people can do that without necessarily a super deep technical programming background. There's all these things to do around that, other things that support programming, that support services, like what AWS and Google and Azure have, but what you did there, it's like... Running OS/2 Warp in 1999 and now focusing on all things serverless and cloud, there are a lot of skills that you grew in one place and then could apply them to another.
Rebecca: And I'm wondering for people who are like, "Okay, I want to make that leap or I want to make that jump, or perhaps I want to start focusing on serverless and cloud, but I've been in companies who haven't done that, or haven't had the opportunity to challenge myself in that way, or whatever it is," do you have any ideas around what transferable skills you've taken with you that have really allowed you to move into the serverless and cloud and GraphQL space?
Ant: I think there's a couple things. There's the old cliche of just learning how to learn is probably the number one skill. I've been able to jump and do multiple different things in tech because I've always been open to it. I mean, in example, in the '90s, a good friend of my parents lost his job, he was working for a large multinational in South Africa and they moved to SAP. This is around '99, 2000, and my parents' friend lost the job because he was a old mainframe dev and this particular company was shutting down their mainframe, and he basically had the same job for 30, 40 years., and basically, instant early retirement kind of situation. And one of my friends was actually on the project going in to implement SAP. I told myself, "I never want to be in that situation where I've done the same thing for 30, 40 years, and suddenly I get replaced." One of the advantages we've got knows is with the birth of the internet, it's so much easier to learn. Information's so much more accessible, and the pace of innovation has increased.
Ant: It also means a lot of technologies we're using today are not very old, and actually, there's new stuff being invented all the time. So actually to become an expert on a piece of tech, now you don't need to have 20 years experience in it. You could be writing React for five years, and you'll be a world-acknowledged expert. There are folks who are acknowledged experts in the world with two to three years experience writing React. That's a really good example. And they're totally valid because it's a relatively new technology, you don't need to have 20 years experience in tech to be an expert in that niche. So you can learn something really quickly, you can become a service expert really, really quickly.
Ant: I've seen folks who adopted Serverless last year and they've been doing nothing but Serverless over these pandemic times, and now they're creating courses, launching products, and they're really going for it. So part of that is being open and being able to learn, and also don't think just because you've got six months' experience, or a years' experience, or not a lot of experience in area that your experience is not valid. All these new areas of tech, you don't need a lot of experience in there to be good at it. If you're doing it every day, you're becoming a craftsman in it. You can get there in six months to a year, two years, you can be an expert in that area.
Ant: So I think don't let yourself get in front of yourself. Don't stop yourself just because something's new. Actually, if something's new, that's when you should be jumping on it because, hey, there's no one else doing this thing, or very few other people doing this thing. I think that's one bit, so learn how to learn. Don't be afraid of new stuff. You can become an expert really quickly, you could be speaking at conferences really, really quickly. Particularly the newer the tech. There's less knowledge about that and any knowledge you've gained through learning, reading blogs, doing the stuff, becoming a practitioner, you can then share with folks. Yeah, learn how to learn. Don't be afraid, and also share. The whole learning and public thing. What you've learned, share with others, and that's a good way to advance your career. And definitely don't let yourself get pigeonholed. Don't be like my parents' friend, who basically was almost forced into early retirement because he had a bunch of skills that were no longer relevant, and in his late 50s, mentally couldn't bring himself to re-skill. Don't do that. Don't mentally shut yourself down into one particular area.
Jeremy: And I think like they say, there's no compression algorithm for experience. But that's the thing, everything that I've done in serverless, I mean, I've been doing this stuff since 1997, not serverless, but all this other stuff, and essentially, you start seeing these patterns that emerge that are, "Oh, I remember this. I did this before, or I did something very similar to this." And again, the flavor of the week, there are a lot of flavor of the week type things. I would say serverless is not a flavor of the week, that is going to be a dominant paradigm when it comes to cloud computing. I think things like React and Vue and SPAs, I mean, we're already seeing a shift now to server-side rendering and all the problems that come with some of the heavy... You mentioned the 16 megabyte or whatever it was for Firestore.
Jeremy: There's a lot of reasons why things fall out of favor, but I think these new technologies need the experience of people who have been doing this stuff in the past, and learning a new framework, or learning a new programming language or a new service or whatever, taking your past experience and applying that to this new thing is incredibly powerful, and I think hugely important for the people who are building it to give them that feedback because I've seen a lot of things being built by people who are very young, who don't have that experience, and then they just run into the same problems that we all experienced 15, 20 years ago. So also an interesting sort of take on that.
Ant: Yeah, definitely. So in terms of switching, your experience will always be valid, particularly is your development experience. I've done everything from dev to sysadmin work, to the works. It was really interesting with the Facebook outage recently and all the BGP issues, I'm going, "Hey, I know that stuff," because I used to work for network companies with level three communications, so I have a pretty good handle on network. I haven't had to deal with it for five, six years. Every now and again, it comes in useful where I have to deal with the VPC, but I try to avoid them. But yeah, your past experience is always going to be valid. I think a lot of your experience, your past experience will be things like problem-solving, just patience, just dealing with learning how to learn, and soft skills are definitely the more transferable ones, and then your architectures and patterns always transferable. Serverless hasn't introduced any new architecture or patterns, it's just made them easier. Event-driven architecture design existed before serverless. Microservices existed before serverless. So all this stuff is transferable.
Jeremy: Yeah, and you got it so much easier today too. If you haven't cut yourself trying to work inside a physical server, then you haven't lived, and now these kids have it's so easy today. It's crazy.
Rebecca: Kids these days.
Ant: I've got some war stories. I've never bled inside a data center, but I've done other stupid things.
Rebecca: I've never bled inside a data center, that is honestly the first time I think that has ever been said on this show in the history of all 120 episodes, or whatever it's been.
Jeremy: Well, anyways. All right, we're pretty much out of time, Ant, but anything that you want to kind of talk about in terms of training or just like... What is that now? I mean, you're big into training now with Senzo and the stuff you're doing there, I know you're, again, trying to do some pivots and trying to figure out the best way to move some things forward, but what advice I guess would you have beyond the transferable skill thing in just terms of people who want to get started with serverless? What's the best training out there? I mean, are certifications worth it? I mean, where do people want to go?
Ant: I think, honestly, the best way to learn... So one of the serverless barriers is understanding your platform first. The best way to learn is always by doing. So at one level, I say go out and do one of these relatively cheap cloud practitioner-type courses. They'll give you a base level understanding of the cloud, and they're pretty easy. Do not get daunted or intimidated by it. They are pretty easy. But then when you want to take the next step into serverless, do it one step at a time and build something, build stuff. Read tutorials, the early stuff. You will learn more by doing. Most of my learning was by doing. I've never done a serverless course in my life. So learn by doing. Find a project to build, whatever the next thing you want to build, build it serverlessly, and also try and focus as well to a certain extent to focus on what you've learned, and what worked well what, didn't work well.
Ant: And then the other thing is interact with the community. There's big communities out there. ServerlessDays events, which will be start kicking off a lot more next year. GitHub Issues as a weird place to ask questions, but some of the serverless framework folk on the GitHub Issue is actually quite helpful. And get Stack Overflow, join the communities and Slacks and the Discords, get involved. Actively put yourself out there. Don't sit behind a computer screen and get frustrated if stuff doesn't work. Learn by doing. Definitely learn by doing is for me the best way to do this.
Rebecca: I can only 110% agree with start interacting with the community. I think in terms of digital communities, non-physical communities, being in the serverless community or around it and watching how people interacted with it is a truly special experience, even as someone who doesn't come from a background of programming, right? And so there's such a welcoming nature there, that all I can say is double down what Ant said, get interactive with the community. But Ant, that about wraps it up. I want to say keep learning, essentially, is a great way to end, but also I've never bled in a data center, I think is probably the best way to end. So with that, thank you so much for joining us and for sharing your knowledge with all of us and listeners and the community. How can listeners find out more about you? So your Twitter, what's up with Senzo, ServerlessDays, organize your information, we'll put it all in the show notes, but give us a little rundown about how people should get in touch with you, where they should find out more.
Ant: Twitter's probably the easiest way, @iamstan on Twitter. Senzo.io at and homeschool.dev are the two websites, but they're both going to be refactored, relaunched in the coming month or two. We're going to be doing a few things to change things up there. Our focus is going to be enabling experts to teach others, and we're going to be doubling down a bit more on that. Yeah, there'll be a few changes here, but just follow me on twitter, @iamstan.
Jeremy: And check out ServerlessDays.io for any upcoming ServerlessDays conferences. Some of them happening in-person again. Still a lot virtual, but so much learning to be done there as well.
Ant: Yeah.
Jeremy: All right, awesome. Thanks, Ant.
Rebecca: Thank you so much, Ant. It was so good to have you.
Ant: Yeah, it's been great to chat, as always.
