Episode #108: Mulling over Multi-cloud with Corey Quinn
August 30, 2021 • 60 minutes
On this episode, Jeremy and Rebecca chat with Corey Quinn about the unlikely success of cloud agnostic projects, the myth of portability, how accounting feels about metered cloud/serverless billing models, how big tech handles criticism, and much more.
Corey Quinn is the Cloud Economist at The Duckbill Group. Corey’s unique brand of snark combines with a deep understanding of AWS’s offerings, unlocking a level of insight that’s both penetrating and hilarious. He lives in San Francisco with his spouse and daughter.
Transcript
Jeremy: Hi everyone, I'm Jeremy Daly.
Rebecca: And I'm Rebecca Marshburn.
Jeremy: And this is Serverless Chats. Rebecca, how you doing? Welcome back.
Rebecca: I am doing well. It's good to be here. Welcome back to you, Jeremy.
Jeremy: Yes, it's been busy the last few weeks. What have you been up to lately?
Rebecca: Me? Well, I've done a lot of poetry writing and nothing is that easy to rhyme with serverless?
Jeremy: I don't know. Actually, I found some things to rhyme with serverless, but anyways.
Rebecca: Me too, but what can you do.
Jeremy: This is a monumental guest. This is a big get for us-
Rebecca: Renowned, illustrious.
Jeremy: ... because he is renowned. And so our guest today, he's the chief cloud economist at the Duckbill Group. Author of the Last Week in AWS newsletter, host of Screaming in the Cloud and Morning Brief Podcast. He's a regular cloud speaker, a writer. He instigates on Twitter, which is always fun. And of course, he's a constant thorn in Amazon Web Services' side. So I like to call him DJ Snarky Snark, I don't know if that's a good name for him or not, but Corey Quinn is with us today. Hey Corey, thanks for joining us.
Corey: Thank you, Jeremy, Rebecca, it's always great to have the opportunity to both speak with you folks and indulge my ongoing love affair with the sound of my own voice.
Jeremy: It is great to have you here. I've wanted to have you on for a long time.
Corey: It really is, isn't it?
Jeremy: It really is. And I know, Rebecca, you and I were talking earlier and you have a funny story the first time you met Corey, right?
Rebecca: Yeah. I second the motion that it's great to have you here, for us to have you or whatever it is that you said. I don't know if it's funny, I was super excited when you were coming on and I was remembering back to when I first met you in person. You're this classic internet instigator and you're larger than life type of presence even in regular life. And I think it was 2016 at the San Francisco summit for AWS. And I pulled you in because I wanted to interview leaders in the cloud space and we were doing quick things on social media. And I misstated that you were a hero because I'm like, "This is a larger than life person. Of course, this person's a hero."
Rebecca: I was like, "Hey, how does it feel to be an AWS hero?" And you said, "Well, I'm not." And I was like, "Oh." And you were like, "I am too damn crass." Maybe not verbatim, but essentially that.
Corey: That was community villain, these days, I would also accept anti-hero. They never invited me to the program, but it's probably not my speed at this point, just because it's... Authenticity is important, as much fun as it would be to sit there and get a bunch of free credits and return for remarkably little work, there are challenges with that. I'm trying to, in many respects, build a platform where my story has been and remains that you can buy my attention but never my opinion, and the value beyond this is great.
Corey: I don't get a whole lot of free gimmes from AWS and I don't need or want those things, what I want is them to fix some of the problems bedeviling customers, and that's really been my position ever since I started doing this. It also would have been 2017 to 2018-
Rebecca: 2017.
Corey: ... because I started the newsletter in March of 2017. Before that it's, "So what's it like to be a hero in 2016?" My question would be, "What's that?" I've never dragged a child out of a burning building. Lately I've been dragging the child out from in front of the television watching Paw Patrol, but that's not here nor there.
Rebecca: Yeah. Thank you for that fact check. You are right, I think my conception of time is a little off these days, to be honest.
Corey: It's still 2020, right?
Rebecca: Yeah.
Jeremy: Exactly. Well, let's go back to 2019. Corey, the first time you and I met, I believe it was in the basement of the Sands Convention Center before re:Invent on a Sunday morning. And I remember so even though maybe you don't get the perks from AWS, everyone that was walking by from AWS certainly wanted to come and have a conversation with you, you seemed to know quite a few people there.
Corey: I've got to be direct on this. I really am a little concerned about what the next in-person re:Invent is going to look like because lockdown hit last year. My proxy for how large my audience is generally Twitter followers, not that validates or means anything, but that is an easy number to grasp and reason about over time. And at this point now, because I basically doubled down on shit posting during the pandemic, my audience has tripled in size since then, and it is AWS centric. So I'm wondering like, are people going to be following me into the bathroom when I'm at re:Invent?
Corey: Because there were long stretches and previous years at those events where I would be sitting around and looking for people to talk to, no one knew who I was, and that was okay. At some level, it's one of those doors once you walk through, you can't really walk back, but we'll see. It's the best kind of problem. Well, second best kind of problem, the best kind is someone else's. The second best kind of problem is one you have to deal with now.
Jeremy: Well, fame comes with a lot of burdens.
Corey: I would agree, but it's not all it's cracked up to be, I think it was Bill Murray who said... It wasn't bill, I forgot who it was, but the thing was, "Oh, you want to be rich and famous. Great. Start off with rich and see if that gets the things you think you want."
Jeremy: That's a good point. That's a good point.y, fame is interesting. So let's get into a conversation here because besides just the snarky posts on Twitter and occasionally rubbing some people the wrong way, maybe, which I think is always fun to watch, you actually run a business. And you've been you've been doing this cloud economist thing for quite some time. And you write a lot, you speak a lot, you talk to a lot of people. And I know for me, speaking to people on the podcast is the best way for me to learn something new and dive deeper into different things. I know you'll know a lot about this, and I know this is probably one of your favorite topics, which is multi-cloud.
Jeremy: And you wrote a post about this, I think it was maybe middle of last year where you talked about what multi-cloud was and what the problems with it, and summed it up as multi-cloud doesn't actually exist. And I always look at it in three different ways. There's this cloud agnostic aspect of it, where like, I can just move my workloads around all these different clouds, which I think is probably not the right thing to do and probably not even possible. Then you have this like cloud diversification way of doing it, where I'll build some apps here, some apps there.
Jeremy: But then you have what I think is a better definition of multi-cloud, which is using best of breed services when it makes sense. So I'm just curious, maybe you can tell the audience here how you think about multi-cloud. And maybe why trying to pursue that as fool's errand.
Corey: Let me begin by with a disclaimer that this is for the general case. When I talk to individuals who say, "Oh, I do the thing you said not to do at our company. What do you think about that?" My answer is, "Well, I think you're probably on the right path," because I'm talking in the general case, you have more context into your constraints than I do, you're almost certainly right. What I'm against is this idea of building something from day one to run in every cloud provider just because it seems like a best practice. No, it's dumb, don't do it because it keeps you from being able to leverage any differentiated offerings across the board.
Corey: I wrote a, well, let's let's charitably call it an application that I use to build out my weekly newsletter that goes out to the Last Week in AWS subscribers. Last count it was over 30 Lambdas function about four or five APIs gateway, and that was it. It just ties together a whole bunch of different things. The code that it runs and stuff lives on GitHub, so I guess that uses Azure and I use Retool, which also runs on Azure, but I don't have to care about that or know that. And a lot of the things that I write long form in there get posted to my blog as well, which there's WP engine, but under the hood that's right, LastWeekinAWS.com lives on GCP.
Corey: So I don't care about those things, and that makes sense, what I'm not building is that entire application to run simultaneously across different providers. Although I've done some digging, I could port it to GCP or Oracle Cloud in a week. Azure would take about a month, and IBM Cloud basically when hell freezes over, because we're talking about clouds here.
Jeremy: That's a good point.
Corey: But I have the strategic exodus option, which means I don't have to actually do anything. When I wind up building something that leverages AWS services into this, I don't stop and think, "Oh, that's going to be hard to replace if I have to move," because why would I realistically for my business needs have to move? I don't see that there's a valid answer to that.
Jeremy: And that's a thing that comes with lock-in anyways, is that the second you choose any service, you're going to lock yourself in in some way. Another thing you mentioned in there that I thought was super important, and this is something Rebecca, maybe you want to dig into a little bit more, is just the people aspect of that. This idea where if you announced that you're migrating to a different cloud, how many engineers are like, "Oh yeah, that's great. I'm going to throw away what I've just learned over the last six years and learn something completely new"?
Corey: I should be clear, I'm not a particular partisan here. The reason my business focuses on AWS builds is because that's where the expensive problem is. My goal for this industry is in five or 10 years, if I want to start some nonsense Twitter for pet style startup, I want the question of what cloud provider I use to be a difficult one. I want there to be multiple viable, great answers, not the obvious answer and then a bunch of also rants. And I worry that monocultures are not good for customers.
Rebecca: Yeah. And I wanted to follow up on that a little bit, this idea of... The way to say it would be a potentially controversial thing, but that's not scary to talk to you about potentially controversial things is, let's say no matter what migration you did, let's say you were going to lose a third of your engineers. In the long vision horizonless space of time, is that such a bad thing or is that actually indicating something different where you're like, if I have a third of engineers who don't want to learn something new, ultimately I don't necessarily want those engineers to be building my product. That's a very simplified question or way to say it, but that's something serious about-
Corey: Sure. I would disagree on some level with the idea that the... For example, let's say that I give all of the speaking to microphone pieces up and I go back to being a SRE. Great. I know AWS super well because I've been working with it for 10 years. That is the set that I am known for, and that I have honed. I can do that job rather well. If I have to migrate to a different cloud provider, because that's what my company decides it's going to do, well, I do I want to continue focusing on this highly in demand, highly valuable skill, or do I want to go back to basics and learn a second cloud provider?
Corey: If I decide I want to continue on the AWS path, it's not about not learning something new so much as it is continuing to hone a very specific, very in-demand skill. And I have a very hard time dinging people who make that choice. I don't think that is an unreasonable decision for them to make. It does speak as well to the fact that you are already locked in through your choice of how security is handled in different environments, through how identity is managed. And of course the staff, because not, everyone's going to be on board with a cloud migration.
Corey: Now, from a business perspective, okay, we're going to have some turnover concerns if we do that migration. That is not inherently a problem, it just adds cost. Cloud migrations are already expensive as hell, regardless because you're going to in almost every case have to borrow from a feature velocity in order to do it, and that's expensive. The opportunity cost as well as the actual cost, that adds up to a fairly large number. So the question that needs to be answered is, what is the upside that justifies that level of expense? And very often, there isn't one.
Rebecca: Yeah. And I'm curious, let's say there is an upside in-
Corey: Oh there can be, there absolutely can be.
Rebecca: What is a good way? Is there a good way or the best of the worst ways to do immigration? Is it democratic where you're like, we're going to see how many people would be interested in doing this, or is it just straight up numbers gaming? What are some good ways to implement when you're like, "Okay, we actually need to do this?"
Corey: You don't run those things like a democracy in almost every case. It's a strategic decision that business has to get into. If you don't pull the engineers for things like that, like, well it seems like it'd be a fun afternoon project. It's not. The most common way to do a migration is to attempt to migrate from one cloud to another, get halfway there, give up because it's hard and declare multi-cloud victory. We see this with a lot of hybrid stories as well. That's fine. The reality is what happens instead of what you plan on. The challenge is that we always tell ourselves a great lie and we never get away from it that, "After this next sprint, suddenly we're going to make all the right decisions. We're going to start building things as they should be."
Corey: And that day never comes. Technical debt is load bearing and it's there for a reason. And we love to, as engineers talk in condescending terms like legacy, which is engineering speak for it makes money. And it is like it is for a reason. No one shows up to work in the morning expecting to do a terrible job at work today unless they work in the Facebook ethics department. So people are doing the best they have with what they have. The sign of a junior consultant to someone who walks into an environment looks what's here and says, "Well, this is crap."
Corey: Great. Why is it like that? Was it a lack of understanding? Is there a constraint perhaps you don't see? Did this predate the easy answer that you're about to suggest? What drives this? Why is it the way that it is? And without that context, you just sound like a tool. Don't do that.
Jeremy: Right. Going back to the migration piece of this, there's always potentially that chance where you say, "Okay, we need to make this migration." And you're right, there are reasons why you can do it, maybe there's not-
Corey: I've been dumping on it, let me give you an example of a good one where, all right, I'm a SaaS company, or I sell a product and I build a thing on AWS. And someone comes to me and says, "Hey, I'd like to buy your service and spend a lot of money on it. Does it run on Azure?" And the answer is no, it doesn't. And they say, "Great, but Walmart is one of our customers and they refuse any of their data there. So we can't make this giant and commitment with you unless it runs on Azure."
Corey: And I'm like, "Oh, Azure, of course it runs on Azure or will, by the time that contract gets signed." And then you have a valid reason to move the segment of that workload over. You obviously need to honor your customers and be transparent with them, but even in that story, you're going to be able to run the application on top of Azure, but you're not going to move your marketing site, your billing system, a lot of your internal processes over to multiple clouds, you're going to take the workload that you are delivering to a customer that handles data at a minimum, and then evaluate from there.
Corey: You have to meet customers where they are, and if they're in other cloud providers, well as an ISV, independent software vendor, that's where you're going. And that's not a foolish thing to do.
Jeremy: That's where my question is, going back to what Rebecca was saying about, are there easy ways to do this? And I think from a migration standpoint, a lot of people believe that if we build on Kubernetes, then we can just go ahead and move things over to cloud B from cloud A. But with the proliferation of all these managed services, whether it's DynamoDB or Cosmos DB or Firestore, or any of these other things, is that really a reality? Have you seen anybody actually do this, say like, "Hey, we can just go ahead and move our compute workload over"?
Jeremy: Because it seems to me moving a couple of Lambda functions to a different fast service, for example, is not a ton of work, and transporting an entire Kubernetes cluster over without the data, because it's probably not part of it, would be a much more challenging escapade, I guess.
Corey: It is no longer... Let me reframe that. Workloads do exist, but they are uncommon. An example would be that I am going to bring some amount of data in from either public sources or the data is small enough that egress charges don't matter to my workload. I'm going to do a lot of processing on that data. Maybe it's a bunch of compile thumb jobs or a build farm or something like that and it's not particularly time dependent, and then I'm going to return the data which is relatively small to some centralized repository. In scenarios like that, it is strictly about, where can I run a container that is economically optimized for me?
Corey: That's when you can start doing things like arbitraging between different providers at different times of day and whatnot. And that takes some cost and some complexity and some engineering, but at certain points of scale, it becomes viable. That's not most workloads, to be very clear. There are things that tie you to particular environments. Data gravity is very real and egress costs don't help with any of that. Where your data lives is going to be your primary cloud provider in almost every case.
Corey: And when we see folks who have multiple cloud vendors, regardless of what sense you want to use, IaaS an all of those places, for example, not the, "Well I use G Suite so now I'm a Google Cloud customer," where we see folks doing workload deployments. In most cases, we're seeing 80 to 90% of their spend is on their primary, and then there's a long tail there. And there are valid use cases for that too. I do not believe for example, that AWS is going to suffer a sudden and irrevocable loss of S3 data across multiple regions instantly. It is not how they build things.
Corey: They have a near complete control plan separation or anything like that, etc, etc. But it's easier for me to back up that data to Google Cloud say than it is for me to explain that concept to an auditor every time they come back and have to talk to me about this or my board, when they asked that question. They rehydrate the business level, backup of core data to another cloud provider. That's not foolish at all.
Jeremy: Interesting.
Corey: But make sure you understand why you're doing what you're doing, that's the thing that gets missed.
Jeremy: All right. The other thing that I'd love to talk about, because this is something that I have this conversation with a lot of people and you being a billing expert, I know you make a lot of jokes about the cost of your Lambda functions are really low. But there's other cloud costs. I think people are starting to build more sophisticated cloud applications, even if they're not entirely serverless, they're using a lot of these metered services to do it. So I'm just really curious, that serverless pay per use billing model, is that a nightmare for the procurement and the accounting departments in some of these things, or is it starting to become more of an acceptable practice?
Corey: Cloud it is a nightmare for a lot of those folks too, because there's a giant misunderstanding in many respects that distills down to the cloudier that something is, and by which I mean the closer it gets to serverless versus a data center, the less expensive it becomes, but also the less predictable. There's a baseline cost to service your first customer, a bunch of things that need to exist in your environment. And the additional cost beyond that distills down to a model of unit economics.
Corey: Something we saw during the pandemic in many shops that prided themselves on auto scaling, for example, saw 80% drop off and user traffic, but their bill remained largely intact at a flat line or doing what AWS bills always doing, and so increasing with time, because data doesn't delete itself. Great. Auto scaling is more of an aspirational thing in some cases, and people use the term elasticity in cloud to mean, "Oh, I can scale up." And you need to scale up because if not, you're dropping customer traffic on the ground. Scaling down has never really been a primary area of focus for companies.
Corey: Because at that point, it's just extra money, which I know sounds like it's for steal, but it's not really. Losing customers and disappointing them is way more important than, "We spent a little bit too much on our infrastructure." So people don't emphasize the scale-down part of the story. Now, as we wind up seeing the whole serverless approach, there are benefits to procurement done right. Simon Worldly framed at once is tracing the flow of capital through your organization. For a long time, it was hard to find expensive Lambda bills. There was a blog post somewhat recently on the AWS Management and Governance blog on July 22nd, How The Washington Post's Arc XP uses CloudWatch Metrics Explorer to reduce costs.
Corey: And it talks about how The Washington Post had access into the early beta of CloudWatch Metrics Explorer, and how there was an internal proprietary algorithm that they use based upon a variety of different metrics that they then gather and then figure out how to handle provision concurrency, and how much of that's being used and dynamically address it in real time. And AWS strongly talked about this as if they were indispensable at creating this. Yeah, The Washington Post is a reference client of ours. Who exactly do you think came up with a lot of those principles and ideas during the course of an engagement there?
Corey: And The Washington Post team is great at this. They are phenomenal. They are further ahead down the serverless path in some of their workloads than most companies you've talked to. And what's also interesting about this, and I want to be very clear, this is the sort of thing that people will read and think, "Oh, I need to absolutely implement something like this," and they'll go and do this and spend time on it. And I'll check their AWS bill and talk about these efforts, and they're spending 70 bucks a month on Lambdas function. Maybe that's not the thing you want to be building that advances the state of your business.
Corey: Let's admit, The Washington Post is very clearly a scaled out company. They are doing a lot of neat stuff with serverless and they are clearly doing things that are interesting enough to be at the vanguard of what we're seeing as far as cost optimization goes in the world of serverless. They wouldn't have been featured in the AWS blog if they weren't.
Rebecca: Let's talk a little bit more about that about serverless billing and how the serverless model shifts a lot of operational decisions to developers. And I'm wondering if that leaves them responsible for infrastructure costs. And I'm wondering, is that a good thing because it can add agility is an extra burden that can have terrible consequences. How do you see companies mitigating this, and should they? Should they be mitigating it, that $70 bill that was made at an operational decision from a developer standpoint?
Corey: Context matters. And I think that there's a fallacy inherent in the question that suddenly engineers are responsible for the infrastructure bill. Who is responsible for the bill is a question in many organizations. And invariably, the answer of the responsibility is who owns the P&L for the group, the workload, whatever it happens to be, and they can delegate the work, but not the responsibility. So yelling at engineers when they have a bill mistake is fruitless. We've all had bill mistakes.
Corey: I have screwed things up on the bill this year. If I can do it, I promise. So can you it's it is something that happens. That is the nature of how this works. The thing that I think engineers get wrong in many cases, when it comes to bill work is that they don't have access or insight to strategy in many cases, and that's fine. There's a certain engineering mindset type, and I am clearly, one of them are, I wouldn't have started the business that I did, where I will spend six to eight weeks just knocking 200 bucks a month off of my developer environment.
Corey: Yeah, I've embezzling more than an office supplies during that time span. So maybe that's not the thing that's going to determine the best use of the best use of my time. The thing that gets missed is that we've learnt to do this in many cases in our spare time in school or whatnot, when our time has remarkably little value. To your employer, you are expensive. You are more expensive than the AWS environment you're working on in almost every case. Payroll dwarfs Amazon bills. Rent in the office of corporate office is usually right up there too depending on the specifics in terms it places. Yeah. Full-time remote suddenly has some financial advantages too.
Jeremy: Yeah. And I'm actually curious more about that. I make the assumption that the developer becomes more responsible for the bill. And when I say responsible, and I don't think Rebecca means this either, that they have to own up to it that it's somehow their fault. But the idea of them owning that decision, as opposed to the old operational way of doing it, of like, "Well, we're going to procure 100 servers and we're going to spin these up." And I know there's a lot of scalability and things in the cloud, EC2 instances and auto scaling groups and all that kind of stuff that can really change depending on what the workloads are.
Jeremy: But choosing Kinesis and having to scale up that with multiple streams and things like that, or using Dynamo DB poorly can cost a lot of money. I'm just curious if you've seen this, especially with the number of bills you've been looking at and your group's been looking at, have you seen any of these like mistakes that add up to something that you'd be like, "Oh, this was a really big mistake?" Or are we still talking about small dollars here?
Corey: It depends is always the answer. Because remember as well that every bill aspect of every dollar amount is either wildly insignificant or wildly significant, depending upon constraints. If I'm a student in a dorm room learning how AWS works and I fall into one of their cleverly laid traps, like the managed NAT gateway and my I'm in the free tier, why did I just get charged $70 this month? That is disastrous because that's my food budget. I'm not going to sit here and tell them they're wrong.
Corey: Conversely, if I'm spending, I don't know, $180 million a year on AWS, and I leak my credentials to GitHub, for example, and someone spins up a bunch of things to mine Bitcoin, it's very hard to impact the bill by a meaningful percentage, because that's a lot of Bitcoin. The mistakes get magnified at certain tiers, and then go away at other tiers as well. Let me explain how a bill scenario typically plays out in most companies. The Amazon bill shows up at the end of the month to someone in finance and they se, "Oh, Amazon, the bookstore. How the hell much are people reading? It's way this month than it was last month."
Corey: And they dig into it and there's this thing called S3 on it because that's how the bill breaks things down. It doesn't break things down like by the data science team or whatnot. So they play this game of corporate telephone. And by the time it gets to someone who can impact the bill and many environments, they're not allowed to see the bill, which is just a genius move, complete genius, great job team. They are then asked about this, but what they hear after it's filtered through all of that game of telephone is, "You're spending too much," as opposed to the bill is higher.
Corey: The question becomes, "Is that the new normal? What does that mean for our projections?" And critically, especially at large scale, the way that discounting works in all cloud providers, but AWS, most notably, we do an awful lot of enterprise discount program negotiation with AWS on behalf of our customers, one of our core consulting offerings. We help them figure out how to handle their bill, the long-term contractual obligations, the way the discounting works is it's. How much are you committing to spend over a period of time. And if you commit too much while you owe that money regardless but you're not getting anything for it, that's a miss.
Corey: Conversely, if you wildly overshoot your commitment, well, you could've gotten a better discount, so you left money on the table, what should that commitment level be? And answering that question is one of the things we do, but it's a hard question to answer. And when you have a sudden bill increase like that, it's, "Okay, what does it mean for our commitments? Should we be reaching out to our account manager out of cycle to wind up potentially talking about negotiating an extension or that gets us better discounting, or is this a temporary aberration that's going to go away next month once we make a good business decision deleting that second set of backups or firing Pat."
Corey: Great. That is something that becomes a nuanced conversation. That nuance is very often lost on the engineers who all they hear is, you spent too much on this. And very often that's not the right answer.
Jeremy: Yup. And I've got so many follow-up questions to that, but I'm going to limit it to two. First of all, I'll make a comment. Tags, aren't that the way to break it down that it was the data science team versus the other thing? You don't have to answer that question because I already know the answer. And then the other question I would have is, isn't that just a bad practice not to show teams what their actual cost is, or somehow tie those as a cost of goods sold, like, "Why did we spend more as a data team this month than we did last month? Oh, well, because we ran additional experiments."
Jeremy: And maybe there's projections in there as to what that generates for revenue down the road or whatever it is. But doesn't that just seem... it sounds like you'd think it's backwards, but why would companies not do that? Why would you not pass that information through? And why would now that you can bill and almost a lot of the stuff is paper use, that you can tie it to your actual productive use. You can trace that through the business. Why would you not expose that back to the teams?
Corey: In theory, you're spot on correct. Go ahead and build that, I'll wait. Let me give you an example, talking back to you just said, I'm going to run Kubernetes on an AWS account. Great. It has a whole bunch of different things working inside of it and it winds up having shared services between different teams, and okay, now there's a spike. From everything AWS sees about that, regardless of whether it's EKS, regardless of whether you're running it yourself on EC2 instances, it doesn't matter.
Corey: It looks like one really weirdly behaved application that talks across availability zones sometimes, but other times it doesn't, because it has no sense of zone affinity. And you're seeing weird use patterns start and stop. And what the workloads are inside of that is completely impenetrable to any form of tagging system that currently exists unless you build a metadata system outside of it, which you first. And at that point, how do you allocate that when you have 40 teams that are all interacting with that Kubernetes environment, how do you isolate down into who did what?
Corey: The answer in many cases, and I'm not making this up, is I go diving into VPC flow logs to figure out the network traffic time of day port, what's talking to what, then you start talking to people, "What's likely to be doing this particular behavior pattern at 3:00 in the morning?" And they're, "Oh yeah, that's that job that fires off." "Was there a change recently?" "No, except we wrote the whole thing three weeks ago, does that count?" "Yeah." And then we start having those conversations, but how do you get exposure to that?
Corey: And how do you, more than that, distill that down into something that someone in finance is going to be able to reason about? The answer, at a certain point of scale is you have a team that's devoted to doing this and that team is cross-functional with different memberships, different membership that comprises different functional specialties, because what we've done at The Duckbill Group is not something that you should have to do at most companies, which is take engineers and teach them how to operate at finance.
Corey: Spoiler by the way, it is easier to do that than is teach finance people to work as engineers. They're both hard disciplines to be very clear, but you can at least get to a point of understanding the relevant terminology and its impact faster by going from engineering to finance than if you're coming from a deep finance background, trying to figure out just what the hell of far gate might be.
Jeremy: Sounds a lot like forensic cloud accounting.
Corey: Absolutely. And be careful, you always find a body.
Rebecca: Yeah. And I don't know how there's not already a primetime TV show about this, on NBC it's like NTIS and then forensic data accounting with The Duckbill Group.
Corey: I don't understand how I don't have my own Netflix comedy special yet about cloud computing.
Rebecca: I think it's coming.
Jeremy: Well, yes, I would love to see that as well. They make documentaries about pretty much everything now, so I don't see why there there's not a Corey Quinn hour or something like that. Another thing you mentioned, and I love that example of the managed NAT gateway, because that drives me nuts. It's like, "Hey, secure everything. Oh, but if you want to get outside of it now, you've got to pay $35 a month," or whatever it is as an absolute baseline for some of those things.
Corey: It's a tax on going about your business, and it drives me up a wall. Let me be clear here for folks who have not delved into the joy of this, if you have a private subnet in AWS, it by default cannot speak to anything outside of that subnet. Cool. You can add a managed NAT gateway, which lets it do that. But it adds a data... There's an instance fee of, I think something like what is it? Four or five cents an hour in that range? I have not memorized the entirety of the pricing catalog. And on top of that, it charges-
Jeremy: I was going to quiz you on that, but that's all right.
Corey: At the top of that, there's also a 4.50 per gigabyte data processing fee that is layered on top of any data transfer fees as well, which means that if you have not taken the extra step of setting up an S3 gateway endpoint inside the private subnet, which is free, storing data in S3, goes from free, by having that to incurring a 4.50 per gigabyte data processing fee. That is the same as if you had stored that data for two months in S3. It is ludicrous and it is offensive. And if you were to run your own NAT instance yourself, and that wasn't managed, that data processing could also be absent and all you'd have to pay for is the instance fee, which would not be particularly egregious.
Corey: It is something that is obnoxious, customers hate it. I've never yet found a customer who thought it was fair. It's one of the easy examples I continually go to of trying to make a dollar eroding customer trust because with things like that happen, everyone has learned now that whenever AWS launches a new service, the first thing you look at is not the shiny marketing page or the customer story, you pull up the pricing page, because how is this going to screw me? Is the question that everyone either learns or is about to learn to ask before you delve into anything, because pricing is also marketing on some level it's who is this targeted for?
Corey: Amazon Kendra, they did a great job at talking about that for 20 minutes on stage at re:Invent when it first came out, like, "This is amazing." And I looked into it, "Oh, it starts at 7,500 bucks a month." You realize I can hire someone whose full-time job it is to be my internal repository search person and get me whatever data I asked for and come out ahead? So, not for me, it's for big enterprises. If they'd said that at the beginning, I would have not bothered paying attention.
Rebecca: Yeah. Let's talk it a little bit more about the enigmas of pricing and needing to be able to first go to that pricing page, even be like, "What is this going to be for me?" I know that Jeremy and I, and I think a lot of people talked about this, heard about this, that it is data transfer costs are another enigma to most people. Are they justified? How much does it add to a company's total bill? What do you think about AWS data transfer?
Corey: Their benchmark's at about 10% or so, but we see significant deviations from that. Fun example. Do you know that it costs more for me to move data between an availability zone in one region to an availability zone in that same region than it costs me to move that data from Virginia to Ohio or vice versa? It costs twice as much to do that. And that seems like a really weird thing. Their data transfer pricing, their egress pricing specifically, has not materially moved in ages, and it still feels like 1998 pricing. It starts at nine cents a gigabyte to cents something externally to the internet.
Rebecca: Good year though.
Corey: Oh, sure. The other side of it is that you look at this and it's awful because you're reasoning about this, "Well, data transfer bandwidth has to be super expensive." Yeah, but if I send something into AWS from the outside, it's free with an asterisk next to it because there are exception cases, but it's generally free. Huh, that's a weird misaligned expectation. Then you have companies like Oracle Cloud showing up where they are charging a 10th of what AWS does for egress bandwidth to the internet.
Corey: And I have a laundry list of complaints about Oracle that you can probably guess, but the two things I will say for them without qualification or reservation are that one, their free tier is actually free. And two, their data transfer pricing is amazing. If you want me to add a third one in, it's that their service offering is legitimate. I'd kick the tires on a bunch of them and I like them a lot. To be clear, they are sponsoring my podcast these days. And my rule has always been, you cannot buy my opinion. I think that from that perspective, Oracle Cloud is great.
Corey: There are a bunch of challenges with them, to be clear, I am not a shell, but I will call out what's good. And to be clear, I would still be saying this if they were not sponsoring what I am doing. And I know this because for over a year, I've been saying this in public loudly already. Well, I just want to be very clear on that point right there. I don't shell for anyone.
Jeremy: No. And if we dig in a little bit more, I do want to spend some time talking about you, Corey, and then chat a little bit more about you. And that is one thing that I think comes through, is authenticity, and that's always something where you've never really been afraid to say something good or bad about a company. And with AWS, sometimes you say it in good fun, other times you really hold their feet to the fire, and I personally admire that. But I'm curious in terms of AWS in general, they seem to be very much so publicly. They're like, "Oh yeah, we love Corey." And sometimes maybe they seem a little bit annoyed and so forth.
Jeremy: But I'm curious, you had a New York Times article written about this snark and holding AWS accountable. I'm really curious, it seems like they accepted, but what's your relationship actually with them? Do you find that they handle criticism well, or maybe not just AWS in general, but all companies?
Corey: An example of how this works is I say what I think about these things, and I have been since I started this place and I had no audience worth mentioning, it's begin as you mean to go on. And that was something that I held to across the board. There's also a misunderstanding that there is no TED Amazon who is going to sit there and have these single corporate opinion. Big companies are comprised of lots of people. For those of us who consider a big company, one that has 200 people in it, that's a bit of a challenge to wrap my head around, but it's true.
Corey: Different service teams have different opinions on me, different groups have different opinions, but the by and large, the most common one even now is that they have not heard of me and that I find just egregious, but it's okay. Now, that New York Times profile as well said at one point that AWS has paid Corey for advice. I could not possibly comment on that, but I am citing that is what was included in The New York Times article. Companies do not generally want a bunch of people to do ego stroking, well, not successful ones. They want to know what they're doing wrong so that they can improve it on some level.
Corey: And I have a laundry list of criticisms I can give you about AWS, but I have never for a moment doubted the authenticity of their customer obsession approach. Now, in aspects of Amazon, like the commercial retail business, that expresses itself and it feels to me as being something that's about making customers do things that advantage Amazon in certain ways, whereas in the AWS space, it's about how can we best serve customers. And there is a key distinction there. I looking on all the time I've spent working with AWS, with all of the things I have said, they have never tried to muzzle me in what I say.
Corey: They have tried to correct the record at times when they believe what I am saying is not accurate. If I agree, I will correct the record, if not, I will tell them so. I never once caught someone speaking, they have AWS speaking to me in a lie. People can be wrong, don't misunderstand me, but when they tell me things and I back channel check them, I have never once found it to be inaccurate. That counts for something.
Rebecca: Yeah. And to grow in this a little bit, your wits and your humor, and sometimes your crassness, whatever that might be, it's always really quick. And so, even though I think you can almost feel like what you're saying is off the cuff, I think it's very clear that there's a ton of thought that you put into what you write and what you say. And I'm wondering actually, similar to AWS having leadership principles, or tenants that they build stuff around, do you have a specific set of tenants or principles written down somewhere that you look to when you're crafting your critiques or whether or not you're deciding to critique or defend?
Rebecca: Is there like something pinned up to your corkboard that's like this, "These are my tenants, and these are how I evaluate certain things"?
Corey: Great question. I've taken them a step further and I've tweeted about them, there's seven of them, and at The Duckbill Group.
Rebecca: I got to see this.
Corey: I can read the quick list to you without domination.
Rebecca: Please.
Corey: It's our marketing beliefs and our rules. Rule one is consent is required. Number two is humor is necessary. Three is punch up, not down. Four is that the competition doesn't matter. Rule five is that trust is earned over years and lost in a second. Rule six is, help our customers be awesome. And rule seven, which is more of a marketing thing, it is protect subscriber information. It goes back to trust is easily destroyed. Honestly, we're looking at reworking on number seven to a subset of another, but it's so important, I want to call it out on its own.
Corey: Again, articulating these things as a challenge, but that is how we wind up viewing the world. And there's a reason that I have a pet domain of twitterforpets.com when I need to make fun of some random startup, because making fun of an actual startup is shitty. You're basically crapping all over someone's life work at this point, and that doesn't feel good. By the time you're a multi-trillion dollar company, you can take my slings and arrows, and there is still nuance to that. One of the things I got wrong was holding to that belief absolutely in the early days.
Corey: AWS releases a new service, and the first thing I do is dunk on it. There's a relatively small team that spent the last 18 months of blood, sweat, and tears building that thing. And if the first thing that I do is make fun of it and talk about how it's bad, that doesn't feel good, and that's not who I aspire to be. So I find ways to talk about services most of the time in a way that is making fun of things that aren't going to horribly offend people. For example, no one spent 18 months naming Trainium. And if they did, maybe they should feel bad today.
Corey: It comes down to where do you draw the line? Where do you focus on it? Occasionally, they will release something that is just so terrible, I cannot even think of a nice thing to say, and I will excoriate it publicly. And I try to do that in a very limited degrees, but there are times where, when you're putting this out into the world and attempting to charge customers money for it and it's crap, I'm sorry, whoever it is that has put the amalgamation of things together has failed the team that built it and has failed their customers. Now, I will say this for AWS, services don't get worse over time.
Corey: And a lot of things that I savaged such as EFS, their Elastic File System, when it first came out, has now become a staple of some of my workflows, and I recommend it to people on an ongoing basis. That is valuable. What I think people get wrong, and I see this all the time, and it's why I started the newsletter to be honest with you, it was for my own purposes of once people learn something, they don't go back and validate that it's still true. I talk to people who believe that you still have a 10-tag limit on resources. I talked to people who do not know that availability zones names are randomized in an account.
Corey: I further to people who do not know that they have since started putting a zone ID into some of their API calls where you can actually disambiguate those between different accounts. And these are things that were not true once upon a time and then become true, but people don't update their knowledge and insight into it. That's what I do.
Jeremy: You mentioned a little bit earlier that you were a little bit annoyed that there are some people at AWS who don't know who you are.
Corey: Let's be clear here, I'm annoyed there are people who don't know who I am. I don't need to bound that to AWS.
Rebecca: Not after today. Full world.
Jeremy: You don't need to bound that to just AWS. Well, speaking of people who are semi annoyed that AWS doesn't know who they are, Ben Kehoe recently published a very interesting article on Medium that I know you commented on Twitter about AWS not having a central understanding of who a human being is, and being able to carry over a lot of those preferences and personalizations, even if they're working across multiple companies and they've got separate logins, and there should be no SSO where you can login and access everything, that could that could be very dangerous.
Jeremy: But just this idea of knowing and owning an identity for a human being. And you extended this to say, "Why do I have to fill out the same 18 fields on the re:Invent form or anytime by signing up for a webinar or things like that?" I'm just really curious, and you said you wholeheartedly agree with Ben on that, what's the problem with that? Why is it such a problem that a company like AWS, it's so customer obsessed and so customer focused as you mentioned, maybe doesn't actually know who their customers are?
Corey: To be honest, it's a legacy story because AWS clearly learns from its own customers, and that's useful in many respects. Originally, the rule was you're one customer, one AWS account. Great. There's still a company that we talk to running 10,000 instances in a single account in a single region instead of having multiple accounts, like a reasonable best practice should that. That takes time. Well, that's a one way door to customers faster. And they didn't build their systems with that in mind. There are a number of companies that nail this, Google Cloud, for example.
Corey: I'm already logged into my Google account in my browser, I go to cloud.google.com and say, "Oh, I want to try this." It automatically knows who I am, it doesn't ask things. Honestly, I've got to level with you, one of the most disappointing aspects of AWS from a team perspective, and I lay this to be clear at leadership, not the individual hardworking people there, is AWS marketing. They are in many cases disjointed, they don't understand who their customer is, they don't understand the value of storytelling. The counterexample is, look at Emily Freeman who recently started there, at any of the talks that she has given and the things she says online and juxtapose that with everything else AWS marketing does.
Corey: It's clear she's not a culture fit, but I'm hoping that that changes and the culture shifts to become more like her. If she leaves and that doesn't happen, they have failed. I want to be very clear on this. But as we look at this, they talk about how data is so important, data is the key and data is the future, and why do I have to fill out that same form a bunch of times? Why do you talk about how important it is to... They launched a whole service called Amazon Personalize, you can present different experiences based upon who someone is and you build profiles of them over time.
Corey: Great. Why don't you try dogfooding that before you start talking about how great it is? The thing that makes a company amazing is also the thing that makes it suck. And marketing's a hard job. I want to be clear on this, because Amazon is a company comprised of micro service organizations, and anytime they have to do anything in a shared sense, it's messy, it's a seam, and it's a challenge. The bill is a great example of this, the console is another, and marketing is another. They build a bunch of building blocks and say, "You can build an amazing house like that."
Corey: "Really? What kind of house?" "To answer that question, here's our guest speaker on stage from Netflix." "Great. I'm a bank, I don't necessarily want to do exactly what Netflix does." Or let's be clear, I think I want to do what Netflix does and I should be actively prevented from doing those things because they stream movies and you are a bank. So let's make sure they contextualize this. But speak to challenges specific to individual industries, if I want to deploy something, and this is, I think, on some level, the blessing and curse of serverless.
Corey: If I want to deploy something into my environment that's going to solve a business problem, I would vastly prefer to just take a package solution, then implement it myself. Let me give you a real example of this. My serverless bill is something like seven bucks a month right now or so for the entire system that builds my stuff. And five of that is something I think was comprehend though I might switch some of that to Anguilla. So I've spent basically no money on this thing, whereas in return, I spend until last week, 120 bucks every month to Epsilon to instrument that application and tell me what's broken and put that all in one place.
Corey: Now, all that data lives already within AWS services, but I don't want to go to five different tabs and cross reference between them to answer this. I want to go to one. And what they are paying at Epsilon to service my account is nothing compared to the $120 a month I'm paying them. And I pay it with a smile on my face because it's the value that it gives me drastically outweighs the cost. AWS is focused on the building block primitives, and well, are we going to charge a markup on the cost? And they're not talking about value because they don't know how to talk about value, culturally speaking.
Corey: You see it in other ways where they cancel, reinforce their Strategic Security Conference, but only after canceling the Analyst Summit a month or two beforehand and scheduling the America Online Summit for the same day and time. Security is job zero, that means it's no one's job apparently, because you see these things, they canceled the whole thing, a week goes by, and then they say, "Oh, it's now a virtual event." Who's minding the store over there? It's is incoherent. They're talking now about their virtual experience for re:Invent that as of this recording is still on.
Corey: It's going to be a live stream keynotes, and afterwards, they'll have the breakout session videos up on the internet. Oh, they could do every year. That the virtual experience is worth having. Why bother? And with travel restrictions being what they are and the next wave of cloud customers being folks who aren't going to go to Las Vegas to learn about these things, build a virtual first event. And the argument is like, "Well, it's sort of sales conference because they make an awful lot of money from sponsorships."
Corey: I do my re-Quinn event, sidecar virtual event at the same time, and I assure you, I am not charging small money to sponsors and they're paying it with smiles on their faces because surprise, it delivers an audience, it delivers value, it tells a story about what they do. The idea of, well, we can't charge anywhere near what we can offer virtual events we can for physical events is untrue. You just haven't figured out how to tell a story and deliver those people what they want. I'm empathetic, truly I am, but even their online event is badly done.
Corey: The reasons people like what I do is I distill it down, because no one's going to watch three hours of one keynote, then the other four keynote, then watch through over 1,000 breakout session videos. If they're absolutely going to have, "Here's my re:Invent in five minutes spiel, my 30-minute keynote rebuttal given hours after the keynote, where I make fun of the things that they announced while rounding them up, is great because it has that veneer of authenticity to it, which is... Veneer of authenticity is sort of a contradiction in term, but it has the ring of authenticity to it because this is what I think about this, shooting from the hip.
Corey: This is what I think it has the potential to do, the potential not to do. And people think that my snark and humor is there because of deep-seated personality defects. And while that's completely true, it also is because without that, the stuff is incredibly boring and it is how I approach bringing it to life. People should be listening to my nonsense for those jokes, not because I'm the only coherent place you can get a roundup of what the hell they're trying to inarticulately say.
Jeremy: Right. And that goes back to the personalization thing. And I will say too, I think AWS as an organization has a hard time connecting with individuals, people within AWS. Rebecca and I met through her being part of the Heroes program or running the Heroes program. And I have so many really good contacts and friends in AWS now, but I think as you said, as a whole, it is very much so disconnected.
Corey: And again, I want to be very clear here, 95% of what AWS does is great. I believe that sincerely, but sitting here patting them on the back for, yeah, S3 is a marvel and S3 storage lens is incredible. And nimble studio, I think is inspired as far as changing some of the trajectory of these things. That's great and all, but what's the value in talking about those things? And belaboring the obvious, at some point, that's marketing's job. They should be singing those praises, but marketing is constrained.
Corey: They have to be at every company, where they can say that something that they do is great, or they can say nothing about it, but they can't say, "Yeah, there are problems of what we're offering," because that's not how marketing works. I tend to take a different view and I don't work for these companies.
Rebecca: Speaking about your humor, bringing things to life and what it takes to be authentic, Jeremy and I feel like you might be a Russian Twitter bot with how active you are. Can you confirm or deny that?
Corey: I have interesting expression of ADHD in many respects. The things that I wind up gravitating towards are... Again, the thing that makes you rock also makes you suck. I am terrible at preparing in advance for things like that, which means that when you suck at preparation, you've got to be good at improv. There's no other answer. You have to be able to deliver on demand, which makes my podcast recording super easy because, "Oh yeah, here's what we think we're going to talk. Nope, don't need to see it. Ask questions, I'll answer or vice versa."
Corey: Organic conversations, it's helpful. I can't do the preparation thing for conference talks and the like, but knowing how I handle on the spot questions is useful, but it's less work on my part. And it also means that I can be on panels, or I can do and ask me anything on a stage to a crowd and not worry that they're going to ask me something that's going to put me in a very uncomfortable position, because I know how to handle those things. Again, this is me. I'm not suggesting other people pursue this. And I actively encourage people not to, because I hurt people as I was learning how to do this at various times throughout my life.
Corey: And the failure mode of clever is being an ass, and the world doesn't need more of that. There are lots of paths to wind up audience building, to driving engagement. Humor is one of them, and it's the path that I've taken, but there are many others. Find the thing that works for you.
Jeremy: Yeah. Well, I think you do it really well.
Corey: Thank you.
Jeremy: I know I enjoy reading it and seeing how you're going to react to something. I always love your commentary on naming things as well.
Corey: There's another reason behind naming to be very clear on this, because if I sit there and I make fun of some esoteric expression of how there's 17th or 18th service that manages containers winds up working, that's an inside joke that people are going to need... It requires reading first for this joke to make sense. That's a silly name, is something that anyone of any skill level can appreciate and participate in. Making jokes and everything I do more broadly accessible is key. Give me a choice between giving a 100-level talk and a 400-level talk, I'll buy us for the 100-level talk every time, because I want to be as broadly accessible to folks whenever possible.
Jeremy: Love it. Hey, for those few people who don't know who you are, if they do want to follow you or sign up for your newsletter, how do they do that?
Corey: Lastweekinaws.com is where it all begins. That links to my Twitter profile, that asks you to sign up for the newsletter, which I assure you want to sign up for, and other things as well. We're about to launch our charity t-shirt campaign, for example. We try and keep things interesting and keep it light and new, and I crave novelty. So doing the same joke over and over, it doesn't work super well, how do we express it in new and interesting ways? More to come on that over the next year or so?
Jeremy: Awesome. Thank you, Corey.
Corey: Thank you both for having me.
Rebecca: Thanks, Corey. Good to talk to you.
