September 16, 2019 • 36 minutes
Jeremy chats with Forrest Brazeal about the CI/CD challenges facing enterprises, how to take a pragmatic approach to building pipelines for your serverless projects, and what tools are available to help you.
Forrest Brazeal is an AWS Serverless Hero and a Senior Cloud Architect at Trek10, where he hosts the Think FaaS serverless podcast and contributes to their open source efforts. He understands the challenges faced by enterprises moving to the cloud and loves building solutions that provide maximum business value for minimal cost. Outside of his day job, he interviews the top names in cloud for his “Serverless Superheroes” series at A Cloud Guru and also creates the FaaS and Furious webcomic. He is the co-chair of ServerlessConf and regularly speaks at workshops and other events in the serverless community.
Jeremy: Hi, everyone. I'm Jeremy Daly, and you're listening to Serverless Chats. This week, I'm chatting with Forrest Brazeal. Hey, Forrest, Thanks for being here.
Forrest: Hey, Jeremy. Thanks for having me. It's great to be on the show.
Jeremy: So you are an AWS Serverless Hero as well as a Senior Cloud Architect at Trek10. And I think many people in the serverless community, as well as the underground technology rap scene, know who you are. But why don't you explain to listeners a little bit about your background and what Trek10 does?
Forrest: Sure thing. Well, and I think the underground tech rap community is basically just me. So that's a small pond there, but yeah, so my name is Forrest Brazil. I'm a Senior Cloud Architect at Trek10. Trek10 is an AWS advanced consulting partner. That means we work with AWS Technologies. We focus primarily on the cloud native side of things. So think serverless, IoT, basically anything you can build while using the best practices that AWS wants you to use. I spend a lot of time helping clients put things like that together. I also spend a fair amount of time being community-facing. I love to educate and advocate for serverless technologies wherever I can. That's why I do the serverless hero thing. And it's great to be here with you now.
Jeremy: Awesome. And I think I could probably talk to you about — I think I have talked to you about pretty much everything that serverless has. But I think what I want to do today is focused more on the sort of CI/CD process for enterprises. And maybe I know you've done a lot of stuff in that space, you and Jared Short have, and maybe you can start by telling us, sort of what's the difference between sort of your typical CI/CD process and one that involves serverless now?
Forrest: Sure, Ultimately, there's not necessarily that much difference. I mean, the underlying goal is the same. I've got code. I want to get it off my laptop. I want to get it into production and serving my users as safely and quickly and efficiently as I can. And that goal doesn't change because you're suddenly using different infrastructure, or less infrastructure hopefully, but I think where it gets a little bit interesting for serverless is you all of a sudden start having the cloud become part of your software development lifecycle a whole lot earlier than you may have been used to it in the past. So you don't write this code locally, and then you throw it over the wall and it gets deployed on some infrastructure that you're not thinking about. So much of your development process now actually is tied up in that configuration and figuring out permissions, thinking about latency at the cloud level, right? You're simply not getting a very realistic picture of what your application's going to look like if you're trying to mock all that and develop it locally. So we see people trying to figure out how can I get this app into the cloud tested earlier on in my lifecycle? How can I do that collaboratively? Or how can I do that without stepping on other members of my team who may be working on different features, perhaps in a shared account? That's where we try to put some best practices together that make things easier for serverless developers.
Jeremy: Right. Great. Okay. So why don't we start talking about it specifically with enterprises then. So you've worked with a lot of enterprises. I've worked with a lot of enterprises. We know that there are certainly challenges facing enterprises when moving, just to the cloud, let alone you know, just moving to serverless. But just from a CI/CD perspective, what are some of these challenges that enterprises face?
Forrest: I think any time you're in a large organization where you're contending with multiple teams, teams that have different priorities, different technology stacks that they're comfortable with, the biggest challenge you face is just that. Despite the temptation, there's really not a one-size-fits-all solution that you can impose. So we see a lot of these central cloud teams now, and a lot of times they have a cute name, like the Cumulonimbus team or something like that. You can always spot them a mile away. You see these teams come in and they say, “Well, we gotta solve the CI/CD problem.” They've been given that problem to solve and understandably, it's a problem. Right? Code is not getting into production fast enough or it is, and there's all this shadow IT happening, right? So people want to find a way to get around that. They give this task to a central IT team. They come up with some great solution that involves CodePipeline or something like that, and they put it in production, and they say, “Hey, everyone, come use this.” And they're rather shocked to discover when no one uses it. And the reason for that is, as I said, is that you've got teams that all want to do things their own way and they view the centrally imposed CI/CD standards is being just another thing that are stopping them from getting code to production quickly, which, of course, is exactly the opposite of what CI/CD is supposed to do. And so what you have happened then is the shadow IT problem just gets worse and your development teams continue to work around your central team, and you, frankly, wind up with a worse problem than he had in the first place. So when you're working with an enterprise then, the CI/CD challenge is how do you put something together that solves the centralization problems you have, the governance problems where you really do need some amount of rigor around who can deploy to production, when it can happen and what those deployments look like. And then you marry that to the also very real necessity for dev teams to be able to accomplish their work the way they want to. So that's the problem that we try to solve.
Jeremy: Yeah, and I think you get this other problem where nobody knows who owns the process, right, especially if you have certain teams working on different components, you know that throw-it-over-the-wall sort of DevOps mentality in a fast-paced serverless environment, anyways doesn't really work anymore, right?
Forrest: That's exactly right. And it's not that it can't work. It's that it doesn't work the way people expect that it will, because they tend to think about these problems at the level of the team. Once you move into a large enterprise again, the moving pieces rapidly increase in number, and you've got to find a way to coordinate those people together.
Jeremy: So what is the typical, or maybe let's say, a basic CI/CD pipeline look like and maybe we can call this the “hello world” version, which I think most people read a blog post, they see this laid out, they know what that typical, or they maybe feel comfortable with that typical deployment process. But why doesn't that work in enterprises?
Forrest: Sure, the “hello world” pipeline, as you call it, would involve some combination of a few different steps. Obviously, you want to start with a source control repository. You want to push your code to source control. After that happens, you want to run some number of build tests on it. You want to run your unit tests. You want to do your linting things that happens statically on the code, perhaps static analysis if you're doing some security stuff. And then you move on from there to a deployment phase where you want to actually pick up that code and you want to push it out into some sort of non-production environment where you can test it and people have all kinds of different environments they like to involve there, whether it's dev, whether it's staging or QA, non-prod, whatever you want to call it and then at some point, that code will be validated. Ideally, you'll run some kind of smoke tests or blackbox tests, end-to-end tests on the deployed environment. Once that's out there, then you will want to deploy to production. And a lot of people like to have some kind of a gate, some kind of approval there, where someone has to manually give a thumbs up and say, “Yes, I agree. The automated tests look good and this is ready to go out.” That's the “hello world” idea I think people see that laid out, and they say this is going to be great. The challenges then that you run into with that in the enterprise, of course, you've got, as I said, different teams, but you've also got different branching strategies. You've got these problems where oh, I had a problem that I need to hot fix to production immediately and my boss is telling I simply don't have time to go through all this normal process. So how can I just circumvent all that, because the business reality is I need to push it out now. That's you know, all of the hypotheticals that you've put down on paper sound great until you are at three in the morning and you really need that IAM permission to be corrected all at once.
Jeremy: And I don't think that's a common case where somebody has to rush a hot fix out to production, is it? Come on now.
Forrest: No. Although, you know, I've seen cases where people put these branching strategies together where you go to master and then you have, like, staging branches and production branches above that and one of the challenges people have there, of course, is they want to go to hot fixes and they hot fixed a production. They've got to figure out a way to merge those branches back down onto each other, and you very quickly end up with a true version of your code, which is deployed in prod, and then several different half in-progress releases in different other environments, and it rapidly collapses in on itself, and no one knows of the state of any of the environments. And that, of course, is worse than what people had at the start.
Jeremy: Definitely. Alright, so let's talk about repositories for a second, because this is another thing. There's the age-old debate between mono repo versus multiple repos and so forth. And I've seen this a lot, and I'm sure you've seen this a lot, where sort of the strategy they choose, that the enterprise chooses, for their repositories in terms of, you know, multi versus mono repo, that can also be a challenge when developing a CI/CD pipeline as well.
Forrest: I think that's true. I don't see a lot of organizations going with a true mono repo. I mean, nobody's Google, and I think people understand that. So you don't see an entire organization that's putting literally everything in the same repository. So when people talk realistically about mono repo versus multi repo or micro repo, or whatever you want to say, they're typically thinking about it in the context of some smaller problem that they have their heads around. So for this particular application or service that I'm designing, should I have a single repository or should I be splitting this up somehow by pipeline or by microservice or component of my app? That's really what the question is, and I think maybe that terminology gets confusing. What I've come down on, I guess, and I've spent a lot of time trying to get my head around this, multi repos are great and it a lot of times makes sense to have a repo per pipeline if you have multiple pipelines out of a single repository that can get confusing and hard to manage. But where people struggle is they try to break that down too much, and what they've done is they've created dependencies that span multiple repositories. So if you repeatedly find yourself having to encompass two or three different repositories or more in a single feature that you're pushing out, you're probably creating problems for yourself that you could solve if you had all that stuff in one repository to begin with. So I see people doing things like trying to create these orchestration systems on top of multiple repositories, and on top of multiple pipelines, just to try to control when dependencies go out and that just, it's adding an extra level of complexity that I don't think needs to be there. You know, don't be afraid of git-merge, and merge conflicts. That's actually a protection the repository is providing for you, and when you get away from that, you have to start reinventing wheels that I don't think are necessary.
Jeremy: Yeah, I like that idea too about being very careful about having dependencies or too much coupling between multiple repos, because that's always that problem where you're launching into a new environment or into a staging environment or something and you need to launch each deployment, or each pipeline needs to run an order. Otherwise, the other ones won't be able to access bits of information from it. Alright, so what about feature branches? Because that's another thing that I think helps, especially when you go to a smaller repos — maybe not the micro repose, but packaging individual services — when you're launching different features to have those separate. What are you thoughts on feature branches?
Forrest: I'm a fan of feature branches. I think that in my experience, it's the pattern that is best suited for enabling teams at your standard development shop as opposed to something like trunk-based development. So when we talk about feature branches, we're talking about a flow that would look something like: I make a branch off of master or off of my stable branch. I check that out. I commit some amount of code that solves a problem for me. I test it. And then when I'm ready, I pull request that branch to master, goes through code review from the team. And then once that code is merged into master, at that point, it's off to the races and it's being sent through whatever staging QA accounts, and it's on its way to production, and that process hopefully happens very quickly. It's mostly automated, and it's not like it's sitting in limbo off of master for days, but it's not necessarily an environment where you're creating a lot of bespoke release branches off of master. It's very much once I merge this feature, and then I'm ready to go to prod, or I should be ready to go to prod if the tests pass. I’ve seen that work well for teams because it lets different developers collaborate and it also lets you, if you need to, to make those hot fixes directly to master without a lot of additional fooling around and without worrying about having you know do they also need to be merged into existing releases that I'm adding on to. It just seems to work well for a lot of especially small-to-medium sized teams when they understand their code base pretty well.
Jeremy: Yeah, and that's also, you know, sort of one of those processes where when you're developing using feature branches, it's good to break up, have very modular code in there so that you're not always overwriting the index.js file or some some main handler file where you've got some layers of code in there where things can be manipulated without constantly having those conflicts, like you said when you’re merging multiple branches. So then, before we move on to the next thing, I just wanted to talk quickly about some of the common pitfalls with CI/CD, because I know I mean, I've seen this before, especially when you have multiple teams that are merging into some sort of centralized repository or centralized, you know, process that you get these QA teams that are constantly getting updates and is there a way to sort of mitigate against that? So as you have all these different teams that the QA team should kind of stay on top of what they're testing, as opposed to always getting new updates.
Forrest: Yeah, so you're talking about, like release cadences and things like that, and every team is different here and their process is different, and a lot depends on the maturity of where they are and what they already have from a testing perspective. I think that it it does make sense to have a single staging environment where things come together before prod. But it's important to not mistake that environment for production. No matter how similar you make it, and how automated it is, you won't really know how the app’s performing until you get it to prod and can test it there. I'm a big fan of, you know, blue-green releases for that reason, and rolling things out that aren't live yet, but that are in the production environment so that you can run tests on the infrastructure that's going to be serving your users before they've actually had a chance to see it. And you can get obviously more complicated with that, with feature flags and and canary releases as well. But that's a whole discipline that I think a lot of teams are just not ready for yet. It's not something I'm seeing in most places. There's a lot of work to do before we get there, where teams are comfortable with that.
Jeremy: Alright, Cool. Alright, so speaking about all this work to do, you wrote a very good post earlier this year called Enterprise CI/CD on AWS: a pragmatic approach and I really liked it. Thought it was really interesting how you sort of split these things up. So let's talk about that. Let's talk about this pragmatic approach to CI/CD. And in there you mentioned, you know, you run into this Conway's law problem where every organization is different. Every organization does things differently. So maybe we can talk about that. What is your pragmatic approach?
Forrest: Sure. So at some point, designing a process like this for a large enough enterprise is almost entirely an organizational problem and a human problem rather than a technical problem. The technical challenge is relatively easy to solve with the tooling we have today. I can set up a pipeline and I can set up approval gates, no problem, and I can figure out a way to pass artifacts back and forth. Big deal. It's been done a million times. But when I have a whole bunch of different teams that have different technology stacks, as I said early on in the show, that all have their own processes for how they like to get to production. But I also have some need for central governance and a need to control who actually has access to deploy things in prod, it gets more challenging. And the way that seems to work for a lot of folks to tackle this is to split build and release. So when I say split build-and-release, I mean you're taking the build piece of your pipeline, which a lot of times maps onto the CI that continuous integration half of CI/CD. That would be something that typically is very tightly controlled by an individual dev team. So they would be using a source control repository of their choice. Could be GitHub. Could be CodeCommit — probably not CodeCommit. Let's be honest, but I suppose it could be. And, you know, whatever they're choosing there, that'll be hooked into a build pipeline of their choice, which could be Jenkins-based, could be CodePipeline, GitLab, who knows and then that will run build tests that they write, unit tests they write. It'll run their code coverage stuff. They'll probably have to hit certain standards that are set by the organization, in terms of hey, you have to have this percent of code coverage. You have to have this sort of security scanning that's running on your code, but they get to choose how they how they implement that and what kind of pipeline they put it in. And I find that most dev teams now you know, have that capacity or want to have that capacity, to be able to put something like that together. The output of that build pipeline, that build process, is, quite simply put, one or more deployment artifacts. It's a deployable chunk that can be put out into production into other accounts, and that's something that could go into Artifactory, could go into S3, it just needs to go somewhere where you're release pipeline can access it. The release pipeline then is the piece that makes more sense to be owned by a central team and the central team then would take care of actually doing the code promotion across accounts. So their pipeline will be the one that spans, you know, not just the development account, but staging production, QA, whatever. They would have the permissions to deploy into production. They would have the approval gates on there, they would have the timing in terms of whether there needs to be some kind of a maintenance window before something gets rolled out. They can own all of that. Is this you know, the healthiest possible paradigm in a perfect world, right, where you have build-and-release split that way instead of the dev team owning things themselves all the way to production? Probably not. But this is actually the way most large enterprises function and will function for the foreseeable future is that there's got to be a little bit more governance around who could do what. And it also takes some of that need to juggle all those accounts away from the individual dev team. They can focus on what they do best. Your SRE team can focus on what they do best, and this seems to work fairly well.
Jeremy: So then, for something like feature pipelines would that be on the build side? Would you deploy your own feature pipelines there? Or is that something that also goes to the release side?
Forrest: Yeah, that's a great question. And that would definitely be on the build side. You would want them to be able to spin up and test their own infrastructure before the code gets merged to the master branch, again envisioning this feature-to-master kind of flow for your source control. So you can have your central team help thereby providing some scaffolding, possibly even something that gets vended out at account creation time. So having really good process around how you create another AWS accounts and how you see them with tooling and processes for people is a really big part of this. But yeah, having that ability for people to push code and have it spin up a feature branch should be in control of the dev team.
Jeremy: All right. And you mentioned security like some static analysis and things like that. But really, where does the security fall? Does it sort of fall on both sides here?
Forrest: Yes, absolutely. And that's where I mentioned, even if the dev team has some control over what tooling they're running there, and, of course, the dev team, they're writing the code. They know better than anybody, you know, what they should be looking for when it comes to certain security things. There do need to be standards across the org and there needs to be communication in terms of what's getting looked at. So from an AWS perspective, you know things like stars and IAM policies, there needs to be policies around “hey, eyeballs are on this” and you know, that if we're seeing the show up in a code review, there needs to be a conversation happening. And that's just something that you work with the organization over time to develop that excellence.
Jeremy: Alright, so what about testing? So you mentioned unit testing and linting on the build side. But is there more testing that would be done sort of on the release side as well?
Forrest: Absolutely. Big fan of any kind of automated smoke testing, blackbox, end-to-end testing you can do. In fact, I think you can't go to production without it. So when that infrastructure is deployed, whether it's QA, staging, prod — even, and especially in prod — you want to have tests that run as part of your release pipeline immediately following that that are doing deep health checks, that are, you know, provisioning test records, making sure you can get all the way back to the database and back, things like that. And you know that's just a no-brainer. The challenge there, too, is those tests, you want them to be run and deployed by your release team in these various accounts, but you actually want them to be written by your dev team because they're the only ones who actually know how the application’s supposed to behave. So I built things in the past where you'll have a standard, an interface that's set up between your build-and-release team, where the dev team will, for example, let's say you're working in a CodeBuild CodePipeline type of an environment, your dev team will go ahead and write some scripts that launch their end-to-end tests. So it could be, for example, Cyprus if you're using that tool for end-to-end testing, and they will go ahead and create build specs that call their Cyprus test, and those will be stored in a known location in the repository. So by the time your release artifacts get out to release, the release pipelines, which are standardized, will know to go look in that location, find whatever test files have been defined, and run those end-to-end tests.
Jeremy: So you mentioned approval gates in there, and this is something — so again, I think we are talking about continuous delivery, not continuous deployment, when we talk about CD in this context anyway. So I don't know how many organizations are confident enough to just automatically, once they pass the automated test, to go ahead and have those deployed to production. So obviously, we're putting in approval gates here, and that's a human function that needs to happen. So what about multiple gates though? That seems to be something that people look for.
Forrest: I think more people think they want that than actually want that and what you're talking about I think is something like a nuclear key approval, you know. So envision the two guys in the nuclear command center, each of whom has their hand on one key and both keys out returned for the missile to launch. People sometimes think they want that for a software deployment process where you need tWo individual people who probably belonged to separate IAM approval groups or whatever to be able to approve something — and you can certainly implement that in CodePipeline. I think I've got a blog post out there somewhere that explains how you can do it — but over time, that seems to be a little bit frustrating for folks. It actually does slow them down a bit. And what seems to be better in most of those cases is to have a trusted group of people who can approve, but then also just to maintain really good auditing. So if something does go wrong, you know what happened and you can go back and address it.
Jeremy: So I also see that sort of being helpful for escalating a problem, because we would only fail a build or we would only not approve something if there was something seriously wrong with it, right? So is escalation another sort of reason to have these approval gates?
Forrest: I think so. Escalation is another big topic, and we could probably spend a whole podcast talking about that. In general, and especially in these big org, where you just have huge multiplications of AWS accounts now being vended out through AWS organizations, or there's even people who've outgrown the capability of AWS organizations, how do you keep track of all these accounts? How do you figure out who can be escalated to access different functionality inside of them, you know? And is there such a thing as a central team that has to have eyes on all of that? Or can you delegate that permission to trusted people inside of the individual dev teams? That's a question that more and more folks are starting to answer. There absolutely are ways you could do that with escalation. We actually do that here at Trek10. We've got a nice little Slackbot that lets us temporarily attach policies to our roles, to be able to deploy things in production for various accounts. But I think that's a little bit of a separate topic.
Jeremy: Okay. Alright, well, so let's move on to tooling, right. So you mentioned, you know, obviously the CodeBuild and different code repositories, GitHub, Bitbucket, things like that. I mean, obviously that's part of it, having that. But then CodeBuild and all these other tools that do it. So first, let's start with this. Can we use standard build tools to deploy our serverless applications, like the Travis and Circle CI and Jenkins and those sort of things?
Forrest: Absolutely can, and people do it every day. However, if you are really getting into the serverless mindset, which is, I don't want to maintain infrastructure that's not directly relevant to me in my business, it would seem that a massive piece of infrastructure that will be nice to not have to worry about anymore is your build server. I think all of us have spent time troubleshooting a giant Jenkins instance that's been running at a disk space because of one rogue job that's affecting everything else, right? That's not a problem we should have to have in a serverless world. We want to be using tools like CodeBuild, which reliably gives me an ephemeral build job, and it doesn't affect any other jobs that are running at the same time. And if it goes away, I just spin up another one and I only pay for what I consume, right? Those are all great serverless fundamentals that we want to be working with. So, yeah, you can provision any old build server and launch your serverless apps. But I think it's much better if you can, to try to find a way to deploy your serverless app serverlessly. And there's more and more tools now that are letting you do that.
Jeremy: And so what about for people that are starting to move to serverless or are starting to migrate parts of their application? You know, would you suggest that they rebuild their existing CI/CD pipelines using these tools? I mean, I would think that some of these legacy tools would sort of be slowing down the process.
Forrest: They can. And I do recommend that wherever possible, it's ultimately just a really long process though usually. It's interesting. We see a lot of organizations now that are a few years into their cloud journey. But for them that means that they have five-year-old tooling that in some respects is sort of legacy now. So they’re that first wave of cloud folks and perhaps they've developed this large multifarious Jenkins instance that’s serving an awful lot of cloud teams and they're outgrowing that and they're running into all kinds of problems with it. But at the same time, it took a lot of years to get people actually onboard and using that and using probably shared tools and libraries and plugins that are accessible to everybody. So it's no joke. You can't just unwind that overnight, but it can be done and part of the way that can be done, as I was saying earlier, is trying to hand some of that control back to the individual teams, so you don't make a central team a bottleneck for that in the first place.
Jeremy: Yeah, so what about some of these tools that are coming out that are specific for serverless? So there's seed.run, you know, something like ZEIT or Architect, and I think Begin is part of the Architect framework, and serverless.com now has their own. I mean, I like these tools because I think they're great for small jobs, but what are your thoughts? Do they fit into the enterprise world?
Forrest: So you know, a lot of these tools are early on in the process, and I don't want to speak individually to them right now. Some of them I've seen do great things. I'm excited about some of what Begin seems to be offering. I don't think it's quite released yet. I think the important lesson to take away from those tools right now because they're so young and so early — so who knows where exactly they'll go on an individual basis — but I think the important thing to take away is that this is obviously a big problem that has not been adequately solved yet, at least not at the 80% level and that's why some of these tools are starting to proliferate because people very rightly are saying, “Oh, it's challenging. It's difficult to be able to put together a reasonable CI/CD pipeline that works for my developers and works for serverless without a lot of frustration.” So they're trying to automate some of that away, and we'll see how that goes over the next few years if more of those features are just available natively in the cloud providers or if one of these tools really takes off, but I think we would be foolish to dismiss the obvious gap that there currently exists for developers to be able to do this more easily,
Jeremy: Right, and so you mean basically, I don't think any organization, no matter what we build or what someone builds, there's gonna be a turnkey solution for enterprises.
Forrest: No, I mean, what do they say about enterprise software? It's solving the problems of today by creating the problems of tomorrow. I think that's probably true for any kind of CI/CD system as well, whether you build it yourself or whether you buy it from somebody, but it's just because it's an organizational thing at that level, as I said, it's always going to be growing and changing. There will always be interesting new challenges to solve.
Jeremy: Alright, so another tool that is out there was built by you and Trek10 which is this Quick Start CI/CD for AWS. So I'm not sure this is the best thing to walk through step-by-step on a podcast if someone's listening in their car, or mowing their lawn. This might not be the easiest thing to digest, but maybe we could just go over it quickly because I do like how it's set up. I do like what it does, all the different features that provides. So do you want to just kind of walk us through it at a high level?
Forrest: Yeah, let me give you the elevator pitch for it. Maybe that will help. So this grew out of a couple of different projects. Well, really, I guess multiple projects we've been doing for various clients at Trek10. And we were doing exactly what I've been describing over the course of this podcast, which is going at enterprises and helping them figure out how to set up that feature branch-based workflow and how to get serverless apps deployed using these ephemeral services like CodeBuild and get them pushed out across multiple AWS accounts. And we were solving some of the same problems over and over again and had all this CloudFormation that we were using. And so we said, “Hey, let's let's spruce this up and let's make it available for everybody.” And that's what the Quick Start is. Some of the things that the Quick Start does I think probably won't necessarily be needed long term as AWS continues to roll out more features. But right now a lot of this you can't do natively today, so you do need some additional CloudFormation to come along and help you with that. The biggest thing there is we really wanted to enable people to dynamically create feature branch deployments. So as I was, I think saying earlier, when I push code to a feature branch as a developer with a serverless app on AWS, I want that feature branch to automatically spin up for me a deployment of my app that's namespaced according to my branch; it doesn't conflict with anybody else. So if I need to, I could have 20 developers all on the same account, all deploying. Nobody steps on anybody else. Harder to do in the serverless world than you would think, but at the same time, it's also very cost effective because you're not paying for that infrastructure unless you're actively using it. So we built some Lambda, and some SNS that spawns those dynamic pipelines for you and you can check that all out. It's open source. There is also the ability for you to promote those things across accounts and you know it's basic. It's got the full end-to-end life cycle in there. No doubt as you work through this, if you decide to work through this, you're going to find things you'll want to tweak to your use case. There's the concept in there of a shared services account, so we actually take the pipeline infrastructure. We segregate that off into its own AWS environment, and then it reaches out and passes roles cross account to be able to deploy in dev, and staging and prod. And that's a very best practice thing to do. Some people want to actually create multiple shared services accounts, or sometimes they call them tools accounts, one that has production access and one that does not. But ultimately, you know, you can play with that. The goal is you want to make it easy for your devs to deploy and spawning those pipelines on their behalf seems to be a good way to do that.
Jeremy: Now, when you're building like a development version or something that's launched to a different environment, obviously with serverless, we usually have some naming conventions in there for functions and for DynamoDB tables and things like that. So are you generating, are you rebuilding the artifacts at every step here, or is there some way that you're creating artifacts that can then be promoted?
Forrest: In this case, and this is me casting my mind back a few months, so apologies for any inaccuracies here, but I think in this case, we did end up with a separate set of artifacts for the feature branches. And then we rebuilt the artifacts when you push to master but then that same set of artifacts is used across all the accounts you promote once you merge code into the master branch. And definitely, if you split up your tools accounts between dev and your prod workflow, you would have two sets of artifacts that were generated. I don't know that that's necessarily a big deal, but if it is, of course, you'd want to go with a different approach.
Jeremy: Okay, great. Alright. So maybe we can wrap this up by, you know, giving some advice. So for the enterprise that is looking, or is having some trouble with CI/CD or looking to go serverless — I'm asking you multiple questions here, but let's say for the enterprise looking to go serverless, has some question about their CI/CD process, besides calling Trek10, what would be your best advice?
Forrest: That's a great question. So obviously call Trek10, but even before you do that, check out the Quick Start. You can play with that yourself. It'll give you a good feel, I think, for how far you can go on your own. I definitely would recommend trying to use some of the AWS native services if you are in AWS. If you get to where you need something that is a little bit more flexible or something that enables you to work with additional tools that are not natively integrated with the AWS services yet, at that point I would suggest looking into that build release split that I discussed earlier and that you can find in that pragmatic enterprise CI/CD post.
Jeremy: Awesome. Alright, well, I will get that stuff into the show notes, but anyways, listen Thank you. This was awesome. This is certainly a complex topic that we probably could go into detail about and spend hours talking about. But anyways, why don't you tell the listeners how they can find out more about you?
Forrest: Sure. Well, I'm pretty easy to find. You can find me on Twitter @forrestbrazeal. Also on LinkedIn. And of course, I do write a number of things for the Trek10 blog. I have a podcast there called Think FaaS — functions as a service. We've been a little bit lax about putting out episodes of that recently, but we are doing a bunch of live episodes of Think FaaS next month at ServerlessConf in New York City. Jeremy, I think you're speaking as well.
Jeremy: I am, yes.
Forrest: Yeah, you got a full size talk. They're not a Think FaaS talk, but yeah, we'll be putting those out on the podcast here in the coming months. I am co-chairing the conference this year. So if you have not signed up for a ServerlessConf yet, please do that. I would love to see you there. I think It's going to be a great couple days and you can hear Jeremy speak on — what are you speaking on, Jeremy?
Jeremy: I am speaking on building resilient serverless systems with non-serverless components.
Forrest: There you go. You don't want to miss that.
Jeremy: Alright, Awesome. Alright. Thanks again, Forrest.
Forrest: Absolutely. Thanks, Jeremy. It was a blast.