the Stack Overflow podcast, a place to talk to all things software and technology. I'm Ryan Donovan. I am at the blog here at Stack Overflow and today I'm joined by two great guests, Finn Glover, CEO and co-founder of Schematic and Ben Papalone, the CTO and also co-founder of Schematic. We're going to be talking about feature flags, how to manage them on both a business and technical level. So, gentlemen, welcome to the podcast. Glad you're here. Thank you so much for having us, Ryan. Thanks for having us.
Of course. So at the top of the show, we'd like to get a little info about our guests. How did you get into software and technology? What's your origin story? Ben, do you want to lead or do you want me to? Sure. I'll jump into it. I've been kind of a startup journeyman for a better part of it.
couple of decades here, joined a startup in like 2006 as a high school intern, a bunch of Virginia Tech students that were building an email product and kind of got bit by the startup bug doing that, went to college and just kept studying computer science and learning how to program and doing different startups. And here we are 20 years later, almost just still working on B2B SaaS, you know, so I've been through
you know, a variety of different products throughout there, but generally have stayed fairly consistent as kind of a early stage engineering kind of persona on products like that. And if you want to explain yourself, Heckman.
I guess I'm at Journeyman is the right word. I've been a startup Journeyman as well. I founded my first company in 2012 and originally I was kind of a lifestyle business and about four years in we saw an opportunity to pivot this services company into a software company.
And when we made that decision, we needed somebody who could lead the company technically. And that sort of is how I eventually got introduced to Ben. This would have been 2016 or so. He was living in Austin, Texas. I was in Chattanooga, Tennessee. For some reason, he agreed to come to Chattanooga, join that company. We ended up raising a series A, moving the company to Atlanta. That company ended up being acquired a couple of years later.
Ben went to another growth stage software company. I went to another growth stage software company, but behind the scenes at night, we were trying to figure out what's next business to start together. You know, that led to what we were building now, which happy to get into, but we founded this company about 18 months ago. And so now I think Ben and I have been working together for the better part of 10 years on a few different companies and a bunch of nighttime projects that never got off the ground, but this one has gotten off the ground. Okay.
Yeah, well, let's talk talk about that project the future flags I see articles about how hard it is the issues with it, whether you should run experiments with future flags all the things what's what is the
the problem with them. Why is it so hard to get future flags right? Yeah, I think there's a lot of reasons. I think there's kind of like a theoretical background of how future flags are supposed to be used and then kind of an in practice, what ends up happening. And there's a big gap there. So basically like the premise with feature flags is like they should
Mostly be used for operational purposes like you're releasing a new feature, but not all the pieces are done yet. So parts of it stay behind the flag till it's done or maybe you're ready to release it and you want to do a slow rollout. So you use the flag to do your slow rollout. AB testing is another one that starts to kind of like get out of the scope of like DevOps or CI CD or operational use cases into more like product.
or kind of business adjacent use cases. That's a common one like experimentation. But generally like the rule of thumb you see a lot is like these should be short lived, right? Like keep the flag in your code base for as long as it takes you to get the feature fully rolled out. And then you should kind of like go back and tidy it up. But if you don't, that would be considered like technical depth.
Right and I would say like the main reason for that premise is like one they seem like such a common constant presence now
But, you know, feature flags did kind of like emerge as part of like the DevOps trend in the mid aughts. And obviously just checking a flag in your code base was a thing before that. But like as a kind of like a tool, you know, like it emerged during that period, like especially with LaunchDarkly and companies like that getting traction. So like part of it is just the fact that it's like a, you know, comes out of that DevOps world. And so those are the use cases that got a lot of adoption initially for like feature flag services.
The other part of it is just like, if you have a bunch of flag checks in your code base and you don't actually need them, then either you have a lot of untested code paths in your code base, or you have a lot of tests to write. You know what I mean? So that's the downside of leaving a lot of flagging in your code base.
My experience especially with fast-moving software companies, which is hopefully most of them, is there tends to be not a lot of appetite to go back after, to clean up post-facto work with your features.
And it just sometimes becomes, it's not always made clear to the engineers, like, when are we actually done with this feature, right? Like they might finish there, all of their work with the feature, put up on the flag, and then maybe there's another team that manages to roll out. And there's like never a moment where they go back and say like, okay, you're cool to remove this flag. Now we're done with it. There's just sometimes that disconnect, but for that, or any of several reasons, like flags oftentimes don't get cleaned up. So you end up with this sprawl regardless. Flagging tools will often like have
you know, capabilities in them to help with this, like a last check that date, like if that's still getting checked and it shouldn't be like, that's a, you know, a red flag to somebody to go be like, Hey, we should remove this flag or vice versa. But it's like a fairly unsolved problem for the most part. That's one thing that's very hard. Like that's what we sometimes think of as like zombie flags. I was saying, you know, what is the actual issue with leaving this flag in there? Say it's, you know, there's no
actual feature behind it, there's still a flag in the code base that's checked. And when it's checked, it actually does nothing. Does it cause any harm on either like a product or tech level? I think the hard cause is generally just add complexity. Like there's so much that engineers have to worry about already, like having unnecessary things to worry about is just kind of like a, you know, it's like a cognitive load cost thing that
If you don't need it, you shouldn't have it, which is kind of the argument I just made about the excess code paths, but there's a flip side too, which is like maybe the flag's been removed and now it's just cruft in the FM tool, right? Like the flag just shows up and launched our clear whatever schematic or whatever tool.
But flipping it does nothing and so that has it that has some harm in the sense that like i've been in situations where like let's say a product manager said i'm gonna give this feature to this user flip the flag said hey user you're all set and then the user said you know what i'm pretty upset that you said i got this feature i didn't get it.
product manager is confused, goes, talks to the engineer, turns out the switch does nothing, you know, it's like pushing a button, trying to cross the road and it's not plugged in to anything, right? So there's, there are some harms there, right? They're kind of like all within the context of like communication and unnecessary complexity, I'd say. So any sort of like business consideration side, like, is there
a way of thinking about feature flags that can minimize the eventual sprawl, the eventual tech debt from them. If you're only using it to differentiate product tiers, is that a reasonable way to think about feature flags or is that missing out on the full power of? I think it's a reasonable way. What I would argue and what we would probably argue as being the schematic perspective in general is this case I've been making for feature flags for all being a problem.
is something that we should maybe just accept. It's kind of like...
the Planet Fitness approach to feature flags. Like, it's going to be this way regardless, you know, like, it's just kind of like, you're going to have the sprawl, right? And so a better way of looking at it is probably like, let's make sure you have the tools and the kind of practices with how you approach feature flags that you just kind of see it as being okay to have long lived flags in your code base. Because not only does that mean you don't have to feel bad about it, but also like it means you can support use cases like you mentioned, like,
entitlements and plans and things like this which are like long-term feature flags so I mean I'm kind of it's kind of a dodgy answer but I would almost say like you should set yourself up so you're okay with it I think you should accept what pains are gonna be there at the beginning yeah and so like there's a lot of things you can do like you know like a let's say you're thinking about it in like a UI context like maybe you have a component based system like react you know you could have like a
Rappler component that manages a lot of like this conditionality as being common and include that in your maybe your routing system or something like that so that It's not necessarily like a sprawl of a bunch of if statements throughout your front end It's more of like a consistent pattern similar thing with like if you're using it for like maybe your rate limiting or like API access use case Do with a middleware so it's more of like a setting that says like check this flag Don't check this flag that you could include in a config
things like this where like you kind of think of feature flags or I do sometimes as being like if statements as a service but in practice like you know anytime in code where you see like lots of statements all over the place that tends to be something you see as like a smell and you probably look for like patterns to try to like make that code more legible. This is no different really you know it's just like the idea of like repairing yourself ahead of time with your architectural choices to think that way with regards to feature flags as well. This is almost a way to like
not do multiple builds of a product in a lot of ways to be like, I'm just going to compile once, have it out there. You know, I can manage all the, the tiering experiments, the slow rollouts, can manage user access control and have it all, you know, like you said, a bunch of nested if statements. But at what point do you say, you know, maybe you're trying to do too much with feature flags? At what point are you like? This is kind of a wonky,
architectural design, like, is there a better way to do this? Yeah, it's a great question. I think you could hit that wall in a few different ways. I think, yeah, for one thing, definitely like this rat's nest of if statements kind of approach, like could be an area where it's like, okay,
The feature flags are maybe fine, but like the way we're using them is making it really hard to test our product, right? Because like if you are doing this mindset of like ship one build and then there's tons of options on it that can be altered at runtime, which is effectively what you're doing when you use a feature flags system that.
runs at runtime is you now have like so many combinations of things that in theory you should be testing before you release a change, right? So taking a more systematic approach like having a dedicated middleware that does the checking or a dedicated like component layer in your router that does the checking like that can sometimes make it easier to kind of like have patterns in your test that like manage your combinatorial complexity of that.
Another way you can hit the wall is maybe just having so many flags and not having an organizational system around them. Some flags might be met for an A.B. test that somebody did six months ago in a sense.
Move to another team and doesn't even care about it anymore. This next flag might be super critical because it's actually gauging access to a feature that's on your pricing page. People pay for it. There are some downsides having different use cases for flags that manifest.
potentially an identical interface in your code base but even that like in our own code base and we of course use schematic like to manage our own flags like we use like kind of like custom hooks around that so that like certain features of feature flags like for example if it's like an entitled man we have a certain like a React hook for that so that we can say like
Okay, let's pull out like usage of this. Are they over a limit different states and things like that? Whereas if it's just an operational flag, we will just use it. So to some extent, like we've chosen in our case to have our call sites look different depending on the use case of the flag, but it is at least like a unified service that's managing all of them.
Which yeah, that has your question implies that can be an advantage and a disadvantage depending on the context You talk about the sort of combinatorial complexity of all these slides. Do you think it's better to?
Have the flags sort of individually own features or is it better to have sort of larger flags? Maybe these larger flags flip other flags down the line. So you're thinking like the kind of like dependent flags like this or actually there's a pattern. People talk about Keystone flags. I think we're essentially it's like dependent flags where you need one flag in order to have another flag on almost like another way of thinking about it would be like targeting inheritance potentially could be like a way of looking at it. That's what you mean.
Yeah, whether you do that, the sort of inheritance or you have a sort of massive super flag that's like, this is just all the enterprise stuff all at once. Yeah, I mean, ideally you would have like a massive super flag for like an atomic unit of value, like just to use the like kind of entitlements and plans.
use case as an example, like you might have on your pricing page, you get 10 widgets per month with this plan and 20 widgets per month for the next plan. Like there might be 90 different places throughout your front and your back end of your different code bases where like you need to check different things related to this widget feature. So you might have potentially like different flags under that heading.
Related to like maybe soft limit hard limits things like this again depends on your tool because some tools will have all these capabilities built into a single flag that has like states to it
but ideally you would have like one feature that's like the feature flag for widgets, right? But if you use a super massive flag, if that's like the technical term for the entire like enterprise plan versus the basic plan, you've kind of robbed yourself of some flexibility that you kind of like, ideally like the purpose of having like a runtime configuration system, like a feature management tool is to have flexibility. So like if you want to say like,
Oh, the enterprise has always exclusively had access to widgets, but now the basic tier is going to get that too, because we're trying to expand our user base. Ideally, you'd be able to do that with the future management tool, not with the code changes to the code base, right? So it does take some planning to decide like, what's my atomic unit of value that I want to gate in this way? But that would be my recommendation. I'm going to get your sense of what you see people using otherwise to manage this in the market. Like I could see, you know,
a big file of hard-coded booleens or whatever.
simplest version, but I'm sure there are complex ways to manage all this. I'll speak kind of broadly about the market for a second, and then I'll just speak to kind of like how we started to get really opinionated about this space. But Ben mentioned this, like our point of view is launched directly in many ways, created this category about sourcing future management 10 plus years ago. It's now become, you know, a relatively like ubiquitous primitive that everybody's familiar with.
What gets sort of interesting and maybe sort of is like what's maybe kind of underappreciated, I think, by a lot of people is the complexity that entitlements can effectively kind of place onto businesses as they grow. And often what that kind of means is like a business achieves product market fit with a simple product, but eventually it evolves to support a variety of good market motions, maybe a variety of building models like seat-based or usage-based.
And eventually, of course, multiple products too. It's just quite a bit of dimensionality to support from a pricing and packaging perspective. And a lot of that complexity often falls on product and engineering to go figure out how to support. The business comes back to product and engineering, and they're like, we want to price and package and sell like this.
What Ben and I and our other co-founders learned is that often what people were doing to try to handle this is they were hard-coded entitlements into the application code and then tying them to a plan ID in the billing system. That was the most common approach.
And of course, like that's really easy to do early on, like it's really quick and dirty. But over time as you go on that journey, that creates a lot of, you might call it technical debt, but you might also call it commercial debt because now the business can't evolve pricing and packaging without asking a team of developers to go figure this out. The other things that we saw people do is they would try to use feature flags, but kind of the core vendors in the space didn't really have sort of a first-class citizen in their products for entitlements.
It's something that you could try to manipulate those tools to handle, but there weren't out-of-the-box integrations with the downstream business systems, notably the billing systems or the CRMs or the ERPs. Then we saw a long tail of other ways that people would handle it. Sometimes people would handle it with GWT tokens trying to do it through their auth provider. Sometimes people would handle it just in their database.
I think the most common was hard coding early on because it was so easy and to do anything different would be to future proof. Why do that? But it ends up being quite problematic for companies if they achieve scale and really want to start evolving pricing and packaging.
When Ben and I started schematic, our kind of observation as well, if you reimagine feature flags to go beyond deployment and beyond rollout and into monetization and metering and packaging, that's actually a really valuable use case for the primitive because then the business can effectively achieve a ton of pricing and packaging agility and remove developers from having to support pricing and billing initiatives, which they often don't like to do.
but it also allows you to be really flexible with like the individual customer, like handle overrides and handle exceptions and deal with customers again without having to bring developers into the play to do that. That can be handled by somebody in success or BizOps or product. So yeah, that's, that's kind of like some of what we learned over the last kind of like two years exploring this is from how people approach this perspective. Yeah, that's interesting. You actually like give the management of the flags to
non-developers, is that right? I think part of what we envision here is, of course, CICD is earned by developers, and there are abstractions on top of flags, and entitlement is an abstraction on top of flag that has a real commercial connotation, because it's effectively like, what should you have in the product based on your contract or your subscription?
That entitlement should be manageable by somebody who owns monetization for pricing and packaging often and i think more and more frequently lives in product. Yeah i would just add on like feature flex is such a familiar interface that kind of like piggybacking on that.
is powerful because then like Finn was explaining like it potentially alleviates like future dev work if you can set up your monetization using a system that uses such a primitive. But the way we kind of think about it schematic is we want to give like appropriate interfaces to every kind of.
persona in this like kind of story right so like to the developer it's like report usage using like kind of like maybe a segment like API because folks are very familiar with like events and then like check access using a flag because that's a very familiar interface and so forth but. From the perspective of like a business user like they should have like inappropriate kind of like a role within the like.
administrative web app that controls the system to where they can manage the plans, but maybe their role isn't necessarily relevant to lower level targeting that pertains more to CICD. Something we kept hearing is pure play, feature management tools. You can really do entitlements with them, and people sometimes do.
but it is considered like a hack in a way, partially just because the interfaces aren't ideal for those business users, the targeting interfaces is usually something like Boolean logic form. It was very consistent in our customer research that agility you'd want to get out of using flags for entitlements wasn't obtained in these contexts because the user type of a customer success or product or whatever just wasn't,
as used to that type of interface, you know, so like you'd want something where it's more like managed plan rather than like manage this like Boolean formula of feature targeting, right? So sorry if that was long winded, but like that's a big part of our vision with schematic is like all of these appropriate interfaces for different user types. Yeah, I think that makes sense. Like you give the non-technical folks something that they understand and you almost, you know, bundle all these feature flags in the end, right? You don't have them turn them on and off individually.
Yeah, and a lot of times they feel strongly about their tools. If you've ever tried to get a sales team to stop using Salesforce, you have a story of failure probably, or with Stripe, billing integration, whatever it might be. Maybe they're not as attached to the tool as the sales team
but they're certainly going to be pretty finicky about integrating in the exact way they want because the money needs to be right or whatever. Are you touching that? Those tools themselves might be the appropriate interface. Maybe they can manage things from within the CRM, close this deal just by virtue of the company being close in a certain place.
correctly through the FM tool, like that's kind of a nice way to picture it to make it kind of, again, appropriate, convenient. However, you want to say it for like that user in the set of users. So do you ultimately aim to automate it through these other interfaces, like just have the click of a button, Salesforce, flip a bunch of flags? That's definitely a way it can work, yes. And like I said, I think my ultimate goal, or our ultimate goal is to have
Basically this work well and in a way that each user type thinks is like correct for them, right? Like if they don't want to go into some other tool to manage their plan that some customers on, they shouldn't have to do that, right? Especially if like probably they have to do that in Salesforce already, right? So like we don't want to make people just do two things when they could do one. We do want the tool to be powerful enough that especially for like an early stage company that hasn't like expanded out on all these different teams that have their own
SaaS contracts and things like you could just start right with schematic and grow from there, and that becomes kind of like your mission control of everything. So we kind of want it to be possible to approach in any direction. To me, what's so interesting about this conversation, like,
We started this conversation all about feature flags and why they're hard, what are patterns, what's the original theory, what's the real reality. We're moving into their extension into pricing and packaging and maybe even our vision a little bit. But I think what was so fascinating to us about the discovery around this product that we're building in the overall space was
that feature flagging could do more for businesses. I mean, it's done an immense amount for production engineering teams and especially developers and especially DevOps, but it could do a lot for businesses. And you could sort of hear that sort of pent up vision for what they could do from a lot of people that had worked on great feature flagging products, like they saw the power.
But there was still sort of like the market just hadn't sort of come to recognize whether it be from like inertia or like not enough tooling that that could exist. And you could build a world in which like the feature flag runs the ICD, but then the business can achieve all this flexibility and it's reconciling between application code and downstream business system. So you're not dealing with
huge homegrown systems that developers are constantly building and maintaining to support the way the business evolves. Like that was, I think to us, the thing that was so interesting is like, you know, we're focused on startups, but we were interviewing growth stage, $20 million, $30 million businesses that were spending millions of dollars in engineering resources every year to just try to manage monetization. And like none of it was core functionality. It's interesting you said earlier that you're basically, you know, piggybacking on this known feature fighting
process. Do you think there are additional sort of features you can piggyback on with the feature flagging? Is there more that they can do?
I'll speak from my vantage and then you lay around. I think there are a couple things that we have heard a lot of pull around. And the first is, if you are powering an entitlement system, you have some really interesting context about a business. You have the usage data at granular feature level,
sort of combined with the financial data of the customer or even the cohort of customers on the plan. That presents really interesting sort of opportunities for obviously business analysis and business intelligence. And there's a lot you can do there. I think the second thing is, I think companies are especially companies that pick dev tools are under an enormous amount of pressure to deliver these like very consumer grade purchasing experiences to the end customer.
I've run growth at a business, like everywhere you go, every time you hear the word growth, it's all about reducing friction for the end buyer. That kind of necessitates that you go build a lot of different purchasing experiences from upgrade paths to downgrade paths to usage meters to customer portals. All of that is effectively components that could be easily dropped and embedded into an application.
But it sort of relies on this underlying entitlements infrastructure so that it's serving up the right information to the right customer at the right time. And so those are maybe kind of two examples of things that we think you could do on top of this, on top of this foundation. Then what would you say? That second one about kind of UI components, I think is like a big, not obvious one just because like doing actual UI is obviously pretty far a field of feature flags themselves, but it was kind of a light bulb moment for us.
The original idea was very much in FM space, but then it was a light bulb moment for us once we realized like, well, we have all the usage data about your features. We know what all your plans and how much you charge for everything. And we know who has access to what. So like we kept hearing and customer interviews like a common pain point from like CTOs was like,
Maybe I have this page on my app that's like my plan page and it sucks to work on. Basically, we can't redo it because it'll break everything and like no one wants to work on it because it sucks to work on. And like half of my customers, when they go to that page, they don't even see the right thing because they're on some legacy plan and we don't have that condition handled.
I felt like that was something that like just nobody wanted to deal with and we were just kind of well positioned to kind of like swallow up all that complexity and just give you like a drop in thing. Because really, again, it's just kind of like outsourcing a bunch of conditional logic to another service.
So it seems like it doesn't really fit into the feature management world, but in our conception of it, it kind of was like a very intuitive lateral move to grow into that. So that's a big one. Yeah, like analytics type use cases that been mentioned. People ask us for that. I tend to kind of.
push back until I feel like we've really nailed some of the more core stuff, just because like, I feel like there's a lot of really good analytics products out there, you know, but we'll probably get there at some point, you know, there's a lot to do within planned
want to be really exceptional at like, okay, I want to try out some new pricing, you know, like, let's just try out like maybe this packaging only on some cohort of customers, like that's the kind of thing that's like really, really hard and annoying to do. And like, might have no results. So those ideas, like those tend to live and die in like the Google Doc and the meeting. And then by the time it comes down to like, let's implement this, everyone kind of realizes like, the results are going to be
We're not sure what they're going to be yet and the cost is going to be high so we're not going to do this experiment. We want to unlock more of that. Finn was mentioning feature flags, getting all that adoption and being a DevOps trend. I've been a big believer in the original DevOps vision and DevOps manifesto and all that stuff. A big part of it that feature flags are a part of is like,
The ability to do continuous delivery, I feel like as a big has been a big boon to just software quality overall, you know, because you can run your tests, you can really isolate like when something went wrong with just this one release, just this one change. And I feel like there's like similar kind of operational things we can offer to businesses like.
Hey, if you want to just tweak this plan by this tiny amount and see what result that has on your business, you can do that now. It has like borderline no cost to you. Or like if you want to make this feature an add-on that you can buy all a cart rather than something included, just change that. You can do that experiment and find out what impact it has on your business. Like those are kind of things that are like really part of the core value prop, but I just think they're going to have so much impact if people start adopting them that I really like to see that play out.
I mean, I can see why people are asking for analytics if you're running all these little experimental packages. Like, did that work? There are a couple of things that, again, like we started our company 18 months ago. We're an early stage company. We're really excited about like the companies that are adopting the product because, to us, they seem like really good companies who are kind of buying into this vision of flags can go do way more. But when I look, when I kind of reflect a little bit back on like the research before we even started the company,
I think there were like three things that we kind of consistently heard, and two of them were from like engineering, and one of them was from like the business at large, and it was interesting to compare the overlap. From engineering, we often heard
root cause of the complexities entitlements, like that is a really hard problem. And then the second thing we would often hear is, especially from startups, I hired Stripe to earn billing and payments. But then as I grew, I had to build all this additional infrastructure around Stripe or between Stripe and my application. And I don't want my team to have to be doing that. I don't want my engineers to have to be on billing initiatives every six months.
So those are kind of the two things we heard from engineering and then on from on the business side, we would hear statements like, to Ben's point, we've stopped talking about pricing because it would require pulling a core team into a sprint to do this. And like, we just have to go ship new features. And the more we heard that, the more we started to say to yourself, what's kind of crazy.
that a really fast moving business that's shipping all this new product and investing in all this new product can actually monetize that product. The product roadmap is so disconnected from the pricing and packaging roadmap in most companies. And that's a real kind of missed opportunity for companies, especially that are like, you know, kind of so much of their kind of core expenses of business is going into application development. Well,
Thank you very much, folks. It's that time of the show where we shout out somebody who came on Stack Overflow and asked a good question, dropped some knowledge, helped out other developers. Today we're shouting out Remy Labue for winning a lifeboat batch. The given answer that reached the score of 20 more on his question that had a score of negative three or less. The question was, how to fix standard logic error? What basic string
M, construct null, not valid errors. If that makes sense to you, go check out the question. It made sense to almost 600,000 other people. I'm Ryan Donovan. I edit the blog here at Stack Overflow. You can find the blog at stackoverflow.blog. And if you want to reach out to me with comments, questions, topics to cover, you can find me on LinkedIn.
I'm Ben Papillon. I'm the CTO at Schematic. If you want to find out more about Schematic, go to schematichq.com. If you're looking for more about me, I can be found on Blue Sky and X.
I'm Finn Glover. I'm the co-founder and CEO of Schematic. I've worked with Ben for eight years. And if this has been interesting to you, I'd encourage you to read his essay on feature flags, theory versus reality. It's ripe with Simpson's images, which makes it all the more enjoyable. And then you can find me on LinkedIn or Twitter X. And we'll include the article in the show notes. Thank you, everybody. And we'll talk to you next time.