Taming Irreversibility with Feature Flags (in Python)

Hugo Bessa
May 21, 2018
<p>Feature Flags are a very simple technique to make features of your application quickly toggleable. The way it works is, everytime we change some behavior in our software, a logical branch is created and this new behavior is only accessible if some specific configuration variable is set or, in certain cases, if the application context respects some rules.</p><p>There are many ways of implementing them, the simplest one would be defining a place to store some configuration variables and retrieve their values in your application, changing the flow only if the value is true. For instance, in a simple Python application, you can store the configurations as Environment Variables, and use them as the condition for an <code>if</code> clause.</p><pre><code># Environment TOGGLE_FEATURE_ONE=1 </code></pre><pre><code class="language-python">import os def is_feature_active(feature_name): env_var_value = os.environ.get(upper('toggle_'+feature_name)) return env_var_value == '1' def my_function(): if is_feature_active('feature_one'): do_something() else: do_something_else() </code></pre><p>Yes, feature flags are that simple to use, but it doesn’t mean you can’t do big stuff with it. If well used, they can bring a lot of benefits to your development flow. We'll discuss in this article what are these benefits, which precautions should we take when using feature flags, and what are the best practices to maintain a healthy codebase and increase your development team’s peace of mind.</p><p>This technique has already been very well discussed in great articles like <a href="https://martinfowler.com/articles/feature-toggles.html">this one</a> from Martin Fowler (one of the signers in the Agile Manifest) and <a href="https://www.facebook.com/notes/kent-beck/taming-complexity-with-reversibility/1000330413333156/">this other one</a> written by Kent Beck (former technical coach on Facebook). Here, I’m going to approach the topic in a very practical way, with tips we've learned while using feature flags in Vinta's projects.</p><hr><h2 id="the-great-benefits-we-found-by-using-feature-flags">The great benefits we found by using feature flags</h2><h4 id="1-improving-team-s-response-time-to-bugs"><strong>1. Improving team’s response time to bugs</strong></h4><p>The most obvious benefit is to be able to turn features on and off in production on the fly. This is great when you're launching new features. If users start experiencing some bug triggered by that feature, you can quickly turn it off and no more users are going to be affected. Then you can thoroughly fix the issue and reactivate the feature later, avoiding new complaints, and, even better, to avoid having your database getting more and more corrupted.</p><h4 id="2-making-possible-to-sync-code-more-frequently"><strong>2. Making possible to sync code more frequently</strong></h4><p>Also, you can merge your code changes more often, even when the feature you were development is only partially complete. Since the old code will still be there with the flag off, the new feature wouldn’t impact other developers. But the greatest part is that they would <strong>have a better knowledge of where the code is going</strong> and already be doing their tasks adapted to the feature that is still incomplete.</p><p>By doing that, you’ll get <strong>less complex conflicts</strong> if there is more than one developer working on the same part of the code because it would be easier to always have a more up to date version of the code.</p><h4 id="3-having-a-more-fluid-feature-launching-flow"><strong>3. Having a more fluid feature launching flow</strong></h4><p>Another advantage of using feature flags happens when you’re testing features in <strong>sandbox environments</strong>. If your team has scarce resources and cannot have many sandboxes to test each staging feature in a dedicated server, you probably have more than one feature in the same staging git branch (or other source control manager’s). If one of these features is reproved on QA process/black box tests, you have to revert the broken feature or fix the issue before deploying to production. With feature flags, you can deploy the broken feature and keep it inactive until the fix is deployed as well.</p><p>This helps to create a more fluid development flow, without many reverts and rollbacks. This is good because the tools you use to perform the reverts/rollbacks can be tricky sometimes, especially if your feature involves data migrations.</p><p>In all Vinta’s projects, we have a staging environment where we validate if things are really working. The server simulates an environment that looks a lot like production, so we can find bugs we’d only in production. We have a git branch to sync with staging, and when things are tested in there, this branch is merged to production branch.</p><p>When our projects teams started growing, we had a lot of features being merged to staging at the same time. That led to the problem we mentioned in this section: buggy features locking the whole deploy. We solved this by using feature flags because now we can turn the buggy features off, and they won’t cause any problem when deployed.</p><h4 id="4-validate-your-features-with-users-the-right-way"><strong>4. Validate your features with users the right way</strong></h4><p>In this article’s intro, it was mentioned that feature flags can use the application context instead of just the configuration variables. This is very useful, for instance, to decide if a feature is active or not based on the authenticated user. This way, you can easily create user groups and make A/B tests and soft launches (enable features to your users gradually, instead of launching for everybody at once), making it possible to better evaluate your feature before releasing it definitely.</p><p>The partial feature release is also great to diminish the effect of bugs since new bugs are going to affect only a small part of your user base before the final release.</p><hr><h2 id="using-feature-flags-in-the-real-world">Using feature flags in the real world</h2><p>As was said in the intro, the concept behind feature flags is simple, but there are a lot of ways to implement. Here we’ll talk about some of these implementations, their benefits and the concerns related to them. The main challenges about feature flags are these three: deciding which flag's storage method fits better with your application, how to always maintain the application consistent in any combination of the flags' states and maintaining the code quality/readability. Here we'll discuss all of these in detail.</p><h3 id="storing-the-flags">Storing the flags</h3><p>The storage method adds or reduces the complexity of your flags usage depending on your needs. We'll now talk about the four more relevant ways of storing flags.</p><h4 id="configuration-files"><strong>Configuration files</strong></h4><p>This is only a good option if we're talking about an offline application. In this method, every time a flag needs to be changed, you have to change the application files, which usually means software updates or even a new deploy. But the good part is that it’s very simple to implement since there are many tools to read configuration files in any language/framework you might be using.</p><h4 id="environment-variables"><strong>Environment Variables</strong></h4><p>Earlier, we’ve explained the simplest way to store the flags, which are environment variables, but they are not always the best option.</p><p>Environment variables are great when restarting the app is not a concern, because everytime you change the values you'll have to retrieve them again for the whole application, which usually means to restart the servers. For instance, in a web application that can’t afford being restarted, it would be better to store these flags in the database, in an in-memory data structure store like <a href="https://redis.io/">Redis</a>, or in a third-party service.</p><p>Another concern of using environment variables is that you have to save your flags as strings. If you want to add more information than just a flag (boolean), like the user group names that have access to the feature you'll have to handle serialization/deserialization.</p><p>One last concern regarding environment variables is that they’re related to the environment your application is running on. If your application is distributed or has instances in more than one environment, you have to make sure the values are consistent between them.</p><h4 id="database"><strong>Database</strong></h4><p>Storing flags on a remote database is a safer option for web and distributed applications because the consistency between the flags values between instances will be maintained. At the same time, restarts are not a concern, since most DBMS (database management systems) already handle this for you.</p><p>One thing you have to be aware of is that, since your application doesn't restart when you toggle flags, some requests and async operations may be running in the exact moment the flags' values are updated. This may lead to inconsistent application states. So, every time you're changing values, make sure you're handling all possible states and maybe only toggle big features when the access rate is lower, or when your async tasks/cron jobs aren't running. We're going to explore this problem in more details in the Asynchronous tasks and pending requests section.</p><p>The choice of which database to use may impact in the time your application takes to retrieve the flags' values. For instance, if you use Redis, which stores values in-memory, your application will probably retrieve flags' values faster than if you use a database platform like PostgreSQL. Also, Redis has a pub/sub system that can notify your application, when a flag changes. This way you’d not have to fetch flags’ values all the time.</p><h4 id="third-party-sdk"><strong>Third party SDK</strong></h4><p>Furthermore, there are some services for managing feature flags like <a href="https://launchdarkly.com/">LaunchDarkly</a>. The features they provide vary from service to service, but one of the best things is having a dashboard for managing your flags, making the activation/deactivation of features more accessible, even for a non-technical person.</p><h3 id="maintaining-application-consistency-in-active-and-inactive-states">Maintaining application consistency in active and inactive states</h3><p>When you have two possible states for each feature (on and off), you have to make sure both of them work as expected. In some cases, you even need to ensure that intermediary states also work (like requests that had a flag value when they started running and a different one at the end).</p><h4 id="data-migrations-and-backward-compatibility"><strong>Data migrations and backward compatibility</strong></h4><p>When you're developing a new feature, it’s very common that you need to add columns to existing tables or even create new tables. If there are only new tables in the feature, the work is easier, because active and inactive states don't share the same table. In this case, your only concern is to maintain data consistency between the new tables and the old ones for the case where you need to turn the feature off. You also have to make sure you don't delete data/tables you may need in the inactive state of the feature.</p><p>In case the feature’s active and inactive states share tables, you have to create workarounds to make sure the data is being stored correctly for both states. Sometimes this means creating redundancy by having two or more columns meaning the same data but in a different format.</p><h4 id="asynchronous-tasks-and-pending-requests"><strong>Asynchronous tasks and pending requests</strong></h4><p>When you toggle a feature on or off, many things may be happening in the background of your application. In some features, you can just ignore this and update flags without worrying about broken application/database states. But sometimes a simple feature flag toggle with the wrong timing may cause dangerous results to your application.</p><p>Imagine you are running a task that takes a long time in the background, fetching data from an external API that only sends you the same data once. This is very common for notifications systems: when you retrieve a notification, the message is destroyed from the original server queue/database. Let's suppose we've developed this new feature that saves every notification in the application database and emails the user for each one. This feature has a feature flag.</p><p>But suddenly, in the middle of the notifications retrieval, you notice your application is sending every notification to every user, not only to the ones that should receive them. This is a huge privacy problem, so you need to desperately turn the feature off to stop sending notification emails.</p><p>But then, with the feature off, you fetched some notifications, but did not store them in the database nor sent the emails. The notifications that were retrieved cannot be recovered anymore because they were deleted from the original source. This loss is irreversible.</p><p>In a case like this, you have to treat data very carefully. Even if the store and email feature are disabled, you should still store the data you retrieve in the database, even if you don't show this data anywhere, because when you fix the bug and reactivate the feature, no message will have been lost.</p><p>The same case could happen if the notifications retrieval was made in an HTTP request, so you also have to be careful about which requests are being made when toggling features.</p><p>One way to handle this is to consider every edge case on every affected part of the feature and run fix scripts to adjust database/application state after you toggle a feature on or off.</p><p>Another way to handle this in a more generic fashion is comparing the time the flag was activated and the time the request/async task started before considering the new value.</p><h4 id="automatic-tests"><strong>Automatic tests</strong></h4><p>When you're developing or updating a feature, you can usually delete some old tests that got outdated and do not describe the new application behavior correctly anymore. With feature flags, these tests still need to work, since the old behavior is still accessible when the feature is turned off. But how to differentiate the feature’s active state from the inactive state?</p><p>If you're using feature flags with EnvVars or config files, you can simulate the flags’ states by using mocks. Python has a built-in module for mocks and you can easily override some global configurations and test both states of each flag.</p><p>If you're using the database to store flags, you can change the value of each flag before running your usual tests so the flag value matches the behavior you're testing.</p><h3 id="managing-code-quality-and-the-size-of-the-test-suite">Managing code quality and the size of the test suite</h3><p>When you look at your code with only one feature flag, it's very readable. But imagine when you have dozens of features, each one with its own flag, or even worse: nested features (consequently, nested flags). The code is going to look pretty bad.</p><pre><code class="language-python">if is_flag_active('my-first-feature'): do_something() if is_flag_active('my-second-feature'): do_second_something() if is_flag_active('my-third-feature'): do_third_something() if is_flag_active('my-forth-feature'): do_forth_something() else: do_forth_something_else() else: do_second_something_else() </code></pre><figure class="kg-card kg-image-card kg-card-hascaption"><img src="https://s3.amazonaws.com/vinta-cms/media/bp-feature-flags-identation-hadouken.jpg" class="kg-image" alt="Indentation Hadouken"><figcaption>Almost something like this</figcaption></figure><p>To avoid this, there's one very simple step to follow: <strong>remove old flags</strong> from the code as soon as they're validated. Or even better, develop your new features thinking you will have to remove the flag later (leaving only the active state and having to back to the old code in case you want to go back). Here we'll list some tips to make flags easier to read and to remove later.</p><h4 id="1-make-your-features-modular"><strong>1. Make your features modular</strong></h4><p>The goal here is to make it so your features depend on your current app as little as possible. By doing this, you have few and very clear contact points where the old behavior and the new meet. This way it'll be easier to understand the logic since you don't have to know the whole application to understand what each part of the code is for. It'll also make it easier to find which parts of the code you'll have to modify in order to remove the flag once the feature is validated (or not).</p><p>Another reason to write modular features is that you may avoid bugs occasioned by an accidental partial removal of the flag's code since there'll be fewer and more separate lines of code to be removed.</p><p>A great way to start building a more modular application is by using proper design patterns. <a href="https://sourcemaking.com/design_patterns/strategy">Strategy</a>, <a href="https://sourcemaking.com/design_patterns/chain_of_responsibility">chain of responsibility</a> are good examples of design patterns that can help you maintain your code more modular.</p><h4 id="2-avoid-long-logical-branches-in-your-code"><strong>2. Avoid long logical branches in your code</strong></h4><p>Use helper functions or classes when you have to branch your code according to the active and inactive state of a feature. If you write both states' logic in the same place, your code may become denser and less readable, so the best option here is not having logic directly in the if/else statements' scope (the ones that check the flag state).</p><pre><code class="language-python"># bad if is_feature_active(‘my-feature’): do_something() do_something_else() do_something_more() do_log_something() else: do_some_other_thing() # … # good def do_feature_one_stuff(): do_something() do_something_else() do_something_more() do_log_something() # ... def do_feature_one_inactive_stuff(): do_some_other_thing() #... if is_feature_active(‘my-feature’): do_feature_one_stuff() else: do_feature_one_inactive_stuff() </code></pre><h4 id="3-write-different-unit-tests-for-the-active-and-inactive-state-of-features"><strong>3. Write different unit tests for the active and inactive state of features</strong></h4><p>If you don't do this, it'll be harder to separate which tests are related to which state when you're removing the flag later. Removing a flag must be as easy as possible since it's a very uninteresting task that doesn't aggregate anything to the product value (it's only a technical debt).</p><pre><code class="language-python">def test_my_function_do_something_with_flag_one_active(): set_flag_active(‘flag_one’) my_function() assert do_something def test_my_function_do_something_else_with_flag_one_inactive(): set_flag_inactive(‘flag_one’) my_function() assert do_something_else </code></pre><h4 id="4-schedule-the-flag-s-removal-and-stick-to-the-schedule-"><strong>4. Schedule the flag's removal (and stick to the schedule)</strong></h4><p>It's important to have in mind when any feature flag is ready to be removed. A great way to make sure it's validated or not without much work is to give them some framed time to validate. This way you can create reminders to tell you when to remove the flags and plan the removals on your sprints.</p><p>Remember: removing old flags is really important. Otherwise, your code readability will quickly become poor and you’ll have a lot of accumulated work to improve it.</p><h4 id="5-measure-your-feature-s-success"><strong>5. Measure your feature’s success</strong></h4><p>There are a lot of tools and services to store and see analytics of how your application is being used in more or fewer details, like <a href="https://analytics.google.com/">Google Analytics</a>, <a href="https://web.facebook.com/business/learn/facebook-ads-pixel">Facebook Pixel</a>, <a href="https://mixpanel.com/">Mixpanel</a> and <a href="https://www.hotjar.com">Hotjar</a>. Measuring usage is very important when you're using feature flags, so you can properly validate if your new features are really improving user experience and metrics, and even find bugs and unhandled edge cases.</p><hr><h2 id="don-t-reinvent-the-wheel-use-the-right-tools">Don't reinvent the wheel, use the right tools</h2><p>There are a lot of python packages that can help you implement feature flags. Each one has its specific implementation strategies, so you’ll have to choose which one is the best fit for your project.</p><h3 id="gutter-former-gargoyle-">Gutter (former Gargoyle)</h3><p><a href="https://github.com/disqus/gutter">Github link</a></p><p>This is a library maintained by <a href="https://disqus.com">Disqus</a>. It's great for creating really complex context checks to decide if a flag is active or not.</p><p>It is really flexible, but it's also a little bit hard to use (there are a lot of concepts to understand). If your goal is to have the maximum flexibility, this is a good pick. If you're prioritizing simplicity, this may be too much.</p><h3 id="feature-ramp">Feature Ramp</h3><p><a href="https://github.com/venmo/feature_ramp">Github link</a></p><p>This is a very simple python API for creating and fetching flags using a Redis backend. It doesn't support context checking, only a boolean value for each feature.</p><h3 id="framework-specific-tools">Framework specific tools</h3><p>There are many tools for specific frameworks. The ones we’ve tested here were for <a href="https://www.djangoproject.com/">Django</a> and <a href="http://flask.pocoo.org/">Flask</a>, and these are our recommendations.</p><h4 id="django-waffle"><strong>Django Waffle</strong></h4><p><a href="https://github.com/jsocol/django-waffle/">Github link</a></p><p>Very flexible database-based feature flags API. It has a lot of helper functions to abstract its features, as well as some decorators and mixins to help you build your views (depending on whether you're using function views or <a href="https://docs.djangoproject.com/en/2.0/topics/class-based-views/">class-based views</a>).</p><h4 id="flask-featureflags"><strong>Flask FeatureFlags</strong></h4><p><a href="https://github.com/trustrachel/Flask-FeatureFlags">Github link</a></p><p>This package has a very simple but customizable API. By default, it stores the flags in a config file, but it gives you the options of using custom backends and storing the flags wherever you want.</p><hr><h2 id="conclusion">Conclusion</h2><p>We've had a great experience with feature flags here at Vinta so far. We took advantage of all the benefits listed in this article, but the path has not always been easy.</p><p>We’ve learned and continue to learn a lot about how to have a healthy development flow with feature flags, preserving code quality and safety for the developers.</p><p>These were some of the tips we found to be useful in our experience. If you have something else worth mentioning, please leave a comment! We'd love to hear your input!</p><p>Special thanks to the people that helped me writing this post: <a href="https://www.vinta.com.br/blog/author/luca/">@Luca Bezerra</a>, <a href="https://www.vinta.com.br/blog/author/laisvarejao/">@Lais Varejão</a>, @Igor Calabria, <a href="http://www.depedro.me">@Pedro Torres</a> and <a href="https://twitter.com/dinizz">@Thiago Diniz</a>.</p>