CD

CD: Continuous Deployment

fjavierm, Leave a comment

At this point, we are an organization that uses Continuous Integration and Continuous Delivery but, we can go one step farther and start using Continuous Deployment.

Now we work in a continuous delivery environment but we still need to push a button to deploy things into our production environments, not a big deal but still not moving as fast as we want. Deploying to production can be a risky and costly exercise that sometimes requires putting all development on hold. And, we are still dependent on a deployment cycle and the more commits we fit into a release the more risky it is and the more possibilities we have to introduce an error.

Continuos deployment tries to solve all these drawbacks by automatically shipping every change pushed to the main repository to production. We can obtain benefits like:

  • No one is required to drop their work to make a deployment because everything is automated.
  • Releases become smaller and easier to understand.
  • The feedback loop with our customers is faster: new features and improvements go straight to production when they are ready.

STEPS WE NEED TO TAKE

We are already applying continuous integration what means we have plenty of tests, they cover our functionality and guarantee that we are not introducing breaking changes but, are they good enough?

One of the things we need is that the quality of our test suite will determine the level of risk for our release, and our team will need to make automating testing a priority during development. This was important before but now, takes a new level of importance. This means implementing tests for every new feature, as well as adding tests for any regressions discovered after release.

Another thing we need is that fixing a broken build for the main branch should also be first on the list. Why? Because if we don not do that, changes are going to be accumulated and we are going to end up killing the benefits of our new continuous deployment environment.

We need to start using some kind of coverage tools. We can do a rough estimation of our coverage but this does not work. Just try it. If we do the rough estimation and after that use a coverage tool, we will be surprised. It is said that a good goal is to aim for 80% coverage but, there is a big but. Coverage must be meaningful it is not enough writing tests that go through every line of code, the tests need to challenge the code. The tests need to cover business cases, edge conditions and any possible problem we can think of. Code reviews help with this, in addition with helping to transfer knowledge. Our test suite is now the keeper of our production environment, we want to have the best and strongest keeper we can achieve.

Once our code has been deployed to production we need to monitor the situation, we need to check that everything is working as expected and we are serving our customers appropriately. And this monitoring needs to be real-time monitoring. There are some tools out there that can help us with this step. We do not just need to monitor our system is up and running, we need to monitor CPU, memory consumption, average request time… All the parameters we decide we need to keep our systems healthy and to receive alerts if something is off.

No matter how confident we are in our pre-deployment process and our monitoring capabilities, after every deployment we should have some kind of tests running. These test are smoke test, now they earn a lot of importance. We do not need to run anything big, just a  few simple tests loading some static pages and a few more that require to use all production services, micro services, 3rd parties, databases… Our smoke tests should be able to guarantee us that everything is working properly.

At this point, I am sure that some of you (I was doing it) are thinking, what happen with my QA team now? Fair question. Now, the QA team should be working closely with the product manager and the development team to define the risks associated with new improvements. They can help to define cases that need to be tested not just increasing coverage but increasing the quality coverage. They can work in automatization and, they can respond and define properly together with the development team possible bugs found into production.

If any of you readers is a project manager, delivery manager or similar, I am sure by now you are thinking, what about the release notes? There is not space for release notes unless we want to spam our clients. In a big continuous deployment environment we can release code hundreds or thousands of times per day. The advice here is to focus on key announcements for features or enhancements. If you fix a bug that it is particularly affecting one of your customers, just inform this customer. Everything else, you use one of the infinity number of management tools out there to handled changes like JIRA or Trello.

I can be a hard path to move from continuous delivery to continuous deployment. If the effort is worth it or not, it is a decision that the companies and teams need to take. The only advice here is start small and build up your continuous deployment knowledge and experience. If you have a new project, try to apply everything we have mention here, even you can start building your production infrastructure before you even code and push your changes after that. Once you have manage to use continuous deployment in one project, try to apply all the lessons learned to the rest of your projects, the existing ones or even the legacy ones if it is worth it.

CD: Continuous Deployment

CD: Continuous Delivery

fjavierm, 1 Comment

Nowadays, our development teams use Agile methodologies what means that we have accelerate our processes trying to deliver small chunks of functionalities to receive early and real feedback from users to continue iterating our ideas.

Now, we are an organization that follow and use Continuous Integration practices but, we are still missing something. We are not able to receive this feedback if it takes ages between our developers finishing their tasks and the code been deployed in to production where our users can use it. In addition, when a lot of changes or features are delivered at the same time, it makes more difficult to debug bugs on solve possible errors. For a long time, the deployment process has been seen as a risky process that requires a lot of preparation but, this needs to change if we want to be truly Agile.

Here it comes Continuous Delivery (CD). Continuous Delivery is a practice that tries to make tracking and deploying software trivial. The goal is to ship changes to our users early and often, multiple times a day if possible, to help us minimize the risk of releasing, and giving our developers the opportunity to get feedback as soon as possible.

As I have said before, we should have already a Continuous Integration environment to ensure that all changes pushed to the main repository are tested and ready to be deployed. Can it be done without a CI environment? Yes, probably, but more probably we are going to create a machine that it is just going to push our bugs faster to production increasing our risks.

Steps we need to take

Create a continuous delivery pipeline

The continuous delivery pipeline is a list of steps that happen every time our code changes till it finds its way to production. It includes building and testing the application as part of the CI process and extends it with the ability to deploy to and test staging and production environments.

With this we will achieve two things:

  • Our code will be always ready to be deployed to production.
  • Releasing changes will be as simple as clicking a button.

Create a staging environment

All of us, generally, have been writing code in some point in our lives, some of us still do it, some of us have evolve our careers but, something that I am sure all of us remember are these cases of “it worked on my machine…”. Configurations, differences in environments, networks… There are thousands of things that can be different and can go wrong, and probably it will go. No matter how many test we have locally, how many precautions we take, our local conditions are not going to be the same than our production conditions. For this reason we need an intermediate environment, the “staging” environment.

The “staging” environment does not need to support the same scale as our production environment but it should be as close as possible to the production configuration to ensure that the software is tested in similar conditions.

The “staging” environment should allow us to see things before it happen. If something is going to break, it should break in this environment. Using this environment, our release workflow should be similar to:

  1. Developers build and test locally a feature.
  2. Developers push their changes to the main repository where CI tests run automatically against their commits.
  3. If the build is green, the changes are released to the staging environment.
  4. Acceptance tests are run against the staging environments to make sure nothing is broken.
  5. Changes are now ready to be deployed to production.

An additional advantage is that it allows our QA team and product owners to verify the software works as intended before releasing to our users, and without requiring a special deployment or access to a local developer machine.

Automate our deployment

Or, we need to create a “green button” that once it has been pressed, without any other human intervention is going to deploy our code in staging or production.

We can start building some scripts that we run from our development machines and, after that, we can add them to a any CI platform.

There are many ways to deploy software, but there are common rules that maybe we can use as guidance:

  • Version the deployment scripts with our code. That way we will be able to audit changes to our deployment configuration easily if necessary.
  • Do not store passwords in our script. Instead, use environment variables that can be set before launching the deployment script.
  • Use SSH keys when possible to access the deployment servers. They will allow us to connect to our servers without providing a password and will resist a brute force attack.
  • Make sure that any build tools involved in the pipeline does not prompt for user input. Use a non-interactive mode or provide an option to automatically assume yes when installing dependencies.
  • Test it, test it and test it again. Make sure everything deploys as expected, that nothing is missing no matter what kind of changes you are doing.
  • Maybe, it is a good moment to write some smoke test, if you do not have it, to check if your machines are up an running.

Include data structure changes

Let’s face it, when our application changes the code is not the only thing changing. The data structures change too. And, we are not going to have a CD environment if this changes are not added to our automatic deployments.

First, create backups. No matter how good we are, how little is the change, something is going to fail in some point and we want to be able to restore the previous state of the application.

Second, there are multiple tool that can help manage data structure changes as code. Some frameworks bring their own to the table but, if not, we can find tools that fit our technologies and ideas. Just learn them and use them.

All together and the last detail, the CI server

Arriving here, we have set a CI environment with a CI server and we have a “button” to run our deployments, now, why not put everything together?

To do this, we just need to add a manual step in our pipeline to release (press the button) the code to our different environments.

Now, every time our code is merged and ready to production, following business needs, we just need to push the button and release. This give us a great control of our deployments. The only precaution we need to take is not to leave to many commits accumulated.

CD: Continuous Delivery

CI, CD and CD

fjavierm, , , , 1 Comment

When we talk about moder development practices, we often listen some acronyms among we can find CI and CD when we refer the way we build and release software. CI is pretty straightforward and stands for continuous integration. But CD can either mean continuous delivery or continuous deployment. All these practices have things in common but also, they have some significant differences. We are going to explain these similarities and differences.

Continuous integration

In environments where continuous integration is used, developers merge their changes in the main branch as often as the can. These changes are validated by creating a build and running automated tests against the build. Doing this, we avoid the old times painful releases when everything was merged in the last minute.

Continuous integration practice puts a big emphasis on automation testing to keep a healthy build each time the commits are merged in the main branch warning quickly about possible problems.

Continuous delivery

Continuous delivery is the next step towards the release of your changes. This practice make sure you can release to your customers as often and quickly as you want. This means that on top of having automated your testing, you also have automated your release process and you can deploy your application at any point of time by clicking on a button.

With continuous delivery, you can decide to release daily, weekly, fortnightly, or whatever suits your business requirements. However, if you truly want to get the benefits of continuous delivery, you should deploy to production as soon as possible to make sure that you release small batches, that are easy to troubleshoot in case of problems.

Continuous deployment

But, we can go another step farther, and this step is continuous deployment. With this practice, every change that passes all stages of your production pipeline is released to your customers. There is no human intervention (no clicking a button to deploy), and only a fail in test time will prevent a new change to be deployed to production.

Continuous deployment is an excellent way to accelerate the feedback loop with your customers and take pressure off the team as there is not a ‘release day’ anymore. Developers can focus on building software, and they see their work go live minutes after they have finished working on it. Basically, when a developer merges a commit in the main branch, this branch is build, tested and, if everything goes well, deployed to production environments.

Can I use all of them together?

Of course you can, as I have said, each one of them its just a step closer to the production environment. You can set your continuous integration environment, after that, once the team is comfortable, you can add continuous delivery and, finally, continuous deployment can be added to the picture.

PIPELINE
Example of CI, CD and CD pipeline

Is it worth it?

Continuous integration:

What it needs from you:

  • Your team will need to write automated tests for each new feature, improvement or bug fix.
  • You need a continuous integration server that can monitor the main repository and run the tests automatically for every new commits pushed.
  • Developers need to merge their changes as often as possible, at least once a day.

What it gives to you:

  • Less bugs get shipped to production as regressions are captured early by the automated tests.
  • Building the release is easy as all integration issues have been solved early.
  • Less context switching as developers are alerted as soon as they break the build and can work on fixing it before they move to another task.
  • Testing costs are reduced drastically – your CI server can run hundreds of tests in the matter of seconds.
  • Your QA team spend less time testing and can focus on significant improvements to the quality culture.

Continuous delivery

What it needs from you:

  • You need a strong foundation in continuous integration and your test suite needs to cover enough of your codebase.
  • Deployments need to be automated. The trigger is still manual but once a deployment is started there should not be a need for human intervention.
  • Your team will most likely need to embrace feature flags so that incomplete features do not affect customers in production.

What it gives to you:

  • The complexity of deploying software has been taken away. Your team does not have to spend days preparing for a release anymore.
  • You can release more often, thus accelerating the feedback loop with your customers.
  • There is much less pressure on decisions for small changes, hence encouraging iterating faster.

Continuous deployment

What it needs from you:

  • Your testing culture needs to be at its best. The quality of your test suite will determine the quality of your releases.
  • Your documentation process will need to keep up with the pace of deployments.
  • Feature flags become an inherent part of the process of releasing significant changes to make sure you can coordinate with other departments (Support, Marketing, PR…).

What it gives to you:

  • You can develop faster as there is no need to pause development for releases. Deployments pipelines are triggered automatically for every change.
  • Releases are less risky and easier to fix in case of problem as you deploy small batches of changes.
  • Customers see a continuous stream of improvements, and quality increases every day, instead of every month, quarter or year.

As said before, you can adopt continuous integration, continuous delivery and continuous deployment. How you do it depends on your needs and your situation. If you are just starting a project and you do not have customers yet you can go for it and implement the three of them and just iterate on them at the same time you iterate on your project and your needs grow. If you have already a project in production you can just go step by step and adopting the practices first in your staging environments.

CI, CD and CD