What does it mean to have a good configuration for your infrastructure?
Let’s imagine that you are a system administrator and you are around the year 2000. You have only one server. How does your configuration look? It’s quite possible that you don’t have any reusable configuration and everything is in your head. You are the best, you know how to set up and keep some type of Linux system alive and it’s enough for you and for people who pay for your work. How does a situation look when you have to set up a new server? Probably you start from scratch. Your knowledge is evolving, you read a lot, everything is under your control.
Let’s make a small jump forward. Imagine we have 10 servers that we have to set up. They look quite similar but there are some small differences. Hardware can be different, basic services are the same but a lot of details may vary. You are a good specialist, so you know script languages like Bash, Perl, Python. You are a smart guy so it’s nothing strange in this that you start building some configuration structure. Some scripts can be included in the others and there are elements that can be reused with different parameters. Now it’s starting to look quite serious. Where you keep your configuration, probably in a safe place in your personal computer file system, additionally you make copies to be sure that you don’t lose your work effects. Now answer me one serious question. Where did you keep your passwords? Around the year 2000, there weren’t so many fancy tools to store the passwords. If you are a brave man and you like the risk, you could keep them in the scripts, or if you were not brave but a reasonable person, you kept it somewhere safe in a separate place. Maybe this was slowing down your daily work, but let’s be serious. We need to keep even the lowest security level.
Everything is changing, so from time to time, you have to update your scripts, you have to remember about updating all copies, make some changelog. The next serious question we have is, what is the difference between my scripts and configuration version on my servers. Of course, I’m a professional, so I know what exactly I’m doing and where. I note all maintenance events. It takes time but keeps everything under control.
Now let’s forget about the year 2000. I hope it’s only history and no one acts like this today. In the last 20 years, a lot of things have changed. Computers are more and more important every day. We need more computing power, we need more and more complicated services, and there can be some dependencies between them. We have more passwords, more security rules, we have to work fast and as transparently as possible.
In the meantime, a lot of tools came out and IaC (Infrastructure as Code) started to be popular. Now we can be much more professional, and your life can be simpler. We know tools like Chef, Puppet, Ansible, and many others that can be very helpful with all these problems I tried to highlight in the first part of this text. This could be the end of my story but life is dynamic and our needs are more and more complicated every day. First of all, now we have clouds and our focus moved from detailed system configuration to environment configuration and new projects stopped being monolithic and stateful. Now we have a bunch of small services with different integrations, scaled separately and they are written in different technologies. Fortunately, we have containerization now, so we don’t have to worry about how to run some completely strange new technologies. We are mature enough to force containerization in our organization and now we run Docker containers everywhere. Our services are stateless which makes our infrastructure even more powerful. Now we don’t have to think about complicated recovery and in many cases we can use even spot instances.
From a configuration perspective, it’s starting to look like a nightmare. We have more and more tasks, and if our company is reasonable there is more than one specialist responsible for the infrastructure. Now we have to be team players. If we have a team, we have to answer the next questions. Do we have a consistent idea for our configuration? Do all team members make everything the same? Do we have some procedures? This is now the real-world look. To be professional these days, we have to be a different person. Now we can’t act as lonely ninjas. We are team players with all pros and cons of this.
In the first moment, we can think that all consequences from the growing team are negative, but it’s of course not true. It’s completely the opposite. In the last 20 years, software engineering, development tools, and project management methodologies became more mature. Now we know much better how to make good and stable software in a huge group of people and get benefits from it. It would be quite a long text to write about all consequences, so I focus only on the most crucial. Let’s forget about thinking that configuration is a bunch of scripts even with some simple tools that can control versioning and applying. What would happen if we treat our configuration as a regular software project with all the benefits that we can have from it. So what exactly can we get from it?
First of all in the modern software project we have a completely different knowledge distribution than it was 20 years ago. Now code review is the standard part of the development process. What can it change? Now you don’t write code without thinking and keep fingers crossed that it will work and no one will see how ugly code we made. Now you have the second pair of eyes, and what you write should be transparent enough to understand by someone else. If you had some experience with code review you know how beneficial it can be in a well-prepared development process.
The second thing is version control. Git is the most popular source code repository in the world. In the software project we know how to conduct development in many branches, make releases, tags, and things like that. This knowledge is more complicated than keeping a few files on the personal computer, but the benefits are humongous. We can use all this knowledge to move in time with our configuration project and if we have some technical problems with the tools, it’s always easy to find an answer because it’s a world standard.
The last thing, but the most important for me is testing. Probably I don’t have to write here how tests are important in the modern software project. In old-fashioned script-based configuration sometimes it’s impossible to write some fair automated tests. In a software project, we usually have many more tools that can help us, but if we forget about simple scripting languages or configuration tools with limited functionalities, and start to use a real programming language with all its benefits, our configuration will jump to the next level. If you add to your project power of unit and integration tests, finally we can say that we have something that is stable and the maintenance is more predictable.
Imagine, you have a stable configuration where the writing approach is standardized, all changes are versioned, all dependent libraries and tools are managed by some standard dependency management tool configured in your project, where you can run tests before you apply your configuration. It’s a completely different situation.
Sounds great, but as usual it’s not so simple. Today it’s been around 2 years since we started writing DevOpsBox, this is our tool where we treat configuration as a code. What do we know after this time? First of all, the DevOps tools market is very young, we bet on the best open source technologies, but during this time we have to replace some of them. This is a rather common thing in software development but maybe not in configuration. The next thing is that configuration as a project is quite complicated. For software developers, it’s still a small project. For DevOps or SysOps Engineers it’s probably something huge, and it can be hard to understand. Also, if we are talking about a modern programming language as a fundamental element for making a solution like this, we have to understand that usual SysOps don’t have experience in software development and they need a lot of time to understand the code.
This is a huge change. Now we expect our former system administrator will be a good software developer. For some people this transformation is impossible, for others, it’s a huge jump forward. But do we really need it? The answer is YES. Now our expectations for the configuration projects are much higher than10 years ago. It’s a very expensive and time-intensive project (https://www.devopsbox.io/profit-calculator). We want to have something that will be future proof predictable investment. Without the power of programming languages, it’s hard to meet such demanding requirements.
P.S.
If 10 years ago I had told some backend developer that we would write completely different separate frontend applications, he would have told me I was crazy. Things are changing very fast so it’s better to flow with them than try to fight against.
