DevOps Engineer — Do you know who you want to hire?
Time goes fast, and here we are in the cloud market for over two years. During this time, we had many occasions to talk with DevOps Engineers working for various size organizations, and we had to answer what candidate type would be the best employee for us. We came across many job offers, and some organizations wanted to hire us as well. DevOps Engineer is a very popular position, and it should be clear what skills we can expect and what should be a career path for this role. I wrote here “should be” and in reality, it looks completely different.
What I have to say here is that before you start reading the next parts of my story, it’s very important you understand that I don’t want to insult anyone, make them feel worse or better. In the last few years, we had a quite huge revolution in IT. We have Agile, DevOps, NoOps, Microservices, Clouds, and many others. Everything should help us do software faster and better. Unfortunately, the truth is that all these have technical and also organizational impact, so many people don’t understand why our projects now go slower than it was before the revolutions.
The problem
DevOps is still a buzzword. Now, no one wants to have an Administrator in his organization. It is exactly the same when people want to have only Agile projects in their organizations and no other, but the truth is completely different. There are different types of methodologies and sometimes there is no methodology but a mess. DevOps is not a person responsible for everything and nothing, there are more roles around infrastructure in organizations, and it’s very important to know who should be responsible for what and know what we can expect. Some people can think that it’s very comfortable for System Administrators to start calling them DevOps, but the truth is slightly different. It’s good to have a new popular position without learning anything new, but it’s a short-sighted plan. Long term, it is much better to know your own responsibilities and to have a plan of what would be good to learn or try in the future to have a good position in the market and deliver the best solution for your own organization.
The Definition
To find the right answer to my first question, we have to start from the beginning. I don’t want to copy the definition of DevOps, there are many other sources where you can find it. Important is that DevOps itself is not a role but cultural philosophy. Unfortunately, I’m sure many people who call themself a DevOps or want to hire a DevOps Engineer have never read this definition. As an effect, we have a huge frustration and lack of understanding of why our organization doesn’t work as we expect.
There are many articles about the difference between SysOps and DevOps, but I think that is the best is to read about the differences between Developer and DevOps Engineer. It’s easy to find there that a Developer is someone who is focused on writing business code, where DevOps is someone who is focused on internal development problems. Usually, she/he is focused on automating everything possible and making development as simple and as fast as possible. She/he is rather a Developer with knowledge about development processes automation and tools configuration, definitely a team player. It’s quite far from the typical System Administrator focused on separate tasks, who has no experience dealing with millions of lines of code everyday. The situation isn’t simple, for the complete view you need to look at it from a different perspective.
All organizations want to be Agile now, so when we build any solution, our development is driven by business needs. The question is if the Product Owner is able to understand why we need to have so many technical tasks in the product backlog. It’s possible to dig deeper and find many differences between Agile and DevOps ideologies, but for us, it is enough. I recommend you to find definitions and articles about the role called Service Owner and what is the difference between Product and Service Owners. I’ll write more about this role as well, and in the end, I’ll place useful links where you can find more information about each of the previously mentioned terms and comparisons.
I think this little too long introduction shows us perfectly where we are in our projects today.
Let’s say we have to start a new project, and this project will be conducted using the Scrum methodology. We have a System Administrator in the team — someone called him DevOps, we have Developers who want to use the best and the newest technologies as possible, and we have a Product Owner completely without any technical background.
I think you should be very close to feel the pain and the frustration of our team members right now. To be more specific, I will use a real case. Let’s say that Frontend Developers want to keep all frontend services in a monorepo, and Backend Developers want to have microservices, each in a separate code repository. Frontend guys can explain a lot of the advantages of their idea. The Product Owner completely doesn’t understand what’s going on, but they are experienced team players, so they know how to convince him. They know how to use the language of benefits, the use of keywords: cheaper, faster, reusable. The Product Owner can understand this language, and he wants to have these benefits. Now let’s wear the skin of our Administrator/DevOps. It’s hard for him to understand the benefits, but he thinks he can understand the needs. Monorepo is very simple to add, but making releases from it can be tricky, especially if we have a separate part of our project where we have typical processes with multiple repositories. Using tools he already knows, he starts to decompose this problem and prepares the tasks. Let’s say here he is the only DevOps in the team, so it will take him around six weeks, especially that he has a lot of different needs from other team members. He wants to plan them, so he goes to the Product Owner, and now we have the first fight. “Whaaaaat ?!?!?!?!?. How long will it take?” “I understand that for this time I won’t be able to see any working part of our solution, because it won’t be possible to release anything?” Typical advice from the PO in this situation is: maybe you could talk with other team members about how to decompose these tasks, to make it possible to see some outcome after every sprint? The ideal situation would be if other team members could help our Administrator and make this task faster, but it’s quite possible that our Administrator can use tools that are completely strange for the developers. And now this is the moment when the team members are starting to add some additional tasks, just only to show that they are doing something useful. Of course, it’s not the end of problems. It’s quite possible that the developers have a never-ending conflict with the Administrator, and he won’t like them. For many years administrators were protecting systems against developers, and they were sure that developers never think about security and performance issues. It’s hard to change this point of view and start to think that we are all in one team and together we are responsible for everything, so there shouldn’t be “stupid devs” or any other “stupid” people in any team. As a side effect of this friendship, we can misunderstand the needs of the developers, slower and more expensive processes in our project.
As you can see in this whole situation, there is a lot of frustration and stress for all team members. The Product Owner will have his product slower than he expects. Developers won’t be happy because they will feel that no one understands their needs, and they have to do additional tasks just only to “paint the grass”. Even the best administrator, flexible and dynamic, will have problems to solve problems when he is pressed by developers to learn and deliver solutions faster, and the Product Owner doesn’t understand why everything goes so slow.
How can we fix this situation? Our organization needs to understand that we have non-functional requirements in our project. They don’t deliver direct business outcomes, but they are very important during the development process and future maintenance costs. I mentioned the Service Owner role, probably it is the best idea to have a separate person responsible for the technical perspective of the final product. With him, we can be sure that technical requirements find the right place in our backlog, and communication in our team will be easier. By the way, please try to find job offers for Service Owners, it can surprise you how unpopular this role is.
Ok, so how should we recruit a DevOps Engineer to our organization? First of all, we should answer if we have a DevOps culture? If not, maybe we should stop using the word DevOps. Of course, our world isn’t only black or white, so the next question we should answer is if we expect our “DevOps” to be responsible for the network and detailed systems configuration, or maybe only for automation of these processes? These are slightly different things, and if we need only a strong network and system knowledge, we should stop calling this role DevOps and start to use SysOps or Sysadmin. The next question is what experience our DevOps should have. Probably in the past, they should be a Developer or an Administrator. Both roles have their own advantages. Administrators have great knowledge about security, scalability, network, backups, and things like that, unfortunately they usually don’t have experience in developing automation tasks, code quality checking, and test automation. For all these developers are better. DevOps Engineer should be focused on developing automation and optimization, so who will be better for this than an experienced developer. If our company is in the cloud, the best would be if he has some background in a cloud configuration.
Someone can say now. Wait, but now the developers are very narrowly specialized, and they don’t want to know about all this infrastructure stuff. We can go back to our first question. If we have developers focused only on coding, are we sure that we have a DevOps culture in our organization? Probably our project isn’t even Agile.
I don’t want to finish my story leaving you with the feeling that we don’t need SysOps/Sysadmins/Administrators, my point is completely opposite. They are very important, but please let them use their knowledge because it suddenly didn’t disappear and all this is still very important in our projects. Just don’t try to save money by assigning them tasks that should be performed by someone else.
Useful links:
Role comparisons:
https://blog.udacity.com/2019/07/what-is-the-difference-between-dev-and-devops.html
https://venturi-group.com/devops-and-sysadmin-whats-the-difference/
Definitions:
https://aws.amazon.com/devops/what-is-devops/
https://www.edureka.co/blog/devops-vs-agile-everything-you-need-to-know/
https://uwm.edu/itsm/definitions-and-roles-responsibilities-2/
