There are a lot of parts of ops that handle things that aren't easily scriptable. What do usage patterns look like so you can better manage when to bring servers up/down to handle normal usage flow and have servers warmed up when they're needed? Are you using the best build options for your VMs? Are you using the best VMs for your application? Are you at the best datacenter for your application? Even with certain build options having better automation tools than before, there are still a lot of places where it the world is far from a few scripts.
With the rise of devops system engineering knowledge is being lost.
I often see developers trying to replace it with bloated, complex piles of hipster software.
While a devops team is attempting to build a toy Google-like infrastructure on 50 nodes for weeks, another company is deploying code just fine using some 15 years old setup with netboot + disk imaging system + OS packages.
I don't think this is true at all. Which is not to defend or even endorse the Kubernetes wank, "devops" is also "hey, we have codified what our systems look like in code (with Chef/CloudFormation/etc.)." Which is a hell of a step up from what existed prior. That I occasionally have to then go debug why AWS's Xen network drivers are spitting the bit or why this application's disk access patterns are making EBS sad certainly keeps me very firmly in the details of the systems in question, too.
(As an aside: I mentally replace the words "hipster software" whenever I see it with "things I don't understand" and the sentences never seem to change their meaning.)
> Which is a hell of a step up from what existed prior.
Not accurate. A lot of new-stuff is just old-stuff we've had forever with a new name and branding. Worse, a lot of "new" stuff is too low quality to be reliable and/or gonna die/be obsoleted very soon.
Old shops who took development seriously have developed internal tooling, customized to their internal needs. It can be surprising how well it can hold the comparison against new hype software.
But they don't both suck. Newer tools reflect the needs of newer problems. Like, I can point to exactly what I get from my Chef/Cfer stack--I can wrangle hundreds of concurrent machines that need to update to a policy while also getting, in a standardized and predictable format, a full itemization of all changes applied to all systems. I can leverage the community, too, to help me bootstrap new features and functionality because we're speaking a shared language.
They might be new tools to you, and being skeptical of new stuff is fine (even healthy), but there's a point at which skepticism turns into Ludditism and it's well before the point where one starts to bloviate about "hipster software".
It's funny you'd quote Chef (or puppet), because they've been replaced by Ansible/Salt.
I spend a lot of time evaluating new tools, possibly more so than anyone else in this tread. There are very few novelties which solve a problem notably better than what existed before AND are reliable enough for serious usage AND don't come with so many drawbacks that it nullify their interests.
Chef hasn't been replaced by Ansible or Salt. Each tool has a different value prop that makes sense in different contexts. I might use Ansible in some environments where I wouldn't use Chef--there aren't a ton of these environments, IME, but I've chosen to use Ansible in the past despite it not being my favorite for reasons like this. I would use Chef in other contexts where Ansible's ecosystem or tooling is lacking--I find that chef-zero and berkshelf are a better way to self-bootstrap on AWS when using CloudFormation, for example--or where I desire the greater flexibility and expressiveness of a Ruby-based DSL.
The black-and-white thing you're painting is, if I'm gonna be frank...a little messed up. I mean this in the best of ways: please enhance your chill. It's not that bad out here.
That question doesn't make any sense at all. That's exactly the same as saying: with the rise of do it yourself home repair, how often are plumbers different from carpenters?
The question is great. DevOps is a field that I really don't want to touch as a developer. I build frontend and backend as well, design databases and focus on UX and configure a LAMP stack, but that's where sysadmin ends for me.
What I don't want to mess with is the "DevOps" tools, like Docker and rest of cloud orchestration stuff. It's a very complex field and I couldn't care less. If I work on a project like that there's always somebody who can do that kind of setup.
DevOps, at its core, is just treating your configuration files and artifacts (IP/DNS, firewall rules, mail config, service and web app artifacts, libs and other stock software, SQL/NoSQL deployment descriptors, queue configs, users/roles/certificates) as development artifacts, and have an automated, reproducible deployment procedure. Going to a box and manually configure something in an ad-hoc fashion using fancy GUIs is considered a no-go. Rather, you're supposed to craft/test your configuration artifacts and then check it in to eg. git. If anything, this should be a workflow familiar to developers, hence "DevOps".
Docker and other Docker-like containers like Rocket don't have anything to do with DevOps per se, but they facilitate easier automated deployment of said configuration artifacts. Technically, Docker & Co. are just remedies to classical DLL hell situations, and have been used mostly for development and software evaluation purposes. Though they technically don't provide much more isolation than simple chroot-jails, increasingly they're also used in place of VMs because of density reasons, eg. because you can run a whole bunch of services on a single machine with less footprint than VMs (on Linux, that is).
The frenzy with Docker and the other cloud orchestration stuff, as you say, is IMHO mostly because of Sillycon startups with insane amounts of venture capital buying their way into people's minds.
Edit: ok, the last remark was a bit snarky; sure there's a need of mass-digestible tool for collaboratively editing DevOps artifacts and deployment plans, but I don't know any I like; feel free to point me to one (preferably without JSON and/or yaml configuration files)
A plumber who is also a carpenter is usually known as a "handyman". There's a reason apartment complexes big enough to have maintenance staff tend to hire that sort of person over a dedicated specialist.
If you're a large business you should be automating your sysadmins with devops/SRE techniques. If you're a small business you should be outsourcing that work to companies that have automated it for you.
That's why cloud providers and associated services have seen enormous growth.
I dunno, I think the question makes perfect sense. I am a software developer. I very often write Chef and a metric buttload of glue code in Ruby and Bash and occasionally PowerShell. That I can also write a Rails app and a React-Native app and a Dropwizard app (all things that I've touched this week) in addition to being able to set up Kafka in a fault-tolerant and scalable way is because it's all code.
Chef etc. aren't "home repair." (Well, maybe Docker, and it shows.) They're a way of expressing subject matter knowledge. Still gotta be a developer to do that expressing in a way that isn't going to kill you, either directly or because you write code that your fellows burn you at the stake for.
There's reasons we have these classes of people with certain labels. Until the 18th century, buildings were designed and built by "artistan craftsmen", people who had no formal training or education in building things (or in any capacity) but said, sure, I can make you a house. Then it would fall down and kill a family of five. All around the world today, architects and engineers are required to be licensed and prove they know what they're talking about so this doesn't happen. As a result of this process, the whole profession developed and advanced the state of how we build things.
We're lucky that we work in an industry where our work is most likely not going to kill people. We don't have to be licensed to make a claim of who we are or what we can do. But it would be very disingenuous for me to call myself a professional software engineer just because i've written hundreds of thousands of lines of code, and for you to call yourself a sysadmin because you've set up some software. The mark of a professional is understanding at a deep level all of the hidden detail that goes into the work, and if we hold them up to a standard, the titles will reflect that.
So, how often are software developers different from sysadmins? They're always different. They're completely different practices. As a person, you can indeed have two separate successful careers and become both, but that doesn't have anything to do with devops, which is merely the practice of dev and ops collaborating together.
I've read your post two or three times now, and I straight-up don't get what you're driving at. Yes, there are levels of subject matter expertise that you need to know--or at least be conversant enough with to deep-dive when necessary--for system administration. That's also the case for database programming or bioinformatics or graphics programming or whatever. But it is still expressed through the writing of code, in 2016--and that code, largely today and with increasing focus going forward, needs to be well-structured, testable, so on and so forth. It is still the development of software. It's a specialization within the same field. It may not have been twenty years ago; if it isn't generally today, it will be before I retire, and probably a lot sooner than that. And to that end, the question that was originally asked makes a lot of sense, to which the answer is "a sysadmin is-or-had-better-be a software developer; a software developer may or may not be a sysadmin."
"Set up some software," though? The condescension is both unnecessary and built upon some pretty shaky projections on your part. Can we not dance that dance, please? (ETA: Not least because I think your posts, for the most part, are really good and I have a lot of respect for you, especially around security topics.)
I would be horrified if you had to be a software developer to be a sysadmin. It would be like requiring an auto mechanic to be an engineer, or a cook to be a chemist or something.
In actual fact, the complexity of software design is antithetical to the job of system administration. If at all possible you should use the least code possible to do a given job, and rely on the reuse of tools to serve various functions. This is not coding, just like auto repair is not engineering. If you're writing code you are not adminning, you are developing.
There is a time and a place for software development in Ops, but it should be based around projects led by teams to serve specific functions that existing tools won't solve, again similar to auto repair because some tools are needed simply to repair things more efficiently, but then you use those tools and don't keep engineering new ones. The distinction is small but highly important, because code is often a source of headaches in Ops. This is coming from a guy who is usually hired to write code for Ops departments.
A software developer can of course perform some of the functions of a sysadmin, but again the complexity of the whole becomes too much to understand just by doing single tasks. Either job is too complex to learn without a lot of study.
I just find comparing the two in the context of one becoming the other invites oversimplification of either role, and devops doesn't help that comparison.
> If at all possible you should use the least code possible to do a given job, and rely on the reuse of tools to serve various functions.
I agree, on both counts--and you should write the least code possible and reuse the most tools possible when you're building a web app, too! And when I am managing systems and infrastructure, I am reusing tools that are configured and operated through the expression of domain-specific code in Ruby. When I'm solving a problem, I'm writing code in Chef or in Cfer/CloudFormation (if I'm touching a machine by hand that's outside of my burn-it-down test environment, something's going really, really wrong). I'm consciously writing that code in such a way that I can put it into an automated test environment based on preconditions and postconditions that map to the business logic of the system to ensure correctness when that code is then changed. If I need to solve a related problem later, I'll generalize the code for least-effort reuse at that point.
This is literally software development, to me. That the output involves a configured system instead of a web app or whatever is orthogonal to being software development. It still feels like you are drawing a distinction without a difference.
Not the same at all. Let me rephrase the question.
Reading the article, the sysadmin is described as the person who constructs and maintains the environment the code runs in, including the physical machines, but also things like dependency management (I infer).
Given that just about everything except the physical machine can be handled through devops automation (from experience), and those things don't require "special" skills other than docker/ansible knowledge, why are sysadmins still considered separate from software devs?
On a more opinionated note, I think that using people, and not code to setup the operating environment of their application is pretty inefficient nowadays.
I started as a dev many years ago, then moved into sysadmin / devops stuff for over a decade, and then moved back to more hands on dev stuff again a few years ago. I think your statement that "those things don't require "special" skills other than docker/ansible knowledge" is over-simplifying things a bit.
Writing a production quality Dockerfile or Ansible Playbook requires more than just a bunch of "apt-get install" statements. You need a bunch of knowledge about how each bit of software should be configured and tuned for each individual use-case.
Once it's been written, then the devs can happily automate all the things as much as they like. Writing it in the first place is where a lot of what was traditionally sysadmin knowledge comes into play.
Anyone with a bit of patience and practice can follow a recipe in a cook-book. Not everyone can write one.
I'll agree that I definitely oversimplified it. I think what I was getting at was that it was a similar set of skills to other types of software development.
Thinking like this is what leads to tens of thousands of Mongo databases being exposed online.
Anyone can follow a "how to install X" guide and install a few packages and get a working config to get going and then automate it with their tool of choice but that doesn't come anywhere close to a production ready system.
When your job focus is shipping product that's all you care about, get it out the door as fast as possible so why learn anything more than the bare minimum?
Software is complicated and error prone and it takes a good amount of skill and time to write it well. Personally, I'm not impressed at all with most software made in an Ops environment. I think it usually sucks and that sucky quality, along with the complexity of debugging and maintaining it, makes it a net deficit to productivity. I have been writing software in Ops for a long time and it is just annoying how little work is done correctly in this space. I think most Ops automation is so error prone that you might as well hire an intern to stare at your network and you'll get as good if not better system stability.
In fact, this is how Ops used to work. You'd use humans to oil the machinery of the basic automation you put in place, and they were damn good at keeping uptime high and catching problems nobody thought of in their automation. Now people just automate and pray, it seems, and then play catch-up indefinitely, or waste time redesigning.
We totally automate the physical machines too. But wherever possible by using tools, not writing software. Honestly I wouldn't want to insult the practice of software engineering by comparing my work to theirs.
> Software is complicated and error prone and it takes a good amount of skill and time to write it well.
This we can agree on.
> Personally, I'm not impressed at all with most software made in an Ops environment. I think it usually sucks and that sucky quality,
Not sure what is an ops environment although I'll agree with the general statement that most software I see sucks and is of sucky quality, but I think that relates back to your first point.
> In fact, this is how Ops used to work. You'd use humans to oil the machinery of the basic automation you put in place, and they were damn good at keeping uptime high and catching problems nobody thought of in their automation. Now people just automate and pray, it seems, and then play catch-up indefinitely, or waste time redesigning.
I'm not proposing that the rise of devops precludes the need for people specialized in it's use and implications. Instead that sysadmin is essentially becoming a branch of software engineering with the rise of devops. When setting up a system IS writing code, why is the "software engineer" different from the "sysadmin"?
They're not really and they shouldn't be (although systems knowledge here becomes increasingly important). This is the point of DevOps, that operations can be treated as a software problem and managed in an automated fashion. The role of "SysAdmin" as presented in this article is fading quickly.
with the rise of cargo cult style management, how often are managers different then a script that purchases items from Amazon randomly and bulk emails randomly generated business plans?
