43 free and open-source tools that put the Ops in DevOps
A full DevOps deploy pipeline should consist of more than just build and developer tools. It also must accommodate creating and administering the production environment through code. Yet for some reason, actually running production (what used to be called Ops) through automation seems to get short shrift.
The DevOps tools listed below will ease admins' minds. You can Snap and connect these together to create a push-button environment from scratch, or use them to do the slow spade work of bringing a legacy system to a modern support cadence.
Organized by category, these free and open-source tools, which you can download and explore immediately, range from configuration management to logging, monitoring, and provisioning—creating new virtual servers at the push of a button, in a private or public cloud.
Once a production server is built, it just sort of exists. Operations upgrades the operating and system software occasionally, security hardens the server, and we hope nothing goes wrong. Some companies keep a checklist of how to re-create a server, so it can be done in hours instead of days, but mostly we hope they never go down. Configuration management (CM) tools store the information about systems; from software and versions to drivers and hardware. With the advent of virtualization, we found a new possible kind of virtualization: We could either store an actual disk image of a point in time in CM Software, or store all the information in order to re-create the server on demand.
Puppet and Chef provide that second sort of CM. They store the state of the data center and allow admins to create simple, textlike commands (Chef calls them "recipes"), then run those commands on a variety of servers. That makes deploying a change to a large server farm with multiple development, test, and production servers as easy as a Unix command-line operation. These tools also add the classic programming (the "Dev" in "DevOps") benefit of versioning, code review, workflow process, and delivery. Chef also offers a usable dashboard to create and manage configurations.
CFEngine is designed to serve large, complex product environments and has an emphasis on performance, security, and compliance. SaltStack offers an open-source CM product. This tool provides the typical CM functionality but allows its users to download and modify product code. There is also a large community network available for support and training.
If you have production servers and don't know how they got there, or why, you could re-create them with CM tools and virtualization, then test to see if they work. Or use a tool such as Blueprint, which reverse engineers servers and software products to discover new software installations and configuration changes. Each change is captured in a format that can be ported to another environment through Blueprint's sibling tool, Blueprint I/O.
Sometimes the problem is converting text from one type of markup to another. Pandoc is designed to automate that work.
Bcfg2 and ActiveMQ are alternative open-source products that may suit companies that are working on smaller budgets and are well versed in dealing with open-source software.
As a first step, consider storing configuration files in git, svn, or other traditional version-control tools, then develop tools and procedures for managing the files and system changes. Git is a more modern product that supports advanced branching and merge techniques.
The days of hand-grepping through log files to look for specific errors are long gone for most administrators. There is just too much data, coming in too fast. Administrators today need to aggregate log data, getting insights into status and alerts in real time—or have powerful analytic tools to look for trends. Log management tools help collect logs in one place so that IT staff can easily discover important system events.
Rsyslog, short for "the rocket fast system for log processing," sells itself on performance. This product claims the ability to process over a million messages per second. Rsyslog can take input from many different sources and output to different destinations quickly.
The combination of ElasticSearch, Logstash, and Kibana, sometimes referred to as the ELK Stack, is a very popular set of tools for log management. The combination of these three tools allows the Ops person to gather logs and search through them. Kibana is a data visualization tool that sits on top of Logstash and ElasticSearch to produce easy-to-read reports and real-time summaries of streaming data.
StatsD is a free front-end product written in Node by an Etsy employee. This product has implementations in many different languages and is designed to sit on top of a Graphite metrics server. Graphite is a tool that consumes and stores log data for use by display tools. Graylog2 is an open-source, all-in-one log management tool. This product will gather logs from a variety of sources and produce real-time reports based on streaming data.
In small or new software companies, deploying new software to production is often a step-by-step process involving someone typing in commands and waiting for feedback. Because it involves turning off the web servers, copying files, and restarting, deployment often has to happen at around midnight on a weekend. Deployment tools simplify that process for small companies and make deploying complex software a possibility for larger companies.
Fabric is a Python-based deployment tool that provides the Ops person a suite of tools to make it easier to use SSH. Fabric is based on usage of defined configuration files. Capistrano is a tool written in Ruby that can be used to deploy products in any language. A new hosted (cloud) version of Capistrano is available to teams that do not want to manage deployment from a local server. Ansible is a mass SSH deployment language that's very easy to get started with. Puppet, mentioned previously as a configuration management tool, doubles as a deployment management tool. Foreman is a front-end deployment tool that many people pair with Puppet.
Administering a server farm often requires the same command to run on many systems; that is where central controller tools come in. The category involves running remote SSH commands on many different systems. Func is a lightweight tool for managing remote systems. There are no complicated installs or configurations to manage. Fabric, mentioned previously, offers a streamlined set of operations for performing actions on several different remote systems.
Varnish Cache is more complex. This tool can perform the duties of a reverse HTTP proxy to cache copies of web pages the first time a user visits. All visits after that are served by the Varnish Cache. This is a combination load balancer, firewall, and HTTP router.
Continuous integration (CI) tools can be thought of as complex task managers. In modern software development environments, they are an absolute must. The main function of CI systems is to incorporate new source code, create a product build, run automated tests, and then produce a report. For server administrators, CI means the ability to gather all the artifacts for a production release, up to and including a full virtual server, for every build. That makes it much easier to find where a problem was introduced, do an audit, and roll back.
Jenkins and Hudson are probably the most popular CI tools available. Both are fully featured tools that will do just about anything a modern software development team needs. GoCD manages the build/test/release cycle for companies that want to release to an environment automatically several times a day. Shippable is a continuous delivery tool that is built around supporting cloud infrastructure, especially companies using Docker.
Concourse is a highly configurable CI tool that automates deployment pipelines and enables continuous delivery. Concourse can perform builds, create Docker images, merge in multiple branches, and deploy if unit tests pass. While Jenkins and Hudson can also do that work, Concourse was built with that kind of complex setup in mind.
Monitoring and measurement
Programmers tend to focus on increasing mean time between failures, but the team can also get value from it's opposite: minimizing time to identify and recover, or mean time to recovery (MMTR). Logfiles can be helpful to minimize MTTR, but in addition, there is plenty of other information a server might have available, such as CPU use, number of 404 or 500 errors, exceptions handled, database queries, response times, uptime, and so on.
Graphite and StatsD, previously mentioned in this article, are popular tools for aggregating and displaying log information from software and hardware in a way that is usable for Ops staff. Programmers can make statsD commands at any time, over UDP (so it won't impact the production network), and Graphite can display those statistics on monitors, allowing programmers to deploy and "watch the screens" to make sure a change does not impact server performance.
CollectD, Ganglia, and Munin are similar tools that run as daemons and produce very simple reports. These tools are free and open source but are also portable and performance-optimized. Nagios and Zabbix are suitable for large, complex environments with many remote servers.
Cacti is a graphing solution that uses real-time data feeds to produce reports for operations departments, much like Graphite.
Bluepill is a daemon manager that is useful for managing tools such as Munin.
Icinga is a modular monitoring tool that can be extended to suit the needs of your product and environment.
Software environments often exist on virtual machines rather than dedicated hardware. If the software isn't deployed that way, developers and testers are using a virtualized strategy to create and destroy new completely software environments in minutes.
OpenStack is an open-source solution for creating public and private clouds. Clouds created using OpenStack can be managed through dashboards built on the OpenStack API. Vagrant is an environment manager specifically for creating quick development environments. This tool can drastically simplify the first day for new employees or anyone moving from one project to another in the same company. Rulter is an open-source library, but this one is focused on isolating deployment scripts via Docker containers.
Docker is a modular system for building environments. Each Docker container holds everything needed to run a piece of software—product code, runtime environment, and system libraries. Docker builds can be stored in CM tools to improve auditing, to know exactly what was deployed when.
Putting it all together
Sadly, there is no simple DevOps tool for administrators "to bind them all." Instead, most companies start with software to build the virtual servers (OpenStack for a private cloud; sadly, the public cloud is not free), then perform the software builds, administer them, log, and monitor against them.
This guide provides a bit of a road map, and it can be intimidating. Don't boil the ocean. Instead, find a single serious pain point that one of these tools can address and go after it. Plan this as an experiment. If it works, or even if it doesn't, find the next pain point and move on.
So get going. And tell us how it turned out.