If you face the requirements to run a big bunch of environments for a single app, if you work with microservice architectures, have diverse of apps or an environment where an app is deployed for every tennant, if you develop a high performance scalable application, it means that DC/OS is what you should pay attention to.
Let’s imagine infrastructure as a layer-cake: a lower layer is physical servers, higher virtualized environment goes and higher there is a containerized environment where apps reside. Each following layer is more abstract, more hardware independent, goal-oriented and more user-friendly.
Physical servers. These are the most basic, the least specified layer. This layer isn’t always convenient to use for apps as it doesn’t provide fault tolerance and proper application isolation, which reside within one operating system. To overcome these limitations, virtualized and containerized environments started being developed.
Virtualized environment. This one is a level of abstract servers, which are allocated in such a way to maximally serve the purpose and provide maximum utilization of underlying hardware. One hardware server runs from a few to hundreds of virtualized servers.
Containerized environment. Containers are even more specific and serve for application isolation. The most known containerized environments are Docker, LXC, OpenVZ, Virtuozzo, FreeBSD Jail, Solaris Containers. This layer can reside right on hardware servers or in virtual servers. It depends on a goal and convenience. As a rule, there can be dozens and hundreds of containers even on a virtual server. Usually, entities proportion between levels differs significantly. One can readily mark that to run such a “zoo” can be quite expensive. For the virtualized environment, such orchestration systems as OpenStack, Apache CloudStack, VMWare vSphere, OpenNebula, and others are used for a long time. They allow users to abstract “hardware” and manage all the virtual machines in a convenient form, maintain them and develop. Often, further graduation stops here – administrators create virtual servers for some peculiar apps and use them.
However, in a modern DevOps world such state of affairs isn’t a preferred approach as, currently, a virtual server isn’t considered convenient enough. It is too complex and it doesn’t tend to be convenient when apps continuously evolve. Contemporary approaches of application development and deployment dictate new requirements. The most important among which is environment immutability. But in the case of a virtual server, environment immutability isn’t ensured since setting up: updating, deleting or reconfiguration lead to environmental changes.
Developers got to love Docker especially because it gave an opportunity of supporting of “clean” immutable environment with high resource isolation level. Developers inspired by this opportunity started using Docker for app delivery. Now, you can launch one app of any version in any configuration on one real device or virtual machine. You can delete unnecessary deployments whenever necessary, while environment stays clean.
If you try to launch many instances of MySQL on one Linux server, you will understand how difficult it is, while with Docker it’s really a piece of cake.
On the other hand, moving to smaller services within microservice architecture adds fuel to the fire. Such a death star as Netflix demonstrates how many microservices can be used in a complex app and each of them is run in a separate container in dozens, hundreds and even thousands copies to provide with high efficiency of a service:
The principle “divide et impera” is in use. Just imagine that every microservice can be deployed in several code versions simultaneously.
In reality, running these “premises” isn’t possible without special orchestration tools. There Kubernetes, Docker Swarm, Apache Mesos and its user-friendly incarnation DC/OS occur. These tools gather all the abstraction environment levels into one universal ecosystem.
Our company widely uses DC/OS and Docker Swarm for developing apps. So far we don’t use Kubernetes since we haven’t faced tasks where current tools look weak, besides, we don’t have corresponding requests from customers.
Apache Mesos and Docker Swarm perform next tasks:
- controlling available cluster resources;
- launching and maintaining apps in a specified condition.
In general, Apache Mesos seems more universal to us as Docker Swarm performs only functions as Mesos with Marathon framework, while there are dozens of other frameworks for Mesos, which are suitable for specific tasks, for instance, deployment, control and scaling clusters of Cassandra, HDFS or Kafka.
Apache Mesos is a complex tool and requires significant efforts to deploy and manage it:
Luckily, the main project’s sponsor, a company Mesosphere, has developed computing environment DC/OS that is an integrated platform for Mesos and many frameworks for diverse requirements. This environment can be easily set up on virtual or physical servers. It works sustainably and contains all necessary components for app orchestration by means of Apache Mesos.
DC/OS gives next facilities:
- Bring CPU, RAM, storage systems of virtual and physical servers together into one resource pool.
- Launch container or native apps on this joint pool with CPU, RAM and storage system resource reservation.
- Supports scheduled execution of tasks with the help of Metronome and Chronos frameworks.
- Allows running apps upgrade in a cluster with zero downtime.
- Provides app high availability by means of health monitoring and restarting for erroneous apps; provides availability reservation and load balancing between them.
- Maintains app groups which have unified policies for running, stopping and upgrading.
- Sustains network isolation and provides a dedicated address to every application.
- Distributes the load between instances of one app.
- Allows adding and deleting extra nodes.
- Provides maintained application catalog for specific tasks, for example, HDFS, Kafka, Elasticsearch, Cassandra, Redis, MySQL, PostgreSQL, that are set up with one click.
- Provides complete support of GPGPU, Ceph clustered file system, and Amazon EBS.
So, DC/OS is a meta operational system which uses scheduler Mesos to manage resources. This is how it is deciphered “Data-Center Operating System”.
The value of DC/OS is that it presents a completely integrated environment where developers can:
- perform CI/CD;
- deploy an app version for each VCS branch;
- route traffic with the help of a balancer and DNS between different app versions, for example, app-develop.cluster, app-release.cluster, app-feature-x.cluster;
- carry out experiments with a code in an environment indistinguishable from a real one;
- perform a software-defined automated deployment from the perspective of abstract communicating applications rather than low-level operating system configuration settings;
- deploy infrastructure components with one click (for instance, absolutely working cluster Apache Kafka, or replicated MongoDB);
- do deployment in the style of A/B, Canary, Blue-green;
- use private Docker registry;
- develop apps which use big data processing and machine learning.
All facilities are integrated and described in documentation or articles. You don’t have to think about how to implement something in a development environment so that it works in a production environment. We have come up to a common approach in BitWorks which allows using DS/OS for delicate app development using DevOps and CI/CD practices. In the next article, we will take a detailed look at ways of organizing a complete development, testing and performing environment by means of DC/OS and Gitlab CE.
When using a right approach you will stop being afraid of losing all infrastructure since in a matter of hours you will be able to deploy everything from Git, you will be able easily to update apps with a new version available, you will provide all interested parties with all necessary code versions for testing and demonstrations, you will be able to easily run thousands of app versions for diverse customers in a single cluster. For more detailed information about DC/OS see an official page of the project.
Please, share the post with friends if you like it.