News

Apache CloudStack extension for hybrid event bus

logo
Today we would like to share a new plug-in event bus module – a hybrid event bus that combines two buses: inMemory Event Bus – to handle events inside CloudStack, and Kafka or RabbitMQ Event Bus – to interact with external handlers.

We use Kafka Event Bus in our CloudStack installations. Recently, after testing a new plug-in in the product environment, we found that this bus does not have event subscription functionality: internal event receivers can not subscribe to events, as there is simply no implementation.

This is a nasty thing, because it limits the CloudStack API. It is almost impossible to predict who would like to subscribe to events even inside the CloudStack core, for example, to implement deferred processing. Using the standard Kafka Event Bus, some of the event bus capabilities are no longer available, and troubleshooting this regression is very difficult. To overcome this, we implemented a “hybrid” bus that uses the inMemory bus for internal purposes and the Kafka or RabbitMQ bus for external communications.

You can find the extension on GitHub published under Apache 2 License.

Successful completion of the project to create a personalized ad platform

After six months of development, the project of ONELYA, in which our team acted as a key subcontractor, was successfully completed. Purpose of the project – creation of an advertising platform designed to generate personalized advertising offers based on automatic analysis of user behavior on sites and mobile applications. In this project we managed to demonstrate not only our well-known abilities to develop high-performance, scalable advertising delivery platforms, but also to apply modern approaches based on machine learning and real-time data analysis.

The project from our side was developed by a team of sixteen professionals:

  • two project managers;
  • three backend engineers;
  • two frontend engineers;
  • three QA engineers;
  • three ML engineers;
  • systems analyst;
  • technical writer;
  • DevOps expert.

Delivery time: more than six man-years.

Key technologies:

  • Languages: Scala/Akka, Python3, TypeScript;
  • Repositories: Apache Cassandra, Elasticsearch, HDFS, RocksDB;
  • Message Queuing: Apache Kafka;
  • Frameworks for big data: Apache Spark, PySpark;
  • DBMS: PostgreSQL, Liquibase;
  • UI: Angular 6, Fingerprint2, Swagger;
  • Infrastructure: Ansible, DC / OS, Docker, HAProxy, NGINX;
  • Automation E2E QA: Postman, Protractor;
  • Management: Gitlab, Gitlab CI, Atlassian JIRA, Atlassian Confluence.

Practices:

  • Agile/Scrum;
  • Continuous integration, continuous delivery;
  • Git-flow;
  • automated testing of backend, frontend, ML;
  • infrastructure as a code.


Apache CloudStack Virtual Machines Log Management Plug-in

logo
From 2017 we actively contribute into open source cloud management platform Apache CloudStack. Those efforts had began from developing of CloudStack-UI, a modern CloudStack frontend interface, during development of that product we became involved deeper into core product and suggested several product improvements and fixed number of bugs. CloudStack is an extensible product with very flexible plug-in interface which enables the creation of extensions which change system behavior or implement new, previously missing features.

Today, we are proud to present our new contribution to CloudStack - cloud-vm-logs, a plug-in purposed for VM log records collecting, storage and management. The plug-in permits administrators to integrate log management with CloudStack, shifting the accent from the infrastructure provisioning and managemnt to the application management. The plug-in is built around Filebeat, Logstash, Elasticsearch, a proven scalable stack for log management.

The extension allows users filter records by time, journal files and keywords. In the next published version we plan to provide a console utility for logs access and implement additional API features for a log statistics and journal retention configuration.

We granted it to the community under Apache License 2. Source code in Java is uploaded on GitHub. If you find the plug-in useful, please give us a feedback and suggest improvements.