At Bitworks, we develop scalable highly available comprehensive software systems. Each project is unique in its own way, and the criteria of the product quality differ significantly from case to case. Moreover, quality is a relative concept: customers, developers and end users may have different expectations from the product, and it is very important to achieve common understanding and agreement among all stakeholders.
In order to always create products that meet the universally accepted quality standards, provide the most appropriate solutions for the problem, while meeting the expectations of all stakeholders, we identified a number of high-level ideals which we rely on to build and use an effective development process.
The Scrum framework is used as the foundation for the development process at Bitworks, however, as the project grows, depending on the customer’s requirements, domain specificity, available resources, and other factors, this process is adjusted to achieve maximum efficiency and ensure compliance with the ideals that we identified.
In order to familiarize readers of our blog with the development process in Bitworks and the decisions we take to optimize it (and most importantly, the reasons behind them), we have decided to publish a series of articles, in which we are going to tell you in detail about the approach we use for development of software systems by using one of our projects as an example.
In this series of articles we would like to tell about:
- High-level ideals and generally accepted standards, which determine not only the development process, but also the functioning of the company as a whole.
- Key stages of the development process, connections between them, their nuances and possible alternative approaches to their organization.
- Participants and other stakeholders of the development process, and how the mutual agreement of their needs is achieved.
- Technical and methodological features of the development process organization, which ensure the harmonious interaction of all its individual components.
Throughout the cycle, as an illustrative example, we will use one of our current projects. So, without further ado, meet Placeholder, a digital advertising exchange system.
Of course, the project is not called Placeholder (no matter how suitable this name might be for the system associated with advertising): we intentionally want to depersonalize the product in order not to create incorrect associations and not to disclose information that can not be divulged.
We will not go into the details of the problem that the Placeholder product solves, we will not touch the specifics of the domain and the technical details of the implementation, instead we will concentrate only on the development process. However, for the sake of completeness, we will point out some key features of Placeholder:
- The domain of the product is automation of business processes in digital marketing.
- Target market of the product - the US as a starting point, further plans are currently under discussion.
- Users of the system are large publishers and advertising agencies (less often - individual advertisers).
- The system interacts with many external systems, including financial accounting systems, CRM-systems, blockchain.
- The project involves a separate (from Bitworks) product team, which deals with requirements analysis, interface design, customer service and product promotion.
The project has been in development for about two years and is currently preparing for a public beta release. Currently, the Placeholder team has 20 members (excluding the product team).
In this introductory article, we want to outline the main principles of the development process in Placeholder, its stages, participants and the main artifacts, in order to later be able to consider them in more detail in the context provided.
As with many other projects at Bitworks, the work on Placeholder was initially guided by a development process based on the Scrum framework. Over time, under the influence of various factors (the growth of the development team, the specificity of interaction with the product team, etc.), strict implementation of all the requirements of the Scrum framework became impossible, and the process was significantly modified.
The resulting process can not formally be attributed to Scrum, however, it adopts many of the principles and practices of the framework. The terminology has partially been preserved as well: for example, the Scrum-master role has preserved its name. In this article as well as in the following ones, we will pay special attention to the deviations from Scrum and will try to explain in detail their causes and effects.
The Placeholder development process is iterative and incremental, and it uses fixed duration sprints (according to the sprint definition in Scrum) lasting two weeks each.
Contrary to Scrum, the Placeholder development team has a further division into subteams:
- the Front end development team,
- the Back end development team,
- the Quality Assurance team (QA).
In charge of each of the teams there is a team lead, who represents it at all general events of the process.
Creation of a hierarchy and separation of responsibilities are major violations of the Scrum Guide, as is the increase in size of the development team over 9 people. The complexity of the project and the amount of work required a narrower specialization of the team members, and the need to maintain a high pace of development led to the expansion of the team, and subsequently to the emergence of the team leaders responsible for the fields.
In addition to teams and their members, the project has the following roles:
- Architect;
- Technical writer;
- DevOps specialist;
- Scrum-master.
The stages of a single iteration of the development process are shown in the following diagram:
The state of the project is monitored by the Scrum master at each stage. They develop and approve standards, ensure that the process conforms to these standards, optimize individual steps and the process as a whole. In this sense, it would be more correct to call them a process master (similar to how the Scrum-master helps to follow Scrum, the process master helps to follow the existing process), but the old name got roots and in the end remained unchanged.
At the beginning of each iteration the requirements are processed. Most of the requirements reach the development team in a “raw” form: they can include the analytical workings of the product team, the “wants” of the users, the refinements that arose on one of the previous iterations, the technical debt. In order for the requirement to become a task and be included into the iteration plan, it is necessary to:
- formalize it;
- break down into tasks;
- order tasks by priority.
In one of the following articles we will discuss in more detail the possible sources of requirements, the tools used to organize informal and formal requirements, the process of formalizing and decomposing requirements, and working with priorities in a large number of stakeholders.
Based on the results of the requirements refinement, team leads create an iteration plan: they formulate tasks for engineers, refine the assessment, organize tasks according to their allocation into teams (back-end, front end, QA) and draw up an approximate schedule for their implementation.
After this, in accordance with the plan, specialists begin to work on the tasks. Developers implement the necessary functionality, conduct code reviews and then send the result to the QA department for validation. QA specialists check functionality for compliance with the requirements and either accept the work or send it for revision.
The stage of task execution itself requires special attention and deserves a separate article.
The DevOps specialist provides Continuous Integration for several separate environments, in order to have independent versions of the system available for different participants of the process. In one of the following articles we will describe in more detail, which environments are used in Placeholder (and why), as well as other responsibilities of the DevOps and their place in the development process.
At the end of this phase, the regression testing of the entire functionality of the system is performed, alongside with preparation for the demonstration of the iteration results (sprint review). The demonstration allows all participants of the process to see the full picture for the first time. The results are later discussed at the Sprint Retrospective events: first inside individual teams, and then among the team leads.
After that, preparation for the release begins. Usually this is a very straightforward and entirely organizational stage: all the necessary arrangements are completed during the preparation for the demonstration, and it remains only to make final adjustments. At the same time the teams begin a new iteration.
In parallel with the iterative incremental process, Placeholder practices execution of intermediate projects. The introduction of the project-based approach to the iterative development process is another deviation from Scrum, but the proper allocation of time and resources among sprint tasks and intermediate projects allows, in addition to the increment, to solve such isolated tasks as improving the infrastructure and conducting security audits. We will talk more about intermediate projects in subsequent articles.
The application of transparency, inspection, adaptation, and other fundamental principles of the Scrum framework, as well as the preserved possibility to introduce significant changes to the process, all this allows the Placeholder team to achieve maximum efficiency in the face of constant growth of the project.