Overview

This solution provides a comprehensive, secure, and automated approach to managing containerized applications. By integrating Gitlab CI for continuous integration, Argo CD for continuous delivery, and leveraging the scalability and resilience of Kubernetes, it empowers development, platform, and management teams to deliver value faster and more reliably.

The GitOps-centric design ensures that infrastructure and application configurations are managed as code, promoting auditability, version control, and automated reconciliation.

Features

This solution is packed with features that streamline the development and deployment process.

Easy building & testing is achieved through optimized CI pipelines, simplifying the creation and validation of containerized applications. These robust pipelines help to ensure code quality and reduce the risk of errors.

Further enhancing efficiency, the solution allows super fast deployments via Argo CD, which automates the deployment process to Kubernetes.

These deployments, driven by delivery via GitOps, manage infrastructure and application configurations as code in Git, delivering auditability, version control, and automated reconciliation.

The solution is also compatible with any containerized software, working seamlessly with applications packaged as Docker containers and orchestrated by Kubernetes. Moreover, it provides dynamically created on-demand/preview environments, temporary spaces useful for showcasing and testing features before merging to production.

In addition, the platform is properly secured with the “restricted” Pod Security Standard to enforce strict security policies and protect applications and infrastructure. Finally, you can generate SBOM for all artifacts, ensuring transparency and aiding in vulnerability management.

Benefits

The solution yields significant benefits across development, platform, and management teams, driving efficiency, security, and innovation.

For Development Teams

This solution significantly empowers development teams with a focus on streamlined processes and reduced complexities.

Unified deliveries are made possible through standardized Gitlab CI templates, unified Dockerfiles, and application profiles, ensuring consistency across projects.

Unified change management simplifies the process of modifying both the platform and applications, as changes are delivered through the same automated pipelines.
Development cycles become shorter, leading to reduced TTM (Time to Market), as the elimination of manual steps and long wait times allows for faster feature releases.

Faster onboarding of new projects is also possible, as the ready-made and automated infrastructure enables developers to focus on delivering value rather than dealing with setup and configuration.

In addition, build platforms exhibit excellent scalability, growing to meet changing demands.
Finally, more efficient testing is possible through dynamic preview environments.

For Platform Teams

Platform teams experience a significant reduction in operational overhead and improved control.
Unified change management translates to less chaos and promotes best practices across the organization, simplifying platform management. This in turn leads to reduced errors, fewer “fires,” less stress, greater stability, and fewer calls during on-call duty.

Platform teams will also benefit from better control over changes / increased transparency, thanks to the ability to easily track any change through GitOps.

High availability is achieved via the GitLab architecture which provides a continuously available build platform.
Finally, including an SBOM makes it easier to track down all the software components for security compliance.

For Management

For management, the benefits translate to strategic advantages and cost savings.
A faster development cycle results in reduced TTM (Time to Market), fostering faster innovation and a competitive edge. Projects are onboarded much faster due to pre-configured environments, allowing teams to focus on delivering value, not on maintenance.

Management can rely on high availability of the build platform, a crucial capability for organizations to deploy software fast and reliably.
They can also count on increased security because SBOM is included in the solution, which is designed with security in mind.
Finally, this solution is cost-effective thanks to being highly scalable and reducing waste that is common with implementing dedicated solutions.

Ingredients

This solution relies on several essential components working in concert.

A Gitlab CI instance, running on Kubernetes, forms the foundation for continuous integration. This instance is created and maintained with code via GitOps, ensuring that its configuration is version controlled and reproducible.

Argo CD, with separate dev and prod instances installed on Kubernetes clusters, handles continuous delivery.

Kubernetes itself is a critical ingredient, requiring at least two clusters – one for development and one for production. An optional dedicated cluster can further isolate the build environment, hosting the Gitlab CI instance, Gitlab runners, and Docker executors.

Finally, the organization of Git repositories is critical to effective operation. The solution uses the following repositories:

• Applications: To store the source code for individual applications.
• CI: To store CI templates and custom CI configurations.
• CD: To store deployment manifests and configurations.
• Management: To store infrastructure and platform configurations.

Description

There are three major areas of the solution. Each has its own purpose and is built using the ingredients mentioned previously.

Continuous Integration (CI)

Individual application code resides in their respective application repositories.
A dedicated CI repository stores reusable CI templates and custom CI configurations, including GitLab CI templates and a custom container image for docker executor with custom tools (e.g. Kustomize, Skaffold, Buildpacks, curl, etc.).
Gitlab CI is triggered by commits to an app repo and runs within the GitLab CI instance. The Gitlab CI instance uses a custom docker image and the builtin registry.

The pipeline uses templates from ci and runs on a container created from the custom image. The pipeline uses the API for GitLab to push, pull, check status, create merge requests, and create comments. This builds, tests, generates image and publishes it.

Finally, the template ends by generating manifests and publishing them to CD repository. These can be:

• Static manifests
• Modified Kustomize overlays
• Helm Chart values

Continuous Deployment (CD)

Deployment is centered around the CD repository, which houses deployment manifests and configurations. This repository acts as the single source of truth for the desired state of your applications.

The CI pipeline ensures that the CD repo always reflects the latest successful build by performing automated pushes after a successful build and test cycle. These automated pushes trigger the creation of a MR created by CI templates. This merge request contains the proposed changes to the deployment manifests and configurations.
To ensure that changes are properly reviewed and validated, the merge request is subject to code reviews and automated checks by authorized team members.
These automated checks can include linting, security scanning, and integration tests to ensure the changes meet the required quality and security standards.

After the checks pass, the changes undergo an approval -> merge process, managed by authorized teams. This multi-stage approval process ensures that only validated and authorized changes are deployed to the environment.

For each environment (development, pre-production, and production), a specific Argo CD instance is deployed, such as the Argo CD dev instance. This instance is responsible for monitoring the CD repo and synchronizing the environment to match the desired state defined in the manifests.
It has the following characteristics:

• Runs in Kubernetes dev cluster: The Argo CD dev instance is deployed and runs within the development Kubernetes cluster, providing a dedicated deployment management environment for the development team.
• Managed with code from the management repository: The configuration and management of the Argo CD instance itself are managed as code within the management repository, ensuring consistent and reproducible deployments.
• Restricted access based on roles: To maintain security and prevent unauthorized changes, access to the Argo CD instance is restricted using role-based access control (RBAC), with most users granted read-only access. This ensures that only authorized personnel can make changes to the deployment configurations.
• Changes managed using resources defined in code: Argo CD applies changes based on Application resources, which define the desired state of the application.
  • Points to a path in the cd repository: Application resources point to a specific path in the CD repo containing the deployment manifests.
  • Can create namespaces for on-demand environments: Argo CD can create namespaces dynamically for on-demand/preview environments if required for a complex testing scenario.

Platform Management

The management repository serves as the central control plane for the entire platform, encompassing infrastructure configuration, shared service deployments, and security policies.
The core principle behind this repository is to manage “Everything as Code”, enabling GitOps-driven operations for consistent, auditable, and reproducible platform management.

The entire platform is managed as code and deployed using GitOps principles, enabling automated reconciliation and drift detection. The management repository contains the complete configuration for all Kubernetes resources required for the platform, including:

• Quotas: Resource quotas for limiting resource consumption by applications.
• Secrets: Secure storage of sensitive information, such as passwords and API keys.
• Netpol: Network policies for controlling network traffic between applications and services.
• RBAC: Role-based access control for managing user and application permissions.
• GitLab instance configuration: Configuration files for managing and configuring the GitLab instance
• Shared services configuration: Definition and configuration of shared services, for example such as:
  • Databases: Configuration for deploying and managing shared database instances.
  • Kafka: Configuration for deploying and managing shared Kafka clusters.
  • Critical components: Management and configuration of critical infrastructure components.
  • Backups, upgrades, tuning, troubleshooting: Procedures described in code and configurations for backing up, upgrading, tuning, and troubleshooting shared services.

The management repository is managed by the dedicated platform team, responsible for maintaining the stability, security, and performance of the platform.
The structure of the management repository can be organized as a single repository or multiple repositories in a hierarchical layout, depending on the complexity and organizational structure.

Similar to the development instance, an Argo CD prod instance exists, responsible for deploying and managing applications in the production environment. This instance adheres to the same principles as the development instance but with stricter security and access controls:

• Even more restricted access: The production Argo CD instance operates similarly to the development instance, but with enhanced security measures and stricter access control policies to safeguard the production environment.
• Runs in Kubernetes production cluster: The Argo CD instance operates in the production Kubernetes cluster to manage its configuration and the deployment of the applications.
• Managed with code: Like the development instance, it’s managed as code from the management repository.

Use Scenarios

Several key use cases benefit from this approach, highlighting the advantages of automation, security, and agility it provides.

1. Rollback flawed version of an app

Imagine a scenario where a new version of your application has just been deployed to production, but shortly after deployment, critical bugs are discovered, impacting users. Time is of the essence to minimize the damage. The solution facilitates a swift rollback by leveraging the CD repo.

First, you check for commit information to identify the specific commit associated with the problematic deployment, either by inspecting Argo CD’s UI, examining pod labels, or reviewing the CD repo details. Knowing the faulty commit hash is crucial for identifying the problematic changes.
Next, you find the MR and revert changes in the CD repository, effectively undoing the problematic deployment configuration. This is typically achieved by creating a new merge request that reverts the changes introduced in the previous deployment.
Finally, you wait for Argo CD to sync, which automatically applies the older, stable state of the application to the production environment. Argo CD constantly monitors the CD repository, and upon detecting the reverted changes, it automatically reconciles the production environment to match the previous, stable configuration.

This process allows for a rapid return to a functioning state, minimizing disruption and preserving user experience. The entire rollback process is automated, auditable, and repeatable, significantly reducing the manual effort and risk associated with traditional rollback procedures.

2. Implement “Quick Fix” for an app

When a critical bug or security vulnerability is identified in your application, a rapid response is essential to mitigate potential risks.
In cases where a rollback isn’t feasible or has already failed, perhaps due to data migrations or incompatible changes, this solution allows you to implement a quick fix and get it into production swiftly.

The process begins with developing the solution, making the necessary code changes to address the issue. This might involve bug fixes, security patches, or configuration adjustments.
Next, you push to the git repo, triggering the automated CI process. The build is triggered, running tests to validate the fix, and a container image is built and published to the container registry.
Comprehensive testing is performed to ensure the fix resolves the issue without introducing new problems.
After this, a new kustomize overlay is created with a new image, configmap, and other necessary changes to incorporate the fix. This overlay allows the platform teams to change certain parts of the application config (like the image version), without having to overwrite the whole configuration.
This new overlay is pushed to a repo in a new branch, creating new merge requests for development, pre-production, and production environments. This makes sure, that changes are only applied if other team members approve of the changes.
The CI finishes its process, and auto checks are performed on the newly created MR.
These automated checks can include linting, security scanning, and integration tests to ensure the changes meet the required quality and security standards.
Once the auto-checks pass, someone with the proper role accepts the MR and merges changes to the main branch. Upon merging, Argo CD automatically syncs changes on the dev cluster.

This allows developers to quickly check the changes and see, if their changes work. After initial validations on the dev cluster, you perform some additional tests.
The same sequence is repeated for pre-production - by merging the same changes, then let ArgoCD sync. This allows for testing the changes in a staging like environment. Lastly, because of the importance of the change the approval step on prod is manual. An operator carefully checks the changes and approves the MR. Argo CD syncs changes on the prod cluster.

After a few minutes from git commit new app is running on prod.

3. Implement big change in a new version of app

When introducing significant changes to your application, such as a major feature release, database schema migration, or architectural refactoring, thorough testing and validation are paramount to ensure stability and user satisfaction.
This solution enables you to leverage on-demand preview environments for comprehensive testing before deploying to production.

The process commences with developing changes and pushing to the app repo in a new branch. Creating a dedicated branch allows for isolating the new features and changes from the main development branch, preventing disruptions to existing functionalities.
A container image is built and pushed to the registry, and Kubernetes resources are rendered and pushed to the CD repo. Creating dedicated images for new features allows teams to try out new versions without having to deploy them to production.
After that, a new merge request is created. To leverage the new features, the developer initiates tests on a preview environment with a proper comment in the MR. A comment in the MR signals the CI pipeline to do something special, like deploying the created image to a preview environment.
The CI pipeline adjusts manifests to include the preview environment.
The Argo CD picks up manifests, creates a namespace for the preview environment, and deploys the app.

After a few minutes, the app is ready for thorough testing on a brand new env. This environment offers isolated, realistic conditions for thorough testing and validation of the new features, ensuring a stable and high-quality release to production. Once testing is complete, preview env is destroyed when MR is closed or merged.

4. Grant access to external service for an app

Applications often require access to external services, such as databases, APIs, or message queues, to fulfill their functionality. Securely and reliably granting this access requires careful management of network policies, secrets, and service accounts.
In this scenario, a platform team member develops changes in manifests. These changes might include modifying network policies to allow traffic to the external service, providing secrets containing credentials for authentication, or assigning roles to the service account to grant appropriate permissions.

For example: If an application needs to connect to a database, the network policies must allow access to the database port and the secret must include the database password.

After the changes are proposed, automatic checks are run to validate changes. These checks can include validating the syntax of the manifests, ensuring the network policies do not conflict with existing policies, and verifying the secrets are properly encrypted and stored.
Afterwards, a code review and approval process starts. This process is initiated to make sure, that changes are only applied with the consent of other members of the platform team.
Finally, the changes are merged to the main branch. Once merged, Argo CD picks up changes and applies them on the cluster either automatically or manually for critical areas.

5. Implement new policy

Organizations often face new requirements driven by evolving policies, regulations, or industry standards. Complying with these requirements can necessitate changes across multiple components of the CI pipeline and platform.
This scenario outlines the steps to implement such changes in a structured and controlled manner.

First, introduce new ci templates for apps. Before enforcing a new change it is important to create new solutions and get some feedback.
Then ask a few teams to start using it - open a feedback loop and collect their feedback.
Fixes are applied, and after that, the improved template can be published as the new version of the CI template. That triggers a new announcement of the version to encourage teams to migrate and give a grace period for the new changes. Before the final enforcement, teams must be made aware of what is coming.
Finally, the old versions must be deprecated by forcing the teams to upgrade and provide help during migrations.

During all those steps, the platform teams must introduce new changes, validate and revert if needed. These can include setting webhooks, Kyverno policies or other custom resources.

Implementation

This solution offers a flexible and efficient implementation path, designed to integrate seamlessly with your organization’s existing tools and processes while adhering to industry best practices for security and automation.

Duration

The initial setup of this solution can be remarkably fast.
In as little as two weeks, a fully functional environment can be established, complete with the ability to deploy and manage your first application.

This rapid deployment time ensures a quick return on investment and allows your development teams to start leveraging the benefits of the solution immediately.

Method

Everything within the solution is managed as code, employing GitOps principles for consistent and automated configuration management.
This is facilitated through the use of operators, such as the GitLab operator and Argo CD operator, which automate the deployment and management of these critical components within your Kubernetes environment.

Configurations are developed throughout the implementation process to align precisely with your organization’s specific context, needs, and existing workflows. This includes customizing several key aspects:

• CI Pipeline Templates: The implementation tailors CI pipeline templates to your specific needs and standards to fit existing tools and business processes. These templates encapsulate your organization’s preferred build, test, and deployment processes, ensuring consistency and compliance across all applications.
• Application Profiles: The creation of application profiles allows teams to group CI processes (build, test) by technology or framework. Each profile also includes deployment source information specifying whether deployments are managed via helm chart or kustomize with overlays.
• Security Controls: Robust security controls are established during implementation, with a strong emphasis on access control.
  • Access Control to Git: Secure and controlled access to Git repositories is essential for maintaining the integrity and confidentiality of your code. This access control acts as a single point of truth and control, preventing unauthorized modifications and ensuring compliance with security policies.
  • Access Control to Argo CD: Similarly, strict access control to Argo CD is implemented to restrict access to deployment configurations and prevent unauthorized deployments. This measure safeguards your environments from malicious or accidental changes.

Requirements

To effectively utilize this solution, several requirements must be met:

1. The applications must already be containerized, indicating that they should be packaged as Docker containers for seamless integration with the system.

2. An optional existing SSO solution can be integrated to manage user authentication. If no SSO solution exists, Gitlab accounts can be used for access to both Git and Argo CD.

3. A working Kubernetes environment is required. This environment can be hosted on a cloud platform (e.g., EKS, AKS, GKE) or deployed on-premise, provided it has DNS configured for services and access to Let’s Encrypt or another mechanism for providing trusted TLS certificates.

4. A reliable storage solution for Git repositories, the container registry, and the build cache, with additional object storage (e.g., S3) recommended for storing backups.