Generators

A crucial element of our GitOps platform is the generator functionality within ArgoCD. This capability holds significant importance for our platform as it automatically creates numerous components of our platform for each discovered service. We achieve this by integrating generators that draw on the data from our environments repository.

The ‘SCM Provider’ generator plays a crucial role in this process. It monitors our GitHub instance, searching for repositories that align with the following structure:

filters:
  - pathsExist:
      [chart/Chart.yaml, pipeline/Chart.yaml, Dockerfile]

This ability to easily identify new repositories and services facilitates their seamless integration into the platform. However, the ‘SCM Provider’ generator alone doesn’t provide the full context needed for complete service integration. To supplement this, we employ the ‘git generator’. This generator allows us to create Applications based on specific files within a Git repository or structured according to the directory hierarchy of a Git repository.

In essence, for each new service we identify, based on the matching pattern above, we search the environments repository for a yaml file that shares the same name as the discovered service. For instance, if we establish a repository named github.com/dx-book/service-1, the ‘git generator’ will search for an additional metadata file within the environments repository, aptly named service-1.yaml.

The contents of this service-1.yaml file are then utilized as input to generate a range of essential components that each new service requires. These components encompass aspects such as namespaces, pipelines, secrets, service accounts, and so on. This approach ensures the smooth and efficient integration of new services into the platform, enabling them to benefit from the suite of capabilities the platform offers.

Under the mechanisms previously explained, involving the discovery of repositories and the enrichment of their data with a metadata file, we’ve subdivided our generators into different categories. This segregation not only streamlines comprehension but also simplifies the maintenance process. We’ve divided our generators into different sections. We’ll see each of them in detail in the next section.

The chart below shows how it links together.

graph LR
A[ArgoCD] -->B(Platform)
    B --> |Discover repositories| C[Generators]
    C --> |For each discovered repository| D[Generates]
    D --> E[Dependencies]
    D --> F[Triggers]
    D --> G[Pipelines]
    D --> H[Deployments]
    

Activate the Generators

The generators are not activated by default, since it might bring tons of apps and make things a bit heavier, we decided to have a second step to bring them in.

This process is a one-time requirement designed to enhance clarity and comprehension about how infrastructure components are managed. With this foundation, our focus shifts towards integrating applications into the platform, rather than solely managing infrastructure components.

To initiate the bootstrap process for all applications discovered in our environments repository, please execute the following command:

helm upgrade platform-name platform/chart --set generators=true

This command will ensure that all the discovered applications defined with their own yaml files are appropriately included in the setup process on the platform. This is done through the generators.yaml pointing to the applications folder under environments repository.

The environments repository current looks like this:

├── README.md
├── applications
│   ├── dev
│   │   └── service-1.yaml
│   └── prod
│       └── service-1.yaml
└── shared.yaml

At this point, if everything is sucessful you can see a new ArgoCD Application called environments. If you see an error like Unknown desc = authentication required. you’ll need to update the secret you used for GitHub, since the one provided doesn’t seem to be valid.

The secret that gives the permission for this repository can be investigated under the file below. Please note this is a encrypted sealed-secret and you might need to reencrypt, either manually or using our ./configure.sh script, from the install the platform section.

cat chart/templates/base/secrets/github-environments-encrypted.yaml

Now, let’s move to look into each generator in more detail. First, with the dependencies generator.