Best Practices for Argo CD
What are some best practices when using Argo CD?
Within this blog post, we’ll be highlighting some best practices tied to Argo CD, that allow you to leverage GitOps easily within your deployment workflow.
Below we will explain the following:
- What is Argo CD
- What are some best practices with Argo CD?
What is Argo CD
Argo CD is the most popular and fastest-growing GitOps tool for Kubernetes. When following a GitOps deployment pattern, Argo CD makes it easy to define a set of applications with their desired state in a repository and where they should be deployed. After a deployment, Argo CD constantly monitors the state and can even catch configuration drift.
Argo CD's core component is the Applications Controller that executes continuous monitoring of applications and then compares them to the live application state against your target state defined within your Git repository.
The Application controller retrieves the desired resource manifest from the Git repository and compares it to the live resource from your Kubernetes cluster.
This approach enables GitOps for your deployment workflow by leveraging Git as your source of truth. Now, let's further explore how we can leverage Argo CD best for our deployments!
The image above shows the primary Argo CD dashboard with a single Argo application successfully deployed.
The dashboard above provides a detailed view of the same Argo application deployed in the image above. However, it also provides an understanding of the status of the Kubernetes resources.
Now, let's explore how we can apply best practices to our Argo CD deployments!
Argo CD Best Practices
Argo CD has several best practices; however, we will review some of the most important ones we've gathered from the Argo community and prioritized below.
1. Separate your Git repositories
It's super important to separate your configurations and source code into different Git repositories. Separating your configs in their repository limits commit access to avoid something being pushed to production environments. For example, if you accidentally push manifest changes to the config repo, it can trigger an infinite loop of build jobs and Git commit triggers. Using an automated CI pipeline, you can also prevent pushing manifest changes to the same repo.
Separating your repositories is also more secure and restricts commit access, so someone does not accidentally misconfigure an application.
Argo CD does not connect to your source code repository. However, if you make a change to your code and you want to deploy the change to a specific environment, you can do the following:
- Build a new application code version and then merge a pull request (PR) in the configuration repo to use the new application version you created.
- Utilize automation for the update you made to your source code by using a pipeline in the source code repository.
- Argo CD Image Updater allows you to update your container images of the Kubernetes workload managed by Argo CD.
These approaches allow you to maintain logs for any auditing process. You can quickly identify development activity and reference your Git history for easy traceability.
2. Create a directory structure to enable a multi-application system for your Argo CD deployments
Once you've separated your source code and configs into separate repositories, it's essential to set up a directory structure that applies GitOps for Argo CD deployments. You can leverage several approaches for your Git repository structure that best serves your organization's needs.
However, we'll highlight some tips to best structure your directory when using Argo CD.
- Do: We suggest modeling your environments or clusters using different folders instead of branches in your configuration repository (e.g., prod, staging, testing, etc.).
- Do: Make sure your cluster and environment configurations repositories are separated (i.e., separate your prod configuration in a different repository from staging).
- Do: Utilize some sort of manifest management, such as a raw Kubernetes YAML file, Kustomize, or Helm for your environment definitions for your apps.
- Do: Create an 'argocd' folder in your configuration repository for each cluster and create an Argo CD Application manifest for each app in the cluster's repository.
- By creating the separate 'argocd' folder, you can also implement role-based access control for different clusters if you wish with Git repository permissions.
- Do: Leverage a multi-folder or a multi-repo structure instead of a multi-branch approach. You should NOT have permanent branches for your clusters or environments.
- Don't: Never put any independent applications or applications managed by different teams in the same repository.
Implementing these strategies above for your directory structure provides several advantages. Those advantages include improved security, easy rollbacks, audit log, easier testing, and automating your manifest updates. It also allows you to create and delete applications dynamically. If you'd like to see some examples of these directory structures, look here.
3. Determine a promotion strategy
Now that you've established a directory structure, you may face a challenge regarding the best promotion practice between clusters. When deploying multiple applications with Argo CD, it's best to pick one promotion strategy that suits your directory structure and stick with it. Below we'll highlight how you can do this based on your own needs.
Group your applications
First and foremost, once you've established a way to manage your applications and deployments, and there are too many apps to keep track of - the ApplicationSet is your best solution. We should also note that the ApplicationSet was previously an external controller that you had to install on your own, and now it's integrated and makes things much easier! However, you need some extra support when using the ApplicationSet instead of the App of Apps pattern that is easy enough to use right away without any additional help or a learning curve.
Yet, when deploying to production, an ApplicationSet is the best choice. This is because an ApplicationSet is a Kubernetes controller/custom resource definition (CRD) that enables automation and allows flexibility when managing multiple applications across several clusters.
The ApplicationSet consists of two main components:
You are essentially allowing the ApplicationSet to act as a templating agent.
The generators used with the ApplicationSet are responsible for generating params, utilizing the template to consume the variables, and then applying them to the cluster as Argo CD applications. So, any changes made to an ApplicationSet template and its fields will automatically be applied to each application created.
This way, you can simultaneously deploy your Argo apps to multiple Kubernetes clusters.
When managing ten or fewer applications, it's best to use the App of Apps pattern.
App of Apps was the pioneer before ApplicationSets that allowed us to deploy multiple applications at once with ease.
You are leveraging the actual app itself, aka a "Root App," to contain the other applications instead of individual Kubernetes objects. This, in turn, allows you to manage a group of applications that you can deploy declaratively. Essentially this pattern supports the declaration of children apps in a recursive way.
Image from the Argo CD documentation
Another possible approach to promoting changes is the Argo CD Image Updater. It's still a relatively new project, but it provides a manual way to update your manifest versions.
However, we should note that this approach is not GitOps friendly. Meaning it goes against the declarative nature of GitOps, but perhaps its functionality is what you and your team needs.
Which to choose?
One does not "need" a promotion strategy when beginning to deploy with Argo CD. The strategies listed above are needed when trying to solve "too many apps." This problem dramatically depends on the number of Argo CD applications you are managing and whether or not you already have a deployment process in place or not.
The Argo community has developed the strategies listed above to help you manage all your Argo CD application deployments, depending on your scale, directory structure, and manifest types, amongst other factors.
However, it's ultimately up to you and your workload, which is best for you and your team.
4. Manage your secrets securely
No one solution for secret management will work for all organizations. However, some common approaches for how to best handle your secrets with Argo CD can help based on these two significant factors:
Encrypt your secrets directly in your Git repository:
Sealed secrets are one way to encrypt secrets created by anyone but can only be decrypted by the controller running within the target cluster.
SOPS (Secret OperationS) is an open source solution that allows you to encrypt and decrypt an entire file or field for your Kubernetes secrets. This approach will enable you to store your secrets, and other Kubernetes manifests directly in your Git repository.
Externalizing your secrets from your Git repository:
This solution creates debate about whether it's GitOps or not. Nonetheless, it's a solution using Kubernetes auth in Vault. The Argo CD repo server authorizes Vault to use the service account token in the secret manifest, substituting the required value and creating a secret for you.
- Cloud Provider Secrets
Cloud provider secrets are created by your cloud service providers (CSP) depending on the cloud service. It depends on what your CSP offers and if their secret management solution meets your needs. Ensure you evaluate which strategy secures your data best.
Firstly, storing your secrets in a Git repository isn't easy! Perhaps your organization has a different policy for secrets, or you already have a system in place, so in this case, you may want to externalize your secrets.
However, if you want to store your secrets in your Git repository, plaintext secrets cannot be stored in your repo, and you never want to risk storing sensitive data and exposing it!
But, Git can truly be the source of truth, including your secrets, and you won't have to risk getting fired when they are encrypted asymmetrically. You can use Argo CD to manage your deployment state from Git into your cluster, including your secrets, by ensuring processes are put in place, and the tools above are used safely.
Essentially, we cannot say what the best practice for you and your team is to access Argo CD. However, we can advise you and your team to choose which approach works best for your organization and set a standard for your team.
Developers require access to Argo CD, so you need to set security measures and role-based access control (RBAC).
This approach allows your developers to access the Argo CD UI and ensure that each team has the appropriate access to their applications and their applications only.
Argo CD provides an RBAC configuration that includes two pre-defined roles:
- Read-only
- Admin
You can also implement a permissions definition for your applications and other resource types with Argo CD. You can then sync these roles and permissions together to create a policy to define access to your system.
No one within your organization requires access to Argo CD.
Some organizations do not want anyone to access the Argo CD UI or the API server. Perhaps the organization would like to use the automation but does not want to leverage the UI component. In this case, you can use a variant of Argo CD, also known as Argo CD Core. This allows developers to make a commit and ignore the UI aspects and reduce complexity. However, we should note that you lose multi-tenancy benefits by installing Argo CD Core.
So, whichever approach you choose above - as long as it benefits you and your team, you're making the right choice.
If you're using Argo CD for Continuous Delivery (CD), you want to elevate the rest of your workflow to automate everything in Kubernetes to reduce downtime. Then, you should check out the other Argo projects and learn more about how they can help.
We don't encourage you to use any specific projects to leverage best practices, but instead, we encourage you to use those that will help you and your system best.
In the meantime, we can explore how we could use Argo Events, Workflows, and Rollouts to enhance your system if you choose to.
Progressive Delivery
Progressive delivery is a set of practices that roll out new features gradually instead of all at once.
Rollouts provide advanced deployment capabilities and rolling updates for progressive delivery approaches you might already know, such as blue-green, canary, etc.
Advanced Deployment with CI
Argo Workflows and Events require you to have an existing Continuous Integration (CI) process to leverage these projects.
Workflows allow you to build and orchestrate parallel jobs and utilize a pipeline on Kubernetes.
Events is an event-driven workflow automation framework that is used with Kubernetes.
By utilizing these other Argo projects, you can easily manage your clusters, run workflows, and implement GitOps for your Kubernetes system! However, not all organizations are ready to implement automation for everything in Kubernetes. Combining all these tools requires time and understanding of how your system works before identifying which tool will best help your team's process.
Conclusion
Argo CD and the other Argo projects certainly make life easier by allowing you to automate each step for a production release, migration, and the operation workflows for your system!
By leveraging these best practices above, you're on the right track to empower your teams and apply GitOps for your infrastructure so you can deploy to prod more frequently.
I'd like to thank those who are part of the Argo community who participated and provided feedback for this blog. If you have any questions or think there are better ways to work with Argo CD, please reach out and let me know!