Skip to main content

Managing Credentials

Kargo Warehouses frequently require read-only access to private repositories. Promotion processes often require this as well, and may also require read/write access to Git repositories.

This section focuses on an operator's role in providing Kargo Projects with necessary credentials.

info

Not what you were looking for?

If you're a user looking to learn more about managing credentials at the project level, refer instead to the Managing Credentials section of the User's Guide.

info

Whether you're installing Kargo using Helm or via Argo CD, the next two sections assume familiarity with procedures for configuring that installation.

Repository Credentials as Secret Resources​

Kargo expects repository credentials it uses to have been stored as specially labeled Kubernetes Secret resources containing specially-formatted data. These Secrets generally take the following form:

apiVersion: v1
kind: Secret
metadata:
  name: <name>
  namespace: <project namespace>
  labels:
    kargo.akuity.io/cred-type: <type>
stringData:
  repoURL: <repo url>
  username: <username>
  password: <password>

The names of Secret resources are inconsequential because Kargo matches credentials to repositories by repository type and URL. Secret names may therefore observe any naming convention preferred by the user.

The label key kargo.akuity.io/cred-type and its value, one of git, helm, image, or generic is important, as it designates the Secret as representing credentials for a Git, Helm chart, or container image repository, or something else, respectively.

Secrets labeled as git, image, or helm credentials must generally contain the following keys:

  • repoURL:

    • The full URL of the repository the credentials are for.

    OR

    • A regular expression matching the URLs of multiple repositories for which the credentials may be used, with the repoURLIsRegex key additionally set to true.

      info

      This is useful if, for example, your project accesses many GitHub repositories, all beginning with https://github.com/example-org, and can use the same token for accessing all of them.

  • Either:

    • username: The username to use when authenticating to the repository.

    • password: A password or personal access token.

      info

      If the value of the password key is a personal access token, the value of the username field is often inconsequential. You should consult your repository's documentation for more information.

    OR:

    • sshPrivateKey: A PEM-encoded SSH private key. Applicable only to Git repositories using SSH-style URLs -- for instance git@github.com:example/repo.git.

Global Credentials​

Credentials are generally managed at the project level by project admins, but in cases where one or more sets of credentials are needed widely across many or all Kargo projects, an operator may opt into designating one or more namespaces as containing "global" credentials, accessible to all projects. It is then the operator's responsibility to create and manage such credentials as well.

When Kargo searches for repository credentials, these additional namespaces are searched only after finding no matching credentials in the project's own namespace.

note

Precedence

When Kargo searches for repository credentials in a "global" namespace, it first iterates over all appropriately labeled Secrets without repoURLIsRegex set to true looking for a repoURL value matching the repository URL exactly.

Only if no exact match is found does it iterate over all appropriately labeled Secrets with repoURLIsRegex set to true looking for a regular expression matching the repository URL.

When searching for an exact match, and then again when searching for a pattern match, appropriately labeled Secrets are considered in lexical order by name.

info

Because Kargo matches credentials to repositories by repository type and URL, users do not need to be informed of the details (e.g. names) of any global credentials, except possibly that they exist.

Enabling Global Credentials​

To designate one or more namespaces as containing "global" credentials, list them under the Kargo Helm chart's controller.globalCredentials.namespaces option at installation time.

Operators must also manually ensure Kargo controllers receive read-only access to Secrets in the designated namespaces. For example, if kargo-global-creds is designated as a global credentials namespace, the following RoleBinding should be created within that namespace:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
    name: kargo-controller-read-secrets
    namespace: kargo-global-creds
subjects:
    - kind: ServiceAccount
      name: kargo-controller
      namespace: kargo
roleRef:
    kind: ClusterRole
    name: kargo-controller-read-secrets
    apiGroup: rbac.authorization.k8s.io
note

The kargo-controller-read-secrets ClusterRole is predefined by the Kargo Helm chart and grants read-only access to Secret resources.

info

By default, Kargo controllers lack cluster-wide permissions on Secret resources. Instead, the Kargo management controller dynamically expands and contracts controller access to Secrets on a namespace-by-namespace basis as new Projects are created and deleted.

It is because this process does not account for "global" credential namespaces that these bindings must be created manually by an operator.

warning

Setting the controller.serviceAccount.clusterWideSecretReadingEnabled option to true at installation will grant Kargo controllers cluster-wide read permission on Secret resources.

This is highly discouraged, especially in sharded environments where this permission would have the undesirable effect of granting remote Kargo controllers read permissions on all Secrets throughout the Kargo control plane's cluster -- including Secrets having nothing to do with Kargo.

Ambient Credentials​

This section provides guidance on configuring Kargo and various cloud platforms to support "ambient" credentials. Kargo users are presumed not to have sufficient access to those platform to configure these options themselves, so this section is intended for operators and cloud platform administrators.

Amazon Elastic Container Registry (ECR)​

Kargo can be configured to authenticate to ECR repositories using EKS Pod Identity or IAM Roles for Service Accounts (IRSA).

If Kargo locates no Secret resources matching a repository URL and is deployed within an EKS cluster, it will attempt to use EKS Pod Identity or IAM Roles for Service Accounts (IRSA) to authenticate. Leveraging either eliminates the need to store ECR credentials in a Secret resource.

Follow this overview to set up EKS Pod Identity in your EKS cluster or this one to set up IRSA. For either, you will assign an IAM role to the kargo-controller ServiceAccount within the namespace in which Kargo is (or will be) installed.

note

To use IRSA, you will additionally need to specify the ARN of the controller's IAM role as the value of the controller.serviceAccount.iamRole setting in Kargo's Helm chart at installation.

At this point, an IAM role will be associated with the Kargo controller, however, that controller acts on behalf of multiple Kargo projects, each of which may require access to different ECR repositories. To account for this, when Kargo attempts to access an ECR repository on behalf of a specific project, it will first attempt to assume an IAM role specific to that project. The name of the role it attempts to assume will always be of the form kargo-project-<project name>. It is this role that should be granted read-only access to applicable ECR repositories.

info

The name of the IAM role associated with each Kargo project is deliberately not configurable to prevent project admins from attempting to coerce Kargo into assuming arbitrary IAM roles.

caution

For optimal adherence to the principle of least privilege, the IAM role associated with the kargo-controller ServiceAccount should be limited only to the ability to assume project-specific IAM roles. Project-specific IAM roles should be limited only to read-only access to applicable ECR repositories.

info

If the Kargo controller is unable to assume a project-specific IAM role, it will fall back to using its own IAM role directly. For organizations without strict tenancy requirements, this can eliminate the need to manage a large number of project-specific IAM roles. While useful, this approach is not strictly recommended.

Tokens Kargo obtains for accessing any specific ECR repository on behalf of any specific Kargo project, are valid for 12 hours and cached for 10. A controller restart clears the cache.

Google Artifact Registry (GAR)​

Kargo can be configured to authenticate to Google Artifact Registry (GAR) repositories using Workload Identity Federation (WIF).

If Kargo locates no Secret resources matching a repository URL, and if Kargo is deployed within a Google Kubernetes Engine (GKE) cluster with WIF enabled, it will attempt to use it to authenticate. Leveraging this option eliminates the need to store credentials in a Secret resource. WIF can be enabled when creating a new cluster or can be added to an existing cluster.

note

Clusters managed by GKE Autopilot have WIF enabled automatically.

With WIF enabled, GCP Identity and Access Management (IAM) automatically understands a principal identifier of the following form to be a reference to the Kargo controller's Kubernetes Service Account (KSA):

principal://iam.googleapis.com/projects/<gcp project number>/locations/global/workloadIdentityPools/<gcp project name>.svc.id.goog/subject/ns/<kargo namespace>/sa/kargo-controller
note

There is no need to annotate the Kargo controller's KSA in any specific way to enable the above.

Because the Kargo controller acts on behalf of multiple Kargo projects, each of which may require access to different GAR repositories, when accessing a repository on behalf of a given project, it will attempt to impersonate a project-specific Google Service Account (GSA). The name of the GSA that the controller will attempt to impersonate will always be of the form kargo-project-<kargo project name>@<gcp project name>.iam.gserviceaccount.com.

info

The name of the GSA associated with each Kargo project is deliberately not configurable to prevent project admins from attempting to coerce Kargo into impersonating arbitrary GSAs.

To enabled this, each project-specific GSA must:

  • Have an IAM policy that permits the Kargo controller's KSA to impersonate the GSA by creating a token (roles/iam.serviceAccountTokenCreator).

  • Be granted read-only access (roles/artifactregistry.reader) to the specific GAR repositories with which it interacts.

caution

Following the principle of least privilege, the IAM principal associated with the Kargo controller's GSA should be granted no permissions beyond the ability to impersonate project-specific GSAs.

note

Beginning with Kargo v1.5.0, if maintaining a separate GSA for every Kargo project is deemed too onerous and strict adherence to the principle of least privilege is not a concern, permissions may be granted directly to the Kargo controller's KSA. In the event that a project-specific GSA does not exist or cannot be impersonated, Kargo will fall back on using the controller's KSA directly to access GAR repositories. While useful, this approach is not strictly recommended.

Tokens Kargo obtains for accessing any specific GAR repository on behalf of any specific Kargo project, are valid for 60 minutes and cached for 40. A controller restart clears the cache.

Azure Container Registry (ACR)​

Kargo can be configured to authenticate to ACR repositories using Azure Workload Identity.

If Kargo locates no Secret resources matching a repository URL and is deployed within an AKS cluster with workload identity enabled, it will attempt to use it to authenticate. Leveraging this eliminates the need to store ACR credentials in a Secret resource. Workload Identity can be enabled when creating a new cluster or can be added to an existing cluster.

danger

Azure Workload Identity can be complex to configure and difficult to troubleshoot.

Before continuing, be certain of the following:

  • Your AKS cluster has the OIDC Issuer feature enabled.
  • Your AKS cluster has the Workload Identity feature enabled.

For Workload Identity to work, the Kargo controller's Kubernetes ServiceAccount will need to be federated with a managed identity. Follow these instructions to create one and these to federate it with the controller's ServiceAccount.

info

Federating the managed identity to the Kargo controller's ServiceAccount establishes a trust relationship. In AKS clusters with Workload Identity enabled, a mutating admission webhook will intercept the creation of any Pod resource labeled with azure.workload.identity/use: "true" and using a ServiceAccount that's been federated to a managed identity. Knowing such a Pod is authorized to act on behalf of the associated managed identity, the webhook will modify the Pod's spec to inject credentials in a well-known location for discovery by any Azure clients executing within any of its containers.

To access container images or Helm charts hosted in ACR, the managed identity must be granted the AcrPull role on the registry or on individual repositories within it.

danger

Before continuing, be certain of the following:

  • You have created a User-Assigned Managed Identity.

    ⚠️ This is different from an App Registration!

  • You have created a Federated Identity Credential that associates the managed identity with the Kubernetes ServiceAccount used by the Kargo controller. (In a typical installation of Kargo, this is the kargo-controller ServiceAccount in the kargo namespace.)

  • The managed identity has been granted the AcrPull role on your ACR registry or specific repositories within it.

For Workload Identity to inject credentials into any Pod, two specific Kargo configuration settings are required:

  1. Controller Pods must be labeled with azure.workload.identity/use: "true".

    This label can be affixed to Kargo controller Pods by using the controller.podLabels setting in Kargo's Helm chart at the time of installation or upgrade.

  2. The controller's ServiceAccount must be annotated with azure.workload.identity/client-id: <managed identity client id>.

    warning

    Azure documentation states this annotation is optional, however, in practice, it often is required.

    This annotation can be affixed to the Kargo controller's ServiceAccount by using the controller.serviceAccount.annotations setting in Kargo's Helm chart at the time of installation or upgrade.

Example Helm values:

controller:
  podLabels:
    azure.workload.identity/use: "true"
  serviceAccount:
    annotations:
      azure.workload.identity/client-id: <managed identity client id>
info

For further guidance on this, refer to the advanced installation guides for Helm or Argo CD

warning

If the azure.workload.identity/use: "true" label is present on the Kargo controller's Pod and the azure.workload.identity/client-id annotation is also present on the Kargo controller's ServiceAccount, but the Pod was started prior to Workload Identity having been enabled in the cluster or prior to the controller's ServiceAccount having been federated with a managed identity, the Pod will not have been injected with necessary credentials. Such a Pod should be deleted. The controller's Deployment will create a replacement Pod which will be injected with necessary credentials.

caution

For optimal adherence to the principle of least privilege, the managed identity associated with the kargo-controller ServiceAccount should be limited only to the AcrPull role on the specific ACR repositories required by your Kargo projects.

Tokens Kargo obtains for accessing any specific ACR repository are valid for approximately 3 hours and cached for 2.5 hours. A controller restart clears the cache.

note

When authenticating to ECR using EKS Pod Identity or IRSA (Amazon), or when authenticating to GAR using Workload Identity Federation (Google), the option exists for strict adherence to the the principle of least privilege by granting the identity associated with the Kargo controller no permissions other than those required to assume/impersonate other, Project-specific identities. Project-specific identities can then be granted access only to the specific registries or repositories.

Assuming/impersonating a project-specific identity in Azure is considerably more complex than doing so in AWS or GCP. As a result, the Kargo controller lacks the option described above for Azure Workload Identity / ACR.