Secure and Simplify Kubernetes Secrets Management with External Secrets

In the world of Kubernetes, managing secrets securely is vital. Kubernetes secrets are essential for storing sensitive data like database credentials, API keys, and certificates. However, handling these secrets directly within Kubernetes presents challenges around security, accessibility, and synchronization. This is where the External Secrets Operator (ESO) comes into play. By integrating with external secret management systems, ESO allows Kubernetes to securely access and update secrets automatically, providing both security and efficiency.

This post explores ESO’s architecture, key components, and its value from an SRE perspective. We’ll also go through a practical demo, setting up an EKS cluster via OpenTofu, deploying ESO, and securely managing database secrets stored in AWS Secret Manager.

Why Use External Secrets Operator?

In any production environment, securely managing secrets is non-negotiable. For SREs, securely propagating secrets across clusters and keeping them in sync without manual intervention is critical. The External Secrets Operator (ESO) addresses these needs by:

• Centralizing Secret Management: Integrates external secret managers like AWS Secret Manager or HashiCorp Vault, centralizing secret storage for secure access.
• Automating Secret Synchronization: Keeps Kubernetes secrets automatically updated, reducing manual handling.
• Enhancing Security with External Stores: Prevents direct exposure of secrets in Kubernetes and aligns with industry best practices for secret management.

ESO Architecture and Core Components

The External Secrets Operator extends Kubernetes with custom resources that define where secrets are stored and how to synchronize them. Let’s break down its main components and their benefits for SREs:

SecretStore

The SecretStore is a namespaced resource in Kubernetes that defines the connection to an external secrets manager, specifying how to authenticate. It’s ideal for SREs managing different secrets in isolated namespaces. It handles the credentials and access details required to connect to external APIs (e.g., AWS Secrets Manager, Google Secret Manager).

Example configuration:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-store
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-north-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: aws-creds
            key: access-key
          secretAccessKeySecretRef:
            name: aws-creds
            key: secret-access-key

ClusterSecretStore

The ClusterSecretStore is a cluster-scoped version of SecretStore, providing a central gateway to external secrets for all namespaces in the cluster. This resource supports integration with multiple secret providers, allowing each namespace to access a shared secret backend while maintaining isolation. It’s ideal for multi-namespace applications, enabling centralized but controlled access across multiple environments.

Example configuration:

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: example
spec:
  provider:
    aws:
      service: SecretsManager
      region: eu-north-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key
    vault:
      server: "https://vault.acme.org"
      path: "secret"
      version: "v2"
      auth:
        tokenSecretRef:
          name: "my-secret"
          namespace: "secret-admin"
          key: "vault-token"
  conditions:
    - namespaces:
        - "namespace-a"
        - "namespace-b"

ExternalSecret

The ExternalSecret specifies the data to fetch from an external provider, using secretStoreRef to access the appropriate SecretStore or ClusterSecretStore. This resource allows the operator to sync secrets into Kubernetes as per the specified configuration.

Example configuration:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-store
    kind: SecretStore
  target:
    name: db-credentials-k8s
    creationPolicy: Owner
  data:
    - secretKey: username
      remoteRef:
        key: my-database-secret
        property: username
    - secretKey: password
      remoteRef:
        key: my-database-secret
        property: password

ClusterExternalSecret

ClusterExternalSecret is a cluster-scoped resource that manages ExternalSecret resources in specific namespaces. It allows SREs to centrally manage secrets and propagate them across namespaces using selectors.

Example configuration:

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: hello-world
spec:
  externalSecretName: hello-world-es
  namespaceSelectors:
    - matchLabels:
        team: "devops"
  refreshTime: "1m"
  externalSecretSpec:
    secretStoreRef:
      name: example-store
      kind: ClusterSecretStore
    target:
      name: my-secret
      creationPolicy: "Merge"
    data:
      - secretKey: db-user
        remoteRef:
          key: db-credentials
          property: user

PushSecret

The PushSecret allows Kubernetes secrets to be pushed to an external secret manager. It uses a selector to identify the source secret, making it useful for syncing changes from Kubernetes to external secret stores.

Example configuration:

apiVersion: external-secrets.io/v1alpha1
kind: PushSecret
metadata:
  name: pushsecret-example
  namespace: default
spec:
  updatePolicy: Replace
  deletionPolicy: Delete
  refreshInterval: 10s
  secretStoreRefs:
    - name: aws-parameterstore
      kind: SecretStore
  selector:
    secret:
      name: source-secret

Security and Access Control in ESO

ESO supports fine-grained control over secrets, crucial for regulatory compliance and multi-tenant environments:

• Least Privilege Access: Define strict IAM policies to limit ESO’s access to only required secrets, ensuring regulatory compliance with standards like SOC2 and ISO 27001.
• Namespace Isolation: Isolate secrets based on namespaces, limiting access to applications that need them.
• Audit and Monitoring: Regularly audit ESO’s access and monitor for unauthorized changes.

For multi-tenant setups, using multiple controllers within a cluster allows each controller to handle specific namespaces or ClusterSecretStores. This helps enforce security policies by limiting which namespaces or applications can access specific secrets.

Conclusion

The External Secrets Operator is a powerful tool for securely managing secrets in Kubernetes environments. It centralizes secret storage, reduces manual management, and enhances security by integrating with AWS Secrets Manager, HashiCorp Vault, and other providers. From an SRE perspective, ESO automates secrets management across clusters, making it scalable and secure.

For full configurations, check out my GitHub repository. By adopting ESO, SRE teams can streamline secret management and focus on delivering resilient, secure applications. Try ESO and experience secure, automated secret management in Kubernetes!