Verify Images

Check image signatures and add digests

Warning

Image verification is a beta feature. It is not ready for production usage and there may be breaking changes. Normal semantic versioning and compatibility rules will not apply.

Sigstore is a Linux Foundation project focused on software signing and transparency log technologies to improve software supply chain security. Cosign is a sub-project that provides image signing, verification, and storage in an OCI registry.

The Kyverno verifyImages rule uses Cosign to verify container image signatures, attestations and more stored in an OCI registry. The rule matches an image reference (wildcards are supported) and specifies a public key to be used to verify the signed image or attestations. Background scanning is also supported with this rule type which means reports will show related entries.

Verifying Image Signatures

Container images can be signed during the build phase of a CI/CD pipeline using Cosign. An image can be signed with multiple signatures, for example at the organization level and at the project level.

The policy rule check fails if the signature is not found in the OCI registry, or if the image was not signed using the specified key.

The rule also mutates matching images to add the image digest if the digest is not already specified. Using an image digest has the benefit of making image references immutable. This helps ensure that the version of the deployed image does not change and, for example, is the same version that was scanned and verified by a vulnerability scanning and detection tool.

The imageVerify rule executes as part of the mutation webhook as the applying policy may insert the image digest. The imageVerify rules execute after other mutation rules are applied but before the validation webhook is invoked. This order allows other policy rules to first mutate the image reference if necessary, for example, to replace the registry address, before the image signature is verified.

The imageVerify rule can be combined with auto-gen so that policy rule checks are applied to Pod controllers.

Here is a sample image verification policy:

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: check-image
 5spec:
 6  validationFailureAction: enforce
 7  background: false
 8  webhookTimeoutSeconds: 30
 9  failurePolicy: Fail
10  rules:
11    - name: check-image
12      match:
13        any:
14        - resources:
15            kinds:
16              - Pod
17      verifyImages:
18      - image: "ghcr.io/kyverno/test-verify-image:*"
19        key: |-
20          -----BEGIN PUBLIC KEY-----
21          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8nXRh950IZbRj8Ra/N9sbqOPZrfM
22          5/KAQN0/KjHcorm/J5yctVd7iEcnessRQjU917hmKO6JWVGHpDguIyakZA==
23          -----END PUBLIC KEY-----

This policy will validate that all images that match ghcr.io/kyverno/test-verify-image:* are signed with the specified key.

A signed image can be run as follows:

1kubectl run signed --image=ghcr.io/kyverno/test-verify-image:signed
2pod/signed created

The deployed Pod will be mutated to use the image digest.

Attempting to run an unsigned image will produce a policy error as follows:

1kubectl run unsigned --image=ghcr.io/kyverno/test-verify-image:unsigned
2Error from server: admission webhook "mutate.kyverno.svc" denied the request:
3
4resource Pod/default/unsigned was blocked due to the following policies
5
6check-image:
7  check-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:unsigned:
8    signature not found'

Similarly, attempting to run an image which matches the specified rule but is signed with a different key will produce an error:

1kubectl run signed-other --image=ghcr.io/kyverno/test-verify-image:signed-by-someone-else
2Error from server: admission webhook "mutate.kyverno.svc" denied the request:
3
4resource Pod/default/signed-other was blocked due to the following policies
5
6check-image:
7  check-image: 'image verification failed for ghcr.io/kyverno/test-verify-image:signed-by-someone-else:
8    invalid signature'

Signing images

To sign images, install Cosign and generate a public-private key pair.

1cosign generate-key-pair

Next, use the cosign sign command and specifying the private key in the -key command line argument.

1# ${IMAGE} is REPOSITORY/PATH/NAME:TAG
2cosign sign --key cosign.key ${IMAGE}

This command will sign your image and publish the signature to the OCI registry. You can verify the signature using the cosign -verify command.

1cosign verify --key cosign.pub ${IMAGE}

Refer to the Cosign documentation for usage details and OCI registry support.

Using private registries

To use a private registry, you must create an image pull secret in the Kyverno namespace and specify the secret name as an argument for the Kyverno deployment:

Configure the image pull secret:

1kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-password> --docker-email=<your-email> 
2-n kyverno

Update the Kyverno deployment to add the --imagePullSecrets=regcred argument:

 1apiVersion: apps/v1
 2kind: Deployment
 3metadata:
 4  labels:
 5    app.kubernetes.io/component: kyverno
 6   ...
 7
 8
 9spec:
10  replicas: 1
11  selector:
12    matchLabels:
13      app: kyverno
14      app.kubernetes.io/name: kyverno
15  template:
16    spec:
17      containers:
18      - args:
19        ...
20        - --webhooktimeout=15
21        - --imagePullSecrets=regcred

Using a signature repository

To use a separate registry to store signatures use the COSIGN_REPOSITORY environment variable when signing the image. Then in the Kyverno policy rule specify the repository for each image:

 1
 2...
 3
 4verifyImages:
 5- image: "ghcr.io/kyverno/test-verify-image:*"
 6  repository: "registry.io/signatures"
 7  key: |-
 8    -----BEGIN PUBLIC KEY-----
 9    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8nXRh950IZbRj8Ra/N9sbqOPZrfM
10    5/KAQN0/KjHcorm/J5yctVd7iEcnessRQjU917hmKO6JWVGHpDguIyakZA==
11    -----END PUBLIC KEY-----    
12
13...
14

Verifying Image Attestations

Container image signatures prove that the image was signed by the holder of a matching private key. However, signatures do not provide additional data and intent that frameworks like SLSA (Supply chain Levels for Software Artifacts) require.

An attestation is metadata attached to software artifacts like images. Signed attestations provide verifiable information required for SLSA.

The in-toto attestation format provides a flexible scheme for metadata such as repository and build environment details, vulnerability scan reports, test results, code review reports, or any other information that is used to verify image integrity. Each attestation contains a signed statement with a predicateType and a predicate. Here is an example derived from the in-toto site:

 1{
 2  "payloadType": "https://example.com/CodeReview/v1",
 3  "payload": {
 4    "_type": "https://in-toto.io/Statement/v0.1",
 5    "predicateType": "https://example.com/CodeReview/v1",
 6    "subject": [
 7      {
 8        "name": "registry.io/org/app",
 9        "digest": {
10          "sha256": "b31bfb4d0213f254d361e0079deaaebefa4f82ba7aa76ef82e90b4935ad5b105"
11        }
12      }
13    ],
14    "predicate": {
15      "author": "alice@example.com",
16      "repo": {
17        "branch": "main",
18        "type": "git",
19        "uri": "https://git-repo.com/org/app"
20      },
21      "reviewers": [
22        "bob@example.com"
23      ]
24    }
25  },
26  "signatures": [
27    {
28      "keyid": "",
29      "sig": "MEYCIQDtJYN8dq9RACVUYljdn6t/BBONrSaR8NDpB+56YdcQqAIhAKRgiQIFvGyQERJJYjq2+6Jq2tkVbFpQMXPU0Zu8Gu1S"
30    }
31  ]
32}

The imageVerify rule can contain one or more attestation checks that verify the contents of the predicate. Here is an example that verifies the repository URI, the branch, and the reviewers.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: attest-code-review
 5  annotations:
 6    pod-policies.kyverno.io/autogen-controllers: none
 7spec:
 8  validationFailureAction: enforce
 9  background: false
10  webhookTimeoutSeconds: 30
11  failurePolicy: fail
12  rules:
13    - name: attest
14      match:
15        any:
16        - resources:
17            kinds:
18              - Pod
19      verifyImages:
20      - image: "registry.io/org/*"
21        key: |-
22          -----BEGIN PUBLIC KEY-----
23          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEHMmDjK65krAyDaGaeyWNzgvIu155
24          JI50B2vezCw8+3CVeE0lJTL5dbL3OP98Za0oAEBJcOxky8Riy/XcmfKZbw==
25          -----END PUBLIC KEY-----          
26        attestations:
27          - predicateType: https://example.com/CodeReview/v1
28            conditions:
29              - all:
30                - key: "{{ repo.uri }}"
31                  operator: Equals
32                  value: "https://git-repo.com/org/app"            
33                - key: "{{ repo.branch }}"
34                  operator: Equals
35                  value: "main"
36                - key: "{{ reviewers }}"
37                  operator: In
38                  value: ["ana@example.com", "bob@example.com"]

The policy rule above fetches and verifies that the attestations are signed with the matching private key, decodes the payloads to extract the predicate, and then applies each condition to the predicate.

Each verifyImages rule can be used to verify signatures or attestations, but not both. This allows the flexibility of using separate signatures for attestations.

Signing attestations

To sign attestations, use the cosign attest command.

1# ${IMAGE} is REPOSITORY/PATH/NAME:TAG
2cosign attest --key cosign.key --predicate <file> --type <predicate type>  ${IMAGE}

This command will sign your attestations and publish them to the OCI registry. You can verify the attestations using the cosign verify-attestation command.

1cosign verify-attestation --key cosign.pub ${IMAGE}

Refer to the Cosign documentation for additional details including OCI registry support.

Verifying Image Annotations

Cosign has the ability to add annotations when signing and image, and Kyverno can be used to verify these annotations with the verifyImage.annotations field. For example, this policy checks for the annotation of sig: original.

 1apiVersion: kyverno.io/v1
 2kind: ClusterPolicy
 3metadata:
 4  name: check-image
 5spec:
 6  validationFailureAction: enforce
 7  background: false
 8  webhookTimeoutSeconds: 30
 9  failurePolicy: fail
10  rules:
11    - name: check-image
12      match:
13        any:
14        - resources:
15            kinds:
16              - Pod
17      verifyImages:
18      - image: "ghcr.io/kyverno/test-verify-image:*"
19        key: |-
20          -----BEGIN PUBLIC KEY-----
21          MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE8nXRh950IZbRj8Ra/N9sbqOPZrfM
22          5/KAQN0/KjHcorm/J5yctVd7iEcnessRQjU917hmKO6JWVGHpDguIyakZA==
23          -----END PUBLIC KEY-----          
24        annotations:
25          sig: "original"

Known Issues

Prometheus metrics and the Kyverno CLI are currently not supported. Check the Kyverno GitHub for a complete list of pending issues.

Last modified February 06, 2022 at 6:32 PM PST: updates for 1.6.0 (3279396)