New & Improved Package Structure: efficient OCI storage, remote & local signing, OCI import, sget deprecation

[jeff-mccoy]

Implementation is starting over at Issue #1319.
You can track this work in milestone 0.25.x

New universal package structure

This proposal seeks to define a normalized package structure for Zarf packages. In addition to providing signing without depending on cosign sget, it will also allow treating local or remote packages the same way. Additionally, this proposal discusses a new command, zarf package publish for allowing users to publish packages to a remote registry for deployment or reuse in other packages.

Context Driving Change (OCI Packaging)

• Transform structure of Zarf packages to OCI compliant artifacts
  • Current State: Packages are BLOB (binary large object)
• OCI compliant packages are native to both kubernetes and package registries.
  • Downstream platforms are converging on using OCI layers as default standard for packaging on the platform.
  • CNCF-defined standard already used.
  • Distribution tools natively support this format already to save custom work and schema for scanning and signing.
• Helps with overall distribution and allowing zarf packages being uploaded to a docker registry
• Moving to OCI helps with mission compliance and compatibility for mission partners

We know we are successful when:

1. (Priority) Users can use to Zarf can natively publish an OCI compliant Zarf package to an OCI compliant registry
2. (Secondary goal) Package creators can sign Zarf packages to enable Package deployers can trust a packages supply chain security

Package Structure Changes

• outer tarball no longer compressed
• zarf.yaml flag metadata.uncompressed now changes compression of ./components/<name>.tar.
• each component folder is now a tarball instead of a directory
• images.tar is now expanded into ./images/
• backwards compatibility is maintained via only using new capabilities if they exist:
  • if the ./images/ dir exists, it will be used instead of ./images.tar
  • if ./components/<name>.tar.? exists, it will be used instead of the ./components/<name>/ dir
  • if ./checksums.txt exists it will be processed
    • if metadata.checksumSignature is also not an empty string, it will used to verify the signature of the checksums file
  • if ./sboms/.tar.zst exists it will be used instead of the ./sboms/ dir
  • if ./zarf.yaml.sig exists it will be used to verify the signature of ./zarf.yaml

Top Level Structure

./checksums.txt
./components/<name>.tar.zst
./images/**
./sboms.tar.zst
./zarf.yaml
./zarf.yaml.sig

Old Package Structure

zarf-package-kafka-strimzi-demo-arm64.tar.zst
├── components
│   ├── baseline # -> baseline.tar.zst
│   │   ├── charts
│   │   │   └── strimzi-kafka-operator-0.29.0.tgz
│   │   ├── manifests
│   │   │   └── manifests
│   │   │       ├── kafka-topic.yaml
│   │   │       └── kafka.yaml
│   │   └── values
│   │       └── strimzi-kafka-operator-0.29.0-0
│   └── kafka-tools # -> kafka-tools.tar.zst
│       └── files
│           └── 0
├── images.tar # -> images/*
├── sboms # -> sboms.tar.zst
│   ├── compare.html
│   ├── quay.io_strimzi_kafka_0.29.0-kafka-3.2.0.json
│   ├── quay.io_strimzi_operator_0.29.0.json
│   ├── sbom-viewer-quay.io_strimzi_kafka_0.29.0-kafka-3.2.0.html
│   ├── sbom-viewer-quay.io_strimzi_operator_0.29.0.html
│   ├── sbom-viewer-zarf-component-kafka-tools.html
│   └── zarf-component-kafka-tools.json
└── zarf.yaml

New Package Structure

zarf-package-kafka-strimzi-demo-arm64.tar
├── checksums.txt # sha256sums of all files in the package except the zarf.yaml* files
├── components
│   ├── baseline.tar.zst # compressed from components/baseline
│   └── kafka-tools.tar.zst # compressed from components/kafka-tools
├── images # expanded from images.tar
│   ├── 01b1f1f1de1aa5d9a192c9308f23e3aac2f70e49b5138d20c015e903e594166f.tar.gz
│   ├── 036197be467ed17bb645dab4976d18829a54c64aad8ce4b58c50308cd1214161.tar.gz
│   ├── 374bf787cd457fd983f262c45c2dd1f1f16b4525461164d0fbfdbdd088ec9851.tar.gz
│   ├── 40c5950cb2f4d9e4a5e4f3231ffc4a80584b3a593c7a30bac482ed213e84f7f2.tar.gz
│   ├── 4b4bf95f1bc1a0d0acb9bafc4c38a1734c80b96b8a8e33464d5edeb94b50f101.tar.gz
│   ├── 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1.tar.gz
│   ├── 52c4b995397666a08ee66ad7e5b146c93a835d5e0cef9d7f43668005de0f8721.tar.gz
│   ├── 5a3cb477fb6927486e1de4005a9f6d303105318a10f3aaf20b9ac96b75b4063d.tar.gz
│   ├── 5ece2c7bdc1cb7a7d0e8fdd82a6b718c26481bdf7106a0aa9b07bfb49f15390c.tar.gz
│   ├── 607cadd9fc076f6aa4c1c08b269c986cc6de8dd2f21a4bfbaf8fd0f1ad25f0e8.tar.gz
│   ├── 676abaef5cf46c0214d6895f00c1c9de9e39e3867b1001c70233531a06f39081.tar.gz
│   ├── 6b941698d1b1d32850d0503afc02c8d0cda80973fe3fc4557f91a815d9979791.tar.gz
│   ├── 6ea53c55475943271e40630d710aa7273879ed87f51176c5aaab07760231864e.tar.gz
│   ├── 9c9d0e98f33016ea08c3ce3e8b14bbc972210c7b4d376879ff1138a4ed76e6ea.tar.gz
│   ├── a5b9f2a044430ec591cab07d63b05c59a513bc2095cc1a0794671d9a225c2ca5.tar.gz
│   ├── a73cc92a765199bc51556a56e450dd53f3240b3a1622d6095731ba0c0caad2e0.tar.gz
│   ├── b2ad865460cc2204b2e2884a0555bb41760d4449f3fe0e36bb441d740c3e9bc6.tar.gz
│   ├── cc86879c05681cf853ca3a6773d171c33549c7e70d6a5a645ba75851a4744741.tar.gz
│   ├── ccd4421c3004e570a9ca0040b297624bf11cf8e7d6a5f4c4e2f6fe3bf3af3b2e.tar.gz
│   ├── d2ca1bcb8952a0df0ef3f48ac2e1b1c829a4c8f3a50fb8c4fe115518bf901430.tar.gz
│   ├── d7fc738c2b482c640f65d249d9a992cc3884eaf92a4f7336eee81905a15e68bf.tar.gz
│   ├── eb8600330c7fda365d3580238595890e252f1f326c133c71777a0b7f1e5a13ed.tar.gz
│   ├── ef37c8e9c16fe19c428ac96ae2f254e0bbf3ab46bc5bf6ac5fa9493746bc0dfd.tar.gz
│   ├── f91a5b4860e95380f988cdc91fb3b192cec41f3527aed80364cb7df3f1e0178c.tar.gz
│   ├── manifest.json
│   ├── sha256:7af7a0f44d24587add4881175ce4328c4743e39f044e78454a243019e2f04277
│   └── sha256:bd73ec29f4295bb439cd618b35fd8ae4cceb1d6312cd45f1864e4541784ceb69
├── sboms.tar.zst # compressed from sboms/
├── zarf.yaml
└── zarf.yaml.sig # signature of zarf.yaml

Zarf Package Config (Kafka Strimzi Example)

kind: ZarfPackageConfig
metadata:
  name: "kafka-strimzi-demo"
  description: "Demo tiny Zarf Kafka deployment"
  architecture: "arm64"
  version: "v0.1.0"
build:
  terminal: "minion.local"
  user: "jeff"
  architecture: "arm64"
  timestamp: "Mon, 09 Jan 2023 12:10:43 -0600"
  version: "v0.23.2-26-gb33622f1"
  checksumSignature: "MEQCIAU4wPBpl/U5Vtdx/eJFgR0nICiiNCgyWPWarupH0onwAiAv5ycIKgztxHNVG7bzUjqHuvK2gsc4MWxwDgtDh0JINw=="

Publishing Zarf Packages

Using the new package structure detailed above, publishing to an OCI Distribution Registry will assume that the layers will be written/optimized for reuse which means that the OCI artifact will operate in lieu of a tarball. When the package is published to a registry, two different artifacts will be published: the package and a skeleton package. The skeleton package will be a copy of the package with all layers that can be pulled remotely removed. This is intended to be used to extend a package or in the case of zarf package create. The standard package will be published for directly deploying a package.

Publish a package will follow the convention <repo>/<package-name>:<version>-<arch> and <repo>/<package-name>:<version>-skeleton. Using the package example above:

Command:

zarf package publish defenseunicorns

Result:

Create and publish OCI artifacts:
- defenseunicorns/kafka-strimzi-demo:v0.1.0-arm64
- defenseunicorns/kafka-strimzi-demo:v0.1.0-skeleton

Skeleton Package

A skeleton package is a special package that is essentially a copy/paste of the local folder where the zarf.yaml is defined. Note, in this solution referencing files outside of the current zarf.yaml folder path will not work properly. One particular note is Kustomizations, they will not be processed or converted into manifests but will simply be copied into the OCI artifact for later composition if imported.

Deploying OCI Packages

Deploying will be very similar to prior sget behavior. Using the package example above:

zarf package deploy oci://defenseunicorns/kafka-strimzi-demo:v0.1.0-arm64

Alternatively, we could look at defining a new command:

zarf pacakge deploy-oci defenseunicorns/kafka-strimzi-demo:v0.1.0-arm64

Eventually the Zarf UI could include the ability to search a registry for packages and deploy or download them.

Importing OCI Packages/Components

It can be confusing, but you cannot directly import a package in Zarf. A component is an atomic unit imported/extended within a package. We refer to this as composable components. It is possible to import a package/component that is only a single component. The challenge with OCI imports of packages/components is that we will have to copy things locally before finishing the composition. Additionally, composable components today do not have a concept of special handling for remote assets already pulled into a package/local location. This is where the skeleton component proposal comes in. The skeleton package will be loaded into a temp directory and referenced like a local package/component. This will occur at the normal composition stage, before the confirm step on zarf package create.

Example OCI import

kind: ZarfPackageConfig
metadata:
  name: "demo-with-terraform-import"

components:
  - name: terraform
    import:
      oci: defenseunicorns/terraform:1.3.7

This example would then call oci://defenseunicorns/terraform:1.3.7-skeleton and include it as a part of the package definition on zarf package create. As with local imports, this would be completely transparent on zarf package deploy.

Decision: I'm not sure if we should include a checksum or signature file for skeleton files as they will be incomplete and regenerated anyway during package create.

[jeff-mccoy]

Need to add a section about signing--idea would be to focus on cosign blob signing using normal key-based signing if the user defines/provides a key to perform the signing.

[Madeline-UX]

There is value in adding the signing to the Zarf deploy UI as well. Here is an example of what I am thinking:

[runyontr]

This is great @jeff-mccoy! I had some clarifying questions about what some of the details are going to look like, sorry for they hop all over in ideas

OCI Layers

We should make a decision about whether images/artifacts within an imported zarf package should be re-generated by the parent zarf package creation process, or should copy those artifacts hosted remotely on the registry (in the non-skeleton zarf package).

Regardless, I think the ability to import artifacts from a registry may be needed for situations like air-gap builds.

Given the assumption that I'd like to be able to use the artifacts from the zarf package in my import:

• is each image.tar.gz file its own OCI layer?
• How does Zarf know which image layers it should pull for a given component?

Nested References

For the skeleton package, if my package imports a component from an OCI hosted skeleton file would my zarf package/skeleton:

1. Copy the reference as a reference to the external OCI object
2. Cache the skeleton of the referenced object inside the skeleton for the package

Skeleton Tag

If the skeleton tag was the same as the original zarf package, minus the images directory, it might be very simple to create a new tag/OCI image with no additional layers/storage being used. Or we could bypass the need for the tag regardless if the zarf.yaml (and signature) were always the last 2 (or first 2) layers in the zarf package. Zarf could then just pull those particular layers for import (and validation) without the need for a second tag.

[jeff-mccoy]

OCI Layers - this proposal doesn't change any existing behavior in Zarf. Those layers are collected either by Crane or Docker and stored in a tarball. The only change here would be to extract the layers from the tarball so we can copy them individually into the OCI artifact. By how the layers work, each layer itself will incur a single reuse penalty on the publishing registry because layers are tarred/compressed in OCI. So even though an image may have three layers published to the registry, the first time those layers are published as layers themselves, they will have their own SHA256 hash and, therefore, not actually realize reuse until another Zarf package reuses them. Needing to worry about rebuilding those layers from a local copy (besides just a raw tarball which we now support) will require a good bit more work in the codebase concerning how we parse/build image copies, and feels like a very specific edge case. You would only need this if you were attempting to do composable components across an airgap, which is something we have no support for at this time.

[jeff-mccoy]

Nested References - these will behave like any other imported asset; they will collect all the references and keep unique tmp folders for each import's local artifacts and not pull remote resources until after package confirm.

[jeff-mccoy]

Skeleton Tag - we discussed this, but it's just not a major concern. With how much is custom within components and the fact that the images are stored outside of the components directory, optimizing for that small amount of reuse doesn't provide a substantial gain. This solution also lends itself well to more atomic, capability-based units. E.g. if you have a package with components that assume terraform is available, simply importing a terraform component first as a separate component will still maximize layer reuse as the terraform component will likely be reused by other packages as well.

[bburky]

Might want to look into using the OCI Image Layout Specification for the structure of the images/ directory. It should allow you to share layers' blobs across images, but it's unclear if there's a supported way to put multiple images in the same directory(org.opencontainers.image.ref.name is supposed to be a tag, not a whole image name I think). In theory it's the standardized on disk format for OCI images, and is already compatible with some other tools.

[wirewc]

Implementation is starting over at Issue #1319.

[bburky]

The "skeleton" package looks very interesting. If I understand correctly, it would be the full package minus any images, storing references instead. It's a little unclear, would OCI imported components (oci:) also be included by reference?

Talking to the team during the Zarf Weekly Team Sync, it sounds like the (initial?) target of skeleton packages is something more like a "template" for other zarf packages (with OCI component import?), not to be used to deploy to clusters (it won't work with zarf package deploy)?

When do the referenced images actually pulled? During zarf package create when used via OCI import?

---

If images are referenced by Docker tag, there is some risk that the images may change between package and deploy. Perhaps during package resolve the images to their sha256 digest? Then at package time, you have confidence that the image did not change since package. This could be optional, maybe it's desired that the skeleton package automatically updates with tag changes?

---

I would be interested in the skeleton mode expanding to be usable for deploying to clusters too. I have been a little worried with the push towards large full stack Zarf packages, but I think this could enable some workflows to make the large package actually quite lightweight in packaged size and enable data reuse.

Often every component in a full stack does not get updated at once. Perhaps a user has some very large (10s of GB) images in one component, but a different component has been updated (maybe just BB values updated) and zarf package create is rerun. If this is a skeleton package, then the update can be incremental because the unmodified old component's OCI reference did not change and will be reused. Only when performing zarf package deploy to the cluster, would the zarf CLI finally pull all the images and components, and the images would be pushed to the cluster as usual.

This also makes some potential CI pipeline development workflows a little nicer (imagine all of the following could be independent pipelines, each reusing data from the previous):

1. Build and push images to registry
2. Create a single component zarf skeleton package referencing the images. Push it to a registry.
   • Optionally, deploy this package to a test cluster and perform tests on just this component
   • Using a skeleton package, this would be very small and not include images
3. A final pipeline could assemble multiple single component packages (with OCI import). All components and images are only included by reference.
   • This is a skeleton package that references components from other skeleton packages. The package size would be small because it contains very little data: the images or referenced OCI components are not included
4. Only when this is combined multi-component package is deployed, will all the child OCI components and images be pulled.

I still want zarf package deploy to push all the images to the cluster, I'd just like to optimize the package storage in the registry. Unlike YOLO (which also avoids copying images by not using an in-cluster registry at all?), I still want to use the in-cluster zarf docker registry. I just want to avoid copying around large packages and reuse data anywhere possible.