zbstore

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 15, 2025 License: MIT Imports: 30 Imported by: 0

Documentation

Index

Examples

Constants

View Source 
const DefaultDerivationOutputName = "out"

DefaultDerivationOutputName is the name of the primary output of a derivation. It is omitted in a number of contexts.

View Source 
const DerivationExt = ".drv"

DerivationExt is the file extension for a marshalled Derivation.

Variables

This section is empty.

Functions

func HashPlaceholder 

func HashPlaceholder(outputName string) string

HashPlaceholder returns the placeholder string used in leiu of a derivation's output path. During a derivation's realization, the backend replaces any occurrences of the placeholder in the derivation's environment variables with the temporary output path (used until the content address stabilizes).

func IsSourceContentAddress 

func IsSourceContentAddress(ca nix.ContentAddress) bool

IsSourceContentAddress reports whether the given content address describes a "source" store object. "Source" store objects are those that are hashed by their NAR serialization and do not have a fixed (non-SHA-256) hash. This typically means source files imported using the "path" function, but can also mean content-addressed build artifacts.

func IsValidOutputName 

func IsValidOutputName(name string) bool

IsValidOutputName reports whether the given string is valid as a derivation output name.

func ReceiveExport 

func ReceiveExport(receiver NARReceiver, r io.Reader) error

ReceiveExport processes a stream of NARs in `nix-store --export` format, returning the first error encountered.

ReceiveExport will not read beyond the end of the export, so there may still be data remaining in r after a call to ReceiveExport.

func Rewrite 

func Rewrite(f io.ReadWriteSeeker, baseOffset int64, newDigest string, rewriters []Rewriter) error

Rewrite applies rewriters to f. newDigest is the string to replace self-references with. The length of newDigest must be the same as the length of the digest passed to SourceSHA256ContentAddress. Rewrites are performed in the same order as they are present in the slice.

f is treated like it starts at baseOffset bytes from the beginning of a NAR serialization. This can be used to apply a subset of rewrites from SourceSHA256ContentAddress to a particular file inside the store object.

func UnknownCAOutputPlaceholder 

func UnknownCAOutputPlaceholder(ref OutputReference) string

UnknownCAOutputPlaceholder returns the placeholder for an unknown output of a content-addressed derivation.

func ValidateContentAddress 

func ValidateContentAddress(ca nix.ContentAddress, refs References) error

ValidateContentAddress checks whether the combination of the content address and set of references is one that will be accepted by a zb store. If not, it returns an error describing the issue.

Types

type ContentAddress 

type ContentAddress = nix.ContentAddress

A ContentAddress is a content-addressibility assertion.

type ContentAddressOptions 

type ContentAddressOptions struct {
	// Digest is the temporary path digest (as given by [Path.Digest])
	// Digest is used to detect self-references.
	// If the store object is known to not contain self-references,
	// Digest may be the empty string.
	Digest string
	// CreateTemp is called to create temporary storage
	// for parts of a store object that require multi-pass analysis.
	// If CreateTemp is nil, multi-pass analyses are performed in-memory.
	// This is generally not recommended, as the files can be large.
	CreateTemp bytebuffer.Creator
	// If Log is not nil, it is called to provide additional diagnostics about the analysis process.
	// The messages passed in are human-readable and should not be parsed by applications.
	Log func(string)
}

ContentAddressOptions holds optional parameters for SourceSHA256ContentAddress.

type Derivation 

type Derivation struct {
	// Dir is the store directory this derivation is a part of.
	Dir Directory

	// Name is the human-readable name of the derivation,
	// i.e. the part after the digest in the store object name.
	Name string
	// System is a string representing the OS and architecture tuple
	// that this derivation is intended to run on.
	System string
	// Builder is the path to the program to run the build.
	Builder string
	// Args is the list of arguments that should be passed to the builder program.
	Args []string
	// Env is the environment variables that should be passed to the builder program.
	Env map[string]string

	// InputSources is the set of source filesystem objects that this derivation depends on.
	InputSources sets.Sorted[Path]
	// InputDerivations is the set of derivations that this derivation depends on.
	// The mapped values are the set of output names that are used.
	InputDerivations map[Path]*sets.Sorted[string]
	// Outputs is the set of outputs that the derivation produces.
	Outputs map[string]*DerivationOutputType
}

A Derivation represents a store derivation: a single, specific, constant build action.

func ParseDerivation 

func ParseDerivation(dir Directory, name string, data []byte) (*Derivation, error)

ParseDerivation parses a derivation from ATerm format. name should be the derivation's name as returned by Path.DerivationName.

func (*Derivation) Clone 

func (drv *Derivation) Clone() *Derivation

Clone returns a deep copy of drv.

func (*Derivation) Export 

func (drv *Derivation) Export(hashType nix.HashType) ([]byte, *ExportTrailer, error)

Export marshals the derivation to a NAR containing ATerm format and computes the derivation's store metadata using the given hashing algorithm.

At the moment, the only supported algorithm is nix.SHA256.

func (*Derivation) InputDerivationOutputs 

func (drv *Derivation) InputDerivationOutputs() iter.Seq[OutputReference]

InputDerivationOutputs returns an iterator over the output references this derivation uses as inputs. The iterator will produce references in lexicographic order of the derivation path, then in lexicographic order of the output name within a derivation path.

func (*Derivation) MarshalText 

func (drv *Derivation) MarshalText() ([]byte, error)

MarshalText converts the derivation to ATerm format.

func (*Derivation) OutputPath 

func (drv *Derivation) OutputPath(outputName string) (Path, error)

OutputPath returns a fixed output's store object path. OutputPath returns an error if the output's path cannot be known ahead of realization.

func (*Derivation) References 

func (drv *Derivation) References() References

References returns the set of other store paths that the derivation references. Derivations will never have a self-reference.

type DerivationOutputType 

type DerivationOutputType struct {
	// contains filtered or unexported fields
}

A DerivationOutputType describes the content addressing scheme of an output of a Derivation.

func FixedCAOutput 

func FixedCAOutput(ca nix.ContentAddress) *DerivationOutputType

FixedCAOutput returns a DerivationOutputType that must match the given content address assertion.

func FlatFileFloatingCAOutput 

func FlatFileFloatingCAOutput(hashAlgo nix.HashType) *DerivationOutputType

FlatFileFloatingCAOutput returns a DerivationOutputType that must be a single file and will be hashed with the given algorithm. The hash will not be known until the derivation is realized.

func RecursiveFileFloatingCAOutput 

func RecursiveFileFloatingCAOutput(hashAlgo nix.HashType) *DerivationOutputType

RecursiveFileFloatingCAOutput returns a DerivationOutputType that is hashed as a NAR with the given algorithm. The hash will not be known until the derivation is realized.

func (*DerivationOutputType) FixedCA 

func (out *DerivationOutputType) FixedCA() (_ ContentAddress, ok bool)

FixedCA returns a fixed hash output's content address. ok is true only if the output was created by FixedCAOutput.

func (*DerivationOutputType) HashType 

func (t *DerivationOutputType) HashType() (_ nix.HashType, ok bool)

HashType returns the hash type of the derivation output, if present.

func (*DerivationOutputType) IsFixed 

func (t *DerivationOutputType) IsFixed() bool

IsFixed reports whether the output was created by FixedCAOutput.

func (*DerivationOutputType) IsFloating 

func (t *DerivationOutputType) IsFloating() bool

IsFloating reports whether the output's content hash cannot be known until the derivation is realized. This is true for outputs returned by FlatFileFloatingCAOutput and RecursiveFileFloatingCAOutput.

func (*DerivationOutputType) IsRecursiveFile 

func (t *DerivationOutputType) IsRecursiveFile() bool

IsRecursiveFile reports whether the derivation output uses recursive (NAR) hashing.

type Directory 

type Directory string

Directory is the absolute path of a zb store. 

const (
	// DefaultUnixDirectory is the default zb store directory
	// on Unix-like systems.
	DefaultUnixDirectory Directory = "/opt/zb/store"

	// DefaultWindowsDirectory is the default zb store directory
	// on Windows systems.
	DefaultWindowsDirectory Directory = `C:\zb\store`
)

func CleanDirectory 

func CleanDirectory(path string) (Directory, error)

CleanDirectory cleans an absolute POSIX-style or Windows-style path as a Directory. It returns an error if the path is not absolute.

func DefaultDirectory 

func DefaultDirectory() Directory

DefaultDirectory returns the default zb store directory for the running operating system. This will be one of DefaultUnixDirectory or DefaultWindowsDirectory.

func DirectoryFromEnvironment 

func DirectoryFromEnvironment() (Directory, error)

DirectoryFromEnvironment returns the zb store Directory in use based on the ZB_STORE_DIR environment variable, falling back to DefaultDirectory if not set.

func (Directory) IsNative 

func (dir Directory) IsNative() bool

IsNative reports whether the directory uses the same path style as the running operating system.

func (Directory) Join 

func (dir Directory) Join(elem ...string) string

Join joins any number of path elements to the store directory separated by the store directory's separator type.

func (Directory) Object 

func (dir Directory) Object(name string) (Path, error)

Object returns the store path for the given store object name.

func (Directory) ParsePath 

func (dir Directory) ParsePath(path string) (storePath Path, sub string, err error)

ParsePath verifies that a given absolute path begins with the store directory and names either a store object or a file inside a store object. On success, it returns the store object's name and the slash-separated relative path inside the store object, if any.

type ExportTrailer 

type ExportTrailer struct {
	StorePath      Path
	References     sets.Sorted[Path]
	Deriver        Path
	ContentAddress ContentAddress
}

ExportTrailer holds metadata about a Nix store object used in the `nix-store --export` format.

type Exporter 

type Exporter struct {
	// contains filtered or unexported fields
}

An Exporter serializes zero or more NARs to a stream in `nix-store --export` format.

func NewExporter 

func NewExporter(w io.Writer) *Exporter

NewExporter returns a new Exporter that writes to w. The caller is responsible for calling Exporter.Close on the returned exporter to finish the stream.

func (*Exporter) Close 

func (imp *Exporter) Close() error

Close writes the footer of the export to the exporter's underlying writer. Close returns an error if a store object has been written but Exporter.Trailer has not been called. Close does not close the underlying writer.

func (*Exporter) Trailer 

func (imp *Exporter) Trailer(t *ExportTrailer) error

Trailer marks the end of a store object in the stream. Subsequent calls to Exporter.Write will be part of a new store object.

func (*Exporter) Write 

func (imp *Exporter) Write(p []byte) (int, error)

Write writes bytes of a store object to the exporter's underlying writer.

type MachOSignatureRewrite 

type MachOSignatureRewrite struct {
	// ImageStart is the offset in bytes that the Mach-O file starts
	// relative to the beginning of the NAR file.
	ImageStart int64
	// CodeEnd is the offset in bytes to the first byte of the code signature section
	// relative to the beginning of the NAR file.
	// CodeEnd is also the end of the image signature range.
	CodeEnd int64
	// PageSize is the number of bytes to hash for each hash slot.
	// A value <=1 indicates "infinity",
	// meaning that the entirety of the image signature range
	// turns into a single signature.
	PageSize int
	// HashType is the hash algorithm in use.
	HashType nix.HashType
	// HashOffset is the offset in bytes of the hash slot element at index zero
	// relative to the beginning of the NAR file.
	HashOffset int64
}

MachOSignatureRewrite is a Rewriter that represents a recomputation of an ad hoc Mach-O code signature due to self references inside the executable section. A multi-architecture Mach-O file may contain multiple rewrites: one for each architecture.

func (*MachOSignatureRewrite) CodeLimit 

func (rewrite *MachOSignatureRewrite) CodeLimit() uint64

CodeLimit returns the number of bytes that need to be hashed.

func (*MachOSignatureRewrite) HashSize 

func (rewrite *MachOSignatureRewrite) HashSize() int

HashSize returns the total size of all hash slots in bytes.

func (*MachOSignatureRewrite) HashSlots 

func (rewrite *MachOSignatureRewrite) HashSlots() int

HashSlots returns the number of hashes that this rewrite will produce.

func (*MachOSignatureRewrite) IsPageSizeInfinite 

func (rewrite *MachOSignatureRewrite) IsPageSizeInfinite() bool

IsPageSizeInfinite reports whether the entirety of the image signature range turns into a single signature.

func (*MachOSignatureRewrite) ReadRange 

func (rewrite *MachOSignatureRewrite) ReadRange() (start, end int64)

ReadRange returns (rewrite.ImageStart, rewrite.CodeEnd).

func (*MachOSignatureRewrite) Rewrite 

func (rewrite *MachOSignatureRewrite) Rewrite(newDigest string, context io.Reader) ([]byte, error)

Rewrite is equivalent to rewrite.Sign(nil, context).

func (*MachOSignatureRewrite) Sign 

func (rewrite *MachOSignatureRewrite) Sign(dst []byte, code io.Reader) ([]byte, error)

Sign computes the code signature hash slots for the Mach-O image and appends them to dst. code should be a reader that starts at rewrite.ImageStart. Sign will not read more than *MachOSignatureRewrite.CodeLimit bytes from code.

func (*MachOSignatureRewrite) WriteOffset 

func (rewrite *MachOSignatureRewrite) WriteOffset() int64

WriteOffset returns rewrite.HashOffset.

type MachOUUIDRewrite 

type MachOUUIDRewrite struct {
	// ImageStart is the offset in bytes that the Mach-O file starts
	// relative to the beginning of the NAR file.
	ImageStart int64
	// UUIDStart is the offset in bytes that the Mach-O UUID starts
	// relative to the beginning of the NAR file.
	UUIDStart int64
	// CodeEnd is the offset in bytes to the first byte of the code signature section
	// relative to the beginning of the NAR file.
	// CodeEnd is also the end of the image signature range.
	CodeEnd int64
}

MachOUUIDRewrite is a Rewriter that replaces the LC_UUID command value with one based on the hash of the data from ImageStart to CodeEnd.

func (*MachOUUIDRewrite) ReadRange 

func (rewrite *MachOUUIDRewrite) ReadRange() (start, end int64)

ReadRange returns (rewrite.ImageStart, rewrite.CodeEnd).

func (*MachOUUIDRewrite) Rewrite 

func (rewrite *MachOUUIDRewrite) Rewrite(newDigest string, context io.Reader) ([]byte, error)

Rewrite is equivalent to rewrite.Sign(nil, context).

func (*MachOUUIDRewrite) WriteOffset 

func (rewrite *MachOUUIDRewrite) WriteOffset() int64

WriteOffset returns rewrite.UUIDStart.

type NARReceiver 

type NARReceiver interface {
	io.Writer
	ReceiveNAR(trailer *ExportTrailer)
}

A type that implements NARReceiver processes multiple NAR files. After the NAR file has been written to the receiver, ReceiveNAR is called to provide metadata about the written NAR file. Subsequent writes will be for a new NAR file. If the Write method of a NARReceiver returns an error, the NARReceiver should not receive further calls.

type OutputReference 

type OutputReference struct {
	DrvPath    Path
	OutputName string
}

OutputReference is a reference to an output of a derivation.

func ParseOutputReference 

func ParseOutputReference(s string) (OutputReference, error)

ParseOutputReference parses the result of OutputReference.String back into an OutputReference.

Example 
package main

import (
	"fmt"

	"zb.256lights.llc/pkg/zbstore"
)

func main() {
	ref, err := zbstore.ParseOutputReference("/opt/zb/store/ib3sh3pcz10wsmavxvkdbayhqivbghlq-hello-2.12.1.drv!out")
	if err != nil {
		panic(err)
	}
	fmt.Println(ref.DrvPath)
	fmt.Println(ref.OutputName)
}

Output:

/opt/zb/store/ib3sh3pcz10wsmavxvkdbayhqivbghlq-hello-2.12.1.drv
out

func (OutputReference) IsZero 

func (ref OutputReference) IsZero() bool

IsZero reports whether the reference is the zero value.

func (OutputReference) MarshalText 

func (ref OutputReference) MarshalText() ([]byte, error)

MarshalText returns the output reference in the same format as OutputReference.String.

func (OutputReference) String 

func (ref OutputReference) String() string

String returns the path and the output name separated by "!".

func (*OutputReference) UnmarshalText 

func (ref *OutputReference) UnmarshalText(text []byte) error

UnmarshalText parses the output reference like ParseOutputReference into ref.

type Path 

type Path string

Path is a zb store path: the absolute path of a zb store object in the filesystem. For example: "/opt/zb/store/s66mzxpvicwk07gjbjfw9izjfa797vsw-hello-2.12.1" or "C:\zb\store\s66mzxpvicwk07gjbjfw9izjfa797vsw-hello-2.12.1".

func FixedCAOutputPath 

func FixedCAOutputPath(dir Directory, name string, ca nix.ContentAddress, refs References) (Path, error)

FixedCAOutputPath computes the path of a store object with the given directory, name, content address, and reference set.

func ParsePath 

func ParsePath(path string) (Path, error)

ParsePath parses an absolute path as a store path (i.e. an immediate child of a zb store directory).

func (Path) Base 

func (path Path) Base() string

Base returns the last element of the path.

func (Path) DerivationName 

func (path Path) DerivationName() (drvName string, isDrv bool)

DerivationName returns Path.Name with a suffix of DerivationExt stripped. If the path does not end in DerivationExt, DerivationName returns ("", false).

func (Path) Digest 

func (path Path) Digest() string

Digest returns the digest part of the last element of the path.

func (Path) Dir 

func (path Path) Dir() Directory

Dir returns the path's directory.

func (Path) IsDerivation 

func (path Path) IsDerivation() bool

IsDerivation reports whether the name ends in DerivationExt.

func (Path) IsNative 

func (path Path) IsNative() bool

IsNative reports whether the path uses the same path style as the running operating system.

func (Path) Join 

func (path Path) Join(elem ...string) string

Join joins any number of path elements to the store path separated by the store path's separator type.

func (Path) MarshalText 

func (path Path) MarshalText() ([]byte, error)

MarshalText returns a byte slice of the path or an error if it's empty.

func (Path) Name 

func (path Path) Name() string

Name returns the part of the last element of the path after the digest, excluding the separating dash.

func (*Path) UnmarshalText 

func (path *Path) UnmarshalText(data []byte) error

UnmarshalText validates and cleans the path in the same way as ParsePath and stores it into *path.

type References 

type References struct {
	// Self is true if the store object contains one or more references to itself.
	Self bool
	// Others holds paths of other store objects that the store object references.
	Others sets.Sorted[Path]
}

References represents a set of references to other store paths that a store object contains for the purpose of generating a Path. The zero value is an empty set.

func MakeReferences 

func MakeReferences(self Path, refSet *sets.Sorted[Path]) References

MakeReferences converts a set of complete store paths into a References value.

func (References) IsEmpty 

func (refs References) IsEmpty() bool

IsEmpty reports whether refs represents the empty set.

func (References) ToSet 

func (refs References) ToSet(self Path) *sets.Sorted[Path]

ToSet converts the references to a set of paths given the store object's own path.

type Rewriter 

type Rewriter interface {
	// Rewrite returns the bytes to replace in the store object
	// starting at WriteOffset.
	// context is a reader over the bytes in the store object in ReadRange.
	Rewrite(newDigest string, context io.Reader) ([]byte, error)
	// WriteOffset returns the offset from the beginning of the NAR file
	// of the first byte that needs to be rewritten.
	WriteOffset() int64
	// ReadRange returns the offset from the beginning of the NAR file
	// of the first byte that needs to be read
	// to the last byte (exclusive) that needs to be read
	// to compute the rewrite.
	// If no context is required to compute the rewrite,
	// ReadOffset shall return the same value as WriteOffset for both start and end.
	// If ReadRange overlaps with WriteOffset,
	// the content of the bytes at WriteOffset are not defined and should be ignored.
	ReadRange() (start, end int64)
}

A Rewriter represents a modification to a store object to account for self-references. It must be safe to call methods on a Rewriter from multiple goroutines concurrently.

type SelfReference 

type SelfReference interface {
	Rewriter

	// AppendReferenceText appends the textual representation of the self-reference location
	// as used in computing the content address to dst.
	// Such strings must not contain a "|" character.
	AppendReferenceText(dst []byte) ([]byte, error)
}

SelfReference is a Rewriter that represents a self-reference.

type SelfReferenceAnalysis 

type SelfReferenceAnalysis struct {
	// Rewrites is the sequence of rewrites for the NAR serialization
	// required to account for self-reference digests.
	Rewrites []Rewriter
	// Paths is the set of paths in the NAR serialization that contain self-reference digests.
	// They should be in ascending order of ContentOffset.
	Paths []nar.Header
}

SelfReferenceAnalysis holds additional information about self-references computed by SourceSHA256ContentAddress.

func SourceSHA256ContentAddress 

func SourceSHA256ContentAddress(sourceNAR io.Reader, opts *ContentAddressOptions) (nix.ContentAddress, *SelfReferenceAnalysis, error)

SourceSHA256ContentAddress computes the content address of a "source" store object from its NAR serialization. See IsSourceContentAddress for an explanation of "source" store objects.

func (*SelfReferenceAnalysis) HasSelfReferences 

func (analysis *SelfReferenceAnalysis) HasSelfReferences() bool

HasSelfReferences reports whether the analysis is non-empty.

func (*SelfReferenceAnalysis) Path 

func (analysis *SelfReferenceAnalysis) Path(name string) *nar.Header

Path returns the header for the given path.

func (*SelfReferenceAnalysis) RewritesInRange 

func (analysis *SelfReferenceAnalysis) RewritesInRange(start, end int64) []Rewriter

RewritesInRange returns a slice of analysis.Rewrites whose WriteOffset values are in the range [start, end).

type SelfReferenceOffset 

type SelfReferenceOffset int64

A SelfReferenceOffset is a SelfReference that represents a simple textual reference to the store path's digest. It is stored as an offset in bytes relative to the beginning of the NAR file.

func (SelfReferenceOffset) AppendReferenceText 

func (offset SelfReferenceOffset) AppendReferenceText(dst []byte) ([]byte, error)

AppendReferenceText implements SelfReference by appending the decimal representation of the offset to dst.

func (SelfReferenceOffset) ReadRange 

func (offset SelfReferenceOffset) ReadRange() (start, end int64)

ReadRange returns (int64(offset), int64(offset)).

func (SelfReferenceOffset) Rewrite 

func (offset SelfReferenceOffset) Rewrite(newDigest string, context io.Reader) ([]byte, error)

Rewrite returns newDigest.

func (SelfReferenceOffset) WriteOffset 

func (offset SelfReferenceOffset) WriteOffset() int64

WriteOffset returns the offset as an int64.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.