filesystem

package
v0.0.0-...-82353ca Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 14, 2026 License: Apache-2.0 Imports: 31 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func CreateChunkDiscardingFileMerkleTree 

func CreateChunkDiscardingFileMerkleTree(ctx context.Context, parameters *FileCreationParameters, f filesystem.FileReader) (model_core.PatchedMessage[*model_filesystem_pb.FileContents, dag.ObjectContentsWalker], error)

CreateChunkDiscardingFileMerkleTree is a helper function for creating a Merkle tree of a file and immediately constructing an ObjectContentsWalker for it. This function takes ownership of the file that is provided. Its contents may be re-read when the ObjectContentsWalker is accessed, and it will be released when no more ObjectContentsWalkers for the file exist.

func CreateDirectoryMerkleTree 

func CreateDirectoryMerkleTree[TDirectory, TFile model_core.ReferenceMetadata](
	ctx context.Context,
	concurrency *semaphore.Weighted,
	group *errgroup.Group,
	directoryParameters *DirectoryCreationParameters,
	directory CapturableDirectory[TDirectory, TFile],
	capturer DirectoryMerkleTreeCapturer[TDirectory, TFile],
	out *CreatedDirectory[TDirectory],
) error

CreateDirectoryMerkleTree creates a Merkle tree that corresponds to the contents of a given directory. Upon success, a Directory message corresponding with the root directory is returned.

Computation of the Merkle trees of the individual files can be done in parallel. These processes may terminate asynchronously, meaning that group.Wait() needs to be called to ensure that the capturer is invoked for all objects, and the output message is set.

func CreateFileMerkleTree 

func CreateFileMerkleTree[T model_core.ReferenceMetadata](ctx context.Context, parameters *FileCreationParameters, f io.Reader, capturer FileMerkleTreeCapturer[T]) (model_core.PatchedMessage[*model_filesystem_pb.FileContents, T], error)

CreateFileMerkleTree creates a Merkle tree structure that corresponds to the contents of a single file. If a file is small, it stores all of its contents in a single object. If a file is large, it creates a B-tree.

Chunking of large files is performed using the MaxCDC algorithm. The resulting B-tree is a Prolly tree. This ensures that minor changes to a file also result in minor changes to the resulting Merkle tree.

TODO: Change this function to support more efficient creation of Merkle trees for sparse files.

func DirectoryGetContents 

func DirectoryGetContents[TReference any](
	ctx context.Context,
	reader model_parser.MessageObjectReader[TReference, *model_filesystem_pb.DirectoryContents],
	directory model_core.Message[*model_filesystem_pb.Directory, TReference],
) (model_core.Message[*model_filesystem_pb.DirectoryContents, TReference], error)

DirectoryGetContents is a helper function for obtaining the contents of a directory.

func DirectoryGetLeaves 

func DirectoryGetLeaves[TReference any](
	ctx context.Context,
	reader model_parser.MessageObjectReader[TReference, *model_filesystem_pb.Leaves],
	directory model_core.Message[*model_filesystem_pb.DirectoryContents, TReference],
) (model_core.Message[*model_filesystem_pb.Leaves, TReference], error)

DirectoryGetLeaves is a helper function for obtaining the leaves contained in a directory.

func FileContentsEntryToProto 

func FileContentsEntryToProto[TReference object.BasicReference](
	entry *FileContentsEntry[TReference],
) model_core.PatchedMessage[*model_filesystem_pb.FileContents, dag.ObjectContentsWalker]

FileContentsEntryToProto converts a FileContentsEntry back to a Protobuf message.

TODO: Should this function take a model_core.ExistingObjectCapturer?

func GetDirectoryWithContents 

func GetDirectoryWithContents[TMetadata model_core.ReferenceMetadata](
	createdDirectory *CreatedDirectory[TMetadata],
	externalObject *model_core.Decodable[model_core.MetadataEntry[TMetadata]],
	patcher *model_core.ReferenceMessagePatcher[TMetadata],
) *model_filesystem_pb.Directory

GetDirectoryWithContents creates a Directory message that either contains or references a DirectoryContents message. This function is typically called as part of inlinedtree.Build(), as that function makes the decision whether there is enough space to inline the contents of a directory.

func NewCapturedDirectoryWalker 

func NewCapturedDirectoryWalker(directoryParameters *DirectoryAccessParameters, fileParameters *FileCreationParameters, rootDirectory CapturedDirectory, rootObject *model_core.CreatedObjectTree, decodingParameters []byte) dag.ObjectContentsWalker

NewCapturedDirectoryWalker returns an implementation of ObjectContentsWalker that is capable of walking over a hierarchy of Directory and Leaves objects that were created using CreateDirectoryMerkleTree and captured using FileDiscardingDirectoryMerkleTreeCapturer. This makes it possible to upload such directory hierarchies to a storage server.

These Merkle trees do not contain any file contents, but it is permitted for the storage server to request them. If that happens, we must reobtain them from the underlying file system. This is why the caller must provide a handle to the root directory on which the provided Merkle tree is based.

func NewCapturedFileWalker 

func NewCapturedFileWalker(fileParameters *FileCreationParameters, r filesystem.FileReader, fileReference object.LocalReference, fileSizeBytes uint64, fileObject *model_core.CreatedObjectTree, decodingParameters []byte) dag.ObjectContentsWalker

NewCapturedFileWalker returns an implementation of ObjectContentsWalker that is capable of walking over a hierarchy of file contents list and chunk objects that were created using CreateFileMerkleTree and captured using ChunkDiscardingFileMerkleTreeCapturer. This makes it possible to upload such files to a storage server.

These Merkle trees only contain file contents lists and no chunks, but it is permitted for the storage server to request chunks. If that happens, we must reobtain them from the underlying file system. This is why the caller must provide a handle to the file on which the provided Merkle tree is based.

func NewDirectoryClusterObjectParser 

func NewDirectoryClusterObjectParser[TReference any]() model_parser.MessageObjectParser[TReference, DirectoryCluster]

NewDirectoryClusterObjectParser creates an ObjectParser that is capable of parsing directory objects, and exposing them in the form of a DirectoryCluster.

func NewFileContentsListObjectParser 

func NewFileContentsListObjectParser[TReference object.BasicReference]() parser.ObjectParser[TReference, FileContentsList[TReference]]

NewFileContentsListObjectParser creates an ObjectParser that is capable of parsing FileContentsList messages, turning them into a list of entries that can be processed by FileContentsIterator.

Types

type CapturableDirectory 

type CapturableDirectory[TDirectory, TFile model_core.ReferenceMetadata] interface {
	// Identical to filesystem.Directory.
	Close() error
	ReadDir() ([]filesystem.FileInfo, error)
	Readlink(name path.Component) (path.Parser, error)

	// Enter a directory, so that it may be traversed. The
	// implementation has the possibility to return an existing
	// Directory message. This can be of use when computing a Merkle
	// tree that is based on an existing directory structure that is
	// altered slightly.
	EnterCapturableDirectory(name path.Component) (*CreatedDirectory[TDirectory], CapturableDirectory[TDirectory, TFile], error)

	// Open a file, so that a Merkle tree of its contents can be
	// computed. The actual Merkle tree computation is performed by
	// calling CapturableFile.CreateFileMerkleTree(). That way file
	// Merkle tree computation can happen in parallel.
	OpenForFileMerkleTreeCreation(name path.Component) (CapturableFile[TFile], error)
}

CapturableDirectory is an interface for a directory that can be traversed by CreateFileMerkleTree().

type CapturableFile 

type CapturableFile[TFile model_core.ReferenceMetadata] interface {
	CreateFileMerkleTree(ctx context.Context) (model_core.PatchedMessage[*model_filesystem_pb.FileContents, TFile], error)
	Discard()
}

CapturableFile is called into by CreateFileMerkleTree() to obtain a Merkle tree for a given file. Either one of these methods will be called exactly once.

func NewSimpleCapturableFile 

func NewSimpleCapturableFile[TFile model_core.ReferenceMetadata](fileContents model_core.PatchedMessage[*model_filesystem_pb.FileContents, TFile]) CapturableFile[TFile]

NewSimpleCapturableFile creates a CapturableFile that returns a FileContents message that was computed previously. This can be used to place already existing files in directory Merkle trees.

type CapturedDirectory 

type CapturedDirectory interface {
	Close() error
	EnterCapturedDirectory(name path.Component) (CapturedDirectory, error)
	OpenRead(name path.Component) (filesystem.FileReader, error)
}

CapturedDirectory is called into by CapturedDirectoryWalker to traverse a directory hierarchy and read file contents.

type CreatedDirectory 

type CreatedDirectory[TDirectory model_core.ReferenceMetadata] struct {
	Message                        model_core.PatchedMessage[*model_filesystem_pb.DirectoryContents, TDirectory]
	MaximumSymlinkEscapementLevels *wrapperspb.UInt32Value
}

CreatedDirectory contains the message of the root directory of the directory hierarchy for which a Merkle tree was created. It also contains all of the metadata that needs to be available when creating a DirectoryReference message for referring to the root directory.

func NewCreatedDirectoryBare 

func NewCreatedDirectoryBare[TDirectory model_core.ReferenceMetadata](message model_core.PatchedMessage[*model_filesystem_pb.DirectoryContents, TDirectory]) (*CreatedDirectory[TDirectory], error)

NewCreatedDirectoryBare creates a CreatedDirectory, recomputing MaximumSymlinkEscapementLevels from the provided Directory message.

This function can be used to embed an existing directory in a Merkle tree for which no DirectoryReference exists (e.g., because it was not the root directory of a directory cluster). In cases where a DirectoryReference is available, this function should not be used.

func (*CreatedDirectory[TDirectory]) ToDirectoryReference 

func (cd *CreatedDirectory[TDirectory]) ToDirectoryReference(reference *model_core_pb.DecodableReference) *model_filesystem_pb.DirectoryReference

ToDirectoryReference creates a DirectoryReference message that corresponds to the current directory.

type Directory 

type Directory struct {
	Directory *model_filesystem_pb.DirectoryContents

	// Indices at which child directories with inline contents are
	// accessible within the same cluster. If the child directory
	// has external contents, the index is set to -1.
	ChildDirectoryIndices []int
}

Directory contained in a DirectoryCluster.

type DirectoryAccessParameters 

type DirectoryAccessParameters struct {
	// contains filtered or unexported fields
}

DirectoryAccessParameters contains parameters that were used when creating Merkle trees of directories that should also be applied when attempting to access its contents afterwards. Parameters include whether files were compressed or encrypted.

func NewDirectoryAccessParametersFromProto 

func NewDirectoryAccessParametersFromProto(m *model_filesystem_pb.DirectoryAccessParameters, referenceFormat object.ReferenceFormat) (*DirectoryAccessParameters, error)

NewDirectoryAccessParametersFromProto creates an instance of DirectoryAccessParameters that matches the configuration stored in a Protobuf message. This, for example, permits a server to access files that were uploaded by a client.

func (*DirectoryAccessParameters) GetEncoder 

func (p *DirectoryAccessParameters) GetEncoder() model_encoding.DeterministicBinaryEncoder

GetEncoder returns the encoder that should be used to encode or decode directory contents and leaves objects.

type DirectoryCluster 

type DirectoryCluster []Directory

DirectoryCluster is a list of all Directory messages that are contained in a single object in storage. Directories are stored in topological order, meaning that the root directory is located at index zero.

The advantage of a DirectoryCluster over a plain Directory message is that it's possible to refer to individual directories using an index.

type DirectoryComponentWalker 

type DirectoryComponentWalker[TReference object.BasicReference] struct {
	// contains filtered or unexported fields
}

DirectoryComponentWalker can be used to resolve the properties of a file contained in a directory that has been written to storage.

func NewDirectoryComponentWalker 

func NewDirectoryComponentWalker[TReference object.BasicReference](
	ctx context.Context,
	directoryContentsReader model_parser.MessageObjectReader[TReference, *model_filesystem_pb.DirectoryContents],
	leavesReader model_parser.MessageObjectReader[TReference, *model_filesystem_pb.Leaves],
	onUpHandler func() (path.ComponentWalker, error),
	currentDirectory model_core.Message[*model_filesystem_pb.Directory, TReference],
	directoriesStack []model_core.Message[*model_filesystem_pb.DirectoryContents, TReference],
) *DirectoryComponentWalker[TReference]

NewDirectoryComponentWalker creates a new DirectoryComponentWalker that starts resolution within the provided directories.

func (*DirectoryComponentWalker[TReference]) GetCurrentDirectoriesStack 

func (cw *DirectoryComponentWalker[TReference]) GetCurrentDirectoriesStack() ([]model_core.Message[*model_filesystem_pb.DirectoryContents, TReference], error)

GetCurrentDirectoriesStack returns the stack of directories that the component walker currently uses to perform resolution. This list can, for example, be copied if the directory walker needs to be cloned.

func (*DirectoryComponentWalker[TReference]) GetCurrentDirectory 

func (cw *DirectoryComponentWalker[TReference]) GetCurrentDirectory() model_core.Message[*model_filesystem_pb.Directory, TReference]

GetCurrentDirectory returns the lowest directory that the component walker currently uses to perform resolution.

func (*DirectoryComponentWalker[TReference]) GetCurrentFileProperties 

func (cw *DirectoryComponentWalker[TReference]) GetCurrentFileProperties() model_core.Message[*model_filesystem_pb.FileProperties, TReference]

GetCurrentFileProperties returns the properties of the file corresponding to the path that was performed. If the path resolved to a directory, the file properties will not be set.

func (*DirectoryComponentWalker[TReference]) OnDirectory 

func (cw *DirectoryComponentWalker[TReference]) OnDirectory(name path.Component) (path.GotDirectoryOrSymlink, error)

OnDirectory is called when the path to be resolved has a leading directory pathname components.

func (*DirectoryComponentWalker[TReference]) OnTerminal 

func (cw *DirectoryComponentWalker[TReference]) OnTerminal(name path.Component) (*path.GotSymlink, error)

OnTerminal is called when the path to be resolved has a trailing pathname component.

func (*DirectoryComponentWalker[TReference]) OnUp 

func (cw *DirectoryComponentWalker[TReference]) OnUp() (path.ComponentWalker, error)

OnUp is called when the path to be resolved contains a ".." component.

type DirectoryCreationParameters 

type DirectoryCreationParameters struct {
	*DirectoryAccessParameters
	// contains filtered or unexported fields
}

DirectoryCreationParameters contains parameters such as encoders, and maximum object sizes that need to be considered when creating Merkle trees of directories.

func NewDirectoryCreationParametersFromProto 

func NewDirectoryCreationParametersFromProto(m *model_filesystem_pb.DirectoryCreationParameters, referenceFormat object.ReferenceFormat) (*DirectoryCreationParameters, error)

NewDirectoryCreationParametersFromProto converts the directory creation parameters that are stored in a Protobuf message to its native counterpart. It also validates that the provided sizes are in bounds.

type DirectoryMerkleTreeCapturer 

type DirectoryMerkleTreeCapturer[TDirectory, TFile any] interface {
	CaptureFileNode(TFile) TDirectory
	CaptureDirectory(ctx context.Context, createdObject model_core.CreatedObject[TDirectory]) (TDirectory, error)
	CaptureLeaves(ctx context.Context, createdObject model_core.CreatedObject[TDirectory]) (TDirectory, error)
}

DirectoryMerkleTreeCapturer is called into by CreateDirectoryMerkleTree() to capture Merkle tree objects.

If CreateDirectoryMerkleTree() is only invoked to compute the hash of a directory, a simple implementation may be provided that discards all objects. If the directory is small enough, an implementation may be used to that stores all data in memory. For large directories it is suggested to discard file nodes and only keep directory objects in memory, as Merkle trees for files can always be recomputed if needed. 

var FileDiscardingDirectoryMerkleTreeCapturer DirectoryMerkleTreeCapturer[model_core.CreatedObjectTree, model_core.NoopReferenceMetadata] = fileDiscardingDirectoryMerkleTreeCapturer{}

FileDiscardingDirectoryMerkleTreeCapturer is an instance of DirectoryMerkleTreeCapturer that keeps any Directory and Leaves objects, but discards FileContents list and file chunk objects.

Discarding the contents of files is typically the right approach for uploading directory structures with changes to only a small number of files. The Merkle trees of files can be recomputed if it turns out they still need to be uploaded.

func NewSimpleDirectoryMerkleTreeCapturer 

func NewSimpleDirectoryMerkleTreeCapturer[TMetadata any](capturer model_core.CreatedObjectCapturer[TMetadata]) DirectoryMerkleTreeCapturer[TMetadata, TMetadata]

NewSimpleDirectoryMerkleTreeCapturer creates a DirectoryMerkleTreeCapturer that assumes that directories and leaves need to be captured the same way, and that file metadata uses the same type as directory metadata.

type EscapementCountingScopeWalker 

type EscapementCountingScopeWalker struct {
	// contains filtered or unexported fields
}

EscapementCountingScopeWalker determines the number of levels a given path escapes the current directory. This is achieved by counting the number of leading ".." components.

If the path is absolute or if ".." components are placed after named components (e.g., "a/../b"), it determines the number of levels is not bounded.

func NewEscapementCountingScopeWalker 

func NewEscapementCountingScopeWalker() *EscapementCountingScopeWalker

NewEscapementCountingScopeWalker creates an EscapementCountingScopeWalker that is in the initial path parsing state (i.e., corresponding with path ".").

func (*EscapementCountingScopeWalker) GetLevels 

func (sw *EscapementCountingScopeWalker) GetLevels() *wrapperspb.UInt32Value

GetLevels returns the number of levels the parsed path escapes the current directory. If nil is returned, the escapement is not bounded.

func (*EscapementCountingScopeWalker) OnAbsolute 

func (sw *EscapementCountingScopeWalker) OnAbsolute() (path.ComponentWalker, error)

OnAbsolute is called when the path for which the number of escapement levels is computed starts with a slash.

func (*EscapementCountingScopeWalker) OnDriveLetter 

func (sw *EscapementCountingScopeWalker) OnDriveLetter(drive rune) (path.ComponentWalker, error)

OnDriveLetter is called when the path for which the number of escapement levels is computed starts with a Windows drive letter.

func (*EscapementCountingScopeWalker) OnRelative 

func (sw *EscapementCountingScopeWalker) OnRelative() (path.ComponentWalker, error)

OnRelative is called when the path for which the number of escapement levels is computed is relative.

func (*EscapementCountingScopeWalker) OnShare 

func (sw *EscapementCountingScopeWalker) OnShare(server, share string) (path.ComponentWalker, error)

OnShare is called when the path for which the number of escapement levels is computed starts with a Windows network share.

type FileAccessParameters 

type FileAccessParameters struct {
	// contains filtered or unexported fields
}

FileAccessParameters contains parameters that were used when creating Merkle trees of files that should also be applied when attempting to access its contents afterwards. Parameters include whether files were compressed or encrypted.

func NewFileAccessParametersFromProto 

func NewFileAccessParametersFromProto(m *model_filesystem_pb.FileAccessParameters, referenceFormat object.ReferenceFormat) (*FileAccessParameters, error)

NewFileAccessParametersFromProto creates an instance of FileAccessParameters that matches the configuration stored in a Protobuf message. This, for example, permits a server to access files that were uploaded by a client.

func (*FileAccessParameters) GetChunkEncoder 

func (p *FileAccessParameters) GetChunkEncoder() model_encoding.DeterministicBinaryEncoder

GetChunkEncoder returns the encoder that was used to create chunks of a file. This can be used to subsequently decode the chunks.

func (*FileAccessParameters) GetFileContentsListEncoder 

func (p *FileAccessParameters) GetFileContentsListEncoder() model_encoding.DeterministicBinaryEncoder

GetFileContentsListEncoder returns the encoder that was used to create file contents lists of large files. This can be used to subsequently decode these lists.

type FileChunkReaderForTesting 

type FileChunkReaderForTesting = model_parser.DecodingObjectReader[object.LocalReference, []byte]

FileChunkReaderForTesting is used for generating mocks that are used by FileReader's unit tests.

type FileContentsEntry 

type FileContentsEntry[TReference object.BasicReference] struct {
	// contains filtered or unexported fields
}

FileContentsEntry contains the properties of a part of a concatenated file. This type is equivalent to the FileContents message, except that counters like size and sparseness are cumulative with respect to the list in which the entries are stored. This allows for efficient lookups.

func NewFileContentsEntryFromBinary 

func NewFileContentsEntryFromBinary(r io.ByteReader, getDecodingParametersSizeBytes func(isFileContentsList bool) int) (FileContentsEntry[object.LocalReference], error)

NewFileContentsEntryFromBinary constructs a FileContentsEntry from a binary representation that was previously obtained by calling FileContentsEntry.AppendBinary().

func NewFileContentsEntryFromProto 

func NewFileContentsEntryFromProto[TReference object.BasicReference](fileContents model_core.Message[*model_filesystem_pb.FileContents, TReference]) (FileContentsEntry[TReference], error)

NewFileContentsEntryFromProto constructs a FileContentsEntry based on the contents of a single FileContents Protobuf message, refering to the file as a whole.

func (*FileContentsEntry[TReference]) AppendBinary 

func (fce *FileContentsEntry[TReference]) AppendBinary(b []byte) []byte

AppendBinary converts a FileContentsEntry to a compact binary representation. This can, for example, be embedded in the file handle of a file exposed via an NFSv4 mount.

func (*FileContentsEntry[TReference]) GetEndBytes 

func (fce *FileContentsEntry[TReference]) GetEndBytes() uint64

GetEndBytes returns the byte offset at which the current entry ends, with respect to the file contents list in which the entry is placed.

func (*FileContentsEntry[TReference]) GetReference 

func (fce *FileContentsEntry[TReference]) GetReference() *model_core.Decodable[TReference]

GetReference returns the reference of the chunk or file contents list stored in this entry. If the current entry is a hole, this function returns nil.

type FileContentsIterator 

type FileContentsIterator[TReference object.BasicReference] struct {
	// contains filtered or unexported fields
}

FileContentsIterator is a helper type for iterating over the chunks of a concatenated file sequentially.

func NewFileContentsIterator 

func NewFileContentsIterator[TReference object.BasicReference](root FileContentsEntry[TReference], initialOffsetBytes uint64) FileContentsIterator[TReference]

NewFileContentsIterator creates a FileContentsIterator that starts iteration at the provided offset within the file. It is the caller's responsibility to ensure the provided offset is less than the size of the file.

func (*FileContentsIterator[TReference]) GetCurrentPart 

func (i *FileContentsIterator[TReference]) GetCurrentPart() (reference *model_core.Decodable[TReference], offsetBytes, sizeBytes uint64)

GetCurrentPart returns the reference of the part of the file that contain the data corresponding with the current offset. It also returns the offset within the part from which data should be read, and the expected total size of the part.

It is the caller's responsibility to track whether iteration has reached the end of the file. Once the end of the file has been reached, GetCurrentPart() may no longer be called.

func (*FileContentsIterator[TReference]) PushFileContentsList 

func (i *FileContentsIterator[TReference]) PushFileContentsList(list FileContentsList[TReference]) error

PushFileContentsList can be invoked after GetCurrentPart() to signal that the current part does not refer to a chunk of data, but another FileContentsList. After calling this method, another call to GetCurrentPart() can be made to retry resolution of the part within the provided FileContentsList.

func (*FileContentsIterator[TReference]) ToNextPart 

func (i *FileContentsIterator[TReference]) ToNextPart()

ToNextPart can be invoked after GetCurrentPart() to signal that the current part refers to a chunk of data. The next call to GetCurrentPart() will return the reference of the part that is stored after the current one.

type FileContentsList 

type FileContentsList[TReference object.BasicReference] []FileContentsEntry[TReference]

FileContentsList contains the properties of parts of a concatenated file. Parts are stored in the order in which they should be concatenated, with any counters contained within being cumulative to permit binary searching.

func NewFileContentsListFromProto 

func NewFileContentsListFromProto[TReference object.BasicReference](l model_core.Message[[]*model_filesystem_pb.FileContents, TReference]) (FileContentsList[TReference], error)

NewFileContentsListFromProto converts a list of FileContents messages that were previously written to storage to its native counterpart.

type FileContentsListReaderForTesting 

type FileContentsListReaderForTesting = model_parser.DecodingObjectReader[object.LocalReference, FileContentsList[object.LocalReference]]

FileContentsListReaderForTesting is used for generating mocks that are used by FileReader's unit tests.

type FileCreationParameters 

type FileCreationParameters struct {
	*FileAccessParameters
	// contains filtered or unexported fields
}

FileCreationParameters contains parameters such as encoders, and minimum/maximum object sizes that need to be considered when creating Merkle trees of files.

func NewFileCreationParametersFromProto 

func NewFileCreationParametersFromProto(m *model_filesystem_pb.FileCreationParameters, referenceFormat object.ReferenceFormat) (*FileCreationParameters, error)

NewFileCreationParametersFromProto converts the file creation parameters that are stored in a Protobuf message to its native counterpart. It also validates that the provided sizes are in bounds.

func (*FileCreationParameters) EncodeChunk 

func (p *FileCreationParameters) EncodeChunk(data []byte) (model_core.Decodable[*object.Contents], error)

EncodeChunk encodes the data of a small file, or a region of a large file into an object that can be written to storage.

type FileMerkleTreeCapturer 

type FileMerkleTreeCapturer[TMetadata any] interface {
	CaptureChunk(ctx context.Context, contents *object.Contents) (TMetadata, error)
	CaptureFileContentsList(ctx context.Context, createdObject model_core.CreatedObject[TMetadata]) (TMetadata, error)
}

FileMerkleTreeCapturer is provided by callers of CreateFileMerkleTree to provide logic for how the resulting Merkle tree of the file should be captured.

A no-op implementation can be used by the caller to simply compute a reference of the file. An implementation that actually captures the provided contents can be used to prepare a Merkle tree for uploading.

The methods below return metadata. The metadata for the root object will be returned by CreateFileMerkleTree. 

var ChunkDiscardingFileMerkleTreeCapturer FileMerkleTreeCapturer[model_core.CreatedObjectTree] = chunkDiscardingFileMerkleTreeCapturer{}

ChunkDiscardingFileMerkleTreeCapturer is an implementation of FileMerkleTreeCapturer that only preserves the FileContents messages of the Merkle tree. This can be of use when incrementally replicating the contents of a file. In those cases it's wasteful to store the full contents of a file in memory.

func NewSimpleFileMerkleTreeCapturer 

func NewSimpleFileMerkleTreeCapturer[TMetadata any](capturer model_core.CreatedObjectCapturer[TMetadata]) FileMerkleTreeCapturer[TMetadata]

NewSimpleFileMerkleTreeCapturer creates a FileMerkleTreeCapturer that assumes that chunks and file contents lists need to be captured the same way.

type FileMerkleTreeCapturerForTesting 

type FileMerkleTreeCapturerForTesting FileMerkleTreeCapturer[model_core.ReferenceMetadata]

FileMerkleTreeCapturerForTesting is used to generate mocks for unit testing CreateFileMerkleTree().

type FileReader 

type FileReader[TReference object.BasicReference] struct {
	// contains filtered or unexported fields
}

FileReader is a helper type for reading the contents of files that have been written to storage, using traditional interfaces like io.Reader, io.ReaderAt, io.Writer, etc.

func NewFileReader 

func NewFileReader[TReference object.BasicReference](
	fileContentsListReader model_parser.DecodingObjectReader[TReference, FileContentsList[TReference]],
	fileChunkReader model_parser.DecodingObjectReader[TReference, []byte],
	fileChunkReaderConcurrency *semaphore.Weighted,
) *FileReader[TReference]

NewFileReader creates a FileReader that uses the specified object readers for chunks and file contents lists.

func (*FileReader[TReference]) FileOpenRead 

func (fr *FileReader[TReference]) FileOpenRead(ctx context.Context, fileContents FileContentsEntry[TReference], offsetBytes uint64) *SequentialFileReader[TReference]

FileOpenRead opens a file that has been written to storage, returning a handle that allows the file to be read sequentially.

func (*FileReader[TReference]) FileOpenReadAt 

func (fr *FileReader[TReference]) FileOpenReadAt(ctx context.Context, fileContents FileContentsEntry[TReference]) io.ReaderAt

FileOpenReadAt opens a file that has been written to storage, returning an io.ReaderAt so that the file can be read at random.

func (*FileReader[TReference]) FileReadAll 

func (fr *FileReader[TReference]) FileReadAll(ctx context.Context, fileContents FileContentsEntry[TReference], maximumSizeBytes uint64) ([]byte, error)

FileReadAll reads the entire contents of a file that has been written to storage, returning the contents as a byte slice.

func (*FileReader[TReference]) FileReadAt 

func (fr *FileReader[TReference]) FileReadAt(ctx context.Context, fileContents FileContentsEntry[TReference], p []byte, offsetBytes uint64) (int, error)

FileReadAt performs a single read of a portion of a file that has been written to storage. The provided offset and output array MUST reside within the boundaries of the file.

func (*FileReader[TReference]) FileWriteTo 

func (fr *FileReader[TReference]) FileWriteTo(ctx context.Context, fileContents FileContentsEntry[TReference], file io.WriterAt) error

FileWriteTo writes the contents of a file residing in the Object Store to a local file. It is assumed that the file already has the desired size and consists of a single hole.

func (*FileReader[TReference]) GetDecodingParametersSizeBytes 

func (fr *FileReader[TReference]) GetDecodingParametersSizeBytes(isFileContentsList bool) int

GetDecodingParametersSizeBytes returns the expected size of the decoding parameters that a reference to a file contents list or chunks should have.

type SectionWriter 

type SectionWriter struct {
	// contains filtered or unexported fields
}

SectionWriter provides an implementation of io.Writer on top of io.WriterAt. It is similar to io.SectionReader, but then for writes.

func NewSectionWriter 

func NewSectionWriter(w io.WriterAt) *SectionWriter

NewSectionWriter creates a new SectionWriter that starts writing data to the output, starting at offset zero.

func (*SectionWriter) GetOffsetBytes 

func (w *SectionWriter) GetOffsetBytes() int64

GetOffsetBytes returns the offset at which the next call to Write() or WriteString() will start writing.

func (*SectionWriter) Write 

func (w *SectionWriter) Write(p []byte) (int, error)

Write binary data to the underlying writer at the current offset, and progress the current offset.

func (*SectionWriter) WriteString 

func (w *SectionWriter) WriteString(s string) (int, error)

WriteString writes string data to the underlying writer at the current offset, and progress the current offset.

type SequentialFileReader 

type SequentialFileReader[TReference object.BasicReference] struct {
	// contains filtered or unexported fields
}

SequentialFileReader can be used to sequentially read data from a file that has been written to storage. SequentialFileReader buffers the last read chunk of data, meaning that small reads will not trigger redundant reads against the underlying storage backend.

func (*SequentialFileReader[TReference]) Read 

func (r *SequentialFileReader[TReference]) Read(p []byte) (int, error)

Read data from a file that has been written to the Object Store.

func (*SequentialFileReader[TReference]) ReadByte 

func (r *SequentialFileReader[TReference]) ReadByte() (byte, error)

ReadByte reads a single byte of data from a file that has been written to the Object Store.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.