core

package
v0.0.0-...-82353ca Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 14, 2026 License: Apache-2.0 Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ComputeTopLevelMessageReference 

func ComputeTopLevelMessageReference[TMessage proto.Message, TReference object.BasicReference](
	m TopLevelMessage[TMessage, TReference],
	referenceFormat object.ReferenceFormat,
) (object.LocalReference, error)

ComputeTopLevelMessageReference computes the local reference of an object whose contents are equal to that of a provided top-level message after it has been referenced, without any encoding applied.

This function should typically only be used to compute tag key hashes, as it only produces references of fictive objects. In practice you almost always want to encode objects prior to storing them in an object.

func DecodableLocalReferenceToString 

func DecodableLocalReferenceToString[TReference object.BasicReference](reference Decodable[TReference]) string

DecodableLocalReferenceToString converts a reference containing decoding parameters to a string of the format "${reference}.${decodingParameters}", where both parts are base64. Strings of this format may be printed as part of logs and other diagnostic output.

func DecodableLocalReferenceToWeakProto 

func DecodableLocalReferenceToWeakProto[TReference object.BasicReference](reference Decodable[TReference]) *model_core_pb.WeakDecodableReference

DecodableLocalReferenceToWeakProto converts a reference containing decoding parameters to a Protobuf message, so that it may be used as a weak reference as part of RPC request/response bodies or objects in storage.

func FlattenReference 

func FlattenReference[TReference any](m Message[*model_core_pb.Reference, TReference]) (TReference, error)

FlattenReference returns the actual reference that is associated with a given Reference Protobuf message.

func FlattenReferenceSet 

func FlattenReferenceSet[TReference any](m Message[*model_core_pb.ReferenceSet, TReference]) (object.OutgoingReferences[TReference], error)

FlattenReferenceSet returns the actual references that are contained in a given ReferenceSet Protobuf message.

func GetIndexFromReferenceMessage 

func GetIndexFromReferenceMessage(referenceMessage *core.Reference, degree int) (int, error)

GetIndexFromReferenceMessage extracts the one-based index contained in a Reference message and converts it to a zero-based index that can, for example, be provided to OutgoingReferences.GetOutgoingReference().

func MarshalTopLevelMessage 

func MarshalTopLevelMessage[TMessage proto.Message, TReference object.BasicReference](m TopLevelMessage[TMessage, TReference]) ([]byte, error)

MarshalTopLevelMessage converts the contents of a top-level Protobuf message to a byte sequence. This byte sequence can be used to sort messages along a total order, or to act as a map key.

func MessagesEqual 

func MessagesEqual[
	TMessage proto.Message,
	TReference1, TReference2 object.BasicReference,
](m1 Message[TMessage, TReference1], m2 Message[TMessage, TReference2]) bool

MessagesEqual returns true if two messages contain the same data.

Nested messages belonging to different objects may contain the same data, but use different reference numbers. This function therefore needs to convert the provided messages to top-level messages, which is expensive. We therefore compare the message size first.

func MessagesEqualList 

func MessagesEqualList[
	TMessage proto.Message,
	TReference1, TReference2 object.BasicReference,
](m1 Message[[]TMessage, TReference1], m2 Message[[]TMessage, TReference2]) bool

MessagesEqualList returns true if two messages contain the same list of Protobuf message.

Nested messages belonging to different objects may contain the same data, but use different reference numbers. This function therefore needs to convert each of the provided pairs of elements to top-level messages, which is expensive. We therefore compare the message size first.

func NewProtoBinaryMarshaler 

func NewProtoBinaryMarshaler(message proto.Message) encoding.BinaryMarshaler

NewProtoBinaryMarshaler converts a Protobuf message to a BinaryMarshaler. This prevents further access to the message's contents, but does allow it to be passed along in contexts where the data is only expected to be marshaled and stored.

func NewProtoListBinaryMarshaler 

func NewProtoListBinaryMarshaler[TMessage proto.Message](messages []TMessage) encoding.BinaryMarshaler

NewProtoListBinaryMarshaler converts a list of Protobuf message to a BinaryMarshaler. This prevents further access to the message's contents, but does allow it to be passed along in contexts where the data is only expected to be marshaled and stored.

When marshaled, each message is prefixed with its size.

func NewProtoListObjectFormat 

func NewProtoListObjectFormat(m proto.Message) *model_core_pb.ObjectFormat

NewProtoListObjectFormat creates a new ObjectFormat message that corresponds to a list of the type of the provided Protobuf message.

func NewProtoObjectFormat 

func NewProtoObjectFormat(m proto.Message) *model_core_pb.ObjectFormat

NewProtoObjectFormat creates a new ObjectFormat message that corresponds to the type of the provided Protobuf message.

func NewRawBinaryMarshaler 

func NewRawBinaryMarshaler(data []byte) encoding.BinaryMarshaler

NewRawBinaryMarshaler converts a byte slice to a BinaryMarshaler. The BinaryMarshaler does nothing apart from returning the original byte slice. This can be used to store blobs in objects.

func ObjectFormatToPath 

func ObjectFormatToPath(objectFormat *model_core_pb.ObjectFormat) ([]string, bool)

ObjectFormatToPath converts an object format to a set of pathname components. This corresponds to the paths accepted by tools like bonanza_browser.

func PatchedMessagesEqual 

func PatchedMessagesEqual[
	TMessage proto.Message,
	TMetadata1, TMetadata2 ReferenceMetadata,
](m1 PatchedMessage[TMessage, TMetadata1], m2 PatchedMessage[TMessage, TMetadata2]) bool

PatchedMessagesEqual returns true if two patched messages contain the same data.

Nested messages belonging to different objects may contain the same data, but use different reference numbers. This function therefore needs to convert the provided messages to top-level messages, which is expensive. We therefore compare the message size first.

func TopLevelMessagesEqual 

func TopLevelMessagesEqual[
	TMessage proto.Message,
	TReference1, TReference2 object.BasicReference,
](m1 TopLevelMessage[TMessage, TReference1], m2 TopLevelMessage[TMessage, TReference2]) bool

TopLevelMessagesEqual returns true if two top-level messages contain the same data. This function can only be called against top-level messages, as only those consistently number any outgoing references.

Types

type CreatedObject 

type CreatedObject[TMetadata any] struct {
	*object.Contents
	Metadata []TMetadata
}

CreatedObject holds the contents of an object that was created using ReferenceMessagePatcher. It also holds the metadata that was provided to ReferenceMessagePatcher.AddReference(), provided in the same order as the outgoing references of the created object.

func MarshalAndEncodeKeyed 

func MarshalAndEncodeKeyed[TMetadata ReferenceMetadata](
	m PatchedMessage[encoding.BinaryMarshaler, TMetadata],
	referenceFormat object.ReferenceFormat,
	encoder model_encoding.KeyedBinaryEncoder,
	decodingParameters []byte,
) (CreatedObject[TMetadata], error)

MarshalAndEncodeKeyed marshals a patched message, encodes it with explicitly provided decoding parameters, and converts it to an object that can be written to storage.

func (CreatedObject[TMetadata]) Capture 

func (o CreatedObject[TMetadata]) Capture(ctx context.Context, capturer CreatedObjectCapturer[TMetadata]) (MetadataEntry[TMetadata], error)

Capture the contents of a created object, returning the metadata of the object in the form of a MetadataEntry[TMetadata], which can be provided to ReferenceMessagePatcher.AddReference() to let a message refer to the object.

type CreatedObjectCapturer 

type CreatedObjectCapturer[TMetadata any] interface {
	CaptureCreatedObject(ctx context.Context, createdObject CreatedObject[TMetadata]) (TMetadata, error)
}

CreatedObjectCapturer can be used as a factory type for reference metadata. Given the contents of an object and the metadata of all of its children, it may yield new metadata. 

var DiscardingCreatedObjectCapturer CreatedObjectCapturer[NoopReferenceMetadata] = discardingCreatedObjectCapturer{}

DiscardingCreatedObjectCapturer is an implementation of CreatedObjectCapturer that discards any created objects. This can be used if the goal is to simply compute the root node of the Merkle tree, discarding any children that were created in the process. 

var WalkableCreatedObjectCapturer CreatedObjectCapturer[dag.ObjectContentsWalker] = walkableCreatedObjectCapturer{}

WalkableCreatedObjectCapturer is an implementation of ObjectCapturer that creates a dag.ObjectContentsWalker for each created object. This ends up keeping objects in memory and only allows them to be traversed as part of the upload process.

This implementation is sufficient when given existing Merkle trees in the form of a dag.ObjectContentsWalkers that need to be combined into single Merkle tree.

type CreatedObjectCapturerFunc 

type CreatedObjectCapturerFunc[TMetadata any] func(ctx context.Context, createdObject CreatedObject[TMetadata]) (TMetadata, error)

CreatedObjectCapturerFunc has the same signature as CreatedObjectCapturer.CaptureCreatedObject(). It is provided to allow a single type to provide multiple capturing functions, and to pass each of these methods to a function taking a CreatedObjectCapturer.

func (CreatedObjectCapturerFunc[TMetadata]) CaptureCreatedObject 

func (f CreatedObjectCapturerFunc[TMetadata]) CaptureCreatedObject(ctx context.Context, createdObject CreatedObject[TMetadata]) (TMetadata, error)

CaptureCreatedObject forwards the call to create reference metadata to the underlying CreatedObjectCapturerFunc.

type CreatedObjectTree 

type CreatedObjectTree CreatedObject[CreatedObjectTree]

CreatedObjectTree is CreatedObject applied recursively. Namely, it can hold the contents of a tree of objects that were created using ReferenceMessagePatcher in memory.

func (CreatedObjectTree) Discard 

func (CreatedObjectTree) Discard()

Discard any resources owned by the CreatedObjectTree.

func (CreatedObjectTree) ToObjectContentsWalker 

func (t CreatedObjectTree) ToObjectContentsWalker() dag.ObjectContentsWalker

ToObjectContentsWalker returns a ObjectContentsWalker that allows traversing all objects contained in the CreatedObjectTree.

type Decodable 

type Decodable[T any] struct {
	Value T
	// contains filtered or unexported fields
}

Decodable can be used to annotate an object with parameters that are used to decode an object. For example, Decodable[CreatedObject[T]] can hold a created object that can subsequently be accessed. Similarly, Decodable[object.LocalReference] can be used to refer to an object that can subsequently be accessed.

Decodable is comparable if its value is comparable as well.

func CopyDecodable 

func CopyDecodable[T1, T2 any](from Decodable[T1], to T2) Decodable[T2]

CopyDecodable extracts the decoding parameters of a given Decodable[T] and attaches it to another object.

func FlattenDecodableReference 

func FlattenDecodableReference[TReference any](m Message[*model_core_pb.DecodableReference, TReference]) (Decodable[TReference], error)

FlattenDecodableReference returns the actual reference that is associated with a given DecodableReference Protobuf message, and attaches decoding parameters to it.

func MarshalAndEncodeDeterministic 

func MarshalAndEncodeDeterministic[TMetadata ReferenceMetadata](
	m PatchedMessage[encoding.BinaryMarshaler, TMetadata],
	referenceFormat object.ReferenceFormat,
	encoder model_encoding.DeterministicBinaryEncoder,
) (Decodable[CreatedObject[TMetadata]], error)

MarshalAndEncodeDeterministic marshals a patched message, encodes it deterministically, and converts it to an object that can be written to storage.

func NewDecodable 

func NewDecodable[T any](value T, decodingParameters []byte) (Decodable[T], error)

NewDecodable is a helper function for creating instances of Decodable[T].

func NewDecodableLocalReferenceFromString 

func NewDecodableLocalReferenceFromString(referenceFormat object.ReferenceFormat, s string) (Decodable[object.LocalReference], error)

NewDecodableLocalReferenceFromString converts a string of the format "${reference}.${decodingParameters}", where both parts are base64 encoded, to a decodable local reference. Strings of this format may be printed as part of logs and other diagnostic output.

func NewDecodableLocalReferenceFromWeakProto 

func NewDecodableLocalReferenceFromWeakProto(referenceFormat object.ReferenceFormat, m *model_core_pb.WeakDecodableReference) (Decodable[object.LocalReference], error)

NewDecodableLocalReferenceFromWeakProto converts a WeakDecodableReference Protobuf messages to its native counterpart.

func (*Decodable[T]) GetDecodingParameters 

func (d *Decodable[T]) GetDecodingParameters() []byte

GetDecodingParameters returns the parameters needed to decode the object associated with the value.

type ExistingObjectCapturer 

type ExistingObjectCapturer[TReference, TMetadata any] interface {
	CaptureExistingObject(TReference) TMetadata
}

ExistingObjectCapturer can be used as a factory type for reference metadata. Given a reference of an object that already exists in storage, it may yield metadata.

type LeakCheckingObjectManager 

type LeakCheckingObjectManager[TReference any, TMetadata ReferenceMetadata] struct {
	// contains filtered or unexported fields
}

LeakCheckingObjectManager is a wrapper around ObjectManager that keeps track of the number of ReferenceMetadata values that are still valid. This can be used to detect and report leaks of ReferenceMetadata.

func NewLeakCheckingObjectManager 

func NewLeakCheckingObjectManager[TReference any, TMetadata ReferenceMetadata](base ObjectManager[TReference, TMetadata]) *LeakCheckingObjectManager[TReference, TMetadata]

NewLeakCheckingObjectManager creates a LeakCheckingObjectManager that is in the initial state, meaning that it has zero valid ReferenceMetadata objects.

func (*LeakCheckingObjectManager[TReference, TMetadata]) CaptureCreatedObject 

func (om *LeakCheckingObjectManager[TReference, TMetadata]) CaptureCreatedObject(ctx context.Context, createdObject CreatedObject[*LeakCheckingReferenceMetadata[TMetadata]]) (*LeakCheckingReferenceMetadata[TMetadata], error)

CaptureCreatedObject captures the contents of an object that was just created, returning reference metadata that tracks any leaks.

func (*LeakCheckingObjectManager[TReference, TMetadata]) CaptureExistingObject 

func (om *LeakCheckingObjectManager[TReference, TMetadata]) CaptureExistingObject(reference TReference) *LeakCheckingReferenceMetadata[TMetadata]

CaptureExistingObject creates reference metadata for an object that already exists in storage. The reference metadata that is returned performs tracking of leaks.

func (*LeakCheckingObjectManager[TReference, TMetadata]) GetCurrentValidMetadataCount 

func (om *LeakCheckingObjectManager[TReference, TMetadata]) GetCurrentValidMetadataCount() int

GetCurrentValidMetadataCount returns the current valid number of ReferenceMetadata objects. When non-zero, ReferenceMetadata objects may have been leaked.

func (*LeakCheckingObjectManager[TReference, TMetadata]) ReferenceObject 

func (om *LeakCheckingObjectManager[TReference, TMetadata]) ReferenceObject(entry MetadataEntry[*LeakCheckingReferenceMetadata[TMetadata]]) TReference

ReferenceObject converts the reference metadata back to a reference. As references don't have explicit lifetimes, this means that no leak tracking is performed from this point on.

type LeakCheckingReferenceMetadata 

type LeakCheckingReferenceMetadata[TMetadata ReferenceMetadata] struct {
	// contains filtered or unexported fields
}

LeakCheckingReferenceMetadata is a wrapper around ReferenceMetadata that keeps track of whether it's still valid. When discarded, it decrements the valid metadata count.

func (*LeakCheckingReferenceMetadata[TMetadata]) Discard 

func (rm *LeakCheckingReferenceMetadata[TMetadata]) Discard()

Discard the reference metadata, releasing any resources associated with it.

func (*LeakCheckingReferenceMetadata[TMetadata]) Unwrap 

func (rm *LeakCheckingReferenceMetadata[TMetadata]) Unwrap() TMetadata

Unwrap a LeakCheckingReferenceMetadata, returning the ReferenceMetadata that it contains. This destroys the LeakCheckingReferenceMetadata in the process.

type Message 

type Message[TMessage any, TReference any] struct {
	Message            TMessage
	OutgoingReferences object.OutgoingReferences[TReference]
}

Message is a piece of data, typically a Protobuf message, that has zero or more outgoing references associated with it.

func Nested 

func Nested[TMessage1, TMessage2, TReference any](parent Message[TMessage1, TReference], child TMessage2) Message[TMessage2, TReference]

Nested is a helper function for creating instances of Message that refer to a message that was mebedded into another one.

func NewMessage 

func NewMessage[TMessage, TReference any](
	m TMessage,
	outgoingReferences object.OutgoingReferences[TReference],
) Message[TMessage, TReference]

NewMessage is a helper function for creating instances of Message.

func NewRawMessage 

func NewRawMessage(contents *object.Contents) Message[[]byte, object.LocalReference]

NewRawMessage creates a Message whose contents correspond to the payload of an object in storage. Typically additional processing needs to be performed to consume this data, such as decoding and parsing.

func NewSimpleMessage 

func NewSimpleMessage[TReference, TMessage any](m TMessage) Message[TMessage, TReference]

NewSimpleMessage is a helper function for creating instances of Message for messages that don't contain any references.

func (*Message[TMessage, TReference]) Clear 

func (m *Message[TMessage, TReference]) Clear()

Clear the Message, releasing the data and outgoing references that are associated with it.

func (Message[TMessage, TReference]) IsSet 

func (m Message[TMessage, TReference]) IsSet() bool

IsSet returns true if the instance of Message is initialized.

type MetadataEntry 

type MetadataEntry[TMetadata any] struct {
	object.LocalReference
	Metadata TMetadata
}

MetadataEntry contains both the key (reference) and value (metadata) of which ReferenceMessagePatcher tracks.

func CaptureExistingObject 

func CaptureExistingObject[TReference object.BasicReference, TMetadata any](
	capturer ExistingObjectCapturer[TReference, TMetadata],
	reference TReference,
) MetadataEntry[TMetadata]

CaptureExistingObject creates reference metadata for an object that already exists, and returns it in the form of a MetadataEntry that can be passed to ReferenceMessagePatcher.AddReference().

type NoopReferenceMetadata 

type NoopReferenceMetadata struct{}

NoopReferenceMetadata is a trivial implementation of ReferenceMetadata that does not capture anything. This can be used if the goal is to simply compute the root node of the Merkle tree, discarding any children that were created in the process.

func (NoopReferenceMetadata) Discard 

func (NoopReferenceMetadata) Discard()

Discard all resources owned by this instance of NoopReferenceMetadata. This method is merely provided to satisfy the ReferenceMetadata interface.

type ObjectCapturer 

type ObjectCapturer[TReference, TMetadata any] interface {
	CreatedObjectCapturer[TMetadata]
	ExistingObjectCapturer[TReference, TMetadata]
}

ObjectCapturer is a combination of CreatedObjectCapturer and ExistingObjectCapturer, allowing the construction of metadata both for newly created objects and ones that exist in storage.

func NewDiscardingObjectCapturer 

func NewDiscardingObjectCapturer[TReference any]() ObjectCapturer[TReference, NoopReferenceMetadata]

NewDiscardingObjectCapturer creates is an implementation of ObjectCapturer that discards any created objects, and stores no metadata for any existing objects. This can be used if the goal is to simply compute the root node of the Merkle tree, discarding any children that were created in the process.

type ObjectExporter 

type ObjectExporter[TInternal, TExternal any] interface {
	ExportReference(ctx context.Context, internalReference TInternal) (TExternal, error)
	ImportReference(externalReference TExternal) TInternal
}

ObjectExporter can be used to exported references obtained from ObjectManager to those of another format, and vice versa.

Calls to ExportReference() also need to ensure that any objects referenced by the internal reference format are flushed to storage, so that the external reference becomes valid. bonanza_builder may use this to flush actions to storage, so that any request to execute an action on another worker doesn't fail due to missing objects.

type ObjectManager 

type ObjectManager[TReference, TMetadata any] interface {
	ObjectCapturer[TReference, TMetadata]
	ObjectReferencer[TReference, TMetadata]
}

ObjectManager represents an environment in which references and metadata can be interchanged freely.

type ObjectReferencer 

type ObjectReferencer[TReference, TMetadata any] interface {
	ReferenceObject(MetadataEntry[TMetadata]) TReference
}

ObjectReferencer is the inverse of ObjectCapturer, allowing metadata to be converted back to references. This can be of use in environments where objects also need to be accessible for reading right after they have been constructed, without explicitly waiting for them to be written to storage.

type PatchedMessage 

type PatchedMessage[TMessage any, TMetadata ReferenceMetadata] struct {
	Message TMessage
	Patcher *ReferenceMessagePatcher[TMetadata]
}

PatchedMessage is a tuple for storing a Protobuf message that contains model_core_pb.Reference messages, and the associated ReferenceMessagePatcher that can be used to assign indices to these references.

func BuildPatchedMessage 

func BuildPatchedMessage[TMessage any, TMetadata ReferenceMetadata](
	builder func(*ReferenceMessagePatcher[TMetadata]) (TMessage, error),
) (PatchedMessage[TMessage, TMetadata], error)

BuildPatchedMessage is a convenience function for constructing PatchedMessage that has a newly created ReferenceMessagePatcher attached to it. A callback is invoked for creating a message that is managed by the ReferenceMessagePatcher. If the callback returns an error, any intermediate metadata is discarded.

func FlattenPatchedSingletonList 

func FlattenPatchedSingletonList[TMessage any, TMetadata ReferenceMetadata](m PatchedMessage[[]TMessage, TMetadata]) PatchedMessage[TMessage, TMetadata]

FlattenPatchedSingletonList converts a PatchedMessage containing a single element list to a PatchedMessage containing just that single element.

A dedicated function for this is provided, as this is one of the few cases in which obtaining a PatchedMessage for a nested message value is safe. In the general case it is not, as it would cause the resulting PatchedMessage to contain unused reference metadata.

func MarshalAny 

func MarshalAny[TMessage proto.Message, TMetadata ReferenceMetadata](m PatchedMessage[TMessage, TMetadata]) (PatchedMessage[*model_core_pb.Any, TMetadata], error)

MarshalAny wraps a patched message into a model_core_pb.Any message. References stored in the original message are kept intact.

func MustBuildPatchedMessage 

func MustBuildPatchedMessage[TMessage any, TMetadata ReferenceMetadata](
	builder func(*ReferenceMessagePatcher[TMetadata]) TMessage,
) PatchedMessage[TMessage, TMetadata]

MustBuildPatchedMessage is equivalent to BuildPatchedMessage, except that the callback that is invoked may not fail.

func NewPatchedMessage 

func NewPatchedMessage[TMessage any, TMetadata ReferenceMetadata](
	message TMessage,
	patcher *ReferenceMessagePatcher[TMetadata],
) PatchedMessage[TMessage, TMetadata]

NewPatchedMessage creates a PatchedMessage, given an existing Protobuf message and reference message patcher.

func NewPatchedMessageFromExisting 

func NewPatchedMessageFromExisting[
	TMessage proto.Message,
	TMetadata ReferenceMetadata,
	TReference object.BasicReference,
](
	existing Message[TMessage, TReference],
	createMetadata ReferenceMetadataCreator[TMetadata],
) PatchedMessage[TMessage, TMetadata]

NewPatchedMessageFromExisting creates a PatchedMessage, given an existing Protobuf message that may contain one or more references to other objects. For each reference that is found, a callback is invoked to create metadata to associate with the reference.

func NewSimplePatchedMessage 

func NewSimplePatchedMessage[TMetadata ReferenceMetadata, TMessage any](v TMessage) PatchedMessage[TMessage, TMetadata]

NewSimplePatchedMessage is a helper function for creating instances of PatchedMessage for messages that don't contain any references.

func Patch 

func Patch[
	TMessage proto.Message,
	TMetadata ReferenceMetadata,
	TReference object.BasicReference,
](
	capturer ExistingObjectCapturer[TReference, TMetadata],
	m Message[TMessage, TReference],
) PatchedMessage[TMessage, TMetadata]

Patch an existing message, copying the message and causing all containing references to be managed by a ReferenceMessagePatcher. For each references contained within, metadata is created by calling into an ExistingObjectCapturer.

func PatchList 

func PatchList[
	TMessage any,
	TMessagePtr interface {
		*TMessage
		proto.Message
	},
	TMetadata ReferenceMetadata,
	TReference object.BasicReference,
](
	capturer ExistingObjectCapturer[TReference, TMetadata],
	existingList Message[[]*TMessage, TReference],
) PatchedMessage[[]*TMessage, TMetadata]

PatchList is identical to Patch(), except that it patches a list of messages.

func ProtoListToBinaryMarshaler 

func ProtoListToBinaryMarshaler[TMessage proto.Message, TMetadata ReferenceMetadata](m PatchedMessage[[]TMessage, TMetadata]) PatchedMessage[encoding.BinaryMarshaler, TMetadata]

ProtoListToBinaryMarshaler is a convenience function for converting a PatchedMessage containing a list of Protobuf messages to one that is marshalable.

func ProtoToBinaryMarshaler 

func ProtoToBinaryMarshaler[TMessage proto.Message, TMetadata ReferenceMetadata](m PatchedMessage[TMessage, TMetadata]) PatchedMessage[encoding.BinaryMarshaler, TMetadata]

ProtoToBinaryMarshaler is a convenience function for converting a PatchedMessage containing a Protobuf message to one that is marshalable.

func (*PatchedMessage[T, TMetadata]) Clear 

func (m *PatchedMessage[T, TMetadata]) Clear()

Clear the instance of PatchedMessage, disassociating it from its message and reference message patcher. The reference message patcher is not discarded, meaning that any resources owned by reference metadata is not released.

func (*PatchedMessage[T, TMetadata]) Discard 

func (m *PatchedMessage[T, TMetadata]) Discard()

Discard the reference message patcher, releasing any resources owned by reference metadata.

func (PatchedMessage[T, TMetadata]) IsSet 

func (m PatchedMessage[T, TMetadata]) IsSet() bool

IsSet returns true if the PatchedMessage is assigned to a message and its associated reference message patcher.

func (PatchedMessage[T, TMetadata]) Merge 

func (m PatchedMessage[T, TMetadata]) Merge(patcher *ReferenceMessagePatcher[TMetadata]) T

Merge the references contained in the patched message into a provided patcher, and return the message that is now owned by the provided patcher.

func (*PatchedMessage[T, TMetadata]) Move 

func (m *PatchedMessage[T, TMetadata]) Move() PatchedMessage[T, TMetadata]

Move the message out of the PatchedMessage by copying it and clearing the original instance.

func (PatchedMessage[T, TMetadata]) SortAndSetReferences 

func (m PatchedMessage[T, TMetadata]) SortAndSetReferences() (TopLevelMessage[T, object.LocalReference], []TMetadata)

SortAndSetReferences assigns indices to outgoing references.

type PatchedMessageList 

type PatchedMessageList[TMessage any, TMetadata ReferenceMetadata] []PatchedMessage[TMessage, TMetadata]

PatchedMessageList is a list of PatchedMessage objects, which have not been merged into a single PatchedMessage yet.

func (PatchedMessageList[TMessage, TMetadata]) Discard 

func (l PatchedMessageList[TMessage, TMetadata]) Discard()

Discard all messages contained in the list.

func (PatchedMessageList[TMessage, TMetadata]) Merge 

func (l PatchedMessageList[TMessage, TMetadata]) Merge() PatchedMessage[[]TMessage, TMetadata]

Merge all the messages in the list, so that references contained in all messages are managed by a single ReferenceMessagePatcher.

type ReferenceMessagePatcher 

type ReferenceMessagePatcher[TMetadata ReferenceMetadata] struct {
	// contains filtered or unexported fields
}

ReferenceMessagePatcher keeps track of all Reference messages that are contained within a given Protobuf message. For each of these Reference messages, it stores the actual object.LocalReference that is associated with them.

Once the Protobuf message has been fully constructed, SortAndSetReferences() can be used to return a sorted, deduplicated list of all outgoing references of the message, and to update the indices in the Reference messages to refer to the correct outgoing reference.

func MapReferenceMessagePatcherMetadata 

func MapReferenceMessagePatcherMetadata[TOld, TNew ReferenceMetadata](pOld *ReferenceMessagePatcher[TOld], mapMetadata func(MetadataEntry[TOld]) TNew) *ReferenceMessagePatcher[TNew]

MapReferenceMessagePatcherMetadata replaces a ReferenceMessagePatcher with a new instance that contains the same references, but has metadata mapped to other values, potentially of another type.

func MapReferenceMetadataToWalkers 

func MapReferenceMetadataToWalkers[TMetadata WalkableReferenceMetadata](p *ReferenceMessagePatcher[TMetadata]) *ReferenceMessagePatcher[dag.ObjectContentsWalker]

MapReferenceMetadataToWalkers replaces all the reference metadata in a patched message with instances of dag.ObjectContentsWalker.

This function can be used in contexts where we only care about transmitting the contents of a DAG to a server, and no longer need to access any of the additional metadata that was gathered.

func NewReferenceMessagePatcher 

func NewReferenceMessagePatcher[TMetadata ReferenceMetadata]() *ReferenceMessagePatcher[TMetadata]

NewReferenceMessagePatcher creates a new ReferenceMessagePatcher that does not contain any Reference messages.

func (*ReferenceMessagePatcher[TMetadata]) AddDecodableReference 

func (p *ReferenceMessagePatcher[TMetadata]) AddDecodableReference(capturedObject Decodable[MetadataEntry[TMetadata]]) *core.DecodableReference

AddDecodableReference is a helper function for AddReference that can be called in the common case where a DecodableReference needs to be emitted.

func (*ReferenceMessagePatcher[TMetadata]) AddReference 

func (p *ReferenceMessagePatcher[TMetadata]) AddReference(capturedObject MetadataEntry[TMetadata]) *core.Reference

AddReference allocates a new Reference message that is associated with a given object.LocalReference and caller provided metadata.

func (*ReferenceMessagePatcher[TMetadata]) CaptureAndAddDecodableReference 

func (p *ReferenceMessagePatcher[TMetadata]) CaptureAndAddDecodableReference(
	ctx context.Context,
	createdObject Decodable[CreatedObject[TMetadata]],
	capturer CreatedObjectCapturer[TMetadata],
) (*core.DecodableReference, error)

CaptureAndAddDecodableReference is a helper function for AddDecodableReference, which first captures an object and then creates a decodable reference message for it which can be attached to a parent object.

func (*ReferenceMessagePatcher[TMetadata]) Discard 

func (p *ReferenceMessagePatcher[TMetadata]) Discard()

Discard all resources owned by all metadata managed by this reference message patcher.

func (*ReferenceMessagePatcher[TMetadata]) GetHeight 

func (p *ReferenceMessagePatcher[TMetadata]) GetHeight() int

GetHeight returns the height that the object of the Protobuf message would have if it were created with the current set of outgoing references.

func (*ReferenceMessagePatcher[TMetadata]) GetReferencesSizeBytes 

func (p *ReferenceMessagePatcher[TMetadata]) GetReferencesSizeBytes() int

GetReferencesSizeBytes returns the size that all of the outgoing references would have if an object were created with the current set of outgoing references.

func (*ReferenceMessagePatcher[TMetadata]) Merge 

func (p *ReferenceMessagePatcher[TMetadata]) Merge(other *ReferenceMessagePatcher[TMetadata])

Merge multiple instances of ReferenceMessagePatcher together. This method can be used when multiple Protobuf messages are combined into a larger message, and are eventually stored as a single object.

func (*ReferenceMessagePatcher[TMetadata]) SortAndSetReferences 

func (p *ReferenceMessagePatcher[TMetadata]) SortAndSetReferences() (object.OutgoingReferencesList[object.LocalReference], []TMetadata)

SortAndSetReferences returns a sorted list of all outgoing references of the Protobuf message. This list can be provided to object.NewContents() to construct an actual object for storage. In addition to that, a list of user provided metadata is returned that sorted along the same order.

type ReferenceMetadata 

type ReferenceMetadata interface {
	Discard()
}

ReferenceMetadata is freeform metadata that can be associated with any reference managed by a ReferenceMessagePatcher.

Simple implementations of ReferenceMessagePatcher may capture the contents of the object referenced, causing the full Merkle tree to reside in memory. Alternatively, implementations may also store information on how the data that is referenced was obtained, so that it may be recomputed if needed.

As ReferenceMatadata may contain things like file descriptors, a Discard() method is provided to release such resources in case ReferenceMessagePatcher encounters duplicate references.

type ReferenceMetadataCreator 

type ReferenceMetadataCreator[T any] func(index int) T

ReferenceMetadataCreator is invoked by NewPatchedMessageFromExisting for every reference that is encountered, allowing metadata for the outgoing reference to be created.

type ReferenceMetadataCreatorForTesting 

type ReferenceMetadataCreatorForTesting ReferenceMetadataCreator[ReferenceMetadata]

ReferenceMetadataCreatorForTesting is an instantiation of ReferenceMetadataCreator for generating mocks to be used by tests.

type TopLevelMessage 

type TopLevelMessage[TMessage any, TReference any] struct {
	Message            TMessage
	OutgoingReferences object.OutgoingReferences[TReference]
}

TopLevelMessage is identical to Message, except that it may only refer to a message for which all embedded references are consecutively numbered, and the outgoing references list contains no references in addition to the ones embedded in the message.

As top-level messages are considered being canonical, this type provides some additional methods for generating keys and performing comparisons.

func FlattenAny 

func FlattenAny[TReference any](m Message[*model_core_pb.Any, TReference]) (TopLevelMessage[*anypb.Any, TReference], error)

FlattenAny extracts the anypb.Any message that's embedded in a model_core_pb.Any message. The resulting message only has access to the outgoing references of the inner message.

func MarshalTopLevelAny 

func MarshalTopLevelAny[TMessage proto.Message, TReference any](m TopLevelMessage[TMessage, TReference]) (TopLevelMessage[*anypb.Any, TReference], error)

MarshalTopLevelAny wraps a message in an anypb.Any.

This function assumes that the provided message is a top-level message. Only top-level messages are permitted to be wrapped in an anypb.Any, as it's impossible to create a model_core_pb.Any for messages having outgoing references with sparse indices.

func NewSimpleTopLevelMessage 

func NewSimpleTopLevelMessage[TReference, TMessage any](m TMessage) TopLevelMessage[TMessage, TReference]

NewSimpleTopLevelMessage creates a TopLevelMessage that has no outgoing references.

func NewTopLevelMessage 

func NewTopLevelMessage[TMessage, TReference any](
	m TMessage,
	outgoingReferences object.OutgoingReferences[TReference],
) TopLevelMessage[TMessage, TReference]

NewTopLevelMessage creates a TopLevelMessage that corresponds to an already existing Protobuf message.

func UnmarshalAnyNew 

func UnmarshalAnyNew[TReference any](m Message[*model_core_pb.Any, TReference]) (TopLevelMessage[proto.Message, TReference], error)

UnmarshalAnyNew extracts the message contained in a model_core.Any, and ensures that any references contained within are accessible.

func UnmarshalTopLevelAnyNew 

func UnmarshalTopLevelAnyNew[TReference any](m TopLevelMessage[*anypb.Any, TReference]) (TopLevelMessage[proto.Message, TReference], error)

UnmarshalTopLevelAnyNew extracts the message contained in an anypb.Any, and ensures that any references contained within are accessible.

This function assumes that the anypb.Any is a top-level message, meaning that the inner message has access to the same set of references as the outer message.

func UnmarshalTopLevelMessage 

func UnmarshalTopLevelMessage[
	TMessage any,
	TMessagePtr interface {
		*TMessage
		proto.Message
	},
](referenceFormat object.ReferenceFormat, data []byte) (TopLevelMessage[TMessagePtr, object.LocalReference], error)

UnmarshalTopLevelMessage performs the inverse of MarshalTopLevelMessage. It can be used to convert a byte sequence to a top-level Protobuf message.

func Unpatch 

func Unpatch[TMessage, TReference any, TMetadata ReferenceMetadata](
	referencer ObjectReferencer[TReference, TMetadata],
	m PatchedMessage[TMessage, TMetadata],
) TopLevelMessage[TMessage, TReference]

Unpatch performs the opposite of Patch(). Namely, it assigns indices to all references contained in the current message. Metadata that is tracked by the ReferenceMessagePatcher is converted to references by calling into an ObjectReferencer.

func WrapTopLevelAny 

func WrapTopLevelAny[TReference any](m TopLevelMessage[*anypb.Any, TReference]) TopLevelMessage[*model_core_pb.Any, TReference]

WrapTopLevelAny performs te inverse of FlattenAny. Namely, it wraps an anypb.Any into a model_core_pb.Any. This allows one or more anypb.Any messages to be stored in a single object, together with other data and references.

func (TopLevelMessage[TMessage, TReference]) Decay 

func (m TopLevelMessage[TMessage, TReference]) Decay() Message[TMessage, TReference]

Decay a top-level message to a plain message.

type WalkableReferenceMetadata 

type WalkableReferenceMetadata interface {
	ReferenceMetadata

	ToObjectContentsWalker() dag.ObjectContentsWalker
}

WalkableReferenceMetadata can be implemented by reference metadata to indicate that they can be converted to a dag.ObjectContentsWalker, allowing the contents of any captured objects to be obtained up to once. This allows them to be written to a storage server.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.