dulwich.pack

module documentation

Classes for dealing with packed git objects.

A pack is a compact representation of a bunch of objects, stored using deltas where possible.

They have two parts, the pack file, which stores the data, and an index that tells you where the data is.

To find an object you look in all of the index files 'til you find a match for the object name. You then use the pointer got from this as a pointer in to the corresponding packfile.

Class	`DeltaChainIterator`	Abstract iterator over pack data based on delta chains.
Class	`FilePackIndex`	Pack index that is based on a file.
Class	`MemoryPackIndex`	Pack index that is stored entirely in memory.
Class	`Pack`	A Git pack object.
Class	`PackChunkGenerator`	No class docstring; 0/3 instance variable, 1/4 method documented
Class	`PackData`	The data contained in a packfile.
Class	`PackIndex`	An index in to a packfile.
Class	`PackIndex1`	Version 1 Pack Index file.
Class	`PackIndex2`	Version 2 Pack Index file.
Class	`PackIndexer`	Delta chain iterator that yields index entries.
Class	`PackInflater`	Delta chain iterator that yields ShaFile objects.
Class	`PackStreamCopier`	Class to verify a pack stream as it is being read.
Class	`PackStreamReader`	Class to read a pack stream.
Class	`SHA1Reader`	Wrapper for file-like object that remembers the SHA1 of its data.
Class	`SHA1Writer`	Wrapper for file-like object that remembers the SHA1 of its data.
Class	`UnpackedObject`	Class encapsulating an object unpacked from a pack file.
Exception	`PackFileDisappeared`	Undocumented
Function	`apply_delta`	Based on the similar function in git's patch-delta.c.
Function	`bisect_find_sha`	Find a SHA in a data blob with sorted SHAs.
Function	`chunks_length`	Undocumented
Function	`compute_file_sha`	Hash a portion of a file into a new SHA.
Function	`create_delta`	Use python difflib to work out how to transform base_buf to target_buf.
Function	`deltify_pack_objects`	Generate deltas for pack objects.
Function	`iter_sha1`	Return the hexdigest of the SHA1 over a set of names.
Function	`load_pack_index`	Load an index file by path.
Function	`load_pack_index_file`	Load an index file from a file-like object.
Function	`obj_sha`	Compute the SHA for a numeric type and object chunks.
Function	`pack_header_chunks`	Yield chunks for a pack header.
Function	`pack_object_chunks`	Generate chunks for a pack object.
Function	`pack_object_header`	Create a pack object header for the given object info.
Function	`pack_objects_to_data`	Create pack data from objects
Function	`read_pack_header`	Read the header of a pack file.
Function	`read_zlib_chunks`	Read zlib data from a buffer.
Function	`take_msb_bytes`	Read bytes marked with most significant bit.
Function	`unpack_object`	Unpack a Git object.
Function	`write_pack`	Write a new pack data file.
Function	`write_pack_data`	Write a new pack data file.
Function	`write_pack_header`	Write a pack header for the given number of objects.
Function	`write_pack_index_v1`	Write a new pack index file.
Function	`write_pack_index_v2`	Write a new pack index file.
Function	`write_pack_object`	Write pack object to a file.
Function	`write_pack_objects`	Write a new pack data file.
Constant	`DEFAULT_PACK_DELTA_WINDOW_SIZE`	Undocumented
Constant	`DELTA_TYPES`	Undocumented
Constant	`OFS_DELTA`	Undocumented
Constant	`REF_DELTA`	Undocumented
Variable	`has_mmap`	Undocumented
Class	`_PackTupleIterable`	Helper for Pack.pack_tuples.
Function	`_compute_object_size`	Compute the size of a unresolved object for use with LRUSizeCache.
Function	`_delta_encode_size`	Undocumented
Function	`_encode_copy_operation`	Undocumented
Function	`_load_file_contents`	Undocumented
Constant	`_MAX_COPY_LEN`	Undocumented
Constant	`_ZLIB_BUFSIZE`	Undocumented

def apply_delta(src_buf, delta): ¶

Based on the similar function in git's patch-delta.c.

Parameters
src_buf	Source buffer
delta	Delta instructions

def bisect_find_sha(start, end, sha, unpack_name): ¶

Find a SHA in a data blob with sorted SHAs.

Returns: Index of the SHA, or None if it wasn't found

Parameters
start	Start index of range to search
end	End index of range to search
sha	Sha to find
unpack_name	Callback to retrieve SHA by index

def chunks_length(chunks): ¶

Undocumented

def compute_file_sha(f, start_ofs=0, end_ofs=0, buffer_size=1<<16): ¶

Hash a portion of a file into a new SHA.

Returns: A new SHA object updated with data read from the file.

Parameters
f	A file-like object to read from that supports seek().
start_ofs	The offset in the file to start reading at.
end_ofs	The offset in the file to end reading at, relative to the end of the file.
buffer_size	A buffer size for reading.

def create_delta(base_buf, target_buf): ¶

Use python difflib to work out how to transform base_buf to target_buf.

Parameters
base_buf	Base buffer
target_buf	Target buffer

def deltify_pack_objects(objects, window_size=None): ¶

Generate deltas for pack objects.

Returns: Iterator over type_num, object id, delta_base, content: delta_base is None for full text entries

Parameters
objects	An iterable of (object, path) tuples to deltify.
window_size	Window size; None for default

def iter_sha1(iter): ¶

Return the hexdigest of the SHA1 over a set of names.

Returns: 40-byte hex sha1 digest

Parameters
iter	Iterator over string objects

def load_pack_index(path): ¶

Load an index file by path.

Returns: A PackIndex loaded from the given path

Parameters
path	Path to the index file

def load_pack_index_file(path, f): ¶

Load an index file from a file-like object.

Returns: A PackIndex loaded from the given file

Parameters
path	Path for the index file
f	File-like object

def obj_sha(type, chunks): ¶

Compute the SHA for a numeric type and object chunks.

def pack_header_chunks(num_objects): ¶

Yield chunks for a pack header.

def pack_object_chunks(type, object, compression_level=-1): ¶

Generate chunks for a pack object.

Returns: Chunks

Parameters
type	Numeric type of the object
object	Object to write
compression_level	the zlib compression level

def pack_object_header(type_num, delta_base, size): ¶

Create a pack object header for the given object info.

Returns: A header for a packed object.

Parameters
type_num	Numeric type of the object.
delta_base	Delta base offset or ref, or None for whole objects.
size	Uncompressed object size.

def pack_objects_to_data(objects): ¶

Create pack data from objects

Returns: Tuples with (type_num, hexdigest, delta base, object chunks)

Parameters
objects	Pack objects

def read_pack_header(read): ¶

Read the header of a pack file.

Returns: Tuple of (pack version, number of objects). If no data is: available to read, returns (None, None).

Parameters
read	Read function

def read_zlib_chunks(read_some, unpacked, include_comp=False, buffer_size=_ZLIB_BUFSIZE): ¶

Read zlib data from a buffer.

This function requires that the buffer have additional data following the compressed data, which is guaranteed to be the case for git pack files.

Returns: Leftover unused data from the decompression.

Parameters
read_some	Read function that returns at least one byte, but may return less than the requested size.
unpacked	An UnpackedObject to write result data to. If its crc32 attr is not None, the CRC32 of the compressed bytes will be computed using this starting CRC32. After this function, will have the following attrs set: * comp_chunks (if include_comp is True) * decomp_chunks * decomp_len * crc32
include_comp	If True, include compressed data in the result.
buffer_size	Size of the read buffer.
Raises
`zlib.error`	if a decompression error occurred.

def take_msb_bytes(read, crc32=None): ¶

Read bytes marked with most significant bit.

Parameters
read	Read function
crc32	Undocumented

def unpack_object(read_all, read_some=None, compute_crc32=False, include_comp=False, zlib_bufsize=_ZLIB_BUFSIZE): ¶

Unpack a Git object.

Returns: A tuple of (unpacked, unused), where unused is the unused data

leftover from decompression, and unpacked in an UnpackedObject with the following attrs set:

obj_chunks (for non-delta types)
pack_type_num
delta_base (for delta types)
comp_chunks (if include_comp is True)
decomp_chunks
decomp_len
crc32 (if compute_crc32 is True)

Parameters
read_all	Read function that blocks until the number of requested bytes are read.
read_some	Read function that returns at least one byte, but may not return the number of bytes requested.
compute_crc32	If True, compute the CRC32 of the compressed data. If False, the returned CRC32 will be None.
include_comp	If True, include compressed data in the result.
zlib_bufsize	An optional buffer size for zlib operations.

def write_pack(filename, objects, deltify=None, delta_window_size=None, compression_level=-1): ¶

Write a new pack data file.

Returns: Tuple with checksum of pack file and index file

Parameters
filename	Path to the new pack file (without .pack extension)
objects	(object, path) tuple iterable to write. Should provide __len__
deltify	Whether to deltify pack objects
delta_window_size	Delta window size
compression_level	the zlib compression level

def write_pack_data(write, num_records=None, records=None, progress=None, compression_level=-1): ¶

Write a new pack data file.

Returns: Dict mapping id -> (offset, crc32 checksum), pack checksum

Parameters
write	Write function to use
num_records	Number of records (defaults to len(records) if None)
records	Iterator over type_num, object_id, delta_base, raw
progress	Function to report progress to
compression_level	the zlib compression level

def write_pack_header(write, num_objects): ¶

Write a pack header for the given number of objects.

def write_pack_index_v1(f, entries, pack_checksum): ¶

Write a new pack index file.

Returns: The SHA of the written index file

Parameters
f	A file-like object to write to
entries	List of tuples with object name (sha), offset_in_pack, and crc32_checksum.
pack_checksum	Checksum of the pack file.

def write_pack_index_v2(f, entries, pack_checksum): ¶

Write a new pack index file.

Returns: The SHA of the index file written

Parameters
f	File-like object to write to
entries	List of tuples with object name (sha), offset_in_pack, and crc32_checksum.
pack_checksum	Checksum of the pack file.

def write_pack_object(write, type, object, sha=None, compression_level=-1): ¶

Write pack object to a file.

Returns: Tuple with offset at which the object was written, and crc32

Parameters
write	Write function to use
type	Numeric type of the object
object	Object to write
sha	Undocumented
compression_level	the zlib compression level

def write_pack_objects(write, objects, delta_window_size=None, deltify=None, compression_level=-1): ¶

Write a new pack data file.

Returns: Dict mapping id -> (offset, crc32 checksum), pack checksum

Parameters
write	write function to use
objects	Iterable of (object, path) tuples to write. Should provide __len__
delta_window_size	Sliding window size for searching for deltas; Set to None for default window size.
deltify	Whether to deltify objects
compression_level	the zlib compression level to use

DEFAULT_PACK_DELTA_WINDOW_SIZE: int = ¶

Undocumented

Value

DELTA_TYPES = ¶

Undocumented

Value

(OFS_DELTA, REF_DELTA)

OFS_DELTA: int = ¶

Undocumented

Value

REF_DELTA: int = ¶

Undocumented

Value

has_mmap: bool = ¶

Undocumented

def _compute_object_size(value): ¶

Compute the size of a unresolved object for use with LRUSizeCache.

def _delta_encode_size(size): ¶

Undocumented

Returns
`bytes`	Undocumented

def _encode_copy_operation(start, length): ¶

Undocumented

def _load_file_contents(f, size=None): ¶

Undocumented

_MAX_COPY_LEN: int = ¶

Undocumented

Value

_ZLIB_BUFSIZE: int = ¶

Undocumented

Value