Aqueous Archive Format

Editor’s Draft,

Issue Tracking:
GitHub
Inline In Spec
Editor:
Kang Seonghoon
Repository:
github.com/lifthrasiir/aqueous/tree/main/spec

Abstract

This specification defines the Aqueous archive format, a compressed archive format designed for efficient storage and retrieval of files, and the Aqueous overlay archive format, which describes a directory tree as changes to an existing one.

1. Introduction

This section is non-normative.

The Aqueous archive format stores a directory tree, or a single file, as a sequence of blocks. This section gives an overview of how the parts of the format fit together; each paragraph points to the chapter defining the part it describes.

An archive is a header, a sequence of blocks and a footer (§ 3 Container). Every block starts with its type and its size, so a decompressor can skip any block without understanding it, and the type tells whether the block can be ignored when it is not understood (§ 3.1.1 Block types). The footer points back to the last metadata block, so that a decompressor can start reading from the end of the archive.

Blocks are grouped into segments, each ending with a metadata block (§ 4 Segments and entries). The metadata block names the entries of the segment, i.e. directories, files, symbolic links and other nodes, and points each file to the block holding its content. A later segment adds, replaces and removes entries of earlier segments, so that an archive can be extended by appending a segment without rewriting what precedes it (§ 3.7 Appending to an archive). An archive without any metadata block simply decompresses into a single unnamed file, as a gzip file does.

The content of a file is given by a single leaf block. The simplest leaf block is a data block storing the content itself, optionally compressed with Zstandard and a dictionary shared by many small blocks (§ 5 Stored data). A formula block is a leaf block that describes the content instead, as a small stack machine program over non-leaf blocks following it (§ 6 Formulas). The program takes slices of them, concatenates them, patches them, and applies WebAssembly modules to them to undo a transformation made by the compressor (§ 7 Modules). A leaf block and the non-leaf blocks following it form a parcel, which is typically all that a decompressor reads to extract a single file, and the formula block serves as an index for random access within it.

An overlay archive describes a directory tree as changes to an existing one, called its base (§ 8 Overlay archives and imports). Import blocks read byte ranges of files in the base, which formulas can patch, so that an overlay archive works as an update or a delta of the base.

The whole archive may be encrypted with a passphrase using age [AGE] (§ 9 Encrypted archives). Checksums cover every block except, optionally, leaf data blocks, and except shorthand blocks, which name registered dictionaries and modules by their hashes (§ 4.2.2 Registered dictionaries and modules). Mirror blocks hold copies of the blocks whose damage would affect many files (§ 10 Integrity and recovery). Metadata that is not necessary to read the files, such as timestamps, comments and permissions, is encoded in CBOR and may be ignored by a decompressor (§ 11 Optional metadata, § 12 Extraction).

2. Conventions

2.1. Expressions

All expressions are interpreted according to the standard arithmetics, with the following extensions:

All expressions result in an integer between -(264 - 1) and 264 - 1, or an array of those integers with the number of elements not exceeding 264 - 1. Note that subexpressions still may exceed this range.

2.2. Types

Types are a set of values and associated encoding scheme. A type or a structure is implicitly-sized when its byte length is only available through the context. Unless stated as being implicitly-sized, all types have the defined byte length out of the context.

u8 is an unsigned 8-bit integer, occupying 1 byte. Its range is [0, 28).

u16le is an unsigned 16-bit integer, occupying 2 bytes, stored in little-endian byte order (least significant byte first). Its range is [0, 216).

u32le is an unsigned 32-bit integer, occupying 4 bytes, stored in little-endian byte order. Its range is [0, 232).

u64le is an unsigned 64-bit integer, occupying 8 bytes, stored in little-endian byte order. Its range is [0, 264).

u(n), where n is an integer less than 264, is the smallest integer type that can represent n if n > 0, and a zero-sized type if n = 0. In other words,

leb128 is an unsigned 64-bit integer encoded using the variable number of bytes with the LEB128 encoding. Each byte, except the last, has its most significant bit (MSB) set to 1, indicating that more bytes follow. The remaining 7 bits of each byte are used to store the integer’s value. The least significant 7 bits of the first byte are the least significant bits of the integer, and so on. The last byte has its MSB set to 0. Its range is [0, 264): an encoding MUST be at most 10 bytes long, and MUST NOT encode a value of 264 or more.

This allows multiple encodings for any number less than 263, padded with bytes of zero bits, unless the shortest encoding is required.

a[n], where n is an integer, denotes an ordered array of as. n’s range is [0, 264).

a[*] denotes an implicitly-sized ordered array of as with an unspecified number of elements. The number of elements is implicitly determined from the context. This specification uses two kinds of such contexts:

2.3. Structure

Structure is a logical group of one or more ordered fields of possibly differing types concatenated together. Each field has the following properties:

Field names can only coincide when the field’s type varies by conditions, in which case at most one such condition is applicable at any time. The name can be omitted when the field’s name is the same as the previous field’s.

The field type may be offset, which is a special zero-sized type interpreted as the offset to the following byte relative to the unspecified file position (origin). Offsets are only used to measure the number of bytes within a set of fields. The syntax and semantics of this specification don’t depend on the choice of origin.

The optional constraint does not affect the field’s encoding and only narrows down the possible set of values. Constraints can be one of the following:

The conditions may be one of the following:

Structures are implicitly-sized only when their byte length is not available through the context and at least one implicitly-sized field exists therein.

2.4. Constants

This specification defines and uses the following constants, all of which are denoted in UPPERCASE.

Name Value
HEADER_TYPE 27816
OVERLAY_HEADER_TYPE 6916
FOOTER_TYPE 90 5BCB 957D16
OVERLAY_FOOTER_TYPE 7FC16
ENCRYPTED_HEADER_TYPE 155 04DC A57916
HEADER_MAGIC (F8 04 03 6A D6 BE)16
OVERLAY_HEADER_MAGIC (69 0D 15 AF F6 B2)16
FOOTER_MAGIC (FD AA AE DE 85 12)16
OVERLAY_FOOTER_MAGIC (FC 0F 6E 7B 4B AC)16
ENCRYPTED_HEADER_MAGIC (F9 CA F2 A6 D0 2A 01 00)16
ENCRYPTED_MIRRORED_HEADER_MAGIC (F9 CA F2 A6 D0 2A 02 00)16
MIRROR_MAGIC (02 AA F5 FE BA 2B AE 23)16
OVERLAY_MIRROR_MAGIC (01 05 34 FA BD 11 AF 53)16
ENCRYPTED_MIRROR_MAGIC (78 D7 3F 69 0A FF 44 83)16
RECOVERY_MAGIC (69 0A D1 79 CD 15 7A B6)16
OVERLAY_RECOVERY_MAGIC (45 E7 34 55 EA F2 E1 0D)16
The encoding of HEADER_TYPE, OVERLAY_HEADER_TYPE, FOOTER_TYPE, OVERLAY_FOOTER_TYPE and ENCRYPTED_HEADER_TYPE as a block type (§ 3.1 Block) is a prefix of HEADER_MAGIC, OVERLAY_HEADER_MAGIC, FOOTER_MAGIC, OVERLAY_FOOTER_MAGIC and ENCRYPTED_HEADER_MAGIC respectively, and that of ENCRYPTED_HEADER_TYPE is also a prefix of ENCRYPTED_MIRRORED_HEADER_MAGIC.

2.5. Semantics and failures

This specification defines the meaning of an archive as a function from a byte sequence to an archive value, which also takes a base for an overlay archive. Every algorithm in this specification is declarative: it defines what the result is, not how or in which order it has to be computed, and a conforming implementation MAY compute it in any other way that produces the same result.

The result of interpreting an archive is either an archive value or a failure, which is one of the following:

An unsupported failure tells that a decompressor implementing a newer version may be able to interpret the input, which is not the case for an invalid archive failure in general. It takes precedence over an invalid archive failure in any block (§ 4.4 Interpreting an archive), since a future version may give a different meaning to the other blocks.

A conforming compressor, given an archive value, MUST produce a byte sequence whose interpretation is that archive value. When it produces an overlay archive, it is also given a base, and the interpretation with that base MUST be that archive value. This specification doesn’t constrain how the compressor chooses blocks, block sizes, dictionaries or formulas.

3. Container

3.1. Block

Blocks are a typed byte sequence. The entirety of an unencrypted archive and the beginning of an encrypted archive consist of either blocks or block-like structures.

Blocks have the following generic structure:

Name Type Conditions
block_start offset
type leb128 ∉ {FOOTER_TYPE, OVERLAY_FOOTER_TYPE}
size u8 If type % 4 = 0
u16le If type % 4 = 1
u32le If type % 4 = 2
u64le If type % 4 = 3
data u8[size] Depends on type
block_end offset
padding u8 If (block_end - block_start) % 2 = 1

Some type values, including FOOTER_TYPE and OVERLAY_FOOTER_TYPE, are also used to denote block-like structures. For example the archive footer has to have a distinct block type in order to be placed after blocks. This specification doesn’t assign any meaning to such block-like structures unless explicitly stated.

type MUST use the shortest leb128 encoding.

The presence of padding ensures that there are even number of bytes in any block. Conforming compressors MUST set padding to zero, but its value doesn’t affect the result of interpreting an archive.

padding carries no information, so its damage is not a reason to reject an otherwise readable archive. On the other hand, type determines the width of size and where the next block starts, so a single encoding for each type leaves no ambiguity in finding block boundaries, e.g. by scanning.

Later sections describe the contents of data field according to each defined type. Since type also determines the byte length of size, there are generally multiple type values assigned for each logical block type.

3.1.1. Block types

The following table lists every range of block types. A range of block types for a single logical block type is also described in binary, e.g. 0001 0mss, where letters denote bits that determine the parameters of the block. ss always denotes the two least significant bits, which determine the width of size as above: 0 for u8, 1 for u16le, 2 for u32le and 3 for u64le. Whether a reserved block type is critical is determined by its leading digit (§ 3.1.1.1 Reserved block types).

Block types of 2 hexadecimal digits are as follows:

Block Types Critical Definition
0016 — 0316 Yes § 3.3 Spacer block
0416 Yes § 3.4 Size hint block
0516 — 0F16 Yes Reserved
1016 — 1716 Yes § 4.3 Metadata block
1816 — 1F16 Yes Reserved
2016 — 3716 Yes § 5.1 Data block
3816 — 3F16 Yes Reserved
4016 — 4B16 Yes § 5.1 Data block
4C16 — 4F16 Yes Reserved
5016 Yes § 5.2 Dictionary block, as a shorthand block (§ 4.2.2 Registered dictionaries and modules)
5116 — 5716 Yes § 5.2 Dictionary block
5816 — 5F16 Yes § 8.1 Import block
6016 Yes § 7.1 Module block, as a shorthand block (§ 4.2.2 Registered dictionaries and modules)
6116 — 6316 Yes § 7.1 Module block
6416 — 6716 Yes § 7.3 Resource block
6816 Yes Reserved
6916 Yes Reserved for OVERLAY_HEADER_TYPE (§ 3.1.1.1 Reserved block types)
6A16 — 6F16 Yes Reserved
7016 — 7716 Yes § 6.1 Formula block
7816 — 9F16 Yes Reserved
A016 — BF16 No Reserved
C016 No Reserved (§ 10.5 Recovery sets)
C116 — C316 No Parity block (§ 10.5 Recovery sets)
C416 — C716 No Index block (§ 10.5 Recovery sets)
C816 — CF16 No Reserved
D016 — DF16 No Reserved for development environments
E016 — EF16 No Reserved for experiments
F016 — FF16 No Reserved

Block types of 3 or more hexadecimal digits are as follows:

Block Types Critical Definition
10016 — 9FF16, 100016 — 9FFF16, … Yes Reserved, including block-like structures (§ 3.1.1.1 Reserved block types)
A0016 — BFF16, A00016 — BFFF16, … No Reserved
C0016 — C0F16 No Reserved
C1016 — C1716 No § 10.3 Mirror block for § 4.3 Metadata block
C1816 — C4F16 No Reserved
C5016 — C5716 No § 10.3 Mirror block for § 5.2 Dictionary block
C5816 — C5F16 No § 10.3 Mirror block for § 8.1 Import block
C6016 — C6316 No § 10.3 Mirror block for § 7.1 Module block
C6416 — C6716 No § 10.3 Mirror block for § 7.3 Resource block
C6816 — C6F16 No Reserved
C7016 — C7716 No § 10.3 Mirror block for § 6.1 Formula block
C7816 — CFF16, C00016 — CFFF16, … No Reserved
D0016 — DFF16, D00016 — DFFF16, … No Reserved for development environments
E0016 — EFF16, E00016 — EFFF16, … No Reserved for experiments
F0016 — FFF16, F00016 — FFFF16, … No Reserved
3.1.1.1. Reserved block types

The leading digit of a block type is the first digit of its hexadecimal representation, padded with zeros to at least 2 digits. For example, the leading digits of 416, 4B16 and C1016 are 0, 4 and C respectively.

A block type is critical if its leading digit is 0 through 9, and non-critical if it is A through F. A block whose type is not defined by this specification:

Every block type defined by this specification is critical except for mirror blocks and recovery blocks, which a decompressor not using them can skip. The non-critical types with the leading digit C are kept for blocks used for recovery. Future versions of this specification will use a critical type for a block without which an archive can’t be correctly interpreted, e.g. a data block with a new compression method in 3816 through 3F16, and a non-critical type for a block that can be safely skipped. The block-like structures below also have critical types, so they are never skipped by mistake.

Blocks whose type’s leading digit is D (i.e. D016 through DF16, D0016 through DFF16, …) are reserved for development environments. This specification doesn’t assign any meanings to these blocks. These blocks MUST be written or read only with specific instructions to enable them.

Blocks whose type’s leading digit is E (i.e. E016 through EF16, E0016 through EFF16, …) are reserved for experiments. This specification doesn’t assign any meanings to those blocks.

Blocks reserved for development environments and experiments are non-critical, so a decompressor not enabling them ignores them. An implementation enabling an experiment may treat its blocks as critical, since the experiment is expected to move to a block type defined by this specification once finished.

The following block types are defined as invalid due to the use as block-like structures. A valid archive MUST NOT include such block types as a proper block.

3.2. Shared external structure

Many block types share the same external structure, only varying its internal structure. The structure may or may not be compressed, and uses a compression_method and a checksum_method specified by the parent structure. Each compression method defines the structure of its parameters (compression_params), whose byte length is determined by parsing it. Each checksum_method also has an associated, fixed checksum_size. Checksummed Zstandard data is the compressed external structure (compressed_data) with Zstandard as compression_method and XXH64 as checksum_method.

The compressed external structure (compressed_data) consists of the following:

Name Type Conditions
original_size_delta leb128
original_size_shrink leb128 If original_size_delta = 0
compression_params Parameters of compression_method Depends on compression_method
data_start offset
data u8[*]
data_end offset
checksum u8[checksum_size] If checksum_method is not None, depends on checksum_method

The compressed size (compressed_size) is inferred from the block size: compressed_size = data_end - data_start. The original size (original_size) is encoded relative to compressed_size:

Each original size thus has exactly one encoding, where the original size equal to compressed_size is encoded as two zero bytes.

Compressed data is usually shorter than the original, and costs a single field in that case. The original size can still be smaller, e.g. when Zstandard stores incompressible data in raw blocks, each of which has 3 bytes of overhead. This matters for blocks that always use Zstandard, such as import blocks whose import_checksum doesn’t compress, while a block that can use None as its compression_method typically does so instead.

The uncompressed external structure (uncompressed_data) consists of the following:

Name Type Conditions
data u8[*]
checksum u8[checksum_size] If checksum_method is not None

In both structures, the checksum is computed over the decoded data (§ 5.3 Block contents), i.e. the data after decompression, rather than the stored bytes (§ 10.1 Checksums).

3.2.1. Zstandard compression method

This specification only defines a single compression method, Zstandard [RFC8878]. A Zstandard block is the "Block" structure in the Zstandard frame ([RFC8878], Section 3.1.1.2). Its parameters (compression_params) are as follows:

Name Type
window_descriptor u8

The meaning of the window_descriptor field is identical to the Zstandard’s Window_Descriptor byte, which determines Window_Size as in [RFC8878], Section 3.1.1.1.2. Consequently each Zstandard block holds at most Block_Maximum_Size = min(Window_Size, 128 KiB) bytes of data.

Conforming decompressors MUST support any Window_Size up to 227 bytes (128 MiB), and MAY return a resource failure for a larger Window_Size. Conforming compressors SHOULD NOT use a Window_Size larger than 227 bytes unless explicitly requested.

A dictionary is a byte sequence supplied to the compression method as if it preceded the data, typically shared by multiple small blocks to compensate for the lack of shared compression context. If a dictionary is in use, its content, entropy tables and repeat offsets are given by the dictionary block as described in § 5.2 Dictionary block.

Data compressed by this method is a sequence of Zstandard blocks, without an additional framing provided by Zstandard frames. data MUST consist of one or more such blocks without any other bytes, where only the last block is signalled as the final block by its block header.

The end of the compressed data is thus given both by data_end and by the final block, and they have to agree.

3.2.2. XXH64 checksum method

This specification only defines a single checksum method, XXH64 [XXHASH]. Its seed MUST be zero and the result MUST be stored as u64le. The resulting checksum_size is 8.

3.3. Spacer block

Blocks 016 through 316 are spacer blocks. Their data consists of the following:

Name Type
space u8[size]

A spacer block occupies any even number of bytes from 2, which is sufficient since every block occupies an even number of bytes due to padding.

Conforming compressors MUST fill space with zero bytes in order to avoid data leak, but its content doesn’t affect the result of interpreting an archive. Conforming compressors are RECOMMENDED to utilize sparse files for performance.

3.4. Size hint block

Block 416 is a size hint block. Its data consists of the following:

Name Type
size_hint leb128 ≠ 0

If this block is present, it MUST be the first block after the header and its size_hint field MUST be equal to the footer’s size_hint field.

3.5. Unencrypted archive

An unencrypted archive consists of a header, zero or more blocks and a footer.

Name Type Conditions
header_magic u8[6] ∈ {HEADER_MAGIC, OVERLAY_HEADER_MAGIC}
archive_start offset
size_hint_block size_hint_block Optional
blocks block[*]
archive_end offset
footer_magic u8[6] ∈ {FOOTER_MAGIC, OVERLAY_FOOTER_MAGIC}
footer_size u16le = footer_end - footer_start
footer_start offset
size_hint leb128 = archive_end - archive_start
metadata_backward_offset leb128
footer_end offset
footer_size_copy u16le = footer_size

header_magic is a fixed 6-byte sequence to identify the file as an unencrypted Aqueous archive. This header also marks the beginning of the archive, when the archive doesn’t start at the beginning of the file.

footer_magic is a fixed 6-byte sequence to mark the beginning of the footer. The footer also marks the end of the archive, when the archive doesn’t end at the end of the file.

An unencrypted archive is an overlay archive if header_magic is OVERLAY_HEADER_MAGIC, and a standalone archive otherwise. footer_magic MUST be OVERLAY_FOOTER_MAGIC in an overlay archive and FOOTER_MAGIC otherwise. An overlay archive is interpreted against a base, an existing directory tree or file given separately from the overlay archive: it starts from the base instead of an empty directory and applies its segments to it (§ 4.4 Interpreting an archive), and only an overlay archive can contain external import blocks reading the base (§ 8.1 Import block). The format of overlay archives is called the Aqueous overlay archive format, and has its own media type (§ 15.2 application/aqo Media Type).

A standalone archive is interpreted as an overlay archive would be with an empty directory as its base. Confusing the other way is harmful: an overlay archive that only adds and removes files, without any external import, would be silently interpreted without any file of its base. This can happen when an overlay archive with a damaged header is repaired into a standalone archive. Therefore every structure that can be found without the header tells whether the archive is an overlay archive: the footer, found from the end of the archive, and mirror blocks, found by scanning. Metadata blocks are only found through them, and an encrypted archive tells it in its decrypted payload.

The offset between header_magic and footer_magic is recorded as size_hint, which MAY also be replicated as size_hint_block if random access is desired.

The footer size, which is the size of all fields between footer_size and footer_size_copy (both exclusive), is recorded both in footer_size and in footer_size_copy, so that the footer can be read from the end of the archive (§ 3.6 Locating an archive).

footer_size and footer_size_copy are u16le instead of leb128. Therefore the footer size is limited to 65,535 bytes, and a single read of 65,545 bytes (including footer_magic, footer_size and footer_size_copy) is sufficient to guarantee that the entire footer is read.

metadata_backward_offset is the offset from footer_start back to the beginning of the last metadata block, just like previous_metadata_offset is the offset from the beginning of the current metadata block back to the previous one. It MUST be 0 if and only if there is no metadata block, so that a decompressor can tell such archives without scanning blocks.

blocks are logically divided into segments, each consisting of content blocks and a single metadata block describing them, with optional spacer, mirror and ignored blocks in between. Each segment can be independently processed and decompressed in the reading order, except that internal imports (§ 8.1 Import block) read the result of preceding segments. Segments are described in § 4 Segments and entries.

Alternatively, blocks can have no metadata block. Such an archive decompresses into a single unnamed file, whose content is the concatenation of the contents of all leaf blocks in the order of appearance. It decompresses into an empty file if there is no leaf block.

This allows a single file of unknown size to be compressed in a streaming fashion, by writing each unit of random access as a separate leaf block, and thus a separate parcel, as soon as it is compressed. It is analogous to a gzip file that doesn’t need to contain the name of the decompressed file. Compressing multiple files in a streaming fashion is not supported: a file spanning multiple non-leaf blocks requires a parcel starting with a formula block, which declares the length of every slice of the non-leaf blocks before any of them.

3.6. Locating an archive

A file may contain an archive with leading bytes, e.g. a self-extracting executable, or with trailing bytes, e.g. appended by a transfer. Since interpreting an archive takes the archive alone, a decompressor accepting such a file has to locate the archive first.

A header magic and a footer magic match when they are HEADER_MAGIC and FOOTER_MAGIC, or OVERLAY_HEADER_MAGIC and OVERLAY_FOOTER_MAGIC.

To locate an archive in a byte sequence file:

  1. If file starts with ENCRYPTED_HEADER_MAGIC or ENCRYPTED_MIRRORED_HEADER_MAGIC, return file.

  2. Let back be null. Let n be the last 2 bytes of file as u16le, and let f be the length of file − n − 10. If f ≥ 0, the bytes of file from f are FOOTER_MAGIC or OVERLAY_FOOTER_MAGIC followed by n as u16le, and the leb128 from f + 8 is s such that h = f − s − 6 ≥ 0 and the bytes of file from h are the header magic matching that footer magic, set back to h.

  3. If file starts with HEADER_MAGIC or OVERLAY_HEADER_MAGIC:

    1. If back is 0, return the result of trimming file.

    2. Read blocks from the offset 6 by the generic structure (§ 3.1 Block), with data of every block taken as a byte sequence without parsing, until the bytes at the current offset e start with FOOTER_MAGIC or OVERLAY_FOOTER_MAGIC. If it fails to read a block before that, or the footer magic doesn’t match the header magic, return failure.

    3. Let m be the 2 bytes of file from e + 6 as u16le. If file is shorter than e + m + 10 bytes, return failure. Otherwise, return the first e + m + 10 bytes of file.

  4. If back is not null, return the result of trimming the bytes of file from back.

  5. Return failure.

To trim a located archive a, a byte sequence starting with a header magic and ending with a footer whose magic matches it:

  1. Let f be the offset of the footer at the end of a. If its metadata_backward_offset is 0, return a.

  2. Let last be the block starting metadata_backward_offset bytes before the footer_start of that footer. If it can’t be read by the generic structure (§ 3.1 Block) or is not a metadata block, return a.

  3. Let blocks be a list containing last, and m be last. While the decoded data of m can be parsed as far as its previous_metadata_offset, and the block starting previous_metadata_offset bytes before m can be read by the generic structure and is a metadata block, set m to that block and prepend it to blocks.

  4. For each block m of blocks, in order:

    1. Let e be the offset after the padding of m (the block_end if it has none).

    2. While the bytes of a from e are a whole spacer block read by the generic structure, set e to the offset after it.

    3. If e is not f and the bytes of a from e start with the footer magic of a followed by a u16le n such that e + n + 10 ≤ f, return the first e + n + 10 bytes of a.

  5. Return a.

A footer directly after a metadata block, other than the footer at the end, is what appending to an archive leaves while it writes: the archive still ends at that footer, as reading the blocks forward in the step 3.2 finds, and the bytes after it are trailing bytes until the appending is committed. Trimming finds it from the end of the file by the metadata blocks alone, not reading every block, as a decompressor interpreting the archive reads every metadata block and the bytes after it anyway; where several follow metadata blocks, the first ends the archive, as reading forward finds. A footer anywhere else before the footer at the end makes the archive found from the end invalid.

3.7. Appending to an archive

This section is non-normative, except where noted.

Since a later segment adds, replaces and removes the entries of earlier ones (§ 4.4.2 Merging segments), an unencrypted archive can be extended by appending segments after its last metadata block, without rewriting what precedes it. The footer has to move to the new end, which is done so that the file holds a valid archive at every point: either the archive before the appending or the archive after it. To commit one or more segments to an archive a in a file, whose footer F starts at the offset p of the file:

  1. If the length of F is odd, write a zero byte after it. Write the blocks of the segments after it, and a footer for the archive ending with them, directly after their last metadata block or after a spacer block following it (§ 3.3 Spacer block). Make sure that all of them are durably written.

  2. Overwrite the first bytes of F with the type and size of a spacer block occupying the bytes of F and its padding byte if any, and make sure that they are durably written.

  3. Overwrite the rest of F with zero bytes, as the space of a spacer block requires.

Until the step 2 completes, the file holds a, followed by trailing bytes that trimming leaves out. Afterwards it holds the extended archive, as F is no longer a footer. If the file stops being written at any point, the next appending finds a or the extended archive and either removes the trailing bytes or completes the step 2 if they end with a whole footer.

The step 2 overwrites 2 bytes if F is 257 bytes or shorter (a spacer block of the type 0), 3 bytes up to 65,538 bytes, and 5 bytes otherwise. It relies on that write being done entirely or not at all, which storage devices guarantee within an aligned sector of 512 bytes or a multiple of it. Therefore a compressor that is to append to an archive later places each footer so that the bytes the step 2 would overwrite, at their offset in the file, don’t cross a multiple of 512 bytes, putting a spacer block of 2 bytes after the last metadata block when they would. Since an archive may be put after leading bytes afterwards, e.g. those of a self-extracting executable, a compressor appending to an archive whose footer crosses that boundary can’t commit as above, and instead removes the footer and writes the segments and a new footer in its place. Such an archive stopped being written before the new footer is whole loses its footer, which can be reconstructed (§ 10.6 Repair).

A compressor appending to an archive MUST NOT leave a footer other than the one at the end, except while it is writing; another compressor appending to such an archive MUST first remove the trailing bytes or complete the commit. Each metadata block whose segment ends a commit SHOULD have segment-status (§ 11.3 segment-status: Commits and the end of an archive).

An encrypted archive can’t be appended to. The last encrypted chunk of its payload is marked as the last one, and continuing the payload would encrypt that chunk again under the same nonce with a different plaintext, which compromises the confidentiality of both.

The located archive is then interpreted, which may still fail. A decompressor SHOULD warn the user when file has bytes outside the located archive. It MAY search for HEADER_MAGIC and OVERLAY_HEADER_MAGIC when the archive can’t be located otherwise, e.g. when file has both leading and trailing bytes.

An encrypted archive is located only at the beginning of a file and extends to its end, because the end of its payload can’t be known without decrypting it (§ 9 Encrypted archives).

A block can’t start with FOOTER_MAGIC or OVERLAY_FOOTER_MAGIC, since they start with the shortest leb128 encodings of FOOTER_TYPE and OVERLAY_FOOTER_TYPE (§ 2.4 Constants), which are not valid block types. The step 3.2 therefore finds the end of the first archive in file, while the step 4 finds the archive at the end of file without reading the bytes before it. Archives concatenated together are not a single archive, and only the first one is located.

4. Segments and entries

4.1. Archive model

An archive value is a struct with the following items:

A file node is a struct with the following items:

A directory node is a struct with the following items:

A link node is a struct with the following items, representing a symbolic link:

A special node is a struct with the following items, representing a FIFO or a device file:

A device number is stored as a pair, not as a platform-specific dev_t encoding, as tar and cpio do and as makedev takes it. Its meaning is up to the platform extracting it: the same pair generally denotes unrelated devices on different platforms, e.g. /dev/null is (1, 3) on Linux but (3, 2) on macOS, and this specification doesn’t record which platform the number came from.

An unknown node is a struct with the following items, representing a kind of node not defined by this version of this specification:

Every file node, directory node, link node, special node and unknown node also has forks, an ordered map from fork names to byte sequences, which represents named data streams other than the content of a file, e.g. resource forks of macOS and alternate data streams of NTFS (§ 11.7 forks: Resource forks and alternate data streams). A fork name is a CBOR data item matching aqr-text (§ 11.2 Comments) that is not empty, and two fork names are equal if and only if they are identical CBOR data items. A node without any fork has an empty map as its forks, as does every node created by an algorithm in this specification unless its forks are given.

The root can’t be a link node, a special node or an unknown node.

A tombstone is a struct with no items, marking the removal of a node. It only occurs in children of the result of interpreting a segment, and is removed by merging it into the preceding result (§ 4.4.2 Merging segments), so it never occurs in an archive value.

An entry name is a struct with the following items:

Two entry names are equal if and only if both items are equal. In particular, a "wtf-8" name and a "raw" name are never equal even when their bytes are.

No bytes of an entry name can contain 0016 or 2F16 (/), or be equal to (2E)16 (.) or (2E 2E)16 (..).

These restrictions make every path in an archive value a plain sequence of components, so that no name can refer to the current, parent or nested directory on any common file system. Platform-specific restrictions are handled at the extraction time (§ 12 Extraction).

An entry name is encoded as a byte sequence in one of the following ways, neither of which contains a zero byte:

Byte C116 was chosen because it can’t occur in WTF-8 or any other known extension to UTF-8.

Conforming compressors are RECOMMENDED to use the "wtf-8" encoding, and SHOULD only use the "raw" encoding as a round-trip measure.

To decode an entry name from a entry_name field bytes:

  1. If bytes is not empty and bytes[0] is C116, let name be an entry name whose encoding is "raw" and bytes are bytes without the first byte.

  2. Otherwise, if bytes is a valid WTF-8 string, let name be an entry name whose encoding is "wtf-8" and bytes are bytes.

  3. Otherwise, return an invalid archive failure.

  4. If name violates any restriction of an entry name above, return an invalid archive failure.

  5. Return name.

A link target is a struct with the following items:

Unlike an entry name, a link target is a path which may contain 2F16 (/) as a separator, and components equal to . or ... It may be absolute, and may refer to a path that doesn’t exist in the archive value or anywhere else. The separator is always /, even for a link target from a platform using \ as the separator, such as Windows, where every \ is represented as /. Otherwise a link target is not normalized, and its meaning, e.g. of a drive letter, is left to the platform. There is no limit on its length other than implementation-defined limits.

/ can’t occur in a file name on Windows, and isn’t interpreted as a separator in a link target there, so this replacement is lossless. A decompressor on Windows replaces / back with \ when creating a link (§ 12 Extraction).

On Windows, the target of an absolute link or a junction is stored as an NT path starting with \??\, which has no meaning on other platforms. A compressor is advised to convert it to a Win32 path first, e.g. \??\C:\foo to C:/foo, \??\UNC\server\share\x to //server/share/x and \??\Volume{GUID}\x to //?/Volume{GUID}/x.

Platforms differ in the maximum length of a link target: 4095 bytes on Linux (less on some file systems, e.g. 1023 bytes on XFS), 1023 bytes on macOS and FreeBSD, and about 8,000 UTF-16 code units on Windows. A link that can’t be created is handled at the extraction time (§ 12 Extraction).

To decode a link target from a link_target field bytes, do the same as to decode an entry name from bytes, except that the result is a link target and restrictions of a link target apply instead.

A link target is never resolved within the archive model. Every path in an archive value is still a plain sequence of components, and a lookup of a path in an archive value never follows a link node, as lstat doesn’t. In particular, no node is a descendant of a link node.

This makes the classic attack of extracting a/passwd through a preceding link a pointing to /etc inexpressible in the archive model. A tool MAY offer a lookup following links, but it then has to limit the number of links followed.

A link group is an opaque value that is only compared for equality, and is either a pair of a segment and a segment-local integer or an archive-wide integer (§ 11.5 link-group: Hard links). Two archive values are equal if and only if they become identical by some one-to-one renaming of the link groups of one of them.

4.2. Content blocks

Any block other than spacer, size hint, metadata, mirror and ignored blocks is a content block which has to be described by a subsequent metadata block, either by its fields or by forks in its optional metadata (§ 11.7 forks: Resource forks and alternate data streams). A metadata block and all content blocks between the previous metadata block (or the beginning of the archive if none) and that metadata block form a segment. As an exception, an archive without a metadata block (§ 3.5 Unencrypted archive) forms a single segment with no metadata block. Each segment MUST be self-contained: every reference from a block in a segment, whether by an index or by an offset, MUST point to a block in the same segment. An internal import (§ 8.1 Import block) refers to a node of preceding segments by its path rather than to a block, and is the only way for a segment to use the content of another segment.

Content blocks are classified as follows:

Dictionary, module and resource blocks are respectively indexed in the order of appearance within the segment. Two conventions are used for these indices:

4.2.1. Block order

Blocks in a segment MUST be ordered so that a formula block is resolved by blocks preceding it and all data required to evaluate it follows it:

These rules make every reference from a formula block to a non-leaf block a forward reference. Therefore, when a non-leaf block is read in a streaming fashion, all formula blocks referencing it have already been read, and the decompressor knows exactly when the block can be discarded. Likewise, the decompressor never has to retain a formula block after all non-leaf blocks it references have been read. An import block whose imports are referenced by many formula blocks follows all of them, so a streaming decompressor retains those formula blocks until the import block is read. A compressor bounds this by spreading imports over multiple import blocks.

A leaf block and all subsequent content blocks up to the next leaf block or the end of the segment form a parcel. When no leaf block is interleaved with non-leaf blocks, a parcel consists of a leaf block, the non-leaf blocks it references if it is a formula block, and possibly auxiliary blocks. In that case a decompressor can read a single file by reading its parcel and the auxiliary blocks it uses, and a streaming decompressor never has to retain a formula block beyond its parcel.

module formula non-leaf non-leaf leaf formula non-leaf metadata leaf metadata parcel parcel parcel parcel segment segment
An archive with two segments, each ending with the metadata block describing it, and four parcels, each starting with a leaf block. Arrows denote references from formula blocks to non-leaf blocks (solid) and to a module block (dashed). The module block precedes the first leaf block in its segment, so it belongs to no parcel. Dictionary, resource and spacer blocks are omitted, and no leaf block is interleaved with non-leaf blocks.

4.2.2. Registered dictionaries and modules

A registered dictionary is an entry of the Aqueous Dictionaries Registry (§ 15.5 Aqueous Dictionaries Registry), and a registered module is an entry of the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry). Each entry is identified by its registered hash, the SHA-512/256 hash [FIPS180-4] of its bytes: the decoded data of a dictionary block for a registered dictionary (§ 5.2 Dictionary block), and wasm_module for a registered module (§ 7.1 Module block).

Block 5016 (a dictionary block) and block 6016 (a module block) are shorthand blocks. The data of a shorthand block is a registered hash alone, 32 bytes long, instead of checksummed Zstandard data, and the shorthand block stands for a block of the same kind whose decoded data is the bytes of that entry:

The two registries share the hash function but not the entries, and a shorthand block is resolved only against the registry of its own kind. Knowing an entry of one registry never makes a decompressor know an entry of the other: in particular, the hash of a registered module in block 5016 doesn’t denote a dictionary of the module’s bytes, unless the same bytes are separately registered as a dictionary, in which case they are two unrelated entries.

A shorthand block has no checksum, and the registered hash is only looked up and never verified. Therefore damage to the hash can’t be distinguished from a hash that was never registered, and both result in an unsupported failure rather than an invalid archive failure.

Registry entries are never modified or removed, so a registered hash keeps denoting the same bytes, and a decompressor may know only some of the entries. Keeping the registries apart lets a decompressor carry a native implementation of a registered module (§ 7.1 Module block) instead of its bytes, which it couldn’t if the hash could also denote a dictionary. A damaged hash is found as unknown with an overwhelming probability, and a mirror block of the shorthand block can recover it.

4.3. Metadata block

Blocks 1016 through 1716 are metadata blocks, whose type is 0001 0mss in binary (§ 3.1.1 Block types). Their data uses the shared external structure (§ 3.2 Shared external structure), with XXH64 as checksum_method and compression_method given by m: None if m is 0, and Zstandard if m is 1.

The internal structure after decompression is as follows:

Name Type
previous_metadata_offset leb128
num_modules leb128
num_resources leb128
num_dictionaries leb128
num_directories leb128
num_files leb128
num_specials leb128
entries_size leb128 = entries_end - entries_start
optional_metadata_size leb128
module_offsets u(previous_metadata_offset)[num_modules]
resource_offsets u(previous_metadata_offset)[num_resources]
dictionary_offsets u(previous_metadata_offset)[num_dictionaries]
entry_parents u(num_directories)[num_directories + num_files + num_specials - 1]
file_params file_param[num_files]
entry_offsets u(entries_size)[num_directories + num_files + num_specials - 1]
entries_start offset
entries entry[num_directories + num_files + num_specials]
entries_end offset
optional_metadata u8[optional_metadata_size]

All relative offsets are measured backward from the beginning of the current metadata block: a relative offset o refers to the block starting o bytes before the current metadata block, implying that the offset of zero is invalid. All valid relative offsets are also even.

Each metadata block describes the content blocks in its segment, i.e. blocks appearing after the previous metadata block, if any, and until the current metadata block. The previous metadata block is linked via previous_metadata_offset, which is also the maximum possible offset that can occur in this particular metadata block. If there is no prior metadata block, previous_metadata_offset is the relative offset to the end of header_magic in the unencrypted archive structure (§ 3.5 Unencrypted archive), plus 1 so that it never points to a valid block.

num_modules, num_resources and num_dictionaries are the numbers of module, resource and dictionary blocks in the segment respectively. Relative offsets to those blocks are later recorded as module_offsets, resource_offsets and dictionary_offsets respectively, in the order of appearance.

The metadata block represents zero or more directories, zero or more files and zero or more special entries, collectively called entries and numbered together. The root directory is numbered 0, subsequent directories are numbered from 1 to num_directories, individual files are numbered from (num_directories + 1) to (num_directories + num_files), and special entries are numbered from (num_directories + num_files + 1) to (num_directories + num_files + num_specials). There MUST be at least one entry other than the root directory in the first metadata block of an archive, i.e. num_directories + num_files + num_specials ≥ 1. A later metadata block MAY have none, e.g. to add only the journal (§ 11.4 journal: Points to return to) of a state to the archive; its segment then has no content blocks, and entry_parents, entry_offsets and entries are empty. entry_parents[k - 2] is the parent of the entry k for each k ≥ 2, and MUST be a directory less than k, i.e. in [0, min(k - 1, num_directories)], so that directories can’t form a cycle. The root directory (entry 0) has no parent and the parent of the entry 1 is always the root directory, so they are omitted.

File parameters (file_param) occur for each file (but not for directories or special entries), and consist of the following:

Name Type Conditions
file_dictionary u(num_dictionaries)
file_offset u(previous_metadata_offset)
file_output leb128 If file_offset is odd

file_dictionary is the dictionary used by the block that file_offset refers to as a 1-based index, or zero if no dictionary is used. Therefore file_dictionary = k refers to the block at dictionary_offsets[k - 1], and num_dictionaries is the maximum possible value for it. A relative offset to the corresponding leaf block, which starts the file’s parcel, is recorded as file_offset, plus 1 if the file is another output of the block than the output 0. The relative offset can’t be zero. Since every relative offset is even, file_offset % 2 tells which: the file output index is 0 if file_offset is even, and file_output + 1 if it is odd. Multiple files MAY refer to the same leaf block and output, in which case they have the same content.

This is how an archive stores files with the same content only once, including files that were hard links to each other (§ 11.5 link-group: Hard links). It costs only a file_param and an entry per file.

Entry structure (entry) then occurs for each directory, file and special entry, in this order, and consists of the following:

Name Type Conditions
entry_name u8[*] ≠ 0
entry_name_sentinel u8 = 0
original_size leb128 If this entry is for a file
special_kind leb128 If this entry is for a special entry
link_target u8[*] ≠ 0 If this entry is for a special entry and special_kind = 1
link_target_sentinel u8 = 0 If this entry is for a special entry and special_kind = 1
device_major leb128 If this entry is for a special entry and special_kind is 3 or 4
device_minor leb128 If this entry is for a special entry and special_kind is 3 or 4
special_data u8[*] If this entry is for a special entry and special_kind > 4

Each entry is named according to entry_name, which encodes an entry name as described in § 4.1 Archive model and is decoded there. Entry names from ZIP are discussed in § 16.1 Entry names from ZIP.

A special entry represents a node other than a file or a directory, as determined by special_kind:

special_kind Node
0 Tombstone, which removes the node at the same path in preceding segments or in the base (§ 4.4.2 Merging segments)
1 Link node, with link_target as its target
2 Special node of the kind "fifo"
3 Special node of the kind "char-device", with device_major and device_minor as its device number
4 Special node of the kind "block-device", with device_major and device_minor as its device number
5 or greater Unknown node, reserved for future versions of this specification

link_target uses the same encodings as entry_name. special_data is not interpreted by this version of this specification; its length is determined by the beginning of the next entry in entry_offsets, or entries_end for the last entry.

Special entries are numbered separately from files, as directories are, so that file_params remain directly indexed by entry numbers. A link target is stored in the entry rather than in the optional metadata, because it defines the node itself and can’t be ignored. Since the metadata block is compressed as a whole, similar targets compress well.

Unknown kinds are not an invalid archive failure, so that a decompressor can still extract the rest of an archive using a kind defined in the future. Every special entry can be skipped without knowing its kind, thanks to entry_offsets. An unknown node replaces an earlier node at the same path like any other non-directory node (§ 4.4.2 Merging segments), so a future kind can’t change how segments are merged. This is why tombstones are defined by this version, as the first kind: a decompressor treating a tombstone as an unknown node would keep it in place of the node it was meant to remove.

The entries array is the only part that has the unspecified number of bytes, which is therefore available as entries_size beforehand. Any other field can be randomly accessed after reading the initial set of leb128 fields.

As an exception to the relative offsets above, entry_offsets are measured from entries_start. The entry 1 always starts at entries_start, so its offset is omitted, and entry_offsets[k - 2] is the offset to the beginning of the entry k for each k ≥ 2. They MUST agree with the actual positions of entries.

optional_metadata is described in Optional metadata (§ 11 Optional metadata).

4.4. Interpreting an archive

To interpret an archive given a byte sequence input, an optional passphrase passphrase and an optional base base, which is an archive value:

  1. If input starts with ENCRYPTED_HEADER_MAGIC or ENCRYPTED_MIRRORED_HEADER_MAGIC:

    1. Parse input as the encrypted archive structure (§ 9 Encrypted archives) in its entirety. If it fails to parse, return an invalid archive failure.

    2. If header_checksum is present and doesn’t match, return an invalid archive failure.

    3. If lg_work_factor exceeds an implementation-defined limit, return a resource failure.

    4. If passphrase is not given, or the file key can’t be unwrapped from encrypted_file_key with passphrase as described in [AGE], return a decryption failure.

    5. Using the unwrapped file key, verify age_header_mac against the reconstructed age header and decrypt the age payload as described in [AGE]. If either fails, return an invalid archive failure.

    6. Set input to the decrypted payload. It MUST satisfy the additional constraints in § 9 Encrypted archives, or an invalid archive failure is returned.

  2. If input parses as the unencrypted archive structure (§ 3.5 Unencrypted archive) in its entirety, with data of every block taken as a byte sequence without parsing, and either the type of any of blocks is critical and not defined by this specification, or any of blocks is a shorthand block of 32 bytes of data whose registered hash the decompressor doesn’t know (§ 4.2.2 Registered dictionaries and modules), return an unsupported failure.

  3. Parse input as the unencrypted archive structure (§ 3.5 Unencrypted archive) in its entirety. If it fails to parse, return an invalid archive failure.

  4. If input is an overlay archive and base is not given, return a base failure.

  5. If input is a standalone archive, set base to an archive value whose root is a directory node with an empty map as children and null as extra, and whose extra is an empty map.

  6. If blocks contain no metadata block:

    1. Let content be the concatenation of the file contents of all leaf blocks in blocks, in the order of appearance, each for every output index from 0 in order (all outputs of a formula block, and the only one of a data block).

    2. Return an archive value whose root is a file node with content as content, null as link group and null as extra, and whose extra is an empty map.

  7. If the root of base is a directory node, let root be a copy of it. Otherwise let root be a directory node with an empty map as children and null as extra.

  8. Let extra be a copy of the extra of base.

  9. For each segment segment in blocks, in the order of appearance:

    1. Let segment root be the result of interpreting segment.

    2. Merge segment root into root.

    3. For each key k and value v in the top-level map of the segment’s optional metadata other than entry-extras, segment-status (§ 11.3 segment-status: Commits and the end of an archive) and journal (§ 11.4 journal: Points to return to), set extra[k] to v.

  10. Return an archive value with root as root and extra as extra.

The archive base is base after the step 5, and the preceding result of a segment is root just before the segment is interpreted in the step 9, or the root of the archive base for an archive without a metadata block. They are what external imports and internal imports respectively refer to (§ 8.2 Resolving an import). A given base doesn’t affect the interpretation of a standalone archive.

The preceding result of the first segment of an overlay archive is its base, so an internal import can also read a file of the base, as long as no preceding segment has replaced or removed it. An external import reads the base as given, even after a preceding segment has replaced the file. An overlay archive without a metadata block replaces its base with a single file, e.g. to patch a single file given as the base.

The input to this algorithm is exactly the archive, from the first byte of header_magic to the last byte of footer_size_copy.

A file may contain an archive with leading or trailing bytes, which a decompressor removes by locating the archive (§ 3.6 Locating an archive).

Since every content block has to be described by a subsequent metadata block (§ 4.2 Content blocks), an archive with a metadata block can’t have any content block after the last metadata block. previous_metadata_offset and metadata_backward_offset MUST point to the metadata blocks found by parsing blocks as described in § 4.3 Metadata block and § 3.5 Unencrypted archive, so that a decompressor can find each segment either by scanning blocks forward or by following offsets backward.

4.4.1. Interpreting a segment

To interpret a segment segment whose metadata block is metadata block:

  1. Let meta be the decoded data of metadata block, parsed as its internal structure (§ 4.3 Metadata block).

  2. Let d be num_directories, f be num_directories + num_files and n be num_directories + num_files + num_specials of meta.

  3. Let nodes be a list containing a single directory node with an empty map as children, and set the extra and forks of it from the entry extra for the entry 0.

  4. For each k in [1, n]:

    1. Let entry be meta.entries[k - 1].

    2. Let name be the result of decoding entry.entry_name.

    3. Let parent be 0 if k = 1, or meta.entry_parents[k - 2] otherwise.

    4. If k ≤ d, let node be a directory node with an empty map as children.

    5. Otherwise, if k ≤ f:

      1. Let param be meta.file_params[k - d - 1].

      2. Let block be the block at the relative offset param.file_offset - param.file_offset % 2 from metadata block. It MUST be a leaf block in segment.

      3. If param.file_dictionary is zero, block MUST NOT use a dictionary. Otherwise, block MUST use the dictionary block at meta.dictionary_offsets[param.file_dictionary - 1].

      4. Let content be the file content of block for the file output index of param. Its length MUST be entry.original_size.

      5. Let node be a file node with content as content and null as link group.

      6. If the entry extra for the entry k has link-group with a value g, set node.link group to a pair of segment and g if g ≥ 0, or to g otherwise.

    6. Otherwise, if entry.special_kind is 0, let node be a tombstone. The entry extra for the entry k MUST be null.

    7. Otherwise, if entry.special_kind is 1:

      1. Let target be the result of decoding entry.link_target.

      2. Let node be a link node with target as target.

    8. Otherwise, if entry.special_kind is 2, let node be a special node with "fifo" as kind and null as device.

    9. Otherwise, if entry.special_kind is 3 or 4, let node be a special node with "char-device" or "block-device" respectively as kind and (entry.device_major, entry.device_minor) as device.

    10. Otherwise, let node be an unknown node with entry.special_kind as kind and entry.special_data as data.

    11. If node is not a tombstone, set the extra and forks of node from the entry extra for the entry k.

    12. Let children be the children of nodes[parent]. children[name] MUST NOT exist.

    13. Set children[name] to node.

    14. Append node to nodes.

  5. Return nodes[0].

Since every parent is less than k and at most d by § 4.3 Metadata block, nodes[parent] is always a directory node already in nodes.

The entry extra for the entry k is the element at the 0-based index k of entry-extras in the segment’s optional metadata, where the index 0 corresponds to the root directory, or null if entry-extras or such element doesn’t exist. entry-extras MUST NOT have more elements than the entries including the root directory, i.e. num_directories + num_files + num_specials + 1.

To set the extra and forks of a node node from an entry extra e:

  1. If e is null, set the extra of node to null and its forks to an empty map, and return.

  2. Set the extra of node to e without its forks member, if any.

  3. Set the forks of node to an empty map.

  4. For each name → [offset, output, size] of forks in e, if any:

    1. Let block be the block at the relative offset offset from the metadata block of the segment. It MUST be a leaf block in the segment.

    2. Let content be the file content of block for the output index output. Its length MUST be size.

    3. Set node.forks[name] to content.

forks refers to blocks by relative offsets, which have no meaning outside the segment, so it is resolved here and never remains in the extra of a node. Unlike file_params, forks has no dictionary field: a leaf data block names its dictionary by dictionary_index anyway, and file_dictionary only lets a decompressor fetch the dictionary before reading the block.

4.4.2. Merging segments

To merge directory nodes later into earlier:

  1. For each name → node of later.children:

    1. Let existing be earlier.children[name], or null if it doesn’t exist.

    2. If node is a tombstone, remove earlier.children[name] if it exists.

    3. Otherwise, if node is a directory node:

      1. If existing is not a directory node, set existing to a directory node with an empty map as children and null as extra, and set earlier.children[name] to existing.

      2. Merge node into existing.

    4. Otherwise, set earlier.children[name] to node.

  2. If later.extra is not null, set earlier.extra to it and earlier.forks to later.forks.

As a result, a directory appearing in multiple segments has the union of their children, while any other node, or a directory replaced by a node of the different kind, is replaced as a whole with its descendants. In particular, a directory replaced by a link node loses all its descendants, so no node is a descendant of a link node in the result. A tombstone removes the node at its path along with its descendants, and does nothing if there is no such node. A tombstone is never kept in the result, even under a directory that didn’t exist before. A directory with a null extra in a later segment keeps the existing extra and forks, which can instead be replaced with an empty map {}.

Directories necessarily reappear in later segments to hold new files under them, which is why a directory is merged with an existing one rather than replacing it. For the same reason, a reappearing directory doesn’t have to repeat its entry extra.
A directory whose children change entirely, e.g. from A to B, keeps its node and gets tombstones for the names in A but not in B, while the names in B replace or add nodes as usual. A tombstone on a directory states that the directory itself is absent from the result, and a single tombstone removes a whole subtree. So replacing the entire base of an overlay archive needs a tombstone for each name only in the base at the topmost level where the base and the new tree differ.

5. Stored data

5.1. Data block

Blocks 2016 through 3716 and 4016 through 4B16 are data blocks. A data block is a compressed block if its compression_method is not None, and an uncompressed block otherwise.

The type of a data block is 001kk css in binary (§ 3.1.1 Block types) for a leaf block, and 0100 kkss for a non-leaf block, where:

Its data consists of the following:

Name Type Conditions
dictionary_index leb128 If has_dictionary
compressed_data compressed_data If compression_method is not None
uncompressed_data uncompressed_data If compression_method is None

If a dictionary is in use, its 0-based index is specified as dictionary_index. For a leaf data block, the metadata block also gives the dictionary as file_dictionary, which is available before reading the block; for a non-leaf data block, dictionary_index is the only reference to the dictionary. compressed_data and uncompressed_data use the shared external structure (§ 3.2 Shared external structure) with compression_method and checksum_method determined as above. uncompressed_data represents a raw, uninterpreted byte sequence.

A compressed block is a single unit of decompression: reading any part of its content requires decompressing it from the beginning. Content larger than a desired unit of random access is split into multiple blocks:

5.2. Dictionary block

Blocks 5016 through 5716 are dictionary blocks, whose type is 0101 0tss in binary (§ 3.1.1 Block types). Their data is checksummed Zstandard data (§ 3.2 Shared external structure), except that block 5016 is a shorthand block for a registered dictionary (§ 4.2.2 Registered dictionaries and modules).

t determines whether the dictionary has entropy tables (has_entropy_tables): false if 0, and true if 1.

The internal structure after decompression is as follows:

Name Type Conditions
entropy_tables u8[*] If has_entropy_tables
repeat_offsets u32le[3] If has_entropy_tables
content u8[*]

The decoded data of a dictionary block is a dictionary for blocks using it, and corresponds to a Zstandard dictionary ([RFC8878], Section 5) without Magic_Number and Dictionary_ID:

In both cases, content is the Content of the dictionary, which is the only part of the dictionary that sequences can reference as preceding data. content MUST be at least 8 bytes long.

Magic_Number and Dictionary_ID are omitted because they are only meaningful for Zstandard frames and the framing is done by this format. An implementation using libzstd can prepend them (with an arbitrary nonzero Dictionary_ID) to load a dictionary with entropy tables. In that case it has to make sure that no sequence references the prepended bytes or entropy_tables, which some decompression paths of libzstd don’t check.
Entropy tables can be trained independently of the content (e.g. ZDICT_finalizeDictionary) and mainly benefit small blocks: in measurements with per-file blocks of 1–20 KB files, they reduced the size by 0.4–26%, while the benefit vanished for blocks of 64 KB or more. Fast strategies of libzstd (e.g. levels 1–3 of 1.5.7) reuse the dictionary’s entropy tables for every block even when the block’s own tables would be smaller. When the repeat offsets of the dictionary are the initial values, a compressor can avoid this by also compressing a block with the same content as a raw content dictionary and choosing the smaller result, since a block compressed that way never refers to the entropy tables of the dictionary.

5.3. Block contents

The decoded data of a block using the shared external structure (§ 3.2 Shared external structure), given its compression_method and checksum_method, is determined as follows:

The stored content of a data block, whether a leaf block or not, is the decoded data of its compressed_data or uncompressed_data. The stored content of a non-leaf item is the stored content of the non-leaf data block supplying it, or the imported content of the import supplying it (§ 8.2 Resolving an import). A data block using a dictionary uses the dictionary block at dictionary_index within the segment.

The file content of a leaf block for an output index i is:

6. Formulas

6.1. Formula block

Blocks 7016 through 7716 are formula blocks, whose type is 0111 0dss in binary (§ 3.1.1 Block types).

d determines whether a dictionary is used (has_dictionary): false if 0, and true if 1.

Its data consists of the following:

Name Type Conditions
dictionary_index leb128 If has_dictionary
num_outputs leb128
formula_data compressed_data

If a dictionary is in use, its 0-based index is specified as dictionary_index, as in data blocks.

num_outputs is the number of outputs of the formula block, which MUST be at least 1 (§ 6.3 Formula evaluation).

Every instruction pops and pushes a number of items that its operands give, so the number of outputs follows from the instructions alone, without applying any module. num_outputs tells it before formula_data is decompressed, e.g. to a decompressor checking file_output or streaming the outputs to the files using them.
formula_data is checksummed Zstandard data (§ 3.2 Shared external structure).
Formula blocks always use Zstandard, which can still store data verbatim with raw Zstandard blocks. Many small formula blocks are expected to share a dictionary to compensate for the lack of shared compression context.

A formula block describes the content of a file as an expression over non-leaf items following it, evaluated with modules and resources preceding it. The internal structure of formula_data is described in Formula data (§ 6.2 Formula data).

6.2. Formula data

This section describes the internal structure of formula_data in formula blocks (§ 6.1 Formula block), after it has been decompressed.

A formula consists of the lists of blocks it references, followed by instructions of a stack machine. Each instruction pops zero or more byte sequences from the stack and pushes zero or more byte sequences, and the byte sequences remaining on the stack after the last instruction are the outputs of the formula.

Name Type
num_blocks leb128
block_ordinal_deltas leb128[num_blocks]
num_resources leb128
resource_indices leb128[num_resources]
num_modules leb128
module_indices leb128[num_modules]
instructions instruction[*]

block_ordinal_deltas lists the non-leaf items referenced by the formula block. Despite their names, num_blocks, block_ordinal_deltas and slice_block count non-leaf items, which are blocks except for imports. Their 0-based ordinals among the non-leaf items supplied by the non-leaf blocks following the formula block within the segment, in the order of appearance, are o0 = block_ordinal_deltas[0] and oi = oi - 1 + block_ordinal_deltas[i] + 1 for i ≥ 1, so the ordinals are strictly increasing. The i-th referenced item is the non-leaf item with the ordinal oi, which MUST exist.

resource_indices and module_indices list the resource and module blocks referenced by the formula block, as 0-based indices within the segment (§ 4.2 Content blocks). Each of them MUST refer to an existing block, which precedes the formula block by § 4.2.1 Block order.

instructions fill the rest of formula_data. Each item or block listed in the above lists MUST be used by at least one instruction, and is said to be referenced by the formula block only in that case.

Non-leaf items are referenced by ordinals rather than by offsets, so that relocating or recompressing a block, or inserting other blocks such as mirror blocks, doesn’t invalidate formula blocks referencing blocks after it. An offset would also make the result depend on how the block is located: an offset pointing into the middle of a block can still find a valid-looking block there, e.g. in an archive stored inside an uncompressed block, while a decompressor reading blocks in order would find no block at that offset. Ordinals in a typical parcel are consecutive, so block_ordinal_deltas are mostly zero. Resources and modules are referenced by absolute indices, which recur in many formula blocks and compress well with a shared dictionary.

Each instruction consists of an opcode of type u8 followed by its operands, whose structure is determined by the opcode as described in the following sections. opcode MUST be one of the following:

Opcode Instruction Pops Pushes
0016 Nop 0 0
0116 Literal 0 1
0216 ModuleBytes 0 1
0316 ResourceBytes 0 1
0F16 Swap 0 0
1016 Slice 0 1
1216 Concatenate concat_count 1
1316 Pack pack_count 1
1416 Split 1 split_count
1516 Patch 2 1
1616 Apply apply_num_inputs apply_num_outputs
2016 Checksum 0 0

Opcodes are grouped by their roles: 0016 through 0F16 push bytes as they are, in the formula itself or in the blocks it lists, or rearrange the stack, 1016 through 1F16 compute byte sequences from referenced items or the stack, and 2016 through 2F16 check them. Other opcodes are reserved. Opcode 1116 is reserved as well: it was an earlier Apply instruction, which gave no task limits, and a decoder MUST NOT give it another meaning. A formula with the opcode 1116 is an invalid archive failure.

A reference to a preceding block in the same segment, e.g. to patch a file stored earlier in the segment, is not yet defined. A reference to another file is made by an import (§ 8.1 Import block) rather than by an instruction, and the content of a file in preceding segments by an internal import. The candidate is a second list of non-leaf items preceding the formula block, read by Slice instructions as referenced items are, where the instructions of another formula block are copied when its output is needed, instead of referring to the output. It requires a bound on the distance of such a reference declared before the referenced blocks, so that a streaming decompressor knows which blocks to retain, and a way to locate a preceding block without reading the segment from its beginning.

formula_data is not randomly accessible, and a decompressor is expected to read and decompress it in its entirety before evaluating any part of the formula. Every requirement on formula_data that doesn’t depend on the evaluated byte sequences, such as indices into the lists and the number of byte sequences on the stack, can be checked in a single pass over instructions without evaluating anything. The same pass determines the length of every byte sequence on the stack, so formula_data then serves as an index for random access within the output, in the same way as a seek table of other formats. The referenced items are located by skipping blocks following the formula block by their size fields, and an item of an import block by decoding that block.

The index grows linearly with the number of non-leaf items referenced, but it compresses well because most instructions share the same shape and sizes. The decompression of the whole formula_data is a one-time cost per file.

To execute an instruction on a stack, which is a list of byte sequences whose last item is the top, perform the steps described for its opcode in the following sections. To pop n items, the stack MUST have at least n items, and the last n items are removed from the stack and used in their order in the stack.

6.2.1. Nop instruction

Opcode 0016 denotes a Nop instruction, which has no operands and does nothing.

6.2.2. Literal instruction

Opcode 0116 denotes a Literal instruction. Its operands consist of the following:

Name Type
literal_size leb128
literal_data u8[literal_size]

It pushes literal_data.

6.2.3. ModuleBytes instruction

Opcode 0216 denotes a ModuleBytes instruction. Its operands consist of the following:

Name Type
module_bytes_module leb128
module_bytes_size leb128
  1. module_bytes_module MUST be less than num_modules.

  2. Let b be the module block at module_indices[module_bytes_module]. b MUST NOT be a shorthand block.

  3. The length of data of b MUST be module_bytes_size.

  4. Push data of b.

6.2.4. ResourceBytes instruction

Opcode 0316 denotes a ResourceBytes instruction. Its operands consist of the following:

Name Type
resource_bytes_resource leb128
resource_bytes_size leb128
  1. resource_bytes_resource MUST be less than num_resources.

  2. Let b be the resource block at resource_indices[resource_bytes_resource].

  3. The length of data of b MUST be resource_bytes_size.

  4. Push data of b.

ModuleBytes and ResourceBytes instructions push the bytes of a block as stored, i.e. still compressed and with its checksum, rather than its decoded data. They let a file holding the same blocks, typically another Aqueous archive, share them with the archive holding the file: a formula rebuilds such a block from its type and size, given by a Literal instruction, followed by its data, without compressing the module or the resource again. A compressor can take the stored bytes of the inner block as they are for the outer block, as any compressed data decoding to the same bytes is valid. A shorthand block can’t be referenced by a ModuleBytes instruction, since its data is a registered hash while the block it stands for is never stored.

The instructions neither decode nor apply the block: a module referenced only by ModuleBytes instructions isn’t applied, so a conformance level doesn’t allow refusing it, and the checksum of the block covers its decoded data, which the instructions don’t compute (§ 13.2 Partial access). The pushed bytes are covered instead by a Checksum instruction covering a byte sequence computed from them, typically an output checksum. The size is given as an operand, as the sizes of Slice and Patch instructions are, so that the length of every byte sequence on the stack is known from the formula alone.

6.2.5. Swap instruction

Opcode 0F16 denotes a Swap instruction. Its operands consist of the following:

Name Type
swap_depth leb128

The stack MUST have more than swap_depth items. It exchanges the top of the stack with the item swap_depth items below it.

6.2.6. Slice instruction

Opcode 1016 denotes a Slice instruction. Its operands consist of the following:

Name Type
slice_block leb128
slice_start leb128
slice_length leb128
  1. slice_block MUST be less than num_blocks.

  2. Let c be the stored content of the slice_block-th referenced item.

  3. slice_start + slice_length MUST be at most the length of c.

  4. Push the bytes of c in [slice_start, slice_start + slice_length).

6.2.7. Apply instruction

Opcode 1616 denotes an Apply instruction. Its operands consist of the following:

Name Type
apply_module leb128
apply_num_resources leb128
apply_resources leb128[apply_num_resources]
apply_params_size leb128
apply_params u8[apply_params_size]
apply_memory_limit leb128
apply_num_task_limits leb128
apply_task_limits leb128[apply_num_task_limits]
apply_num_inputs leb128
apply_num_outputs leb128
apply_output_sizes leb128[apply_num_outputs]
  1. apply_module MUST be less than num_modules, and each item of apply_resources MUST be less than num_resources.

  2. apply_num_outputs MUST be at least 1. apply_num_task_limits MUST be the number of classes of the module.

  3. Pop apply_num_inputs items as inputs.

  4. Let outputs be the result of applying the module block at module_indices[apply_module] with inputs, the resource blocks at resource_indices[r] for each r of apply_resources, apply_params, apply_output_sizes, apply_memory_limit and apply_task_limits as its task limits (§ 7.4 Module execution).

  5. Push each item of outputs in order.

6.2.8. Concatenate instruction

Opcode 1216 denotes a Concatenate instruction. Its operands consist of the following:

Name Type
concat_count leb128

It pops concat_count items and pushes their concatenation in order.

6.2.9. Pack instruction

Opcode 1316 denotes a Pack instruction. Its operands consist of the following:

Name Type
pack_count leb128

It pops pack_count items and pushes the packed byte sequence of them in order (§ 6.3.1 Packed byte sequences).

6.2.10. Split instruction

Opcode 1416 denotes a Split instruction. Its operands consist of the following:

Name Type
split_count leb128
split_sizes leb128[split_count]
  1. split_count MUST be at least 1.

  2. Pop an item x. The sum of split_sizes MUST be equal to the length of x.

  3. For each n of split_sizes in order, push the next n bytes of x.

6.2.11. Patch instruction

Opcode 1516 denotes a Patch instruction. Its operands consist of the following:

Name Type
patch_size leb128
  1. Pop 2 items as source and patch.

  2. patch MUST consist of one or more Zstandard blocks without any other bytes, where only the last block is signalled as the final block by its block header, and each block holds at most 128 KiB of data.

  3. Let output be the result of decompressing patch as a sequence of Zstandard blocks [RFC8878] with source as the Content of a raw content dictionary, i.e. the entropy tables start empty and the repeat offsets start at their initial values (1, 4, 8). The offset of every match MUST be at most the length of source plus the number of bytes already decompressed.

  4. The length of output MUST be patch_size.

  5. Push output.

A Patch instruction decompresses Zstandard blocks with an earlier byte sequence as a raw content dictionary, as zstd --patch-from does, so a decompressor reuses its Zstandard decompressor for it. The earlier byte sequence is an operand on the stack rather than a dictionary block, so that it can be any byte sequence computed by the formula, e.g. an import or a stream split from it by a module. The patch itself is typically a slice of an uncompressed non-leaf data block, whose checksum covers the stored patch, while an output checksum covers the result.
A Patch instruction has no Zstandard frame and therefore no Window_Size: a match may refer to any byte of source or of the output so far, and the guaranteed limit of Window_Size in § 3.2.1 Zstandard compression method doesn’t apply. The decompressor holds both byte sequences anyway, so a long reach costs no additional memory. With libzstd, this is decompressing a frame after ZSTD_DCtx_refPrefix with source, whose release builds only check offsets against the start of the prefix; a decompressor enforcing Window_Size has to lift that check for the frame. A compressor using libzstd sets ZSTD_c_windowLog large enough to cover source and the output, as zstd --patch-from does.
With an empty source, a Patch instruction decompresses the blocks of a Zstandard frame without a dictionary, which start from the same state. A compressor can then keep a Zstandard file as it is and take the files its content holds (the whole, or the members of a tar archive) from a Split instruction of that output, rather than storing them again. A piece of the Split that no file takes is dropped as the source of another Patch instruction whose patch is an empty raw block, the final one (01 00 00), of an empty output.

6.2.12. Checksum instruction

Opcode 2016 denotes a Checksum instruction. Its operands consist of the following:

Name Type
checksum_value u64le

The stack MUST NOT be empty. The XXH64 checksum of the top of the stack MUST be equal to checksum_value.

A Checksum instruction covers the byte sequence at the top of the stack when it is executed, and the byte sequence remains the same item when moved by Swap instructions. An output checksum is a Checksum instruction covering an output of the formula block. An output is checksummed if an output checksum covers it.

An output checksum covers a single output, not the concatenation of the outputs. A formula block of several outputs, which may be the contents of several files (§ 4.3 Metadata block), therefore has an output checksum for each output that is to be checksummed, each executed with Swap instructions bringing that output to the top of the stack and back, so that each file can be verified by itself.

6.3. Formula evaluation

To evaluate a formula block block:

  1. Let data be the decoded data of formula_data of block, parsed as in § 6.2 Formula data.

  2. Let stack be an empty list of byte sequences.

  3. For each insn of instructions of data in order, execute insn on stack as described for its opcode in § 6.2 Formula data.

  4. The number of items of stack MUST be num_outputs of block.

  5. Return stack as the outputs of block, numbered from 0 in order from the bottom of the stack.

Every byte sequence on the stack is used exactly once, as an input of another instruction or as an output, so a formula is a tree of instructions rooted at its outputs, except that an instruction pushing multiple items, an Apply or a Split instruction, is shared by them. Byte sequences are defined as fully evaluated values. Lazy, streaming or partial evaluation is an implementation strategy and doesn’t change the result (§ 13.2 Partial access). A Checksum instruction covering an intermediate byte sequence, such as the output of an Apply instruction, can be verified by a decompressor that has computed that byte sequence but not the entire output.

6.3.1. Packed byte sequences

The packed byte sequence of a list of byte sequences items is the concatenation of the following, where n is the number of items:

A byte sequence p is a packed byte sequence of exactly one list items, if any. To unpack p:

  1. If p is shorter than 8 bytes, return failure.

  2. Let n be the first 8 bytes of p as u64le, and let h be 8 × (n + 1). If h exceeds the length of p, return failure.

  3. Let ei be the 8 bytes of p from the offset 8 × (i + 1) as u64le for each i in [0, n), and let e-1 be 0.

  4. If ei - 1 > ei for any i in [0, n), return failure.

  5. If h + en - 1 is not the length of p, return failure.

  6. Return the list of the bytes of p in [h + ei - 1, h + ei) for each i in [0, n).

Packed byte sequences let a module take a list of byte sequences whose number varies between applications, e.g. one per section of an executable, as a single input, since the number of inputs of a module is fixed by its entry point (§ 7.4 Module execution). The boundaries are not stored: a Pack instruction derives them from the lengths of its items, which are known without evaluating anything (§ 6.2 Formula data). A module still has to unpack its input and fail on a failure, since the same bytes can be given in any other way, e.g. by a Literal instruction followed by a Concatenate instruction. The fixed-width boundaries let a module find any item without parsing the others, and precede the items so that a part module can request them first.

In the other direction, a module producing a list of byte sequences outputs their plain concatenation, which a Split instruction separates by the sizes declared in the formula. The output of such a module is passed to another module taking a packed byte sequence by a Split instruction followed by a Pack instruction.

7. Modules

7.1. Module block

Blocks 6016 through 6316 are module blocks, whose type is 0110 00ss in binary (§ 3.1.1 Block types). Their data is checksummed Zstandard data (§ 3.2 Shared external structure), except that block 6016 is a shorthand block for a registered module (§ 4.2.2 Registered dictionaries and modules).

The internal structure after decompression gives a WebAssembly module in the binary format [WASM-CORE-3], wasm_module, in one of two forms told by its first byte:

wasm_module MUST decode and validate as a module of the features allowed in § 7.4 Module execution, and MUST conform to the profile and the interface given there; otherwise an invalid archive failure occurs. In short, the module defines no memory of its own and imports every memory and host function from the module name aqr, has no start function, and exports a single entry point that the host invokes once per application. Custom sections are allowed and don’t affect the result. The stored form adds nothing around the module, so that the decoded data can be inspected and run by any WebAssembly tool. A module block for a registered module has the same wasm_module as the entry, whatever its form, and a shorthand block stands for one of the stored form.

The module hash of a module block is the SHA-512/256 hash [FIPS180-4] of its wasm_module, which is also the registered hash of a registered module (§ 4.2.2 Registered dictionaries and modules). Since every application of a module depends only on wasm_module and its arguments (§ 7.4 Module execution), a decompressor MAY apply a module whose module hash it knows with an implementation of its own (e.g. native code) instead of instantiating wasm_module, as long as the result is indistinguishable from the definition, including invalid archive failures. That includes the memory limits an application declares (§ 7.4 Module execution): an implementation of its own MUST fail an application with a resource failure where the module’s memories would exceed them, which it can tell from the application’s memory requirements before it runs (or else by applying the module with an engine, where the limits are enforced as it runs). Such a decompressor needs neither the bytes of the module nor a WebAssembly engine, whether the module is given by a shorthand block or in full. A decompressor MAY refuse to apply a module as its conformance level allows (§ 13.1 Conformance levels), in which case a resource failure occurs.

A compressor can use a shorthand block for a registered module to save the size of the module, at the cost of an unsupported failure in a decompressor that doesn’t know the entry. A module given in full can be read by any decompressor with a WebAssembly engine.
A decompressor chooses how to apply each module, and none of the choices requires translating modules to native code. For a module whose module hash it knows, such as a registered module at level 1, it can either:
  • apply an implementation of its own, which is the fastest but has to be made for every module known. Translating the module by automated means (e.g. wasm2c) is preferred to writing it anew, since the result has to be indistinguishable from the module, including its failures; or

  • instantiate the module with a WebAssembly engine without isolating it, since the hash is strong enough that no other module can have it. This needs only the hash for a module given in full, and the bytes only for a shorthand block.

For a module whose module hash it doesn’t know, it can either:

For example, a decompressor at level 1 needs neither an engine nor a sandbox if it implements every registered module natively, and one at level 2 needs both but no native implementation.

7.2. Split modules

The split form of a module block keeps wasm_module as 14 streams, in which the instructions of its code are split by field, and its data segments' bytes and its custom sections are apart from the rest. Every byte of wasm_module comes back from the streams as it was, LEB128s of any length included, but for the sizes of the sections and the functions split and for the local indices, which are written in the shortest form. Its structure is as follows:

Name Type
count u8
lengths leb128[count]
payload u8[*]

count MUST be 13 (0D16), one less than the number of the streams. The stream si for each i in [0, 13) is the lengths[i] bytes of payload after those of s0 through si - 1, and the stream s13 is the rest of payload. If the sum of lengths exceeds the length of payload, an invalid archive failure occurs. The streams are named as follows:

i Name Holds
0 skeleton the header; the sections' ids, sizes and counts; how each function is kept
1 sections the sections kept as they are but custom sections, and the functions kept as they are
2 custom the custom sections' contents
3 locals the functions' declarations of locals
4 op opcodes, memory arguments' flags and the first bytes of their offsets, the immediates of i32.const and call
5 local the codes of local indices
6 label the labels of br and br_if
7 brtable the labels of br_table
8 blocktype block types
9 offset memory arguments' offsets but their first bytes
10 i64 the immediates of i64.const
11 bytes float and vector constants, shuffle lanes, lane indices
12 misc the other immediates, and the data segments but their bytes
13 data the data segments' bytes

wasm_module is joined from the streams by reading each from its start, each read taking its next bytes, and appending to the output, initially empty, as follows. The following terms are used:

  1. Copy 8 bytes of skeleton. They MUST be 00 61 73 6D 01 00 00 00 (hexadecimal).

  2. While skeleton has bytes left, join a section:

    1. Copy a byte id of skeleton.

    2. If id is 10 (the code section) or 11 (the data section), read a byte k of skeleton, which is not copied and MUST be 0 or 1. If k is 0, join the content of the section (§ 7.2.1 Code and data sections), and continue to the next section.

    3. Copy a LEB128 n of skeleton, then n bytes of custom if id is 0, or of sections otherwise.

  3. Every stream MUST have no bytes left.

Reading past the end of a stream, a LEB128 not within its bound, a byte not allowed, or any other condition stated with MUST not holding, results in an invalid archive failure.

7.2.1. Code and data sections

The content of a code or data section split is joined to a separate byte sequence c, which is then appended to the output preceded by its length written. The length MUST be less than 232. c is joined as follows:

  1. Copy a LEB128 count of skeleton.

  2. Repeat count times:

    • For the code section (id = 10), read a byte f of skeleton, which is not copied and MUST be 0 or 1.

      • If f is 0, join a function body to a separate byte sequence b, and append b preceded by its length written, which MUST be less than 232. The body is a LEB128 n of locals copied, then n times a LEB128 of locals and a byte of locals copied, then an expression (§ 7.2.2 Split expressions).

      • If f is 1, copy a LEB128 n of sections, then n bytes of sections.

    • For the data section (id = 11), copy a data segment: a LEB128 kind of misc; if kind is 2, a LEB128 of misc; unless kind is 1, an expression (§ 7.2.2 Split expressions); then a LEB128 n of misc and n bytes of data.

7.2.2. Split expressions

An expression is joined an instruction at a time, keeping a nesting depth d, initially 0, and the recent locals (a list of integers, initially empty) and next (initially 0) made anew for the expression, until its end. An instruction is its opcode, a byte of op copied, then its immediates as the following table tells, copied in order. A byte not in the table is not allowed.

Opcode Immediates
02, 03, 04 a LEB128 of blocktype; then d increases by 1
0B none; the expression ends if d is 0, and d decreases by 1 otherwise
0C, 0D a LEB128 of label
0E a LEB128 n of brtable, then n + 1 LEB128s of brtable
10, 12 a LEB128 of op
11, 13 two LEB128s of misc
1C a LEB128 n of misc, then n bytes of misc
20, 21, 22 a local index (see below)
23–26, 3F, 40, D0, D2 a LEB128 of misc
28–3E a memory argument (see below)
41 a LEB128 of op
42 a LEB128 of at most L64 bytes of i64
43 4 bytes of bytes
44 8 bytes of bytes
00, 01, 05, 0F, 1A, 1B, 45–C4, D1 none
FC a LEB128 x of op, then by x: none for 0–7; two LEB128s of misc for 8, 10, 12 and 14; a LEB128 of misc for 9, 11, 13 and 15–17; any other x is not allowed
FD a LEB128 x of op, then by x: a memory argument for 0–11, 92 and 93; 16 bytes of bytes for 12 and 13; a byte of bytes for 21–34; a memory argument then a byte of bytes for 84–91; none for 14–20, 35–83 and 94–255; any other x is not allowed

Opcodes are hexadecimal, and x is decimal. A memory argument is a LEB128 a of op; if bit 6 of a (26) is set, a LEB128 of misc; then the offset, a LEB128 of at most L64 bytes whose first byte is of op and, if the MSB of that byte is set, whose other bytes (at most 9) are of offset.

A local index i is coded by a LEB128 c of at most L64 bytes of local, which is read and not copied. Let r be the length of the recent locals:

  1. If c < r, i is the item of the recent locals at c (from 0). Otherwise, i is next + unzz(c - r), which MUST be in [0, 232).

  2. Write i.

  3. Remove i from the recent locals if there, insert it at the front, and remove the items after the first 256. Then next becomes the greater of next and i + 1.

The streams are the inputs of the registered module wasm-recons@1 in order (§ 15.6 Aqueous Modules Registry), which rebuilds a module the same way and checks its declared length too, so a compressor can make both from the same splitting. A compressor splits only the parts it can walk to their ends: a section or a function whose size or local indices aren’t in the shortest form, or with an opcode not in the table, is kept as it is, its bytes in sections. The walk takes any byte the table allows, whether WebAssembly does or not; wasm_module has to validate all the same.

The split form has no length of wasm_module, but wasm_module is at most five times as long as payload: every byte is copied once, but for a byte of local or of skeleton (k, f) each giving at most five bytes written.

A local index is coded by its rank among those last used, so that the same fragment of code in two functions is coded the same even where they number their locals differently. Over the modules of an implementation (3.6 MB, each module compressed alone by Zstandard at level 22), the split form saves about 16% of the module blocks' data.

7.3. Resource block

Blocks 6416 through 6716 are resource blocks, whose type is 0110 01ss in binary (§ 3.1.1 Block types). Their data is checksummed Zstandard data (§ 3.2 Shared external structure).

The internal structure after decompression is an uninterpreted byte sequence, which is supplied to modules by formula blocks referencing it (§ 6.2 Formula data, § 7.4 Module execution).

7.4. Module execution

This section is tentative.

A module is a WebAssembly module [WASM-CORE-3] stored as wasm_module, instantiated and invoked through the embedding interface defined in the Appendix of [WASM-CORE-3]. It MUST conform to the following profile, or an invalid archive failure occurs:

Whether the guaranteed limit of 227 bytes (128 MiB) for the Zstandard Window_Size (§ 3.2.1 Zstandard compression method) should also apply to the memory limit.

Every import of a module is from the module name aqr, under one of the following names. The names of imports and exports are prefixed with aqr_ to avoid collisions in the toolchains and hosts that link them into one namespace. Any other import is an invalid archive failure.

aqr_memory

A memory, the working memory.

aqr_params

A memory of the parameters.

aqr_resourcei

A memory of the i-th resource, where i is a decimal number without leading zeros.

aqr_inputk

A memory of the k-th input, where k is a decimal number without leading zeros.

aqr_outputj

A memory of the j-th output, where j is a decimal number without leading zeros.

aqr_request, aqr_release, aqr_commit

Functions of a part module, defined below. A whole module MUST NOT import them.

aqr_heartbeat

A function defined below.

aqr_spawn, aqr_result, aqr_join, aqr_take

Functions of tasks, defined in § 7.4.1 Tasks.

aqr_chan, aqr_send, aqr_recv, aqr_close

Functions of channels, defined in § 7.4.1 Tasks.

aqr_log

The diagnostic function, defined below.

A memory is sized to fit a length L when both its minimum and maximum are ⌈L / 65536⌉ pages, and sized to fit L within a window of m pages when both are min(⌈L / 65536⌉, m) pages.

To apply a module module given a list of input byte sequences inputs, a list of resource blocks resources, a parameter byte sequence params, a list of declared output sizes sizes, a memory limit and a list of task limits, one for each class:

  1. sizes MUST NOT be empty. The number of the task limits MUST be the number of classes of module.

  2. Every aqr_resourcei imported MUST satisfy i < the number of resources, every aqr_inputk imported MUST satisfy k < the number of inputs, and every aqr_outputj imported MUST satisfy j < the number of sizes.

  3. Create the memories imported, each initially zero-filled:

    • The working memory has the minimum size of its import type.

    • The memory of the parameters is sized to fit the length of params, and holds params from offset 0.

    • The memory of the i-th resource is sized to fit its stored content c, and holds c from offset 0.

    • The memory of the k-th input is sized to fit the length of inputs[k], within its window if its import type has one. If module is a whole module, it holds inputs[k] from offset 0. Otherwise let Wk be its size in bytes and the visible range of the k-th input be [0, 0).

    • The memory of the j-th output is sized to fit sizes[j], within its window if its import type has one. If module is a part module, let Wout,j be its size in bytes.

  4. Let outputs be a list of as many empty byte sequences as sizes.

  5. If the total size of the memories in bytes exceeds the memory limit, return a resource failure. The memory limit applies to the total size of the memories of the application’s instance on its own, and the c-th task limit (from 0) to that of each unit of class c on its own: the memories of the task for a class that is not a gang class, and the memories of the tasks of the gang and the messages queued in the channels of the gang for a gang class. An implementation MAY return a resource failure here, before it instantiates module, if the memory limit and the sum of the task limits (a concurrency of 1 for each class, the least it can choose) exceed what the implementation has, which is implementation-defined, but only for an application whose minimal bound is over 512 MiB (see below).

  6. Instantiate module with the memories and the host functions below.

  7. Invoke the entry point with the arguments of type i64, in order: the length of params, the length of the stored content of each item of resources, the length of each item of inputs and each item of sizes. The entry point MUST have these parameters and no results. If it traps or any host function below signals an error, return an invalid archive failure. If it exceeds the memory limit or any implementation-defined limit, return a resource failure. If any task of the application fails, return its failure (that of any of them if several fail). If every instance of the application waits, return an invalid archive failure. Once it returns, every task it spawned MUST be taken, and every channel it made MUST have been given to a gang.

  8. If module is a whole module, set each outputs[j] to the first sizes[j] bytes of the memory of the j-th output, if imported, and to sizes[j] zero bytes otherwise.

  9. The length of each outputs[j] MUST be sizes[j].

  10. Return outputs.

A memory.grow instruction exceeds the memory limit when the total size to which that limit applies would exceed it (the memory limit, or the task limit of the class of the task executing it, for the unit it belongs to; see step 5 above), and MUST succeed otherwise if the maximum of the import type of the working memory allows.

The minimal bound of an application is its memory limit plus the sum of its task limits: the most its memories come to at a concurrency of 1 for each class. A decoder MUST support every application whose minimal bound is at most 512 MiB (229 bytes): step 5’s resource failure before instantiating is only for a larger one. An encoder SHOULD NOT produce an application whose minimal bound is more.

Note: The reference encoder’s default bound is 256 MiB, so that a decoder may run applications side by side; it falls back, for the data concerned, to what it archives without the recompressor whose application would exceed it; an option raises the bound for an archive whose decoders are known to have the memory.

Each application instantiates the module anew, so nothing remains from an earlier one. The size of every memory except the working memory is fixed by the length of its data and the module’s window, if any, so an application declares no sizes of memories. A part module with a window smaller than its data uses the memory as a ring buffer, addressing offset o at o % W, and the memory limit counts the window rather than the data, which bounds the application’s memory whatever the length of its data. A module needing a buffer whose size depends on its data keeps the bytes in the working memory. The memory of an output of a whole module can be passed to the module consuming it as the memory of a whole module’s input, as long as the bytes from offset sizes[j] are zeroed, since the result is the same.

Host functions are defined as follows, where lok and hik denote the visible range of the k-th input, k, j, ptr and len are of type i32, and every offset is of type i64. Called by a task, aqr_request, aqr_release and aqr_commit signal an error.

aqr_request(k, hi)

The k-th input MUST be imported. hi MUST be in [hik, length of inputs[k]], and hi - lok MUST be at most Wk. For each offset o in [hik, hi), write inputs[k][o] to the memory of the k-th input at o % Wk. Then set hik to hi.

aqr_release(k, lo)

The k-th input MUST be imported. lo MUST be in [lok, hik]. Set lok to lo.

aqr_commit(j, hi)

The j-th output MUST be imported. hi MUST be in [length of outputs[j], sizes[j]], and hi - length of outputs[j] MUST be at most Wout,j. For each offset o in [length of outputs[j], hi), append the byte at o % Wout,j of the memory of the j-th output to outputs[j].

aqr_heartbeat()

Does nothing.

aqr_log(ptr, len)

Does nothing. Implementations MAY present the len bytes of the working memory from the offset ptr as a diagnostic message, which is untrusted data from the archive, but doing so MUST NOT affect the application, even when the range is out of bounds. The bytes are a UTF-8 [RFC3629] string, with no terminator: a module SHOULD write well-formed UTF-8, and an implementation presenting them MUST read them as UTF-8, replacing each ill-formed subsequence with U+FFFD, never as another encoding. It is the only diagnostic function; a module reports a failure by aqr_log followed by a trap.

A violation of any MUST in host functions is signalled as an error to the module invocation.

The host writes input memories of a part module only within aqr_request and reads its output memories only within aqr_commit, so the content of every memory depends only on the order of calls made by the module. Implementations MAY block in aqr_request until the requested bytes are available and in aqr_commit until consumers catch up, as long as the result is indistinguishable from this definition.

7.4.1. Tasks

A task is an instance of the module of its own, of a class, run on the bytes of the working memory of the instance that spawns it that the spawn names and giving back a byte sequence, its result. The application’s instance and every task are the instances of the application; each instance has a list of handles, initially empty, each of which is a list of the tasks spawned at once. A task is either pending, running, done (with a result), failed (with an invalid archive failure or a resource failure), or taken. The application’s instance is of no class: the classes, numbered from 0, are those of tasks, and the memory limit, not a task limit, applies to the instance. An instance that is a task of class c spawns only tasks of a class above c, and the application’s instance tasks of any class.

The tasks spawned at once by one aqr_spawn of a gang class are a gang. A unit is a task of a class that is not a gang class, and a gang otherwise; it is pending until it is started, and then running until all of its tasks are done or failed. The tasks of a gang are started together: all of them or none. A pending task is not yet an instance of the application: it has no memories, and it does not wait.

The units of each class start in the order they were spawned over the application (by handle, then by index; no unit overtakes another), and an implementation chooses, for each class c, its concurrency, any number from 1 to the task slots of c: whenever fewer units of class c than that are running, the first of them pending is started. A running task stops running when it is done or failed.

A channel links instances of an application: a queue of messages (byte sequences) of at most its capacity, whether it is closed, its sender and receiver, instances initially none, and its owner, the gang it belongs to, initially none. The application has a list of channels, initially empty. Each instance has a list of the channels it made that have not yet been given to a gang, initially empty. When an instance ends (returns or fails), every channel of which it is the sender is closed. An instance waits while it is within aqr_join, aqr_send or aqr_recv and the condition it waits for does not hold.

To run a task given a list of regions (each a byte offset d and a byte sequence B), arg and i:

  1. Create the memories imported: the working memory has the larger of the minimum of its import type and ⌈E / 65536⌉ pages, E the largest d + B.length of the regions (0 for none), and the maximum of its import type; every other memory has the minimum and maximum 0.

  2. Instantiate the module with the memories and the host functions, and then, for each region (d, B) in order, write B to the working memory from the offset d.

  3. Invoke the task entry point with arg and i. If it traps or any host function signals an error, the task fails with an invalid archive failure. If it exceeds the memory limit or any implementation-defined limit, the task fails with a resource failure.

  4. Once it returns, every task it spawned MUST be taken and every channel it made MUST have been given to a gang, or the task fails with an invalid archive failure.

  5. The task is done, with the result last set by aqr_result, or an empty byte sequence if none.

The host functions of tasks are defined as follows, for the instance I calling them, where arg, n, h, i, ptr, dst and len are of type i32, unsigned:

aqr_spawn(class, count, records, arg, n)

class MUST be less than the number of classes of the module, and above the class of I where I is a task. records + 12 × count MUST be at most the size of the working memory of I. Read count region records from the working memory of I from the offset records, each three u32les: src, len and dst, of which src + len MUST be at most the size of the working memory and dst + len at most 232; let the regions be, for each in order, (dst, the len bytes of the working memory of I from the offset src). If class is a gang class, the channels in the list of I of those not yet given are given to the gang of this spawn: set their owner to it, and empty the list. Otherwise that list MUST be empty. Append to the handles of I a list of n new pending tasks of class class, the i-th (from 0) of which is to run a task given the regions, arg and i once it is started (as the concurrency of the class allows), and return the index of that list among the handles of I.

aqr_result(ptr, len)

I MUST be a task, and ptr + len MUST be at most the size of its working memory. Set the result of I to the len bytes of its working memory from the offset ptr.

aqr_join(h, i)

The i-th task of the h-th handle of I MUST exist and not be taken. Wait until it is done or failed. If it failed, the instance I fails with its failure. Otherwise return the length of its result.

aqr_take(h, i, dst, len)

The i-th task of the h-th handle of I MUST be done (joined), len MUST be at most the length of its result, and dst + len MUST be at most the size of the working memory of I. Copy the first len bytes of its result to the working memory of I from the offset dst, and the task is taken.

The host functions of channels are defined as follows, where c, ptr, dst, len and cap are of type i32, unsigned, the channel C is the c-th (from 0) of the application, which MUST exist, and taking an end of C for I means: the owner of C MUST NOT be none, and I MUST be a task of it or the instance that spawned it; the other end of C MUST NOT be I; and if that end is none, set it to I, and otherwise it MUST be I. A task of a class that is not a gang class therefore takes no end of a channel.

aqr_chan(cap)

cap MUST be at least 1. Append to the channels of the application a new channel of capacity cap, and its index to the list of channels of I not yet given, and return its index.

aqr_send(c, ptr, len)

Take the sender of C for I. C MUST NOT be closed, and ptr + len MUST be at most the size of the working memory of I. Wait until fewer messages than the capacity of C are queued, and then enqueue the len bytes of the working memory of I from the offset ptr.

aqr_recv(c, dst, cap)

Take the receiver of C for I. Wait until a message is queued or C is closed. If no message is queued, return −1 (0xFFFFFFFF). Otherwise let M be the first message. If the length of M exceeds cap, return it, leaving M queued. Otherwise dst + the length of M MUST be at most the size of the working memory of I; dequeue M, copy it to the working memory of I from the offset dst, and return its length.

aqr_close(c)

Take the sender of C for I, and close C.

A task shares nothing with the instance that spawned it but the regions of the working memory it starts with, the result it gives back and the messages of the channels whose ends they are, and each end of a channel is an instance’s: the instances are a Kahn process network, so the outputs do not depend on when, in what order or how many at once the tasks run. Implementations MAY run a task at any time from its start and on any thread, as long as an instance waiting does not keep the others from running. A pending unit holds no memory under any limit (what implementations keep of its regions is their own), and the concurrency of a class, which an implementation picks between 1 and its task slots to fit the memory it has, bounds the units of the class running at once. The memory of an application is then at most its memory limit and, for each class c, the concurrency of c times the task limit of c, which is at most the task slots of c times that limit, however deep the tasks spawn tasks and whatever the order the tasks run in. Lowering a concurrency never deadlocks an application that ends with larger ones: a unit of the highest class never waits for a pending unit of a lower class, and a task of a gang waits only for the other tasks of the gang, which were started with it, for the instance that spawned the gang, which is running, and for units of higher classes, so, by induction from the highest class, every unit runs to its end. Since every channel is within a gang, an application needing more tasks at once than the task slots, such as a pipeline of more stages, spawns them as one gang, which takes one slot. A channel is closed by its sender ending only if it sent to it or closed it, so an instance that may send no message to a channel closes it itself. Every memory but the working memory is empty to a task, so the data a task works on is in its regions; its globals are those of a new instance. A region the module does not overwrite where its data segments and stack are, such as one after the minimum of the working memory, leaves the task’s own data as instantiation made it. The tasks of a spawn start from the same regions, which implementations MAY share between them by copying on write.
Implementations MAY keep decoding archives of earlier versions of the format, which use the reserved opcode 1116 and an earlier aqr_spawn of two parameters, for the modules they know; this is outside this specification.

Whether the memories other than the working memory, read-only, should be visible to tasks, which would require rejecting the instructions writing them; and the windows of a part module in tasks.

8. Overlay archives and imports

8.1. Import block

Blocks 5816 through 5F16 are import blocks, which are non-leaf blocks supplying byte ranges of files outside the segment. Their type is 0101 1ess in binary (§ 3.1.1 Block types), and their data is checksummed Zstandard data (§ 3.2 Shared external structure).

e determines the source (external): preceding segments if 0, and the base if 1.

An import block is an internal import block if external is false, and an external import block otherwise. An external import block MUST NOT occur in an archive other than an overlay archive (§ 3.5 Unencrypted archive).

The internal structure after decompression is as follows:

Name Type
num_imports leb128 ≠ 0
imports import[num_imports]

Each import consists of the following:

Name Type
import_path u8[*] ≠ 0
import_path_sentinel u8 = 0
import_offset leb128
import_size leb128
import_checksum u8[32]

An import in an internal or external import block is an internal import or an external import respectively. Each import supplies a single non-leaf item, so the import block supplies num_imports items in the order of imports. The stored content of the item is the imported content of the import, given by resolving it, the bytes in [import_offset, import_offset + import_size) of the file found at import_path:

import_path is a sequence of zero or more components separated by 2F16 (/). Each component uses the same encodings as entry_name, and its decoded name is subject to the same restrictions as an entry name, so no component can be empty. An empty import_path denotes the root itself, which is a file when the base is a single file.

import_checksum is the SHA-512/256 hash [FIPS180-4] of the imported content.

Since an entry name can’t contain / or be . or .., splitting import_path is unambiguous, and no path can refer to anything outside the root. A path is relative to the root rather than to the directory of the file using it, because a formula block referencing an import is evaluated before the metadata block names that file, and a single leaf block may be shared by files in different directories. The prefixes shared by paths in the same block compress well.
An import refers to a byte range rather than a whole file, so that a decompressor reading a part of a large file only reads and verifies the part actually imported. Multiple ranges of the same file are separate imports.
import_checksum tells a wrong or modified base before anything is evaluated from it. This matters because a wrong base often doesn’t make the evaluation fail, e.g. a Patch instruction produces a different output from a different source. Unlike other checksums in this specification, it uses a cryptographic hash: the base is not part of the archive, so an archive authenticated by other means (e.g. a signature) would otherwise not authenticate the result. It costs 32 bytes per import, which don’t compress.
An import is often used through a module splitting the imported content into streams, i.e. the preprocessing part of a transformation, so that only the streams that changed have to be patched before the module joining them back is applied. Such a module only has to perform the split decided by the compressor, given as its parameters.

8.2. Resolving an import

To resolve an import import of an import block in a segment:

  1. Split import.import_path at every 2F16 into components, unless it is empty, and let names be the result of decoding each component as an entry name. If any component fails to decode, return an invalid archive failure.

  2. Let node be the preceding result of the segment if import is an internal import, or the root of the archive base otherwise.

  3. For each name of names in order:

    1. node MUST be a directory node, and node.children[name] MUST exist.

    2. Set node to node.children[name].

  4. node MUST be a file node.

  5. import.import_offset + import.import_size MUST be at most the length of node.content.

  6. Let content be the bytes of node.content in [import.import_offset, import.import_offset + import.import_size).

  7. The SHA-512/256 hash [FIPS180-4] of content MUST be equal to import.import_checksum.

  8. Return content as the imported content of import.

A violation of any MUST in this algorithm is a base failure in an overlay archive, and an invalid archive failure in a standalone archive.

As any other lookup in an archive value, resolving an import never follows a link node, since a link node is not a directory node. In a standalone archive the preceding result is determined by the archive alone, so a failure is a property of the archive. In an overlay archive it depends on the base even for an internal import, whose path may lead into the part of the base no preceding segment has replaced.

8.3. Reading a base

Reading a base from a file system is outside the archive model, as is extraction. A decompressor doing so maps a directory or a single file to an archive value, and is subject to the following requirements:

An archive can copy any byte of its base into its result, and the base failure of an import with a guessed content tells whether the guess is right. An untrusted archive therefore gets as much access to the base as to its own result, which is why the base is only used when explicitly given. A decompressor is advised to report the files of the base it has read.

A decompressor MAY write the result of an overlay archive into its base directory, i.e. apply it in place. In that case:

Ordering the writes alone is not sufficient in general: a file may be patched from another file which is in turn patched from the former, e.g. when two files are swapped.

9. Encrypted archives

An archive might have been encrypted, in which case it consists of a block-like header of type ENCRYPTED_HEADER_TYPE followed by encrypted chunks. An encrypted archive is mirrored if it starts with ENCRYPTED_MIRRORED_HEADER_MAGIC, in which case copies of its header are placed between encrypted chunks.

Name Type Conditions
checksum_start offset
encrypted_header_magic u8[8] ∈ {ENCRYPTED_HEADER_MAGIC, ENCRYPTED_MIRRORED_HEADER_MAGIC}
lg_work_factor u8 ∈ [0C16, 3F16)
scrypt_ephemeral_secret u8[16]
encrypted_file_key u8[32]
age_header_mac u8[32]
payload_nonce u8[16]
checksum_end offset
header_checksum u64le If mirrored
chunks u8[*]

The 97 bytes from lg_work_factor to payload_nonce are called the header fields.

header_checksum is the XXH64 [XXHASH] hash with the seed zero of all bytes from checksum_start to checksum_end.

header_checksum tells a damaged header from a wrong passphrase before running scrypt. Without it, damage to lg_work_factor, scrypt_ephemeral_secret or encrypted_file_key results in a decryption failure for any passphrase, and damage to lg_work_factor may make scrypt run for an arbitrarily long time.

All fields from lg_work_factor to age_header_mac are used to reconstruct an age v1 textual file header [AGE] as follows:

age-encryption.org/v1
-> scrypt scrypt_ephemeral_secret_base64 lg_work_factor_decimal
encrypted_file_key_base64
--- age_header_mac_base64

Where,

The age payload consists of payload_nonce followed by the encrypted chunks in chunks and is interpreted in accordance with the age v1 format. Every encrypted chunk except the last one is 65,552 bytes long, i.e. 64 KiB of plaintext followed by a 16-byte authentication tag.

If the archive is not mirrored, chunks consists of the encrypted chunks alone. Otherwise chunks also contains mirrored headers: for each encrypted chunk Cm numbered from 0, a mirrored header immediately precedes Cm if and only if m + 1 is either a power of two from 2 to 32 or a positive multiple of 64. The k-th mirrored header numbered from 1 therefore precedes Cm where m = 2k − 1 for k ≤ 6 and m = 64(k − 5) − 1 for k ≥ 6, and no mirrored header follows the last encrypted chunk. A mirrored header consists of the following:

Name Type
checksum_start offset
encrypted_mirror_magic u8[8] = ENCRYPTED_MIRROR_MAGIC
mirror_index u32le = k
mirrored_fields u8[97] = header fields
checksum_end offset
mirror_checksum u64le

mirror_checksum is the XXH64 [XXHASH] hash with the seed zero of all bytes from checksum_start to checksum_end.

As with mirror blocks, the constraints on the fields of a mirrored header are requirements on conforming compressors. A mirrored header always occupies 117 bytes at its position, and its content doesn’t affect the result of interpreting an archive. A mirrored header is intact if its mirror_checksum verifies and its fields other than mirrored_fields satisfy the constraints above. A decompressor MUST NOT use the mirrored_fields of a mirrored header that is not intact.

Mirrored headers allow the archive to be decrypted when its header is damaged. Since every encrypted chunk except the last one has the same length, the k-th mirrored header is always 113 + 65,552m + 117(k − 1) bytes after the start of encrypted_header_magic, with m as above. A single mirrored header found by scanning for ENCRYPTED_MIRROR_MAGIC and verified by mirror_checksum thus gives the start of the archive, the position of every encrypted chunk and every field needed to decrypt them. Unlike mirror blocks, mirrored headers need no offset to tell archives apart: scrypt_ephemeral_secret and payload_nonce are random, so mirrored headers with the same mirrored_fields belong to the same archive.
The interval between mirrored headers doubles from 1 to 64 encrypted chunks and stays at 64 chunks afterwards, which equals the distance of 4 MiB recommended in § 10.3 Mirror block measured in plaintext, so that small archives have mirrored headers soon after the header and the overhead stays at 117 bytes per 64 chunks, about 0.003%, for large archives.

The interval of 64 encrypted chunks follows the tentative distance of 4 MiB in § 10.3 Mirror block and changes with it.

The age payload MUST decrypt into a valid unencrypted archive (§ 3.5 Unencrypted archive), either a standalone archive or an overlay archive, with the following additional constraints:

Conforming compressors MUST NOT encrypt two different plaintexts of an encrypted chunk with the same file key, payload_nonce and chunk number, because the age payload derives the nonce of each chunk from them and the flag of the last chunk alone, so that doing so reveals the difference of the two plaintexts and allows the chunk to be forged. In particular, an encrypted archive is not modified in place, e.g. appended to (§ 3.7 Appending to an archive); an archive changed after its chunks are encrypted is encrypted again as a whole with a new payload_nonce.

A compressor can satisfy the requirement without comparing the plaintexts by encrypting each chunk at most once for a given payload_nonce. A chunk whose plaintext isn’t final yet, such as the first chunk until the size hint is known or the chunks of a room left for blocks written later, is held unencrypted and encrypted once when all of it is known, which the chunks being of a fixed length allows in any order; a chunk already encrypted that has to change, e.g. when the room left turns out to be too short, leads to a new payload_nonce and every chunk encrypted again.
Spacer blocks (§ 3.3 Spacer block) are allowed in the payload as in any other archive. A compressor uses them to fill a room left for blocks written later, and to keep a block and its mirror block an encrypted chunk apart (§ 10.3 Mirror block).

10. Integrity and recovery

10.1. Checksums

The following table summarizes every checksum defined by this specification:

Structure Checksum Computed over Presence
Metadata, dictionary, import, module and resource blocks, except shorthand blocks XXH64 Decoded data Always
Shorthand blocks None (the registered hash identifies the entry but is not checked) Never
Non-leaf data blocks XXH64 Decoded data Always
Leaf data blocks XXH64 Decoded data By type
formula_data of formula blocks XXH64 Decoded data Always
Byte sequences in a formula XXH64 (Checksum instruction) The byte sequence covered Optional
Imports SHA-512/256 (import_checksum) Imported content Always
Mirror blocks XXH64 (mirror_checksum) Stored bytes Always
Recovery blocks XXH64 (recovery_checksum) Stored bytes Always
Slices of a recovery set XXH64 (slice_hashes) Stored bytes within the range If the set has an index block
Header and mirrored headers of an encrypted archive XXH64 (header_checksum, mirror_checksum) Stored bytes If mirrored

The encrypted chunks of an encrypted archive are also authenticated by [AGE].

Unless explicitly requested otherwise, conforming compressors SHOULD use a checksum_method other than None for every leaf block.

Unless explicitly requested otherwise, conforming compressors SHOULD make every output of every formula block checksummed.

An output checksum is the only checksum verifying a module application or a patch, so it matters most for an output computed by an Apply instruction or a Patch instruction. An output computed only by other instructions is already covered by the checksums of formula_data and of the non-leaf items it is computed from, but its output checksum still lets the content of the file be verified by itself, so that whether every file is covered by a checksum doesn’t depend on the instructions of the formula, as the archival profile requires.
A checksum over the stored bytes would only verify that they have been read correctly. A checksum over the decoded data also verifies the decompression, and lets a decompressor reading only some non-leaf blocks verify the file content it has actually read.
Unlike leaf blocks, the checksum is mandatory for metadata, dictionary, import, module and resource blocks. Names and the structure of entries are not covered by any other checksum, and the damage to these blocks affects many files at once. The cost is negligible because these blocks are a small fraction of the archive and each is read only once. It also allows a decompressor to detect the damage to these blocks by reading them alone, which is a prerequisite for using mirror blocks.
The checksum is mandatory for non-leaf blocks. The output checksum of a formula block covers an entire output, so a decompressor that only reads a part of the output can’t verify it without reading the whole output. Instead it can verify the checksums of non-leaf blocks it has actually read. The checksum also lets damage to a non-leaf block be detected before the block is used by a module application, so that the damage is always an invalid archive failure rather than a resource failure, as for formula_data below.
The checksum of formula_data is mandatory and distinct from the optional output checksums declared inside formula_data (§ 6.2.12 Checksum instruction). formula_data is always decoded in its entirety before evaluation, so its checksum can be verified even by a decompressor reading only a part of the output. It lets a damaged formula block be detected by reading the block alone and before evaluating anything, so that a damaged formula block is always an invalid archive failure rather than a resource failure caused by damaged resource declarations, and is distinguished from damage to the non-leaf blocks it references. An output checksum, on the other hand, also verifies the evaluation including module applications, but can only be verified after computing the entire output.
The checksum is optional for leaf blocks to allow the following:

Neither is a concern for long-term preservation, where the archival profile requires the checksum.

10.2. Archival profile

An archive conforms to the archival profile when the content of every file is covered by a checksum:

A conforming compressor producing an archive for long-term preservation SHOULD conform to the archival profile, and SHOULD make it a mirrored archive. A tool verifying an archive SHOULD report whether it conforms to the archival profile.

Other blocks always have checksums, so an archive conforming to the archival profile detects any damage to the content of a file by its decompression alone. Whether an archive conforms can be determined from block types and formula blocks, without decompressing data blocks. An archive conforming to the archival profile depends on nothing outside itself, neither a base nor the registries, so that it can be read by a decompressor at level 2 (§ 13.1 Conformance levels) that knows no registry entry. It may still use modules not registered, which a decompressor at a lower level may refuse to apply.

10.3. Mirror block

A mirror block holds a verbatim copy of another block, called its target block, so that the target block can be restored when it is damaged. Mirror blocks are only defined for block types whose damage affects more than a single parcel and can be detected by reading the block alone, i.e. that have a checksum:

Block Types Target Block Types
C1016 — C1716 1016 — 1716 (§ 4.3 Metadata block)
C5016 — C5716 5016 — 5716 (§ 5.2 Dictionary block)
C5816 — C5F16 5816 — 5F16 (§ 8.1 Import block)
C6016 — C6316 6016 — 6316 (§ 7.1 Module block)
C6416 — C6716 6416 — 6716 (§ 7.3 Resource block)
C7016 — C7716 7016 — 7716 (§ 6.1 Formula block)

As with any other block, the size field of a mirror block is determined by its own type and counts the bytes of its own data. Since a mirror block is longer than its target block, its size field may have to be wider. Therefore a mirror block is either:

The target block MUST be of the target type, which MUST be a block type listed above for the mirror block’s type. Whether a mirror block is widened is signalled by archive_offset.

For example, a mirror block of type C1316 mirrors a metadata block of type 1316, which is uncompressed and has a u64le size field, if unwidened, or a metadata block of type 1216 with a u32le size field if widened. Since C0016 % 4 = 0, both stay in the same row of the table above.

Its data consists of the following:

Name Type
mirror_magic u8[8] ∈ {MIRROR_MAGIC, OVERLAY_MIRROR_MAGIC}
archive_offset leb128
target_offset leb128
target_data u8[*]
checksum_end offset
mirror_checksum u64le

mirror_magic is a fixed byte sequence that allows mirror blocks to be found by scanning when the surrounding structure, such as the header, the footer or the metadata chain, is damaged. It MUST be OVERLAY_MIRROR_MAGIC in an overlay archive and MIRROR_MAGIC otherwise.

archive_offset is the offset from archive_start to the block_start of the mirror block, plus 1 if the mirror block is widened. Since the offset of every block from archive_start is even, archive_offset % 2 tells whether the mirror block is widened, and archive_offset − (archive_offset % 2) is the offset itself.

archive_offset distinguishes mirror blocks of this archive from those of another archive stored inside it, which can be found by scanning and verified by themselves just like the mirror blocks of this archive. Such a mirror block may point to bytes that are not its target block in this archive, for example when the inner archive is split into multiple blocks. Since the inner archive is stored after archive_start of this archive, the start of the archive implied by its archive_offset is always greater. The widened bit is kept in archive_offset, which is read only to tell such archives apart, so that target_offset has its own bit for the direction below.

target_offset is the distance between the block_start of the target block and the block_start of the mirror block, plus 1 if the target block follows the mirror block. Since the distance between any two blocks is even and not 0, target_offset % 2 tells which of the two comes first, and target_offset is at least 2. The target block either precedes or follows the mirror block, and the two MUST NOT overlap.

target_data is a copy of the data of the target block. The bytes of the target block, from its block_start to its block_end, MUST be equal to the concatenation of the following:

  1. The shortest leb128 encoding of the target type;

  2. The length of target_data, encoded as the size field for the target type;

  3. target_data.

The padding of the target block, if any, is not compared, and is restored as a zero byte. The type of the mirror block itself is 2 bytes long, as every block type uses the shortest leb128 encoding.

mirror_checksum is the XXH64 [XXHASH] hash with the seed zero of all bytes from the block_start of the mirror block to checksum_end, including its type and size fields.

Unlike other checksums in this specification, mirror_checksum is computed over the stored bytes, so that the copy can be verified without decoding it and the mirror block can be verified when found by scanning. A decompressor normally uses a mirror block only after the checksum of its target block fails, while a verifier can also compare the copy with the target block directly.
A mirror block is a copy of bytes, not a block to be interpreted at its own position. This is necessary because relative offsets in a metadata block are measured from the metadata block itself.

A mirror block SHOULD be unwidened unless its data doesn’t fit in the size field for the target type.

The data of a mirror block is at least 18 bytes longer than that of its target block, so this is only relevant when the target block is close to the limit of its size field. A single step is always sufficient, since the next width can hold far more than the additional bytes. The width of the target block’s size field can’t be inferred from the size of target_data, because a block may use a wider size field than necessary.
A block with a u64le size field can’t be mirrored when its data is too close to 264 bytes, since its mirror block can’t be widened. This is not a practical concern, because such a block is far larger than 263 bytes, which is beyond what most systems with signed 64-bit file offsets can store in a single file.
To find mirror blocks by scanning, a decoder can look for MIRROR_MAGIC and try each of four possible widths of the size field preceding it. The candidate is correct when the 2 bytes before the size field encode a mirror block type with the matching width and mirror_checksum verifies. The candidate is then discarded unless the start of the archive it implies, its block_start minus the offset given by archive_offset, equals archive_start, because it belongs to another archive stored inside this archive otherwise. If archive_start can’t be determined from header_magic or size_hint, the smallest value implied by verified candidates can be used instead. A decoder that doesn’t know whether the archive is an overlay archive, e.g. because both the header and the footer are damaged, looks for both magics, and the magic of verified candidates tells it.
Mirror blocks have a distinct magic in an overlay archive for the same reason as the header and the footer (§ 3.5 Unencrypted archive): every structure that can be found without the header has to tell whether the archive is an overlay archive.

A mirror block doesn’t belong to any segment and MAY occur anywhere in blocks, before or after its target block, including after the last metadata block. A single target block MAY have multiple mirror blocks. The requirements on a mirror block in this section are requirements on conforming compressors, and a mirror block violating them doesn’t make the archive an invalid archive. Mirror blocks don’t affect the result of interpreting an archive at all.

A mirror block is intact if its mirror_checksum verifies and it satisfies the requirements above other than being equal to its target block. A decompressor MUST NOT restore a target block from a mirror block that is not intact, and SHOULD report a mirror block that is not intact or differs from its intact target block when it reads one.

A mirror block is as likely to be damaged as any other block, but its damage loses nothing as long as its target block is intact. Treating it as an invalid archive would reject an archive whose files are all readable, and only by decompressors that happen to read mirror blocks.

A mirror block and its target block are expected to be damaged together when they are close to each other. Conforming compressors SHOULD separate a mirror block and its target block, from the block_end of the earlier one to the block_start of the later one, by at least 4096 bytes, so that they never share an aligned 4 KiB unit of the underlying storage, and SHOULD separate them by at least 4 MiB (4,194,304 bytes) when possible.

The recommended distances are tentative: a larger one would also cover the loss of a flash erase block, pending the measurement of the overhead of mirror blocks. The interval between mirrored headers in an encrypted archive (§ 9 Encrypted archives) changes with the latter distance.

A damaged encrypted chunk fails to authenticate as a whole, and an aligned 4 KiB unit of the underlying storage may span two adjacent encrypted chunks. Therefore, in place of the distance of 4096 bytes recommended above, conforming compressors SHOULD place a mirror block and its target block in an encrypted archive so that at least one entire encrypted chunk lies between them.

A compressor writing an archive from its start can keep a copy of each block to be mirrored and write all the mirror blocks after the last metadata block, in the order of their target blocks. The mirror block of a block is then as far from it as the blocks after it and the mirror blocks before its own, and the mirror block of the last metadata block, written last, is as far from it as all the other mirror blocks. A spacer block before a mirror block nearer than 4096 bytes makes up the rest. Nothing before the mirror blocks changes, so that the offsets in metadata and formula blocks stay as they are, and only the footer is written again, with size_hint and metadata_backward_offset moved past them. The copies are held until the end, about as many bytes as the blocks mirrored.

A truncation of the archive, however, removes the last metadata block together with all the mirror blocks after it. The compressor can instead hold back the blocks written in the last 4 MiB or more before the last metadata block, and write the mirror block of the last metadata block before them, so that a truncation shorter than them leaves the mirror block and the metadata block can be restored (§ 10.6 Repair). The relative offsets in the metadata block to the blocks before its mirror block, and previous_metadata_offset, then count the mirror block, whose length depends on the metadata block: the compressor encodes the metadata block for a room of a given length for its mirror block, and encodes it again with a longer room until the mirror block fits, filling the rest of the room with a spacer block. Holding back the blocks costs the memory of at least 4 MiB of them, and more when the last ones held back are longer.

In an archive of several segments, a metadata block only becomes available at the end of its segment, so its mirror block can instead be placed after a sufficient amount of the next segment has been written. The mirror block of the last metadata block can’t be distant from it unless the compressor delays the last segment, and closing segments periodically bounds the loss in that case.

10.4. Mirrored archives

An archive is a mirrored archive when every block whose damage affects more than a single parcel can be restored:

A tool verifying an archive SHOULD report whether it is mirrored, and SHOULD report the blocks that have no intact mirror block, whose mirror blocks are all nearer than above, and that differ from an intact mirror block.

Being mirrored is a property separate from the archival profile. The archival profile is about detecting damage and is determined from block types and formula blocks, while being mirrored is about restoring blocks and is determined by comparing mirror blocks with their target blocks. An archive may also be protected by redundancy outside it, e.g. a file system or a separate recovery file, where the archival profile still matters but mirror blocks don’t.
The distance of 4 MiB recommended in § 10.3 Mirror block is not a condition, since an archive smaller than that can’t satisfy it. An archive placing its mirror blocks as described in § 10.3 Mirror block still satisfies the condition of 4096 bytes: the mirror block of the last metadata block is the nearest one when all the mirror blocks are placed after it, and the blocks held back before the metadata block are at least 4096 bytes long when its mirror block is placed before them.

10.5. Recovery sets

A recovery set protects a range of bytes of the archive with parity computed by a Reed–Solomon code, so that damaged parts of the range can be restored, including the content of files that no mirror block covers. The range is divided into slices, each of which is either intact or lost as a whole, and up to as many lost slices as the set has parity shards can be restored. A recovery set consists of the following blocks, called recovery blocks:

Block Types Definition
C116 — C316 Parity block, holding a single parity shard
C416 — C716 Index block, holding the hashes of every data shard

Block type C016 is reserved, as a parity block is always longer than 255 bytes: its shard alone takes a slice size of at least 4096 bytes.

A recovery set without any parity shard consists of index blocks only, which locate damage to the range without restoring it.

Its data consists of the following:

Name Type Conditions
recovery_magic u8[8] ∈ {RECOVERY_MAGIC, OVERLAY_RECOVERY_MAGIC}
archive_offset leb128
range_offset leb128
range_length leb128 ≥ 1
lg_slice_size u8, 12 ≤ lg_slice_size ≤ 63
parity_count leb128
shard_index leb128 < parity_count If a parity block
shard u8[slice size] If a parity block
slice_hashes u64le[data shard count] If an index block
checksum_end offset
recovery_checksum u64le

recovery_magic is a fixed byte sequence that allows recovery blocks to be found by scanning, as mirror_magic does for mirror blocks. It MUST be OVERLAY_RECOVERY_MAGIC in an overlay archive and RECOVERY_MAGIC otherwise.

archive_offset is the offset from archive_start to the block_start of the recovery block, for the same purpose as archive_offset of a mirror block.

The protected range of the set is [range_start, range_end), where range_start is the block_start of the recovery block minus range_offset and range_end is range_start + range_length. The range MUST lie within [archive_start, archive_end), and every recovery block MUST start at or after range_end, i.e. range_offset ≥ range_length. The range MAY start or end in the middle of a block, and MAY contain recovery blocks of other recovery sets and mirror blocks.

The slice size S is 2lg_slice_size, and the number of parity shards p is parity_count. Recovery blocks of an archive with the same range_start, range_length, lg_slice_size and parity_count belong to the same recovery set, and all others belong to other recovery sets.

recovery_checksum is the XXH64 [XXHASH] hash with the seed zero of all bytes from the block_start of the recovery block to checksum_end, including its type and size fields.

The parameters of the set determine its parity shards entirely, so no identifier is needed to tell recovery sets apart: successive sets have distinct ranges, and the sets of another archive stored inside this archive imply another start of the archive by archive_offset.

10.5.1. Slices and shards

The slice grid of an archive has its origin at the start of its header_magic, i.e. at archive_start − 6, which is the start of the payload in an encrypted archive. The slice k of a recovery set, for every integer k ≥ 0, consists of the bytes from the offset k × S to the offset (k + 1) × S from the origin of the slice grid.

The data shards of a recovery set are its slices overlapping its range, in their order. Let k0 be ⌊(range_start − o) / S⌋, where o is the origin; the data shard count n is ⌈(range_end − o) / S⌉ − k0, and the data shard di for 0 ≤ i < n is the slice k0 + i with every byte outside the range taken as zero, whatever the archive holds there. Only the first and the last data shards may be partly outside the range.

Every recovery set with p ≥ 1 also has p parity shards P0, …, Pp−1 of S bytes each, computed from its data shards as in § 10.5.2 Parity shards. If p ≥ 1, n + m MUST NOT exceed 65,536, where m is the smallest power of two that is p or greater.

The shard of a parity block is the parity shard Pj for j = shard_index, and MUST be equal to it. A parity block MUST NOT belong to a recovery set with p = 0.

The slice_hashes of an index block has an entry for every data shard, and the entry i MUST be the XXH64 [XXHASH] hash with the seed zero of the bytes of di within the range, without the zeros outside it.

The slice grid is aligned to the archive rather than to the range, so that a slice of every recovery set is a unit of 2s bytes aligned to the start of the file for an archive starting at the start of the file, where it coincides with the aligned units of the underlying storage that are damaged or lost together; a range aligned to archive_start, 6 bytes after the start of the file, would have every such unit span two slices. In an encrypted archive, a slice of a multiple of 64 KiB is a whole number of encrypted chunks, so that a chunk failing to authenticate loses a single slice. The data shard count is therefore that of the slices the range overlaps, not R / S for the length R of the range: a range of R bytes has ⌈R / S⌉ or ⌈R / S⌉ + 1 data shards depending on where it starts.

10.5.2. Parity shards

A shard of S bytes is a sequence of S / 2 16-bit symbols. The symbol c of a shard, for 0 ≤ c < S / 2, has its least significant byte at the offset 64 × ⌊c / 32⌋ + (c mod 32) in the shard and its most significant byte at the offset 64 × ⌊c / 32⌋ + 32 + (c mod 32). In other words, every 64 bytes of a shard hold 32 symbols, their least significant bytes followed by their most significant bytes.

Symbols are elements of the finite field GF(216) represented as GF(2)[x] / (x16 + x5 + x3 + x2 + 1), in which a polynomial is written as a 16-bit integer whose bit b is the coefficient of xb. A symbol v, however, denotes the element Σb vb × βb, where vb is the bit b of v and β0, …, β15 are the following elements:

b 0 1 2 3 4 5 6 7
βb 000116 ACCA16 3C0E16 163E16 C58216 ED2E16 914C16 401216
b 8 9 10 11 12 13 14 15
βb 6C9816 10D816 6A7216 B90016 FDB816 FB3416 FF3816 991E16

Addition of symbols is therefore their bitwise exclusive or. For 0 ≤ k < 65,536, ωk denotes the element denoted by the symbol k.

The elements βb form a Cantor basis: β0 = 1 and βb2 + βb = βb−1 for 1 ≤ b ≤ 15. In this basis, ω0, …, ω2t−1 form a subspace for every t, which allows the parity shards to be computed by an additive fast Fourier transform.

The data shards are divided into groups of m shards: the group g ≥ 0 consists of dgm, …, dgm+m−1, where di for i ≥ n is taken as S zero bytes, and the data shard dgm+r is placed at the point ω(g+1)m+r. For each symbol position c and each group g, let fg be the polynomial over the field of degree less than m such that fg(ω(g+1)m+r) is the symbol c of dgm+r for every 0 ≤ r < m. The symbol c of the parity shard Pj, for 0 ≤ j < p, is Σg fg(ωj).

Written out by Lagrange interpolation, the symbol c of Pj is Σi di[c] × Πk (ωj + ωk) / (ωm+i + ωk), where k ranges over the points of the group of di other than m + i. It is also the value at ωj of the polynomial of degree less than 65,536 − m that takes the symbol c of di at ωm+i for every i < 65,536 − m. The data and parity shards are thus a codeword of a shortened and punctured systematic Reed–Solomon code over GF(216) with n + p distinct evaluation points, which is maximum distance separable: any n of the n + p shards determine the others. The data shards beyond n are those shortened, and the parity shards from p to m − 1 are those punctured.
This is the code of the additive-FFT Reed–Solomon codes by Lin, Al-Naffouri, Han and Chung, and the shards are in the layout of their implementations: the high-rate code of reed-solomon-simd (rate::HighRateEncoder) and of Leopard-RS over GF(216) produce these parity shards from the data shards as they are. The high-level interface of the former chooses another code when p > n, and the original interface of the latter uses GF(28) when n + m ≤ 256. The parity shards can be computed while the data shards are read, one group at a time: each group is transformed into the coefficients of fg and accumulated, and the sum is transformed into its values at ω0, …, ωm−1 once at the end, so that a compressor needs about 2 × m × S bytes of memory regardless of the length of the range.

10.5.3. Placement

A recovery block doesn’t belong to any segment and MAY occur anywhere in blocks after the range of its set. A recovery set MAY have multiple parity blocks with the same shard_index and multiple index blocks, and need not have a parity block for every parity shard. As for mirror blocks, the requirements on a recovery block in this section are requirements on conforming compressors, and a recovery block violating them doesn’t make the archive an invalid archive. Recovery blocks don’t affect the result of interpreting an archive at all.

A recovery block is intact if its recovery_checksum verifies and it satisfies the requirements above other than those on the values of shard and slice_hashes, and other than the range lying before archive_end. A decompressor MUST NOT restore any byte from a recovery block that is not intact.

A recovery set with p ≥ 1 SHOULD have at least two index blocks, separated from each other as a mirror block is recommended to be separated from its target block (§ 10.3 Mirror block), since the damage to its data shards can’t be located without one. Conforming compressors SHOULD place the recovery blocks of a set at least 4 MiB (4,194,304 bytes) after its range_end when possible, so that a single burst of damage doesn’t remove a part of the range and the parity shards restoring it together. In an encrypted archive, the slice size SHOULD be 64 KiB or greater.

A burst of damage L bytes long loses at most ⌈L / S⌉ + 1 slices, and every damaged region, however short, loses a whole slice. A recovery set with p parity shards thus restores a single burst of up to about (p − 1) × S bytes, or p scattered damaged regions. For example, a range of 1 GiB with S = 64 KiB has 16,384 or 16,385 data shards, and 164 parity shards (about 1%, 10.25 MiB) restore a burst of about 10 MiB.
A compressor choosing the slice size for a range and a ratio r of parity shards to data shards can take the smallest s from 12 (or 16 in an encrypted archive) on for which n + m ≤ 65,536 holds, where S = 2s, n is the data shard count of the range for S, p = max(1, ⌈r × n⌉) and m is as above. The smallest slice size loses the fewest bytes to every damaged region. A larger s roughly halves both n and p, so that the condition only gets looser and some s satisfies it. The condition holds regardless of where the range starts if its length is at most S × (65,535 − m).
The parity shards of a set can only be written after its range has been read entirely, so that the recovery blocks of the last set of an archive are not in the range of any other set, and a truncation of the archive removes the end of the last range together with its recovery blocks. Closing ranges periodically bounds the loss in that case, as closing segments does for the mirror block of the last metadata block. A compressor keeping a range open across a long block holds the parity shards of the ranges closed meanwhile, or extends the range to the end of the block with a larger slice size. Appending to an archive leaves every existing recovery set valid, since its range precedes the new blocks; a new set starting at the range_end of the last existing one also covers its recovery blocks.
To find recovery blocks by scanning, a decoder can look for RECOVERY_MAGIC and try each of four possible widths of the size field preceding it, as for mirror blocks (§ 10.3 Mirror block). Every recovery block type is encoded in 2 bytes. An index block alone tells which slices are damaged without decompressing anything.

A tool verifying an archive SHOULD report its recovery sets, the slices whose hash doesn’t match, and the recovery sets that couldn’t restore their damaged slices.

In an encrypted archive, recovery sets lie in the payload and protect it as decrypted, so that they are blocks like any other and the slice grid is that of an unencrypted archive. They can’t be used without the passphrase, and an encrypted chunk failing to authenticate loses every slice it overlaps, which is a single slice when the slice size is a multiple of 64 KiB as recommended above.

10.6. Repair

Restoring damaged blocks from mirror blocks and damaged slices from recovery sets is a process separate from interpreting an archive: it takes a damaged byte sequence and returns a byte sequence that is then interpreted as any archive is. A decompressor MAY repair an archive before interpreting it, and SHOULD report the blocks it restored.

To decrypt a damaged encrypted archive c, a byte sequence starting where an encrypted archive starts, given the passphrase:

  1. If c starts with ENCRYPTED_HEADER_MAGIC or ENCRYPTED_MIRRORED_HEADER_MAGIC and its header is whole, with an header_checksum that verifies if it has one, take its header fields and whether it is mirrored.

  2. Otherwise, take the mirrored_fields of a mirrored header found by scanning c for ENCRYPTED_MIRROR_MAGIC that is intact and is at the position its mirror_index tells in an archive starting at the start of c, and take the archive as mirrored. If there is none, return failure.

  3. Unwrap the file key as in § 9 Encrypted archives; a failure there is a failure of this algorithm.

  4. Divide the bytes after the header into encrypted chunks and mirrored headers at the positions § 9 Encrypted archives gives them. If the bytes after the last whole encrypted chunk are 16 or fewer, leave them out.

  5. Decrypt each encrypted chunk with the flag of the last chunk set if it is the last one, and if it is the last one and fails to authenticate, again with the flag not set.

  6. Decrypt each encrypted chunk failing to authenticate anyway, as ChaCha20 [RFC8439] without its authentication tag, and take its bytes as untrusted.

  7. Return the payload, the concatenation of the decrypted chunks, and its untrusted bytes, and whether it is cut after a chunk: whether the last chunk authenticated only with the flag not set.

A decompressor MUST report a payload cut after a chunk and MUST treat it as a failure of the extraction or the test, even if what is left of it can be interpreted: every chunk then authenticates, but whatever came after the last one is lost, including any later segment.

An encrypted chunk failing to authenticate is decrypted correctly but for the bytes actually damaged, as ChaCha20 is a stream cipher, but nothing tells which bytes these are, and the ciphertext may have been modified on purpose, as ChaCha20 is malleable. Untrusted bytes therefore never become a file’s content: repairing and salvaging use them only to tell where blocks are, and only as checked by reaching the start of a block an authenticated mirror block tells (§ 10.7 Salvage). A decompressor MUST NOT use untrusted bytes otherwise.
An archive cut short in its last encrypted chunk has a last chunk too short to hold anything, or one that fails to authenticate with or without the flag of the last chunk; the chunk before it, taken as the last, then fails with the flag and authenticates without it.

To repair an archive a, a byte sequence holding an unencrypted archive or the decrypted payload of an encrypted archive from its header_magic on, with a set untrusted of the bytes of a that are untrusted (those of encrypted chunks that fail to authenticate, decrypt a damaged encrypted archive; none for an unencrypted archive):

  1. Let start be 6 if a starts with HEADER_MAGIC or OVERLAY_HEADER_MAGIC, and null otherwise.

  2. Let found be the list of mirror blocks found by scanning a for MIRROR_MAGIC and OVERLAY_MIRROR_MAGIC as described in § 10.3 Mirror block, in the order of their block_start, that are intact when archive_start is taken as the start of the archive they imply and that have no untrusted byte. If a is the payload of an encrypted archive, found is instead the list of the mirror blocks reached by reading blocks by the generic structure (§ 3.1 Block) through bytes that are not untrusted, from the offset 6 if the first 6 bytes of a are not untrusted, and from the block_start of the metadata block that the footer tells if a is not cut after a chunk and ends with a footer that has no untrusted byte, each reading stopping before the first untrusted byte or where a block can’t be read. Their target blocks may lie beyond the end of a. Let rfound be the list of recovery blocks found likewise by scanning a for RECOVERY_MAGIC and OVERLAY_RECOVERY_MAGIC (§ 10.5 Recovery sets), that are intact and have no untrusted byte. If start is not null, remove from found and rfound every block implying a start of the archive other than start.

  3. If start is null:

    1. Set start to the smallest start of the archive implied by the blocks of found and rfound. Remove from found and rfound every block implying another start.

    2. If start is 6, replace the first 6 bytes of a with OVERLAY_HEADER_MAGIC if the blocks of found and rfound have OVERLAY_MIRROR_MAGIC or OVERLAY_RECOVERY_MAGIC, and with HEADER_MAGIC otherwise.

  4. For each target block position t of the mirror blocks of found, in the order of found:

    1. Let restored be the bytes that the first mirror block of found for t requires of its target block.

    2. If the bytes of a from t are those required by any mirror block of found for t, remove the bytes of the target block from untrusted and continue. If a is the payload of an encrypted archive and the target block lies within a and has no untrusted byte, continue.

    3. If a is shorter than t, continue.

    4. If a is shorter than t + the length of restored, append bytes to a up to that length.

    5. Replace the bytes of a from t with restored, and the following byte with 0 if the length of restored is odd and a has a following byte. Remove those bytes from untrusted.

  5. For each recovery set of the blocks of rfound, in the order of decreasing range_end, restore a from it.

  6. If the step 5 restored any byte that no earlier repetition of it has restored, repeat from the step 2.

  7. If start is 6, reconstruct the footer of a.

  8. Return a and untrusted.

To restore from a recovery set with the blocks r of rfound in repair an archive, taking start as archive_start:

  1. If p is 0, return.

  2. Let h be the slice_hashes of the first index block of r, or null if there is none.

  3. A data shard di is erased if any of its bytes within the range is untrusted, or if h is not null and the hash of its bytes within the range (§ 10.5.1 Slices and shards) is not h[i]. If no data shard is erased, return.

  4. Let q be the set of every j for which r has a parity block with shard_index j, with the shard of the first such parity block as Pj.

  5. If there is exactly one choice of the bytes within the range of the erased data shards for which the data shards and Pj for every j in q are as § 10.5.2 Parity shards requires, and every erased data shard has the hash h tells if h is not null, replace those bytes of a with that choice, and remove them from untrusted.

Since any n of the shards of a recovery set determine the others, there is at most one such choice when no more data shards are erased than q has parity shards. It is found by decoding with as many parity shards as there are erased data shards, and exists if the result also agrees with the other parity shards and with the hashes. Without an index block, damage to an unencrypted archive is not located, and only the data shards with an untrusted byte are erased. A restored data shard is trusted, as the parity shards and the hashes lie in encrypted chunks that do authenticate, and so do the data shards not erased.
The steps are repeated because restoring from a recovery set may make others usable: a recovery set restores the recovery blocks of the earlier sets in its range, and the mirror blocks and target blocks restored by either reduce the erased data shards of the others. With recovery blocks placed after their ranges as recommended in § 10.5.3 Placement, restoring from the sets in the order of decreasing range_end restores everything in a single pass, and the repetition finds nothing more.
The bytes of a target block that equal an intact mirror block, or that are restored from one, are trusted even in an encrypted chunk that fails to authenticate, since the mirror block lies in chunks that do authenticate. An unencrypted archive has no untrusted bytes: its bytes are only as trustworthy as the checksums verifying them, damaged or not.
The payload of an encrypted archive is not scanned for mirror blocks: a file’s content, stored as it is, may hold bytes of an intact mirror block, e.g. a file of an attacker in the encrypted archive of someone else, and taking it would replace a block of the archive with the attacker’s once the attacker damages or truncates the ciphertext. The blocks read from the header and from the last metadata block, where the mirror blocks are placed as § 10.3 Mirror block recommends, are those that the encryption authenticates as blocks. For the same reason, a block that authenticates is never replaced.

To reconstruct the footer of a byte sequence a starting with a header magic:

  1. Read blocks from the offset 6 by the generic structure (§ 3.1 Block), with data of every block taken without parsing, until the bytes at the current offset start with the shortest leb128 encoding of FOOTER_TYPE or OVERLAY_FOOTER_TYPE, until a ends at or before the end of the block being read, or until the bytes at the current offset are not a leb128 in the shortest encoding.

  2. If the reading reached a footer whose footer_size tells that it ends at the end of a and that has no untrusted byte, return.

  3. If the reading stopped otherwise than by a ending or by reaching the start of a footer, return.

  4. Let m be the last block read that is a metadata block. If there is none, or any block read after m is neither a spacer block nor of a non-critical type, or the bytes after the last block read start with the shortest leb128 encoding of a critical type other than those of spacer blocks, FOOTER_TYPE and OVERLAY_FOOTER_TYPE, return.

  5. Let e be the offset after the padding of the last block read (the block_end if it has none). Truncate a to e bytes, appending zero bytes if it is shorter.

  6. Append a footer with the footer magic matching the header magic of a, size_hint e − 6 and metadata_backward_offset from footer_start back to the block_start of m.

  7. If the first block read is a size hint block whose data is 10 bytes long, set its size_hint to e − 6, encoded in 10 bytes as an encrypted archive’s payload has it.

Reconstructing the footer restores an archive cut short after its last metadata block, where the mirror blocks and the footer after it have been lost; with the mirror block of the last metadata block placed before the last blocks (§ 10.3 Mirror block), also an archive cut short within its last metadata block, which the earlier steps restore beyond the end of a. The footer is not reconstructed when a critical block other than a spacer block follows the last metadata block, whole or cut short, since every content block is followed by a metadata block describing it, nor when no metadata block is read: an archive without a metadata block decompresses into the concatenation of its leaf blocks, and the archive of a single file cut short would otherwise decompress into a shorter file without any failure. An archive cut short before its last metadata block has lost content blocks, and isn’t repaired into a byte sequence that interprets as an archive.
A decompressor can find the same mirror blocks more cheaply by reading the blocks of a one after another by the generic structure (§ 3.1 Block) when that reaches a footer ending at the end of a, taking the blocks of mirror block types that are intact. The two differ only when the data of another block holds an intact mirror block of the same archive verbatim at the same position, which a compressor never writes. A damaged size field usually makes the reading fail to reach the footer, and a mirror block in the data of another block is then found only by scanning.
The header magic is not restored: it is a constant, and an archive whose header magic is damaged is restored from the mirror blocks only, whose implied start of the archive tells where the header magic was.

10.7. Salvage

Salvaging an archive extracts what can still be read from an archive that is an invalid archive even after it is repaired, e.g. one cut short before its last metadata block or one with a damaged leaf block that no mirror block covers. Its result is not the result of interpreting an archive: a decompressor MAY salvage an archive it can’t interpret, and then MUST report every file it doesn’t extract and why, and MUST NOT report the archive as valid.

To salvage an archive a, a byte sequence as for repair an archive:

  1. Repair a, except that, if a starts with a header magic, a target block position t beyond the end of a isn’t skipped in the step 4.3:

    1. Read blocks from the offset 6 by the generic structure (§ 3.1 Block) as far as they are whole, and let e be the offset after the padding of the last block read (6 if none).

    2. If e > t or t − e is odd, skip t.

    3. Truncate a to e bytes, and append a spacer block of t − e bytes if t > e. The bytes from e to t are lost.

    4. Continue with the step 4.4.

    Before reconstructing the footer in the step 7, if any byte of a is still untrusted, lose the untrusted blocks of a. When reconstructing the footer returns in its step 4 only because of blocks after m or a block cut short after the last block read, the bytes from the end of m (after its padding) to the end of a are lost: truncate a to the end of m and continue with its step 5. The blocks in those bytes, as far as they are whole and have no untrusted byte, MAY be salvaged apart as the blocks of a standalone archive of their own without a metadata block, each of its leaf blocks evaluated by itself as in the step 2: the outputs of those evaluated are files and forks whose names are not known, which a decompressor MUST report as such if it extracts them.

  2. Interpret a as in interpret an archive, with the following exceptions, each of which makes the files and forks concerned lost rather than a an invalid archive:

    • A file or a fork whose leaf block offset doesn’t refer to a leaf block is lost.

    • The module_offsets, resource_offsets and dictionary_offsets may list more blocks than there are, if those there are the first ones listed.

    • A formula block whose formula_data fails to decode or to parse, including a reference to a non-leaf block that isn’t there, makes the files and forks of its outputs lost.

    • If any file or fork is lost, a non-leaf block that no formula block references is not a failure.

    • Every leaf block is evaluated by itself: a failure of its evaluation, including a checksum that doesn’t verify, makes the files and forks of its outputs lost, and the other leaf blocks are evaluated as usual.

To lose the untrusted blocks of a byte sequence a starting with a header magic, given the positions known of the block_start of every mirror block of found in repair an archive and of its target block, of every recovery block of rfound, and of the footer if one ends at the end of a and has no untrusted byte:

  1. Let o be 6.

  2. While o is less than the length of a:

    1. Read blocks from o by the generic structure as far as they go, until the start of a footer.

    2. Let s be the block_start of the first block read whose type or size has an untrusted byte, if any. If there is such a block, and the reading from s doesn’t reach exactly a position in known greater than s before failing, reaching the end of a, or passing such a position:

      1. Let k be the smallest position in known greater than s, or the length of a if there is none.

      2. The bytes from s to k are lost: replace them with a spacer block of their length.

      3. Set o to k and continue.

    3. Every block read with an untrusted byte is lost: replace its bytes, padding included, with a non-leaf data block of the same length without compression whose checksum doesn’t verify if it is a non-leaf data block, and with a spacer block of the same length otherwise.

    4. If the reading stopped at a footer with an untrusted byte, truncate a to the start of that footer.

    5. Stop.

A lost non-leaf data block is replaced with one whose checksum doesn’t verify, not with a spacer block, so that the non-leaf blocks after it keep their ordinals in the formula blocks referring to them: a formula block slicing the lost block then fails and its files are lost, while the formula blocks after it slice the blocks they refer to. The bytes from a block read astray to the next known position are lost as a single spacer block, as the blocks there can’t be told apart; a formula block before them whose non-leaf blocks were among them then refers to the wrong non-leaf blocks, which its output checksums detect.
Salvaging gives every file whose bytes are all there and verify where they have checksums, and only those files within its parcel where the damage hits it. A file covered by no checksum, i.e. one of a leaf block without a checksum or of a formula block without an output checksum, is given as it decodes, since nothing tells whether it is damaged; an archive conforming to the archival profile has no such file. The bytes lost by a truncation become a spacer block, so that the blocks before them and the metadata block restored after them are read by the generic structure, while the blocks that were in them are no longer there and their files are lost. A leaf block partly lost is lost as a whole, even without a checksum. An archive cut short in a segment whose metadata block can’t be restored is salvaged as the segments before it: the blocks of that segment after the last metadata block whole are lost together, but for the contents of its leaf blocks still whole, which can be salvaged without their names, as a segment refers only to blocks of its own. An archive cut short just after a metadata block other than the last, however, is repaired into the archive of the segments before the cut, as nothing after the metadata block remains to tell that more followed; only the absence of segment-status in it tells that segments may have been lost (§ 11.3 segment-status: Commits and the end of an archive).
Evaluating every leaf block by itself and reporting the failures of some of them is what a decompressor computing a partial result (§ 13.2 Partial access) of the other files does anyway. Salvaging adds that the structure may refer to blocks that aren’t there.
A repaired encrypted chunk can’t be encrypted again with the original file key and payload_nonce unless its plaintext is fully restored, because doing so otherwise reuses a nonce for a different plaintext. Since a damaged chunk usually contains leaf blocks that mirror blocks don’t cover, a repaired encrypted archive is expected to be decrypted, or encrypted again with a new file key.

11. Optional metadata

The optional_metadata field of the metadata block (§ 4.3 Metadata block) holds the optional metadata, encoded in CBOR [RFC8949]. This part encodes any metadata that is not strictly necessary to read the file, including associated dates and times, access control information and comments.

The optional metadata is described by the following CDDL [RFC8610] definitions:

aqr-opt-metadata = {
  aqr-archive-comment
  ? &(entry-extras: -2) => [* aqr-entry-extra / null]  ; Indexed by entry numbers, starting from the root directory
  aqr-segment-status
  aqr-journal

    * $$aqr-opt-metadata-member  ; For future extensions
}

$$aqr-opt-metadata-member = ()  ; No extension is defined by this specification

aqr-entry-extra = {
  aqr-entry-timestamps
  aqr-entry-comment
  aqr-entry-link-group
  aqr-entry-xattrs
  aqr-entry-forks
  aqr-entry-windows
  aqr-entry-unix
  aqr-entry-macos
  aqr-entry-zip

    * $$aqr-entry-extra-member  ; For future extensions
}

$$aqr-entry-extra-member = ()  ; No extension is defined by this specification

; Key -8 is reserved for access control lists.

It MUST be a single well-formed CBOR data item [RFC8949] without any following bytes, and no map in it can have duplicate keys. The value of every key defined by this specification MUST match its definition. If optional_metadata_size is 0, the optional metadata is an empty map {}.

The members of aqr-entry-extra are grouped by their purposes, and each group is described in the following sections. Keys of aqr-entry-extra from -1 to -59 are platform-neutral. Keys from -60 to -69 are specific to Windows, -70 to -79 to Unix, -80 to -89 to macOS, and -90 to -99 to other platforms. Keys from -100 are specific to the formats an entry was converted from. Key -8 is reserved for access control lists in a future version of this specification, and a compressor MUST NOT use it.

11.1. Timestamps

The keys crtime, ctime, mtime, atime and arctime of aqr-entry-extra record the timestamps that file systems and archive formats associate with a file, a directory or a link. Each is an aqr-time described below, and is absent if the source doesn’t have the corresponding timestamp.

aqr-entry-timestamps = (
  ? &(crtime: -1) => aqr-time
  ? &(ctime: -2) => aqr-time
  ? &(mtime: -3) => aqr-time
  ? &(atime: -4) => aqr-time
  ? &(arctime: -5) => aqr-time
)
crtime

The creation time, also known as the birth time: when the file was created at its location. It is not updated afterwards, but copying a file usually creates a new one with the time of copying. Examples include the creation time of NTFS and FAT, the create date of HFS+ and APFS, i_crtime of ext4 and otime of Btrfs, which are exposed as st_birthtime on BSD and macOS and as stx_btime of statx on Linux.

ctime

The status change time: when the file’s metadata (e.g. permissions, ownership or link count) or its content was last changed. It is the POSIX st_ctime, and the change time of NTFS and the attribute modification date of HFS+ are similar. POSIX doesn’t allow it to be set to an arbitrary time, so an extracting decompressor usually can’t restore it.

mtime

The modification time: when the file’s content was last changed. For a directory, it is when an entry was last added, removed or renamed in it. This is the timestamp that virtually all file systems and archive formats have.

atime

The access time: when the file’s content was last read. Many systems update it lazily or not at all for performance (e.g. relatime and noatime on Linux, and the access time updates of NTFS that can be disabled), and FAT only keeps the date. Reading a file for archiving can itself update it.

arctime

The backup time: when the file was last backed up, as recorded by backup software. File systems don’t update it by themselves. An example is the backup date of HFS and HFS+, exposed as ATTR_CMN_BKUPTIME of getattrlist on macOS. It is the time stored by the source, not the time when the archive itself was made.

The name "ctime" is ambiguous. Windows and formats originating from it use "C" for the creation time: st_ctime in the Microsoft C runtime, the Ctime of ZIP extra field 0x000a, and the CTime of 7z and RAR all go to crtime, not ctime. POSIX formats such as pax use it for the status change time, which goes to ctime.

aqr-time represents a point in time or a local date and time, together with a quantization interval and optional time zone information. It is described by the following CDDL definitions, where etime is the CDDL type name for the extended time defined in [RFC9581]:

aqr-time =
  $aqr-base-time /                          ; UTC timestamp, scaled and quantized
  [$aqr-base-time] /                        ; Local time in unknown time zone, encoded like UTC (*)
  [$aqr-instant, offset: aqr-tz-offset] /   ; UTC timestamp with time zone offset ($)
  etime /                                   ; Extended time defined in [[!RFC9581]]
  [etime]                                   ; Local time in unknown time zone, encoded like UTC

aqr-tz-offset = -48..56  ; In 15 minutes, positive east of UTC as in ISO 9660

$aqr-base-time  = #6.100(int)    ; 1d (FAT atime*), date as in [[!RFC8943]]
$aqr-base-time /= $aqr-instant  ; Can be given a time zone offset

$aqr-instant  = #6.121(int)  ; 2s (FAT mtime*, exFAT atime$, ZIP*, CAB*)
$aqr-instant /= int          ; 1s (~ext3, HFS/HFS+, tar, ISO 9660$)
$aqr-instant /= #6.122(int)  ; 10ms (FAT ctime*, ISO 9660$, exFAT$)
$aqr-instant /= #6.123(int)  ; 1ms
$aqr-instant /= #6.124(int)  ; 1us (UDF)
$aqr-instant /= #6.125(int)  ; 100ns (NTFS, RAR*, 7z)
$aqr-instant /= #6.126(int)  ; 1ns (ext4, ZFS, Btrfs, APFS)

A base time value $aqr-base-time is an integer, either untagged or tagged as follows. The integer multiplied by the scale is the number of seconds since 1970-01-01T00:00:00Z, not counting leap seconds as in the CBOR tag 1 [RFC8949]; negative values denote the time before that epoch. The tag also determines the quantization interval, which is relative to the time value and specified in § 15.4 Quantization Interval. #6.100 is the number of days since 1970-01-01 [RFC8943], and is the only base time value that is not an $aqr-instant, i.e. it can’t be given a time zone offset.

Encoding Quantization Interval (s) Scale (s) Examples
#6.100(int) [0, 86400] 86400 FAT atime*
#6.121(int) [0, 2] 2 FAT mtime, exFAT atime$, ZIP, CAB*
int [0, 1] 1 ext3 and earlier, HFS/HFS+, tar, ISO 9660$ (directory records)
#6.122(int) [0, 10-2] 10-2 FAT ctime*, ISO 9660$ (volume descriptors), exFAT$
#6.123(int) [0, 10-3] 10-3 -
#6.124(int) [0, 10-6] 10-6 UDF
#6.125(int) [0, 10-7] 10-7 NTFS, RAR*, 7z
#6.126(int) [0, 10-9] 10-9 ext4, ZFS, Btrfs, APFS

In the Examples column, formats marked with * store (or can store) a local time in an unknown time zone, and formats marked with $ store a local time with its offset from UTC.

Tags 121 to 126 are the tags for enumerated alternatives 0 to 5 [CBOR-NOTABLE-TAGS], whose meanings are given by the schema in use rather than by the tags themselves. They are therefore meaningful only as $aqr-instant, and are not shortcuts for the extended time in general.
Since a time zone offset is a multiple of 15 minutes, converting a local time to UTC keeps it a multiple of 2 seconds. A local date, however, doesn’t stay a multiple of 1 day unless the offset is zero, so a date with a known time zone offset can only be given as an extended time (e.g. 1001({1: 1697641200, -14: 86400, -10: "+09:00"}) for 2023-10-19 in UTC+9).

Each alternative of aqr-time has the following meaning:

$aqr-base-time

A point in time without any time zone information.

[$aqr-base-time]

A local date and time in an unknown time zone, encoded as if it were UTC. It does not identify a single point in time.

[$aqr-instant, aqr-tz-offset]

A point in time, as in the bare $aqr-base-time, with the offset of the time zone where it was recorded, in 15-minute increments as in ISO 9660. The offset is positive east of UTC: UTC+x hours is 4x and UTC−x hours is −4x. The local time is the base time value plus 900 times the offset in seconds.

etime

An extended time [RFC9581] (tag 1001), for any value which doesn’t fit the other alternatives. The time zone information, if any, is given as a Time Zone Hint (key -10 or 10), and the quantization interval is given as a key described in § 15.4 Quantization Interval.

[etime]

A local date and time in an unknown time zone, encoded as if it were UTC. The extended time MUST NOT contain a Time Zone Hint (key -10 or 10), and its timescale (key -1, -13 or 13), if present, MUST be 0 (UTC).

etime or [etime] MUST NOT be used if another alternative of aqr-time represents the same time value, the same quantization interval and the same time zone information. A Time Zone Hint is the same as an aqr-tz-offset only when it is a numeric offset (e.g. "+09:00") equal to that offset; an extended time without any quantization interval is never the same as other alternatives.

As a consequence, timestamps from common file systems and archive formats never need the extended time, and an implementation without full support for [RFC9581] can still handle them. Such an implementation can ignore an aqr-time using the extended time, as an extracting decompressor is allowed to ignore any extra metadata (§ 12 Extraction).
Some precisions found in the wild are only representable with the extended time, e.g. minutes in ASN.1 UTCTime, 2-32 s in NTP and 1/300 s in SQL Server DATETIME, and so are time zone offsets that are not multiples of 15 minutes as in UDF. They are rare enough in file metadata not to deserve their own encodings.

Two aqr-time values in different encodings are different even when they denote the same point in time. For example, 1 and 122(100) differ in their quantization intervals. A compressor converting a timestamp from a file system or another archive format SHOULD preserve its precision, i.e. use an encoding whose quantization interval matches the source.

Comparing timestamps from file systems of different precisions is a common source of problems, e.g. an unchanged file copied from ext4 to FAT appears to be modified. Recording the quantization interval for each timestamp, rather than for each archive, lets any consumer tell whether two timestamps can denote the same time, even after an archive mixes multiple sources by editing.
The following are possible encodings of 2023-10-19T14:12:34.87Z, or 23:12:34.87 in UTC+9, depending on the source:

Timestamps from ZIP are discussed in § 16.2 Timestamps from ZIP.

11.2. Comments

archive-comment of aqr-opt-metadata and comment of aqr-entry-extra are free-form comments attached to the whole archive and to each entry respectively, e.g. an archive comment and a file comment of ZIP or RAR. Both are an aqr-text, which is a Unicode string, a WTF-8 [WTF-8] string with lone surrogates, or a byte sequence in an unknown encoding:

aqr-archive-comment = (
  ? &(archive-comment: -1) => aqr-text
)

aqr-entry-comment = (
  ? &(comment: -6) => aqr-text
)

aqr-text =
  tstr /          ; Unicode
  #6.273(bstr) /  ; WTF-8, i.e. UTF-16 + surrogate compat
  bstr            ; In unknown encoding

An aqr-text MUST be a tstr if it can be represented as one, i.e. if it is a sequence of Unicode scalar values. The byte string in #6.273(bstr) MUST be a valid WTF-8 string that is not a valid UTF-8 string, i.e. it MUST contain at least one encoded lone surrogate. A bstr is not subject to these requirements even if it happens to be a valid UTF-8 string, as its encoding is not known.

Comments from ZIP are discussed in § 16.3 Comments from ZIP.

11.3. segment-status: Commits and the end of an archive

segment-status of aqr-opt-metadata tells whether the compressor meant its writing to end with the segment, and whether that writing was complete:

aqr-segment-status = (
  ? &(segment-status: -3) => true / aqr-incomplete
)

aqr-incomplete = {
  ? &(files: 0) => uint  ; Number of files left out
  ? &(bytes: 1) => uint  ; Their total size in bytes
}

It doesn’t affect the result of interpreting an archive (§ 4.4 Interpreting an archive): a later segment may still follow it, e.g. a segment appended to the archive afterwards, and it is not a part of the extra of the result. Conforming compressors SHOULD put true in the optional metadata of the last segment of a writing that completes, and aqr-incomplete in that of the last segment of every other commit, and nothing in any other segment.

A decompressor SHOULD report an archive whose last segment has aqr-incomplete as incomplete, telling what was left out if given, while it still interprets the archive as usual. A decompressor that reconstructs the footer of an archive after a metadata block without segment-status SHOULD report that segments after it may have been lost, and SHOULD NOT report the archive as intact.

An archive cut short just after a metadata block other than the last can’t be told by its blocks from an archive that has lost only its footer and the mirror blocks before it, as either ends with a metadata block followed by nothing. The footer is reconstructed either way (§ 10.6 Repair), and segment-status tells the two apart where the compressor meant a footer to follow only the last segment it wrote. An archive cut short just where a segment was later appended to it still reads as whole, as it was before the appending. The hint is lost with the optional metadata of a damaged metadata block, which then can’t be relied on for anything else either.
true and aqr-incomplete are exclusive, as a writing either completed or didn’t, and only the last segment tells about the archive: a writing that continues one left incomplete ends with true without undoing anything of the segments before.

11.4. journal: Points to return to

journal of aqr-opt-metadata names the states of the archive that its writer meant to be returned to, e.g. the states after each run of a backup:

aqr-journal = (
  ? &(journal: -4) => {* aqr-journal-time => aqr-text}
)

aqr-journal-time = uint  ; Milliseconds since 1970-01-01T00:00:00Z, leap seconds not counted

Each key is the time of a journal point and its value is the label of the point, which may be empty. The state of a journal point in the journal of a segment is the result of interpreting the archive cut after that segment, i.e. with the segments after it removed. Several points of a segment therefore name the same state, as when a writing found nothing to change and added a segment of no entries to name the state again. Journal points need not be unique in their labels.

The journal doesn’t affect the result of interpreting an archive (§ 4.4 Interpreting an archive), and it is not a part of the extra of the result. A decompressor MAY be asked for the state of a point, and then interprets only the segments up to the point’s. Which point is meant by a time or a label, if the user gives one, is up to the decompressor: it SHOULD take the latest point whose time is not after the given time, and the last of the points of the label. Where the same time occurs in several segments, or the times don’t increase with the order of segments, the order of the segments tells which point is the later, and a decompressor SHOULD warn of the latter.

Conforming compressors SHOULD put journal only in the metadata block of a segment with segment-status true (§ 11.3 segment-status: Commits and the end of an archive), as the state after a commit of an incomplete writing is not one to return to. They SHOULD NOT put one unless the user asked for it, as the time is information about the user.

aqr-entry-link-group = (
  ? &(link-group: -7) => int  ; File nodes only
)

link-group records that file nodes were originally the same file, i.e. hard links to the same inode on Unix or to the same file record on NTFS. It has no effect on other nodes, and a compressor SHOULD NOT give it to them. A non-negative value is segment-local and denotes the same group only within the same segment, while a negative value is archive-wide and denotes the same group in every segment. The resulting link group in the archive value is determined by interpreting a segment.

A compressor MUST NOT give the same link group to files that were not the same file. It SHOULD NOT give a link group that no other file node has. A file node with link-group SHOULD refer to the same leaf block as other file nodes in the same group, and SHOULD have the same entry extra as them except for link-group and comment.

Segment-local groups allow the inode number (st_ino) to be used as is, as CBOR can encode any 64-bit unsigned integer as a non-negative int, and can be assigned without reading earlier segments. Inode numbers are still only unique within a single file system, so a compressor reading from multiple file systems has to renumber some of them. Archive-wide groups link files across segments, e.g. when a compressor periodically starts a new segment while archiving a single tree, or when it appends a segment to an existing archive. A compressor using them is responsible for keeping them consistent with all other segments, e.g. by reading earlier metadata blocks when appending. Inode numbers are not suitable as archive-wide values as is, because they can be reused after the file is removed or come from an unrelated file system, and linking unrelated files with the same content can’t be detected by any decompressor.

A link group is consistent if all file nodes with that group in the final archive value have the same content. An inconsistent group, e.g. after a later segment has replaced only some of its file nodes, is not an invalid archive failure, but has no effect on extraction (§ 12 Extraction).

The consistency can only be determined from the final archive value, because a later segment may replace any file node in the group. Treating an inconsistent group as an invalid archive would make a decompressor reading only some segments unable to detect such failure (§ 13.2 Partial access). A decompressor can instead check the consistency before creating hard links, at no cost when file nodes refer to the same leaf block. Hard links are not part of the archive model, so a decompressor ignoring link-group still extracts the correct files with the same contents. Directory hard links, found in some macOS Time Machine volumes, are not represented.

11.6. xattrs: Extended attributes

aqr-entry-xattrs = (
  ? &(xattrs: -10) => { * aqr-text => bstr }
)

xattrs records the extended attributes of a node, i.e. named values that the file system attaches to it: extended attributes of Linux, macOS and BSDs, and extended attributes (EAs) of NTFS. Each key is the name of an attribute as the platform gives it, including any namespace prefix (e.g. user. and security. on Linux), and MUST NOT be empty. Each value is the value of the attribute.

A compressor MUST NOT put the following attributes into xattrs:

These names are only special on macOS. An attribute of the same name from another platform, e.g. an EA of NTFS, is put into xattrs as any other, though it may not be written on macOS (§ 12 Extraction).

Extended attributes are small on most platforms: Linux limits a value to 64 KiB, and NTFS limits all EAs of a file to 64 KiB. macOS has no such limit, but its large attributes are mostly resource forks, which are stored as forks in leaf blocks rather than in the metadata block.

POSIX access control lists of Linux appear as the attributes system.posix_acl_access and system.posix_acl_default, and can be put into xattrs as is. Key -8 is reserved for access control lists that are not extended attributes, e.g. those of macOS and NTFS; a future version defining it will also decide which takes precedence when both describe the same node.

11.7. forks: Resource forks and alternate data streams

aqr-entry-forks = (
  ? &(forks: -50) => { * aqr-text => aqr-fork }
)

aqr-fork = [
  offset: aqr-seg-offset,
  output: uint,
  size: uint,
]

aqr-seg-offset = uint  ; Relative offset from the metadata block, as in [[#metadata-block]]

forks records the forks of a node, i.e. its named data streams other than the content of a file. Each key is a fork name, and each value refers to the content of the fork: offset is the relative offset to a leaf block in the same segment, measured backward from the beginning of the metadata block as other relative offsets in § 4.3 Metadata block, output is an output index of that block, and size is the length of the content. They are resolved when setting the forks of the node, and a leaf block referenced by forks is described by the metadata block (§ 4.2 Content blocks). Any number of file_params and forks MAY refer to the same leaf block and output.

Forks are stored in leaf blocks rather than in the metadata block, as files are, because they can be as large as files and hold data worth sharing and recompressing, e.g. fonts and images in classic Mac OS resource forks.

Forks of platforms map to fork names as follows:

Any other name from a platform is a fork name as is, e.g. an alternate data stream named com.apple.ResourceFork, or one named AFP_Resource on a directory, though such forks may not be written on macOS (§ 12 Extraction). A compressor SHOULD NOT record how the file system stores a fork, e.g. whether it is sparse or compressed.

11.8. Windows-specific metadata

aqr-entry-windows = (
  ? &(win-link-type: -60) => &(file: 0, directory: 1, junction: 2)  ; Link nodes only
  ? &(win-attrs: -61) => uint
)

win-link-type determines the kind of a link to be created on Windows (§ 12 Extraction). It has no effect on nodes other than link nodes, and a compressor SHOULD NOT give it to them. "file" and "directory" denote a symbolic link to a file and to a directory respectively, and "junction" denotes a junction (a mount point reparse point).

win-attrs holds the file attributes of Windows (FILE_ATTRIBUTE_*), limited to the following bits; any other bit MUST be zero.

Bit Attribute
0x1 FILE_ATTRIBUTE_READONLY
0x2 FILE_ATTRIBUTE_HIDDEN
0x4 FILE_ATTRIBUTE_SYSTEM
0x20 FILE_ATTRIBUTE_ARCHIVE
0x100 FILE_ATTRIBUTE_TEMPORARY
0x2000 FILE_ATTRIBUTE_NOT_CONTENT_INDEXED
The other attributes are either given by the archive model, e.g. FILE_ATTRIBUTE_DIRECTORY, FILE_ATTRIBUTE_REPARSE_POINT and FILE_ATTRIBUTE_NORMAL (the absence of any other), or tell how the file system stores the file rather than anything about the file, e.g. FILE_ATTRIBUTE_SPARSE_FILE, FILE_ATTRIBUTE_COMPRESSED, FILE_ATTRIBUTE_ENCRYPTED and FILE_ATTRIBUTE_INTEGRITY_STREAM, and the states of cloud files such as FILE_ATTRIBUTE_OFFLINE, FILE_ATTRIBUTE_PINNED and FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS. The file attributes in the external file attributes of ZIP [APPNOTE] map to win-attrs in the same way.

11.9. Unix-specific metadata

aqr-entry-unix = (
  ? &(unix-mode: -70) => uint .le 4095  ; Permission bits only (0o7777), without S_IFMT
  ? &(unix-uid: -71) => uint
  ? &(unix-gid: -72) => uint
  ? &(unix-uname: -73) => aqr-text
  ? &(unix-gname: -74) => aqr-text
)

unix-mode only holds the permission bits, including the set-user-ID, set-group-ID and sticky bits. The kind of an entry is solely determined by its node (§ 4.1 Archive model), so unix-mode never includes the file type bits (S_IFMT).

unix-uid and unix-gid are the numeric user and group IDs of the owner, and unix-uname and unix-gname are the names of that user and group, as uname and gname of tar. An entry MAY have a name without the ID, and vice versa.

Unix metadata from ZIP is discussed in § 16.4 Unix metadata from ZIP.

11.10. macOS-specific metadata

aqr-entry-macos = (
  ? &(mac-flags: -80) => uint
)

mac-flags holds the file flags of macOS (st_flags, as set by chflags), limited to the following bits; any other bit MUST be zero.

Bit Flag
0x1 UF_NODUMP
0x2 UF_IMMUTABLE
0x4 UF_APPEND
0x8 UF_OPAQUE
0x8000 UF_HIDDEN
0x10000 SF_ARCHIVED
0x20000 SF_IMMUTABLE
0x40000 SF_APPEND
0x100000 SF_NOUNLINK
The other flags tell how the file system stores the file (UF_COMPRESSED, SF_DATALESS), its state on the volume (UF_TRACKED), or the protection that the system itself sets (UF_DATAVAULT, SF_RESTRICTED, SF_FIRMLINK). Flags beginning with SF_ can only be changed by the superuser.

11.11. zip-extras: ZIP extra fields

zip-extras preserves extra fields [APPNOTE] of the ZIP entry that the entry was converted from, when they carry information about the file that has no other representation in this specification.

aqr-entry-zip = (
  ? &(zip-extras: -100) => aqr-zip-extras
)

aqr-zip-extras = { * aqr-zip-extra-key => bstr .size (0..65531) }
aqr-zip-extra-key =
  0..65535 /   ; Header ID n in the local header
  -65536..-1   ; Header ID n in the central header, as -1 - n
; TODO: multiple extra fields with the same header ID

A key n from 0 to 65535 denotes the extra field with the header ID n in the local header, and a key −1 − n from −65536 to −1 denotes the extra field with the header ID n in the central header. The value is the data of the extra field, excluding its 4-byte header of the header ID and the data size. Therefore a value with a non-negative key is always in the format for the local header, and a value with a negative key is always in the format for the central header.

A compressor converting a ZIP archive MAY put any extra field of an entry into zip-extras, subject to the following requirements:

Extra fields known to be subject to these requirements are listed in § 16.5 Extra fields not to be preserved. Any other extra field, including one with an unregistered header ID, can be put into zip-extras as is.

zip-extras is never authoritative. If information in a preserved extra field disagrees with the archive model or any other key of aqr-entry-extra, the latter MUST be used. A decompressor finding an extra field that a compressor is not allowed to put SHOULD ignore that field only, and SHOULD NOT treat the whole archive as invalid.

zip-extras is designed for a one-way conversion from ZIP; reconstructing the original ZIP archive from the converted archive is not a goal. Extra fields are preserved so that information without its own key yet, e.g. platform-specific attributes, access control lists and resource forks, can be recovered from already converted archives once a dedicated key is defined. Unknown extra fields are preserved for the same reason, even though some of them might be only meaningful to the ZIP container. The local and central header versions of the same header ID often differ in their formats, e.g. the central header version of 0x5455 only has the modification time, and that of many Info-ZIP attribute blocks only has a summary. Distinguishing them by keys lets a consumer interpret each value in the correct format. The key for the central header version is −1 − n instead of −n, so that header ID 0 doesn’t collide with the local header version, and both versions have the same encoded length in CBOR.

12. Extraction

Writing an archive value to a file system is outside the archive model. A decompressor doing so is subject to the following requirements:

The order of applying the metadata of a node matters on common platforms. Changing the owner can clear the set-user-ID and set-group-ID bits, so the owner goes before unix-mode. Forks and extended attributes can’t be written once the permissions forbid writing, so they also go before unix-mode. Writing the content or the children of a directory changes its modification time, so the timestamps of a directory go after all its children. Immutable and append-only flags forbid all of the above, so they go last. Setting mtime earlier than crtime moves crtime back on macOS, so crtime goes after mtime.
The archive model can’t express a path through a link node (§ 4.1 Archive model), but a link created during the extraction can still redirect a later write, e.g. when a link A and a file a/x map to the same directory on a case-insensitive file system. The rule on conflicting names above already prevents this, and the first requirement makes it explicit. Possible implementations include creating all links after all other nodes, or opening each path relative to its parent directory without following links (e.g. openat with O_NOFOLLOW, or RESOLVE_BENEATH on Linux).

A decompressor extracting a link node is subject to the following additional requirements:

A decompressor MAY create hard links among file nodes with the same consistent link group. It MUST NOT create hard links among any other file nodes, even when they have the same content.

Hard links let extracted files share the disk space, and changes made to one of them, which is crucial for e.g. snapshot backups made with rsync --link-dest. But files that were independent have to remain independent.

A decompressor SHOULD NOT create a FIFO or a device file for a special node unless explicitly requested, as with ownership information. A special node not created for this reason, or because the platform can’t represent it (e.g. the lack of privilege, or a device number out of the platform’s range), MUST be reported to the user.

Creating a device file usually needs the superuser privilege, and a device file with lax permissions created by such an extraction can expose the entire disk. FIFOs are harmless by themselves, but are treated the same for simplicity.

A decompressor MUST NOT create anything for an unknown node, and MUST report it to the user. It MAY continue extracting other nodes.

13. Conformance

As well as sections marked as non-normative, all diagrams, examples and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MAY, MUST, MUST NOT, RECOMMENDED, SHOULD and SHOULD NOT in this specification are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

13.1. Conformance levels

A conforming decompressor has a conformance level, one of the following according to the modules (§ 7 Modules) it applies:

Level 0

Applies no module.

Level 1

Applies every registered module, i.e. every module whose module hash is the registered hash of an entry of the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry), and no other module.

Level 2

Applies every module.

A decompressor at level 0 or level 1 MAY refuse to apply a module that its level doesn’t include, in which case a resource failure occurs. A decompressor at level 2 MUST NOT refuse to apply a module, though it may still fail with a resource failure when a module exceeds its limits. A level doesn’t change any other requirement on a decompressor, e.g. a decompressor at level 0 still reads formula blocks without Apply instructions.

A decompressor at level 0 or level 1 never runs untrusted code. A module registered after a decompressor at level 1 was made is not known to it, which results in an unsupported failure if the module is given by a shorthand block and in a resource failure otherwise, until the decompressor is updated. See § 7.1 Module block for how each level can apply its modules.

13.2. Partial access

A decompressor MAY compute only a part of the result of interpreting an archive, such as a subset of entries, a byte range of a content, or entry names without contents. Such a partial result MUST be equal to the corresponding part of the full result, if the full result is an archive value.

A decompressor computing a partial result is not required to detect an invalid archive failure caused only by parts it doesn’t compute. It SHOULD verify every checksum covering only the bytes it has computed.

For example, a decompressor reading a byte range of a file defined by a formula block only reads the non-leaf blocks covering that range, rather than its entire parcel. It can’t verify the output checksum of the formula block, but can verify the checksums of non-leaf blocks it has read. A decompressor looking up a single path has to consider every segment that may replace it, which is possible by reading metadata blocks backward from metadata_backward_offset until the path or one of its ancestors is found as a non-directory entry in a segment. If that entry is the path itself and not a tombstone, it gives the node at the path; otherwise, i.e. if it is a tombstone or an ancestor, the path doesn’t exist. If no segment has such an entry, the path doesn’t exist in a standalone archive, and is looked up in the base of an overlay archive. Likewise, reading a file using imports only needs the imported byte ranges, and an internal import is resolved by the same lookup from the preceding segment.

14. Security considerations

An archive is a container for arbitrary data and doesn’t by itself contain active content, except for WebAssembly modules (§ 7.4 Module execution) that a decompressor executes to reconstruct file contents. Such modules are run under a deterministic profile without access to anything but their inputs and outputs, but a decompressor still has to treat them as untrusted code and isolate them accordingly, unless it knows their module hash (§ 7.1 Module block). Formula blocks and modules can produce outputs far larger than the archive itself, and some fields (e.g. lg_work_factor) directly control the amount of computation. A decompressor is expected to enforce implementation-defined limits and fail with a resource failure when they are exceeded. A decompressor writing an archive to a file system is subject to the requirements in § 12 Extraction, in particular to prevent writing outside the destination directory. An unencrypted archive provides no confidentiality, and its XXH64 checksums (§ 10.1 Checksums) only detect accidental corruption, not intentional modification. An encrypted archive (§ 9 Encrypted archives) provides confidentiality and integrity through [AGE], which is only as strong as the passphrase used. The metadata of an archive, including entry names, timestamps and ownership, may reveal information about the system that created it.

An overlay archive is interpreted against a base, typically a directory on the file system, and can copy any part of the base into its result. A decompressor reading a base is subject to the requirements in § 8.3 Reading a base, in particular to use a base only when explicitly given one and to never read outside it. Applying an overlay archive in place modifies and removes files in the base directory.

15. IANA Considerations

15.1. application/aqr Media Type

This document requests the registration of a new media type in the Media Types Registry ([RFC6838], section 5.6) as follows:

Type name
application
Subtype name
aqr
Required parameters
N/A
Optional parameters
N/A
Encoding considerations
binary
Security considerations
See § 14 Security considerations.
Interoperability considerations

An unencrypted archive may be preceded and followed by arbitrary bytes, e.g. a self-extracting executable stub; its extent is determined by header_magic and the footer (§ 3.6 Locating an archive). An archive without any metadata block decompresses into a single unnamed file rather than a directory tree. An overlay archive has its own media type application/aqo (§ 15.2 application/aqo Media Type); an encrypted archive whose payload is an overlay archive is also of that type.

Published specification
This document.
Applications that use this media type
Archivers and compressors that implement the Aqueous archive format.
Fragment identifier considerations
N/A
Additional information
Deprecated alias names for this type
N/A
Magic number(s)
F8 04 03 6A D6 BE (hexadecimal; HEADER_MAGIC) for an unencrypted archive, possibly preceded by other data; F9 CA F2 A6 D0 2A (hexadecimal; the common prefix of ENCRYPTED_HEADER_MAGIC and ENCRYPTED_MIRRORED_HEADER_MAGIC) at offset 0 for an encrypted archive.
File extension(s)
.aqr
When a single file is compressed into an archive without any metadata block, the extension may be appended to the original file name instead (e.g. example.tar.aqr).
Macintosh file type code(s)
N/A
Person & email address to contact for further information
Kang Seonghoon <public+aqr@mearie.org>
Intended usage
COMMON
Restrictions on usage
N/A
Author
Kang Seonghoon
Change controller
Kang Seonghoon

15.2. application/aqo Media Type

This document requests the registration of a new media type in the Media Types Registry ([RFC6838], section 5.6) as follows:

Type name
application
Subtype name
aqo
Required parameters
N/A
Optional parameters
N/A
Encoding considerations
binary
Security considerations
See § 14 Security considerations.
Interoperability considerations

An overlay archive can’t be interpreted without its base. Its magic numbers differ from those of application/aqr, so that a decompressor not supporting overlay archives doesn’t mistake one for a standalone archive. The interoperability considerations for application/aqr otherwise apply.

Published specification
This document.
Applications that use this media type
Archivers, patching tools and updaters that implement the Aqueous overlay archive format.
Fragment identifier considerations
N/A
Additional information
Deprecated alias names for this type
N/A
Magic number(s)
69 0D 15 AF F6 B2 (hexadecimal; OVERLAY_HEADER_MAGIC) for an unencrypted overlay archive, possibly preceded by other data. An encrypted overlay archive has the same magic numbers as an encrypted archive of application/aqr.
File extension(s)
.aqo
Macintosh file type code(s)
N/A
Person & email address to contact for further information
Kang Seonghoon <public+aqr@mearie.org>
Intended usage
COMMON
Restrictions on usage
N/A
Author
Kang Seonghoon
Change controller
Kang Seonghoon

15.3. +aqr Structured Syntax Suffix

This document requests the registration of a new structured syntax suffix in the Structured Syntax Suffix Registry ([RFC6838], section 6.2) as follows:

Name
Aqueous Archive Format
+suffix
+aqr
References
This document.
Encoding considerations
binary
Interoperability considerations

A media type with the +aqr suffix denotes a standalone archive, either unencrypted or encrypted, whose archive value is given a more specific meaning by that media type. A generic Aqueous decompressor can interpret an archive of such media type without knowing that meaning. The interoperability considerations for application/aqr (§ 15.1 application/aqr Media Type) also apply.

Fragment identifier considerations

The syntax and semantics of fragment identifiers specified for +aqr SHOULD be as specified for application/aqr. (At publication of this document, there is no fragment identification syntax defined for application/aqr.)

The syntax and semantics for fragment identifiers for a specific xxx/yyy+aqr SHOULD be processed as follows:

  • For cases defined in +aqr, where the fragment identifier resolves per the +aqr rules, then process as specified in +aqr.
  • For cases defined in +aqr, where the fragment identifier does not resolve per the +aqr rules, then process as specified in xxx/yyy+aqr.
  • For cases not defined in +aqr, then process as specified in xxx/yyy+aqr.
Security considerations
The security considerations in § 14 Security considerations apply to any media type with the +aqr suffix. Each such media type may have additional security considerations specific to its interpretation of the archive.
Contact
Kang Seonghoon <public+aqr@mearie.org>
Author/Change controller
Kang Seonghoon

15.4. Quantization Interval

This document requests the registration of a new key for the Time Tag Map Keys Registry ([RFC9581], section 7.3) as follows:

Key Name
Quantization Interval
Key Value
-14 (tentative)
Reference
The Quantization Interval section in this document.
Semantics

A quantization interval specifies the interval in which the true time lies, relative to the time value given. That is, if the time value is t and the quantization interval is from a to b, the true time lies between t + a and t + b inclusive. The quantization is any means to truncate or round the true time value to a discrete representation, and notably, is not related to the accuracy of the underlying clock. For example, it is possible that the clock has a high precision but the time values are truncated to the last second, in which case the quantization interval would be [0, 1 second].

The interval is always closed. It may be slightly wider than the exact set of possible true times (e.g. [0, 1 second] for truncation, where the true time can never be exactly 1 second later), but it never excludes any possible true time, including ties in rounding.

A quantization interval can be given in terms of:

  • A numeric value in seconds or a duration y, indicating the interval of [0, y];
  • An array of two elements [x, y] each of which is either a numeric value in seconds or a duration, indicating the interval of [−x, y]. That is, x is the extent before the time value and y is the extent after the time value.

Every numeric value MUST be finite and non-negative, and every duration MUST have a non-negative key 1, so that the time value given is always contained in the interval. The interval MUST NOT be a single point. Therefore a single numeric value or duration MUST NOT be zero, and an array [x, y] MUST NOT have both x and y zero. If x is zero, the array MUST NOT be used and y MUST be given alone instead.

If an extended time doesn’t contain this key, its quantization interval is unknown; in particular, the time value is not implied to be exact.

For example, the first two of the following represent a time value truncated to 10 milliseconds, and the last one represents a time value rounded to the nearest second:

1001({1: 1697724754, -3: 870, -14: 0.01}),
1001({1: 1697724754, -3: 870, -14: {1: 0, -3: 10}}),
1001({1: 1697724754, -14: [0.5, 0.5]})

The following CDDL declarations can be used to represent the Quantization Interval key:

quantization-interval = ~time / ~duration / [ 2*2 ~time/~duration ]
$$ETIME-ELECTIVE //= (-14: quantization-interval)

15.5. Aqueous Dictionaries Registry

This document requests IANA to create a new registry titled "Aqueous Dictionaries", listing the registered dictionaries that a shorthand block of type 5016 can refer to (§ 4.2.2 Registered dictionaries and modules). New entries are assigned with the Specification Required policy ([RFC8126], section 4.6). Each entry has the following fields:

Hash
The registered hash, i.e. the SHA-512/256 hash [FIPS180-4] of the dictionary’s bytes, written as 64 lowercase hexadecimal digits.
Name
A short name of the dictionary for humans, which has no effect on the format.
Entropy Tables
Whether the dictionary has entropy tables, i.e. the value of has_entropy_tables.
Data
The bytes of the dictionary, being the decoded data of a dictionary block (§ 5.2 Dictionary block), which MUST be valid as such and deposited with the registration.
Reference
The document describing the dictionary and its intended use.

The designated expert verifies that the hash matches the data and that no entry has the same hash. An entry, once registered, MUST NOT be modified or removed, since archives refer to it only by its hash. An entry is never shared with the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry), even when both have the same data.

The initial contents of the registry are as follows:

Hash Name Entropy Tables Reference
TBD text TBD This document
TBD modules TBD This document

The dictionary "text" is intended for general text, and the dictionary "modules" is intended for module blocks given in full (§ 7.1 Module block).

The data and the hashes of the initial dictionaries are yet to be made.

15.6. Aqueous Modules Registry

This document requests IANA to create a new registry titled "Aqueous Modules", listing the registered modules that a shorthand block of type 6016 can refer to (§ 4.2.2 Registered dictionaries and modules). New entries are assigned with the Specification Required policy ([RFC8126], section 4.6). Each entry has the following fields:

Hash
The registered hash, i.e. the module hash of the module, written as 64 lowercase hexadecimal digits.
Name
A short name of the module for humans, which has no effect on the format, written as the name of the module and its version joined by @ (e.g. flate-set@1). Different versions of a module have the same name before @ and different hashes.
Data
The bytes of the module, being wasm_module of a module block (§ 7.1 Module block), which MUST be valid as such and deposited with the registration.
Reference
The document describing the transformation that the module reverses, its inputs, resources and parameters, and preferably the source code from which the module was built.

The designated expert verifies that the hash matches the data, that no entry has the same hash, and that the module conforms to § 7.4 Module execution. An entry, once registered, MUST NOT be modified or removed, since archives refer to it only by its hash. An entry is never shared with the Aqueous Dictionaries Registry (§ 15.5 Aqueous Dictionaries Registry), even when both have the same data.

The initial contents of the registry are the modules built by the reference implementation of this document, as follows, linking the data (D) and the reference (R) of each entry:

Hash Name Links
6b0b52c29cf4c42c206a4fed017ac64ab76e0b320fb85c92e0c0be10829cc034 ac3-recons@1 D R
0099d929fbc194ff15c8ec271bcd64d822671e780ae7cea046f4ec6c0410fcb9 arsc-recons@1 D R
9e1f1f668a435d34f614f98ea502e00469e15062db1f06e22be60155ea439787 asar-recons@1 D R
fea578bce88fa2c9441388f8add12b2cb79bedf9f40ca4f676561786db0e52fe astc-recons@1 D R
3810380b147efa2fb94a8d32deb9464b9202c831e68e941f2e0ab48acc17a2b9 azo-recons@1 D R
e33ddb74d885e215d86f56a52d2845ddc4ef9ef12090ee8c920bc205423d4a09 b2t-recons-quopri@1 D R
c586cd2eb9ea405891552c5b2ef82d6ec1c4263c3df041744f419220dc3f0599 b2t-recons@1 D R
292160510dc876b2454bd65d404685858012e202c75f22d5d5fb606adf942832 bc6h-recons@1 D R
eeb736245a8e13e936c6f5e0dcae855a3f8c8de09dc618f6a2fd06ee35dbbd3c bc7-recons@1 D R
8f3fd3dbf3966a8ecb590084559a4f93bdde14b6a9a50ad4f313e04bffc07dd9 bcj-arm@2 D R
3e41f707a7655fa019757b913785a462b53e109e6257030e8c279a5156f9dc80 bcj-arm64@2 D R
c073da28c681d39cb1acbe0c90ecafb8cf81baf86b2b916ab1fd251c1e787bde bcj-armthumb@2 D R
2d07cd639eadb1f671367f73a37ef64270dc4e1b324604832ea22cb5194bb9c2 bcj-delta@1 D R
8c7843bcff41125e0bca042bd2ca518cfbf6641114c287318a688b71d7370f04 bcj-ia64@2 D R
0c153b210a4aecf9edff3d3f828beb61726857d58d231025bd78a838983db0b8 bcj-powerpc@2 D R
6f99ba289eb005969ed4292d66da6e630b8ea3de3906ac634a68e8ba9e7f71b1 bcj-riscv@2 D R
908cb8d3f0802b9d6aeb1cc9770414945be8534118f4d60095c4450d4521630f bcj-sparc@2 D R
e844d7465f87c75072153637f3af2923608688c6b281ceea6a46684a251bc19c bcj-swap2@1 D R
333a88f38b91e85050069df2537d7d127cfc531ed2e56c90d24a88b028915df3 bcj-swap4@1 D R
bc5de18653206afc1c83a8179acbb9b04463cf4e62b23269b5d6fcd56f4557a7 bcj-x86@2 D R
4c223c70286d04e9b4c63344706de56a40dd85d3f21bec10cca3ebf49f8f998a bcj2@2 D R
fc0c88b21c5b9be96204333446e41beb5dfc5c6a65a014eae2c35b6f69e82ca9 bcn-recons@1 D R
bc7e3751cff6f7912ff2d3d8f026b04be2b7210087384e7e2ae40b4fbcd6d29e bundle-recons@2 D R
76b6880cf027967acad5ad7434de59a100adfd6b17269b2bb2648d6ff2bb1b45 bvx-recons@2 D R
4fb20ee608bbe840fe3d943fc73279f3283994f591a1f1911b730c89dfa3e554 bz2-alz-recons@2 D R
05b7063a2d3c99f212239d6d323124540eecb39f8f152a3d61848ed548e409f5 bz2-recons@2 D R
f192f3cdfac2ab652448f1ade2a3465600796ff41ed73ef212ed9781744d6b39 clip-recons@3 D R
af79fb4e146afc56f035ee9265b571db81cf3c0847dd42816b71196a13fd8785 dex-recons@2 D R
020628e8ed27f952b018beb0e4413c04a3566bfede4cd21261d55e426d4db77a dh-lods@1 D R
d717646aa3c6c2cf08556b979e86574d26d1dd9e7814ef96a709d781ce6ff9e8 div3-recons@3 D R
ab66f2ea4aac7cda80980ff50e21496bd34915c9553c597d1c43af4a98fb1262 dwarf-apple@1 D R
35289b7ef1d0d75f99c3948cddb76e4e0f9304d592d2fb40e5830fc4516c12a4 dwarf-info@2 D R
9ad20aa3412e699e66cc296cdfd48dda49b86e11cd771cb3f4384e2fd2bcf869 etc-recons@1 D R
dbd3de347ffd906cc61c0b1a1fb685e7f1b47c71d2833e00e0c41e16599be597 exe-recons/arm64-delta@1 D R
465a349b04736a5d34715cd5217ce5e8665feac7905b245eaeb736dad856d291 exe-recons/arm64-remap@1 D R
78cf95749c7f980cd31edf253aa36afb8bc753cf530eb226fd1a3c5c86e80ae3 exe-recons/arm64@2 D R
be9df7c3bc994a855d72983b81d5d48d567a82a1f3f4336438dd72962c8f9afd exe-recons/cil@1 D R
85de1c244f9ca7ae68395320117abebe8825a750c063d0e6cf91555a885f3899 exe-recons/macho-codesign@2 D R
d256bcb992b46b1de858ace1e256889192beb6627ee1514b16f23390c70df4cf exe-recons/macho-symtab@1 D R
b3236ff145458aa4a53efe64fdbabf544e8d4b282aa94188c58cdfb41635bfcd exe-recons/pe-reloc@3 D R
a8d092a4eee443f92f6a14537ef0e1637300105516106fa860bdecbf51868aad exe-recons/x86-remap@2 D R
1ecc85b704093e11c8daa5024468bafb4d4496fc52e3ecf016cfd10af0c51581 exe-recons/x86@6 D R
3e24ee2f6af1d4a1aeab02a6ff46ed13f2ca7d94c8ce806e6de8bbd93d21b657 fasta-recons@3 D R
20a656984d70844d4658cb62af20d2437d100e2e174e8ac4bb58a6301f4a5a27 fits-recons-image@1 D R
5ccaa3f93e4296657a8a325e839804777fe8976d7f301859b079109d3bb6e622 fits-recons-rice@1 D R
aa24b0e80a0856cdc07a600f5023252f7578c66e94c0ac355efd434f85ecd49e fits-recons-table@1 D R
ff919a0955fc9353059af53847f1f84b803b4115e6812fc51111eda80a2599fc flac-recons@5 D R
8f636ea534d9114fac5504a9922d4dcc999e8a660dc9ecc2885f5459fd10e813 flate-inflate@1 D R
e7f26ac070c92a66a8b3cba961051b8e2703c5726bea71760dc5bb7c462b4cd6 flate-set@4 D R
8f53a39d83da2c850989320f0f64709ceaec9d8d4046c03258cf776a9b496288 gif-recons@2 D R
d7cd04da761d4b79f181894a2c67f24f857762afc415beb907cfc7c234533ab2 git-pack-index-recons@2 D R
b7ae16ba012c44fd9881c4d01bf3059aecec77c9b72d9690e87babf9c503a563 git-pack-recons@2 D R
c14b56b9abe1f911ce22d4092244484cac068e3796d838afedb5a72b071d5aeb gnu-gettext-recons@1 D R
9234acad545b70a11c46bdc56e27be21a4c0314d20ecb23566d82be07a7cca04 hdf5-recons-float@1 D R
ebd2abd8e92185a1c0fbaef7785daa90582f39c4c779d2de3672a9c896a60879 ima-recons@2 D R
557f820cce081587adcfa218a3a9de9e0b20bd7cf0aa37735fc4e92fbe43c175 jpeg-lossless-recons@1 D R
bc399e90ff8a4789a500aeb395991029b742fb6d419426a6393988067f188838 jpeg-recons@3 D R
d1379510a62a8e5b41b4e394aa1ff0a7699a48fe7f0470d56e6a7a7bd631f7d8 jsonl-recons@2 D R
e745c2292175052983f2ece3d4a529c4970436bc44341b500fc94a00a6ea5931 leveldb-recons@1 D R
28f2d22eaae6c7175e6e7d56e357b42be53b000679258542d29ea19c2052ab25 lha-recons@2 D R
5b602c335bfb01c9b5aa231c241922efcef2e8a00f30cd8ec87731c8ce0c6739 log-recons@1 D R
57fc6a5976a9a38769a7ad6b2482df4e3fbdaed065dd37207d3731f7bc99676e lz4-recons@2 D R
0a4aae3de201085372643b1d52ffc77cdfb377a7d5613d08223ff0989f5e86b4 lzma-decode@1 D R
3e8fc43baf255f73b9cb90c512eb5795634d2d31560c07ae03e0cebec61ba8f0 lzma-recons@4 D R
c579dbf1ad2c0f7a691421ba2bbd6f6212d3ede3636de21f9c3d33650115c891 lzma2-recons@3 D R
a5a74899b18ee623da7e6014f0642af223f9653f2aa407bb0b22730790d9d54a lznt1-recons@1 D R
a9a9c84319d717df983dd3b56586d7b9ddfbbd16d0c3a856c7d96a759586c1a1 lzo-recons@1 D R
2a357d92e82fa7dd6dc787f54fe6a49a8ff12c52a96dbda95ec296f0c5e245b3 lzx-recons@3 D R
216344f2356e88e2604fbd0db7d499ac9b5381777bf414c548f7b8622dfaacbd mca-recons@2 D R
28a35fb096a31414a7c1c794722a195e2c78d438adf4c7f1ea1cb353aeae7f8a mp3-recons@6 D R
7ed36e1fe53ea5337a3e0950f6e5a464f9984508acfaa3c378adf4fa363552f7 msadpcm-recons@1 D R
57fdb061a4e2bbc30c471c088608d528582ef0d4343b84b340e08e5f396d7352 mszip-recons@2 D R
5af417ac62b361b73251aa5b7a045ea05e2d70d475e18f308ab092264345d436 obj-recons@1 D R
192f4385f28ea6cfb2bebe9d012d8769e2a09cccd7e8d0192ba5430c233cb7fa ogg-recons@1 D R
2d4f76cfc71cacaf918b3f94dc59527b9b1b7303e0d69b4a97c98d3107af8ce3 pcm-recons@2 D R
a3008d37eae4a8bf1474f79975da0e834212a7e79cb98f5992553a8c6ce0130e pdb-msf@1 D R
52ce0b70658a83169d5329170d782a2e75ddab60ef2a2ca767cb6b84db20d981 pdb-symbols@2 D R
33bfc52d05afcbe85d7bf7b7b8c56a17cec6a5442240964ddf905cbeae3ff499 pdb-types@1 D R
fac37428ac2315cb88d97fe30191cf33aa3338abcbf56faf4245f1f8925db312 pdf-recons@3 D R
dbfd85f28b73603d8c58b72d67066c56cde661def7e63102aba976906e2fcb93 png-against@2 D R
5680af51169fb124b42513f39bddbe55b037a18fac3c4b83b18ac0da29e25c9e png-recons@5 D R
f4e9b4f65167e3e7160117cf654f5545d228b2bac9c8cdc2d2a2991ad25c2e95 png-set@3 D R
1f14c4d1f1436b106ccbf51e2e5aeaf89d5b0a321ed450fbf340118cd0cf6937 prores-recons@3 D R
aad61c9304a0294838a7b01cb8045fcf9c44d96364c1ac58b0ad0abf1dce4489 psd-recons@1 D R
160f55905d28d1842ef780b1781dfac3e2dbe8191eef4950dceb89c7320daac2 pvrtc-recons@1 D R
ddf12e6354a898fd2fded827c2d0d9aedc0b332292b15ccd3c4e3ba8d31e5fe6 rar3-recons@2 D R
f65035f8334e69b382b112edf92f28036f2d2072c37002f1b6168665113e61cf rar5-recons@2 D R
2e4858b242829592dbdcc037e32ad37c16a96cae44db0aa8dd38dca680370a4c raster-recons-16@1 D R
63773d1d5a107c856845be86dc2de897bfd61f9b1f4316a1cf0851082a610ca3 raster-recons@1 D R
28b6963b212164a89006f2c0bc6f53d4cd945d9f8a5f956d57d1219be010fc4f recs-entries@1 D R
4279aab3b66a45067ace10cad8f979f46b27ed73ae31624a5728c783e4aefa65 recs-fields@1 D R
dac6184b0cdfd1f3e46cf83c420549cf7af5e90c99770ab6865646766f3dc767 recs-floats@1 D R
59fbb34cc7d041590435682f30fe4edbb704769acb0213e0339417535b2f409f snappy-recons@3 D R
4b544507f2459700d9b1e2683f3366dc2230166e8c8461f4d166f4ec90f71347 sqlite-recons@2 D R
9b46a81e6b86836750b2adcf1babd36ec39dbb81f9a04872b992a7af0459d11c tensors-recons@2 D R
24d85c63a9bb175e5ab281e055832afc1037446abbdf529b77c818a3b4d41eba vorbis-recons@3 D R
7ca6b6cdddb3bce10ad90a975a7c92aef3fef7025976f24abb5686edf0805da9 wasm-recons@1 D R
9d5c184ff4736bf54e10966f57542fc74977c9fe2684f1668c05c4d303a59bae wrt-recons-hangul@2 D R
d7a2ad763a9c9ef48cfb8a35ba24b6654e03fa8fbe2879f50109750c8dacc8f8 wrt-recons@3 D R
bbd506258ef6650e59216eb1538ce1211cf93f5a5746ce1b640ed025ff91779f xpress-recons@1 D R
e24ecbe10de42e77f1f74d524333728f70b7473dace96014c142e1cdad8f2074 xsv-recons@2 D R
60b2944db28669697e24767170b5914cf04a7111bf84c7f044c76781b38238a1 zip-recons@6 D R
1d8087f725c1dd6847001053205a6fdd4d54f870feba0978e6c7bce7f54d1ecc zstd-frames@1 D R

The initial entries are not yet fixed. Their modules and the documents describing them may still change before the first version of this document, and an entry changed until then is replaced in this table, not added as a new version.

16. Converting from ZIP

This section is non-normative.

This section describes how a compressor can convert entries of a ZIP archive [APPNOTE] into an archive value without losing information, where ZIP offers multiple sources or encodings for the same information.

16.1. Entry names from ZIP

A ZIP archive [APPNOTE] stores the path of each entry as a byte sequence in the local and central headers, with components separated by /. Its encoding is not always known from the header alone, and several extra fields carry the Unicode version of the same path. A compressor converting a ZIP archive can determine the encoding of each component from the following sources, where earlier sources are preferred:

  1. Extra field 0x7075 (Info-ZIP Unicode Path) of version 1 [INFOZIP-EXTRAFLD], whose NameCRC32 matches the CRC-32 of the path in the header. The field holds the path in UTF-8. A field with a mismatching CRC-32 is ignored as specified by [APPNOTE], as it may have been left over by a tool that changed the path without updating the field.

  2. The path in the header, if bit 11 (language encoding flag) of the general purpose bit flag is set. The path is in UTF-8.

  3. Extra field 0x554e (Xceed Unicode), which holds the path in UTF-16LE. It is converted to WTF-8, which preserves any lone surrogate.

  4. Extra field 0x5a4c (ZipArchive code page), which holds the path encoded in the code page given in the same field. If the field only has the code page, the path in the header is decoded with it instead.

  5. Extra field 0xe57a (ALZip code page), which gives the code page of the path in the header.

  6. Otherwise, the encoding of the path in the header is not specified. It is traditionally IBM code page 437, but often the system code page of the machine that made the archive, so the compressor has to guess the encoding. If it can’t, the raw byte sequence (prefixed with C116) is used for each component.

When the name comes from any source other than the last one, the original bytes of a path not in UTF-8 are not preserved. They only reflect how the ZIP container encodes the name, while the name of the file itself is the decoded Unicode string.

16.2. Timestamps from ZIP

A ZIP archive [APPNOTE] can carry the same timestamp in multiple places: the DOS date and time in the local and central headers, and extra fields 0x000a (NTFS), 0x5455 (extended timestamp), 0x5855 (Info-ZIP UNIX type 1) and 0x000d (UNIX) [INFOZIP-EXTRAFLD]. A compressor converting a ZIP archive can choose the source of each key as follows, where earlier sources are preferred. The order prefers more precise sources first, and then more recent specifications for the same precision.

Key Sources
crtime 0x000a Ctime; 0x5455 CrTime (local header only)
ctime -
mtime 0x000a Mtime; 0x5455 ModTime; 0x5855 or 0x000d ModTime; DOS date and time
atime 0x000a Atime; 0x5455 AcTime (local header only); 0x5855 or 0x000d AcTime
arctime -

Each source is encoded as follows:

When both the DOS date and time and a UTC source are present for mtime, their difference reveals the time zone offset where the archive was made. If the DOS time minus the UTC time differs from 900k seconds by less than 2 seconds, where k is an integer in the range of aqr-tz-offset, mtime can be encoded as [u, k] where u is the encoding of the UTC source. Otherwise, e.g. due to a clock error or a daylight saving time transition, the DOS date and time is simply discarded.

If two sources for the same key disagree by more than their quantization intervals, a compressor can warn about it, and still uses the preferred source.

16.3. Comments from ZIP

A ZIP archive [APPNOTE] stores a comment for each entry in the central header only, and a comment for the whole archive in the end of central directory record. Both are byte sequences, and an empty comment is no different from the absence of one, so the compressor can omit comment or archive-comment for an empty comment.

A compressor converting a ZIP archive can determine the encoding of each entry comment from the following sources, where earlier sources are preferred:

  1. Extra field 0x6375 (Info-ZIP Unicode Comment) of version 1 in the central header [INFOZIP-EXTRAFLD], whose ComCRC32 matches the CRC-32 of the comment in the header. The field holds the comment in UTF-8. A field with a mismatching CRC-32 is ignored, as for 0x7075 in § 16.1 Entry names from ZIP.

  2. The comment in the header, if bit 11 (language encoding flag) of the general purpose bit flag is set. The comment is in UTF-8.

  3. Extra field 0x554e (Xceed Unicode) in the central header, which holds the comment in UTF-16LE. It is converted to WTF-8, which preserves any lone surrogate.

  4. Otherwise, the encoding of the comment is not specified. The compressor can decode it with the encoding guessed for entry names in the same archive. If it can’t, the raw byte sequence is used as a bstr.

The archive comment has neither the language encoding flag nor an extra field, so its encoding is never specified by the archive itself. The compressor can decode it as in the last source above, and puts the result into archive-comment.

Archive comments are often used for purposes other than human-readable text, e.g. a self-extracting archive may store its configuration there, so decoding one with a guessed encoding can be lossy. A compressor unsure about the encoding can keep the raw byte sequence instead.

16.4. Unix metadata from ZIP

A ZIP archive [APPNOTE] can carry the Unix metadata of an entry in the external file attributes of the central header and in extra fields 0x7875 (Info-ZIP UNIX type 3), 0x7855 (Info-ZIP UNIX type 2), 0x756e (ASi UNIX), 0x5855 (Info-ZIP UNIX type 1) and 0x000d (UNIX) [INFOZIP-EXTRAFLD]. A compressor converting a ZIP archive can choose the source of each key as follows, where earlier sources are preferred and sources in the same item have the same precedence.

Key Sources
unix-mode External file attributes; 0x756e Mode
unix-uid 0x7875 UID; 0x7855 UID (local header only); 0x756e, 0x5855 (local header only) or 0x000d UID
unix-gid 0x7875 GID; 0x7855 GID (local header only); 0x756e, 0x5855 (local header only) or 0x000d GID

Each source is encoded as follows:

If two sources for the same key disagree, a compressor can warn about it, and still uses the preferred source.

16.5. Extra fields not to be preserved

The following extra fields [APPNOTE] [INFOZIP-EXTRAFLD] are not put into zip-extras, because their information is either absorbed into the archive model and other keys, or discarded as only meaningful to the ZIP container.

Header ID Name Treatment
0x0001 Zip64 extended information Discarded: sizes and offsets
0x0007 AV Info Discarded: signature
0x000a NTFS Absorbed: timestamps (§ 16.2 Timestamps from ZIP)
0x000d UNIX (PKWARE) Absorbed: timestamps (§ 16.2 Timestamps from ZIP), unix-uid and unix-gid (§ 16.4 Unix metadata from ZIP), the target of a link node, link-group for a hard link, and the device number of a special node
0x0014 PKCS#7 Store for X.509 Certificates Discarded: signature
0x0015 X.509 Certificate ID and Signature for individual file Discarded: signature
0x0016 X.509 Certificate ID for Central Directory Discarded: signature
0x0017 Strong Encryption Header Discarded: encryption
0x0019 PKCS#7 Encryption Recipient Certificate List Discarded: encryption
0x0021 Policy Decryption Key Record Discarded: encryption
0x0022 Smartcrypt Key Provider Record Discarded: encryption
0x0023 Smartcrypt Policy Key Data Record Discarded: encryption
0x10c5 minizip CMS signature Discarded: signature
0x1986 Pixar USD header ID Discarded: padding for alignment
0x5455 Extended timestamp Absorbed: timestamps (§ 16.2 Timestamps from ZIP)
0x554e Xceed Unicode Absorbed: entry name (§ 16.1 Entry names from ZIP) and comment (§ 16.3 Comments from ZIP)
0x5855 Info-ZIP UNIX type 1 Absorbed: timestamps (§ 16.2 Timestamps from ZIP), unix-uid and unix-gid (§ 16.4 Unix metadata from ZIP)
0x5a4c ZipArchive code page Absorbed: entry name (§ 16.1 Entry names from ZIP)
0x5a4d ZipArchive offsets array Discarded: offsets
0x6375 Info-ZIP Unicode Comment Absorbed: comment (§ 16.3 Comments from ZIP)
0x6c78 libarchive "xl" Absorbed: a copy of the central header fields
0x7075 Info-ZIP Unicode Path Absorbed: entry name (§ 16.1 Entry names from ZIP)
0x7855 Info-ZIP UNIX type 2 Absorbed: unix-uid and unix-gid (§ 16.4 Unix metadata from ZIP)
0x7875 Info-ZIP UNIX type 3 Absorbed: unix-uid and unix-gid (§ 16.4 Unix metadata from ZIP), for version 1 only
0x9901 AE-x (WinZip AES) Discarded: encryption
0x9903 WinZip Reference Discarded: reference to another entry, resolved by conversion
0xa11e Data Stream Alignment (Apache Commons Compress) Discarded: padding for alignment
0xa220 Microsoft Open Packaging Growth Hint Discarded: padding for in-place editing
0xcafe Java JAR marker Discarded: information about the ZIP file itself
0xcdcd minizip Central Directory Discarded: encryption and signature of the central directory
0xd935 Android ZIP Alignment Discarded: padding for alignment
0xe57a ALZip code page Absorbed: entry name (§ 16.1 Entry names from ZIP), if decoded with that code page
Some extra fields are preserved only in particular cases. The local and central header versions are considered separately.

Index

Terms defined by this specification

Terms defined by reference

References

Normative References

[AGE]
Filippo Valsorda. age. Version 1.1.0. URL: https://c2sp.org/age@v1.1.0
[FIPS180-4]
Secure Hash Standard (SHS). FIPS PUB 180-4. URL: https://csrc.nist.gov/pubs/fips/180-4/upd1/final
[INFRA]
Anne van Kesteren; Domenic Denicola. Infra Standard. Living Standard. URL: https://infra.spec.whatwg.org/
[RFC2119]
Scott Bradner. Key words for use in RFCs to Indicate Requirement Levels. BCP 14. URL: https://www.rfc-editor.org/info/rfc2119
[RFC3629]
F. Yergeau. UTF-8, a transformation format of ISO 10646. November 2003. Internet Standard. URL: https://www.rfc-editor.org/info/rfc3629/
[RFC4648]
Simon Josefsson. The Base16, Base32, and Base64 Data Encodings. STD 51. URL: https://www.rfc-editor.org/info/rfc4648
[RFC6838]
N. Freed; J. Klensin; T. Hansen. Media Type Specifications and Registration Procedures. January 2013. Best Current Practice. URL: https://www.rfc-editor.org/info/rfc6838/
[RFC8174]
Barry Leiba. Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words. BCP 14. URL: https://www.rfc-editor.org/info/rfc8174
[RFC8439]
Y. Nir; A. Langley. ChaCha20 and Poly1305 for IETF Protocols. June 2018. Informational. URL: https://www.rfc-editor.org/info/rfc8439/
[RFC8610]
Henk Birkholz; Christoph Vigano; Carsten Bormann. Concise Data Definition Language (CDDL): A Notational Convention to Express Concise Binary Object Representation (CBOR) and JSON Data Structures. URL: https://www.rfc-editor.org/info/rfc8610
[RFC8878]
Yann Collet; Murray Kucherawy. Zstandard Compression and the 'application/zstd' Media Type. URL: https://www.rfc-editor.org/info/rfc8878
[RFC8943]
M. Jones; A. Nadalin; J. Richter. Concise Binary Object Representation (CBOR) Tags for Date. November 2020. Proposed Standard. URL: https://www.rfc-editor.org/info/rfc8943/
[RFC8949]
Carsten Bormann; Paul Hoffman. Concise Binary Object Representation (CBOR). STD 94. URL: https://www.rfc-editor.org/info/rfc8949
[RFC9581]
C. Bormann; B. Gamari; H. Birkholz. Concise Binary Object Representation (CBOR) Tags for Time, Duration, and Period. August 2024. Proposed Standard. URL: https://www.rfc-editor.org/info/rfc9581/
[WASM-CORE-3]
Andreas Rossberg. WebAssembly Core Specification. Release 3.0. URL: https://webassembly.github.io/spec/versions/core/WebAssembly-3.0.pdf
[WTF-8]
Simon Sapin. The WTF-8 encoding. URL: https://wtf-8.codeberg.page/
[XXHASH]
Yann Collet. xxHash fast digest algorithm. Version 0.2.0. URL: https://github.com/Cyan4973/xxHash/blob/v0.8.3/doc/xxhash_spec.md

Non-Normative References

[APPNOTE]
.ZIP File Format Specification. Version 6.3.10. URL: https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
[CBOR-NOTABLE-TAGS]
Carsten Bormann. Notable CBOR Tags. Internet-Draft. URL: https://datatracker.ietf.org/doc/draft-bormann-cbor-notable-tags/
[INFOZIP-EXTRAFLD]
Info-ZIP Extra Field Descriptions (proginfo/extrafld.txt in UnZip 6.0). URL: https://infozip.sourceforge.net/
[RFC8126]
M. Cotton; B. Leiba; T. Narten. Guidelines for Writing an IANA Considerations Section in RFCs. June 2017. Best Current Practice. URL: https://www.rfc-editor.org/info/rfc8126/

CDDL Index

aqr-opt-metadata = {
  aqr-archive-comment
  ? &(entry-extras: -2) => [* aqr-entry-extra / null]  ; Indexed by entry numbers, starting from the root directory
  aqr-segment-status
  aqr-journal

    * $$aqr-opt-metadata-member  ; For future extensions
}

$$aqr-opt-metadata-member = ()  ; No extension is defined by this specification

aqr-entry-extra = {
  aqr-entry-timestamps
  aqr-entry-comment
  aqr-entry-link-group
  aqr-entry-xattrs
  aqr-entry-forks
  aqr-entry-windows
  aqr-entry-unix
  aqr-entry-macos
  aqr-entry-zip

    * $$aqr-entry-extra-member  ; For future extensions
}

$$aqr-entry-extra-member = ()  ; No extension is defined by this specification

; Key -8 is reserved for access control lists.

aqr-entry-timestamps = (
  ? &(crtime: -1) => aqr-time
  ? &(ctime: -2) => aqr-time
  ? &(mtime: -3) => aqr-time
  ? &(atime: -4) => aqr-time
  ? &(arctime: -5) => aqr-time
)

aqr-time =
  $aqr-base-time /                          ; UTC timestamp, scaled and quantized
  [$aqr-base-time] /                        ; Local time in unknown time zone, encoded like UTC (*)
  [$aqr-instant, offset: aqr-tz-offset] /   ; UTC timestamp with time zone offset ($)
  etime /                                   ; Extended time defined in [[!RFC9581]]
  [etime]                                   ; Local time in unknown time zone, encoded like UTC

aqr-tz-offset = -48..56  ; In 15 minutes, positive east of UTC as in ISO 9660

$aqr-base-time  = #6.100(int)    ; 1d (FAT atime*), date as in [[!RFC8943]]
$aqr-base-time /= $aqr-instant  ; Can be given a time zone offset

$aqr-instant  = #6.121(int)  ; 2s (FAT mtime*, exFAT atime$, ZIP*, CAB*)
$aqr-instant /= int          ; 1s (~ext3, HFS/HFS+, tar, ISO 9660$)
$aqr-instant /= #6.122(int)  ; 10ms (FAT ctime*, ISO 9660$, exFAT$)
$aqr-instant /= #6.123(int)  ; 1ms
$aqr-instant /= #6.124(int)  ; 1us (UDF)
$aqr-instant /= #6.125(int)  ; 100ns (NTFS, RAR*, 7z)
$aqr-instant /= #6.126(int)  ; 1ns (ext4, ZFS, Btrfs, APFS)

aqr-archive-comment = (
  ? &(archive-comment: -1) => aqr-text
)

aqr-entry-comment = (
  ? &(comment: -6) => aqr-text
)

aqr-text =
  tstr /          ; Unicode
  #6.273(bstr) /  ; WTF-8, i.e. UTF-16 + surrogate compat
  bstr            ; In unknown encoding

aqr-segment-status = (
  ? &(segment-status: -3) => true / aqr-incomplete
)

aqr-incomplete = {
  ? &(files: 0) => uint  ; Number of files left out
  ? &(bytes: 1) => uint  ; Their total size in bytes
}

aqr-journal = (
  ? &(journal: -4) => {* aqr-journal-time => aqr-text}
)

aqr-journal-time = uint  ; Milliseconds since 1970-01-01T00:00:00Z, leap seconds not counted

aqr-entry-link-group = (
  ? &(link-group: -7) => int  ; File nodes only
)

aqr-entry-xattrs = (
  ? &(xattrs: -10) => { * aqr-text => bstr }
)

aqr-entry-forks = (
  ? &(forks: -50) => { * aqr-text => aqr-fork }
)

aqr-fork = [
  offset: aqr-seg-offset,
  output: uint,
  size: uint,
]

aqr-seg-offset = uint  ; Relative offset from the metadata block, as in [[#metadata-block]]

aqr-entry-windows = (
  ? &(win-link-type: -60) => &(file: 0, directory: 1, junction: 2)  ; Link nodes only
  ? &(win-attrs: -61) => uint
)

aqr-entry-unix = (
  ? &(unix-mode: -70) => uint .le 4095  ; Permission bits only (0o7777), without S_IFMT
  ? &(unix-uid: -71) => uint
  ? &(unix-gid: -72) => uint
  ? &(unix-uname: -73) => aqr-text
  ? &(unix-gname: -74) => aqr-text
)

aqr-entry-macos = (
  ? &(mac-flags: -80) => uint
)

aqr-entry-zip = (
  ? &(zip-extras: -100) => aqr-zip-extras
)

aqr-zip-extras = { * aqr-zip-extra-key => bstr .size (0..65531) }
aqr-zip-extra-key =
  0..65535 /   ; Header ID n in the local header
  -65536..-1   ; Header ID n in the central header, as -1 - n
; TODO: multiple extra fields with the same header ID

quantization-interval = ~time / ~duration / [ 2*2 ~time/~duration ]
$$ETIME-ELECTIVE //= (-14: quantization-interval)

Issues Index

A reference to a preceding block in the same segment, e.g. to patch a file stored earlier in the segment, is not yet defined. A reference to another file is made by an import (§ 8.1 Import block) rather than by an instruction, and the content of a file in preceding segments by an internal import. The candidate is a second list of non-leaf items preceding the formula block, read by Slice instructions as referenced items are, where the instructions of another formula block are copied when its output is needed, instead of referring to the output. It requires a bound on the distance of such a reference declared before the referenced blocks, so that a streaming decompressor knows which blocks to retain, and a way to locate a preceding block without reading the segment from its beginning. ↵
This section is tentative. ↵
Whether the guaranteed limit of 227 bytes (128 MiB) for the Zstandard Window_Size (§ 3.2.1 Zstandard compression method) should also apply to the memory limit. ↵
Whether the memories other than the working memory, read-only, should be visible to tasks, which would require rejecting the instructions writing them; and the windows of a part module in tasks. ↵
The interval of 64 encrypted chunks follows the tentative distance of 4 MiB in § 10.3 Mirror block and changes with it. ↵
The recommended distances are tentative: a larger one would also cover the loss of a flash erase block, pending the measurement of the overhead of mirror blocks. The interval between mirrored headers in an encrypted archive (§ 9 Encrypted archives) changes with the latter distance. ↵
The data and the hashes of the initial dictionaries are yet to be made. ↵
The initial entries are not yet fixed. Their modules and the documents describing them may still change before the first version of this document, and an entry changed until then is replaced in this table, not added as a new version. ↵