1. Introduction
This section is non-normative.
The Aqueous archive format stores a directory tree, or a single file, as a sequence of blocks. This section gives an overview of how the parts of the format fit together; each paragraph points to the chapter defining the part it describes.
An archive is a header, a sequence of blocks and a footer (§ 3 Container). Every block starts with its type and its size, so a decompressor can skip any block without understanding it, and the type tells whether the block can be ignored when it is not understood (§ 3.1.1 Block types). The footer points back to the last metadata block, so that a decompressor can start reading from the end of the archive.
Blocks are grouped into segments, each ending with a metadata block (§ 4 Segments and entries). The metadata block names the entries of the segment, i.e. directories, files, symbolic links and other nodes, and points each file to the block holding its content. A later segment adds, replaces and removes entries of earlier segments, so that an archive can be extended by appending a segment without rewriting what precedes it (§ 3.7 Appending to an archive). An archive without any metadata block simply decompresses into a single unnamed file, as a gzip file does.
The content of a file is given by a single leaf block. The simplest leaf block is a data block storing the content itself, optionally compressed with Zstandard and a dictionary shared by many small blocks (§ 5 Stored data). A formula block is a leaf block that describes the content instead, as a small stack machine program over non-leaf blocks following it (§ 6 Formulas). The program takes slices of them, concatenates them, patches them, and applies WebAssembly modules to them to undo a transformation made by the compressor (§ 7 Modules). A leaf block and the non-leaf blocks following it form a parcel, which is typically all that a decompressor reads to extract a single file, and the formula block serves as an index for random access within it.
An overlay archive describes a directory tree as changes to an existing one, called its base (§ 8 Overlay archives and imports). Import blocks read byte ranges of files in the base, which formulas can patch, so that an overlay archive works as an update or a delta of the base.
The whole archive may be encrypted with a passphrase using age [AGE] (§ 9 Encrypted archives). Checksums cover every block except, optionally, leaf data blocks, and except shorthand blocks, which name registered dictionaries and modules by their hashes (§ 4.2.2 Registered dictionaries and modules). Mirror blocks hold copies of the blocks whose damage would affect many files (§ 10 Integrity and recovery). Metadata that is not necessary to read the files, such as timestamps, comments and permissions, is encoded in CBOR and may be ignored by a decompressor (§ 11 Optional metadata, § 12 Extraction).
2. Conventions
2.1. Expressions
All expressions are interpreted according to the standard arithmetics, with the following extensions:
-
a / b equals floor(a ÷ b).
-
a % b, where b > 0, is a non-negative modulo of a divided by b, and equals a - (a / b) × b.
-
a << b equals a × 2b.
-
a >> b equals a / 2b.
-
XX16 denotes a hexadecimal integer.
-
(a, b, …) denotes an ordered array of values.
-
(XX YY ZZ)16 is a shorthand for (XX16, YY16, ZZ16).
-
a || b concatenates two arrays a and b, treating any non-array value as a singleton array (a) or (b).
-
For array a, an equals a || an-1 when n > 0. a0 equals an empty array ().
-
[a, b] denotes a closed interval between a and b, both inclusive.
-
[a, b) denotes a half-open interval between a (inclusive) and b (exclusive).
All expressions result in an integer between -(264 - 1) and 264 - 1, or an array of those integers with the number of elements not exceeding 264 - 1. Note that subexpressions still may exceed this range.
2.2. Types
Types are a set of values and associated encoding scheme. A type or a structure is implicitly-sized when its byte length is only available through the context. Unless stated as being implicitly-sized, all types have the defined byte length out of the context.
u8 is an unsigned 8-bit integer, occupying 1 byte. Its range is [0, 28).
u16le is an unsigned 16-bit integer, occupying 2 bytes, stored in little-endian byte order (least significant byte first). Its range is [0, 216).
u32le is an unsigned 32-bit integer, occupying 4 bytes, stored in little-endian byte order. Its range is [0, 232).
u64le is an unsigned 64-bit integer, occupying 8 bytes, stored in little-endian byte order. Its range is [0, 264).
u(n), where n is an integer less than 264, is the smallest integer type that can represent n if n > 0, and a zero-sized type if n = 0. In other words,
-
If n = 0, u(n) is zero-sized.
-
If n ∈ [1, 28), u(n) is the same as u8.
-
If n ∈ [28, 216), u(n) is the same as u16le.
-
If n ∈ [216, 232), u(n) is the same as u32le.
-
If n ∈ [232, 264), u(n) is the same as u64le.
leb128 is an unsigned 64-bit integer encoded using the variable number of bytes with the LEB128 encoding. Each byte, except the last, has its most significant bit (MSB) set to 1, indicating that more bytes follow. The remaining 7 bits of each byte are used to store the integer’s value. The least significant 7 bits of the first byte are the least significant bits of the integer, and so on. The last byte has its MSB set to 0. Its range is [0, 264): an encoding MUST be at most 10 bytes long, and MUST NOT encode a value of 264 or more.
a[n], where n is an integer, denotes an ordered array of as. n’s range is [0, 264).
a[*] denotes an implicitly-sized ordered array of as with an unspecified number of elements. The number of elements is implicitly determined from the context. This specification uses two kinds of such contexts:
-
When one or more a[*] fields are in the structure with the defined byte length, the numbers of elements in those fields are increased as needed to exactly fill the required byte length. There MUST be a single set of the numbers of elements that satisfy these conditions.
-
When an a[*] field is given with a constraint, the field extends until, but not including, the first instance of a which doesn’t satisfy the required constraint.
2.3. Structure
Structure is a logical group of one or more ordered fields of possibly differing types concatenated together. Each field has the following properties:
-
The descriptive name of the field (e.g. size, compression_params).
-
The data type of the field (e.g. leb128, u8[size]), with an optional constraint.
-
Optional criteria or conditions that dictate the presence, value, or interpretation of the field.
Field names can only coincide when the field’s type varies by conditions, in which case at most one such condition is applicable at any time. The name can be omitted when the field’s name is the same as the previous field’s.
The field type may be offset, which is a special zero-sized type interpreted as the offset to the following byte relative to the unspecified file position (origin). Offsets are only used to measure the number of bytes within a set of fields. The syntax and semantics of this specification don’t depend on the choice of origin.
The optional constraint does not affect the field’s encoding and only narrows down the possible set of values. Constraints can be one of the following:
-
a ∈ X requires that the value is in the specified set X.
-
a ∉ X requires that the value is not in the specified set X.
-
a = x requires that the value is exactly the given value x. If x is a constant expression, this effectively fixes the byte encoding of this field.
-
a ≠ x requires that the value is not the given value x.
The conditions may be one of the following:
-
"Optional" indicates that conforming implementations MAY include or exclude given field as needed.
-
"If …" indicates that the field MUST be included if and only if the specified condition is true.
-
"Depends on …" indicates that the field’s interpretation varies with the specified set of other fields.
Structures are implicitly-sized only when their byte length is not available through the context and at least one implicitly-sized field exists therein.
2.4. Constants
This specification defines and uses the following constants, all of which are denoted in UPPERCASE.
| Name | Value |
|---|---|
| HEADER_TYPE | 27816 |
| OVERLAY_HEADER_TYPE | 6916 |
| FOOTER_TYPE | 90 5BCB 957D16 |
| OVERLAY_FOOTER_TYPE | 7FC16 |
| ENCRYPTED_HEADER_TYPE | 155 04DC A57916 |
| HEADER_MAGIC | (F8 04 03 6A D6 BE)16 |
| OVERLAY_HEADER_MAGIC | (69 0D 15 AF F6 B2)16 |
| FOOTER_MAGIC | (FD AA AE DE 85 12)16 |
| OVERLAY_FOOTER_MAGIC | (FC 0F 6E 7B 4B AC)16 |
| ENCRYPTED_HEADER_MAGIC | (F9 CA F2 A6 D0 2A 01 00)16 |
| ENCRYPTED_MIRRORED_HEADER_MAGIC | (F9 CA F2 A6 D0 2A 02 00)16 |
| MIRROR_MAGIC | (02 AA F5 FE BA 2B AE 23)16 |
| OVERLAY_MIRROR_MAGIC | (01 05 34 FA BD 11 AF 53)16 |
| ENCRYPTED_MIRROR_MAGIC | (78 D7 3F 69 0A FF 44 83)16 |
| RECOVERY_MAGIC | (69 0A D1 79 CD 15 7A B6)16 |
| OVERLAY_RECOVERY_MAGIC | (45 E7 34 55 EA F2 E1 0D)16 |
2.5. Semantics and failures
This specification defines the meaning of an archive as a function from a byte sequence to an archive value, which also takes a base for an overlay archive. Every algorithm in this specification is declarative: it defines what the result is, not how or in which order it has to be computed, and a conforming implementation MAY compute it in any other way that produces the same result.
The result of interpreting an archive is either an archive value or a failure, which is one of the following:
-
An invalid archive failure occurs when the input violates any requirement of this specification. It is a property of the input alone, and all conforming decompressors agree on it.
-
A resource failure occurs when the evaluation exceeds a declared or implementation-defined limit. It is not a property of the input, and conforming decompressors MAY disagree on it.
-
A decryption failure occurs when an encrypted archive can’t be decrypted with the given passphrase, if any. It is a property of the input and the passphrase, and all conforming decompressors agree on it.
-
A base failure occurs when an overlay archive is interpreted without a base, or an import can’t be resolved in the given base (§ 8.2 Resolving an import). It is a property of the input and the base, and all conforming decompressors agree on it.
-
An unsupported failure occurs when the input contains a block of a critical type not defined by this specification (§ 3.1.1.1 Reserved block types), or a shorthand block whose registered hash the decompressor doesn’t know (§ 4.2.2 Registered dictionaries and modules). It is a property of the input, the version of this specification and the registry entries known, and all conforming decompressors implementing the same version and knowing the same entries agree on it.
A conforming compressor, given an archive value, MUST produce a byte sequence whose interpretation is that archive value. When it produces an overlay archive, it is also given a base, and the interpretation with that base MUST be that archive value. This specification doesn’t constrain how the compressor chooses blocks, block sizes, dictionaries or formulas.
3. Container
3.1. Block
Blocks are a typed byte sequence. The entirety of an unencrypted archive and the beginning of an encrypted archive consist of either blocks or block-like structures.
Blocks have the following generic structure:
| Name | Type | Conditions |
|---|---|---|
| block_start | offset | |
| type | leb128 ∉ {FOOTER_TYPE, OVERLAY_FOOTER_TYPE} | |
| size | u8 | If type % 4 = 0 |
| u16le | If type % 4 = 1 | |
| u32le | If type % 4 = 2 | |
| u64le | If type % 4 = 3 | |
| data | u8[size] | Depends on type |
| block_end | offset | |
| padding | u8 | If (block_end - block_start) % 2 = 1 |
Some type values, including FOOTER_TYPE and OVERLAY_FOOTER_TYPE, are also used to denote block-like structures. For example the archive footer has to have a distinct block type in order to be placed after blocks. This specification doesn’t assign any meaning to such block-like structures unless explicitly stated.
type MUST use the shortest leb128 encoding.
The presence of padding ensures that there are even number of bytes in any block. Conforming compressors MUST set padding to zero, but its value doesn’t affect the result of interpreting an archive.
Later sections describe the contents of data field according to each defined type. Since type also determines the byte length of size, there are generally multiple type values assigned for each logical block type.
3.1.1. Block types
The following table lists every range of block types.
A range of block types for a single logical block type is also described in binary,
e.g. 0001 0mss,
where letters denote bits that determine the parameters of the block.
ss always denotes the two least significant bits, which determine the width of size as above:
0 for u8, 1 for u16le, 2 for u32le and 3 for u64le.
Whether a reserved block type is critical is determined by its leading digit (§ 3.1.1.1 Reserved block types).
Block types of 2 hexadecimal digits are as follows:
| Block Types | Critical | Definition |
|---|---|---|
| 0016 — 0316 | Yes | § 3.3 Spacer block |
| 0416 | Yes | § 3.4 Size hint block |
| 0516 — 0F16 | Yes | Reserved |
| 1016 — 1716 | Yes | § 4.3 Metadata block |
| 1816 — 1F16 | Yes | Reserved |
| 2016 — 3716 | Yes | § 5.1 Data block |
| 3816 — 3F16 | Yes | Reserved |
| 4016 — 4B16 | Yes | § 5.1 Data block |
| 4C16 — 4F16 | Yes | Reserved |
| 5016 | Yes | § 5.2 Dictionary block, as a shorthand block (§ 4.2.2 Registered dictionaries and modules) |
| 5116 — 5716 | Yes | § 5.2 Dictionary block |
| 5816 — 5F16 | Yes | § 8.1 Import block |
| 6016 | Yes | § 7.1 Module block, as a shorthand block (§ 4.2.2 Registered dictionaries and modules) |
| 6116 — 6316 | Yes | § 7.1 Module block |
| 6416 — 6716 | Yes | § 7.3 Resource block |
| 6816 | Yes | Reserved |
| 6916 | Yes | Reserved for OVERLAY_HEADER_TYPE (§ 3.1.1.1 Reserved block types) |
| 6A16 — 6F16 | Yes | Reserved |
| 7016 — 7716 | Yes | § 6.1 Formula block |
| 7816 — 9F16 | Yes | Reserved |
| A016 — BF16 | No | Reserved |
| C016 | No | Reserved (§ 10.5 Recovery sets) |
| C116 — C316 | No | Parity block (§ 10.5 Recovery sets) |
| C416 — C716 | No | Index block (§ 10.5 Recovery sets) |
| C816 — CF16 | No | Reserved |
| D016 — DF16 | No | Reserved for development environments |
| E016 — EF16 | No | Reserved for experiments |
| F016 — FF16 | No | Reserved |
Block types of 3 or more hexadecimal digits are as follows:
| Block Types | Critical | Definition |
|---|---|---|
| 10016 — 9FF16, 100016 — 9FFF16, … | Yes | Reserved, including block-like structures (§ 3.1.1.1 Reserved block types) |
| A0016 — BFF16, A00016 — BFFF16, … | No | Reserved |
| C0016 — C0F16 | No | Reserved |
| C1016 — C1716 | No | § 10.3 Mirror block for § 4.3 Metadata block |
| C1816 — C4F16 | No | Reserved |
| C5016 — C5716 | No | § 10.3 Mirror block for § 5.2 Dictionary block |
| C5816 — C5F16 | No | § 10.3 Mirror block for § 8.1 Import block |
| C6016 — C6316 | No | § 10.3 Mirror block for § 7.1 Module block |
| C6416 — C6716 | No | § 10.3 Mirror block for § 7.3 Resource block |
| C6816 — C6F16 | No | Reserved |
| C7016 — C7716 | No | § 10.3 Mirror block for § 6.1 Formula block |
| C7816 — CFF16, C00016 — CFFF16, … | No | Reserved |
| D0016 — DFF16, D00016 — DFFF16, … | No | Reserved for development environments |
| E0016 — EFF16, E00016 — EFFF16, … | No | Reserved for experiments |
| F0016 — FFF16, F00016 — FFFF16, … | No | Reserved |
3.1.1.1. Reserved block types
The leading digit of a block type is the first digit of its hexadecimal representation, padded with zeros to at least 2 digits. For example, the leading digits of 416, 4B16 and C1016 are 0, 4 and C respectively.
A block type is critical if its leading digit is 0 through 9, and non-critical if it is A through F. A block whose type is not defined by this specification:
-
Results in an unsupported failure (§ 4.4 Interpreting an archive) if its type is critical;
-
Is an ignored block otherwise. Like a mirror block, an ignored block doesn’t belong to any segment, MAY occur anywhere in blocks and doesn’t affect the result of interpreting an archive.
Blocks whose type’s leading digit is D (i.e. D016 through DF16, D0016 through DFF16, …) are reserved for development environments. This specification doesn’t assign any meanings to these blocks. These blocks MUST be written or read only with specific instructions to enable them.
Blocks whose type’s leading digit is E (i.e. E016 through EF16, E0016 through EFF16, …) are reserved for experiments. This specification doesn’t assign any meanings to those blocks.
The following block types are defined as invalid due to the use as block-like structures. A valid archive MUST NOT include such block types as a proper block.
-
Block 27816 (HEADER_TYPE) is used for the unencrypted archive header.
-
Block 6916 (OVERLAY_HEADER_TYPE) is used for the overlay archive header.
-
Block 90 5BCB 957D16 (FOOTER_TYPE) is used for the archive footer.
-
Block 7FC16 (OVERLAY_FOOTER_TYPE) is used for the overlay archive footer.
-
Block 155 04DC A57916 (ENCRYPTED_HEADER_TYPE) is used for the encrypted archive header.
3.2. Shared external structure
Many block types share the same external structure, only varying its internal structure. The structure may or may not be compressed, and uses a compression_method and a checksum_method specified by the parent structure. Each compression method defines the structure of its parameters (compression_params), whose byte length is determined by parsing it. Each checksum_method also has an associated, fixed checksum_size. Checksummed Zstandard data is the compressed external structure (compressed_data) with Zstandard as compression_method and XXH64 as checksum_method.
The compressed external structure (compressed_data) consists of the following:
| Name | Type | Conditions |
|---|---|---|
| original_size_delta | leb128 | |
| original_size_shrink | leb128 | If original_size_delta = 0 |
| compression_params | Parameters of compression_method | Depends on compression_method |
| data_start | offset | |
| data | u8[*] | |
| data_end | offset | |
| checksum | u8[checksum_size] | If checksum_method is not None, depends on checksum_method |
The compressed size (compressed_size) is inferred from the block size: compressed_size = data_end - data_start. The original size (original_size) is encoded relative to compressed_size:
-
If original_size_delta is not 0, original_size = compressed_size + original_size_delta.
-
Otherwise, original_size = compressed_size - original_size_shrink, and original_size_shrink MUST be at most compressed_size.
Each original size thus has exactly one encoding, where the original size equal to compressed_size is encoded as two zero bytes.
The uncompressed external structure (uncompressed_data) consists of the following:
| Name | Type | Conditions |
|---|---|---|
| data | u8[*] | |
| checksum | u8[checksum_size] | If checksum_method is not None |
In both structures, the checksum is computed over the decoded data (§ 5.3 Block contents), i.e. the data after decompression, rather than the stored bytes (§ 10.1 Checksums).
3.2.1. Zstandard compression method
This specification only defines a single compression method, Zstandard [RFC8878]. A Zstandard block is the "Block" structure in the Zstandard frame ([RFC8878], Section 3.1.1.2). Its parameters (compression_params) are as follows:
| Name | Type |
|---|---|
| window_descriptor | u8 |
The meaning of the window_descriptor field is identical to the Zstandard’s Window_Descriptor byte,
which determines Window_Size as in [RFC8878], Section 3.1.1.1.2.
Consequently each Zstandard block holds at most Block_Maximum_Size = min(Window_Size, 128 KiB) bytes of data.
Conforming decompressors MUST support any Window_Size up to 227 bytes (128 MiB),
and MAY return a resource failure for a larger Window_Size.
Conforming compressors SHOULD NOT use a Window_Size larger than 227 bytes
unless explicitly requested.
A dictionary is a byte sequence supplied to the compression method as if it preceded the data, typically shared by multiple small blocks to compensate for the lack of shared compression context. If a dictionary is in use, its content, entropy tables and repeat offsets are given by the dictionary block as described in § 5.2 Dictionary block.
Data compressed by this method is a sequence of Zstandard blocks, without an additional framing provided by Zstandard frames. data MUST consist of one or more such blocks without any other bytes, where only the last block is signalled as the final block by its block header.
3.2.2. XXH64 checksum method
This specification only defines a single checksum method, XXH64 [XXHASH]. Its seed MUST be zero and the result MUST be stored as u64le. The resulting checksum_size is 8.
3.3. Spacer block
Blocks 016 through 316 are spacer blocks. Their data consists of the following:
A spacer block occupies any even number of bytes from 2, which is sufficient since every block occupies an even number of bytes due to padding.
Conforming compressors MUST fill space with zero bytes in order to avoid data leak, but its content doesn’t affect the result of interpreting an archive. Conforming compressors are RECOMMENDED to utilize sparse files for performance.
3.4. Size hint block
Block 416 is a size hint block. Its data consists of the following:
| Name | Type |
|---|---|
| size_hint | leb128 ≠ 0 |
If this block is present, it MUST be the first block after the header and its size_hint field MUST be equal to the footer’s size_hint field.
3.5. Unencrypted archive
An unencrypted archive consists of a header, zero or more blocks and a footer.
| Name | Type | Conditions |
|---|---|---|
| header_magic | u8[6] ∈ {HEADER_MAGIC, OVERLAY_HEADER_MAGIC} | |
| archive_start | offset | |
| size_hint_block | size_hint_block | Optional |
| blocks | block[*] | |
| archive_end | offset | |
| footer_magic | u8[6] ∈ {FOOTER_MAGIC, OVERLAY_FOOTER_MAGIC} | |
| footer_size | u16le = footer_end - footer_start | |
| footer_start | offset | |
| size_hint | leb128 = archive_end - archive_start | |
| metadata_backward_offset | leb128 | |
| footer_end | offset | |
| footer_size_copy | u16le = footer_size |
header_magic is a fixed 6-byte sequence to identify the file as an unencrypted Aqueous archive. This header also marks the beginning of the archive, when the archive doesn’t start at the beginning of the file.
footer_magic is a fixed 6-byte sequence to mark the beginning of the footer. The footer also marks the end of the archive, when the archive doesn’t end at the end of the file.
An unencrypted archive is an overlay archive if header_magic is OVERLAY_HEADER_MAGIC, and a standalone archive otherwise. footer_magic MUST be OVERLAY_FOOTER_MAGIC in an overlay archive and FOOTER_MAGIC otherwise. An overlay archive is interpreted against a base, an existing directory tree or file given separately from the overlay archive: it starts from the base instead of an empty directory and applies its segments to it (§ 4.4 Interpreting an archive), and only an overlay archive can contain external import blocks reading the base (§ 8.1 Import block). The format of overlay archives is called the Aqueous overlay archive format, and has its own media type (§ 15.2 application/aqo Media Type).
The offset between header_magic and footer_magic is recorded as size_hint, which MAY also be replicated as size_hint_block if random access is desired.
The footer size, which is the size of all fields between footer_size and footer_size_copy (both exclusive), is recorded both in footer_size and in footer_size_copy, so that the footer can be read from the end of the archive (§ 3.6 Locating an archive).
metadata_backward_offset is the offset from footer_start back to the beginning of the last metadata block, just like previous_metadata_offset is the offset from the beginning of the current metadata block back to the previous one. It MUST be 0 if and only if there is no metadata block, so that a decompressor can tell such archives without scanning blocks.
blocks are logically divided into segments, each consisting of content blocks and a single metadata block describing them, with optional spacer, mirror and ignored blocks in between. Each segment can be independently processed and decompressed in the reading order, except that internal imports (§ 8.1 Import block) read the result of preceding segments. Segments are described in § 4 Segments and entries.
Alternatively, blocks can have no metadata block. Such an archive decompresses into a single unnamed file, whose content is the concatenation of the contents of all leaf blocks in the order of appearance. It decompresses into an empty file if there is no leaf block.
3.6. Locating an archive
A file may contain an archive with leading bytes, e.g. a self-extracting executable, or with trailing bytes, e.g. appended by a transfer. Since interpreting an archive takes the archive alone, a decompressor accepting such a file has to locate the archive first.
A header magic and a footer magic match when they are HEADER_MAGIC and FOOTER_MAGIC, or OVERLAY_HEADER_MAGIC and OVERLAY_FOOTER_MAGIC.
To locate an archive in a byte sequence file:
-
If file starts with ENCRYPTED_HEADER_MAGIC or ENCRYPTED_MIRRORED_HEADER_MAGIC, return file.
-
Let back be null. Let n be the last 2 bytes of file as u16le, and let f be the length of file − n − 10. If f ≥ 0, the bytes of file from f are FOOTER_MAGIC or OVERLAY_FOOTER_MAGIC followed by n as u16le, and the leb128 from f + 8 is s such that h = f − s − 6 ≥ 0 and the bytes of file from h are the header magic matching that footer magic, set back to h.
-
If file starts with HEADER_MAGIC or OVERLAY_HEADER_MAGIC:
-
If back is 0, return the result of trimming file.
-
Read blocks from the offset 6 by the generic structure (§ 3.1 Block), with data of every block taken as a byte sequence without parsing, until the bytes at the current offset e start with FOOTER_MAGIC or OVERLAY_FOOTER_MAGIC. If it fails to read a block before that, or the footer magic doesn’t match the header magic, return failure.
-
Let m be the 2 bytes of file from e + 6 as u16le. If file is shorter than e + m + 10 bytes, return failure. Otherwise, return the first e + m + 10 bytes of file.
-
-
If back is not null, return the result of trimming the bytes of file from back.
-
Return failure.
To trim a located archive a, a byte sequence starting with a header magic and ending with a footer whose magic matches it:
-
Let f be the offset of the footer at the end of a. If its metadata_backward_offset is 0, return a.
-
Let last be the block starting metadata_backward_offset bytes before the footer_start of that footer. If it can’t be read by the generic structure (§ 3.1 Block) or is not a metadata block, return a.
-
Let blocks be a list containing last, and m be last. While the decoded data of m can be parsed as far as its previous_metadata_offset, and the block starting previous_metadata_offset bytes before m can be read by the generic structure and is a metadata block, set m to that block and prepend it to blocks.
-
For each block m of blocks, in order:
-
Let e be the offset after the padding of m (the block_end if it has none).
-
While the bytes of a from e are a whole spacer block read by the generic structure, set e to the offset after it.
-
If e is not f and the bytes of a from e start with the footer magic of a followed by a u16le n such that e + n + 10 ≤ f, return the first e + n + 10 bytes of a.
-
-
Return a.
3.7. Appending to an archive
This section is non-normative, except where noted.
Since a later segment adds, replaces and removes the entries of earlier ones (§ 4.4.2 Merging segments), an unencrypted archive can be extended by appending segments after its last metadata block, without rewriting what precedes it. The footer has to move to the new end, which is done so that the file holds a valid archive at every point: either the archive before the appending or the archive after it. To commit one or more segments to an archive a in a file, whose footer F starts at the offset p of the file:
-
If the length of F is odd, write a zero byte after it. Write the blocks of the segments after it, and a footer for the archive ending with them, directly after their last metadata block or after a spacer block following it (§ 3.3 Spacer block). Make sure that all of them are durably written.
-
Overwrite the first bytes of F with the type and size of a spacer block occupying the bytes of F and its padding byte if any, and make sure that they are durably written.
-
Overwrite the rest of F with zero bytes, as the space of a spacer block requires.
Until the step 2 completes, the file holds a, followed by trailing bytes that trimming leaves out. Afterwards it holds the extended archive, as F is no longer a footer. If the file stops being written at any point, the next appending finds a or the extended archive and either removes the trailing bytes or completes the step 2 if they end with a whole footer.
The step 2 overwrites 2 bytes if F is 257 bytes or shorter (a spacer block of the type 0), 3 bytes up to 65,538 bytes, and 5 bytes otherwise. It relies on that write being done entirely or not at all, which storage devices guarantee within an aligned sector of 512 bytes or a multiple of it. Therefore a compressor that is to append to an archive later places each footer so that the bytes the step 2 would overwrite, at their offset in the file, don’t cross a multiple of 512 bytes, putting a spacer block of 2 bytes after the last metadata block when they would. Since an archive may be put after leading bytes afterwards, e.g. those of a self-extracting executable, a compressor appending to an archive whose footer crosses that boundary can’t commit as above, and instead removes the footer and writes the segments and a new footer in its place. Such an archive stopped being written before the new footer is whole loses its footer, which can be reconstructed (§ 10.6 Repair).
A compressor appending to an archive MUST NOT leave a footer other than the one at the end,
except while it is writing; another compressor appending to such an archive MUST first remove the trailing bytes or complete the commit.
Each metadata block whose segment ends a commit SHOULD have segment-status (§ 11.3 segment-status: Commits and the end of an archive).
The located archive is then interpreted, which may still fail. A decompressor SHOULD warn the user when file has bytes outside the located archive. It MAY search for HEADER_MAGIC and OVERLAY_HEADER_MAGIC when the archive can’t be located otherwise, e.g. when file has both leading and trailing bytes.
A block can’t start with FOOTER_MAGIC or OVERLAY_FOOTER_MAGIC, since they start with the shortest leb128 encodings of FOOTER_TYPE and OVERLAY_FOOTER_TYPE (§ 2.4 Constants), which are not valid block types. The step 3.2 therefore finds the end of the first archive in file, while the step 4 finds the archive at the end of file without reading the bytes before it. Archives concatenated together are not a single archive, and only the first one is located.
4. Segments and entries
4.1. Archive model
An archive value is a struct with the following items:
-
root, either a directory node or a file node. The latter represents a single unnamed file.
-
extra, an ordered map from integers to CBOR data items [RFC8949] representing the archive-wide part of the optional metadata.
A file node is a struct with the following items:
-
content, a byte sequence.
-
link group, a link group or null.
-
extra, a CBOR data item matching
aqr-entry-extra, or null.
A directory node is a struct with the following items:
-
children, an ordered map from entry names to file nodes, directory nodes, link nodes, special nodes or unknown nodes.
-
extra, a CBOR data item matching
aqr-entry-extra, or null.
A link node is a struct with the following items, representing a symbolic link:
-
target, a link target.
-
extra, a CBOR data item matching
aqr-entry-extra, or null.
A special node is a struct with the following items, representing a FIFO or a device file:
-
kind, one of "
fifo", "char-device" or "block-device". -
device, a pair of non-negative integers (major, minor) if kind is "
char-device" or "block-device", or null otherwise. -
extra, a CBOR data item matching
aqr-entry-extra, or null.
dev_t encoding,
as tar and cpio do and as makedev takes it.
Its meaning is up to the platform extracting it:
the same pair generally denotes unrelated devices on different platforms,
e.g. /dev/null is (1, 3) on Linux but (3, 2) on macOS,
and this specification doesn’t record which platform the number came from.
An unknown node is a struct with the following items, representing a kind of node not defined by this version of this specification:
-
kind, an integer greater than 4.
-
data, a byte sequence.
-
extra, a CBOR data item matching
aqr-entry-extra, or null.
Every file node, directory node, link node, special node and unknown node also has
forks, an ordered map from fork names to byte sequences,
which represents named data streams other than the content of a file,
e.g. resource forks of macOS and alternate data streams of NTFS (§ 11.7 forks: Resource forks and alternate data streams).
A fork name is a CBOR data item matching aqr-text (§ 11.2 Comments) that is not empty,
and two fork names are equal if and only if they are identical CBOR data items.
A node without any fork has an empty map as its forks,
as does every node created by an algorithm in this specification unless its forks are given.
The root can’t be a link node, a special node or an unknown node.
A tombstone is a struct with no items, marking the removal of a node. It only occurs in children of the result of interpreting a segment, and is removed by merging it into the preceding result (§ 4.4.2 Merging segments), so it never occurs in an archive value.
An entry name is a struct with the following items:
-
encoding, either "
wtf-8" or "raw". -
bytes, a non-empty byte sequence which is a valid WTF-8 [WTF-8] string if encoding is "
wtf-8".
Two entry names are equal if and only if both items are equal.
In particular, a "wtf-8" name and a "raw" name are never equal even when their bytes are.
No bytes of an entry name can contain 0016 or 2F16 (/),
or be equal to (2E)16 (.) or (2E 2E)16 (..).
An entry name is encoded as a byte sequence in one of the following ways, neither of which contains a zero byte:
-
If its encoding is "
wtf-8", its bytes as is. WTF-8 [WTF-8] is an extension to UTF-8 and losslessly encodes a lone surrogate in UTF-16.
Conforming compressors are RECOMMENDED to use the "wtf-8" encoding,
and SHOULD only use the "raw" encoding as a round-trip measure.
To decode an entry name from a entry_name field bytes:
-
If bytes is not empty and bytes[0] is C116, let name be an entry name whose encoding is "
raw" and bytes are bytes without the first byte. -
Otherwise, if bytes is a valid WTF-8 string, let name be an entry name whose encoding is "
wtf-8" and bytes are bytes. -
Otherwise, return an invalid archive failure.
-
If name violates any restriction of an entry name above, return an invalid archive failure.
-
Return name.
A link target is a struct with the following items:
-
encoding, either "
wtf-8" or "raw". -
bytes, a non-empty byte sequence not containing 0016, which is a valid WTF-8 [WTF-8] string if encoding is "
wtf-8".
Unlike an entry name, a link target is a path which may contain 2F16 (/) as a separator,
and components equal to . or ...
It may be absolute, and may refer to a path that doesn’t exist in the archive value or anywhere else.
The separator is always /, even for a link target from a platform using \ as the separator, such as Windows,
where every \ is represented as /.
Otherwise a link target is not normalized, and its meaning, e.g. of a drive letter, is left to the platform.
There is no limit on its length other than implementation-defined limits.
/ can’t occur in a file name on Windows, and isn’t interpreted as a separator in a link target there,
so this replacement is lossless.
A decompressor on Windows replaces / back with \ when creating a link (§ 12 Extraction).
On Windows, the target of an absolute link or a junction is stored as an NT path starting with \??\,
which has no meaning on other platforms.
A compressor is advised to convert it to a Win32 path first,
e.g. \??\C:\foo to C:/foo, \??\UNC\server\share\x to //server/share/x
and \??\Volume{GUID}\x to //?/Volume{GUID}/x.
Platforms differ in the maximum length of a link target: 4095 bytes on Linux (less on some file systems, e.g. 1023 bytes on XFS), 1023 bytes on macOS and FreeBSD, and about 8,000 UTF-16 code units on Windows. A link that can’t be created is handled at the extraction time (§ 12 Extraction).
To decode a link target from a link_target field bytes, do the same as to decode an entry name from bytes, except that the result is a link target and restrictions of a link target apply instead.
A link target is never resolved within the archive model.
Every path in an archive value is still a plain sequence of components,
and a lookup of a path in an archive value never follows a link node, as lstat doesn’t.
In particular, no node is a descendant of a link node.
a/passwd through a preceding link a pointing to /etc
inexpressible in the archive model.
A tool MAY offer a lookup following links, but it then has to limit the number of links followed.
A link group is an opaque value that is only compared for equality, and is either a pair of a segment and a segment-local integer or an archive-wide integer (§ 11.5 link-group: Hard links). Two archive values are equal if and only if they become identical by some one-to-one renaming of the link groups of one of them.
4.2. Content blocks
Any block other than spacer, size hint, metadata, mirror and ignored blocks
is a content block which has to be described by a subsequent metadata block,
either by its fields or by forks in its optional metadata (§ 11.7 forks: Resource forks and alternate data streams).
A metadata block and all content blocks between the previous metadata block (or the beginning of the archive if none)
and that metadata block form a segment.
As an exception, an archive without a metadata block (§ 3.5 Unencrypted archive) forms a single segment with no metadata block.
Each segment MUST be self-contained:
every reference from a block in a segment, whether by an index or by an offset, MUST point to a block in the same segment.
An internal import (§ 8.1 Import block) refers to a node of preceding segments by its path rather than to a block,
and is the only way for a segment to use the content of another segment.
Content blocks are classified as follows:
-
A leaf block represents the content of one or more files by itself. Leaf data blocks and formula blocks are leaf blocks.
-
A non-leaf block doesn’t represent any file by itself, and only supplies data referenced by one or more formula blocks. Non-leaf data blocks and import blocks are non-leaf blocks. Each non-leaf block supplies one or more non-leaf items to formula blocks: a non-leaf data block supplies a single item, and an import block supplies one item for each of its imports.
-
An auxiliary block supplies data used to decode other blocks. Dictionary blocks, module blocks and resource blocks are auxiliary blocks.
Dictionary, module and resource blocks are respectively indexed in the order of appearance within the segment. Two conventions are used for these indices:
-
Fields in content blocks use 0-based indices, because the presence of a reference is signalled separately (e.g. by the block type).
-
Fields in the metadata block that may indicate the absence of a reference use 1-based indices, where zero denotes the absence.
4.2.1. Block order
Blocks in a segment MUST be ordered so that a formula block is resolved by blocks preceding it and all data required to evaluate it follows it:
-
A dictionary block MUST precede any block using it.
-
A module or resource block MUST precede any formula block referencing it.
-
Every non-leaf item MUST be referenced by at least one formula block, and every non-leaf block MUST follow all formula blocks referencing its items.
-
Leaf blocks are otherwise not constrained and MAY be interleaved with non-leaf blocks.
A leaf block and all subsequent content blocks up to the next leaf block or the end of the segment form a parcel. When no leaf block is interleaved with non-leaf blocks, a parcel consists of a leaf block, the non-leaf blocks it references if it is a formula block, and possibly auxiliary blocks. In that case a decompressor can read a single file by reading its parcel and the auxiliary blocks it uses, and a streaming decompressor never has to retain a formula block beyond its parcel.
4.2.2. Registered dictionaries and modules
A registered dictionary is an entry of the Aqueous Dictionaries Registry (§ 15.5 Aqueous Dictionaries Registry), and a registered module is an entry of the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry). Each entry is identified by its registered hash, the SHA-512/256 hash [FIPS180-4] of its bytes: the decoded data of a dictionary block for a registered dictionary (§ 5.2 Dictionary block), and wasm_module for a registered module (§ 7.1 Module block).
Block 5016 (a dictionary block) and block 6016 (a module block) are shorthand blocks. The data of a shorthand block is a registered hash alone, 32 bytes long, instead of checksummed Zstandard data, and the shorthand block stands for a block of the same kind whose decoded data is the bytes of that entry:
-
size MUST be 32.
-
The registered hash of block 5016 MUST be that of a registered dictionary, which also gives has_entropy_tables, and that of block 6016 MUST be that of a registered module.
-
If the decompressor doesn’t know the entry of the registered hash in the corresponding registry, an unsupported failure occurs (§ 4.4 Interpreting an archive).
-
A shorthand block is otherwise the same as the dictionary or module block it stands for, e.g. in indexing, block order and mirror blocks.
The two registries share the hash function but not the entries, and a shorthand block is resolved only against the registry of its own kind. Knowing an entry of one registry never makes a decompressor know an entry of the other: in particular, the hash of a registered module in block 5016 doesn’t denote a dictionary of the module’s bytes, unless the same bytes are separately registered as a dictionary, in which case they are two unrelated entries.
A shorthand block has no checksum, and the registered hash is only looked up and never verified. Therefore damage to the hash can’t be distinguished from a hash that was never registered, and both result in an unsupported failure rather than an invalid archive failure.
4.3. Metadata block
Blocks 1016 through 1716 are metadata blocks,
whose type is 0001 0mss in binary (§ 3.1.1 Block types).
Their data uses the shared external structure (§ 3.2 Shared external structure),
with XXH64 as checksum_method
and compression_method given by m: None if m is 0, and Zstandard if m is 1.
The internal structure after decompression is as follows:
| Name | Type |
|---|---|
| previous_metadata_offset | leb128 |
| num_modules | leb128 |
| num_resources | leb128 |
| num_dictionaries | leb128 |
| num_directories | leb128 |
| num_files | leb128 |
| num_specials | leb128 |
| entries_size | leb128 = entries_end - entries_start |
| optional_metadata_size | leb128 |
| module_offsets | u(previous_metadata_offset)[num_modules] |
| resource_offsets | u(previous_metadata_offset)[num_resources] |
| dictionary_offsets | u(previous_metadata_offset)[num_dictionaries] |
| entry_parents | u(num_directories)[num_directories + num_files + num_specials - 1] |
| file_params | file_param[num_files] |
| entry_offsets | u(entries_size)[num_directories + num_files + num_specials - 1] |
| entries_start | offset |
| entries | entry[num_directories + num_files + num_specials] |
| entries_end | offset |
| optional_metadata | u8[optional_metadata_size] |
All relative offsets are measured backward from the beginning of the current metadata block: a relative offset o refers to the block starting o bytes before the current metadata block, implying that the offset of zero is invalid. All valid relative offsets are also even.
Each metadata block describes the content blocks in its segment, i.e. blocks appearing after the previous metadata block, if any, and until the current metadata block. The previous metadata block is linked via previous_metadata_offset, which is also the maximum possible offset that can occur in this particular metadata block. If there is no prior metadata block, previous_metadata_offset is the relative offset to the end of header_magic in the unencrypted archive structure (§ 3.5 Unencrypted archive), plus 1 so that it never points to a valid block.
num_modules, num_resources and num_dictionaries are the numbers of module, resource and dictionary blocks in the segment respectively. Relative offsets to those blocks are later recorded as module_offsets, resource_offsets and dictionary_offsets respectively, in the order of appearance.
The metadata block represents zero or more directories, zero or more files and zero or more special entries,
collectively called entries and numbered together.
The root directory is numbered 0,
subsequent directories are numbered from 1 to num_directories,
individual files are numbered from (num_directories + 1) to (num_directories + num_files),
and special entries are numbered from (num_directories + num_files + 1)
to (num_directories + num_files + num_specials).
There MUST be at least one entry other than the root directory in the first metadata block of an archive,
i.e. num_directories + num_files + num_specials ≥ 1.
A later metadata block MAY have none, e.g. to add only the journal (§ 11.4 journal: Points to return to) of a state to the archive;
its segment then has no content blocks, and entry_parents, entry_offsets and entries are empty.
entry_parents[k - 2] is the parent of the entry k for each k ≥ 2,
and MUST be a directory less than k,
i.e. in [0, min(k - 1, num_directories)], so that directories can’t form a cycle.
The root directory (entry 0) has no parent and the parent of the entry 1 is always the root directory,
so they are omitted.
File parameters (file_param) occur for each file (but not for directories or special entries), and consist of the following:
| Name | Type | Conditions |
|---|---|---|
| file_dictionary | u(num_dictionaries) | |
| file_offset | u(previous_metadata_offset) | |
| file_output | leb128 | If file_offset is odd |
file_dictionary is the dictionary used by the block that file_offset refers to as a 1-based index, or zero if no dictionary is used. Therefore file_dictionary = k refers to the block at dictionary_offsets[k - 1], and num_dictionaries is the maximum possible value for it. A relative offset to the corresponding leaf block, which starts the file’s parcel, is recorded as file_offset, plus 1 if the file is another output of the block than the output 0. The relative offset can’t be zero. Since every relative offset is even, file_offset % 2 tells which: the file output index is 0 if file_offset is even, and file_output + 1 if it is odd. Multiple files MAY refer to the same leaf block and output, in which case they have the same content.
Entry structure (entry) then occurs for each directory, file and special entry, in this order, and consists of the following:
| Name | Type | Conditions |
|---|---|---|
| entry_name | u8[*] ≠ 0 | |
| entry_name_sentinel | u8 = 0 | |
| original_size | leb128 | If this entry is for a file |
| special_kind | leb128 | If this entry is for a special entry |
| link_target | u8[*] ≠ 0 | If this entry is for a special entry and special_kind = 1 |
| link_target_sentinel | u8 = 0 | If this entry is for a special entry and special_kind = 1 |
| device_major | leb128 | If this entry is for a special entry and special_kind is 3 or 4 |
| device_minor | leb128 | If this entry is for a special entry and special_kind is 3 or 4 |
| special_data | u8[*] | If this entry is for a special entry and special_kind > 4 |
Each entry is named according to entry_name, which encodes an entry name as described in § 4.1 Archive model and is decoded there. Entry names from ZIP are discussed in § 16.1 Entry names from ZIP.
A special entry represents a node other than a file or a directory, as determined by special_kind:
| special_kind | Node |
|---|---|
| 0 | Tombstone, which removes the node at the same path in preceding segments or in the base (§ 4.4.2 Merging segments) |
| 1 | Link node, with link_target as its target |
| 2 | Special node of the kind "fifo"
|
| 3 | Special node of the kind "char-device", with device_major and device_minor as its device number
|
| 4 | Special node of the kind "block-device", with device_major and device_minor as its device number
|
| 5 or greater | Unknown node, reserved for future versions of this specification |
link_target uses the same encodings as entry_name. special_data is not interpreted by this version of this specification; its length is determined by the beginning of the next entry in entry_offsets, or entries_end for the last entry.
Unknown kinds are not an invalid archive failure, so that a decompressor can still extract the rest of an archive using a kind defined in the future. Every special entry can be skipped without knowing its kind, thanks to entry_offsets. An unknown node replaces an earlier node at the same path like any other non-directory node (§ 4.4.2 Merging segments), so a future kind can’t change how segments are merged. This is why tombstones are defined by this version, as the first kind: a decompressor treating a tombstone as an unknown node would keep it in place of the node it was meant to remove.
The entries array is the only part that has the unspecified number of bytes, which is therefore available as entries_size beforehand. Any other field can be randomly accessed after reading the initial set of leb128 fields.
As an exception to the relative offsets above, entry_offsets are measured from entries_start. The entry 1 always starts at entries_start, so its offset is omitted, and entry_offsets[k - 2] is the offset to the beginning of the entry k for each k ≥ 2. They MUST agree with the actual positions of entries.
optional_metadata is described in Optional metadata (§ 11 Optional metadata).
4.4. Interpreting an archive
To interpret an archive given a byte sequence input, an optional passphrase passphrase and an optional base base, which is an archive value:
-
If input starts with ENCRYPTED_HEADER_MAGIC or ENCRYPTED_MIRRORED_HEADER_MAGIC:
-
Parse input as the encrypted archive structure (§ 9 Encrypted archives) in its entirety. If it fails to parse, return an invalid archive failure.
-
If header_checksum is present and doesn’t match, return an invalid archive failure.
-
If lg_work_factor exceeds an implementation-defined limit, return a resource failure.
-
If passphrase is not given, or the file key can’t be unwrapped from encrypted_file_key with passphrase as described in [AGE], return a decryption failure.
-
Using the unwrapped file key, verify age_header_mac against the reconstructed age header and decrypt the age payload as described in [AGE]. If either fails, return an invalid archive failure.
-
Set input to the decrypted payload. It MUST satisfy the additional constraints in § 9 Encrypted archives, or an invalid archive failure is returned.
-
-
If input parses as the unencrypted archive structure (§ 3.5 Unencrypted archive) in its entirety, with data of every block taken as a byte sequence without parsing, and either the type of any of blocks is critical and not defined by this specification, or any of blocks is a shorthand block of 32 bytes of data whose registered hash the decompressor doesn’t know (§ 4.2.2 Registered dictionaries and modules), return an unsupported failure.
-
Parse input as the unencrypted archive structure (§ 3.5 Unencrypted archive) in its entirety. If it fails to parse, return an invalid archive failure.
-
If input is an overlay archive and base is not given, return a base failure.
-
If input is a standalone archive, set base to an archive value whose root is a directory node with an empty map as children and null as extra, and whose extra is an empty map.
-
If blocks contain no metadata block:
-
Let content be the concatenation of the file contents of all leaf blocks in blocks, in the order of appearance, each for every output index from 0 in order (all outputs of a formula block, and the only one of a data block).
-
Return an archive value whose root is a file node with content as content, null as link group and null as extra, and whose extra is an empty map.
-
-
If the root of base is a directory node, let root be a copy of it. Otherwise let root be a directory node with an empty map as children and null as extra.
-
Let extra be a copy of the extra of base.
-
For each segment segment in blocks, in the order of appearance:
-
Let segment root be the result of interpreting segment.
-
Merge segment root into root.
-
For each key k and value v in the top-level map of the segment’s optional metadata other than
entry-extras,segment-status(§ 11.3 segment-status: Commits and the end of an archive) andjournal(§ 11.4 journal: Points to return to), set extra[k] to v.
-
-
Return an archive value with root as root and extra as extra.
The archive base is base after the step 5, and the preceding result of a segment is root just before the segment is interpreted in the step 9, or the root of the archive base for an archive without a metadata block. They are what external imports and internal imports respectively refer to (§ 8.2 Resolving an import). A given base doesn’t affect the interpretation of a standalone archive.
The input to this algorithm is exactly the archive, from the first byte of header_magic to the last byte of footer_size_copy.
A file may contain an archive with leading or trailing bytes, which a decompressor removes by locating the archive (§ 3.6 Locating an archive).
Since every content block has to be described by a subsequent metadata block (§ 4.2 Content blocks), an archive with a metadata block can’t have any content block after the last metadata block. previous_metadata_offset and metadata_backward_offset MUST point to the metadata blocks found by parsing blocks as described in § 4.3 Metadata block and § 3.5 Unencrypted archive, so that a decompressor can find each segment either by scanning blocks forward or by following offsets backward.
4.4.1. Interpreting a segment
To interpret a segment segment whose metadata block is metadata block:
-
Let meta be the decoded data of metadata block, parsed as its internal structure (§ 4.3 Metadata block).
-
Let d be num_directories, f be num_directories + num_files and n be num_directories + num_files + num_specials of meta.
-
Let nodes be a list containing a single directory node with an empty map as children, and set the extra and forks of it from the entry extra for the entry 0.
-
For each k in [1, n]:
-
Let entry be meta.entries[k - 1].
-
Let name be the result of decoding entry.entry_name.
-
Let parent be 0 if k = 1, or meta.entry_parents[k - 2] otherwise.
-
If k ≤ d, let node be a directory node with an empty map as children.
-
Otherwise, if k ≤ f:
-
Let param be meta.file_params[k - d - 1].
-
Let block be the block at the relative offset param.file_offset - param.file_offset % 2 from metadata block. It MUST be a leaf block in segment.
-
If param.file_dictionary is zero, block MUST NOT use a dictionary. Otherwise, block MUST use the dictionary block at meta.dictionary_offsets[param.file_dictionary - 1].
-
Let content be the file content of block for the file output index of param. Its length MUST be entry.original_size.
-
Let node be a file node with content as content and null as link group.
-
If the entry extra for the entry k has
link-groupwith a value g, set node.link group to a pair of segment and g if g ≥ 0, or to g otherwise.
-
-
Otherwise, if entry.special_kind is 0, let node be a tombstone. The entry extra for the entry k MUST be null.
-
Otherwise, if entry.special_kind is 1:
-
Let target be the result of decoding entry.link_target.
-
-
Otherwise, if entry.special_kind is 2, let node be a special node with "
fifo" as kind and null as device. -
Otherwise, if entry.special_kind is 3 or 4, let node be a special node with "
char-device" or "block-device" respectively as kind and (entry.device_major, entry.device_minor) as device. -
Otherwise, let node be an unknown node with entry.special_kind as kind and entry.special_data as data.
-
If node is not a tombstone, set the extra and forks of node from the entry extra for the entry k.
-
Let children be the children of nodes[parent]. children[name] MUST NOT exist.
-
Set children[name] to node.
-
Append node to nodes.
-
-
Return nodes[0].
Since every parent is less than k and at most d by § 4.3 Metadata block, nodes[parent] is always a directory node already in nodes.
The entry extra for the entry k is the element at the 0-based index k of entry-extras
in the segment’s optional metadata, where the index 0 corresponds to the root directory,
or null if entry-extras or such element doesn’t exist.
entry-extras MUST NOT have more elements than the entries including the root directory,
i.e. num_directories + num_files + num_specials + 1.
To set the extra and forks of a node node from an entry extra e:
-
If e is null, set the extra of node to null and its forks to an empty map, and return.
-
Set the extra of node to e without its
forksmember, if any. -
Set the forks of node to an empty map.
-
For each name → [offset, output, size] of
forksin e, if any:-
Let block be the block at the relative offset offset from the metadata block of the segment. It MUST be a leaf block in the segment.
-
Let content be the file content of block for the output index output. Its length MUST be size.
-
Set node.forks[name] to content.
-
forks refers to blocks by relative offsets, which have no meaning outside the segment,
so it is resolved here and never remains in the extra of a node.
Unlike file_params, forks has no dictionary field:
a leaf data block names its dictionary by dictionary_index anyway,
and file_dictionary only lets a decompressor fetch the dictionary before reading the block.
4.4.2. Merging segments
To merge directory nodes later into earlier:
-
For each name → node of later.children:
-
Let existing be earlier.children[name], or null if it doesn’t exist.
-
If node is a tombstone, remove earlier.children[name] if it exists.
-
Otherwise, if node is a directory node:
-
If existing is not a directory node, set existing to a directory node with an empty map as children and null as extra, and set earlier.children[name] to existing.
-
Merge node into existing.
-
-
Otherwise, set earlier.children[name] to node.
-
-
If later.extra is not null, set earlier.extra to it and earlier.forks to later.forks.
As a result, a directory appearing in multiple segments has the union of their children,
while any other node, or a directory replaced by a node of the different kind, is replaced as a whole with its descendants.
In particular, a directory replaced by a link node loses all its descendants,
so no node is a descendant of a link node in the result.
A tombstone removes the node at its path along with its descendants, and does nothing if there is no such node.
A tombstone is never kept in the result, even under a directory that didn’t exist before.
A directory with a null extra in a later segment keeps the existing extra and forks,
which can instead be replaced with an empty map {}.
5. Stored data
5.1. Data block
Blocks 2016 through 3716 and 4016 through 4B16 are data blocks. A data block is a compressed block if its compression_method is not None, and an uncompressed block otherwise.
The type of a data block is 001kk css in binary (§ 3.1.1 Block types) for a leaf block,
and 0100 kkss for a non-leaf block, where:
-
kk determines compression_method and whether a dictionary is used (has_dictionary): 0 for None without dictionary, 1 for Zstandard without dictionary, and 2 for Zstandard with dictionary. 3 is reserved for future compression methods.
-
c determines checksum_method: None if 0, and XXH64 if 1. A non-leaf data block always uses XXH64 (§ 10.1 Checksums).
Its data consists of the following:
| Name | Type | Conditions |
|---|---|---|
| dictionary_index | leb128 | If has_dictionary |
| compressed_data | compressed_data | If compression_method is not None |
| uncompressed_data | uncompressed_data | If compression_method is None |
If a dictionary is in use, its 0-based index is specified as dictionary_index. For a leaf data block, the metadata block also gives the dictionary as file_dictionary, which is available before reading the block; for a non-leaf data block, dictionary_index is the only reference to the dictionary. compressed_data and uncompressed_data use the shared external structure (§ 3.2 Shared external structure) with compression_method and checksum_method determined as above. uncompressed_data represents a raw, uninterpreted byte sequence.
A compressed block is a single unit of decompression: reading any part of its content requires decompressing it from the beginning. Content larger than a desired unit of random access is split into multiple blocks:
-
In general, a file is represented by a formula block concatenating multiple non-leaf blocks, which typically form a single parcel together. The formula block then acts as an index for random access within the parcel.
-
In an archive without a metadata block (§ 3.5 Unencrypted archive), the file is simply represented by multiple leaf blocks.
5.2. Dictionary block
Blocks 5016 through 5716 are dictionary blocks,
whose type is 0101 0tss in binary (§ 3.1.1 Block types).
Their data is checksummed Zstandard data (§ 3.2 Shared external structure),
except that block 5016 is a shorthand block for a registered dictionary (§ 4.2.2 Registered dictionaries and modules).
t determines whether the dictionary has entropy tables (has_entropy_tables): false if 0, and true if 1.
The internal structure after decompression is as follows:
| Name | Type | Conditions |
|---|---|---|
| entropy_tables | u8[*] | If has_entropy_tables |
| repeat_offsets | u32le[3] | If has_entropy_tables |
| content | u8[*] |
The decoded data of a dictionary block is a dictionary for blocks using it,
and corresponds to a Zstandard dictionary ([RFC8878], Section 5) without Magic_Number and Dictionary_ID:
-
If has_entropy_tables is true, entropy_tables is the
Entropy_Tablesstructure, whose length is determined by parsing it, and repeat_offsets are the threeRepeat_Offsetsin order. Each repeat offset MUST be in [1, length of content]. -
Otherwise, it is a raw content dictionary: the entropy tables start empty and the repeat offsets start at their initial values (1, 4, 8), even when content begins with the dictionary
Magic_Number.
In both cases, content is the Content of the dictionary,
which is the only part of the dictionary that sequences can reference as preceding data.
content MUST be at least 8 bytes long.
Magic_Number and Dictionary_ID are omitted because they are only meaningful for Zstandard frames
and the framing is done by this format.
An implementation using libzstd can prepend them (with an arbitrary nonzero Dictionary_ID)
to load a dictionary with entropy tables.
In that case it has to make sure that no sequence references the prepended bytes or entropy_tables,
which some decompression paths of libzstd don’t check.
ZDICT_finalizeDictionary)
and mainly benefit small blocks:
in measurements with per-file blocks of 1–20 KB files, they reduced the size by 0.4–26%,
while the benefit vanished for blocks of 64 KB or more.
Fast strategies of libzstd (e.g. levels 1–3 of 1.5.7) reuse the dictionary’s entropy tables for every block
even when the block’s own tables would be smaller.
When the repeat offsets of the dictionary are the initial values,
a compressor can avoid this by also compressing a block with the same content as a raw content dictionary
and choosing the smaller result,
since a block compressed that way never refers to the entropy tables of the dictionary.
5.3. Block contents
The decoded data of a block using the shared external structure (§ 3.2 Shared external structure), given its compression_method and checksum_method, is determined as follows:
-
If compression_method is None, it is data.
-
If compression_method is Zstandard, it is the result of decompressing data as described there, with compression_params and the dictionary given by the dictionary block in use (§ 5.2 Dictionary block), if any. The result MUST have exactly original_size bytes.
-
If checksum_method is not None, the checksum MUST be equal to the checksum of the decoded data computed by the checksum_method.
The stored content of a data block, whether a leaf block or not, is the decoded data of its compressed_data or uncompressed_data. The stored content of a non-leaf item is the stored content of the non-leaf data block supplying it, or the imported content of the import supplying it (§ 8.2 Resolving an import). A data block using a dictionary uses the dictionary block at dictionary_index within the segment.
The file content of a leaf block for an output index i is:
-
Its stored content, if it is a data block. i MUST be 0.
-
The output at the index i of evaluating it, if it is a formula block. i MUST be less than its num_outputs.
6. Formulas
6.1. Formula block
Blocks 7016 through 7716 are formula blocks,
whose type is 0111 0dss in binary (§ 3.1.1 Block types).
d determines whether a dictionary is used (has_dictionary): false if 0, and true if 1.
Its data consists of the following:
| Name | Type | Conditions |
|---|---|---|
| dictionary_index | leb128 | If has_dictionary |
| num_outputs | leb128 | |
| formula_data | compressed_data |
If a dictionary is in use, its 0-based index is specified as dictionary_index, as in data blocks.
num_outputs is the number of outputs of the formula block, which MUST be at least 1 (§ 6.3 Formula evaluation).
A formula block describes the content of a file as an expression over non-leaf items following it, evaluated with modules and resources preceding it. The internal structure of formula_data is described in Formula data (§ 6.2 Formula data).
6.2. Formula data
This section describes the internal structure of formula_data in formula blocks (§ 6.1 Formula block), after it has been decompressed.
A formula consists of the lists of blocks it references, followed by instructions of a stack machine. Each instruction pops zero or more byte sequences from the stack and pushes zero or more byte sequences, and the byte sequences remaining on the stack after the last instruction are the outputs of the formula.
| Name | Type |
|---|---|
| num_blocks | leb128 |
| block_ordinal_deltas | leb128[num_blocks] |
| num_resources | leb128 |
| resource_indices | leb128[num_resources] |
| num_modules | leb128 |
| module_indices | leb128[num_modules] |
| instructions | instruction[*] |
block_ordinal_deltas lists the non-leaf items referenced by the formula block. Despite their names, num_blocks, block_ordinal_deltas and slice_block count non-leaf items, which are blocks except for imports. Their 0-based ordinals among the non-leaf items supplied by the non-leaf blocks following the formula block within the segment, in the order of appearance, are o0 = block_ordinal_deltas[0] and oi = oi - 1 + block_ordinal_deltas[i] + 1 for i ≥ 1, so the ordinals are strictly increasing. The i-th referenced item is the non-leaf item with the ordinal oi, which MUST exist.
resource_indices and module_indices list the resource and module blocks referenced by the formula block, as 0-based indices within the segment (§ 4.2 Content blocks). Each of them MUST refer to an existing block, which precedes the formula block by § 4.2.1 Block order.
instructions fill the rest of formula_data. Each item or block listed in the above lists MUST be used by at least one instruction, and is said to be referenced by the formula block only in that case.
Each instruction consists of an opcode of type u8 followed by its operands, whose structure is determined by the opcode as described in the following sections. opcode MUST be one of the following:
| Opcode | Instruction | Pops | Pushes |
|---|---|---|---|
| 0016 | Nop | 0 | 0 |
| 0116 | Literal | 0 | 1 |
| 0216 | ModuleBytes | 0 | 1 |
| 0316 | ResourceBytes | 0 | 1 |
| 0F16 | Swap | 0 | 0 |
| 1016 | Slice | 0 | 1 |
| 1216 | Concatenate | concat_count | 1 |
| 1316 | Pack | pack_count | 1 |
| 1416 | Split | 1 | split_count |
| 1516 | Patch | 2 | 1 |
| 1616 | Apply | apply_num_inputs | apply_num_outputs |
| 2016 | Checksum | 0 | 0 |
Opcodes are grouped by their roles: 0016 through 0F16 push bytes as they are, in the formula itself or in the blocks it lists, or rearrange the stack, 1016 through 1F16 compute byte sequences from referenced items or the stack, and 2016 through 2F16 check them. Other opcodes are reserved. Opcode 1116 is reserved as well: it was an earlier Apply instruction, which gave no task limits, and a decoder MUST NOT give it another meaning. A formula with the opcode 1116 is an invalid archive failure.
A reference to a preceding block in the same segment, e.g. to patch a file stored earlier in the segment, is not yet defined. A reference to another file is made by an import (§ 8.1 Import block) rather than by an instruction, and the content of a file in preceding segments by an internal import. The candidate is a second list of non-leaf items preceding the formula block, read by Slice instructions as referenced items are, where the instructions of another formula block are copied when its output is needed, instead of referring to the output. It requires a bound on the distance of such a reference declared before the referenced blocks, so that a streaming decompressor knows which blocks to retain, and a way to locate a preceding block without reading the segment from its beginning.
formula_data is not randomly accessible, and a decompressor is expected to read and decompress it in its entirety before evaluating any part of the formula. Every requirement on formula_data that doesn’t depend on the evaluated byte sequences, such as indices into the lists and the number of byte sequences on the stack, can be checked in a single pass over instructions without evaluating anything. The same pass determines the length of every byte sequence on the stack, so formula_data then serves as an index for random access within the output, in the same way as a seek table of other formats. The referenced items are located by skipping blocks following the formula block by their size fields, and an item of an import block by decoding that block.
To execute an instruction on a stack, which is a list of byte sequences whose last item is the top, perform the steps described for its opcode in the following sections. To pop n items, the stack MUST have at least n items, and the last n items are removed from the stack and used in their order in the stack.
6.2.1. Nop instruction
Opcode 0016 denotes a Nop instruction, which has no operands and does nothing.
6.2.2. Literal instruction
Opcode 0116 denotes a Literal instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| literal_size | leb128 |
| literal_data | u8[literal_size] |
It pushes literal_data.
6.2.3. ModuleBytes instruction
Opcode 0216 denotes a ModuleBytes instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| module_bytes_module | leb128 |
| module_bytes_size | leb128 |
-
module_bytes_module MUST be less than num_modules.
-
Let b be the module block at module_indices[module_bytes_module]. b MUST NOT be a shorthand block.
-
The length of data of b MUST be module_bytes_size.
-
Push data of b.
6.2.4. ResourceBytes instruction
Opcode 0316 denotes a ResourceBytes instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| resource_bytes_resource | leb128 |
| resource_bytes_size | leb128 |
-
resource_bytes_resource MUST be less than num_resources.
-
Let b be the resource block at resource_indices[resource_bytes_resource].
-
The length of data of b MUST be resource_bytes_size.
-
Push data of b.
The instructions neither decode nor apply the block: a module referenced only by ModuleBytes instructions isn’t applied, so a conformance level doesn’t allow refusing it, and the checksum of the block covers its decoded data, which the instructions don’t compute (§ 13.2 Partial access). The pushed bytes are covered instead by a Checksum instruction covering a byte sequence computed from them, typically an output checksum. The size is given as an operand, as the sizes of Slice and Patch instructions are, so that the length of every byte sequence on the stack is known from the formula alone.
6.2.5. Swap instruction
Opcode 0F16 denotes a Swap instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| swap_depth | leb128 |
The stack MUST have more than swap_depth items. It exchanges the top of the stack with the item swap_depth items below it.
6.2.6. Slice instruction
Opcode 1016 denotes a Slice instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| slice_block | leb128 |
| slice_start | leb128 |
| slice_length | leb128 |
-
slice_block MUST be less than num_blocks.
-
Let c be the stored content of the slice_block-th referenced item.
-
slice_start + slice_length MUST be at most the length of c.
-
Push the bytes of c in [slice_start, slice_start + slice_length).
6.2.7. Apply instruction
Opcode 1616 denotes an Apply instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| apply_module | leb128 |
| apply_num_resources | leb128 |
| apply_resources | leb128[apply_num_resources] |
| apply_params_size | leb128 |
| apply_params | u8[apply_params_size] |
| apply_memory_limit | leb128 |
| apply_num_task_limits | leb128 |
| apply_task_limits | leb128[apply_num_task_limits] |
| apply_num_inputs | leb128 |
| apply_num_outputs | leb128 |
| apply_output_sizes | leb128[apply_num_outputs] |
-
apply_module MUST be less than num_modules, and each item of apply_resources MUST be less than num_resources.
-
apply_num_outputs MUST be at least 1. apply_num_task_limits MUST be the number of classes of the module.
-
Pop apply_num_inputs items as inputs.
-
Let outputs be the result of applying the module block at module_indices[apply_module] with inputs, the resource blocks at resource_indices[r] for each r of apply_resources, apply_params, apply_output_sizes, apply_memory_limit and apply_task_limits as its task limits (§ 7.4 Module execution).
-
Push each item of outputs in order.
6.2.8. Concatenate instruction
Opcode 1216 denotes a Concatenate instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| concat_count | leb128 |
It pops concat_count items and pushes their concatenation in order.
6.2.9. Pack instruction
Opcode 1316 denotes a Pack instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| pack_count | leb128 |
It pops pack_count items and pushes the packed byte sequence of them in order (§ 6.3.1 Packed byte sequences).
6.2.10. Split instruction
Opcode 1416 denotes a Split instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| split_count | leb128 |
| split_sizes | leb128[split_count] |
-
split_count MUST be at least 1.
-
Pop an item x. The sum of split_sizes MUST be equal to the length of x.
-
For each n of split_sizes in order, push the next n bytes of x.
6.2.11. Patch instruction
Opcode 1516 denotes a Patch instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| patch_size | leb128 |
-
Pop 2 items as source and patch.
-
patch MUST consist of one or more Zstandard blocks without any other bytes, where only the last block is signalled as the final block by its block header, and each block holds at most 128 KiB of data.
-
Let output be the result of decompressing patch as a sequence of Zstandard blocks [RFC8878] with source as the
Contentof a raw content dictionary, i.e. the entropy tables start empty and the repeat offsets start at their initial values (1, 4, 8). The offset of every match MUST be at most the length of source plus the number of bytes already decompressed. -
The length of output MUST be patch_size.
-
Push output.
zstd --patch-from does, so a decompressor reuses its Zstandard decompressor for it.
The earlier byte sequence is an operand on the stack rather than a dictionary block,
so that it can be any byte sequence computed by the formula,
e.g. an import or a stream split from it by a module.
The patch itself is typically a slice of an uncompressed non-leaf data block,
whose checksum covers the stored patch, while an output checksum covers the result.
Window_Size:
a match may refer to any byte of source or of the output so far,
and the guaranteed limit of Window_Size in § 3.2.1 Zstandard compression method doesn’t apply.
The decompressor holds both byte sequences anyway, so a long reach costs no additional memory.
With libzstd, this is decompressing a frame after ZSTD_DCtx_refPrefix with source,
whose release builds only check offsets against the start of the prefix;
a decompressor enforcing Window_Size has to lift that check for the frame.
A compressor using libzstd sets ZSTD_c_windowLog large enough to cover source and the output,
as zstd --patch-from does.
01 00 00), of an empty output.
6.2.12. Checksum instruction
Opcode 2016 denotes a Checksum instruction. Its operands consist of the following:
| Name | Type |
|---|---|
| checksum_value | u64le |
The stack MUST NOT be empty. The XXH64 checksum of the top of the stack MUST be equal to checksum_value.
A Checksum instruction covers the byte sequence at the top of the stack when it is executed, and the byte sequence remains the same item when moved by Swap instructions. An output checksum is a Checksum instruction covering an output of the formula block. An output is checksummed if an output checksum covers it.
6.3. Formula evaluation
To evaluate a formula block block:
-
Let data be the decoded data of formula_data of block, parsed as in § 6.2 Formula data.
-
Let stack be an empty list of byte sequences.
-
For each insn of instructions of data in order, execute insn on stack as described for its opcode in § 6.2 Formula data.
-
The number of items of stack MUST be num_outputs of block.
-
Return stack as the outputs of block, numbered from 0 in order from the bottom of the stack.
6.3.1. Packed byte sequences
The packed byte sequence of a list of byte sequences items is the concatenation of the following, where n is the number of items:
-
n as u64le;
-
For each i in [0, n), the sum of the lengths of items[0] through items[i] as u64le;
-
Each item of items in order.
A byte sequence p is a packed byte sequence of exactly one list items, if any. To unpack p:
-
If p is shorter than 8 bytes, return failure.
-
Let n be the first 8 bytes of p as u64le, and let h be 8 × (n + 1). If h exceeds the length of p, return failure.
-
Let ei be the 8 bytes of p from the offset 8 × (i + 1) as u64le for each i in [0, n), and let e-1 be 0.
-
If ei - 1 > ei for any i in [0, n), return failure.
-
If h + en - 1 is not the length of p, return failure.
-
Return the list of the bytes of p in [h + ei - 1, h + ei) for each i in [0, n).
In the other direction, a module producing a list of byte sequences outputs their plain concatenation, which a Split instruction separates by the sizes declared in the formula. The output of such a module is passed to another module taking a packed byte sequence by a Split instruction followed by a Pack instruction.
7. Modules
7.1. Module block
Blocks 6016 through 6316 are module blocks,
whose type is 0110 00ss in binary (§ 3.1.1 Block types).
Their data is checksummed Zstandard data (§ 3.2 Shared external structure),
except that block 6016 is a shorthand block for a registered module (§ 4.2.2 Registered dictionaries and modules).
The internal structure after decompression gives a WebAssembly module in the binary format [WASM-CORE-3], wasm_module, in one of two forms told by its first byte:
-
If the first byte is 0016, the decoded data is the stored form: wasm_module is the whole decoded data, i.e. the content of an ordinary
.wasmfile, starting with the magic00 61 73 6Dand the version01 00 00 00(hexadecimal). -
If the first byte is 0D16, the decoded data is the split form (§ 7.2 Split modules), and wasm_module is joined from it.
-
Otherwise, including when the decoded data is empty, an invalid archive failure occurs.
wasm_module MUST decode and validate as a module of the features allowed in § 7.4 Module execution,
and MUST conform to the profile and the interface given there;
otherwise an invalid archive failure occurs.
In short, the module defines no memory of its own and imports every memory and host function from the module name aqr,
has no start function, and exports a single entry point that the host invokes once per application.
Custom sections are allowed and don’t affect the result.
The stored form adds nothing around the module, so that the decoded data can be inspected and run by any WebAssembly tool.
A module block for a registered module has the same wasm_module as the entry, whatever its form,
and a shorthand block stands for one of the stored form.
The module hash of a module block is the SHA-512/256 hash [FIPS180-4] of its wasm_module, which is also the registered hash of a registered module (§ 4.2.2 Registered dictionaries and modules). Since every application of a module depends only on wasm_module and its arguments (§ 7.4 Module execution), a decompressor MAY apply a module whose module hash it knows with an implementation of its own (e.g. native code) instead of instantiating wasm_module, as long as the result is indistinguishable from the definition, including invalid archive failures. That includes the memory limits an application declares (§ 7.4 Module execution): an implementation of its own MUST fail an application with a resource failure where the module’s memories would exceed them, which it can tell from the application’s memory requirements before it runs (or else by applying the module with an engine, where the limits are enforced as it runs). Such a decompressor needs neither the bytes of the module nor a WebAssembly engine, whether the module is given by a shorthand block or in full. A decompressor MAY refuse to apply a module as its conformance level allows (§ 13.1 Conformance levels), in which case a resource failure occurs.
-
apply an implementation of its own, which is the fastest but has to be made for every module known. Translating the module by automated means (e.g. wasm2c) is preferred to writing it anew, since the result has to be indistinguishable from the module, including its failures; or
-
instantiate the module with a WebAssembly engine without isolating it, since the hash is strong enough that no other module can have it. This needs only the hash for a module given in full, and the bytes only for a shorthand block.
For a module whose module hash it doesn’t know, it can either:
-
refuse to apply it, failing with a resource failure, unless it is at level 2; or
-
instantiate it with a WebAssembly engine in a sandbox, as untrusted code (§ 14 Security considerations).
For example, a decompressor at level 1 needs neither an engine nor a sandbox if it implements every registered module natively, and one at level 2 needs both but no native implementation.
7.2. Split modules
The split form of a module block keeps wasm_module as 14 streams, in which the instructions of its code are split by field, and its data segments' bytes and its custom sections are apart from the rest. Every byte of wasm_module comes back from the streams as it was, LEB128s of any length included, but for the sizes of the sections and the functions split and for the local indices, which are written in the shortest form. Its structure is as follows:
| Name | Type |
|---|---|
| count | u8 |
| lengths | leb128[count] |
| payload | u8[*] |
count MUST be 13 (0D16), one less than the number of the streams. The stream si for each i in [0, 13) is the lengths[i] bytes of payload after those of s0 through si - 1, and the stream s13 is the rest of payload. If the sum of lengths exceeds the length of payload, an invalid archive failure occurs. The streams are named as follows:
| i | Name | Holds |
|---|---|---|
| 0 | skeleton
| the header; the sections' ids, sizes and counts; how each function is kept |
| 1 | sections
| the sections kept as they are but custom sections, and the functions kept as they are |
| 2 | custom
| the custom sections' contents |
| 3 | locals
| the functions' declarations of locals |
| 4 | op
| opcodes, memory arguments' flags and the first bytes of their offsets, the immediates of i32.const and call
|
| 5 | local
| the codes of local indices |
| 6 | label
| the labels of br and br_if
|
| 7 | brtable
| the labels of br_table
|
| 8 | blocktype
| block types |
| 9 | offset
| memory arguments' offsets but their first bytes |
| 10 | i64
| the immediates of i64.const
|
| 11 | bytes
| float and vector constants, shuffle lanes, lane indices |
| 12 | misc
| the other immediates, and the data segments but their bytes |
| 13 | data
| the data segments' bytes |
wasm_module is joined from the streams by reading each from its start, each read taking its next bytes, and appending to the output, initially empty, as follows. The following terms are used:
-
A *LEB128 of at most m bytes* of a stream is its bytes up to and including the first byte without the MSB set, which MUST be within the m bytes; its value is that of the bits of its groups as for leb128, where the groups beyond the 64th bit are dropped. It doesn’t have to be in the shortest form. L32 denotes 5, and L64 denotes 10.
-
To copy bytes or a LEB128 of a stream is to read them and append them to the output as they are. A LEB128 of s without a bound is a LEB128 of at most L32 bytes of s.
-
To write an integer is to append its leb128 in the shortest form.
-
unzz(v) is v / 2 if v is even, and -(v + 1) / 2 otherwise.
-
Copy 8 bytes of
skeleton. They MUST be00 61 73 6D 01 00 00 00(hexadecimal). -
While
skeletonhas bytes left, join a section:-
Copy a byte id of
skeleton. -
If id is 10 (the code section) or 11 (the data section), read a byte k of
skeleton, which is not copied and MUST be 0 or 1. If k is 0, join the content of the section (§ 7.2.1 Code and data sections), and continue to the next section. -
Copy a LEB128 n of
skeleton, then n bytes ofcustomif id is 0, or ofsectionsotherwise.
-
-
Every stream MUST have no bytes left.
Reading past the end of a stream, a LEB128 not within its bound, a byte not allowed, or any other condition stated with MUST not holding, results in an invalid archive failure.
7.2.1. Code and data sections
The content of a code or data section split is joined to a separate byte sequence c, which is then appended to the output preceded by its length written. The length MUST be less than 232. c is joined as follows:
-
Copy a LEB128 count of
skeleton. -
Repeat count times:
-
For the code section (id = 10), read a byte f of
skeleton, which is not copied and MUST be 0 or 1.-
If f is 0, join a function body to a separate byte sequence b, and append b preceded by its length written, which MUST be less than 232. The body is a LEB128 n of
localscopied, then n times a LEB128 oflocalsand a byte oflocalscopied, then an expression (§ 7.2.2 Split expressions). -
If f is 1, copy a LEB128 n of
sections, then n bytes ofsections.
-
-
For the data section (id = 11), copy a data segment: a LEB128 kind of
misc; if kind is 2, a LEB128 ofmisc; unless kind is 1, an expression (§ 7.2.2 Split expressions); then a LEB128 n ofmiscand n bytes ofdata.
-
7.2.2. Split expressions
An expression is joined an instruction at a time, keeping a nesting depth d, initially 0,
and the recent locals (a list of integers, initially empty) and next (initially 0) made anew for the expression,
until its end.
An instruction is its opcode, a byte of op copied, then its immediates as the following table tells, copied in order.
A byte not in the table is not allowed.
| Opcode | Immediates |
|---|---|
| 02, 03, 04 | a LEB128 of blocktype; then d increases by 1
|
| 0B | none; the expression ends if d is 0, and d decreases by 1 otherwise |
| 0C, 0D | a LEB128 of label
|
| 0E | a LEB128 n of brtable, then n + 1 LEB128s of brtable
|
| 10, 12 | a LEB128 of op
|
| 11, 13 | two LEB128s of misc
|
| 1C | a LEB128 n of misc, then n bytes of misc
|
| 20, 21, 22 | a local index (see below) |
| 23–26, 3F, 40, D0, D2 | a LEB128 of misc
|
| 28–3E | a memory argument (see below) |
| 41 | a LEB128 of op
|
| 42 | a LEB128 of at most L64 bytes of i64
|
| 43 | 4 bytes of bytes
|
| 44 | 8 bytes of bytes
|
| 00, 01, 05, 0F, 1A, 1B, 45–C4, D1 | none |
| FC | a LEB128 x of op, then by x:
none for 0–7;
two LEB128s of misc for 8, 10, 12 and 14;
a LEB128 of misc for 9, 11, 13 and 15–17;
any other x is not allowed
|
| FD | a LEB128 x of op, then by x:
a memory argument for 0–11, 92 and 93;
16 bytes of bytes for 12 and 13;
a byte of bytes for 21–34;
a memory argument then a byte of bytes for 84–91;
none for 14–20, 35–83 and 94–255;
any other x is not allowed
|
Opcodes are hexadecimal, and x is decimal.
A memory argument is a LEB128 a of op;
if bit 6 of a (26) is set, a LEB128 of misc;
then the offset, a LEB128 of at most L64 bytes whose first byte is of op
and, if the MSB of that byte is set, whose other bytes (at most 9) are of offset.
A local index i is coded by a LEB128 c of at most L64 bytes of local,
which is read and not copied.
Let r be the length of the recent locals:
-
If c < r, i is the item of the recent locals at c (from 0). Otherwise, i is next + unzz(c - r), which MUST be in [0, 232).
-
Write i.
-
Remove i from the recent locals if there, insert it at the front, and remove the items after the first 256. Then next becomes the greater of next and i + 1.
wasm-recons@1 in order (§ 15.6 Aqueous Modules Registry),
which rebuilds a module the same way and checks its declared length too,
so a compressor can make both from the same splitting.
A compressor splits only the parts it can walk to their ends:
a section or a function whose size or local indices aren’t in the shortest form, or with an opcode not in the table,
is kept as it is, its bytes in sections.
The walk takes any byte the table allows, whether WebAssembly does or not;
wasm_module has to validate all the same.
The split form has no length of wasm_module,
but wasm_module is at most five times as long as payload:
every byte is copied once, but for a byte of local or of skeleton (k, f)
each giving at most five bytes written.
A local index is coded by its rank among those last used, so that the same fragment of code in two functions is coded the same even where they number their locals differently. Over the modules of an implementation (3.6 MB, each module compressed alone by Zstandard at level 22), the split form saves about 16% of the module blocks' data.
7.3. Resource block
Blocks 6416 through 6716 are resource blocks,
whose type is 0110 01ss in binary (§ 3.1.1 Block types).
Their data is checksummed Zstandard data (§ 3.2 Shared external structure).
The internal structure after decompression is an uninterpreted byte sequence, which is supplied to modules by formula blocks referencing it (§ 6.2 Formula data, § 7.4 Module execution).
7.4. Module execution
A module is a WebAssembly module [WASM-CORE-3] stored as wasm_module, instantiated and invoked through the embedding interface defined in the Appendix of [WASM-CORE-3]. It MUST conform to the following profile, or an invalid archive failure occurs:
-
WebAssembly 3.0 core, executed under the deterministic profile; every NaN result is a canonical NaN.
-
None of the features added in WebAssembly 3.0 are allowed, except multiple memories and 64-bit addresses of the memories other than the working memory. That is, tail calls, extended constant expressions, typed function references, garbage collection, exception handling, relaxed SIMD, 64-bit tables and a 64-bit working memory are not allowed.
-
Imported tables are not allowed.
-
It has no start function.
-
It defines no memory, and imports each memory at most once.
-
Every
memory.growinstruction targets the working memory. -
The import type of every memory other than the working memory has 64-bit addresses, the minimum 0 and no maximum, except that the import type of an input or output memory of a part module MAY have a maximum of at least 1, its window in pages.
-
It exports exactly one of the functions
aqr_apply_wholeandaqr_apply_part, the entry point. The module is a whole module in the former case and a part module in the latter. -
If it imports
aqr_spawn, it exports the functionaqr_task, the task entry point, which has two parameters of typei32and no results, and it has exactly one custom section namedaqr_task_slots, whose content is one or more bytes: the c-th of them (from 0) describes its class c of tasks, and their number is its number of classes. The low 7 bits of the byte are the task slots of the class, which MUST be from 1 to 64, and its top bit is set if the class is a gang class (§ 7.4.1 Tasks). Otherwise it has no custom section of that name and no classes.
Whether the guaranteed limit of 227 bytes (128 MiB) for the Zstandard Window_Size (§ 3.2.1 Zstandard compression method)
should also apply to the memory limit.
Every import of a module is from the module name aqr, under one of the following names.
The names of imports and exports are prefixed with aqr_ to avoid collisions in the toolchains and hosts
that link them into one namespace.
Any other import is an invalid archive failure.
aqr_memory-
A memory, the working memory.
aqr_params-
A memory of the parameters.
aqr_resourcei-
A memory of the i-th resource, where i is a decimal number without leading zeros.
aqr_inputk-
A memory of the k-th input, where k is a decimal number without leading zeros.
aqr_outputj-
A memory of the j-th output, where j is a decimal number without leading zeros.
aqr_request,aqr_release,aqr_commit-
Functions of a part module, defined below. A whole module MUST NOT import them.
aqr_heartbeat-
A function defined below.
aqr_spawn,aqr_result,aqr_join,aqr_take-
Functions of tasks, defined in § 7.4.1 Tasks.
aqr_chan,aqr_send,aqr_recv,aqr_close-
Functions of channels, defined in § 7.4.1 Tasks.
aqr_log-
The diagnostic function, defined below.
A memory is sized to fit a length L when both its minimum and maximum are ⌈L / 65536⌉ pages, and sized to fit L within a window of m pages when both are min(⌈L / 65536⌉, m) pages.
To apply a module module given a list of input byte sequences inputs, a list of resource blocks resources, a parameter byte sequence params, a list of declared output sizes sizes, a memory limit and a list of task limits, one for each class:
-
sizes MUST NOT be empty. The number of the task limits MUST be the number of classes of module.
-
Every
aqr_resourcei imported MUST satisfy i < the number of resources, everyaqr_inputk imported MUST satisfy k < the number of inputs, and everyaqr_outputj imported MUST satisfy j < the number of sizes. -
Create the memories imported, each initially zero-filled:
-
The working memory has the minimum size of its import type.
-
The memory of the parameters is sized to fit the length of params, and holds params from offset 0.
-
The memory of the i-th resource is sized to fit its stored content c, and holds c from offset 0.
-
The memory of the k-th input is sized to fit the length of inputs[k], within its window if its import type has one. If module is a whole module, it holds inputs[k] from offset 0. Otherwise let Wk be its size in bytes and the visible range of the k-th input be [0, 0).
-
The memory of the j-th output is sized to fit sizes[j], within its window if its import type has one. If module is a part module, let Wout,j be its size in bytes.
-
-
Let outputs be a list of as many empty byte sequences as sizes.
-
If the total size of the memories in bytes exceeds the memory limit, return a resource failure. The memory limit applies to the total size of the memories of the application’s instance on its own, and the c-th task limit (from 0) to that of each unit of class c on its own: the memories of the task for a class that is not a gang class, and the memories of the tasks of the gang and the messages queued in the channels of the gang for a gang class. An implementation MAY return a resource failure here, before it instantiates module, if the memory limit and the sum of the task limits (a concurrency of 1 for each class, the least it can choose) exceed what the implementation has, which is implementation-defined, but only for an application whose minimal bound is over 512 MiB (see below).
-
Instantiate module with the memories and the host functions below.
-
Invoke the entry point with the arguments of type
i64, in order: the length of params, the length of the stored content of each item of resources, the length of each item of inputs and each item of sizes. The entry point MUST have these parameters and no results. If it traps or any host function below signals an error, return an invalid archive failure. If it exceeds the memory limit or any implementation-defined limit, return a resource failure. If any task of the application fails, return its failure (that of any of them if several fail). If every instance of the application waits, return an invalid archive failure. Once it returns, every task it spawned MUST be taken, and every channel it made MUST have been given to a gang. -
If module is a whole module, set each outputs[j] to the first sizes[j] bytes of the memory of the j-th output, if imported, and to sizes[j] zero bytes otherwise.
-
The length of each outputs[j] MUST be sizes[j].
-
Return outputs.
A memory.grow instruction exceeds the memory limit when the total size to which that limit applies would exceed it
(the memory limit, or the task limit of the class of the task executing it, for the unit it belongs to; see step 5 above),
and MUST succeed otherwise if the maximum of the import type of the working memory allows.
The minimal bound of an application is its memory limit plus the sum of its task limits: the most its memories come to at a concurrency of 1 for each class. A decoder MUST support every application whose minimal bound is at most 512 MiB (229 bytes): step 5’s resource failure before instantiating is only for a larger one. An encoder SHOULD NOT produce an application whose minimal bound is more.
Note: The reference encoder’s default bound is 256 MiB, so that a decoder may run applications side by side; it falls back, for the data concerned, to what it archives without the recompressor whose application would exceed it; an option raises the bound for an archive whose decoders are known to have the memory.
Host functions are defined as follows, where lok and hik denote the visible range of the k-th input,
k, j, ptr and len are of type i32, and every offset is of type i64.
Called by a task, aqr_request, aqr_release and aqr_commit signal an error.
- aqr_request(k, hi)
-
The k-th input MUST be imported. hi MUST be in [hik, length of inputs[k]], and hi - lok MUST be at most Wk. For each offset o in [hik, hi), write inputs[k][o] to the memory of the k-th input at o % Wk. Then set hik to hi.
- aqr_release(k, lo)
-
The k-th input MUST be imported. lo MUST be in [lok, hik]. Set lok to lo.
- aqr_commit(j, hi)
-
The j-th output MUST be imported. hi MUST be in [length of outputs[j], sizes[j]], and hi - length of outputs[j] MUST be at most Wout,j. For each offset o in [length of outputs[j], hi), append the byte at o % Wout,j of the memory of the j-th output to outputs[j].
- aqr_heartbeat()
-
Does nothing.
- aqr_log(ptr, len)
-
Does nothing. Implementations MAY present the len bytes of the working memory from the offset ptr as a diagnostic message, which is untrusted data from the archive, but doing so MUST NOT affect the application, even when the range is out of bounds. The bytes are a UTF-8 [RFC3629] string, with no terminator: a module SHOULD write well-formed UTF-8, and an implementation presenting them MUST read them as UTF-8, replacing each ill-formed subsequence with U+FFFD, never as another encoding. It is the only diagnostic function; a module reports a failure by
aqr_logfollowed by a trap.
A violation of any MUST in host functions is signalled as an error to the module invocation.
7.4.1. Tasks
A task is an instance of the module of its own, of a class, run on the bytes of the working memory of the instance that spawns it that the spawn names and giving back a byte sequence, its result. The application’s instance and every task are the instances of the application; each instance has a list of handles, initially empty, each of which is a list of the tasks spawned at once. A task is either pending, running, done (with a result), failed (with an invalid archive failure or a resource failure), or taken. The application’s instance is of no class: the classes, numbered from 0, are those of tasks, and the memory limit, not a task limit, applies to the instance. An instance that is a task of class c spawns only tasks of a class above c, and the application’s instance tasks of any class.
The tasks spawned at once by one aqr_spawn of a gang class are a gang. A unit is a task of a class that is not a gang class, and a gang otherwise; it is pending until it is started, and then running until all of its tasks are done or failed. The tasks of a gang are started together: all of them or none. A pending task is not yet an instance of the application: it has no memories, and it does not wait.
The units of each class start in the order they were spawned over the application (by handle, then by index; no unit overtakes another), and an implementation chooses, for each class c, its concurrency, any number from 1 to the task slots of c: whenever fewer units of class c than that are running, the first of them pending is started. A running task stops running when it is done or failed.
A channel links instances of an application: a queue of messages (byte sequences) of at most its capacity, whether it is closed, its sender and receiver, instances initially none, and its owner, the gang it belongs to, initially none. The application has a list of channels, initially empty. Each instance has a list of the channels it made that have not yet been given to a gang, initially empty. When an instance ends (returns or fails), every channel of which it is the sender is closed. An instance waits while it is within aqr_join, aqr_send or aqr_recv and the condition it waits for does not hold.
To run a task given a list of regions (each a byte offset d and a byte sequence B), arg and i:
-
Create the memories imported: the working memory has the larger of the minimum of its import type and ⌈E / 65536⌉ pages, E the largest d + B.length of the regions (0 for none), and the maximum of its import type; every other memory has the minimum and maximum 0.
-
Instantiate the module with the memories and the host functions, and then, for each region (d, B) in order, write B to the working memory from the offset d.
-
Invoke the task entry point with arg and i. If it traps or any host function signals an error, the task fails with an invalid archive failure. If it exceeds the memory limit or any implementation-defined limit, the task fails with a resource failure.
-
Once it returns, every task it spawned MUST be taken and every channel it made MUST have been given to a gang, or the task fails with an invalid archive failure.
-
The task is done, with the result last set by aqr_result, or an empty byte sequence if none.
The host functions of tasks are defined as follows, for the instance I calling them,
where arg, n, h, i, ptr, dst and len are of type i32, unsigned:
- aqr_spawn(class, count, records, arg, n)
-
class MUST be less than the number of classes of the module, and above the class of I where I is a task. records + 12 × count MUST be at most the size of the working memory of I. Read count region records from the working memory of I from the offset records, each three u32les: src, len and dst, of which src + len MUST be at most the size of the working memory and dst + len at most 232; let the regions be, for each in order, (dst, the len bytes of the working memory of I from the offset src). If class is a gang class, the channels in the list of I of those not yet given are given to the gang of this spawn: set their owner to it, and empty the list. Otherwise that list MUST be empty. Append to the handles of I a list of n new pending tasks of class class, the i-th (from 0) of which is to run a task given the regions, arg and i once it is started (as the concurrency of the class allows), and return the index of that list among the handles of I.
- aqr_result(ptr, len)
-
I MUST be a task, and ptr + len MUST be at most the size of its working memory. Set the result of I to the len bytes of its working memory from the offset ptr.
- aqr_join(h, i)
-
The i-th task of the h-th handle of I MUST exist and not be taken. Wait until it is done or failed. If it failed, the instance I fails with its failure. Otherwise return the length of its result.
- aqr_take(h, i, dst, len)
-
The i-th task of the h-th handle of I MUST be done (joined), len MUST be at most the length of its result, and dst + len MUST be at most the size of the working memory of I. Copy the first len bytes of its result to the working memory of I from the offset dst, and the task is taken.
The host functions of channels are defined as follows, where c, ptr, dst, len and cap are of type i32, unsigned,
the channel C is the c-th (from 0) of the application, which MUST exist,
and taking an end of C for I means:
the owner of C MUST NOT be none, and I MUST be a task of it or the instance that spawned it;
the other end of C MUST NOT be I;
and if that end is none, set it to I, and otherwise it MUST be I.
A task of a class that is not a gang class therefore takes no end of a channel.
- aqr_chan(cap)
-
cap MUST be at least 1. Append to the channels of the application a new channel of capacity cap, and its index to the list of channels of I not yet given, and return its index.
- aqr_send(c, ptr, len)
-
Take the sender of C for I. C MUST NOT be closed, and ptr + len MUST be at most the size of the working memory of I. Wait until fewer messages than the capacity of C are queued, and then enqueue the len bytes of the working memory of I from the offset ptr.
- aqr_recv(c, dst, cap)
-
Take the receiver of C for I. Wait until a message is queued or C is closed. If no message is queued, return −1 (0xFFFFFFFF). Otherwise let M be the first message. If the length of M exceeds cap, return it, leaving M queued. Otherwise dst + the length of M MUST be at most the size of the working memory of I; dequeue M, copy it to the working memory of I from the offset dst, and return its length.
- aqr_close(c)
-
Take the sender of C for I, and close C.
aqr_spawn of two parameters, for the modules they know; this is outside this specification.
Whether the memories other than the working memory, read-only, should be visible to tasks, which would require rejecting the instructions writing them; and the windows of a part module in tasks.
8. Overlay archives and imports
8.1. Import block
Blocks 5816 through 5F16 are import blocks,
which are non-leaf blocks supplying byte ranges of files outside the segment.
Their type is 0101 1ess in binary (§ 3.1.1 Block types),
and their data is checksummed Zstandard data (§ 3.2 Shared external structure).
e determines the source (external): preceding segments if 0, and the base if 1.
An import block is an internal import block if external is false, and an external import block otherwise. An external import block MUST NOT occur in an archive other than an overlay archive (§ 3.5 Unencrypted archive).
The internal structure after decompression is as follows:
| Name | Type |
|---|---|
| num_imports | leb128 ≠ 0 |
| imports | import[num_imports] |
Each import consists of the following:
| Name | Type |
|---|---|
| import_path | u8[*] ≠ 0 |
| import_path_sentinel | u8 = 0 |
| import_offset | leb128 |
| import_size | leb128 |
| import_checksum | u8[32] |
An import in an internal or external import block is an internal import or an external import respectively. Each import supplies a single non-leaf item, so the import block supplies num_imports items in the order of imports. The stored content of the item is the imported content of the import, given by resolving it, the bytes in [import_offset, import_offset + import_size) of the file found at import_path:
-
An internal import finds the file in the preceding result of its segment, i.e. the result of applying all preceding segments to the base, which is empty for a standalone archive.
-
An external import finds the file in the base as given.
import_path is a sequence of zero or more components separated by 2F16 (/).
Each component uses the same encodings as entry_name,
and its decoded name is subject to the same restrictions as an entry name, so no component can be empty.
An empty import_path denotes the root itself, which is a file when the base is a single file.
import_checksum is the SHA-512/256 hash [FIPS180-4] of the imported content.
/ or be . or .., splitting import_path is unambiguous,
and no path can refer to anything outside the root.
A path is relative to the root rather than to the directory of the file using it,
because a formula block referencing an import is evaluated before the metadata block names that file,
and a single leaf block may be shared by files in different directories.
The prefixes shared by paths in the same block compress well.
8.2. Resolving an import
To resolve an import import of an import block in a segment:
-
Split import.import_path at every 2F16 into components, unless it is empty, and let names be the result of decoding each component as an entry name. If any component fails to decode, return an invalid archive failure.
-
Let node be the preceding result of the segment if import is an internal import, or the root of the archive base otherwise.
-
For each name of names in order:
-
node MUST be a directory node, and node.children[name] MUST exist.
-
Set node to node.children[name].
-
-
node MUST be a file node.
-
import.import_offset + import.import_size MUST be at most the length of node.content.
-
Let content be the bytes of node.content in [import.import_offset, import.import_offset + import.import_size).
-
The SHA-512/256 hash [FIPS180-4] of content MUST be equal to import.import_checksum.
-
Return content as the imported content of import.
A violation of any MUST in this algorithm is a base failure in an overlay archive, and an invalid archive failure in a standalone archive.
8.3. Reading a base
Reading a base from a file system is outside the archive model, as is extraction. A decompressor doing so maps a directory or a single file to an archive value, and is subject to the following requirements:
-
It MUST NOT use a base unless explicitly given one, and in particular MUST NOT use the current directory or the destination directory as a base by default.
-
It MUST NOT read anything outside the given directory, including through symbolic links or other redirections already present in the base. A symbolic link in the base is a link node and is never followed, so an import whose path leads through or to a link fails to resolve (§ 8.2 Resolving an import).
-
It MUST NOT read a file other than a regular file as the content of a file node. A FIFO or a device file in the base is a special node.
-
It SHOULD map a file system name to an entry name as the reverse of the mapping in § 12 Extraction. If a name can’t be mapped, or multiple names map to the same entry name, it MUST either fail or report it to the user.
-
The base MUST appear the same throughout the interpretation, as if it were read at a single point in time. A decompressor MAY read only the parts of the base needed by imports and by the result, and the checksums of imports detect a part that changed while reading.
A decompressor MAY write the result of an overlay archive into its base directory, i.e. apply it in place. In that case:
-
The result MUST be the same as if every imported content were read before anything is written, e.g. by writing new files under temporary names and renaming them after all of them are written.
-
It MAY remove a node of the base only when the node is absent from the result, and SHOULD leave every node not given by any segment as it is.
9. Encrypted archives
An archive might have been encrypted, in which case it consists of a block-like header of type ENCRYPTED_HEADER_TYPE followed by encrypted chunks. An encrypted archive is mirrored if it starts with ENCRYPTED_MIRRORED_HEADER_MAGIC, in which case copies of its header are placed between encrypted chunks.
| Name | Type | Conditions |
|---|---|---|
| checksum_start | offset | |
| encrypted_header_magic | u8[8] ∈ {ENCRYPTED_HEADER_MAGIC, ENCRYPTED_MIRRORED_HEADER_MAGIC} | |
| lg_work_factor | u8 ∈ [0C16, 3F16) | |
| scrypt_ephemeral_secret | u8[16] | |
| encrypted_file_key | u8[32] | |
| age_header_mac | u8[32] | |
| payload_nonce | u8[16] | |
| checksum_end | offset | |
| header_checksum | u64le | If mirrored |
| chunks | u8[*] |
The 97 bytes from lg_work_factor to payload_nonce are called the header fields.
header_checksum is the XXH64 [XXHASH] hash with the seed zero of all bytes from checksum_start to checksum_end.
All fields from lg_work_factor to age_header_mac are used to reconstruct an age v1 textual file header [AGE] as follows:
age-encryption.org/v1 -> scrypt scrypt_ephemeral_secret_base64 lg_work_factor_decimal encrypted_file_key_base64 --- age_header_mac_base64
Where,
-
scrypt_ephemeral_secret_base64 is the base64 encoding of scrypt_ephemeral_secret without padding ([RFC4648], Section 4);
-
lg_work_factor_decimal is the decimal representation of lg_work_factor without leading zeros;
-
encrypted_file_key_base64 is the base64 encoding of encrypted_file_key without padding;
-
age_header_mac_base64 is the base64 encoding of age_header_mac without padding.
The age payload consists of payload_nonce followed by the encrypted chunks in chunks and is interpreted in accordance with the age v1 format. Every encrypted chunk except the last one is 65,552 bytes long, i.e. 64 KiB of plaintext followed by a 16-byte authentication tag.
If the archive is not mirrored, chunks consists of the encrypted chunks alone. Otherwise chunks also contains mirrored headers: for each encrypted chunk Cm numbered from 0, a mirrored header immediately precedes Cm if and only if m + 1 is either a power of two from 2 to 32 or a positive multiple of 64. The k-th mirrored header numbered from 1 therefore precedes Cm where m = 2k − 1 for k ≤ 6 and m = 64(k − 5) − 1 for k ≥ 6, and no mirrored header follows the last encrypted chunk. A mirrored header consists of the following:
| Name | Type |
|---|---|
| checksum_start | offset |
| encrypted_mirror_magic | u8[8] = ENCRYPTED_MIRROR_MAGIC |
| mirror_index | u32le = k |
| mirrored_fields | u8[97] = header fields |
| checksum_end | offset |
| mirror_checksum | u64le |
mirror_checksum is the XXH64 [XXHASH] hash with the seed zero of all bytes from checksum_start to checksum_end.
As with mirror blocks, the constraints on the fields of a mirrored header are requirements on conforming compressors. A mirrored header always occupies 117 bytes at its position, and its content doesn’t affect the result of interpreting an archive. A mirrored header is intact if its mirror_checksum verifies and its fields other than mirrored_fields satisfy the constraints above. A decompressor MUST NOT use the mirrored_fields of a mirrored header that is not intact.
The interval of 64 encrypted chunks follows the tentative distance of 4 MiB in § 10.3 Mirror block and changes with it.
The age payload MUST decrypt into a valid unencrypted archive (§ 3.5 Unencrypted archive), either a standalone archive or an overlay archive, with the following additional constraints:
-
size_hint_block is no longer optional;
-
No trailing bytes are allowed after the archive.
Conforming compressors MUST NOT encrypt two different plaintexts of an encrypted chunk with the same file key, payload_nonce and chunk number, because the age payload derives the nonce of each chunk from them and the flag of the last chunk alone, so that doing so reveals the difference of the two plaintexts and allows the chunk to be forged. In particular, an encrypted archive is not modified in place, e.g. appended to (§ 3.7 Appending to an archive); an archive changed after its chunks are encrypted is encrypted again as a whole with a new payload_nonce.
10. Integrity and recovery
10.1. Checksums
The following table summarizes every checksum defined by this specification:
| Structure | Checksum | Computed over | Presence |
|---|---|---|---|
| Metadata, dictionary, import, module and resource blocks, except shorthand blocks | XXH64 | Decoded data | Always |
| Shorthand blocks | None (the registered hash identifies the entry but is not checked) | Never | |
| Non-leaf data blocks | XXH64 | Decoded data | Always |
| Leaf data blocks | XXH64 | Decoded data | By type |
| formula_data of formula blocks | XXH64 | Decoded data | Always |
| Byte sequences in a formula | XXH64 (Checksum instruction) | The byte sequence covered | Optional |
| Imports | SHA-512/256 (import_checksum) | Imported content | Always |
| Mirror blocks | XXH64 (mirror_checksum) | Stored bytes | Always |
| Recovery blocks | XXH64 (recovery_checksum) | Stored bytes | Always |
| Slices of a recovery set | XXH64 (slice_hashes) | Stored bytes within the range | If the set has an index block |
| Header and mirrored headers of an encrypted archive | XXH64 (header_checksum, mirror_checksum) | Stored bytes | If mirrored |
The encrypted chunks of an encrypted archive are also authenticated by [AGE].
Unless explicitly requested otherwise, conforming compressors SHOULD use a checksum_method other than None for every leaf block.
Unless explicitly requested otherwise, conforming compressors SHOULD make every output of every formula block checksummed.
-
An uncompressed leaf block without a checksum holds the file content verbatim and nothing derived from it, so the file can be memory-mapped from the archive and modified in place.
-
A compressed leaf block without a checksum can be made from an existing Zstandard frame by copying its Zstandard blocks, without decompressing them to compute the checksum over the decoded data.
Neither is a concern for long-term preservation, where the archival profile requires the checksum.
10.2. Archival profile
An archive conforms to the archival profile when the content of every file is covered by a checksum:
-
Every leaf data block has a checksum_method other than None.
-
Every output of every formula block is checksummed.
-
The archive is a standalone archive.
-
No block is a shorthand block (§ 4.2.2 Registered dictionaries and modules).
A conforming compressor producing an archive for long-term preservation SHOULD conform to the archival profile, and SHOULD make it a mirrored archive. A tool verifying an archive SHOULD report whether it conforms to the archival profile.
10.3. Mirror block
A mirror block holds a verbatim copy of another block, called its target block, so that the target block can be restored when it is damaged. Mirror blocks are only defined for block types whose damage affects more than a single parcel and can be detected by reading the block alone, i.e. that have a checksum:
| Block Types | Target Block Types |
|---|---|
| C1016 — C1716 | 1016 — 1716 (§ 4.3 Metadata block) |
| C5016 — C5716 | 5016 — 5716 (§ 5.2 Dictionary block) |
| C5816 — C5F16 | 5816 — 5F16 (§ 8.1 Import block) |
| C6016 — C6316 | 6016 — 6316 (§ 7.1 Module block) |
| C6416 — C6716 | 6416 — 6716 (§ 7.3 Resource block) |
| C7016 — C7716 | 7016 — 7716 (§ 6.1 Formula block) |
As with any other block, the size field of a mirror block is determined by its own type and counts the bytes of its own data. Since a mirror block is longer than its target block, its size field may have to be wider. Therefore a mirror block is either:
-
Unwidened, where the size fields of the mirror block and its target block have the same width, and its target type is its type minus C0016; or
-
Widened, where the size field of the mirror block is one step wider than that of its target block, and its target type is its type minus C0116. Its type % 4 MUST NOT be 0.
The target block MUST be of the target type, which MUST be a block type listed above for the mirror block’s type. Whether a mirror block is widened is signalled by archive_offset.
Its data consists of the following:
| Name | Type |
|---|---|
| mirror_magic | u8[8] ∈ {MIRROR_MAGIC, OVERLAY_MIRROR_MAGIC} |
| archive_offset | leb128 |
| target_offset | leb128 |
| target_data | u8[*] |
| checksum_end | offset |
| mirror_checksum | u64le |
mirror_magic is a fixed byte sequence that allows mirror blocks to be found by scanning when the surrounding structure, such as the header, the footer or the metadata chain, is damaged. It MUST be OVERLAY_MIRROR_MAGIC in an overlay archive and MIRROR_MAGIC otherwise.
archive_offset is the offset from archive_start to the block_start of the mirror block, plus 1 if the mirror block is widened. Since the offset of every block from archive_start is even, archive_offset % 2 tells whether the mirror block is widened, and archive_offset − (archive_offset % 2) is the offset itself.
target_offset is the distance between the block_start of the target block and the block_start of the mirror block, plus 1 if the target block follows the mirror block. Since the distance between any two blocks is even and not 0, target_offset % 2 tells which of the two comes first, and target_offset is at least 2. The target block either precedes or follows the mirror block, and the two MUST NOT overlap.
target_data is a copy of the data of the target block. The bytes of the target block, from its block_start to its block_end, MUST be equal to the concatenation of the following:
-
The shortest leb128 encoding of the target type;
-
The length of target_data, encoded as the size field for the target type;
The padding of the target block, if any, is not compared, and is restored as a zero byte. The type of the mirror block itself is 2 bytes long, as every block type uses the shortest leb128 encoding.
mirror_checksum is the XXH64 [XXHASH] hash with the seed zero of all bytes from the block_start of the mirror block to checksum_end, including its type and size fields.
A mirror block SHOULD be unwidened unless its data doesn’t fit in the size field for the target type.
A mirror block doesn’t belong to any segment and MAY occur anywhere in blocks, before or after its target block, including after the last metadata block. A single target block MAY have multiple mirror blocks. The requirements on a mirror block in this section are requirements on conforming compressors, and a mirror block violating them doesn’t make the archive an invalid archive. Mirror blocks don’t affect the result of interpreting an archive at all.
A mirror block is intact if its mirror_checksum verifies and it satisfies the requirements above other than being equal to its target block. A decompressor MUST NOT restore a target block from a mirror block that is not intact, and SHOULD report a mirror block that is not intact or differs from its intact target block when it reads one.
A mirror block and its target block are expected to be damaged together when they are close to each other. Conforming compressors SHOULD separate a mirror block and its target block, from the block_end of the earlier one to the block_start of the later one, by at least 4096 bytes, so that they never share an aligned 4 KiB unit of the underlying storage, and SHOULD separate them by at least 4 MiB (4,194,304 bytes) when possible.
The recommended distances are tentative: a larger one would also cover the loss of a flash erase block, pending the measurement of the overhead of mirror blocks. The interval between mirrored headers in an encrypted archive (§ 9 Encrypted archives) changes with the latter distance.
A damaged encrypted chunk fails to authenticate as a whole, and an aligned 4 KiB unit of the underlying storage may span two adjacent encrypted chunks. Therefore, in place of the distance of 4096 bytes recommended above, conforming compressors SHOULD place a mirror block and its target block in an encrypted archive so that at least one entire encrypted chunk lies between them.
A truncation of the archive, however, removes the last metadata block together with all the mirror blocks after it. The compressor can instead hold back the blocks written in the last 4 MiB or more before the last metadata block, and write the mirror block of the last metadata block before them, so that a truncation shorter than them leaves the mirror block and the metadata block can be restored (§ 10.6 Repair). The relative offsets in the metadata block to the blocks before its mirror block, and previous_metadata_offset, then count the mirror block, whose length depends on the metadata block: the compressor encodes the metadata block for a room of a given length for its mirror block, and encodes it again with a longer room until the mirror block fits, filling the rest of the room with a spacer block. Holding back the blocks costs the memory of at least 4 MiB of them, and more when the last ones held back are longer.
In an archive of several segments, a metadata block only becomes available at the end of its segment, so its mirror block can instead be placed after a sufficient amount of the next segment has been written. The mirror block of the last metadata block can’t be distant from it unless the compressor delays the last segment, and closing segments periodically bounds the loss in that case.
10.4. Mirrored archives
An archive is a mirrored archive when every block whose damage affects more than a single parcel can be restored:
-
Every block of a target type listed in § 10.3 Mirror block is equal to an intact mirror block, i.e. its bytes from its block_start to its block_end are as that mirror block requires.
-
For each such block, at least one of those mirror blocks is separated from it by 4096 bytes or more as described in § 10.3 Mirror block, or in an encrypted archive, has at least one entire encrypted chunk between them.
-
An encrypted archive is mirrored.
A tool verifying an archive SHOULD report whether it is mirrored, and SHOULD report the blocks that have no intact mirror block, whose mirror blocks are all nearer than above, and that differ from an intact mirror block.
10.5. Recovery sets
A recovery set protects a range of bytes of the archive with parity computed by a Reed–Solomon code, so that damaged parts of the range can be restored, including the content of files that no mirror block covers. The range is divided into slices, each of which is either intact or lost as a whole, and up to as many lost slices as the set has parity shards can be restored. A recovery set consists of the following blocks, called recovery blocks:
| Block Types | Definition |
|---|---|
| C116 — C316 | Parity block, holding a single parity shard |
| C416 — C716 | Index block, holding the hashes of every data shard |
Block type C016 is reserved, as a parity block is always longer than 255 bytes: its shard alone takes a slice size of at least 4096 bytes.
A recovery set without any parity shard consists of index blocks only, which locate damage to the range without restoring it.
Its data consists of the following:
| Name | Type | Conditions |
|---|---|---|
| recovery_magic | u8[8] ∈ {RECOVERY_MAGIC, OVERLAY_RECOVERY_MAGIC} | |
| archive_offset | leb128 | |
| range_offset | leb128 | |
| range_length | leb128 ≥ 1 | |
| lg_slice_size | u8, 12 ≤ lg_slice_size ≤ 63 | |
| parity_count | leb128 | |
| shard_index | leb128 < parity_count | If a parity block |
| shard | u8[slice size] | If a parity block |
| slice_hashes | u64le[data shard count] | If an index block |
| checksum_end | offset | |
| recovery_checksum | u64le |
recovery_magic is a fixed byte sequence that allows recovery blocks to be found by scanning, as mirror_magic does for mirror blocks. It MUST be OVERLAY_RECOVERY_MAGIC in an overlay archive and RECOVERY_MAGIC otherwise.
archive_offset is the offset from archive_start to the block_start of the recovery block, for the same purpose as archive_offset of a mirror block.
The protected range of the set is [range_start, range_end), where range_start is the block_start of the recovery block minus range_offset and range_end is range_start + range_length. The range MUST lie within [archive_start, archive_end), and every recovery block MUST start at or after range_end, i.e. range_offset ≥ range_length. The range MAY start or end in the middle of a block, and MAY contain recovery blocks of other recovery sets and mirror blocks.
The slice size S is 2lg_slice_size, and the number of parity shards p is parity_count. Recovery blocks of an archive with the same range_start, range_length, lg_slice_size and parity_count belong to the same recovery set, and all others belong to other recovery sets.
recovery_checksum is the XXH64 [XXHASH] hash with the seed zero of all bytes from the block_start of the recovery block to checksum_end, including its type and size fields.
10.5.1. Slices and shards
The slice grid of an archive has its origin at the start of its header_magic, i.e. at archive_start − 6, which is the start of the payload in an encrypted archive. The slice k of a recovery set, for every integer k ≥ 0, consists of the bytes from the offset k × S to the offset (k + 1) × S from the origin of the slice grid.
The data shards of a recovery set are its slices overlapping its range, in their order. Let k0 be ⌊(range_start − o) / S⌋, where o is the origin; the data shard count n is ⌈(range_end − o) / S⌉ − k0, and the data shard di for 0 ≤ i < n is the slice k0 + i with every byte outside the range taken as zero, whatever the archive holds there. Only the first and the last data shards may be partly outside the range.
Every recovery set with p ≥ 1 also has p parity shards P0, …, Pp−1 of S bytes each, computed from its data shards as in § 10.5.2 Parity shards. If p ≥ 1, n + m MUST NOT exceed 65,536, where m is the smallest power of two that is p or greater.
The shard of a parity block is the parity shard Pj for j = shard_index, and MUST be equal to it. A parity block MUST NOT belong to a recovery set with p = 0.
The slice_hashes of an index block has an entry for every data shard, and the entry i MUST be the XXH64 [XXHASH] hash with the seed zero of the bytes of di within the range, without the zeros outside it.
10.5.2. Parity shards
A shard of S bytes is a sequence of S / 2 16-bit symbols. The symbol c of a shard, for 0 ≤ c < S / 2, has its least significant byte at the offset 64 × ⌊c / 32⌋ + (c mod 32) in the shard and its most significant byte at the offset 64 × ⌊c / 32⌋ + 32 + (c mod 32). In other words, every 64 bytes of a shard hold 32 symbols, their least significant bytes followed by their most significant bytes.
Symbols are elements of the finite field GF(216) represented as GF(2)[x] / (x16 + x5 + x3 + x2 + 1), in which a polynomial is written as a 16-bit integer whose bit b is the coefficient of xb. A symbol v, however, denotes the element Σb vb × βb, where vb is the bit b of v and β0, …, β15 are the following elements:
| b | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| βb | 000116 | ACCA16 | 3C0E16 | 163E16 | C58216 | ED2E16 | 914C16 | 401216
| b
| 8
| 9
| 10
| 11
| 12
| 13
| 14
| 15
| |
| βb | 6C9816 | 10D816 | 6A7216 | B90016 | FDB816 | FB3416 | FF3816 | 991E16 |
Addition of symbols is therefore their bitwise exclusive or. For 0 ≤ k < 65,536, ωk denotes the element denoted by the symbol k.
The data shards are divided into groups of m shards: the group g ≥ 0 consists of dgm, …, dgm+m−1, where di for i ≥ n is taken as S zero bytes, and the data shard dgm+r is placed at the point ω(g+1)m+r. For each symbol position c and each group g, let fg be the polynomial over the field of degree less than m such that fg(ω(g+1)m+r) is the symbol c of dgm+r for every 0 ≤ r < m. The symbol c of the parity shard Pj, for 0 ≤ j < p, is Σg fg(ωj).
reed-solomon-simd (rate::HighRateEncoder)
and of Leopard-RS over GF(216) produce these parity shards from the data shards as they are.
The high-level interface of the former chooses another code when p > n,
and the original interface of the latter uses GF(28) when n + m ≤ 256.
The parity shards can be computed while the data shards are read, one group at a time:
each group is transformed into the coefficients of fg and accumulated,
and the sum is transformed into its values at ω0, …, ωm−1 once at the end,
so that a compressor needs about 2 × m × S bytes of memory regardless of the length of the range.
10.5.3. Placement
A recovery block doesn’t belong to any segment and MAY occur anywhere in blocks after the range of its set. A recovery set MAY have multiple parity blocks with the same shard_index and multiple index blocks, and need not have a parity block for every parity shard. As for mirror blocks, the requirements on a recovery block in this section are requirements on conforming compressors, and a recovery block violating them doesn’t make the archive an invalid archive. Recovery blocks don’t affect the result of interpreting an archive at all.
A recovery block is intact if its recovery_checksum verifies and it satisfies the requirements above other than those on the values of shard and slice_hashes, and other than the range lying before archive_end. A decompressor MUST NOT restore any byte from a recovery block that is not intact.
A recovery set with p ≥ 1 SHOULD have at least two index blocks, separated from each other as a mirror block is recommended to be separated from its target block (§ 10.3 Mirror block), since the damage to its data shards can’t be located without one. Conforming compressors SHOULD place the recovery blocks of a set at least 4 MiB (4,194,304 bytes) after its range_end when possible, so that a single burst of damage doesn’t remove a part of the range and the parity shards restoring it together. In an encrypted archive, the slice size SHOULD be 64 KiB or greater.
A tool verifying an archive SHOULD report its recovery sets, the slices whose hash doesn’t match, and the recovery sets that couldn’t restore their damaged slices.
10.6. Repair
Restoring damaged blocks from mirror blocks and damaged slices from recovery sets is a process separate from interpreting an archive: it takes a damaged byte sequence and returns a byte sequence that is then interpreted as any archive is. A decompressor MAY repair an archive before interpreting it, and SHOULD report the blocks it restored.
To decrypt a damaged encrypted archive c, a byte sequence starting where an encrypted archive starts, given the passphrase:
-
If c starts with ENCRYPTED_HEADER_MAGIC or ENCRYPTED_MIRRORED_HEADER_MAGIC and its header is whole, with an header_checksum that verifies if it has one, take its header fields and whether it is mirrored.
-
Otherwise, take the mirrored_fields of a mirrored header found by scanning c for ENCRYPTED_MIRROR_MAGIC that is intact and is at the position its mirror_index tells in an archive starting at the start of c, and take the archive as mirrored. If there is none, return failure.
-
Unwrap the file key as in § 9 Encrypted archives; a failure there is a failure of this algorithm.
-
Divide the bytes after the header into encrypted chunks and mirrored headers at the positions § 9 Encrypted archives gives them. If the bytes after the last whole encrypted chunk are 16 or fewer, leave them out.
-
Decrypt each encrypted chunk with the flag of the last chunk set if it is the last one, and if it is the last one and fails to authenticate, again with the flag not set.
-
Decrypt each encrypted chunk failing to authenticate anyway, as ChaCha20 [RFC8439] without its authentication tag, and take its bytes as untrusted.
-
Return the payload, the concatenation of the decrypted chunks, and its untrusted bytes, and whether it is cut after a chunk: whether the last chunk authenticated only with the flag not set.
A decompressor MUST report a payload cut after a chunk and MUST treat it as a failure of the extraction or the test, even if what is left of it can be interpreted: every chunk then authenticates, but whatever came after the last one is lost, including any later segment.
To repair an archive a, a byte sequence holding an unencrypted archive or the decrypted payload of an encrypted archive from its header_magic on, with a set untrusted of the bytes of a that are untrusted (those of encrypted chunks that fail to authenticate, decrypt a damaged encrypted archive; none for an unencrypted archive):
-
Let start be 6 if a starts with HEADER_MAGIC or OVERLAY_HEADER_MAGIC, and null otherwise.
-
Let found be the list of mirror blocks found by scanning a for MIRROR_MAGIC and OVERLAY_MIRROR_MAGIC as described in § 10.3 Mirror block, in the order of their block_start, that are intact when archive_start is taken as the start of the archive they imply and that have no untrusted byte. If a is the payload of an encrypted archive, found is instead the list of the mirror blocks reached by reading blocks by the generic structure (§ 3.1 Block) through bytes that are not untrusted, from the offset 6 if the first 6 bytes of a are not untrusted, and from the block_start of the metadata block that the footer tells if a is not cut after a chunk and ends with a footer that has no untrusted byte, each reading stopping before the first untrusted byte or where a block can’t be read. Their target blocks may lie beyond the end of a. Let rfound be the list of recovery blocks found likewise by scanning a for RECOVERY_MAGIC and OVERLAY_RECOVERY_MAGIC (§ 10.5 Recovery sets), that are intact and have no untrusted byte. If start is not null, remove from found and rfound every block implying a start of the archive other than start.
-
If start is null:
-
Set start to the smallest start of the archive implied by the blocks of found and rfound. Remove from found and rfound every block implying another start.
-
If start is 6, replace the first 6 bytes of a with OVERLAY_HEADER_MAGIC if the blocks of found and rfound have OVERLAY_MIRROR_MAGIC or OVERLAY_RECOVERY_MAGIC, and with HEADER_MAGIC otherwise.
-
-
For each target block position t of the mirror blocks of found, in the order of found:
-
Let restored be the bytes that the first mirror block of found for t requires of its target block.
-
If the bytes of a from t are those required by any mirror block of found for t, remove the bytes of the target block from untrusted and continue. If a is the payload of an encrypted archive and the target block lies within a and has no untrusted byte, continue.
-
If a is shorter than t, continue.
-
If a is shorter than t + the length of restored, append bytes to a up to that length.
-
Replace the bytes of a from t with restored, and the following byte with 0 if the length of restored is odd and a has a following byte. Remove those bytes from untrusted.
-
-
For each recovery set of the blocks of rfound, in the order of decreasing range_end, restore a from it.
-
If the step 5 restored any byte that no earlier repetition of it has restored, repeat from the step 2.
-
If start is 6, reconstruct the footer of a.
-
Return a and untrusted.
To restore from a recovery set with the blocks r of rfound in repair an archive, taking start as archive_start:
-
If p is 0, return.
-
Let h be the slice_hashes of the first index block of r, or null if there is none.
-
A data shard di is erased if any of its bytes within the range is untrusted, or if h is not null and the hash of its bytes within the range (§ 10.5.1 Slices and shards) is not h[i]. If no data shard is erased, return.
-
Let q be the set of every j for which r has a parity block with shard_index j, with the shard of the first such parity block as Pj.
-
If there is exactly one choice of the bytes within the range of the erased data shards for which the data shards and Pj for every j in q are as § 10.5.2 Parity shards requires, and every erased data shard has the hash h tells if h is not null, replace those bytes of a with that choice, and remove them from untrusted.
To reconstruct the footer of a byte sequence a starting with a header magic:
-
Read blocks from the offset 6 by the generic structure (§ 3.1 Block), with data of every block taken without parsing, until the bytes at the current offset start with the shortest leb128 encoding of FOOTER_TYPE or OVERLAY_FOOTER_TYPE, until a ends at or before the end of the block being read, or until the bytes at the current offset are not a leb128 in the shortest encoding.
-
If the reading reached a footer whose footer_size tells that it ends at the end of a and that has no untrusted byte, return.
-
If the reading stopped otherwise than by a ending or by reaching the start of a footer, return.
-
Let m be the last block read that is a metadata block. If there is none, or any block read after m is neither a spacer block nor of a non-critical type, or the bytes after the last block read start with the shortest leb128 encoding of a critical type other than those of spacer blocks, FOOTER_TYPE and OVERLAY_FOOTER_TYPE, return.
-
Let e be the offset after the padding of the last block read (the block_end if it has none). Truncate a to e bytes, appending zero bytes if it is shorter.
-
Append a footer with the footer magic matching the header magic of a, size_hint e − 6 and metadata_backward_offset from footer_start back to the block_start of m.
-
If the first block read is a size hint block whose data is 10 bytes long, set its size_hint to e − 6, encoded in 10 bytes as an encrypted archive’s payload has it.
10.7. Salvage
Salvaging an archive extracts what can still be read from an archive that is an invalid archive even after it is repaired, e.g. one cut short before its last metadata block or one with a damaged leaf block that no mirror block covers. Its result is not the result of interpreting an archive: a decompressor MAY salvage an archive it can’t interpret, and then MUST report every file it doesn’t extract and why, and MUST NOT report the archive as valid.
To salvage an archive a, a byte sequence as for repair an archive:
-
Repair a, except that, if a starts with a header magic, a target block position t beyond the end of a isn’t skipped in the step 4.3:
-
Read blocks from the offset 6 by the generic structure (§ 3.1 Block) as far as they are whole, and let e be the offset after the padding of the last block read (6 if none).
-
If e > t or t − e is odd, skip t.
-
Truncate a to e bytes, and append a spacer block of t − e bytes if t > e. The bytes from e to t are lost.
-
Continue with the step 4.4.
Before reconstructing the footer in the step 7, if any byte of a is still untrusted, lose the untrusted blocks of a. When reconstructing the footer returns in its step 4 only because of blocks after m or a block cut short after the last block read, the bytes from the end of m (after its padding) to the end of a are lost: truncate a to the end of m and continue with its step 5. The blocks in those bytes, as far as they are whole and have no untrusted byte, MAY be salvaged apart as the blocks of a standalone archive of their own without a metadata block, each of its leaf blocks evaluated by itself as in the step 2: the outputs of those evaluated are files and forks whose names are not known, which a decompressor MUST report as such if it extracts them.
-
-
Interpret a as in interpret an archive, with the following exceptions, each of which makes the files and forks concerned lost rather than a an invalid archive:
-
A file or a fork whose leaf block offset doesn’t refer to a leaf block is lost.
-
The module_offsets, resource_offsets and dictionary_offsets may list more blocks than there are, if those there are the first ones listed.
-
A formula block whose formula_data fails to decode or to parse, including a reference to a non-leaf block that isn’t there, makes the files and forks of its outputs lost.
-
If any file or fork is lost, a non-leaf block that no formula block references is not a failure.
-
Every leaf block is evaluated by itself: a failure of its evaluation, including a checksum that doesn’t verify, makes the files and forks of its outputs lost, and the other leaf blocks are evaluated as usual.
-
To lose the untrusted blocks of a byte sequence a starting with a header magic, given the positions known of the block_start of every mirror block of found in repair an archive and of its target block, of every recovery block of rfound, and of the footer if one ends at the end of a and has no untrusted byte:
-
Let o be 6.
-
While o is less than the length of a:
-
Read blocks from o by the generic structure as far as they go, until the start of a footer.
-
Let s be the block_start of the first block read whose type or size has an untrusted byte, if any. If there is such a block, and the reading from s doesn’t reach exactly a position in known greater than s before failing, reaching the end of a, or passing such a position:
-
Let k be the smallest position in known greater than s, or the length of a if there is none.
-
The bytes from s to k are lost: replace them with a spacer block of their length.
-
Set o to k and continue.
-
-
Every block read with an untrusted byte is lost: replace its bytes, padding included, with a non-leaf data block of the same length without compression whose checksum doesn’t verify if it is a non-leaf data block, and with a spacer block of the same length otherwise.
-
If the reading stopped at a footer with an untrusted byte, truncate a to the start of that footer.
-
Stop.
-
segment-status in it tells that segments may have been lost (§ 11.3 segment-status: Commits and the end of an archive).
11. Optional metadata
The optional_metadata field of the metadata block (§ 4.3 Metadata block) holds the optional metadata, encoded in CBOR [RFC8949]. This part encodes any metadata that is not strictly necessary to read the file, including associated dates and times, access control information and comments.
The optional metadata is described by the following CDDL [RFC8610] definitions:
aqr-opt-metadata= { aqr-archive-comment ? &(entry-extras: -2) => [* aqr-entry-extra / null] ; Indexed by entry numbers, starting from the root directory aqr-segment-status aqr-journal * $$aqr-opt-metadata-member ; For future extensions }$$aqr-opt-metadata-member= () ; No extension is defined by this specificationaqr-entry-extra= { aqr-entry-timestamps aqr-entry-comment aqr-entry-link-group aqr-entry-xattrs aqr-entry-forks aqr-entry-windows aqr-entry-unix aqr-entry-macos aqr-entry-zip * $$aqr-entry-extra-member ; For future extensions }$$aqr-entry-extra-member= () ; No extension is defined by this specification ; Key -8 is reserved for access control lists.
It MUST be a single well-formed CBOR data item [RFC8949] without any following bytes,
and no map in it can have duplicate keys.
The value of every key defined by this specification MUST match its definition.
If optional_metadata_size is 0, the optional metadata is an empty map {}.
The members of aqr-entry-extra are grouped by their purposes, and each group is described in the following sections.
Keys of aqr-entry-extra from -1 to -59 are platform-neutral.
Keys from -60 to -69 are specific to Windows, -70 to -79 to Unix, -80 to -89 to macOS, and -90 to -99 to other platforms.
Keys from -100 are specific to the formats an entry was converted from.
Key -8 is reserved for access control lists in a future version of this specification,
and a compressor MUST NOT use it.
11.1. Timestamps
The keys crtime, ctime, mtime, atime and arctime of aqr-entry-extra
record the timestamps that file systems and archive formats associate with a file, a directory or a link.
Each is an aqr-time described below, and is absent if the source doesn’t have the corresponding timestamp.
aqr-entry-timestamps = (
? &(crtime: -1) => aqr-time
? &(ctime: -2) => aqr-time
? &(mtime: -3) => aqr-time
? &(atime: -4) => aqr-time
? &(arctime: -5) => aqr-time
)
crtime-
The creation time, also known as the birth time: when the file was created at its location. It is not updated afterwards, but copying a file usually creates a new one with the time of copying. Examples include the creation time of NTFS and FAT, the create date of HFS+ and APFS,
i_crtimeof ext4 andotimeof Btrfs, which are exposed asst_birthtimeon BSD and macOS and asstx_btimeofstatxon Linux. ctime-
The status change time: when the file’s metadata (e.g. permissions, ownership or link count) or its content was last changed. It is the POSIX
st_ctime, and the change time of NTFS and the attribute modification date of HFS+ are similar. POSIX doesn’t allow it to be set to an arbitrary time, so an extracting decompressor usually can’t restore it. mtime-
The modification time: when the file’s content was last changed. For a directory, it is when an entry was last added, removed or renamed in it. This is the timestamp that virtually all file systems and archive formats have.
atime-
The access time: when the file’s content was last read. Many systems update it lazily or not at all for performance (e.g.
relatimeandnoatimeon Linux, and the access time updates of NTFS that can be disabled), and FAT only keeps the date. Reading a file for archiving can itself update it. arctime-
The backup time: when the file was last backed up, as recorded by backup software. File systems don’t update it by themselves. An example is the backup date of HFS and HFS+, exposed as
ATTR_CMN_BKUPTIMEofgetattrliston macOS. It is the time stored by the source, not the time when the archive itself was made.
st_ctime in the Microsoft C runtime, the Ctime of ZIP extra field 0x000a,
and the CTime of 7z and RAR all go to crtime, not ctime.
POSIX formats such as pax use it for the status change time, which goes to ctime.
aqr-time represents a point in time or a local date and time,
together with a quantization interval and optional time zone information.
It is described by the following CDDL definitions,
where etime is the CDDL type name for the extended time defined in [RFC9581]:
aqr-time= $aqr-base-time / ; UTC timestamp, scaled and quantized [$aqr-base-time] / ; Local time in unknown time zone, encoded like UTC (*) [$aqr-instant,offset: aqr-tz-offset] / ; UTC timestamp with time zone offset ($) etime / ; Extended time defined in [[!RFC9581]] [etime] ; Local time in unknown time zone, encoded like UTCaqr-tz-offset= -48..56 ; In 15 minutes, positive east of UTC as in ISO 9660$aqr-base-time= #6.100(int) ; 1d (FAT atime*), date as in [[!RFC8943]] $aqr-base-time /= $aqr-instant ; Can be given a time zone offset$aqr-instant= #6.121(int) ; 2s (FAT mtime*, exFAT atime$, ZIP*, CAB*) $aqr-instant /= int ; 1s (~ext3, HFS/HFS+, tar, ISO 9660$) $aqr-instant /= #6.122(int) ; 10ms (FAT ctime*, ISO 9660$, exFAT$) $aqr-instant /= #6.123(int) ; 1ms $aqr-instant /= #6.124(int) ; 1us (UDF) $aqr-instant /= #6.125(int) ; 100ns (NTFS, RAR*, 7z) $aqr-instant /= #6.126(int) ; 1ns (ext4, ZFS, Btrfs, APFS)
A base time value $aqr-base-time is an integer, either untagged or tagged as follows.
The integer multiplied by the scale is the number of seconds since 1970-01-01T00:00:00Z,
not counting leap seconds as in the CBOR tag 1 [RFC8949];
negative values denote the time before that epoch.
The tag also determines the quantization interval,
which is relative to the time value and specified in § 15.4 Quantization Interval.
#6.100 is the number of days since 1970-01-01 [RFC8943],
and is the only base time value that is not an $aqr-instant, i.e. it can’t be given a time zone offset.
| Encoding | Quantization Interval (s) | Scale (s) | Examples |
|---|---|---|---|
#6.100(int)
| [0, 86400] | 86400 | FAT atime* |
#6.121(int)
| [0, 2] | 2 | FAT mtime, exFAT atime$, ZIP, CAB* |
int
| [0, 1] | 1 | ext3 and earlier, HFS/HFS+, tar, ISO 9660$ (directory records) |
#6.122(int)
| [0, 10-2] | 10-2 | FAT ctime*, ISO 9660$ (volume descriptors), exFAT$ |
#6.123(int)
| [0, 10-3] | 10-3 | - |
#6.124(int)
| [0, 10-6] | 10-6 | UDF |
#6.125(int)
| [0, 10-7] | 10-7 | NTFS, RAR*, 7z |
#6.126(int)
| [0, 10-9] | 10-9 | ext4, ZFS, Btrfs, APFS |
In the Examples column, formats marked with * store (or can store) a local time in an unknown time zone, and formats marked with $ store a local time with its offset from UTC.
$aqr-instant,
and are not shortcuts for the extended time in general.
1001({1: 1697641200, -14: 86400, -10: "+09:00"}) for 2023-10-19 in UTC+9).
Each alternative of aqr-time has the following meaning:
$aqr-base-time-
A point in time without any time zone information.
[$aqr-base-time]-
A local date and time in an unknown time zone, encoded as if it were UTC. It does not identify a single point in time.
[$aqr-instant, aqr-tz-offset]-
A point in time, as in the bare
$aqr-base-time, with the offset of the time zone where it was recorded, in 15-minute increments as in ISO 9660. The offset is positive east of UTC: UTC+x hours is 4x and UTC−x hours is −4x. The local time is the base time value plus 900 times the offset in seconds. etime-
An extended time [RFC9581] (tag 1001), for any value which doesn’t fit the other alternatives. The time zone information, if any, is given as a Time Zone Hint (key -10 or 10), and the quantization interval is given as a key described in § 15.4 Quantization Interval.
[etime]-
A local date and time in an unknown time zone, encoded as if it were UTC. The extended time MUST NOT contain a Time Zone Hint (key -10 or 10), and its timescale (key -1, -13 or 13), if present, MUST be 0 (UTC).
etime or [etime] MUST NOT be used
if another alternative of aqr-time represents the same time value,
the same quantization interval and the same time zone information.
A Time Zone Hint is the same as an aqr-tz-offset
only when it is a numeric offset (e.g. "+09:00") equal to that offset;
an extended time without any quantization interval is never the same as other alternatives.
aqr-time using the extended time,
as an extracting decompressor is allowed to ignore any extra metadata (§ 12 Extraction).
UTCTime, 2-32 s in NTP and 1/300 s in SQL Server DATETIME,
and so are time zone offsets that are not multiples of 15 minutes as in UDF.
They are rare enough in file metadata not to deserve their own encodings.
Two aqr-time values in different encodings are different even when they denote the same point in time.
For example, 1 and 122(100) differ in their quantization intervals.
A compressor converting a timestamp from a file system or another archive format
SHOULD preserve its precision, i.e. use an encoding whose quantization interval matches the source.
-
ZIP with a DOS date and time, a local time in an unknown time zone:
[121(848878577)] -
ZIP with an extended timestamp extra field (0x5455) in UTC:
1697724754 -
git, which records a time zone offset in minutes:
[1697724754, 36] -
exFAT modification time with a UTC offset:
[122(169772475487), 36] -
NTFS:
125(16977247548700000) -
ext4:
126(1697724754870000000) -
FAT access date, a local date in an unknown time zone:
[100(19649)] -
A source rounding to the nearest second:
1001({1: 1697724755, -14: [0.5, 0.5]})
Timestamps from ZIP are discussed in § 16.2 Timestamps from ZIP.
11.2. Comments
archive-comment of aqr-opt-metadata and comment of aqr-entry-extra
are free-form comments attached to the whole archive and to each entry respectively,
e.g. an archive comment and a file comment of ZIP or RAR.
Both are an aqr-text, which is a Unicode string,
a WTF-8 [WTF-8] string with lone surrogates, or a byte sequence in an unknown encoding:
aqr-archive-comment= ( ? &(archive-comment: -1) => aqr-text )aqr-entry-comment= ( ? &(comment: -6) => aqr-text )aqr-text= tstr / ; Unicode #6.273(bstr) / ; WTF-8, i.e. UTF-16 + surrogate compat bstr ; In unknown encoding
An aqr-text MUST be a tstr if it can be represented as one,
i.e. if it is a sequence of Unicode scalar values.
The byte string in #6.273(bstr) MUST be a valid WTF-8 string that is not a valid UTF-8 string,
i.e. it MUST contain at least one encoded lone surrogate.
A bstr is not subject to these requirements even if it happens to be a valid UTF-8 string,
as its encoding is not known.
Comments from ZIP are discussed in § 16.3 Comments from ZIP.
11.3. segment-status: Commits and the end of an archive
segment-status of aqr-opt-metadata tells whether the compressor meant its writing to end with the segment,
and whether that writing was complete:
aqr-segment-status= ( ? &(segment-status: -3) => true / aqr-incomplete )aqr-incomplete= { ? &(files: 0) => uint ; Number of files left out ? &(bytes: 1) => uint ; Their total size in bytes }
-
truetells that the segment is the last one of a writing that completed, e.g. of all the files it was given. -
aqr-incompletetells that the segment ends a commit (§ 3.7 Appending to an archive) of a writing that didn’t complete, e.g. because it was stopped or is still going on, with what it left out where known. A later writing may append the rest. -
No
segment-statustells that no footer was meant directly after the segment.
It doesn’t affect the result of interpreting an archive (§ 4.4 Interpreting an archive):
a later segment may still follow it, e.g. a segment appended to the archive afterwards,
and it is not a part of the extra of the result.
Conforming compressors SHOULD put true in the optional metadata of the last segment of a writing that completes,
and aqr-incomplete in that of the last segment of every other commit, and nothing in any other segment.
A decompressor SHOULD report an archive whose last segment has aqr-incomplete as incomplete, telling what was left out if given,
while it still interprets the archive as usual.
A decompressor that reconstructs the footer of an archive after a metadata block without segment-status
SHOULD report that segments after it may have been lost, and SHOULD NOT report the archive as intact.
segment-status tells the two apart where the compressor meant a footer to follow only the last segment it wrote.
An archive cut short just where a segment was later appended to it still reads as whole, as it was before the appending.
The hint is lost with the optional metadata of a damaged metadata block,
which then can’t be relied on for anything else either.
true and aqr-incomplete are exclusive, as a writing either completed or didn’t,
and only the last segment tells about the archive:
a writing that continues one left incomplete ends with true without undoing anything of the segments before.
11.4. journal: Points to return to
journal of aqr-opt-metadata names the states of the archive that its writer meant to be returned to,
e.g. the states after each run of a backup:
aqr-journal= ( ? &(journal: -4) => {* aqr-journal-time => aqr-text} )aqr-journal-time= uint ; Milliseconds since 1970-01-01T00:00:00Z, leap seconds not counted
Each key is the time of a journal point and its value is the label of the point, which may be empty.
The state of a journal point in the journal of a segment is the result of interpreting the archive
cut after that segment, i.e. with the segments after it removed.
Several points of a segment therefore name the same state,
as when a writing found nothing to change and added a segment of no entries to name the state again.
Journal points need not be unique in their labels.
The journal doesn’t affect the result of interpreting an archive (§ 4.4 Interpreting an archive),
and it is not a part of the extra of the result.
A decompressor MAY be asked for the state of a point, and then interprets only the segments up to the point’s.
Which point is meant by a time or a label, if the user gives one, is up to the decompressor:
it SHOULD take the latest point whose time is not after the given time,
and the last of the points of the label.
Where the same time occurs in several segments, or the times don’t increase with the order of segments,
the order of the segments tells which point is the later, and a decompressor SHOULD warn of the latter.
Conforming compressors SHOULD put journal only in the metadata block of a segment with segment-status true
(§ 11.3 segment-status: Commits and the end of an archive), as the state after a commit of an incomplete writing is not one to return to.
They SHOULD NOT put one unless the user asked for it, as the time is information about the user.
11.5. link-group: Hard links
aqr-entry-link-group = (
? &(link-group: -7) => int ; File nodes only
)
link-group records that file nodes were originally the same file,
i.e. hard links to the same inode on Unix or to the same file record on NTFS.
It has no effect on other nodes, and a compressor SHOULD NOT give it to them.
A non-negative value is segment-local and denotes the same group only within the same segment,
while a negative value is archive-wide and denotes the same group in every segment.
The resulting link group in the archive value is determined by interpreting a segment.
A compressor MUST NOT give the same link group to files that were not the same file.
It SHOULD NOT give a link group that no other file node has.
A file node with link-group SHOULD refer to the same leaf block as other file nodes in the same group,
and SHOULD have the same entry extra as them except for link-group and comment.
st_ino) to be used as is,
as CBOR can encode any 64-bit unsigned integer as a non-negative int,
and can be assigned without reading earlier segments.
Inode numbers are still only unique within a single file system,
so a compressor reading from multiple file systems has to renumber some of them.
Archive-wide groups link files across segments,
e.g. when a compressor periodically starts a new segment while archiving a single tree,
or when it appends a segment to an existing archive.
A compressor using them is responsible for keeping them consistent with all other segments,
e.g. by reading earlier metadata blocks when appending.
Inode numbers are not suitable as archive-wide values as is,
because they can be reused after the file is removed or come from an unrelated file system,
and linking unrelated files with the same content can’t be detected by any decompressor.
A link group is consistent if all file nodes with that group in the final archive value have the same content. An inconsistent group, e.g. after a later segment has replaced only some of its file nodes, is not an invalid archive failure, but has no effect on extraction (§ 12 Extraction).
link-group
still extracts the correct files with the same contents.
Directory hard links, found in some macOS Time Machine volumes, are not represented.
11.6. xattrs: Extended attributes
aqr-entry-xattrs = (
? &(xattrs: -10) => { * aqr-text => bstr }
)
xattrs records the extended attributes of a node, i.e. named values that the file system attaches to it:
extended attributes of Linux, macOS and BSDs, and extended attributes (EAs) of NTFS.
Each key is the name of an attribute as the platform gives it, including any namespace prefix
(e.g. user. and security. on Linux), and MUST NOT be empty.
Each value is the value of the attribute.
A compressor MUST NOT put the following attributes into xattrs:
-
The resource fork of macOS (
com.apple.ResourceForkthere), which goes toforksinstead (§ 11.7 forks: Resource forks and alternate data streams). -
An attribute only telling how the file system stores the node, e.g.
com.apple.decmpfsof the transparent compression of macOS.
These names are only special on macOS.
An attribute of the same name from another platform, e.g. an EA of NTFS, is put into xattrs as any other,
though it may not be written on macOS (§ 12 Extraction).
POSIX access control lists of Linux appear as the attributes system.posix_acl_access and system.posix_acl_default,
and can be put into xattrs as is.
Key -8 is reserved for access control lists that are not extended attributes, e.g. those of macOS and NTFS;
a future version defining it will also decide which takes precedence when both describe the same node.
11.7. forks: Resource forks and alternate data streams
aqr-entry-forks= ( ? &(forks: -50) => { * aqr-text => aqr-fork } )aqr-fork= [offset: aqr-seg-offset,output: uint,size: uint, ]aqr-seg-offset= uint ; Relative offset from the metadata block, as in [[#metadata-block]]
forks records the forks of a node, i.e. its named data streams other than the content of a file.
Each key is a fork name, and each value refers to the content of the fork:
offset is the relative offset to a leaf block in the same segment,
measured backward from the beginning of the metadata block as other relative offsets in § 4.3 Metadata block,
output is an output index of that block, and size is the length of the content.
They are resolved when setting the forks of the node,
and a leaf block referenced by forks is described by the metadata block (§ 4.2 Content blocks).
Any number of file_params and forks MAY refer to the same leaf block and output.
Forks of platforms map to fork names as follows:
-
On NTFS, a fork is an alternate data stream of the type
$DATA, named by the stream name without the type, e.g.Zone.Identifierforfile:Zone.Identifier:$DATA. The unnamed data stream is the content of the file, not a fork. -
On macOS, the resource fork, i.e. the extended attribute
com.apple.ResourceForkalso accessible as the pathfile/..namedfork/rsrc, is the fork namedAFP_Resource, after the stream name that SMB servers use for it (e.g.vfs_fruitof Samba). A compressor MUST put the resource fork intoforksunder that name. Other extended attributes of macOS go toxattrsregardless of their sizes (§ 11.6 xattrs: Extended attributes).
Any other name from a platform is a fork name as is,
e.g. an alternate data stream named com.apple.ResourceFork, or one named AFP_Resource on a directory,
though such forks may not be written on macOS (§ 12 Extraction).
A compressor SHOULD NOT record how the file system stores a fork, e.g. whether it is sparse or compressed.
11.8. Windows-specific metadata
aqr-entry-windows = (
? &(win-link-type: -60) => &(file: 0, directory: 1, junction: 2) ; Link nodes only
? &(win-attrs: -61) => uint
)
win-link-type determines the kind of a link to be created on Windows (§ 12 Extraction).
It has no effect on nodes other than link nodes, and a compressor SHOULD NOT give it to them.
"file" and "directory" denote a symbolic link to a file and to a directory respectively,
and "junction" denotes a junction (a mount point reparse point).
win-attrs holds the file attributes of Windows (FILE_ATTRIBUTE_*),
limited to the following bits; any other bit MUST be zero.
| Bit | Attribute |
|---|---|
| 0x1 | FILE_ATTRIBUTE_READONLY
|
| 0x2 | FILE_ATTRIBUTE_HIDDEN
|
| 0x4 | FILE_ATTRIBUTE_SYSTEM
|
| 0x20 | FILE_ATTRIBUTE_ARCHIVE
|
| 0x100 | FILE_ATTRIBUTE_TEMPORARY
|
| 0x2000 | FILE_ATTRIBUTE_NOT_CONTENT_INDEXED
|
FILE_ATTRIBUTE_DIRECTORY, FILE_ATTRIBUTE_REPARSE_POINT and FILE_ATTRIBUTE_NORMAL (the absence of any other),
or tell how the file system stores the file rather than anything about the file,
e.g. FILE_ATTRIBUTE_SPARSE_FILE, FILE_ATTRIBUTE_COMPRESSED, FILE_ATTRIBUTE_ENCRYPTED
and FILE_ATTRIBUTE_INTEGRITY_STREAM, and the states of cloud files such as FILE_ATTRIBUTE_OFFLINE,
FILE_ATTRIBUTE_PINNED and FILE_ATTRIBUTE_RECALL_ON_DATA_ACCESS.
The file attributes in the external file attributes of ZIP [APPNOTE] map to win-attrs in the same way.
11.9. Unix-specific metadata
aqr-entry-unix = (
? &(unix-mode: -70) => uint .le 4095 ; Permission bits only (0o7777), without S_IFMT
? &(unix-uid: -71) => uint
? &(unix-gid: -72) => uint
? &(unix-uname: -73) => aqr-text
? &(unix-gname: -74) => aqr-text
)
unix-mode only holds the permission bits, including the set-user-ID, set-group-ID and sticky bits.
The kind of an entry is solely determined by its node (§ 4.1 Archive model), so unix-mode never includes the file type bits (S_IFMT).
unix-uid and unix-gid are the numeric user and group IDs of the owner,
and unix-uname and unix-gname are the names of that user and group, as uname and gname of tar.
An entry MAY have a name without the ID, and vice versa.
Unix metadata from ZIP is discussed in § 16.4 Unix metadata from ZIP.
11.10. macOS-specific metadata
aqr-entry-macos = (
? &(mac-flags: -80) => uint
)
mac-flags holds the file flags of macOS (st_flags, as set by chflags),
limited to the following bits; any other bit MUST be zero.
| Bit | Flag |
|---|---|
| 0x1 | UF_NODUMP
|
| 0x2 | UF_IMMUTABLE
|
| 0x4 | UF_APPEND
|
| 0x8 | UF_OPAQUE
|
| 0x8000 | UF_HIDDEN
|
| 0x10000 | SF_ARCHIVED
|
| 0x20000 | SF_IMMUTABLE
|
| 0x40000 | SF_APPEND
|
| 0x100000 | SF_NOUNLINK
|
UF_COMPRESSED, SF_DATALESS),
its state on the volume (UF_TRACKED),
or the protection that the system itself sets (UF_DATAVAULT, SF_RESTRICTED, SF_FIRMLINK).
Flags beginning with SF_ can only be changed by the superuser.
11.11. zip-extras: ZIP extra fields
zip-extras preserves extra fields [APPNOTE] of the ZIP entry that the entry was converted from,
when they carry information about the file that has no other representation in this specification.
aqr-entry-zip= ( ? &(zip-extras: -100) => aqr-zip-extras )aqr-zip-extras= { * aqr-zip-extra-key => bstr .size (0..65531) }aqr-zip-extra-key= 0..65535 / ; Header ID n in the local header -65536..-1 ; Header ID n in the central header, as -1 - n ; TODO: multiple extra fields with the same header ID
A key n from 0 to 65535 denotes the extra field with the header ID n in the local header, and a key −1 − n from −65536 to −1 denotes the extra field with the header ID n in the central header. The value is the data of the extra field, excluding its 4-byte header of the header ID and the data size. Therefore a value with a non-negative key is always in the format for the local header, and a value with a negative key is always in the format for the central header.
A compressor converting a ZIP archive MAY put any extra field of an entry into zip-extras, subject to the following requirements:
-
It MUST NOT put the central header version of an extra field if the local header version with the same header ID is identical byte by byte. The central header version without the local header version is only put with a negative key.
-
It MUST NOT put an extra field whose information is fully represented elsewhere, i.e. in the entry name, the archive model (§ 4.1 Archive model) or any other key of
aqr-entry-extra. -
It MUST NOT put an extra field which is only meaningful to the ZIP container itself, e.g. sizes, offsets, padding, alignment, encryption and signatures over the ZIP archive.
-
It SHOULD NOT give
zip-extrasto the root directory, which has no counterpart in ZIP.
Extra fields known to be subject to these requirements are listed in § 16.5 Extra fields not to be preserved.
Any other extra field, including one with an unregistered header ID, can be put into zip-extras as is.
zip-extras is never authoritative.
If information in a preserved extra field disagrees with the archive model or any other key of aqr-entry-extra,
the latter MUST be used.
A decompressor finding an extra field that a compressor is not allowed to put SHOULD ignore that field only,
and SHOULD NOT treat the whole archive as invalid.
zip-extras is designed for a one-way conversion from ZIP;
reconstructing the original ZIP archive from the converted archive is not a goal.
Extra fields are preserved so that information without its own key yet,
e.g. platform-specific attributes, access control lists and resource forks,
can be recovered from already converted archives once a dedicated key is defined.
Unknown extra fields are preserved for the same reason,
even though some of them might be only meaningful to the ZIP container.
The local and central header versions of the same header ID often differ in their formats,
e.g. the central header version of 0x5455 only has the modification time,
and that of many Info-ZIP attribute blocks only has a summary.
Distinguishing them by keys lets a consumer interpret each value in the correct format.
The key for the central header version is −1 − n instead of −n,
so that header ID 0 doesn’t collide with the local header version,
and both versions have the same encoded length in CBOR.
12. Extraction
Writing an archive value to a file system is outside the archive model. A decompressor doing so is subject to the following requirements:
-
It MUST NOT create, modify or remove any file outside the destination directory, including through symbolic links or other redirections already present in the destination or created during the extraction.
-
It SHOULD map an entry name whose encoding is "
wtf-8" to the platform’s native Unicode name (e.g. UTF-16 with lone surrogates on Windows), and an entry name whose encoding is "raw" to the identical byte sequence where the platform allows. -
If an entry name can’t be represented on the platform or has a special meaning there (e.g. a name containing
\or:, or a reserved device name on Windows), it MUST either fail or report a substituted name to the user; it MUST NOT silently use a different name. -
If multiple entry names in a directory map to the same file system name (e.g. due to case insensitivity or Unicode normalization), it MUST either fail or report the conflict, and MUST NOT write a later entry through an earlier one.
-
It MAY ignore the extra of any node, e.g. extra and extra. It SHOULD NOT apply ownership information, i.e.
unix-uid,unix-gid,unix-unameandunix-gname, nor flags only the superuser can change, i.e. theSF_flags ofmac-flags, unless explicitly requested. Nor SHOULD it apply extended attributes, inxattrsor written for forks, that may grant privileges or access, e.g. those outside theuser.namespace on Linux (security.capability,trusted.*,system.posix_acl_access), unless explicitly requested; an attribute not applied for this reason is reported as any other not written. When applying the ownership, it SHOULD useunix-unameandunix-gnamewhere they name a user and a group on the platform, andunix-uidandunix-gidotherwise. -
It SHOULD write the forks of a node where the platform allows: as alternate data streams of the same names on Windows; on macOS, the fork
AFP_Resourceof a file node as the resource fork (com.apple.ResourceFork), and any other fork as an extended attribute of the same name; and elsewhere, as extended attributes of the same names. -
A fork not written MUST be reported to the user, and so MUST an extended attribute in
xattrsnot written when the decompressor appliesxattrs. Reasons include a name invalid or reserved on the platform, e.g.com.apple.ResourceForkfor anything but the resource fork andAFP_Resourceof a non-file node on macOS, and names conflicting with another fork or attribute of the same node, e.g. a fork and an attribute of the same name on macOS, or names differing only in case on NTFS.
unix-mode.
Forks and extended attributes can’t be written once the permissions forbid writing, so they also go before unix-mode.
Writing the content or the children of a directory changes its modification time,
so the timestamps of a directory go after all its children.
Immutable and append-only flags forbid all of the above, so they go last.
Setting mtime earlier than crtime moves crtime back on macOS, so crtime goes after mtime.
A and a file a/x map to the same directory on a case-insensitive file system.
The rule on conflicting names above already prevents this, and the first requirement makes it explicit.
Possible implementations include creating all links after all other nodes,
or opening each path relative to its parent directory without following links
(e.g. openat with O_NOFOLLOW, or RESOLVE_BENEATH on Linux).
A decompressor extracting a link node is subject to the following additional requirements:
-
It MUST NOT create a link whose target escapes the destination directory unless explicitly requested, and SHOULD provide a way to request it. A link target escapes the destination directory if it is absolute or relative to a drive or a root on the platform, or if resolving it from the parent directory of the link, following any link nodes in the archive value on the way, reaches a parent of the destination directory. A link not created for this reason MUST be reported to the user.
Creating such a link doesn’t write outside the destination directory by itself, but the extracted tree can lead later users or programs to do so. Resolving a target lexically is not enough: whend/e/lis a link to../../q, a link tod/e/l/../..lexically resolves todbut actually reaches the parent of the destination directory. -
On Windows, it SHOULD replace every
/in the link target with\. It SHOULD create the kind of link given bywin-link-type, if any. Otherwise, it SHOULD create a symbolic link to a directory if the target resolves to a directory node in the archive value as above, and a symbolic link to a file otherwise. -
If a link can’t be created, e.g. due to the lack of privilege, the file system or the length of its target, it MUST either fail or report it to the user; it MUST NOT silently create a different kind of node. It MAY create a copy of the target instead if it reports so.
-
It MUST NOT apply the extra of a link node to its target. Timestamps, ownership and permissions SHOULD be applied to the link itself where the platform allows (e.g.
lutimes,lchown).
A decompressor MAY create hard links among file nodes with the same consistent link group. It MUST NOT create hard links among any other file nodes, even when they have the same content.
rsync --link-dest.
But files that were independent have to remain independent.
A decompressor SHOULD NOT create a FIFO or a device file for a special node unless explicitly requested, as with ownership information. A special node not created for this reason, or because the platform can’t represent it (e.g. the lack of privilege, or a device number out of the platform’s range), MUST be reported to the user.
A decompressor MUST NOT create anything for an unknown node, and MUST report it to the user. It MAY continue extracting other nodes.
13. Conformance
As well as sections marked as non-normative, all diagrams, examples and notes in this specification are non-normative. Everything else in this specification is normative.
The key words MAY, MUST, MUST NOT, RECOMMENDED, SHOULD and SHOULD NOT in this specification are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.
13.1. Conformance levels
A conforming decompressor has a conformance level, one of the following according to the modules (§ 7 Modules) it applies:
- Level 0
-
Applies no module.
- Level 1
-
Applies every registered module, i.e. every module whose module hash is the registered hash of an entry of the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry), and no other module.
- Level 2
-
Applies every module.
A decompressor at level 0 or level 1 MAY refuse to apply a module that its level doesn’t include, in which case a resource failure occurs. A decompressor at level 2 MUST NOT refuse to apply a module, though it may still fail with a resource failure when a module exceeds its limits. A level doesn’t change any other requirement on a decompressor, e.g. a decompressor at level 0 still reads formula blocks without Apply instructions.
13.2. Partial access
A decompressor MAY compute only a part of the result of interpreting an archive, such as a subset of entries, a byte range of a content, or entry names without contents. Such a partial result MUST be equal to the corresponding part of the full result, if the full result is an archive value.
A decompressor computing a partial result is not required to detect an invalid archive failure caused only by parts it doesn’t compute. It SHOULD verify every checksum covering only the bytes it has computed.
14. Security considerations
An archive is a container for arbitrary data and doesn’t by itself contain active content, except for WebAssembly modules (§ 7.4 Module execution) that a decompressor executes to reconstruct file contents. Such modules are run under a deterministic profile without access to anything but their inputs and outputs, but a decompressor still has to treat them as untrusted code and isolate them accordingly, unless it knows their module hash (§ 7.1 Module block). Formula blocks and modules can produce outputs far larger than the archive itself, and some fields (e.g. lg_work_factor) directly control the amount of computation. A decompressor is expected to enforce implementation-defined limits and fail with a resource failure when they are exceeded. A decompressor writing an archive to a file system is subject to the requirements in § 12 Extraction, in particular to prevent writing outside the destination directory. An unencrypted archive provides no confidentiality, and its XXH64 checksums (§ 10.1 Checksums) only detect accidental corruption, not intentional modification. An encrypted archive (§ 9 Encrypted archives) provides confidentiality and integrity through [AGE], which is only as strong as the passphrase used. The metadata of an archive, including entry names, timestamps and ownership, may reveal information about the system that created it.
An overlay archive is interpreted against a base, typically a directory on the file system, and can copy any part of the base into its result. A decompressor reading a base is subject to the requirements in § 8.3 Reading a base, in particular to use a base only when explicitly given one and to never read outside it. Applying an overlay archive in place modifies and removes files in the base directory.
15. IANA Considerations
15.1.
application/aqr Media Type
This document requests the registration of a new media type in the Media Types Registry ([RFC6838], section 5.6) as follows:
- Type name
- application
- Subtype name
- aqr
- Required parameters
- N/A
- Optional parameters
- N/A
- Encoding considerations
- binary
- Security considerations
- See § 14 Security considerations.
- Interoperability considerations
-
An unencrypted archive may be preceded and followed by arbitrary bytes, e.g. a self-extracting executable stub; its extent is determined by header_magic and the footer (§ 3.6 Locating an archive). An archive without any metadata block decompresses into a single unnamed file rather than a directory tree. An overlay archive has its own media type
application/aqo(§ 15.2 application/aqo Media Type); an encrypted archive whose payload is an overlay archive is also of that type. - Published specification
- This document.
- Applications that use this media type
- Archivers and compressors that implement the Aqueous archive format.
- Fragment identifier considerations
- N/A
- Additional information
-
- Deprecated alias names for this type
- N/A
- Magic number(s)
- F8 04 03 6A D6 BE (hexadecimal; HEADER_MAGIC) for an unencrypted archive, possibly preceded by other data; F9 CA F2 A6 D0 2A (hexadecimal; the common prefix of ENCRYPTED_HEADER_MAGIC and ENCRYPTED_MIRRORED_HEADER_MAGIC) at offset 0 for an encrypted archive.
- File extension(s)
- .aqr
When a single file is compressed into an archive without any metadata block, the extension may be appended to the original file name instead (e.g.example.tar.aqr). - Macintosh file type code(s)
- N/A
- Person & email address to contact for further information
- Kang Seonghoon <public+aqr@mearie.org>
- Intended usage
- COMMON
- Restrictions on usage
- N/A
- Author
- Kang Seonghoon
- Change controller
- Kang Seonghoon
15.2.
application/aqo Media Type
This document requests the registration of a new media type in the Media Types Registry ([RFC6838], section 5.6) as follows:
- Type name
- application
- Subtype name
- aqo
- Required parameters
- N/A
- Optional parameters
- N/A
- Encoding considerations
- binary
- Security considerations
- See § 14 Security considerations.
- Interoperability considerations
-
An overlay archive can’t be interpreted without its base. Its magic numbers differ from those of
application/aqr, so that a decompressor not supporting overlay archives doesn’t mistake one for a standalone archive. The interoperability considerations forapplication/aqrotherwise apply. - Published specification
- This document.
- Applications that use this media type
- Archivers, patching tools and updaters that implement the Aqueous overlay archive format.
- Fragment identifier considerations
- N/A
- Additional information
-
- Deprecated alias names for this type
- N/A
- Magic number(s)
- 69 0D 15 AF F6 B2 (hexadecimal; OVERLAY_HEADER_MAGIC) for an unencrypted overlay archive,
possibly preceded by other data.
An encrypted overlay archive has the same magic numbers as an encrypted archive of
application/aqr. - File extension(s)
- .aqo
- Macintosh file type code(s)
- N/A
- Person & email address to contact for further information
- Kang Seonghoon <public+aqr@mearie.org>
- Intended usage
- COMMON
- Restrictions on usage
- N/A
- Author
- Kang Seonghoon
- Change controller
- Kang Seonghoon
15.3.
+aqr Structured Syntax Suffix
This document requests the registration of a new structured syntax suffix in the Structured Syntax Suffix Registry ([RFC6838], section 6.2) as follows:
- Name
- Aqueous Archive Format
- +suffix
- +aqr
- References
- This document.
- Encoding considerations
- binary
- Interoperability considerations
-
A media type with the
+aqrsuffix denotes a standalone archive, either unencrypted or encrypted, whose archive value is given a more specific meaning by that media type. A generic Aqueous decompressor can interpret an archive of such media type without knowing that meaning. The interoperability considerations forapplication/aqr(§ 15.1 application/aqr Media Type) also apply. - Fragment identifier considerations
-
The syntax and semantics of fragment identifiers specified for
+aqrSHOULD be as specified forapplication/aqr. (At publication of this document, there is no fragment identification syntax defined forapplication/aqr.)The syntax and semantics for fragment identifiers for a specific
xxx/yyy+aqrSHOULD be processed as follows:- For cases defined in
+aqr, where the fragment identifier resolves per the+aqrrules, then process as specified in+aqr. - For cases defined in
+aqr, where the fragment identifier does not resolve per the+aqrrules, then process as specified inxxx/yyy+aqr. - For cases not defined in
+aqr, then process as specified inxxx/yyy+aqr.
- For cases defined in
- Security considerations
- The security considerations in § 14 Security considerations apply to any media type with the
+aqrsuffix. Each such media type may have additional security considerations specific to its interpretation of the archive. - Contact
- Kang Seonghoon <public+aqr@mearie.org>
- Author/Change controller
- Kang Seonghoon
15.4. Quantization Interval
This document requests the registration of a new key for the Time Tag Map Keys Registry ([RFC9581], section 7.3) as follows:
- Key Name
- Quantization Interval
- Key Value
- -14 (tentative)
- Reference
- The Quantization Interval section in this document.
- Semantics
-
A quantization interval specifies the interval in which the true time lies, relative to the time value given. That is, if the time value is t and the quantization interval is from a to b, the true time lies between t + a and t + b inclusive. The quantization is any means to truncate or round the true time value to a discrete representation, and notably, is not related to the accuracy of the underlying clock. For example, it is possible that the clock has a high precision but the time values are truncated to the last second, in which case the quantization interval would be [0, 1 second].
The interval is always closed. It may be slightly wider than the exact set of possible true times (e.g. [0, 1 second] for truncation, where the true time can never be exactly 1 second later), but it never excludes any possible true time, including ties in rounding.
A quantization interval can be given in terms of:
- A numeric value in seconds or a duration y, indicating the interval of [0, y];
- An array of two elements
[x, y]each of which is either a numeric value in seconds or a duration, indicating the interval of [−x, y]. That is, x is the extent before the time value and y is the extent after the time value.
Every numeric value MUST be finite and non-negative, and every duration MUST have a non-negative key 1, so that the time value given is always contained in the interval. The interval MUST NOT be a single point. Therefore a single numeric value or duration MUST NOT be zero, and an array
[x, y]MUST NOT have both x and y zero. If x is zero, the array MUST NOT be used and y MUST be given alone instead.If an extended time doesn’t contain this key, its quantization interval is unknown; in particular, the time value is not implied to be exact.
For example, the first two of the following represent a time value truncated to 10 milliseconds, and the last one represents a time value rounded to the nearest second:
1001({1: 1697724754, -3: 870, -14: 0.01}), 1001({1: 1697724754, -3: 870, -14: {1: 0, -3: 10}}), 1001({1: 1697724754, -14: [0.5, 0.5]})
The following CDDL declarations can be used to represent the Quantization Interval key:
quantization-interval = ~time / ~duration / [ 2*2 ~time/~duration ]
$$ETIME-ELECTIVE //= (-14: quantization-interval)
15.5. Aqueous Dictionaries Registry
This document requests IANA to create a new registry titled "Aqueous Dictionaries", listing the registered dictionaries that a shorthand block of type 5016 can refer to (§ 4.2.2 Registered dictionaries and modules). New entries are assigned with the Specification Required policy ([RFC8126], section 4.6). Each entry has the following fields:
- Hash
- The registered hash, i.e. the SHA-512/256 hash [FIPS180-4] of the dictionary’s bytes, written as 64 lowercase hexadecimal digits.
- Name
- A short name of the dictionary for humans, which has no effect on the format.
- Entropy Tables
- Whether the dictionary has entropy tables, i.e. the value of has_entropy_tables.
- Data
- The bytes of the dictionary, being the decoded data of a dictionary block (§ 5.2 Dictionary block), which MUST be valid as such and deposited with the registration.
- Reference
- The document describing the dictionary and its intended use.
The designated expert verifies that the hash matches the data and that no entry has the same hash. An entry, once registered, MUST NOT be modified or removed, since archives refer to it only by its hash. An entry is never shared with the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry), even when both have the same data.
The initial contents of the registry are as follows:
| Hash | Name | Entropy Tables | Reference |
|---|---|---|---|
| TBD | text | TBD | This document |
| TBD | modules | TBD | This document |
The dictionary "text" is intended for general text, and the dictionary "modules" is intended for module blocks given in full (§ 7.1 Module block).
The data and the hashes of the initial dictionaries are yet to be made.
15.6. Aqueous Modules Registry
This document requests IANA to create a new registry titled "Aqueous Modules", listing the registered modules that a shorthand block of type 6016 can refer to (§ 4.2.2 Registered dictionaries and modules). New entries are assigned with the Specification Required policy ([RFC8126], section 4.6). Each entry has the following fields:
- Hash
- The registered hash, i.e. the module hash of the module, written as 64 lowercase hexadecimal digits.
- Name
- A short name of the module for humans, which has no effect on the format,
written as the name of the module and its version joined by
@(e.g.flate-set@1). Different versions of a module have the same name before@and different hashes. - Data
- The bytes of the module, being wasm_module of a module block (§ 7.1 Module block), which MUST be valid as such and deposited with the registration.
- Reference
- The document describing the transformation that the module reverses, its inputs, resources and parameters, and preferably the source code from which the module was built.
The designated expert verifies that the hash matches the data, that no entry has the same hash, and that the module conforms to § 7.4 Module execution. An entry, once registered, MUST NOT be modified or removed, since archives refer to it only by its hash. An entry is never shared with the Aqueous Dictionaries Registry (§ 15.5 Aqueous Dictionaries Registry), even when both have the same data.
The initial contents of the registry are the modules built by the reference implementation of this document, as follows, linking the data (D) and the reference (R) of each entry:
| Hash | Name | Links |
|---|---|---|
6b0b52c29cf4c42c206a4fed017ac64ab76e0b320fb85c92e0c0be10829cc034
| ac3-recons@1 | D R |
0099d929fbc194ff15c8ec271bcd64d822671e780ae7cea046f4ec6c0410fcb9
| arsc-recons@1 | D R |
9e1f1f668a435d34f614f98ea502e00469e15062db1f06e22be60155ea439787
| asar-recons@1 | D R |
fea578bce88fa2c9441388f8add12b2cb79bedf9f40ca4f676561786db0e52fe
| astc-recons@1 | D R |
3810380b147efa2fb94a8d32deb9464b9202c831e68e941f2e0ab48acc17a2b9
| azo-recons@1 | D R |
e33ddb74d885e215d86f56a52d2845ddc4ef9ef12090ee8c920bc205423d4a09
| b2t-recons-quopri@1 | D R |
c586cd2eb9ea405891552c5b2ef82d6ec1c4263c3df041744f419220dc3f0599
| b2t-recons@1 | D R |
292160510dc876b2454bd65d404685858012e202c75f22d5d5fb606adf942832
| bc6h-recons@1 | D R |
eeb736245a8e13e936c6f5e0dcae855a3f8c8de09dc618f6a2fd06ee35dbbd3c
| bc7-recons@1 | D R |
8f3fd3dbf3966a8ecb590084559a4f93bdde14b6a9a50ad4f313e04bffc07dd9
| bcj-arm@2 | D R |
3e41f707a7655fa019757b913785a462b53e109e6257030e8c279a5156f9dc80
| bcj-arm64@2 | D R |
c073da28c681d39cb1acbe0c90ecafb8cf81baf86b2b916ab1fd251c1e787bde
| bcj-armthumb@2 | D R |
2d07cd639eadb1f671367f73a37ef64270dc4e1b324604832ea22cb5194bb9c2
| bcj-delta@1 | D R |
8c7843bcff41125e0bca042bd2ca518cfbf6641114c287318a688b71d7370f04
| bcj-ia64@2 | D R |
0c153b210a4aecf9edff3d3f828beb61726857d58d231025bd78a838983db0b8
| bcj-powerpc@2 | D R |
6f99ba289eb005969ed4292d66da6e630b8ea3de3906ac634a68e8ba9e7f71b1
| bcj-riscv@2 | D R |
908cb8d3f0802b9d6aeb1cc9770414945be8534118f4d60095c4450d4521630f
| bcj-sparc@2 | D R |
e844d7465f87c75072153637f3af2923608688c6b281ceea6a46684a251bc19c
| bcj-swap2@1 | D R |
333a88f38b91e85050069df2537d7d127cfc531ed2e56c90d24a88b028915df3
| bcj-swap4@1 | D R |
bc5de18653206afc1c83a8179acbb9b04463cf4e62b23269b5d6fcd56f4557a7
| bcj-x86@2 | D R |
4c223c70286d04e9b4c63344706de56a40dd85d3f21bec10cca3ebf49f8f998a
| bcj2@2 | D R |
fc0c88b21c5b9be96204333446e41beb5dfc5c6a65a014eae2c35b6f69e82ca9
| bcn-recons@1 | D R |
bc7e3751cff6f7912ff2d3d8f026b04be2b7210087384e7e2ae40b4fbcd6d29e
| bundle-recons@2 | D R |
76b6880cf027967acad5ad7434de59a100adfd6b17269b2bb2648d6ff2bb1b45
| bvx-recons@2 | D R |
4fb20ee608bbe840fe3d943fc73279f3283994f591a1f1911b730c89dfa3e554
| bz2-alz-recons@2 | D R |
05b7063a2d3c99f212239d6d323124540eecb39f8f152a3d61848ed548e409f5
| bz2-recons@2 | D R |
f192f3cdfac2ab652448f1ade2a3465600796ff41ed73ef212ed9781744d6b39
| clip-recons@3 | D R |
af79fb4e146afc56f035ee9265b571db81cf3c0847dd42816b71196a13fd8785
| dex-recons@2 | D R |
020628e8ed27f952b018beb0e4413c04a3566bfede4cd21261d55e426d4db77a
| dh-lods@1 | D R |
d717646aa3c6c2cf08556b979e86574d26d1dd9e7814ef96a709d781ce6ff9e8
| div3-recons@3 | D R |
ab66f2ea4aac7cda80980ff50e21496bd34915c9553c597d1c43af4a98fb1262
| dwarf-apple@1 | D R |
35289b7ef1d0d75f99c3948cddb76e4e0f9304d592d2fb40e5830fc4516c12a4
| dwarf-info@2 | D R |
9ad20aa3412e699e66cc296cdfd48dda49b86e11cd771cb3f4384e2fd2bcf869
| etc-recons@1 | D R |
dbd3de347ffd906cc61c0b1a1fb685e7f1b47c71d2833e00e0c41e16599be597
| exe-recons/arm64-delta@1 | D R |
465a349b04736a5d34715cd5217ce5e8665feac7905b245eaeb736dad856d291
| exe-recons/arm64-remap@1 | D R |
78cf95749c7f980cd31edf253aa36afb8bc753cf530eb226fd1a3c5c86e80ae3
| exe-recons/arm64@2 | D R |
be9df7c3bc994a855d72983b81d5d48d567a82a1f3f4336438dd72962c8f9afd
| exe-recons/cil@1 | D R |
85de1c244f9ca7ae68395320117abebe8825a750c063d0e6cf91555a885f3899
| exe-recons/macho-codesign@2 | D R |
d256bcb992b46b1de858ace1e256889192beb6627ee1514b16f23390c70df4cf
| exe-recons/macho-symtab@1 | D R |
b3236ff145458aa4a53efe64fdbabf544e8d4b282aa94188c58cdfb41635bfcd
| exe-recons/pe-reloc@3 | D R |
a8d092a4eee443f92f6a14537ef0e1637300105516106fa860bdecbf51868aad
| exe-recons/x86-remap@2 | D R |
1ecc85b704093e11c8daa5024468bafb4d4496fc52e3ecf016cfd10af0c51581
| exe-recons/x86@6 | D R |
3e24ee2f6af1d4a1aeab02a6ff46ed13f2ca7d94c8ce806e6de8bbd93d21b657
| fasta-recons@3 | D R |
20a656984d70844d4658cb62af20d2437d100e2e174e8ac4bb58a6301f4a5a27
| fits-recons-image@1 | D R |
5ccaa3f93e4296657a8a325e839804777fe8976d7f301859b079109d3bb6e622
| fits-recons-rice@1 | D R |
aa24b0e80a0856cdc07a600f5023252f7578c66e94c0ac355efd434f85ecd49e
| fits-recons-table@1 | D R |
ff919a0955fc9353059af53847f1f84b803b4115e6812fc51111eda80a2599fc
| flac-recons@5 | D R |
8f636ea534d9114fac5504a9922d4dcc999e8a660dc9ecc2885f5459fd10e813
| flate-inflate@1 | D R |
e7f26ac070c92a66a8b3cba961051b8e2703c5726bea71760dc5bb7c462b4cd6
| flate-set@4 | D R |
8f53a39d83da2c850989320f0f64709ceaec9d8d4046c03258cf776a9b496288
| gif-recons@2 | D R |
d7cd04da761d4b79f181894a2c67f24f857762afc415beb907cfc7c234533ab2
| git-pack-index-recons@2 | D R |
b7ae16ba012c44fd9881c4d01bf3059aecec77c9b72d9690e87babf9c503a563
| git-pack-recons@2 | D R |
c14b56b9abe1f911ce22d4092244484cac068e3796d838afedb5a72b071d5aeb
| gnu-gettext-recons@1 | D R |
9234acad545b70a11c46bdc56e27be21a4c0314d20ecb23566d82be07a7cca04
| hdf5-recons-float@1 | D R |
ebd2abd8e92185a1c0fbaef7785daa90582f39c4c779d2de3672a9c896a60879
| ima-recons@2 | D R |
557f820cce081587adcfa218a3a9de9e0b20bd7cf0aa37735fc4e92fbe43c175
| jpeg-lossless-recons@1 | D R |
bc399e90ff8a4789a500aeb395991029b742fb6d419426a6393988067f188838
| jpeg-recons@3 | D R |
d1379510a62a8e5b41b4e394aa1ff0a7699a48fe7f0470d56e6a7a7bd631f7d8
| jsonl-recons@2 | D R |
e745c2292175052983f2ece3d4a529c4970436bc44341b500fc94a00a6ea5931
| leveldb-recons@1 | D R |
28f2d22eaae6c7175e6e7d56e357b42be53b000679258542d29ea19c2052ab25
| lha-recons@2 | D R |
5b602c335bfb01c9b5aa231c241922efcef2e8a00f30cd8ec87731c8ce0c6739
| log-recons@1 | D R |
57fc6a5976a9a38769a7ad6b2482df4e3fbdaed065dd37207d3731f7bc99676e
| lz4-recons@2 | D R |
0a4aae3de201085372643b1d52ffc77cdfb377a7d5613d08223ff0989f5e86b4
| lzma-decode@1 | D R |
3e8fc43baf255f73b9cb90c512eb5795634d2d31560c07ae03e0cebec61ba8f0
| lzma-recons@4 | D R |
c579dbf1ad2c0f7a691421ba2bbd6f6212d3ede3636de21f9c3d33650115c891
| lzma2-recons@3 | D R |
a5a74899b18ee623da7e6014f0642af223f9653f2aa407bb0b22730790d9d54a
| lznt1-recons@1 | D R |
a9a9c84319d717df983dd3b56586d7b9ddfbbd16d0c3a856c7d96a759586c1a1
| lzo-recons@1 | D R |
2a357d92e82fa7dd6dc787f54fe6a49a8ff12c52a96dbda95ec296f0c5e245b3
| lzx-recons@3 | D R |
216344f2356e88e2604fbd0db7d499ac9b5381777bf414c548f7b8622dfaacbd
| mca-recons@2 | D R |
28a35fb096a31414a7c1c794722a195e2c78d438adf4c7f1ea1cb353aeae7f8a
| mp3-recons@6 | D R |
7ed36e1fe53ea5337a3e0950f6e5a464f9984508acfaa3c378adf4fa363552f7
| msadpcm-recons@1 | D R |
57fdb061a4e2bbc30c471c088608d528582ef0d4343b84b340e08e5f396d7352
| mszip-recons@2 | D R |
5af417ac62b361b73251aa5b7a045ea05e2d70d475e18f308ab092264345d436
| obj-recons@1 | D R |
192f4385f28ea6cfb2bebe9d012d8769e2a09cccd7e8d0192ba5430c233cb7fa
| ogg-recons@1 | D R |
2d4f76cfc71cacaf918b3f94dc59527b9b1b7303e0d69b4a97c98d3107af8ce3
| pcm-recons@2 | D R |
a3008d37eae4a8bf1474f79975da0e834212a7e79cb98f5992553a8c6ce0130e
| pdb-msf@1 | D R |
52ce0b70658a83169d5329170d782a2e75ddab60ef2a2ca767cb6b84db20d981
| pdb-symbols@2 | D R |
33bfc52d05afcbe85d7bf7b7b8c56a17cec6a5442240964ddf905cbeae3ff499
| pdb-types@1 | D R |
fac37428ac2315cb88d97fe30191cf33aa3338abcbf56faf4245f1f8925db312
| pdf-recons@3 | D R |
dbfd85f28b73603d8c58b72d67066c56cde661def7e63102aba976906e2fcb93
| png-against@2 | D R |
5680af51169fb124b42513f39bddbe55b037a18fac3c4b83b18ac0da29e25c9e
| png-recons@5 | D R |
f4e9b4f65167e3e7160117cf654f5545d228b2bac9c8cdc2d2a2991ad25c2e95
| png-set@3 | D R |
1f14c4d1f1436b106ccbf51e2e5aeaf89d5b0a321ed450fbf340118cd0cf6937
| prores-recons@3 | D R |
aad61c9304a0294838a7b01cb8045fcf9c44d96364c1ac58b0ad0abf1dce4489
| psd-recons@1 | D R |
160f55905d28d1842ef780b1781dfac3e2dbe8191eef4950dceb89c7320daac2
| pvrtc-recons@1 | D R |
ddf12e6354a898fd2fded827c2d0d9aedc0b332292b15ccd3c4e3ba8d31e5fe6
| rar3-recons@2 | D R |
f65035f8334e69b382b112edf92f28036f2d2072c37002f1b6168665113e61cf
| rar5-recons@2 | D R |
2e4858b242829592dbdcc037e32ad37c16a96cae44db0aa8dd38dca680370a4c
| raster-recons-16@1 | D R |
63773d1d5a107c856845be86dc2de897bfd61f9b1f4316a1cf0851082a610ca3
| raster-recons@1 | D R |
28b6963b212164a89006f2c0bc6f53d4cd945d9f8a5f956d57d1219be010fc4f
| recs-entries@1 | D R |
4279aab3b66a45067ace10cad8f979f46b27ed73ae31624a5728c783e4aefa65
| recs-fields@1 | D R |
dac6184b0cdfd1f3e46cf83c420549cf7af5e90c99770ab6865646766f3dc767
| recs-floats@1 | D R |
59fbb34cc7d041590435682f30fe4edbb704769acb0213e0339417535b2f409f
| snappy-recons@3 | D R |
4b544507f2459700d9b1e2683f3366dc2230166e8c8461f4d166f4ec90f71347
| sqlite-recons@2 | D R |
9b46a81e6b86836750b2adcf1babd36ec39dbb81f9a04872b992a7af0459d11c
| tensors-recons@2 | D R |
24d85c63a9bb175e5ab281e055832afc1037446abbdf529b77c818a3b4d41eba
| vorbis-recons@3 | D R |
7ca6b6cdddb3bce10ad90a975a7c92aef3fef7025976f24abb5686edf0805da9
| wasm-recons@1 | D R |
9d5c184ff4736bf54e10966f57542fc74977c9fe2684f1668c05c4d303a59bae
| wrt-recons-hangul@2 | D R |
d7a2ad763a9c9ef48cfb8a35ba24b6654e03fa8fbe2879f50109750c8dacc8f8
| wrt-recons@3 | D R |
bbd506258ef6650e59216eb1538ce1211cf93f5a5746ce1b640ed025ff91779f
| xpress-recons@1 | D R |
e24ecbe10de42e77f1f74d524333728f70b7473dace96014c142e1cdad8f2074
| xsv-recons@2 | D R |
60b2944db28669697e24767170b5914cf04a7111bf84c7f044c76781b38238a1
| zip-recons@6 | D R |
1d8087f725c1dd6847001053205a6fdd4d54f870feba0978e6c7bce7f54d1ecc
| zstd-frames@1 | D R |
The initial entries are not yet fixed. Their modules and the documents describing them may still change before the first version of this document, and an entry changed until then is replaced in this table, not added as a new version.
16. Converting from ZIP
This section is non-normative.
This section describes how a compressor can convert entries of a ZIP archive [APPNOTE] into an archive value without losing information, where ZIP offers multiple sources or encodings for the same information.
16.1. Entry names from ZIP
A ZIP archive [APPNOTE] stores the path of each entry as a byte sequence in the local and central headers,
with components separated by /.
Its encoding is not always known from the header alone,
and several extra fields carry the Unicode version of the same path.
A compressor converting a ZIP archive can determine the encoding of each component from the following sources,
where earlier sources are preferred:
-
Extra field 0x7075 (Info-ZIP Unicode Path) of version 1 [INFOZIP-EXTRAFLD], whose
NameCRC32matches the CRC-32 of the path in the header. The field holds the path in UTF-8. A field with a mismatching CRC-32 is ignored as specified by [APPNOTE], as it may have been left over by a tool that changed the path without updating the field. -
The path in the header, if bit 11 (language encoding flag) of the general purpose bit flag is set. The path is in UTF-8.
-
Extra field 0x554e (Xceed Unicode), which holds the path in UTF-16LE. It is converted to WTF-8, which preserves any lone surrogate.
-
Extra field 0x5a4c (ZipArchive code page), which holds the path encoded in the code page given in the same field. If the field only has the code page, the path in the header is decoded with it instead.
-
Extra field 0xe57a (ALZip code page), which gives the code page of the path in the header.
-
Otherwise, the encoding of the path in the header is not specified. It is traditionally IBM code page 437, but often the system code page of the machine that made the archive, so the compressor has to guess the encoding. If it can’t, the raw byte sequence (prefixed with C116) is used for each component.
16.2. Timestamps from ZIP
A ZIP archive [APPNOTE] can carry the same timestamp in multiple places: the DOS date and time in the local and central headers, and extra fields 0x000a (NTFS), 0x5455 (extended timestamp), 0x5855 (Info-ZIP UNIX type 1) and 0x000d (UNIX) [INFOZIP-EXTRAFLD]. A compressor converting a ZIP archive can choose the source of each key as follows, where earlier sources are preferred. The order prefers more precise sources first, and then more recent specifications for the same precision.
| Key | Sources |
|---|---|
crtime
| 0x000a Ctime; 0x5455 CrTime (local header only)
|
ctime
| - |
mtime
| 0x000a Mtime; 0x5455 ModTime; 0x5855 or 0x000d ModTime; DOS date and time
|
atime
| 0x000a Atime; 0x5455 AcTime (local header only); 0x5855 or 0x000d AcTime
|
arctime
| - |
Each source is encoded as follows:
-
0x000a stores a Windows
FILETIMEt, the number of 100 ns intervals since 1601-01-01T00:00:00Z, which is encoded as125(t − 116444736000000000). A zero value means that the timestamp is absent. -
0x5455, 0x5855 and 0x000d store the number of seconds since 1970-01-01T00:00:00Z, which is encoded as an untagged
int. It is a signed 32-bit integer according to [INFOZIP-EXTRAFLD], though some implementations write times after 2038 as unsigned. The central header version of 0x5455 only hasModTimeeven when its flags indicate others, so the local header version is preferred when available. -
The DOS date and time is a local time in an unknown time zone with the precision of 2 seconds, which is encoded as
[121(n)]where 2n is the local time as if it were UTC.
When both the DOS date and time and a UTC source are present for mtime,
their difference reveals the time zone offset where the archive was made.
If the DOS time minus the UTC time differs from 900k seconds by less than 2 seconds,
where k is an integer in the range of aqr-tz-offset,
mtime can be encoded as [u, k] where u is the encoding of the UTC source.
Otherwise, e.g. due to a clock error or a daylight saving time transition, the DOS date and time is simply discarded.
If two sources for the same key disagree by more than their quantization intervals, a compressor can warn about it, and still uses the preferred source.
16.3. Comments from ZIP
A ZIP archive [APPNOTE] stores a comment for each entry in the central header only,
and a comment for the whole archive in the end of central directory record.
Both are byte sequences, and an empty comment is no different from the absence of one,
so the compressor can omit comment or archive-comment for an empty comment.
A compressor converting a ZIP archive can determine the encoding of each entry comment from the following sources, where earlier sources are preferred:
-
Extra field 0x6375 (Info-ZIP Unicode Comment) of version 1 in the central header [INFOZIP-EXTRAFLD], whose
ComCRC32matches the CRC-32 of the comment in the header. The field holds the comment in UTF-8. A field with a mismatching CRC-32 is ignored, as for 0x7075 in § 16.1 Entry names from ZIP. -
The comment in the header, if bit 11 (language encoding flag) of the general purpose bit flag is set. The comment is in UTF-8.
-
Extra field 0x554e (Xceed Unicode) in the central header, which holds the comment in UTF-16LE. It is converted to WTF-8, which preserves any lone surrogate.
-
Otherwise, the encoding of the comment is not specified. The compressor can decode it with the encoding guessed for entry names in the same archive. If it can’t, the raw byte sequence is used as a
bstr.
The archive comment has neither the language encoding flag nor an extra field,
so its encoding is never specified by the archive itself.
The compressor can decode it as in the last source above, and puts the result into archive-comment.
16.4. Unix metadata from ZIP
A ZIP archive [APPNOTE] can carry the Unix metadata of an entry in the external file attributes of the central header and in extra fields 0x7875 (Info-ZIP UNIX type 3), 0x7855 (Info-ZIP UNIX type 2), 0x756e (ASi UNIX), 0x5855 (Info-ZIP UNIX type 1) and 0x000d (UNIX) [INFOZIP-EXTRAFLD]. A compressor converting a ZIP archive can choose the source of each key as follows, where earlier sources are preferred and sources in the same item have the same precedence.
| Key | Sources |
|---|---|
unix-mode
| External file attributes; 0x756e Mode
|
unix-uid
| 0x7875 UID; 0x7855 UID (local header only); 0x756e, 0x5855 (local header only) or 0x000d UID
|
unix-gid
| 0x7875 GID; 0x7855 GID (local header only); 0x756e, 0x5855 (local header only) or 0x000d GID
|
Each source is encoded as follows:
-
The upper 16 bits of the external file attributes hold the Unix
st_modeif the upper byte of "version made by" is 3 (UNIX). Only its permission bits (0o7777) go tounix-mode, and its file type bits (S_IFMT) are not stored there. -
0x7875 of version 1 stores the user and group IDs as little-endian unsigned integers of variable sizes. [INFOZIP-EXTRAFLD] specifies that it takes precedence over 0x7855. Other versions are not used as a source.
-
0x7855, 0x756e, 0x5855 and 0x000d store the user and group IDs as 16-bit unsigned integers. The central header version of 0x7855 is empty and only indicates that the local header version exists, and the central header version of 0x5855 doesn’t have the user and group IDs.
If two sources for the same key disagree, a compressor can warn about it, and still uses the preferred source.
16.5. Extra fields not to be preserved
The following extra fields [APPNOTE] [INFOZIP-EXTRAFLD] are not put into zip-extras,
because their information is either absorbed into the archive model and other keys,
or discarded as only meaningful to the ZIP container.
| Header ID | Name | Treatment |
|---|---|---|
| 0x0001 | Zip64 extended information | Discarded: sizes and offsets |
| 0x0007 | AV Info | Discarded: signature |
| 0x000a | NTFS | Absorbed: timestamps (§ 16.2 Timestamps from ZIP) |
| 0x000d | UNIX (PKWARE) | Absorbed: timestamps (§ 16.2 Timestamps from ZIP), unix-uid and unix-gid (§ 16.4 Unix metadata from ZIP), the target of a link node, link-group for a hard link, and the device number of a special node
|
| 0x0014 | PKCS#7 Store for X.509 Certificates | Discarded: signature |
| 0x0015 | X.509 Certificate ID and Signature for individual file | Discarded: signature |
| 0x0016 | X.509 Certificate ID for Central Directory | Discarded: signature |
| 0x0017 | Strong Encryption Header | Discarded: encryption |
| 0x0019 | PKCS#7 Encryption Recipient Certificate List | Discarded: encryption |
| 0x0021 | Policy Decryption Key Record | Discarded: encryption |
| 0x0022 | Smartcrypt Key Provider Record | Discarded: encryption |
| 0x0023 | Smartcrypt Policy Key Data Record | Discarded: encryption |
| 0x10c5 | minizip CMS signature | Discarded: signature |
| 0x1986 | Pixar USD header ID | Discarded: padding for alignment |
| 0x5455 | Extended timestamp | Absorbed: timestamps (§ 16.2 Timestamps from ZIP) |
| 0x554e | Xceed Unicode | Absorbed: entry name (§ 16.1 Entry names from ZIP) and comment (§ 16.3 Comments from ZIP) |
| 0x5855 | Info-ZIP UNIX type 1 | Absorbed: timestamps (§ 16.2 Timestamps from ZIP), unix-uid and unix-gid (§ 16.4 Unix metadata from ZIP)
|
| 0x5a4c | ZipArchive code page | Absorbed: entry name (§ 16.1 Entry names from ZIP) |
| 0x5a4d | ZipArchive offsets array | Discarded: offsets |
| 0x6375 | Info-ZIP Unicode Comment | Absorbed: comment (§ 16.3 Comments from ZIP) |
| 0x6c78 | libarchive "xl" | Absorbed: a copy of the central header fields |
| 0x7075 | Info-ZIP Unicode Path | Absorbed: entry name (§ 16.1 Entry names from ZIP) |
| 0x7855 | Info-ZIP UNIX type 2 | Absorbed: unix-uid and unix-gid (§ 16.4 Unix metadata from ZIP)
|
| 0x7875 | Info-ZIP UNIX type 3 | Absorbed: unix-uid and unix-gid (§ 16.4 Unix metadata from ZIP), for version 1 only
|
| 0x9901 | AE-x (WinZip AES) | Discarded: encryption |
| 0x9903 | WinZip Reference | Discarded: reference to another entry, resolved by conversion |
| 0xa11e | Data Stream Alignment (Apache Commons Compress) | Discarded: padding for alignment |
| 0xa220 | Microsoft Open Packaging Growth Hint | Discarded: padding for in-place editing |
| 0xcafe | Java JAR marker | Discarded: information about the ZIP file itself |
| 0xcdcd | minizip Central Directory | Discarded: encryption and signature of the central directory |
| 0xd935 | Android ZIP Alignment | Discarded: padding for alignment |
| 0xe57a | ALZip code page | Absorbed: entry name (§ 16.1 Entry names from ZIP), if decoded with that code page |
-
0x756e (ASi UNIX) is absorbed like 0x000d: the permission bits of its mode go to
unix-mode, the user and group IDs go tounix-uidandunix-gid, and the symbolic link target goes to a link node. Its device number, however, is a rawst_rdevvalue whose bit layout differs among platforms, and can’t be split into the major and minor numbers of a special node. A device file is therefore converted to an empty file node with the original extra field preserved. -
0x4b46 (FWKCS MD5) and 0x1a51 (minizip hash) hold digests of the uncompressed content. They are only preserved when they match the actual content.