Aqueous Archive Format

1. Introduction

This section is non-normative.

The Aqueous archive format stores a directory tree, or a single file, as a sequence of blocks. This section gives an overview of how the parts of the format fit together; each paragraph points to the chapter defining the part it describes.

An archive is a header, a sequence of blocks and a footer (§ 3 Container). Every block starts with its type and its size, so a decompressor can skip any block without understanding it, and the type tells whether the block can be ignored when it is not understood (§ 3.1.1 Block types). The footer points back to the last metadata block, so that a decompressor can start reading from the end of the archive.

Blocks are grouped into segments, each ending with a metadata block (§ 4 Segments and entries). The metadata block names the entries of the segment, i.e. directories, files, symbolic links and other nodes, and points each file to the block holding its content. A later segment adds, replaces and removes entries of earlier segments, so that an archive can be extended by appending a segment without rewriting what precedes it. An archive without any metadata block simply decompresses into a single unnamed file, as a gzip file does.

The content of a file is given by a single leaf block. The simplest leaf block is a data block storing the content itself, optionally compressed with Zstandard and a dictionary shared by many small blocks (§ 5 Stored data). A formula block is a leaf block that describes the content instead, as a small stack machine program over non-leaf blocks following it (§ 6 Formulas). The program takes slices of them, concatenates them, patches them, and applies WebAssembly modules to them to undo a transformation made by the compressor (§ 7 Modules). A leaf block and the non-leaf blocks following it form a parcel, which is typically all that a decompressor reads to extract a single file, and the formula block serves as an index for random access within it.

An overlay archive describes a directory tree as changes to an existing one, called its base (§ 8 Overlay archives and imports). Import blocks read byte ranges of files in the base, which formulas can patch, so that an overlay archive works as an update or a delta of the base.

The whole archive may be encrypted with a passphrase using age [AGE] (§ 9 Encrypted archives). Checksums cover every block except, optionally, leaf data blocks, and except shorthand blocks, which name registered dictionaries and modules by their hashes (§ 4.2.2 Registered dictionaries and modules). Mirror blocks hold copies of the blocks whose damage would affect many files (§ 10 Integrity and recovery). Metadata that is not necessary to read the files, such as timestamps, comments and permissions, is encoded in CBOR and may be ignored by a decompressor (§ 11 Optional metadata, § 12 Extraction).

2. Conventions

2.1. Expressions

All expressions are interpreted according to the standard arithmetics, with the following extensions:

• a / b equals floor(a ÷ b).

• a % b, where b > 0, is a non-negative modulo of a divided by b, and equals a - (a / b) × b.

• a << b equals a × 2^b.

• a >> b equals a / 2^b.

• XX₁₆ denotes a hexadecimal integer.

• (a, b, …) denotes an ordered array of values.

• (XX YY ZZ)₁₆ is a shorthand for (XX₁₆, YY₁₆, ZZ₁₆).

• a || b concatenates two arrays a and b, treating any non-array value as a singleton array (a) or (b).

• For array a, aⁿ equals a || a^n-1 when n > 0. a⁰ equals an empty array ().

• [a, b] denotes a closed interval between a and b, both inclusive.

• [a, b) denotes a half-open interval between a (inclusive) and b (exclusive).

All expressions result in an integer between -(2⁶⁴ - 1) and 2⁶⁴ - 1, or an array of those integers with the number of elements not exceeding 2⁶⁴ - 1. Note that subexpressions still may exceed this range.

2.2. Types

Types are a set of values and associated encoding scheme. A type or a structure is implicitly-sized when its byte length is only available through the context. Unless stated as being implicitly-sized, all types have the defined byte length out of the context.

u8 is an unsigned 8-bit integer, occupying 1 byte. Its range is [0, 2⁸).

u16le is an unsigned 16-bit integer, occupying 2 bytes, stored in little-endian byte order (least significant byte first). Its range is [0, 2¹⁶).

u32le is an unsigned 32-bit integer, occupying 4 bytes, stored in little-endian byte order. Its range is [0, 2³²).

u64le is an unsigned 64-bit integer, occupying 8 bytes, stored in little-endian byte order. Its range is [0, 2⁶⁴).

u(n), where n is an integer less than 2⁶⁴, is the smallest integer type that can represent n if n > 0, and a zero-sized type if n = 0. In other words,

• If n = 0, u(n) is zero-sized.

• If n ∈ [1, 2⁸), u(n) is the same as u8.

• If n ∈ [2⁸, 2¹⁶), u(n) is the same as u16le.

• If n ∈ [2¹⁶, 2³²), u(n) is the same as u32le.

• If n ∈ [2³², 2⁶⁴), u(n) is the same as u64le.

leb128 is an unsigned 64-bit integer encoded using the variable number of bytes with the LEB128 encoding. Each byte, except the last, has its most significant bit (MSB) set to 1, indicating that more bytes follow. The remaining 7 bits of each byte are used to store the integer’s value. The least significant 7 bits of the first byte are the least significant bits of the integer, and so on. The last byte has its MSB set to 0. Its range is [0, 2⁶⁴): an encoding MUST be at most 10 bytes long, and MUST NOT encode a value of 2⁶⁴ or more.

a[n], where n is an integer, denotes an ordered array of as. n’s range is [0, 2⁶⁴).

a[*] denotes an implicitly-sized ordered array of as with an unspecified number of elements. The number of elements is implicitly determined from the context. This specification uses two kinds of such contexts:

• When one or more a[*] fields are in the structure with the defined byte length, the numbers of elements in those fields are increased as needed to exactly fill the required byte length. There MUST be a single set of the numbers of elements that satisfy these conditions.

• When an a[*] field is given with a constraint, the field extends until, but not including, the first instance of a which doesn’t satisfy the required constraint.

2.3. Structure

Structure is a logical group of one or more ordered fields of possibly differing types concatenated together. Each field has the following properties:

• The descriptive name of the field (e.g. size, compression_params).

• The data type of the field (e.g. leb128, u8[size]), with an optional constraint.

• Optional criteria or conditions that dictate the presence, value, or interpretation of the field.

Field names can only coincide when the field’s type varies by conditions, in which case at most one such condition is applicable at any time. The name can be omitted when the field’s name is the same as the previous field’s.

The field type may be offset, which is a special zero-sized type interpreted as the offset to the following byte relative to the unspecified file position (origin). Offsets are only used to measure the number of bytes within a set of fields. The syntax and semantics of this specification don’t depend on the choice of origin.

The optional constraint does not affect the field’s encoding and only narrows down the possible set of values. Constraints can be one of the following:

• a ∈ X requires that the value is in the specified set X.

• a ∉ X requires that the value is not in the specified set X.

• a = x requires that the value is exactly the given value x. If x is a constant expression, this effectively fixes the byte encoding of this field.

• a ≠ x requires that the value is not the given value x.

The conditions may be one of the following:

• "Optional" indicates that conforming implementations MAY include or exclude given field as needed.

• "If …" indicates that the field MUST be included if and only if the specified condition is true.

• "Depends on …" indicates that the field’s interpretation varies with the specified set of other fields.

Structures are implicitly-sized only when their byte length is not available through the context and at least one implicitly-sized field exists therein.

2.4. Constants

This specification defines and uses the following constants, all of which are denoted in UPPERCASE.

2.5. Semantics and failures

This specification defines the meaning of an archive as a function from a byte sequence to an archive value, which also takes a base for an overlay archive. Every algorithm in this specification is declarative: it defines what the result is, not how or in which order it has to be computed, and a conforming implementation MAY compute it in any other way that produces the same result.

The result of interpreting an archive is either an archive value or a failure, which is one of the following:

• An invalid archive failure occurs when the input violates any requirement of this specification. It is a property of the input alone, and all conforming decompressors agree on it.

• A resource failure occurs when the evaluation exceeds a declared or implementation-defined limit. It is not a property of the input, and conforming decompressors MAY disagree on it.

• A decryption failure occurs when an encrypted archive can’t be decrypted with the given passphrase, if any. It is a property of the input and the passphrase, and all conforming decompressors agree on it.

• A base failure occurs when an overlay archive is interpreted without a base, or an import can’t be resolved in the given base (§ 8.2 Resolving an import). It is a property of the input and the base, and all conforming decompressors agree on it.

• An unsupported failure occurs when the input contains a block of a critical type not defined by this specification (§ 3.1.1.1 Reserved block types), or a shorthand block whose registered hash the decompressor doesn’t know (§ 4.2.2 Registered dictionaries and modules). It is a property of the input, the version of this specification and the registry entries known, and all conforming decompressors implementing the same version and knowing the same entries agree on it.

A conforming compressor, given an archive value, MUST produce a byte sequence whose interpretation is that archive value. When it produces an overlay archive, it is also given a base, and the interpretation with that base MUST be that archive value. This specification doesn’t constrain how the compressor chooses blocks, block sizes, dictionaries or formulas.

3. Container

3.1. Block

Blocks are a typed byte sequence. The entirety of an unencrypted archive and the beginning of an encrypted archive consist of either blocks or block-like structures.

Blocks have the following generic structure:

3.1.1. Block types

The following table lists every range of block types. A range of block types for a single logical block type is also described in binary, e.g. 0001 0mss, where letters denote bits that determine the parameters of the block. ss always denotes the two least significant bits, which determine the width of size as above: 0 for u8, 1 for u16le, 2 for u32le and 3 for u64le. Whether a reserved block type is critical is determined by its leading digit (§ 3.1.1.1 Reserved block types).

Block types of 2 hexadecimal digits are as follows:

Block types of 3 or more hexadecimal digits are as follows:

3.1.1.1. Reserved block types

The leading digit of a block type is the first digit of its hexadecimal representation, padded with zeros to at least 2 digits. For example, the leading digits of 4₁₆, 4B₁₆ and C10₁₆ are 0, 4 and C respectively.

A block type is critical if its leading digit is 0 through 9, and non-critical if it is A through F. A block whose type is not defined by this specification:

• Results in an unsupported failure (§ 4.4 Interpreting an archive) if its type is critical;

• Is an ignored block otherwise. Like a mirror block, an ignored block doesn’t belong to any segment, MAY occur anywhere in blocks and doesn’t affect the result of interpreting an archive.

Blocks whose type’s leading digit is D (i.e. D0₁₆ through DF₁₆, D00₁₆ through DFF₁₆, …) are reserved for development environments. This specification doesn’t assign any meanings to these blocks. These blocks MUST be written or read only with specific instructions to enable them.

Blocks whose type’s leading digit is E (i.e. E0₁₆ through EF₁₆, E00₁₆ through EFF₁₆, …) are reserved for experiments. This specification doesn’t assign any meanings to those blocks.

The following block types are defined as invalid due to the use as block-like structures. A valid archive MUST NOT include such block types as a proper block.

• Block 278₁₆ (HEADER_TYPE) is used for the unencrypted archive header.

• Block 69₁₆ (OVERLAY_HEADER_TYPE) is used for the overlay archive header.

• Block 90 5BCB 957D₁₆ (FOOTER_TYPE) is used for the archive footer.

• Block 7FC₁₆ (OVERLAY_FOOTER_TYPE) is used for the overlay archive footer.

• Block 155 04DC A579₁₆ (ENCRYPTED_HEADER_TYPE) is used for the encrypted archive header.

3.2. Shared external structure

Many block types share the same external structure, only varying its internal structure. The structure may or may not be compressed, and uses a compression_method and a checksum_method specified by the parent structure. Each compression method defines the structure of its parameters (compression_params), whose byte length is determined by parsing it. Each checksum_method also has an associated, fixed checksum_size. Checksummed Zstandard data is the compressed external structure (compressed_data) with Zstandard as compression_method and XXH64 as checksum_method.

The compressed external structure (compressed_data) consists of the following:

The uncompressed external structure (uncompressed_data) consists of the following:

In both structures, the checksum is computed over the decoded data (§ 5.3 Block contents), i.e. the data after decompression, rather than the stored bytes (§ 10.1 Checksums).

3.2.1. Zstandard compression method

This specification only defines a single compression method, Zstandard [RFC8878]. A Zstandard block is the "Block" structure in the Zstandard frame ([RFC8878], Section 3.1.1.2). Its parameters (compression_params) are as follows:

Data compressed by this method is a sequence of Zstandard blocks, without an additional framing provided by Zstandard frames. data MUST consist of one or more such blocks without any other bytes, where only the last block is signalled as the final block by its block header.

3.2.2. XXH64 checksum method

This specification only defines a single checksum method, XXH64 [XXHASH]. Its seed MUST be zero and the result MUST be stored as u64le. The resulting checksum_size is 8.

3.3. Spacer block

Blocks 0₁₆ through 3₁₆ are spacer blocks. Their data consists of the following:

A spacer block occupies any even number of bytes from 2, which is sufficient since every block occupies an even number of bytes due to padding.

Conforming compressors MUST fill space with zero bytes in order to avoid data leak, but its content doesn’t affect the result of interpreting an archive. Conforming compressors are RECOMMENDED to utilize sparse files for performance.

3.4. Size hint block

Block 4₁₆ is a size hint block. Its data consists of the following:

3.5. Unencrypted archive

An unencrypted archive consists of a header, zero or more blocks and a footer.

3.6. Locating an archive

A file may contain an archive with leading bytes, e.g. a self-extracting executable, or with trailing bytes, e.g. appended by a transfer. Since interpreting an archive takes the archive alone, a decompressor accepting such a file has to locate the archive first.

A header magic and a footer magic match when they are HEADER_MAGIC and FOOTER_MAGIC, or OVERLAY_HEADER_MAGIC and OVERLAY_FOOTER_MAGIC.

To locate an archive in a byte sequence file:

1. If file starts with ENCRYPTED_HEADER_MAGIC or ENCRYPTED_MIRRORED_HEADER_MAGIC, return file.

2. Let back be null. Let n be the last 2 bytes of file as u16le, and let f be the length of file − n − 10. If f ≥ 0, the bytes of file from f are FOOTER_MAGIC or OVERLAY_FOOTER_MAGIC followed by n as u16le, and the leb128 from f + 8 is s such that h = f − s − 6 ≥ 0 and the bytes of file from h are the header magic matching that footer magic, set back to h.

3. If file starts with HEADER_MAGIC or OVERLAY_HEADER_MAGIC:

   1. If back is 0, return file.

   2. Read blocks from the offset 6 by the generic structure (§ 3.1 Block), with data of every block taken as a byte sequence without parsing, until the bytes at the current offset e start with FOOTER_MAGIC or OVERLAY_FOOTER_MAGIC. If it fails to read a block before that, or the footer magic doesn’t match the header magic, return failure.

   3. Let m be the 2 bytes of file from e + 6 as u16le. If file is shorter than e + m + 10 bytes, return failure. Otherwise, return the first e + m + 10 bytes of file.

4. If back is not null, return the bytes of file from back.

5. Return failure.

The located archive is then interpreted, which may still fail. A decompressor SHOULD warn the user when file has bytes outside the located archive. It MAY search for HEADER_MAGIC and OVERLAY_HEADER_MAGIC when the archive can’t be located otherwise, e.g. when file has both leading and trailing bytes.

4. Segments and entries

4.1. Archive model

An archive value is a struct with the following items:

• root, either a directory node or a file node. The latter represents a single unnamed file.

• extra, an ordered map from integers to CBOR data items [RFC8949] representing the archive-wide part of the optional metadata.

A file node is a struct with the following items:

• content, a byte sequence.

• link group, a link group or null.

• extra, a CBOR data item matching aqr-entry-extra, or null.

A directory node is a struct with the following items:

• children, an ordered map from entry names to file nodes, directory nodes, link nodes, special nodes or unknown nodes.

• extra, a CBOR data item matching aqr-entry-extra, or null.

A link node is a struct with the following items, representing a symbolic link:

• target, a link target.

• extra, a CBOR data item matching aqr-entry-extra, or null.

A special node is a struct with the following items, representing a FIFO or a device file:

• kind, one of "fifo", "char-device" or "block-device".

• device, a pair of non-negative integers (major, minor) if kind is "char-device" or "block-device", or null otherwise.

• extra, a CBOR data item matching aqr-entry-extra, or null.

An unknown node is a struct with the following items, representing a kind of node not defined by this version of this specification:

• kind, an integer greater than 4.

• data, a byte sequence.

• extra, a CBOR data item matching aqr-entry-extra, or null.

The root can’t be a link node, a special node or an unknown node.

A tombstone is a struct with no items, marking the removal of a node. It only occurs in children of the result of interpreting a segment, and is removed by merging it into the preceding result (§ 4.4.2 Merging segments), so it never occurs in an archive value.

An entry name is a struct with the following items:

• encoding, either "wtf-8" or "raw".

• bytes, a non-empty byte sequence which is a valid WTF-8 [WTF-8] string if encoding is "wtf-8".

Two entry names are equal if and only if both items are equal. In particular, a "wtf-8" name and a "raw" name are never equal even when their bytes are.

No bytes of an entry name can contain 00₁₆ or 2F₁₆ (/), or be equal to (2E)₁₆ (.) or (2E 2E)₁₆ (..).

An entry name is encoded as a byte sequence in one of the following ways, neither of which contains a zero byte:

• If its encoding is "wtf-8", its bytes as is. WTF-8 [WTF-8] is an extension to UTF-8 and losslessly encodes a lone surrogate in UTF-16.

• If its encoding is "raw", byte C1₁₆ followed by its bytes.

Conforming compressors are RECOMMENDED to use the "wtf-8" encoding, and SHOULD only use the "raw" encoding as a round-trip measure.

To decode an entry name from a entry_name field bytes:

1. If bytes is not empty and bytes[0] is C1₁₆, let name be an entry name whose encoding is "raw" and bytes are bytes without the first byte.

2. Otherwise, if bytes is a valid WTF-8 string, let name be an entry name whose encoding is "wtf-8" and bytes are bytes.

3. Otherwise, return an invalid archive failure.

4. If name violates any restriction of an entry name above, return an invalid archive failure.

5. Return name.

A link target is a struct with the following items:

• encoding, either "wtf-8" or "raw".

• bytes, a non-empty byte sequence not containing 00₁₆, which is a valid WTF-8 [WTF-8] string if encoding is "wtf-8".

Unlike an entry name, a link target is a path which may contain 2F₁₆ (/) as a separator, and components equal to . or ... It may be absolute, and may refer to a path that doesn’t exist in the archive value or anywhere else. The separator is always /, even for a link target from a platform using \ as the separator, such as Windows, where every \ is represented as /. Otherwise a link target is not normalized, and its meaning, e.g. of a drive letter, is left to the platform. There is no limit on its length other than implementation-defined limits.

To decode a link target from a link_target field bytes, do the same as to decode an entry name from bytes, except that the result is a link target and restrictions of a link target apply instead.

A link target is never resolved within the archive model. Every path in an archive value is still a plain sequence of components, and a lookup of a path in an archive value never follows a link node, as lstat doesn’t. In particular, no node is a descendant of a link node.

A link group is an opaque value that is only compared for equality, and is either a pair of a segment and a segment-local integer or an archive-wide integer (§ 11.3 link-group: Hard links). Two archive values are equal if and only if they become identical by some one-to-one renaming of the link groups of one of them.

4.2. Content blocks

Any block other than spacer, size hint, metadata, mirror and ignored blocks is a content block which has to be described by a subsequent metadata block. A metadata block and all content blocks between the previous metadata block (or the beginning of the archive if none) and that metadata block form a segment. As an exception, an archive without a metadata block (§ 3.5 Unencrypted archive) forms a single segment with no metadata block. Each segment MUST be self-contained: every reference from a block in a segment, whether by an index or by an offset, MUST point to a block in the same segment. An internal import (§ 8.1 Import block) refers to a node of preceding segments by its path rather than to a block, and is the only way for a segment to use the content of another segment.

Content blocks are classified as follows:

• A leaf block represents the content of one or more files by itself. Leaf data blocks and formula blocks are leaf blocks.

• A non-leaf block doesn’t represent any file by itself, and only supplies data referenced by one or more formula blocks. Non-leaf data blocks and import blocks are non-leaf blocks. Each non-leaf block supplies one or more non-leaf items to formula blocks: a non-leaf data block supplies a single item, and an import block supplies one item for each of its imports.

• An auxiliary block supplies data used to decode other blocks. Dictionary blocks, module blocks and resource blocks are auxiliary blocks.

Dictionary, module and resource blocks are respectively indexed in the order of appearance within the segment. Two conventions are used for these indices:

• Fields in content blocks use 0-based indices, because the presence of a reference is signalled separately (e.g. by the block type).

• Fields in the metadata block that may indicate the absence of a reference use 1-based indices, where zero denotes the absence.

4.2.1. Block order

Blocks in a segment MUST be ordered so that a formula block is resolved by blocks preceding it and all data required to evaluate it follows it:

• A dictionary block MUST precede any block using it.

• A module or resource block MUST precede any formula block referencing it.

• Every non-leaf item MUST be referenced by at least one formula block, and every non-leaf block MUST follow all formula blocks referencing its items.

• Leaf blocks are otherwise not constrained and MAY be interleaved with non-leaf blocks.

A leaf block and all subsequent content blocks up to the next leaf block or the end of the segment form a parcel. When no leaf block is interleaved with non-leaf blocks, a parcel consists of a leaf block, the non-leaf blocks it references if it is a formula block, and possibly auxiliary blocks. In that case a decompressor can read a single file by reading its parcel and the auxiliary blocks it uses, and a streaming decompressor never has to retain a formula block beyond its parcel.

4.2.2. Registered dictionaries and modules

A registered dictionary is an entry of the Aqueous Dictionaries Registry (§ 15.5 Aqueous Dictionaries Registry), and a registered module is an entry of the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry). Each entry is identified by its registered hash, the SHA-512/256 hash [FIPS180-4] of its bytes: the decoded data of a dictionary block for a registered dictionary (§ 5.2 Dictionary block), and wasm_module for a registered module (§ 7.1 Module block).

Block 50₁₆ (a dictionary block) and block 60₁₆ (a module block) are shorthand blocks. The data of a shorthand block is a registered hash alone, 32 bytes long, instead of checksummed Zstandard data, and the shorthand block stands for a block of the same kind whose decoded data is the bytes of that entry:

• size MUST be 32.

• The registered hash of block 50₁₆ MUST be that of a registered dictionary, which also gives has_entropy_tables, and that of block 60₁₆ MUST be that of a registered module.

• If the decompressor doesn’t know the entry of the registered hash in the corresponding registry, an unsupported failure occurs (§ 4.4 Interpreting an archive).

• A shorthand block is otherwise the same as the dictionary or module block it stands for, e.g. in indexing, block order and mirror blocks.

The two registries share the hash function but not the entries, and a shorthand block is resolved only against the registry of its own kind. Knowing an entry of one registry never makes a decompressor know an entry of the other: in particular, the hash of a registered module in block 50₁₆ doesn’t denote a dictionary of the module’s bytes, unless the same bytes are separately registered as a dictionary, in which case they are two unrelated entries.

A shorthand block has no checksum, and the registered hash is only looked up and never verified. Therefore damage to the hash can’t be distinguished from a hash that was never registered, and both result in an unsupported failure rather than an invalid archive failure.

4.3. Metadata block

Blocks 10₁₆ through 17₁₆ are metadata blocks, whose type is 0001 0mss in binary (§ 3.1.1 Block types). Their data uses the shared external structure (§ 3.2 Shared external structure), with XXH64 as checksum_method and compression_method given by m: None if m is 0, and Zstandard if m is 1.

The internal structure after decompression is as follows:

4.4. Interpreting an archive

To interpret an archive given a byte sequence input, an optional passphrase passphrase and an optional base base, which is an archive value:

1. If input starts with ENCRYPTED_HEADER_MAGIC or ENCRYPTED_MIRRORED_HEADER_MAGIC:

   1. Parse input as the encrypted archive structure (§ 9 Encrypted archives) in its entirety. If it fails to parse, return an invalid archive failure.

   2. If header_checksum is present and doesn’t match, return an invalid archive failure.

   3. If lg_work_factor exceeds an implementation-defined limit, return a resource failure.

   4. If passphrase is not given, or the file key can’t be unwrapped from encrypted_file_key with passphrase as described in [AGE], return a decryption failure.

   5. Using the unwrapped file key, verify age_header_mac against the reconstructed age header and decrypt the age payload as described in [AGE]. If either fails, return an invalid archive failure.

   6. Set input to the decrypted payload. It MUST satisfy the additional constraints in § 9 Encrypted archives, or an invalid archive failure is returned.

2. If input parses as the unencrypted archive structure (§ 3.5 Unencrypted archive) in its entirety, with data of every block taken as a byte sequence without parsing, and either the type of any of blocks is critical and not defined by this specification, or any of blocks is a shorthand block of 32 bytes of data whose registered hash the decompressor doesn’t know (§ 4.2.2 Registered dictionaries and modules), return an unsupported failure.

3. Parse input as the unencrypted archive structure (§ 3.5 Unencrypted archive) in its entirety. If it fails to parse, return an invalid archive failure.

4. If input is an overlay archive and base is not given, return a base failure.

5. If input is a standalone archive, set base to an archive value whose root is a directory node with an empty map as children and null as extra, and whose extra is an empty map.

6. If blocks contain no metadata block:

   1. Let content be the concatenation of the file contents of all leaf blocks in blocks, in the order of appearance, each for every output index from 0 in order (all outputs of a formula block, and the only one of a data block).

   2. Return an archive value whose root is a file node with content as content, null as link group and null as extra, and whose extra is an empty map.

7. If the root of base is a directory node, let root be a copy of it. Otherwise let root be a directory node with an empty map as children and null as extra.

8. Let extra be a copy of the extra of base.

9. For each segment segment in blocks, in the order of appearance:

   1. Let segment root be the result of interpreting segment.

   2. Merge segment root into root.

   3. For each key k and value v in the top-level map of the segment’s optional metadata other than entry-extras, set extra[k] to v.

10. Return an archive value with root as root and extra as extra.

The archive base is base after the step 5, and the preceding result of a segment is root just before the segment is interpreted in the step 9, or the root of the archive base for an archive without a metadata block. They are what external imports and internal imports respectively refer to (§ 8.2 Resolving an import). A given base doesn’t affect the interpretation of a standalone archive.

The input to this algorithm is exactly the archive, from the first byte of header_magic to the last byte of footer_size_copy.

A file may contain an archive with leading or trailing bytes, which a decompressor removes by locating the archive (§ 3.6 Locating an archive).

Since every content block has to be described by a subsequent metadata block (§ 4.2 Content blocks), an archive with a metadata block can’t have any content block after the last metadata block. previous_metadata_offset and metadata_backward_offset MUST point to the metadata blocks found by parsing blocks as described in § 4.3 Metadata block and § 3.5 Unencrypted archive, so that a decompressor can find each segment either by scanning blocks forward or by following offsets backward.

4.4.1. Interpreting a segment

To interpret a segment segment whose metadata block is metadata block:

1. Let meta be the decoded data of metadata block, parsed as its internal structure (§ 4.3 Metadata block).

2. Let d be num_directories, f be num_directories + num_files and n be num_directories + num_files + num_specials of meta.

3. Let nodes be a list containing a single directory node with an empty map as children and the entry extra for the entry 0 as extra.

4. For each k in [1, n]:

   1. Let entry be meta.entries[k - 1].

   2. Let name be the result of decoding entry.entry_name.

   3. Let parent be 0 if k = 1, or meta.entry_parents[k - 2] otherwise.

   4. If k ≤ d, let node be a directory node with an empty map as children.

   5. Otherwise, if k ≤ f:

      1. Let param be meta.file_params[k - d - 1].

      2. Let block be the block at the relative offset param.file_offset - param.file_offset % 2 from metadata block. It MUST be a leaf block in segment.

      3. If param.file_dictionary is zero, block MUST NOT use a dictionary. Otherwise, block MUST use the dictionary block at meta.dictionary_offsets[param.file_dictionary - 1].

      4. Let content be the file content of block for the file output index of param. Its length MUST be entry.original_size.

      5. Let node be a file node with content as content and null as link group.

      6. If the entry extra for the entry k has link-group with a value g, set node.link group to a pair of segment and g if g ≥ 0, or to g otherwise.

   6. Otherwise, if entry.special_kind is 0, let node be a tombstone. The entry extra for the entry k MUST be null.

   7. Otherwise, if entry.special_kind is 1:

      1. Let target be the result of decoding entry.link_target.

      2. Let node be a link node with target as target.

   8. Otherwise, if entry.special_kind is 2, let node be a special node with "fifo" as kind and null as device.

   9. Otherwise, if entry.special_kind is 3 or 4, let node be a special node with "char-device" or "block-device" respectively as kind and (entry.device_major, entry.device_minor) as device.

   10. Otherwise, let node be an unknown node with entry.special_kind as kind and entry.special_data as data.

   11. If node is not a tombstone, set the extra of node to the entry extra for the entry k.

   12. Let children be the children of nodes[parent]. children[name] MUST NOT exist.

   13. Set children[name] to node.

   14. Append node to nodes.

5. Return nodes[0].

Since every parent is less than k and at most d by § 4.3 Metadata block, nodes[parent] is always a directory node already in nodes.

The entry extra for the entry k is the element at the 0-based index k of entry-extras in the segment’s optional metadata, where the index 0 corresponds to the root directory, or null if entry-extras or such element doesn’t exist. entry-extras MUST NOT have more elements than the entries including the root directory, i.e. num_directories + num_files + num_specials + 1.

4.4.2. Merging segments

To merge directory nodes later into earlier:

1. For each name → node of later.children:

   1. Let existing be earlier.children[name], or null if it doesn’t exist.

   2. If node is a tombstone, remove earlier.children[name] if it exists.

   3. Otherwise, if node is a directory node:

      1. If existing is not a directory node, set existing to a directory node with an empty map as children and null as extra, and set earlier.children[name] to existing.

      2. Merge node into existing.

   4. Otherwise, set earlier.children[name] to node.

2. If later.extra is not null, set earlier.extra to it.

As a result, a directory appearing in multiple segments has the union of their children, while any other node, or a directory replaced by a node of the different kind, is replaced as a whole with its descendants. In particular, a directory replaced by a link node loses all its descendants, so no node is a descendant of a link node in the result. A tombstone removes the node at its path along with its descendants, and does nothing if there is no such node. A tombstone is never kept in the result, even under a directory that didn’t exist before. A directory with a null extra in a later segment keeps the existing extra, which can instead be replaced with an empty map {}.

5. Stored data

5.1. Data block

Blocks 20₁₆ through 37₁₆ and 40₁₆ through 4B₁₆ are data blocks. A data block is a compressed block if its compression_method is not None, and an uncompressed block otherwise.

5.2. Dictionary block

Blocks 50₁₆ through 57₁₆ are dictionary blocks, whose type is 0101 0tss in binary (§ 3.1.1 Block types). Their data is checksummed Zstandard data (§ 3.2 Shared external structure), except that block 50₁₆ is a shorthand block for a registered dictionary (§ 4.2.2 Registered dictionaries and modules).

5.3. Block contents

The decoded data of a block using the shared external structure (§ 3.2 Shared external structure), given its compression_method and checksum_method, is determined as follows:

• If compression_method is None, it is data.

• If compression_method is Zstandard, it is the result of decompressing data as described there, with compression_params and the dictionary given by the dictionary block in use (§ 5.2 Dictionary block), if any. The result MUST have exactly original_size bytes.

• If checksum_method is not None, the checksum MUST be equal to the checksum of the decoded data computed by the checksum_method.

The stored content of a data block, whether a leaf block or not, is the decoded data of its compressed_data or uncompressed_data. The stored content of a non-leaf item is the stored content of the non-leaf data block supplying it, or the imported content of the import supplying it (§ 8.2 Resolving an import). A data block using a dictionary uses the dictionary block at dictionary_index within the segment.

The file content of a leaf block for an output index i is:

• Its stored content, if it is a data block. i MUST be 0.

• The output at the index i of evaluating it, if it is a formula block. i MUST be less than its num_outputs.

6. Formulas

6.1. Formula block

Blocks 70₁₆ through 77₁₆ are formula blocks, whose type is 0111 0dss in binary (§ 3.1.1 Block types).

6.2. Formula data

This section describes the internal structure of formula_data in formula blocks (§ 6.1 Formula block), after it has been decompressed.

A formula consists of the lists of blocks it references, followed by instructions of a stack machine. Each instruction pops zero or more byte sequences from the stack and pushes zero or more byte sequences, and the byte sequences remaining on the stack after the last instruction are the outputs of the formula.

6.3. Formula evaluation

To evaluate a formula block block:

1. Let data be the decoded data of formula_data of block, parsed as in § 6.2 Formula data.

2. Let stack be an empty list of byte sequences.

3. For each insn of instructions of data in order, execute insn on stack as described for its opcode in § 6.2 Formula data.

4. The number of items of stack MUST be num_outputs of block.

5. Return stack as the outputs of block, numbered from 0 in order from the bottom of the stack.

6.3.1. Packed byte sequences

The packed byte sequence of a list of byte sequences items is the concatenation of the following, where n is the number of items:

• n as u64le;

• For each i in [0, n), the sum of the lengths of items[0] through items[i] as u64le;

• Each item of items in order.

A byte sequence p is a packed byte sequence of exactly one list items, if any. To unpack p:

1. If p is shorter than 8 bytes, return failure.

2. Let n be the first 8 bytes of p as u64le, and let h be 8 × (n + 1). If h exceeds the length of p, return failure.

3. Let e_i be the 8 bytes of p from the offset 8 × (i + 1) as u64le for each i in [0, n), and let e_-1 be 0.

4. If e_{i - 1} > e_i for any i in [0, n), return failure.

5. If h + e_{n - 1} is not the length of p, return failure.

6. Return the list of the bytes of p in [h + e_{i - 1}, h + e_i) for each i in [0, n).

7. Modules

7.1. Module block

Blocks 60₁₆ through 63₁₆ are module blocks, whose type is 0110 00ss in binary (§ 3.1.1 Block types). Their data is checksummed Zstandard data (§ 3.2 Shared external structure), except that block 60₁₆ is a shorthand block for a registered module (§ 4.2.2 Registered dictionaries and modules).

The internal structure after decompression is as follows:

7.2. Resource block

Blocks 64₁₆ through 67₁₆ are resource blocks, whose type is 0110 01ss in binary (§ 3.1.1 Block types). Their data is checksummed Zstandard data (§ 3.2 Shared external structure).

The internal structure after decompression is an uninterpreted byte sequence, which is supplied to modules by formula blocks referencing it (§ 6.2 Formula data, § 7.3 Module execution).

7.3. Module execution

This section is tentative and follows docs/xform-design.md §5.

A module is a WebAssembly module [WASM-CORE-3] stored as wasm_module, instantiated and invoked through the embedding interface defined in the Appendix of [WASM-CORE-3]. It MUST conform to the following profile, or an invalid archive failure occurs:

• WebAssembly 3.0 core, executed under the deterministic profile; every NaN result is a canonical NaN.

• None of the features added in WebAssembly 3.0 are allowed, except multiple memories and 64-bit addresses of the memories other than the working memory. That is, tail calls, extended constant expressions, typed function references, garbage collection, exception handling, relaxed SIMD, 64-bit tables and a 64-bit working memory are not allowed.

• Imported tables are not allowed.

• It has no start function.

• It defines no memory, and imports each memory at most once.

• Every memory.grow instruction targets the working memory.

• The import type of every memory other than the working memory has 64-bit addresses, the minimum 0 and no maximum.

• It exports exactly one of the functions aqr_apply_whole and aqr_apply_part, the entry point. The module is a whole module in the former case and a part module in the latter.

• If it imports aqr_spawn, it exports the function aqr_task, the task entry point, which has two parameters of type i32 and no results.

Whether the guaranteed limit of 2²⁷ bytes (128 MiB) for the Zstandard Window_Size (§ 3.2.1 Zstandard compression method) should also apply to the memory limit.

Every import of a module is from the module name aqr, under one of the following names. The names of imports and exports are prefixed with aqr_ to avoid collisions in the toolchains and hosts that link them into one namespace. Any other import is an invalid archive failure.

aqr_memory

A memory, the working memory.

aqr_params

A memory of the parameters.

aqr_resourcei

A memory of the i-th resource, where i is a decimal number without leading zeros.

aqr_inputk

A memory of the k-th input, where k is a decimal number without leading zeros.

aqr_outputj

A memory of the j-th output, where j is a decimal number without leading zeros.

aqr_request, aqr_release, aqr_commit

Functions of a part module, defined below. A whole module MUST NOT import them.

aqr_heartbeat

A function defined below.

aqr_spawn, aqr_result, aqr_join, aqr_take

Functions of tasks, defined in § 7.3.1 Tasks.

aqr_chan, aqr_send, aqr_recv, aqr_close

Functions of channels, defined in § 7.3.1 Tasks.

aqr_log

The diagnostic function, defined below.

A memory is sized to fit a length L when both its minimum and maximum are ⌈L / 65536⌉ pages.

To apply a module module given a list of input byte sequences inputs, a list of resource blocks resources, a parameter byte sequence params, a list of declared output sizes sizes and a memory limit:

1. sizes MUST NOT be empty.

2. Every aqr_resourcei imported MUST satisfy i < the number of resources, every aqr_inputk imported MUST satisfy k < the number of inputs, and every aqr_outputj imported MUST satisfy j < the number of sizes.

3. Create the memories imported, each initially zero-filled:

   • The working memory has the minimum size of its import type.

   • The memory of the parameters is sized to fit the length of params, and holds params from offset 0.

   • The memory of the i-th resource is sized to fit its stored content c, and holds c from offset 0.

   • The memory of the k-th input is sized to fit the length of inputs[k]. If module is a whole module, it holds inputs[k] from offset 0. Otherwise let W_k be its size in bytes and the visible range of the k-th input be [0, 0).

   • The memory of the j-th output is sized to fit sizes[j]. If module is a part module, let W_out,j be its size in bytes.

4. Let outputs be a list of as many empty byte sequences as sizes.

5. If the total size of the memories in bytes exceeds the memory limit, return a resource failure. The memory limit applies to the total size of the memories of the application, those of its tasks included, and of the messages queued in its channels.

6. Instantiate module with the memories and the host functions below.

7. Invoke the entry point with the arguments of type i64, in order: the length of params, the length of the stored content of each item of resources, the length of each item of inputs and each item of sizes. The entry point MUST have these parameters and no results. If it traps or any host function below signals an error, return an invalid archive failure. If it exceeds the memory limit or any implementation-defined limit, return a resource failure. If any task of the application fails, return its failure (that of any of them if several fail). If every instance of the application waits, return an invalid archive failure. Once it returns, every task it spawned MUST be taken.

8. If module is a whole module, set each outputs[j] to the first sizes[j] bytes of the memory of the j-th output, if imported, and to sizes[j] zero bytes otherwise.

9. The length of each outputs[j] MUST be sizes[j].

10. Return outputs.

Windows smaller than their data for a part module, i.e. ring buffers, which would allow import types with a maximum.

A memory.grow instruction exceeds the memory limit when the total size of the memories of the application, those of its tasks and the messages queued in its channels included, would exceed it, and MUST succeed otherwise if the maximum of the import type of the working memory allows.

Host functions are defined as follows, where lo_k and hi_k denote the visible range of the k-th input, k, j, ptr and len are of type i32, and every offset is of type i64. Called by a task, aqr_request, aqr_release and aqr_commit signal an error.

aqr_request(k, hi)

The k-th input MUST be imported. hi MUST be in [hi_k, length of inputs[k]], and hi - lo_k MUST be at most W_k. For each offset o in [hi_k, hi), write inputs[k][o] to the memory of the k-th input at o % W_k. Then set hi_k to hi.

aqr_release(k, lo)

The k-th input MUST be imported. lo MUST be in [lo_k, hi_k]. Set lo_k to lo.

aqr_commit(j, hi)

The j-th output MUST be imported. hi MUST be in [length of outputs[j], sizes[j]], and hi - length of outputs[j] MUST be at most W_out,j. For each offset o in [length of outputs[j], hi), append the byte at o % W_out,j of the memory of the j-th output to outputs[j].

aqr_heartbeat()

Does nothing.

aqr_log(ptr, len)

Does nothing. Implementations MAY present the len bytes of the working memory from the offset ptr as a diagnostic message, which is untrusted data from the archive, but doing so MUST NOT affect the application, even when the range is out of bounds. It is the only diagnostic function; a module reports a failure by aqr_log followed by a trap.

A violation of any MUST in host functions is signalled as an error to the module invocation.

7.3.1. Tasks

A task is an instance of the module of its own, run on a copy of the working memory of the instance that spawns it and giving back a byte sequence, its result. The application’s instance and every task are the instances of the application; each instance has a list of handles, initially empty, each of which is a list of the tasks spawned at once. A task is either running, done (with a result), failed (with an invalid archive failure or a resource failure), or taken.

A channel links instances of an application: a queue of messages (byte sequences) of at most its capacity, whether it is closed, and its sender and receiver, instances initially none. The application has a list of channels, initially empty. When an instance ends (returns or fails), every channel of which it is the sender is closed. An instance waits while it is within aqr_join, aqr_send or aqr_recv and the condition it waits for does not hold.

To run a task given a byte sequence S, arg and i:

1. Create the memories imported: the working memory has S.length / 65536 pages and the maximum of its import type, and every other memory has the minimum and maximum 0.

2. Instantiate the module with the memories and the host functions, and then set the content of the working memory to S.

3. Invoke the task entry point with arg and i. If it traps or any host function signals an error, the task fails with an invalid archive failure. If it exceeds the memory limit or any implementation-defined limit, the task fails with a resource failure.

4. Once it returns, every task it spawned MUST be taken, or the task fails with an invalid archive failure.

5. The task is done, with the result last set by aqr_result, or an empty byte sequence if none.

The host functions of tasks are defined as follows, for the instance I calling them, where arg, n, h, i, ptr, dst and len are of type i32, unsigned:

aqr_spawn(arg, n)

Let S be the content of the working memory of I. Append to the handles of I a list of n new tasks, the i-th (from 0) of which is to run a task given S, arg and i, and return the index of that list among the handles of I.

aqr_result(ptr, len)

I MUST be a task, and ptr + len MUST be at most the size of its working memory. Set the result of I to the len bytes of its working memory from the offset ptr.

aqr_join(h, i)

The i-th task of the h-th handle of I MUST exist and not be taken. Wait until it is done or failed. If it failed, the instance I fails with its failure. Otherwise return the length of its result.

aqr_take(h, i, dst, len)

The i-th task of the h-th handle of I MUST be done (joined), len MUST be at most the length of its result, and dst + len MUST be at most the size of the working memory of I. Copy the first len bytes of its result to the working memory of I from the offset dst, and the task is taken.

The host functions of channels are defined as follows, where c, ptr, dst, len and cap are of type i32, unsigned, the channel C is the c-th (from 0) of the application, which MUST exist, and taking an end of C for I means: if that end is none, set it to I; otherwise it MUST be I.

aqr_chan(cap)

cap MUST be at least 1. Append to the channels of the application a new channel of capacity cap, and return its index.

aqr_send(c, ptr, len)

Take the sender of C for I. C MUST NOT be closed, and ptr + len MUST be at most the size of the working memory of I. Wait until fewer messages than the capacity of C are queued, and then enqueue the len bytes of the working memory of I from the offset ptr.

aqr_recv(c, dst, cap)

Take the receiver of C for I. Wait until a message is queued or C is closed. If no message is queued, return −1 (0xFFFFFFFF). Otherwise let M be the first message. If the length of M exceeds cap, return it, leaving M queued. Otherwise dst + the length of M MUST be at most the size of the working memory of I; dequeue M, copy it to the working memory of I from the offset dst, and return its length.

aqr_close(c)

Take the sender of C for I, and close C.

Whether the memories other than the working memory, read-only, should be visible to tasks, which would require rejecting the instructions writing them; and the windows of a part module in tasks.

8. Overlay archives and imports

8.1. Import block

Blocks 58₁₆ through 5F₁₆ are import blocks, which are non-leaf blocks supplying byte ranges of files outside the segment. Their type is 0101 1ess in binary (§ 3.1.1 Block types), and their data is checksummed Zstandard data (§ 3.2 Shared external structure).

8.2. Resolving an import

To resolve an import import of an import block in a segment:

1. Split import.import_path at every 2F₁₆ into components, unless it is empty, and let names be the result of decoding each component as an entry name. If any component fails to decode, return an invalid archive failure.

2. Let node be the preceding result of the segment if import is an internal import, or the root of the archive base otherwise.

3. For each name of names in order:

   1. node MUST be a directory node, and node.children[name] MUST exist.

   2. Set node to node.children[name].

4. node MUST be a file node.

5. import.import_offset + import.import_size MUST be at most the length of node.content.

6. Let content be the bytes of node.content in [import.import_offset, import.import_offset + import.import_size).

7. The SHA-512/256 hash [FIPS180-4] of content MUST be equal to import.import_checksum.

8. Return content as the imported content of import.

A violation of any MUST in this algorithm is a base failure in an overlay archive, and an invalid archive failure in a standalone archive.

8.3. Reading a base

Reading a base from a file system is outside the archive model, as is extraction. A decompressor doing so maps a directory or a single file to an archive value, and is subject to the following requirements:

• It MUST NOT use a base unless explicitly given one, and in particular MUST NOT use the current directory or the destination directory as a base by default.

• It MUST NOT read anything outside the given directory, including through symbolic links or other redirections already present in the base. A symbolic link in the base is a link node and is never followed, so an import whose path leads through or to a link fails to resolve (§ 8.2 Resolving an import).

• It MUST NOT read a file other than a regular file as the content of a file node. A FIFO or a device file in the base is a special node.

• It SHOULD map a file system name to an entry name as the reverse of the mapping in § 12 Extraction. If a name can’t be mapped, or multiple names map to the same entry name, it MUST either fail or report it to the user.

• The base MUST appear the same throughout the interpretation, as if it were read at a single point in time. A decompressor MAY read only the parts of the base needed by imports and by the result, and the checksums of imports detect a part that changed while reading.

A decompressor MAY write the result of an overlay archive into its base directory, i.e. apply it in place. In that case:

• The result MUST be the same as if every imported content were read before anything is written, e.g. by writing new files under temporary names and renaming them after all of them are written.

• It MAY remove a node of the base only when the node is absent from the result, and SHOULD leave every node not given by any segment as it is.

9. Encrypted archives

An archive might have been encrypted, in which case it consists of a block-like header of type ENCRYPTED_HEADER_TYPE followed by encrypted chunks. An encrypted archive is mirrored if it starts with ENCRYPTED_MIRRORED_HEADER_MAGIC, in which case copies of its header are placed between encrypted chunks.

age-encryption.org/v1
-> scrypt scrypt_ephemeral_secret_base64 lg_work_factor_decimal
encrypted_file_key_base64
--- age_header_mac_base64

10. Integrity and recovery

10.1. Checksums

The following table summarizes every checksum defined by this specification:

The encrypted chunks of an encrypted archive are also authenticated by [AGE].

Unless explicitly requested otherwise, conforming compressors SHOULD use a checksum_method other than None for every leaf block.

Unless explicitly requested otherwise, conforming compressors SHOULD declare an output checksum for every formula block.

10.2. Archival profile

An archive conforms to the archival profile when the content of every file is covered by a checksum:

• Every leaf data block has a checksum_method other than None.

• Every formula block declares an output checksum.

• The archive is a standalone archive.

• No block is a shorthand block (§ 4.2.2 Registered dictionaries and modules).

A conforming compressor producing an archive for long-term preservation SHOULD conform to the archival profile. A tool verifying an archive SHOULD report whether it conforms to the archival profile.

10.3. Mirror block

A mirror block holds a verbatim copy of another block, called its target block, so that the target block can be restored when it is damaged. Mirror blocks are only defined for block types whose damage affects more than a single parcel and can be detected by reading the block alone, i.e. that have a checksum:

As with any other block, the size field of a mirror block is determined by its own type and counts the bytes of its own data. Since a mirror block is longer than its target block, its size field may have to be wider. Therefore a mirror block is either:

• Unwidened, where the size fields of the mirror block and its target block have the same width, and its target type is its type minus C00₁₆; or

• Widened, where the size field of the mirror block is one step wider than that of its target block, and its target type is its type minus C01₁₆. Its type % 4 MUST NOT be 0.

The target block MUST be of the target type, which MUST be a block type listed above for the mirror block’s type. Whether a mirror block is widened is signalled by target_offset.

Its data consists of the following:

A mirror block SHOULD be unwidened unless its data doesn’t fit in the size field for the target type.

A mirror block doesn’t belong to any segment and MAY occur anywhere in blocks, including after the last metadata block, subject only to the requirement that its target block precedes it. A single target block MAY have multiple mirror blocks. The requirements on a mirror block in this section are requirements on conforming compressors, and a mirror block violating them doesn’t make the archive an invalid archive. Mirror blocks don’t affect the result of interpreting an archive at all.

A mirror block is intact if its mirror_checksum verifies and it satisfies the requirements above other than being equal to its target block. A decompressor MUST NOT restore a target block from a mirror block that is not intact, and SHOULD report a mirror block that is not intact or differs from its intact target block when it reads one.

A mirror block and its target block are expected to be damaged together when they are close to each other. Conforming compressors SHOULD separate the end of the target block and the block_start of its mirror block by at least 4096 bytes, so that they never share an aligned 4 KiB unit of the underlying storage, and SHOULD separate them by at least 4 MiB (4,194,304 bytes) when possible.

The recommended distances are tentative, pending the measurement of burst lengths of latent sector errors (docs/recovery-design.md §8, R1). The interval between mirrored headers in an encrypted archive (§ 9 Encrypted archives) changes with the latter distance.

A damaged encrypted chunk fails to authenticate as a whole, and an aligned 4 KiB unit of the underlying storage may span two adjacent encrypted chunks. Therefore, in place of the distance of 4096 bytes recommended above, conforming compressors SHOULD place a mirror block and its target block in an encrypted archive so that at least one entire encrypted chunk lies between them.

10.4. Repair

The repair process is to be specified in this section (docs/recovery-design.md §5.6).

Repairing an encrypted archive with mirror blocks or mirrored headers requires decrypting the encrypted chunks after one that fails to authenticate, which a standard age implementation doesn’t do. This is to be specified with the repair process as a separate step producing the decrypted bytes, where bytes of such chunks are marked as erasures and the interpretation of an archive stays unchanged (docs/recovery-design.md §5.6).

11. Optional metadata

The optional_metadata field of the metadata block (§ 4.3 Metadata block) holds the optional metadata, encoded in CBOR [RFC8949]. This part encodes any metadata that is not strictly necessary to read the file, including associated dates and times, access control information and comments.

The optional metadata is described by the following CDDL [RFC8610] definitions:

aqr-opt-metadata = {
  aqr-archive-comment
  ? &(entry-extras: -2) => [* aqr-entry-extra / null]  ; Indexed by entry numbers, starting from the root directory

    * $$aqr-opt-metadata-member  ; For future extensions
}

$$aqr-opt-metadata-member = ()  ; No extension is defined by this specification

aqr-entry-extra = {
  aqr-entry-timestamps
  aqr-entry-comment
  aqr-entry-link-group
  aqr-entry-windows
  aqr-entry-unix
  aqr-entry-zip

    * $$aqr-entry-extra-member  ; For future extensions
}

$$aqr-entry-extra-member = ()  ; No extension is defined by this specification

; TODO: other extras for round trip

It MUST be a single well-formed CBOR data item [RFC8949] without any following bytes, and no map in it can have duplicate keys. The value of every key defined by this specification MUST match its definition. If optional_metadata_size is 0, the optional metadata is an empty map {}.

The members of aqr-entry-extra are grouped by their purposes, and each group is described in the following sections. Keys of aqr-entry-extra from -1 to -59 are platform-neutral. Keys from -60 to -69 are specific to Windows, -70 to -79 to Unix, -80 to -89 to macOS, and -90 to -99 to other platforms. Keys from -100 are specific to the formats an entry was converted from.

11.1. Timestamps

The keys crtime, ctime, mtime, atime and arctime of aqr-entry-extra record the timestamps that file systems and archive formats associate with a file, a directory or a link. Each is an aqr-time described below, and is absent if the source doesn’t have the corresponding timestamp.

aqr-entry-timestamps = (
  ? &(crtime: -1) => aqr-time
  ? &(ctime: -2) => aqr-time
  ? &(mtime: -3) => aqr-time
  ? &(atime: -4) => aqr-time
  ? &(arctime: -5) => aqr-time
)

crtime

The creation time, also known as the birth time: when the file was created at its location. It is not updated afterwards, but copying a file usually creates a new one with the time of copying. Examples include the creation time of NTFS and FAT, the create date of HFS+ and APFS, i_crtime of ext4 and otime of Btrfs, which are exposed as st_birthtime on BSD and macOS and as stx_btime of statx on Linux.

ctime

The status change time: when the file’s metadata (e.g. permissions, ownership or link count) or its content was last changed. It is the POSIX st_ctime, and the change time of NTFS and the attribute modification date of HFS+ are similar. POSIX doesn’t allow it to be set to an arbitrary time, so an extracting decompressor usually can’t restore it.

mtime

The modification time: when the file’s content was last changed. For a directory, it is when an entry was last added, removed or renamed in it. This is the timestamp that virtually all file systems and archive formats have.

atime

The access time: when the file’s content was last read. Many systems update it lazily or not at all for performance (e.g. relatime and noatime on Linux, and the access time updates of NTFS that can be disabled), and FAT only keeps the date. Reading a file for archiving can itself update it.

arctime

The backup time: when the file was last backed up, as recorded by backup software. File systems don’t update it by themselves. An example is the backup date of HFS and HFS+, exposed as ATTR_CMN_BKUPTIME of getattrlist on macOS. It is the time stored by the source, not the time when the archive itself was made.

aqr-time represents a point in time or a local date and time, together with a quantization interval and optional time zone information. It is described by the following CDDL definitions, where etime is the CDDL type name for the extended time defined in [RFC9581]:

aqr-time =
  $aqr-base-time /                          ; UTC timestamp, scaled and quantized
  [$aqr-base-time] /                        ; Local time in unknown time zone, encoded like UTC (*)
  [$aqr-instant, offset: aqr-tz-offset] /   ; UTC timestamp with time zone offset ($)
  etime /                                   ; Extended time defined in [[!RFC9581]]
  [etime]                                   ; Local time in unknown time zone, encoded like UTC

aqr-tz-offset = -48..56  ; In 15 minutes, positive east of UTC as in ISO 9660

$aqr-base-time  = #6.100(int)    ; 1d (FAT atime*), date as in [[!RFC8943]]
$aqr-base-time /= $aqr-instant  ; Can be given a time zone offset

$aqr-instant  = #6.121(int)  ; 2s (FAT mtime*, exFAT atime$, ZIP*, CAB*)
$aqr-instant /= int          ; 1s (~ext3, HFS/HFS+, tar, ISO 9660$)
$aqr-instant /= #6.122(int)  ; 10ms (FAT ctime*, ISO 9660$, exFAT$)
$aqr-instant /= #6.123(int)  ; 1ms
$aqr-instant /= #6.124(int)  ; 1us (UDF)
$aqr-instant /= #6.125(int)  ; 100ns (NTFS, RAR*, 7z)
$aqr-instant /= #6.126(int)  ; 1ns (ext4, ZFS, Btrfs, APFS)

A base time value $aqr-base-time is an integer, either untagged or tagged as follows. The integer multiplied by the scale is the number of seconds since 1970-01-01T00:00:00Z, not counting leap seconds as in the CBOR tag 1 [RFC8949]; negative values denote the time before that epoch. The tag also determines the quantization interval, which is relative to the time value and specified in § 15.4 Quantization Interval. #6.100 is the number of days since 1970-01-01 [RFC8943], and is the only base time value that is not an $aqr-instant, i.e. it can’t be given a time zone offset.

In the Examples column, formats marked with * store (or can store) a local time in an unknown time zone, and formats marked with $ store a local time with its offset from UTC.

Each alternative of aqr-time has the following meaning:

$aqr-base-time

A point in time without any time zone information.

[$aqr-base-time]

A local date and time in an unknown time zone, encoded as if it were UTC. It does not identify a single point in time.

[$aqr-instant, aqr-tz-offset]

A point in time, as in the bare $aqr-base-time, with the offset of the time zone where it was recorded, in 15-minute increments as in ISO 9660. The offset is positive east of UTC: UTC+x hours is 4x and UTC−x hours is −4x. The local time is the base time value plus 900 times the offset in seconds.

etime

An extended time [RFC9581] (tag 1001), for any value which doesn’t fit the other alternatives. The time zone information, if any, is given as a Time Zone Hint (key -10 or 10), and the quantization interval is given as a key described in § 15.4 Quantization Interval.

[etime]

A local date and time in an unknown time zone, encoded as if it were UTC. The extended time MUST NOT contain a Time Zone Hint (key -10 or 10), and its timescale (key -1, -13 or 13), if present, MUST be 0 (UTC).

etime or [etime] MUST NOT be used if another alternative of aqr-time represents the same time value, the same quantization interval and the same time zone information. A Time Zone Hint is the same as an aqr-tz-offset only when it is a numeric offset (e.g. "+09:00") equal to that offset; an extended time without any quantization interval is never the same as other alternatives.

Two aqr-time values in different encodings are different even when they denote the same point in time. For example, 1 and 122(100) differ in their quantization intervals. A compressor converting a timestamp from a file system or another archive format SHOULD preserve its precision, i.e. use an encoding whose quantization interval matches the source.

Timestamps from ZIP are discussed in § 16.2 Timestamps from ZIP.

11.2. Comments

archive-comment of aqr-opt-metadata and comment of aqr-entry-extra are free-form comments attached to the whole archive and to each entry respectively, e.g. an archive comment and a file comment of ZIP or RAR. Both are an aqr-text, which is a Unicode string, a WTF-8 [WTF-8] string with lone surrogates, or a byte sequence in an unknown encoding:

aqr-archive-comment = (
  ? &(archive-comment: -1) => aqr-text
)

aqr-entry-comment = (
  ? &(comment: -6) => aqr-text
)

aqr-text =
  tstr /          ; Unicode
  #6.273(bstr) /  ; WTF-8, i.e. UTF-16 + surrogate compat
  bstr            ; In unknown encoding

An aqr-text MUST be a tstr if it can be represented as one, i.e. if it is a sequence of Unicode scalar values. The byte string in #6.273(bstr) MUST be a valid WTF-8 string that is not a valid UTF-8 string, i.e. it MUST contain at least one encoded lone surrogate. A bstr is not subject to these requirements even if it happens to be a valid UTF-8 string, as its encoding is not known.

Comments from ZIP are discussed in § 16.3 Comments from ZIP.

11.3. link-group: Hard links

aqr-entry-link-group = (
  ? &(link-group: -7) => int  ; File nodes only
)

link-group records that file nodes were originally the same file, i.e. hard links to the same inode on Unix or to the same file record on NTFS. It has no effect on other nodes, and a compressor SHOULD NOT give it to them. A non-negative value is segment-local and denotes the same group only within the same segment, while a negative value is archive-wide and denotes the same group in every segment. The resulting link group in the archive value is determined by interpreting a segment.

A compressor MUST NOT give the same link group to files that were not the same file. It SHOULD NOT give a link group that no other file node has. A file node with link-group SHOULD refer to the same leaf block as other file nodes in the same group, and SHOULD have the same entry extra as them except for link-group and comment.

A link group is consistent if all file nodes with that group in the final archive value have the same content. An inconsistent group, e.g. after a later segment has replaced only some of its file nodes, is not an invalid archive failure, but has no effect on extraction (§ 12 Extraction).

11.4. Windows-specific metadata

aqr-entry-windows = (
  ? &(win-link-type: -60) => &(file: 0, directory: 1, junction: 2)  ; Link nodes only

  ; TODO: file attributes (e.g. ZIP external file attributes)
)

win-link-type determines the kind of a link to be created on Windows (§ 12 Extraction). It has no effect on nodes other than link nodes, and a compressor SHOULD NOT give it to them. "file" and "directory" denote a symbolic link to a file and to a directory respectively, and "junction" denotes a junction (a mount point reparse point).

11.5. Unix-specific metadata

aqr-entry-unix = (
  ? &(unix-mode: -70) => uint .le 4095  ; Permission bits only (0o7777), without S_IFMT
  ? &(unix-uid: -71) => uint
  ? &(unix-gid: -72) => uint
)

unix-mode only holds the permission bits, including the set-user-ID, set-group-ID and sticky bits. The kind of an entry is solely determined by its node (§ 4.1 Archive model), so unix-mode never includes the file type bits (S_IFMT).

unix-uid and unix-gid are the numeric user and group IDs of the owner.

Unix metadata from ZIP is discussed in § 16.4 Unix metadata from ZIP.

11.6. zip-extras: ZIP extra fields

zip-extras preserves extra fields [APPNOTE] of the ZIP entry that the entry was converted from, when they carry information about the file that has no other representation in this specification.

aqr-entry-zip = (
  ? &(zip-extras: -100) => aqr-zip-extras
)

aqr-zip-extras = { * aqr-zip-extra-key => bstr .size (0..65531) }
aqr-zip-extra-key =
  0..65535 /   ; Header ID n in the local header
  -65536..-1   ; Header ID n in the central header, as -1 - n
; TODO: multiple extra fields with the same header ID

A key n from 0 to 65535 denotes the extra field with the header ID n in the local header, and a key −1 − n from −65536 to −1 denotes the extra field with the header ID n in the central header. The value is the data of the extra field, excluding its 4-byte header of the header ID and the data size. Therefore a value with a non-negative key is always in the format for the local header, and a value with a negative key is always in the format for the central header.

A compressor converting a ZIP archive MAY put any extra field of an entry into zip-extras, subject to the following requirements:

• It MUST NOT put the central header version of an extra field if the local header version with the same header ID is identical byte by byte. The central header version without the local header version is only put with a negative key.

• It MUST NOT put an extra field whose information is fully represented elsewhere, i.e. in the entry name, the archive model (§ 4.1 Archive model) or any other key of aqr-entry-extra.

• It MUST NOT put an extra field which is only meaningful to the ZIP container itself, e.g. sizes, offsets, padding, alignment, encryption and signatures over the ZIP archive.

• It SHOULD NOT give zip-extras to the root directory, which has no counterpart in ZIP.

Extra fields known to be subject to these requirements are listed in § 16.5 Extra fields not to be preserved. Any other extra field, including one with an unregistered header ID, can be put into zip-extras as is.

zip-extras is never authoritative. If information in a preserved extra field disagrees with the archive model or any other key of aqr-entry-extra, the latter MUST be used. A decompressor finding an extra field that a compressor is not allowed to put SHOULD ignore that field only, and SHOULD NOT treat the whole archive as invalid.

12. Extraction

Writing an archive value to a file system is outside the archive model. A decompressor doing so is subject to the following requirements:

• It MUST NOT create, modify or remove any file outside the destination directory, including through symbolic links or other redirections already present in the destination or created during the extraction.

• It SHOULD map an entry name whose encoding is "wtf-8" to the platform’s native Unicode name (e.g. UTF-16 with lone surrogates on Windows), and an entry name whose encoding is "raw" to the identical byte sequence where the platform allows.

• If an entry name can’t be represented on the platform or has a special meaning there (e.g. a name containing \ or :, or a reserved device name on Windows), it MUST either fail or report a substituted name to the user; it MUST NOT silently use a different name.

• If multiple entry names in a directory map to the same file system name (e.g. due to case insensitivity or Unicode normalization), it MUST either fail or report the conflict, and MUST NOT write a later entry through an earlier one.

• It MAY ignore the extra of any node, e.g. extra and extra. It SHOULD NOT apply ownership information such as unix-uid and unix-gid unless explicitly requested.

A decompressor extracting a link node is subject to the following additional requirements:

• It MUST NOT create a link whose target escapes the destination directory unless explicitly requested, and SHOULD provide a way to request it. A link target escapes the destination directory if it is absolute or relative to a drive or a root on the platform, or if resolving it from the parent directory of the link, following any link nodes in the archive value on the way, reaches a parent of the destination directory. A link not created for this reason MUST be reported to the user.

  Creating such a link doesn’t write outside the destination directory by itself, but the extracted tree can lead later users or programs to do so. Resolving a target lexically is not enough: when d/e/l is a link to ../../q, a link to d/e/l/../.. lexically resolves to d but actually reaches the parent of the destination directory.

• On Windows, it SHOULD replace every / in the link target with \. It SHOULD create the kind of link given by win-link-type, if any. Otherwise, it SHOULD create a symbolic link to a directory if the target resolves to a directory node in the archive value as above, and a symbolic link to a file otherwise.

• If a link can’t be created, e.g. due to the lack of privilege, the file system or the length of its target, it MUST either fail or report it to the user; it MUST NOT silently create a different kind of node. It MAY create a copy of the target instead if it reports so.

• It MUST NOT apply the extra of a link node to its target. Timestamps, ownership and permissions SHOULD be applied to the link itself where the platform allows (e.g. lutimes, lchown).

A decompressor MAY create hard links among file nodes with the same consistent link group. It MUST NOT create hard links among any other file nodes, even when they have the same content.

A decompressor SHOULD NOT create a FIFO or a device file for a special node unless explicitly requested, as with ownership information. A special node not created for this reason, or because the platform can’t represent it (e.g. the lack of privilege, or a device number out of the platform’s range), MUST be reported to the user.

A decompressor MUST NOT create anything for an unknown node, and MUST report it to the user. It MAY continue extracting other nodes.

13. Conformance

As well as sections marked as non-normative, all diagrams, examples and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MAY, MUST, MUST NOT, RECOMMENDED, SHOULD and SHOULD NOT in this specification are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

13.1. Conformance levels

A conforming decompressor has a conformance level, one of the following according to the modules (§ 7 Modules) it applies:

Level 0

Applies no module.

Level 1

Applies every registered module, i.e. every module whose module hash is the registered hash of an entry of the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry), and no other module.

Level 2

Applies every module.

A decompressor at level 0 or level 1 MAY refuse to apply a module that its level doesn’t include, in which case a resource failure occurs. A decompressor at level 2 MUST NOT refuse to apply a module, though it may still fail with a resource failure when a module exceeds its limits. A level doesn’t change any other requirement on a decompressor, e.g. a decompressor at level 0 still reads formula blocks without Apply instructions.

13.2. Partial access

A decompressor MAY compute only a part of the result of interpreting an archive, such as a subset of entries, a byte range of a content, or entry names without contents. Such a partial result MUST be equal to the corresponding part of the full result, if the full result is an archive value.

A decompressor computing a partial result is not required to detect an invalid archive failure caused only by parts it doesn’t compute. It SHOULD verify every checksum covering only the bytes it has computed.

14. Security considerations

An archive is a container for arbitrary data and doesn’t by itself contain active content, except for WebAssembly modules (§ 7.3 Module execution) that a decompressor executes to reconstruct file contents. Such modules are run under a deterministic profile without access to anything but their inputs and outputs, but a decompressor still has to treat them as untrusted code and isolate them accordingly, unless it knows their module hash (§ 7.1 Module block). Formula blocks and modules can produce outputs far larger than the archive itself, and some fields (e.g. lg_work_factor) directly control the amount of computation. A decompressor is expected to enforce implementation-defined limits and fail with a resource failure when they are exceeded. A decompressor writing an archive to a file system is subject to the requirements in § 12 Extraction, in particular to prevent writing outside the destination directory. An unencrypted archive provides no confidentiality, and its XXH64 checksums (§ 10.1 Checksums) only detect accidental corruption, not intentional modification. An encrypted archive (§ 9 Encrypted archives) provides confidentiality and integrity through [AGE], which is only as strong as the passphrase used. The metadata of an archive, including entry names, timestamps and ownership, may reveal information about the system that created it.

An overlay archive is interpreted against a base, typically a directory on the file system, and can copy any part of the base into its result. A decompressor reading a base is subject to the requirements in § 8.3 Reading a base, in particular to use a base only when explicitly given one and to never read outside it. Applying an overlay archive in place modifies and removes files in the base directory.

15. IANA Considerations

15.1. application/aqr Media Type

This document requests the registration of a new media type in the Media Types Registry ([RFC6838], section 5.6) as follows:

Type name

application

Subtype name

aqr

Required parameters

N/A

Optional parameters

N/A

Encoding considerations

binary

Security considerations

See § 14 Security considerations.

Interoperability considerations

An unencrypted archive may be preceded and followed by arbitrary bytes, e.g. a self-extracting executable stub; its extent is determined by header_magic and the footer (§ 3.6 Locating an archive). An archive without any metadata block decompresses into a single unnamed file rather than a directory tree. An overlay archive has its own media type application/aqo (§ 15.2 application/aqo Media Type); an encrypted archive whose payload is an overlay archive is also of that type.

Published specification

This document.

Applications that use this media type

Archivers and compressors that implement the Aqueous archive format.

Fragment identifier considerations

N/A

Additional information

Deprecated alias names for this type

N/A

Magic number(s)

F8 04 03 6A D6 BE (hexadecimal; HEADER_MAGIC) for an unencrypted archive, possibly preceded by other data; F9 CA F2 A6 D0 2A (hexadecimal; the common prefix of ENCRYPTED_HEADER_MAGIC and ENCRYPTED_MIRRORED_HEADER_MAGIC) at offset 0 for an encrypted archive.

File extension(s)

.aqr
When a single file is compressed into an archive without any metadata block, the extension may be appended to the original file name instead (e.g. example.tar.aqr).

Macintosh file type code(s)

N/A

Person & email address to contact for further information

Kang Seonghoon <public+aqr@mearie.org>

Intended usage

COMMON

Restrictions on usage

N/A

Author

Kang Seonghoon

Change controller

Kang Seonghoon

15.2. application/aqo Media Type

This document requests the registration of a new media type in the Media Types Registry ([RFC6838], section 5.6) as follows:

Type name

application

Subtype name

aqo

Required parameters

N/A

Optional parameters

N/A

Encoding considerations

binary

Security considerations

See § 14 Security considerations.

Interoperability considerations

An overlay archive can’t be interpreted without its base. Its magic numbers differ from those of application/aqr, so that a decompressor not supporting overlay archives doesn’t mistake one for a standalone archive. The interoperability considerations for application/aqr otherwise apply.

Published specification

This document.

Applications that use this media type

Archivers, patching tools and updaters that implement the Aqueous overlay archive format.

Fragment identifier considerations

N/A

Additional information

Deprecated alias names for this type

N/A

Magic number(s)

69 0D 15 AF F6 B2 (hexadecimal; OVERLAY_HEADER_MAGIC) for an unencrypted overlay archive, possibly preceded by other data. An encrypted overlay archive has the same magic numbers as an encrypted archive of application/aqr.

File extension(s)

.aqo

Macintosh file type code(s)

N/A

Person & email address to contact for further information

Kang Seonghoon <public+aqr@mearie.org>

Intended usage

COMMON

Restrictions on usage

N/A

Author

Kang Seonghoon

Change controller

Kang Seonghoon

15.3. +aqr Structured Syntax Suffix

This document requests the registration of a new structured syntax suffix in the Structured Syntax Suffix Registry ([RFC6838], section 6.2) as follows:

Name

Aqueous Archive Format

+suffix

+aqr

References

This document.

Encoding considerations

binary

Interoperability considerations

A media type with the +aqr suffix denotes a standalone archive, either unencrypted or encrypted, whose archive value is given a more specific meaning by that media type. A generic Aqueous decompressor can interpret an archive of such media type without knowing that meaning. The interoperability considerations for application/aqr (§ 15.1 application/aqr Media Type) also apply.

Fragment identifier considerations

The syntax and semantics of fragment identifiers specified for +aqr SHOULD be as specified for application/aqr. (At publication of this document, there is no fragment identification syntax defined for application/aqr.)

The syntax and semantics for fragment identifiers for a specific xxx/yyy+aqr SHOULD be processed as follows:

• For cases defined in +aqr, where the fragment identifier resolves per the +aqr rules, then process as specified in +aqr.
• For cases defined in +aqr, where the fragment identifier does not resolve per the +aqr rules, then process as specified in xxx/yyy+aqr.
• For cases not defined in +aqr, then process as specified in xxx/yyy+aqr.

Security considerations

The security considerations in § 14 Security considerations apply to any media type with the +aqr suffix. Each such media type may have additional security considerations specific to its interpretation of the archive.

Contact

Kang Seonghoon <public+aqr@mearie.org>

Author/Change controller

Kang Seonghoon

15.4. Quantization Interval

This document requests the registration of a new key for the Time Tag Map Keys Registry ([RFC9581], section 7.3) as follows:

Key Name

Quantization Interval

Key Value

-14 (tentative)

Reference

The Quantization Interval section in this document.

Semantics

A quantization interval specifies the interval in which the true time lies, relative to the time value given. That is, if the time value is t and the quantization interval is from a to b, the true time lies between t + a and t + b inclusive. The quantization is any means to truncate or round the true time value to a discrete representation, and notably, is not related to the accuracy of the underlying clock. For example, it is possible that the clock has a high precision but the time values are truncated to the last second, in which case the quantization interval would be [0, 1 second].

The interval is always closed. It may be slightly wider than the exact set of possible true times (e.g. [0, 1 second] for truncation, where the true time can never be exactly 1 second later), but it never excludes any possible true time, including ties in rounding.

A quantization interval can be given in terms of:

• A numeric value in seconds or a duration y, indicating the interval of [0, y];
• An array of two elements [x, y] each of which is either a numeric value in seconds or a duration, indicating the interval of [−x, y]. That is, x is the extent before the time value and y is the extent after the time value.

Every numeric value MUST be finite and non-negative, and every duration MUST have a non-negative key 1, so that the time value given is always contained in the interval. The interval MUST NOT be a single point. Therefore a single numeric value or duration MUST NOT be zero, and an array [x, y] MUST NOT have both x and y zero. If x is zero, the array MUST NOT be used and y MUST be given alone instead.

If an extended time doesn’t contain this key, its quantization interval is unknown; in particular, the time value is not implied to be exact.

For example, the first two of the following represent a time value truncated to 10 milliseconds, and the last one represents a time value rounded to the nearest second:

1001({1: 1697724754, -3: 870, -14: 0.01}),
1001({1: 1697724754, -3: 870, -14: {1: 0, -3: 10}}),
1001({1: 1697724754, -14: [0.5, 0.5]})

The following CDDL declarations can be used to represent the Quantization Interval key:

quantization-interval = ~time / ~duration / [ 2*2 ~time/~duration ]
$$ETIME-ELECTIVE //= (-14: quantization-interval)

15.5. Aqueous Dictionaries Registry

This document requests IANA to create a new registry titled "Aqueous Dictionaries", listing the registered dictionaries that a shorthand block of type 50₁₆ can refer to (§ 4.2.2 Registered dictionaries and modules). New entries are assigned with the Specification Required policy ([RFC8126], section 4.6). Each entry has the following fields:

Hash

The registered hash, i.e. the SHA-512/256 hash [FIPS180-4] of the dictionary’s bytes, written as 64 lowercase hexadecimal digits.

Name

A short name of the dictionary for humans, which has no effect on the format.

Entropy Tables

Whether the dictionary has entropy tables, i.e. the value of has_entropy_tables.

Data

The bytes of the dictionary, being the decoded data of a dictionary block (§ 5.2 Dictionary block), which MUST be valid as such and deposited with the registration.

Reference

The document describing the dictionary and its intended use.

The designated expert verifies that the hash matches the data and that no entry has the same hash. An entry, once registered, MUST NOT be modified or removed, since archives refer to it only by its hash. An entry is never shared with the Aqueous Modules Registry (§ 15.6 Aqueous Modules Registry), even when both have the same data.

The initial contents of the registry are as follows:

The dictionary "text" is intended for general text, and the dictionary "modules" is intended for module blocks given in full (§ 7.1 Module block).

The data and the hashes of the initial dictionaries are yet to be made.

15.6. Aqueous Modules Registry

This document requests IANA to create a new registry titled "Aqueous Modules", listing the registered modules that a shorthand block of type 60₁₆ can refer to (§ 4.2.2 Registered dictionaries and modules). New entries are assigned with the Specification Required policy ([RFC8126], section 4.6). Each entry has the following fields:

Hash

The registered hash, i.e. the module hash of the module, written as 64 lowercase hexadecimal digits.

Name

A short name of the module for humans, which has no effect on the format, written as the name of the module and its version joined by @ (e.g. flate-set@1). Different versions of a module have the same name before @ and different hashes.

Data

The bytes of the module, being wasm_module of a module block (§ 7.1 Module block), which MUST be valid as such and deposited with the registration.

Reference

The document describing the transformation that the module reverses, its inputs, resources and parameters, and preferably the source code from which the module was built.

The designated expert verifies that the hash matches the data, that no entry has the same hash, and that the module conforms to § 7.3 Module execution. An entry, once registered, MUST NOT be modified or removed, since archives refer to it only by its hash. An entry is never shared with the Aqueous Dictionaries Registry (§ 15.5 Aqueous Dictionaries Registry), even when both have the same data.

The initial contents of the registry are the modules built by the reference implementation of this document, as follows:

The initial entries are not yet fixed. Their modules and the documents describing them may still change before the first version of this document, and an entry changed until then is replaced in this table, not added as a new version.

16. Converting from ZIP

This section is non-normative.

This section describes how a compressor can convert entries of a ZIP archive [APPNOTE] into an archive value without losing information, where ZIP offers multiple sources or encodings for the same information.

16.1. Entry names from ZIP

A ZIP archive [APPNOTE] stores the path of each entry as a byte sequence in the local and central headers, with components separated by /. Its encoding is not always known from the header alone, and several extra fields carry the Unicode version of the same path. A compressor converting a ZIP archive can determine the encoding of each component from the following sources, where earlier sources are preferred:

1. Extra field 0x7075 (Info-ZIP Unicode Path) of version 1 [INFOZIP-EXTRAFLD], whose NameCRC32 matches the CRC-32 of the path in the header. The field holds the path in UTF-8. A field with a mismatching CRC-32 is ignored as specified by [APPNOTE], as it may have been left over by a tool that changed the path without updating the field.

2. The path in the header, if bit 11 (language encoding flag) of the general purpose bit flag is set. The path is in UTF-8.

3. Extra field 0x554e (Xceed Unicode), which holds the path in UTF-16LE. It is converted to WTF-8, which preserves any lone surrogate.

4. Extra field 0x5a4c (ZipArchive code page), which holds the path encoded in the code page given in the same field. If the field only has the code page, the path in the header is decoded with it instead.

5. Extra field 0xe57a (ALZip code page), which gives the code page of the path in the header.

6. Otherwise, the encoding of the path in the header is not specified. It is traditionally IBM code page 437, but often the system code page of the machine that made the archive, so the compressor has to guess the encoding. If it can’t, the raw byte sequence (prefixed with C1₁₆) is used for each component.

16.2. Timestamps from ZIP

A ZIP archive [APPNOTE] can carry the same timestamp in multiple places: the DOS date and time in the local and central headers, and extra fields 0x000a (NTFS), 0x5455 (extended timestamp), 0x5855 (Info-ZIP UNIX type 1) and 0x000d (UNIX) [INFOZIP-EXTRAFLD]. A compressor converting a ZIP archive can choose the source of each key as follows, where earlier sources are preferred. The order prefers more precise sources first, and then more recent specifications for the same precision.

Each source is encoded as follows:

• 0x000a stores a Windows FILETIME t, the number of 100 ns intervals since 1601-01-01T00:00:00Z, which is encoded as 125(t − 116444736000000000). A zero value means that the timestamp is absent.

• 0x5455, 0x5855 and 0x000d store the number of seconds since 1970-01-01T00:00:00Z, which is encoded as an untagged int. It is a signed 32-bit integer according to [INFOZIP-EXTRAFLD], though some implementations write times after 2038 as unsigned. The central header version of 0x5455 only has ModTime even when its flags indicate others, so the local header version is preferred when available.

• The DOS date and time is a local time in an unknown time zone with the precision of 2 seconds, which is encoded as [121(n)] where 2n is the local time as if it were UTC.

When both the DOS date and time and a UTC source are present for mtime, their difference reveals the time zone offset where the archive was made. If the DOS time minus the UTC time differs from 900k seconds by less than 2 seconds, where k is an integer in the range of aqr-tz-offset, mtime can be encoded as [u, k] where u is the encoding of the UTC source. Otherwise, e.g. due to a clock error or a daylight saving time transition, the DOS date and time is simply discarded.

If two sources for the same key disagree by more than their quantization intervals, a compressor can warn about it, and still uses the preferred source.

16.3. Comments from ZIP

A ZIP archive [APPNOTE] stores a comment for each entry in the central header only, and a comment for the whole archive in the end of central directory record. Both are byte sequences, and an empty comment is no different from the absence of one, so the compressor can omit comment or archive-comment for an empty comment.

A compressor converting a ZIP archive can determine the encoding of each entry comment from the following sources, where earlier sources are preferred:

1. Extra field 0x6375 (Info-ZIP Unicode Comment) of version 1 in the central header [INFOZIP-EXTRAFLD], whose ComCRC32 matches the CRC-32 of the comment in the header. The field holds the comment in UTF-8. A field with a mismatching CRC-32 is ignored, as for 0x7075 in § 16.1 Entry names from ZIP.

2. The comment in the header, if bit 11 (language encoding flag) of the general purpose bit flag is set. The comment is in UTF-8.

3. Extra field 0x554e (Xceed Unicode) in the central header, which holds the comment in UTF-16LE. It is converted to WTF-8, which preserves any lone surrogate.

4. Otherwise, the encoding of the comment is not specified. The compressor can decode it with the encoding guessed for entry names in the same archive. If it can’t, the raw byte sequence is used as a bstr.

The archive comment has neither the language encoding flag nor an extra field, so its encoding is never specified by the archive itself. The compressor can decode it as in the last source above, and puts the result into archive-comment.

16.4. Unix metadata from ZIP

A ZIP archive [APPNOTE] can carry the Unix metadata of an entry in the external file attributes of the central header and in extra fields 0x7875 (Info-ZIP UNIX type 3), 0x7855 (Info-ZIP UNIX type 2), 0x756e (ASi UNIX), 0x5855 (Info-ZIP UNIX type 1) and 0x000d (UNIX) [INFOZIP-EXTRAFLD]. A compressor converting a ZIP archive can choose the source of each key as follows, where earlier sources are preferred and sources in the same item have the same precedence.

Each source is encoded as follows:

• The upper 16 bits of the external file attributes hold the Unix st_mode if the upper byte of "version made by" is 3 (UNIX). Only its permission bits (0o7777) go to unix-mode, and its file type bits (S_IFMT) are not stored there.

• 0x7875 of version 1 stores the user and group IDs as little-endian unsigned integers of variable sizes. [INFOZIP-EXTRAFLD] specifies that it takes precedence over 0x7855. Other versions are not used as a source.

• 0x7855, 0x756e, 0x5855 and 0x000d store the user and group IDs as 16-bit unsigned integers. The central header version of 0x7855 is empty and only indicates that the local header version exists, and the central header version of 0x5855 doesn’t have the user and group IDs.

If two sources for the same key disagree, a compressor can warn about it, and still uses the preferred source.

16.5. Extra fields not to be preserved

The following extra fields [APPNOTE] [INFOZIP-EXTRAFLD] are not put into zip-extras, because their information is either absorbed into the archive model and other keys, or discarded as only meaningful to the ZIP container.