# APX VM 2.1 This is the instruction set for APX Virtual Machine v2.1. It's a special purpose VM used for data serialization. The VM can execute in two different modes: - Pack — Serialize an in-memory data structure (variant type) into a byte array. - Unpack — Deserialize a byte array back into an in-memory data structure. The data structure is assumed to be compatible with how most scripting languages represent variables in-memory: - Scalar (int, string, boolean, character). - Array (each element can be scalar, array, or record). - Record/Struct (each member can be a scalar, array, or record). In many programming languages, this way of storing data is called *variant* or *variant type*. Its representation is compatible with JSON. Different APX implementations use different methods of data representation: | Language | Data Representation | |-----------|----------------------------------------------| | C | [dtl_type](https://github.com/cogu/dtl_type) | | C++ | dtl (for C++) | | Python | Python native types (int, list, dict, etc.) | | VBA | Variant, Scripting.Dictionary | ## APX Programs An APX program is a byte array containing serialized (binary) instructions describing how to pack or unpack a data element in a port. The programs are compiled once and then executed repeatedly by the APX Virtual Machine (APX VM) as needed. ## Program encoding APX uses a variable-length program encoding where each instruction starts with a 1-byte instruction header. Depending on the instruction and variant, additional payload bytes (such as array lengths, limit values, or field names) are encoded directly after the instruction header. The next instruction starts immediately after the end of the previous instruction's payload (with zero padding and zero alignment). Before the list of instructions starts, there is a program header which specifies the VM version, program type (PACK or UNPACK), data size, and optional port properties (dynamic arrays, queued ports). ## Program header In VM v2.1, the program header has a variable length (typically between 4 and 12 bytes). The magic number prefix used in VM v2.0 (`"APX"` or `"VM"`) has been dropped in v2.1, allowing the program header to begin directly with the VM version numbers. ### Base Program Header | Byte | Description | Value | |--------------------------|-----------------------------------------------------|---------------| | Byte 0 | APX VM major version number (`'2'` in ASCII) | 0x32 | | Byte 1 | APX VM minor version number (`'1'` in ASCII) | 0x31 | | Byte 2 | Program Flags and Type byte | See below | | Bytes 3..(N-1) (little-endian) | Maximum expected data size (`max_data_size`) | 1, 2, or 4 bytes | ### Byte 2: Program Flags and Type Layout | Bit 7 | Bit 6 | Bit 5 | Bit 4 | Bit 3 | Bits 0-2 | |:-----:|:-----:|:-----:|:-----:|:-----:|:--------:| | Reserved (0) | Reserved (0) | `QUEUED_DATA` (0x20) | `DYNAMIC_DATA` (0x10) | `PACK_PROG` (0x08) | Data Size Variant (0x07) | #### Program Flags and Types | Mask / Bit | Name | Description | |---|---|---| | **Bits 0-2** (`0x07`) | **Data Size Variant** | Determines the byte width of the `max_data_size` integer directly following Byte 2:
• `0` (`UINT8`): 1 byte (0..255)
• `1` (`UINT16`): 2 bytes, little-endian (0..65535)
• `2` (`UINT32`): 4 bytes, little-endian (0..4294967295) | | **Bit 3** (`0x08`) | **PACK_PROG** | Program Type:
• `0`: UNPACK program
• `1`: PACK program | | **Bit 4** (`0x10`) | **DYNAMIC_DATA** | Active (1) if there are any dynamic arrays in the program data. | | **Bit 5** (`0x20`) | **QUEUED_DATA** | Active (1) if this program is for a queued port. | | **Bits 6-7** (`0xC0`) | **RESERVED** | Reserved for future use (must be 0). | ### Queued Port Header Extension When `QUEUED_DATA` (Bit 5) is set, the base header is followed by an embedded `DATA_SIZE` instruction (Opcode 2, Variants 3..11) describing both the element size and queue size: | Byte | Description | Value | |---|---|---| | Byte N | Embedded `DATA_SIZE` instruction header | Opcode 2, Variant 3..11 (Flag = 0) | | Bytes (N+1)..(N+M) | Element Size (`elem_size`) | 1, 2, or 4 bytes (little-endian unsigned integer) | The queue length can be derived using the formula: $$\text{Queue Length} = \frac{\text{MaxDataSize} - \text{QueueStorageSize}}{\text{ElementSize}}$$ where `QueueStorageSize` is the header prefix size (in bytes) reserved in the data buffer to store the current queue length: - 1 byte when Queue Size is `UINT8` (Variants 3, 6, 9) - 2 bytes when Queue Size is `UINT16` (Variants 4, 7, 10) - 4 bytes when Queue Size is `UINT32` (Variants 5, 8, 11) ## Instruction set Each instruction begins with a single 1-byte header: **Instruction Header Format:** | Bit 7 | Bits 4-6 | Bits 0-3 | |:-----:|:--------:|:--------:| | 1 Flag bit | 3 Opcode bits | 4 Variant bits | - **Bit 7** (`0x80`): Flag bit (context-dependent: denotes array type, dynamic array, first field of record, or array limit check). - **Bits 4-6** (`0x70`): Opcode (values 0..7). - **Bits 0-3** (`0x0F`): Variant (values 0..15). ### Summary of Opcodes | Opcode | Name | Description | |:---:|---|---| | `0` | **PACK** | Pack (serialize) an in-memory data item into the byte buffer | | `1` | **UNPACK** | Unpack (deserialize) bytes from the buffer into an in-memory data item | | `2` | **DATA_SIZE** | Array length or element/queue size descriptor | | `3` | **DATA_CTRL** | Data control (record member selection, limit checking) | | `4` | **FLOW_CTRL** | Program flow control (advance to next array element) | | `5` | **RESERVED** | Reserved for future use | | `6` | **RESERVED** | Reserved for future use | | `7` | **RESERVED** | Reserved for future use | --- ### Opcode 0 — PACK Serializes an in-memory value into the binary output buffer. **Flag bit (Bit 7):** - `0`: Scalar data type. - `1`: Array data type. When set, the instruction **must be immediately followed** by an `ARRAY_SIZE` instruction (Opcode 2, Variants 0..2). | Variant ID | Data Type | Data Size (bytes) | Description | |:---:|---|:---:|---| | 0 | `UINT8` | 1 | 8-bit unsigned integer | | 1 | `UINT16` | 2 | 16-bit unsigned integer (little-endian) | | 2 | `UINT32` | 4 | 32-bit unsigned integer (little-endian) | | 3 | `UINT64` | 8 | 64-bit unsigned integer (little-endian) | | 4 | `INT8` | 1 | 8-bit signed integer | | 5 | `INT16` | 2 | 16-bit signed integer (little-endian) | | 6 | `INT32` | 4 | 32-bit signed integer (little-endian) | | 7 | `INT64` | 8 | 64-bit signed integer (little-endian) | | 8 | `BOOL` | 1 | Boolean value (0 = False, 1 = True) | | 9 | `BYTE` | 1 | Raw byte / blob value | | 10 | `RECORD` | N/A | Record / struct definition | | 11 | `ARRAY` | N/A | Reserved for nested arrays / future use | | 12 | `CHAR` | 1 | ASCII character | | 13 | `CHAR8` | 1 | UTF-8 character code unit | | 14 | `CHAR16` | 2 | UTF-16 character code unit (little-endian) | | 15 | `CHAR32` | 4 | UTF-32 character code unit (little-endian) | --- ### Opcode 1 — UNPACK Deserializes binary bytes from the input buffer into an in-memory data item. The variant IDs, payload sizes, and flag bit behavior are identical to **Opcode 0 (PACK)**. --- ### Opcode 2 — DATA_SIZE Used to encode array lengths, or element and queue sizes for queued ports. The integer value is encoded in the program immediately following the 1-byte instruction header in little-endian format. #### Array Size Variants (Variants 0..2) Used directly after a `PACK` or `UNPACK` instruction when its flag bit is set (`is_array = true`). **Flag bit (Bit 7):** - `0`: Fixed-length array. The length directly following the header specifies the exact number of elements. - `1`: Dynamic array. The length directly following the header specifies the **maximum** number of elements. At runtime, the current array length is encoded into or decoded from the data buffer preceding the array elements. | Variant ID | Name | Integer Size (bytes) | Description | |:---:|---|:---:|---| | 0 | `ARRAY_SIZE_UINT8` | 1 | Array length range 0..255 | | 1 | `ARRAY_SIZE_UINT16` | 2 | Array length range 256..65535 (little-endian) | | 2 | `ARRAY_SIZE_UINT32` | 4 | Array length range > 65535 (little-endian) | #### Element and Queue Size Variants (Variants 3..11) Used exclusively within the program header when `HEADER_FLAG_QUEUED_DATA` is set. **Flag bit (Bit 7):** Unused (must be 0). **Payload:** The element size (`elem_size`) encoded as an unsigned integer directly after the instruction header. | Variant ID | Name | Element Size Payload (bytes) | Queue Length Storage (bytes) | |:---:|---|:---:|:---:| | 3 | `ELEMENT_SIZE_U8_QUEUE_SIZE_UINT8` | 1 | 1 | | 4 | `ELEMENT_SIZE_U8_QUEUE_SIZE_UINT16` | 1 | 2 | | 5 | `ELEMENT_SIZE_U8_QUEUE_SIZE_UINT32` | 1 | 4 | | 6 | `ELEMENT_SIZE_U16_QUEUE_SIZE_UINT8` | 2 | 1 | | 7 | `ELEMENT_SIZE_U16_QUEUE_SIZE_UINT16` | 2 | 2 | | 8 | `ELEMENT_SIZE_U16_QUEUE_SIZE_UINT32` | 2 | 4 | | 9 | `ELEMENT_SIZE_U32_QUEUE_SIZE_UINT8` | 4 | 1 | | 10 | `ELEMENT_SIZE_U32_QUEUE_SIZE_UINT16` | 4 | 2 | | 11 | `ELEMENT_SIZE_U32_QUEUE_SIZE_UINT32` | 4 | 4 | --- ### Opcode 3 — DATA_CTRL Controls record field selection and range limit verification. | Variant ID | Name | Payload Bytes after Header | Description | |:---:|---|:---:|---| | 0 | `RECORD_SELECT` | Variable | Selects the record member name | | 1 | `RECORD_END` | 0 | Explicit end-of-record delimiter | | 2 | `LIMIT_CHECK_UINT8` | 2 | Range check for `UINT8` (Lower limit: 1B, Upper limit: 1B) | | 3 | `LIMIT_CHECK_UINT16` | 4 | Range check for `UINT16` (Lower limit: 2B, Upper limit: 2B) | | 4 | `LIMIT_CHECK_UINT32` | 8 | Range check for `UINT32` (Lower limit: 4B, Upper limit: 4B) | | 5 | `LIMIT_CHECK_UINT64` | 16 | Range check for `UINT64` (Lower limit: 8B, Upper limit: 8B) | | 6 | `LIMIT_CHECK_INT8` | 2 | Range check for `INT8` (Lower limit: 1B, Upper limit: 1B) | | 7 | `LIMIT_CHECK_INT16` | 4 | Range check for `INT16` (Lower limit: 2B, Upper limit: 2B) | | 8 | `LIMIT_CHECK_INT32` | 8 | Range check for `INT32` (Lower limit: 4B, Upper limit: 4B) | | 9 | `LIMIT_CHECK_INT64` | 16 | Range check for `INT64` (Lower limit: 8B, Upper limit: 8B) | #### Variant 0 — RECORD_SELECT Selects the name of the next record member. The field name is encoded into the program as a null-terminated ASCII string starting immediately at the byte following the instruction header. **Flag bit (Bit 7):** - `0`: Subsequent record field (`is_first_field = false`). - `1`: This is the **first field** in the record (`is_first_field = true`). #### Variant 1 — RECORD_END Marks the end of a record structure where an explicit termination marker is required. Has no payload. #### Variants 2..10 — LIMIT_CHECK Encodes two integers directly after the instruction header: 1. `lower_limit` (minimum allowed value) 2. `upper_limit` (maximum allowed value) Both limits match the signedness and byte width of the corresponding variant and are stored in little-endian format. **Flag bit (Bit 7):** - `0`: The limit check applies to a scalar value. - `1`: The limit check applies to an array of values (the limit check must be validated for every element in the array). **Instruction Ordering:** - In **PACK** programs: The `LIMIT_CHECK` instruction is encoded **before** the `PACK` instruction to validate in-memory values before serialization. - In **UNPACK** programs: The `LIMIT_CHECK` instruction is encoded **after** the `UNPACK` instruction to validate values after deserialization. --- ### Opcode 4 — FLOW_CTRL Controls VM execution flow. | Variant ID | Name | Payload Bytes | Description | |:---:|---|:---:|---| | 0 | `ARRAY_NEXT` | 0 | Advance to next array element | #### Variant 0 — ARRAY_NEXT Advances the internal array iterator to the next element. This is primarily used when serializing or deserializing arrays of complex types, such as an array of records (structs). --- ### Opcode 5 — RESERVED Reserved for future instruction set extensions. --- ### Opcode 6 — RESERVED Reserved for future instruction set extensions. --- ### Opcode 7 — RESERVED Reserved for future instruction set extensions.