APX VM 2.1

This is the instruction set for APX Virtual Machine v2.1. It’s a special purpose VM used for data serialization.

The VM can execute in two different modes:

  • Pack — Serialize an in-memory data structure (variant type) into a byte array.

  • Unpack — Deserialize a byte array back into an in-memory data structure.

The data structure is assumed to be compatible with how most scripting languages represent variables in-memory:

  • Scalar (int, string, boolean, character).

  • Array (each element can be scalar, array, or record).

  • Record/Struct (each member can be a scalar, array, or record).

In many programming languages, this way of storing data is called variant or variant type. Its representation is compatible with JSON.

Different APX implementations use different methods of data representation:

Language

Data Representation

C

dtl_type

C++

dtl (for C++)

Python

Python native types (int, list, dict, etc.)

VBA

Variant, Scripting.Dictionary

APX Programs

An APX program is a byte array containing serialized (binary) instructions describing how to pack or unpack a data element in a port. The programs are compiled once and then executed repeatedly by the APX Virtual Machine (APX VM) as needed.

Program encoding

APX uses a variable-length program encoding where each instruction starts with a 1-byte instruction header. Depending on the instruction and variant, additional payload bytes (such as array lengths, limit values, or field names) are encoded directly after the instruction header. The next instruction starts immediately after the end of the previous instruction’s payload (with zero padding and zero alignment).

Before the list of instructions starts, there is a program header which specifies the VM version, program type (PACK or UNPACK), data size, and optional port properties (dynamic arrays, queued ports).

Program header

In VM v2.1, the program header has a variable length (typically between 4 and 12 bytes). The magic number prefix used in VM v2.0 ("APX" or "VM") has been dropped in v2.1, allowing the program header to begin directly with the VM version numbers.

Base Program Header

Byte

Description

Value

Byte 0

APX VM major version number ('2' in ASCII)

0x32

Byte 1

APX VM minor version number ('1' in ASCII)

0x31

Byte 2

Program Flags and Type byte

See below

Bytes 3..(N-1) (little-endian)

Maximum expected data size (max_data_size)

1, 2, or 4 bytes

Byte 2: Program Flags and Type Layout

Bit 7

Bit 6

Bit 5

Bit 4

Bit 3

Bits 0-2

Reserved (0)

Reserved (0)

QUEUED_DATA (0x20)

DYNAMIC_DATA (0x10)

PACK_PROG (0x08)

Data Size Variant (0x07)

Program Flags and Types

Mask / Bit

Name

Description

Bits 0-2 (0x07)

Data Size Variant

Determines the byte width of the max_data_size integer directly following Byte 2:
• 0 (UINT8): 1 byte (0..255)
• 1 (UINT16): 2 bytes, little-endian (0..65535)
• 2 (UINT32): 4 bytes, little-endian (0..4294967295)

Bit 3 (0x08)

PACK_PROG

Program Type:
• 0: UNPACK program
• 1: PACK program

Bit 4 (0x10)

DYNAMIC_DATA

Active (1) if there are any dynamic arrays in the program data.

Bit 5 (0x20)

QUEUED_DATA

Active (1) if this program is for a queued port.

Bits 6-7 (0xC0)

RESERVED

Reserved for future use (must be 0).

Queued Port Header Extension

When QUEUED_DATA (Bit 5) is set, the base header is followed by an embedded DATA_SIZE instruction (Opcode 2, Variants 3..11) describing both the element size and queue size:

Byte

Description

Value

Byte N

Embedded DATA_SIZE instruction header

Opcode 2, Variant 3..11 (Flag = 0)

Bytes (N+1)..(N+M)

Element Size (elem_size)

1, 2, or 4 bytes (little-endian unsigned integer)

The queue length can be derived using the formula:

\[\text{Queue Length} = \frac{\text{MaxDataSize} - \text{QueueStorageSize}}{\text{ElementSize}}\]

where QueueStorageSize is the header prefix size (in bytes) reserved in the data buffer to store the current queue length:

  • 1 byte when Queue Size is UINT8 (Variants 3, 6, 9)

  • 2 bytes when Queue Size is UINT16 (Variants 4, 7, 10)

  • 4 bytes when Queue Size is UINT32 (Variants 5, 8, 11)

Instruction set

Each instruction begins with a single 1-byte header:

Instruction Header Format:

Bit 7

Bits 4-6

Bits 0-3

1 Flag bit

3 Opcode bits

4 Variant bits

  • Bit 7 (0x80): Flag bit (context-dependent: denotes array type, dynamic array, first field of record, or array limit check).

  • Bits 4-6 (0x70): Opcode (values 0..7).

  • Bits 0-3 (0x0F): Variant (values 0..15).

Summary of Opcodes

Opcode

Name

Description

0

PACK

Pack (serialize) an in-memory data item into the byte buffer

1

UNPACK

Unpack (deserialize) bytes from the buffer into an in-memory data item

2

DATA_SIZE

Array length or element/queue size descriptor

3

DATA_CTRL

Data control (record member selection, limit checking)

4

FLOW_CTRL

Program flow control (advance to next array element)

5

RESERVED

Reserved for future use

6

RESERVED

Reserved for future use

7

RESERVED

Reserved for future use

Opcode 0 — PACK

Serializes an in-memory value into the binary output buffer.

Flag bit (Bit 7):

  • 0: Scalar data type.

  • 1: Array data type. When set, the instruction must be immediately followed by an ARRAY_SIZE instruction (Opcode 2, Variants 0..2).

Variant ID

Data Type

Data Size (bytes)

Description

0

UINT8

1

8-bit unsigned integer

1

UINT16

2

16-bit unsigned integer (little-endian)

2

UINT32

4

32-bit unsigned integer (little-endian)

3

UINT64

8

64-bit unsigned integer (little-endian)

4

INT8

1

8-bit signed integer

5

INT16

2

16-bit signed integer (little-endian)

6

INT32

4

32-bit signed integer (little-endian)

7

INT64

8

64-bit signed integer (little-endian)

8

BOOL

1

Boolean value (0 = False, 1 = True)

9

BYTE

1

Raw byte / blob value

10

RECORD

N/A

Record / struct definition

11

ARRAY

N/A

Reserved for nested arrays / future use

12

CHAR

1

ASCII character

13

CHAR8

1

UTF-8 character code unit

14

CHAR16

2

UTF-16 character code unit (little-endian)

15

CHAR32

4

UTF-32 character code unit (little-endian)

Opcode 1 — UNPACK

Deserializes binary bytes from the input buffer into an in-memory data item.

The variant IDs, payload sizes, and flag bit behavior are identical to Opcode 0 (PACK).

Opcode 2 — DATA_SIZE

Used to encode array lengths, or element and queue sizes for queued ports.

The integer value is encoded in the program immediately following the 1-byte instruction header in little-endian format.

Array Size Variants (Variants 0..2)

Used directly after a PACK or UNPACK instruction when its flag bit is set (is_array = true).

Flag bit (Bit 7):

  • 0: Fixed-length array. The length directly following the header specifies the exact number of elements.

  • 1: Dynamic array. The length directly following the header specifies the maximum number of elements. At runtime, the current array length is encoded into or decoded from the data buffer preceding the array elements.

Variant ID

Name

Integer Size (bytes)

Description

0

ARRAY_SIZE_UINT8

1

Array length range 0..255

1

ARRAY_SIZE_UINT16

2

Array length range 256..65535 (little-endian)

2

ARRAY_SIZE_UINT32

4

Array length range > 65535 (little-endian)

Element and Queue Size Variants (Variants 3..11)

Used exclusively within the program header when HEADER_FLAG_QUEUED_DATA is set.

Flag bit (Bit 7): Unused (must be 0).

Payload: The element size (elem_size) encoded as an unsigned integer directly after the instruction header.

Variant ID

Name

Element Size Payload (bytes)

Queue Length Storage (bytes)

3

ELEMENT_SIZE_U8_QUEUE_SIZE_UINT8

1

1

4

ELEMENT_SIZE_U8_QUEUE_SIZE_UINT16

1

2

5

ELEMENT_SIZE_U8_QUEUE_SIZE_UINT32

1

4

6

ELEMENT_SIZE_U16_QUEUE_SIZE_UINT8

2

1

7

ELEMENT_SIZE_U16_QUEUE_SIZE_UINT16

2

2

8

ELEMENT_SIZE_U16_QUEUE_SIZE_UINT32

2

4

9

ELEMENT_SIZE_U32_QUEUE_SIZE_UINT8

4

1

10

ELEMENT_SIZE_U32_QUEUE_SIZE_UINT16

4

2

11

ELEMENT_SIZE_U32_QUEUE_SIZE_UINT32

4

4

Opcode 3 — DATA_CTRL

Controls record field selection and range limit verification.

Variant ID

Name

Payload Bytes after Header

Description

0

RECORD_SELECT

Variable

Selects the record member name

1

RECORD_END

0

Explicit end-of-record delimiter

2

LIMIT_CHECK_UINT8

2

Range check for UINT8 (Lower limit: 1B, Upper limit: 1B)

3

LIMIT_CHECK_UINT16

4

Range check for UINT16 (Lower limit: 2B, Upper limit: 2B)

4

LIMIT_CHECK_UINT32

8

Range check for UINT32 (Lower limit: 4B, Upper limit: 4B)

5

LIMIT_CHECK_UINT64

16

Range check for UINT64 (Lower limit: 8B, Upper limit: 8B)

6

LIMIT_CHECK_INT8

2

Range check for INT8 (Lower limit: 1B, Upper limit: 1B)

7

LIMIT_CHECK_INT16

4

Range check for INT16 (Lower limit: 2B, Upper limit: 2B)

8

LIMIT_CHECK_INT32

8

Range check for INT32 (Lower limit: 4B, Upper limit: 4B)

9

LIMIT_CHECK_INT64

16

Range check for INT64 (Lower limit: 8B, Upper limit: 8B)

Variant 0 — RECORD_SELECT

Selects the name of the next record member. The field name is encoded into the program as a null-terminated ASCII string starting immediately at the byte following the instruction header.

Flag bit (Bit 7):

  • 0: Subsequent record field (is_first_field = false).

  • 1: This is the first field in the record (is_first_field = true).

Variant 1 — RECORD_END

Marks the end of a record structure where an explicit termination marker is required. Has no payload.

Variants 2..10 — LIMIT_CHECK

Encodes two integers directly after the instruction header:

  1. lower_limit (minimum allowed value)

  2. upper_limit (maximum allowed value)

Both limits match the signedness and byte width of the corresponding variant and are stored in little-endian format.

Flag bit (Bit 7):

  • 0: The limit check applies to a scalar value.

  • 1: The limit check applies to an array of values (the limit check must be validated for every element in the array).

Instruction Ordering:

  • In PACK programs: The LIMIT_CHECK instruction is encoded before the PACK instruction to validate in-memory values before serialization.

  • In UNPACK programs: The LIMIT_CHECK instruction is encoded after the UNPACK instruction to validate values after deserialization.

Opcode 4 — FLOW_CTRL

Controls VM execution flow.

Variant ID

Name

Payload Bytes

Description

0

ARRAY_NEXT

0

Advance to next array element

Variant 0 — ARRAY_NEXT

Advances the internal array iterator to the next element. This is primarily used when serializing or deserializing arrays of complex types, such as an array of records (structs).

Opcode 5 — RESERVED

Reserved for future instruction set extensions.

Opcode 6 — RESERVED

Reserved for future instruction set extensions.

Opcode 7 — RESERVED

Reserved for future instruction set extensions.