APX Data Serialization Rules

This document defines the binary serialization and deserialization rules for the APX (AUTOSAR Port eXchange) Virtual Machine, detailing how each data type (TypeCode) is encoded into a byte buffer. It also specifies the architectural mechanism for handling dynamic data structures in memory-mapped environments via deterministic slot padding (padded_next).

1. Overview and Core Principles

APX is designed for high-performance inter-process, inter-core, and networked communication in embedded and distributed automotive software systems.

1.1 Packed Binary Serialization

In almost all scenarios, APX serializes data densely packed without inter-field padding or structural alignment. Data fields are placed consecutively into the binary stream byte-for-byte, matching the semantics of packed structures (e.g. #pragma pack(push, 1) in C/C++).

  • No compiler-induced alignment bytes (e.g. aligning 16-bit or 32-bit values to even or 4-byte boundaries).

  • Consecutive fields in records/structs are placed immediately adjacent to one another.

1.2 Endianness

All multi-byte numeric primitives (UINT16, UINT32, UINT64, INT16, INT32, INT64) and multi-byte character code units (CHAR16, CHAR32) are serialized in Standard Little-Endian (LE) byte order, regardless of the host machine’s native CPU architecture.

1.3 Memory-Mapped Determinism

At lower layers of the APX communication pipeline (such as shared memory files, byte port maps, and virtual bus adapters), buffers are memory-mapped. This requires that the maximum memory footprint and offsets of all data structures and fields are statically predictable.

To support variable-length data (such as dynamic arrays and dynamic strings) without causing subsequent data elements to shift in memory offset, APX implements a specialized slot padding mechanism (see Section 3).

2. Serialization Rules by Data Type (TypeCode)

This section specifies how individual data types are serialized and deserialized by the APX Virtual Machine.

2.1 Summary of TypeCodes

TypeCode

ID

Byte Size

Native / Scripting Equivalent

Description

UINT8

0

1

uint8_t / int

8-bit unsigned integer

UINT16

1

2

uint16_t / int

16-bit unsigned integer (little-endian)

UINT32

2

4

uint32_t / int

32-bit unsigned integer (little-endian)

UINT64

3

8

uint64_t / int

64-bit unsigned integer (little-endian)

INT8

4

1

int8_t / int

8-bit signed two’s complement integer

INT16

5

2

int16_t / int

16-bit signed two’s complement integer (little-endian)

INT32

6

4

int32_t / int

32-bit signed two’s complement integer (little-endian)

INT64

7

8

int64_t / int

64-bit signed two’s complement integer (little-endian)

BOOL

8

1

bool

Boolean value (0x00 = false, 0x01 = true)

BYTE

9

1

uint8_t / bytes

Raw byte / blob code unit

RECORD

10

Variable

struct / dict / hash

Composite record structure

RESERVED

11

N/A

N/A

Reserved for future use

CHAR

12

1

char / str

ASCII character code unit

CHAR8

13

1

char8_t / str

UTF-8 character code unit

CHAR16

14

2

char16_t / str

UTF-16 character code unit (little-endian)

CHAR32

15

4

char32_t / str

UTF-32 character code unit (little-endian)

2.2 Unsigned Integers (UINT8, UINT16, UINT32, UINT64)

Unsigned integer types represent non-negative binary integers.

  • UINT8 (1 byte): Encoded as a single byte in range 0..255.

  • UINT16 (2 bytes): Encoded as 2 bytes in little-endian format (least significant byte first), range 0..65,535.

  • UINT32 (4 bytes): Encoded as 4 bytes in little-endian format, range 0..4,294,967,295.

  • UINT64 (8 bytes): Encoded as 8 bytes in little-endian format, range 0..18,446,744,073,709,551,615.

Byte Layout Example (UINT16 = 0x1234):

Offset 0: 0x34 (LSB)
Offset 1: 0x12 (MSB)

Byte Layout Example (UINT32 = 0x01020304):

Offset 0: 0x04 (LSB)
Offset 1: 0x03
Offset 2: 0x02
Offset 3: 0x01 (MSB)

2.3 Signed Integers (INT8, INT16, INT32, INT64)

Signed integer types represent two’s complement binary integers.

  • INT8 (1 byte): Encoded as a single signed byte, range -128..127.

  • INT16 (2 bytes): Encoded as 2 bytes two’s complement in little-endian format, range -32,768..32,767.

  • INT32 (4 bytes): Encoded as 4 bytes two’s complement in little-endian format, range -2,147,483,648..2,147,483,647.

  • INT64 (8 bytes): Encoded as 8 bytes two’s complement in little-endian format, range -9,223,372,036,854,775,808..9,223,372,036,854,775,807.

Byte Layout Example (INT16 = -2 / 0xFFFE):

Offset 0: 0xFE
Offset 1: 0xFF

2.4 Boolean (BOOL)

  • Size: 1 byte.

  • Values:

    • 0x00: false

    • 0x01: true

  • Serialization Behavior: Any non-zero truthy input in the high-level variant (e.g. true, 1) is normalized and written as 0x01.

  • Deserialization Behavior: 0x00 deserializes to false; any non-zero value (!= 0) deserializes to true.

2.5 Raw Bytes / Blob (BYTE)

  • Unit Size: 1 byte per element.

  • Usage: Used for raw binary blobs, payload buffers, or cryptographic digests.

  • Fixed Byte Array (BYTE[N]): Exactly N bytes are copied directly into the buffer. The in-memory payload length must match N exactly.

  • Dynamic Byte Array (BYTE[<=N]): Prefixed by a dynamic length header (1, 2, or 4 bytes), followed by K <= N active bytes. Unused allocated capacity (N - K) is handled via slot padding (see Section 3).

2.6 Characters and Strings (CHAR, CHAR8, CHAR16, CHAR32)

2.6.1 Scalar Characters (Array Length = 0)

When encoded as a scalar character, a single character code unit is packed directly:

  • CHAR / CHAR8: 1 byte (ASCII / UTF-8 code unit).

  • CHAR16: 2 bytes (UTF-16 code unit, little-endian).

  • CHAR32: 4 bytes (UTF-32 code unit, little-endian).

2.6.2 Fixed-Length Strings (CHAR[N], CHAR8[N])

Fixed-length strings allocate exactly N * element_size bytes in the buffer:

  • No dynamic length prefix is written.

  • If the in-memory string length L <= N, the serializer writes the string bytes and zero-fills (null-pads) the remaining N - L bytes up to the full buffer capacity N.

  • If L > N, a buffer error (APX_BUFFER_BOUNDARY_ERROR / VALUE_LENGTH_ERROR) is raised.

Example: Fixed String CHAR[6] with value "Hi":

Offset 0: 'H' (0x48)
Offset 1: 'i' (0x69)
Offset 2: 0x00 (null padding)
Offset 3: 0x00 (null padding)
Offset 4: 0x00 (null padding)
Offset 5: 0x00 (null padding)
Total size: 6 bytes

2.6.3 Dynamic Strings (CHAR[<=N], CHAR8[<=N])

Dynamic strings are variable-length character sequences with a declared maximum capacity N:

  • Length Prefix: 1, 2, or 4 bytes indicating the actual string length K (K <= N).

  • Payload: Exactly K character bytes (no null-terminator is required in the packed payload).

  • Slot Padding: The buffer pointer is padded out to N when moving to subsequent fields.

2.7 Records (Structs)

A record is a composite data structure containing an ordered sequence of named fields.

2.7.1 Field Ordering and Packing

  • Fields are serialized in the exact declaration order defined in the APX IDL specification.

  • No structure alignment or compiler-inserted padding exists between fields. For example, a UINT8 followed by a UINT32 occupies exactly 5 contiguous bytes (\(1 + 4\)).

2.7.2 Nested Records

  • A record field may itself be a child record.

  • The child record’s fields are serialized inline in sequence.

2.7.3 Arrays of Records (RECORD[N] and RECORD[<=N])

  • Fixed Array of Records (RECORD[N]): All N record instances are serialized consecutively. Each record instance occupies its full static maximum record size.

  • Dynamic Array of Records (RECORD[<=N]): Prefixed by the array length integer (\(K \le N\)), followed by \(K\) serialized record instances.

Note: When each record element contains dynamic fields, each record element is padded to its maximum static record size before the next record element begins (see Section 3.3).

2.8 Queued Port Serialization (QUEUED_DATA)

For queued provide/require ports, the port buffer stores a queue of elements:

  • Queue Length Prefix: Encodes the current number of valid elements queued in the buffer (\(0 \le K_{\text{queue}} \le N_{\text{queue}}\)).

  • Element Storage: A pre-allocated array of \(N_{\text{queue}}\) slots, each of size \(\text{ElementSize}\).

  • Queue Header Calculation: $\(\text{Total Port Buffer Size} = \text{QueueStorageSize} + (N_{\text{queue}} \times \text{ElementSize})\)\( \)\(\text{Queue Length} = \frac{\text{MaxDataSize} - \text{QueueStorageSize}}{\text{ElementSize}}\)$

3. The Padding Mechanism for Memory-Mapped Predictability

3.1 The Problem: Fluctuation in Memory-Mapped Offsets

In low-level APX implementations, data communication relies on memory-mapped buffers (such as shared memory files or memory regions mapped directly to device drivers):

  1. Deterministic Offsets: Downstream consumers, receivers, and port decoders rely on fixed byte offsets within the memory map to read individual fields without parsing previous fields.

  2. Dynamic Data Dilemma: When a dynamic array or dynamic string exists inside a record or before other fields, serializing only the active elements (\(K < N_{\text{max}}\)) would cause all subsequent fields to slide forward to a lower memory offset.

  3. Array of Records Dilemma: In an array of records where each record contains a dynamic field, variable record lengths would destroy the uniform stride required to index element \(i\) at \(\text{base} + i \times \text{record\_size}\).

To solve this, APX introduces slot padding (padded_next).

3.2 The Architectural Solution: Slot Padding (padded_next)

Whenever a dynamic array (or dynamic string/record element) is encountered during serialization or deserialization:

  1. Dynamic Length Prefix Determination: The array is prefixed with an unsigned integer indicating the active element count \(K\) (\(0 \le K \le N_\text{max}\)). The prefix size is determined by the declared maximum capacity \(N_\text{max}\):

    Maximum Array Length (\(N_{\text{max}}\))

    SizeType Enum

    Prefix Encoding

    Prefix Size

    \(1 \le N_{\text{max}} \le 255\)

    UINT8 (0)

    uint8_t

    1 byte

    \(256 \le N_{\text{max}} \le 65,535\)

    UINT16 (1)

    uint16_t (little-endian)

    2 bytes

    \(65,536 \le N_{\text{max}} \le 4,294,967,295\)

    UINT32 (2)

    uint32_t (little-endian)

    4 bytes

  2. Calculate Maximum Slot Boundary: The data serialization engine calculates the address or offset where the dynamic array would end if it were populated to its maximum capacity \(N_{\text{max}}\): $\(\text{padded\_next} = \text{current\_buffer\_pos} + \text{LengthPrefixSize} + (N_{\text{max}} \times \text{ElementSize})\)$

  3. Serialize Active Payload: The data serialization engine writes/reads the dynamic length prefix \(K\) and the \(K\) active elements.

  4. Advance Pointer Before Next Item: Before serializing or deserializing the next sibling field or subsequent data element, the data serialization engine invokes prepare_for_buffer_write() / prepare_for_buffer_read(). If padded_next is set, the active buffer pointer (next) is advanced directly to padded_next, safely skipping all unwritten/padding bytes in the slot and resetting padded_next to null.

3.3 Concrete Byte Layout Examples

Example 1: Record with Dynamic String and Sibling Field

Type Definition:

# Record with a dynamic string (max 8 chars) and a uint32 status code
{"Name"a[<=8]"Status"L}

  • Max capacity of "Name": 1 byte (length prefix for \(N \le 255\)) + 8 bytes = 9 bytes.

  • Size of "Status": 4 bytes (UINT32).

  • Total static record size: \(9 + 4 = 13\) bytes.

Scenario A: Full string "ABCDEFGH" (8 chars, Status = 0x12345678):

Offset  0:     0x08        (Length = 8)
Offset  1..8:  "ABCDEFGH" (8 bytes payload)
Offset  9..12: 0x78, 0x56, 0x34, 0x12 (Status, little-endian)
Total written: 13 bytes

Scenario B: Short string "Hi" (2 chars, Status = 0x12345678):

Offset  0:     0x02        (Length = 2)
Offset  1..2:  "Hi"      (2 bytes payload)
Offset  3..8:  [PADDING]   (6 unused bytes, skipped by padded_next)
Offset  9..12: 0x78, 0x56, 0x34, 0x12 (Status, written at deterministic offset 9)
Total slot:    13 bytes

Because prepare_for_buffer_write() jumps from offset 3 to offset 9 before writing "Status", the "Status" field always remains at byte offset 9, preserving memory-mapped integrity.

Example 2: Array of Records Containing a Dynamic String

Type Definition:

# Array of 2 records, each with a dynamic string (max 4 chars) and a uint8 id
{"Label"a[<=4]"Id"C}[2]

  • Each record maximum size: \((1 \text{ byte len} + 4 \text{ bytes payload}) + 1 \text{ byte Id} = 6 \text{ bytes}\).

  • Array total size: \(2 \times 6 = 12\) bytes.

Data to serialize:

  • Element 0: {"Label": "Cat", "Id": 10} (Length = 3)

  • Element 1: {"Label": "A",   "Id": 20} (Length = 1)

Serialized Byte Stream:

-- Element 0 (Record 0, starts at offset 0) --
Offset  0:     0x03          (Label length = 3)
Offset  1:     'C'
Offset  2:     'a'
Offset  3:     't'
Offset  4:     [PADDING]     (1 byte padding up to max 4 chars)
Offset  5:     0x0A          (Id = 10)

-- Element 1 (Record 1, starts at offset 6) --
Offset  6:     0x01          (Label length = 1)
Offset  7:     'A'
Offset  8..10: [PADDING]     (3 bytes padding up to max 4 chars)
Offset 11:     0x14          (Id = 20)

Total buffer size: 12 bytes

Every record instance occupies exactly 6 bytes. array_next() advances across predictable 6-byte boundaries.

4. Implementation References

4.1 C Implementation (c-apx)

  • Serializer (apx/src/serializer.c):

    • write_buffer_reset() initializes padded_next = NULL.

    • serializer_prepare_for_array() computes:

      self->buffer.padded_next = self->buffer.next + length_size + 
    (self->state->max_array_len * self->state->element_size);

    • serializer_prepare_for_buffer_write() is called before every write operation to advance self->buffer.next to self->buffer.padded_next.

  • Deserializer (apx/src/deserializer.c):

    • deserializer_prepare_for_array() sets padded_next = next + (max_array_len * element_size).

    • deserializer_prepare_for_buffer_read() jumps self->buffer.next = self->buffer.padded_next.

4.2 C++ Implementation (cpp-apx)

  • Serializer (apx/src/serializer.cpp):

    • Serializer::prepare_for_array() sets m_buffer.padded_next = m_buffer.next + length_size + (m_state->max_array_len * m_state->element_size);

    • Serializer::prepare_for_buffer_write() (line 1485):

      /*
 * If more data follows after a dynamic array write we must move the write pointer to
 * the first byte after the dynamic array. Otherwise elements after the dynamic array
 * will start move around in the memory map.
 */
apx::error_t Serializer::prepare_for_buffer_write()
{
   if (!is_valid_buffer())
   {
      return APX_MISSING_BUFFER_ERROR;
   }
   if (m_buffer.padded_next != nullptr)
   {
      if ((m_buffer.padded_next < m_buffer.begin) || (m_buffer.padded_next > m_buffer.end))
      {
         return APX_BUFFER_BOUNDARY_ERROR;
      }
      m_buffer.next = m_buffer.padded_next;
      m_buffer.padded_next = nullptr;
   }
   return APX_NO_ERROR;
}

  • Deserializer (apx/src/deserializer.cpp):

    • Deserializer::prepare_for_array() and Deserializer::prepare_for_buffer_read() enforce the same symmetric padding logic for reading.

4.3 Python Implementation Guidelines (py-apx)

  • In py-apx/src/apx/data/serializer.py:

    • WriteBuffer tracks padded_write_pos.

    • prepare_for_buffer_write() must be called before packing any new value, advancing write_pos = padded_write_pos whenever padded_write_pos is not None.

  • In py-apx/src/apx/data/deserializer.py:

    • ReadBuffer tracks padded_read_pos.

    • prepare_for_buffer_read() synchronizes read_pos = padded_read_pos before subsequent unpack operations.

5. Quick Reference Summary Table

Data Structure

Length Header

Inter-field Alignment

Unused Capacity Handling

Buffer Stride / Predictability

Scalar Primitives (UINT8..INT64, BOOL)

None

0 bytes

N/A

Fixed width (1, 2, 4, 8 bytes)

Fixed Array (T[N])

None

0 bytes

N/A

Fixed width (\(N \times \text{elem\_size}\))

Fixed String (CHAR[N] / CHAR8[N])

None

0 bytes

Null-padded (0x00) in place

Fixed width (\(N\) bytes)

Dynamic Array (T[<=N])

1, 2, or 4 bytes

0 bytes

Slot padding via padded_next

Deterministic slot (\(\text{header} + N \times \text{elem\_size}\))

Dynamic String (CHAR[<=N])

1, 2, or 4 bytes

0 bytes

Slot padding via padded_next

Deterministic slot (\(\text{header} + N\) bytes)

Record / Struct

None

0 bytes

Embedded dynamic fields padded via padded_next

Fixed record size (sum of max field sizes)

Array of Records (RECORD[N])

None

0 bytes

Dynamic fields inside records padded

Fixed array size (\(N \times \text{MaxRecordSize}\))

Queued Port

1, 2, or 4 bytes

0 bytes

Tail elements unused in queue buffer

Deterministic port size (\(\text{header} + N_{\text{queue}} \times \text{elem\_size}\))