> For the complete documentation index, see [llms.txt](https://quantumwing.gitbook.io/quantumwing/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://quantumwing.gitbook.io/quantumwing/architecture/canonical-encoding.md).

# Canonical Encoding

Canonical binary encoding for deterministic Dilithium signatures

## Overview

QuantumWing uses **canonical binary encoding** for cryptographic signatures, ensuring **deterministic** Dilithium Mode 3 signatures across all validators.

{% hint style="success" %}
**Status**: ✅ **Production Ready** (June 2026)\
**Tests**: 14/14 passing regression tests\
**Size**: 3409 bytes deterministic format
{% endhint %}

## Why Canonical Encoding?

### The Problem with JSON

**Previously**, validators signed JSON-serialized blocks:

```json
{
  "slot": 42,
  "proposer_index": 0,
  "parent_hash": "0x1234...",
  "state_root": "0x5678..."
}
```

**Issues**:

* ❌ JSON key order is non-deterministic
* ❌ Whitespace varies between implementations
* ❌ Floating-point precision differs
* ❌ Unicode normalization problems
* ❌ Different validators produced **different signatures** for **same block**

### The Solution: Binary Canonical Format

**Current implementation** uses binary-encoded blocks:

```
[32 bytes ParentHash][8 bytes Slot][8 bytes ProposerIndex]
[32 bytes StateRoot][32 bytes BodyRoot][3293 bytes VRFProof]
= 3409 bytes (deterministic)
```

**Benefits**:

* ✅ **Deterministic**: Same input always produces same bytes
* ✅ **Fast**: \~2.5 μs encoding time (1000x faster than JSON)
* ✅ **Compact**: 3409 bytes (vs 5KB+ JSON)
* ✅ **Secure**: No parser ambiguities

## Architecture

### Wire Format vs Canonical Format

QuantumWing separates **transport** from **signing**:

```mermaid
graph LR
    A[Block Creation] --> B[Wire Format<br/>JSON/Protobuf]
    B --> C[Network Transport]
    C --> D[Canonical Format<br/>Binary]
    D --> E[Sign with<br/>Dilithium]
    E --> F[Verify<br/>Signature]
```

**Wire Format** (JSON/Protobuf):

* Used for: REST API, P2P gossip, storage
* Pros: Human-readable, extensible
* Cons: Non-deterministic for signatures

**Canonical Format** (Binary):

* Used for: Signing, verification **ONLY**
* Pros: Deterministic, fast, secure
* Cons: Not human-readable

{% hint style="warning" %}
**NEVER sign JSON directly** - always convert to canonical format first!
{% endhint %}

## Implementation

### SignedHeader Structure

`blockchain/types/signed_header.go`:

```go
type SignedHeader struct {
    ParentHash    [32]byte // Parent block hash
    Slot          uint64   // Slot number
    ProposerIndex uint64   // Validator index
    StateRoot     [32]byte // State merkle root
    BodyRoot      [32]byte // Transactions merkle root
    VRFProof      []byte   // VRF proof (3293 bytes)
}
```

### CanonicalBytes() Method

```go
func (sh *SignedHeader) CanonicalBytes() []byte {
    if len(sh.VRFProof) != VRF_PROOF_LEN {
        panic(fmt.Sprintf("VRF proof must be %d bytes", VRF_PROOF_LEN))
    }

    buf := make([]byte, CANONICAL_HEADER_SIZE) // 3409 bytes
    offset := 0

    // 1. ParentHash (32 bytes)
    copy(buf[offset:], sh.ParentHash[:])
    offset += 32

    // 2. Slot (8 bytes, big-endian)
    binary.BigEndian.PutUint64(buf[offset:], sh.Slot)
    offset += 8

    // 3. ProposerIndex (8 bytes, big-endian)
    binary.BigEndian.PutUint64(buf[offset:], sh.ProposerIndex)
    offset += 8

    // 4. StateRoot (32 bytes)
    copy(buf[offset:], sh.StateRoot[:])
    offset += 32

    // 5. BodyRoot (32 bytes)
    copy(buf[offset:], sh.BodyRoot[:])
    offset += 32

    // 6. VRFProof (3293 bytes)
    copy(buf[offset:], sh.VRFProof)

    return buf
}
```

### Domain-Separated Hashing

```go
func (sh *SignedHeader) Hash(chainID string) [32]byte {
    // 1. Calculate domain separator
    domainString := fmt.Sprintf("QUANTUMWING_BLOCK_V1|%s", chainID)
    domain := sha3.Sum256([]byte(domainString))

    // 2. Get canonical bytes
    canonicalBytes := sh.CanonicalBytes()

    // 3. Hash: Keccak256(domain || canonicalBytes)
    combined := append(domain[:], canonicalBytes...)
    return sha3.Sum256(combined)
}
```

**Domain separation prevents**:

* Cross-chain replay attacks
* Protocol version confusion
* Signature reuse across contexts

## Signing Flow

### Step-by-Step Process

**1. Create Block (Wire Format)**:

```go
block := &Block{
    Header: &BlockHeader{
        ParentHash: previousHash,
        Slot:       currentSlot,
        // ... JSON fields
    },
}
```

**2. Generate VRF Proof**:

```go
vrfProof, err := validator.keyPair.VRFProve(slotSeed)
// vrfProof is 3293 bytes (Dilithium Mode 3)
block.Header.VRFProof = vrfProof
```

**3. Convert to Canonical Format**:

```go
signedHeader := &SignedHeader{
    ParentHash:    block.Header.ParentHash,
    Slot:          block.Header.Slot,
    ProposerIndex: validatorIndex,
    StateRoot:     stateRoot,
    BodyRoot:      bodyRoot,
    VRFProof:      vrfProof,
}
```

**4. Sign Canonical Hash**:

```go
canonicalHash := signedHeader.Hash("quantum-production-1")
signature, err := validator.keyPair.Sign(canonicalHash[:])
// signature is 3293 bytes (Dilithium Mode 3)
```

**5. Attach Signature to Block**:

```go
block.Header.Signature = signature
// Now broadcast block via JSON/Protobuf
```

### Verification Flow

**1. Receive Block (Wire Format)**:

```go
block := receiveBlockFromNetwork()
```

**2. Reconstruct SignedHeader**:

```go
signedHeader := &SignedHeader{
    ParentHash:    block.Header.ParentHash,
    Slot:          block.Header.Slot,
    ProposerIndex: block.Header.ProposerIndex,
    StateRoot:     block.Header.StateRoot,
    BodyRoot:      block.Header.BodyRoot,
    VRFProof:      block.Header.VRFProof,
}
```

**3. Verify Signature**:

```go
canonicalHash := signedHeader.Hash("quantum-production-1")
valid := validator.PublicKey.Verify(canonicalHash[:], block.Header.Signature)
```

## Testing

### Determinism Tests

`blockchain/types/signed_header_test.go`:

```go
func TestCanonicalBytesDeterminism(t *testing.T) {
    sh := &SignedHeader{
        ParentHash:    [32]byte{1, 2, 3},
        Slot:          42,
        ProposerIndex: 0,
        StateRoot:     [32]byte{4, 5, 6},
        BodyRoot:      [32]byte{7, 8, 9},
        VRFProof:      make([]byte, 3293),
    }

    // Generate bytes 3 times
    bytes1 := sh.CanonicalBytes()
    bytes2 := sh.CanonicalBytes()
    bytes3 := sh.CanonicalBytes()

    // All must be identical
    assert.Equal(t, bytes1, bytes2)
    assert.Equal(t, bytes2, bytes3)
}
```

### Size Tests

```go
func TestCanonicalBytesSize(t *testing.T) {
    sh := createValidSignedHeader()
    bytes := sh.CanonicalBytes()
    
    assert.Equal(t, 3409, len(bytes), "Must be exactly 3409 bytes")
}
```

### Multi-Node Consistency

```bash
# Start 3-node testnet
./scripts/start-3node-dht-testnet.sh

# Extract canonical hashes
for node in node1 node2 node3; do
  grep "CANONICAL SIGNATURE" logs/$node/beacon/beacon.log | head -5
done
```

**Expected output**:

```
node1: canonical_hash: 0xabcd1234... (slot 0, validator 0)
node2: canonical_hash: 0xabcd1234... (slot 0, validator 0)  ✅ MATCH
node3: canonical_hash: 0xabcd1234... (slot 0, validator 0)  ✅ MATCH
```

## Performance

| Operation            | Time     | Notes                  |
| -------------------- | -------- | ---------------------- |
| **CanonicalBytes()** | 2.5 μs   | 1000x faster than JSON |
| **Hash()**           | 8 μs     | SHA3-256 on 3441 bytes |
| **Sign()**           | 1.2 ms   | Dilithium Mode 3       |
| **Verify()**         | 0.9 ms   | Dilithium Mode 3       |
| **Total**            | \~2.1 ms | Per block signature    |

**At 12s block time**: 0.02% of time spent on signing/verification.

## Security Properties

### Immutability Guarantees

{% hint style="danger" %}
**CRITICAL**: Never mutate SignedHeader fields after signature generation!
{% endhint %}

```go
// ❌ WRONG - mutating after signature
signedHeader := createSignedHeader()
signature := sign(signedHeader.Hash(chainID))
signedHeader.Slot++ // BUG: Signature now invalid!

// ✅ CORRECT - freeze after signing
signedHeader := createSignedHeader()
signedHeader.validate() // Panic if invalid
signature := sign(signedHeader.Hash(chainID))
// No mutations allowed
```

### Length Validation

```go
func (sh *SignedHeader) CanonicalBytes() []byte {
    if len(sh.VRFProof) != 3293 {
        panic("VRF proof must be exactly 3293 bytes")
    }
    // Fail-fast instead of silent corruption
}
```

### Domain Separation

**Protocol version bump** (future):

```go
// V1 (current)
domain := "QUANTUMWING_BLOCK_V1|quantum-production-1"

// V2 (after protocol upgrade)
domain := "QUANTUMWING_BLOCK_V2|quantum-production-1"
```

Prevents:

* Old validators signing new blocks
* Signature replay across protocol versions

## Production Deployment

### Verification Checklist

Before deploying, ensure:

* [ ] All validators use **same chain ID** (`quantum-production-1`)
* [ ] All validators have **same genesis** hash
* [ ] VRF proofs are **exactly 3293 bytes**
* [ ] Signatures are **exactly 3293 bytes**
* [ ] Canonical hashes **match across nodes** for same slot
* [ ] No JSON signing code remains

### Monitoring

```bash
# Check canonical signature generation
grep "CANONICAL SIGNATURE GENERATED" logs/validators/*.log

# Expected output:
# validator-0: slot: 0, signature_len: 3293, canonical_hash: 0xabcd...
# validator-1: slot: 4, signature_len: 3293, canonical_hash: 0x1234...
```

### Troubleshooting

**Problem**: "Signature verification failed"

**Possible causes**:

1. Different chain IDs between nodes
2. VRF proof length != 3293 bytes
3. JSON signing instead of canonical
4. Mutated SignedHeader after signing

**Solution**:

```bash
# 1. Check chain ID in all logs
grep "chain_id" logs/beacon/*.log

# 2. Check VRF proof length
grep "VRF proof length" logs/validators/*.log

# 3. Verify canonical bytes hex dump (first 64 chars)
grep "canonical_hex" logs/validators/*.log
```

## References

* **Implementation**: `blockchain/types/signed_header.go`
* **Tests**: `blockchain/types/signed_header_test.go`
* **Dilithium Spec**: [NIST FIPS 204](https://csrc.nist.gov/pubs/fips/204/final)
* **SHA3 Spec**: [NIST FIPS 202](https://csrc.nist.gov/pubs/fips/202/final)

## Next Steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>📝 Transaction Format</strong></td><td>How transactions use canonical encoding</td><td><a href="/quantumwing/transactions/format.md">Transaction Format</a></td></tr><tr><td><strong>🔐 Quantum-Safe Crypto</strong></td><td>Dilithium Mode 3 signing</td><td><a href="/quantumwing/cryptography/quantum-safe-crypto.md">Quantum-Safe Crypto</a></td></tr><tr><td><strong>🧪 Testing Guide</strong></td><td>Verify canonical encoding in your network</td><td><a href="/quantumwing/development/testing.md">Testing</a></td></tr></tbody></table>


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://quantumwing.gitbook.io/quantumwing/architecture/canonical-encoding.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `build a script that syncs our docs to a CMS` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.