# Canonical Encoding

## Overview

QuantumWing uses **canonical binary encoding** for cryptographic signatures, ensuring **deterministic** Dilithium Mode 3 signatures across all validators.

{% hint style="success" %}
**Status**: ✅ **Production Ready** (October 2025)\
**Tests**: 14/14 passing regression tests\
**Size**: 3409 bytes deterministic format
{% endhint %}

## Why Canonical Encoding?

### The Problem with JSON

**Previously**, validators signed JSON-serialized blocks:

```json
{
  "slot": 42,
  "proposer_index": 0,
  "parent_hash": "0x1234...",
  "state_root": "0x5678..."
}
```

**Issues**:

* ❌ JSON key order is non-deterministic
* ❌ Whitespace varies between implementations
* ❌ Floating-point precision differs
* ❌ Unicode normalization problems
* ❌ Different validators produced **different signatures** for **same block**

### The Solution: Binary Canonical Format

**Current implementation** uses binary-encoded blocks:

```
[32 bytes ParentHash][8 bytes Slot][8 bytes ProposerIndex]
[32 bytes StateRoot][32 bytes BodyRoot][3293 bytes VRFProof]
= 3409 bytes (deterministic)
```

**Benefits**:

* ✅ **Deterministic**: Same input always produces same bytes
* ✅ **Fast**: \~2.5 μs encoding time (1000x faster than JSON)
* ✅ **Compact**: 3409 bytes (vs 5KB+ JSON)
* ✅ **Secure**: No parser ambiguities

## Architecture

### Wire Format vs Canonical Format

QuantumWing separates **transport** from **signing**:

```mermaid
graph LR
    A[Block Creation] --> B[Wire Format<br/>JSON/Protobuf]
    B --> C[Network Transport]
    C --> D[Canonical Format<br/>Binary]
    D --> E[Sign with<br/>Dilithium]
    E --> F[Verify<br/>Signature]
```

**Wire Format** (JSON/Protobuf):

* Used for: REST API, P2P gossip, storage
* Pros: Human-readable, extensible
* Cons: Non-deterministic for signatures

**Canonical Format** (Binary):

* Used for: Signing, verification **ONLY**
* Pros: Deterministic, fast, secure
* Cons: Not human-readable

{% hint style="warning" %}
**NEVER sign JSON directly** - always convert to canonical format first!
{% endhint %}

## Implementation

### SignedHeader Structure

`blockchain/types/signed_header.go`:

```go
type SignedHeader struct {
    ParentHash    [32]byte // Parent block hash
    Slot          uint64   // Slot number
    ProposerIndex uint64   // Validator index
    StateRoot     [32]byte // State merkle root
    BodyRoot      [32]byte // Transactions merkle root
    VRFProof      []byte   // VRF proof (3293 bytes)
}
```

### CanonicalBytes() Method

```go
func (sh *SignedHeader) CanonicalBytes() []byte {
    if len(sh.VRFProof) != VRF_PROOF_LEN {
        panic(fmt.Sprintf("VRF proof must be %d bytes", VRF_PROOF_LEN))
    }

    buf := make([]byte, CANONICAL_HEADER_SIZE) // 3409 bytes
    offset := 0

    // 1. ParentHash (32 bytes)
    copy(buf[offset:], sh.ParentHash[:])
    offset += 32

    // 2. Slot (8 bytes, big-endian)
    binary.BigEndian.PutUint64(buf[offset:], sh.Slot)
    offset += 8

    // 3. ProposerIndex (8 bytes, big-endian)
    binary.BigEndian.PutUint64(buf[offset:], sh.ProposerIndex)
    offset += 8

    // 4. StateRoot (32 bytes)
    copy(buf[offset:], sh.StateRoot[:])
    offset += 32

    // 5. BodyRoot (32 bytes)
    copy(buf[offset:], sh.BodyRoot[:])
    offset += 32

    // 6. VRFProof (3293 bytes)
    copy(buf[offset:], sh.VRFProof)

    return buf
}
```

### Domain-Separated Hashing

```go
func (sh *SignedHeader) Hash(chainID string) [32]byte {
    // 1. Calculate domain separator
    domainString := fmt.Sprintf("QUANTUMWING_BLOCK_V1|%s", chainID)
    domain := sha3.Sum256([]byte(domainString))

    // 2. Get canonical bytes
    canonicalBytes := sh.CanonicalBytes()

    // 3. Hash: Keccak256(domain || canonicalBytes)
    combined := append(domain[:], canonicalBytes...)
    return sha3.Sum256(combined)
}
```

**Domain separation prevents**:

* Cross-chain replay attacks
* Protocol version confusion
* Signature reuse across contexts

## Signing Flow

### Step-by-Step Process

**1. Create Block (Wire Format)**:

```go
block := &Block{
    Header: &BlockHeader{
        ParentHash: previousHash,
        Slot:       currentSlot,
        // ... JSON fields
    },
}
```

**2. Generate VRF Proof**:

```go
vrfProof, err := validator.keyPair.VRFProve(slotSeed)
// vrfProof is 3293 bytes (Dilithium Mode 3)
block.Header.VRFProof = vrfProof
```

**3. Convert to Canonical Format**:

```go
signedHeader := &SignedHeader{
    ParentHash:    block.Header.ParentHash,
    Slot:          block.Header.Slot,
    ProposerIndex: validatorIndex,
    StateRoot:     stateRoot,
    BodyRoot:      bodyRoot,
    VRFProof:      vrfProof,
}
```

**4. Sign Canonical Hash**:

```go
canonicalHash := signedHeader.Hash("quantum-production-1")
signature, err := validator.keyPair.Sign(canonicalHash[:])
// signature is 3293 bytes (Dilithium Mode 3)
```

**5. Attach Signature to Block**:

```go
block.Header.Signature = signature
// Now broadcast block via JSON/Protobuf
```

### Verification Flow

**1. Receive Block (Wire Format)**:

```go
block := receiveBlockFromNetwork()
```

**2. Reconstruct SignedHeader**:

```go
signedHeader := &SignedHeader{
    ParentHash:    block.Header.ParentHash,
    Slot:          block.Header.Slot,
    ProposerIndex: block.Header.ProposerIndex,
    StateRoot:     block.Header.StateRoot,
    BodyRoot:      block.Header.BodyRoot,
    VRFProof:      block.Header.VRFProof,
}
```

**3. Verify Signature**:

```go
canonicalHash := signedHeader.Hash("quantum-production-1")
valid := validator.PublicKey.Verify(canonicalHash[:], block.Header.Signature)
```

## Testing

### Determinism Tests

`blockchain/types/signed_header_test.go`:

```go
func TestCanonicalBytesDeterminism(t *testing.T) {
    sh := &SignedHeader{
        ParentHash:    [32]byte{1, 2, 3},
        Slot:          42,
        ProposerIndex: 0,
        StateRoot:     [32]byte{4, 5, 6},
        BodyRoot:      [32]byte{7, 8, 9},
        VRFProof:      make([]byte, 3293),
    }

    // Generate bytes 3 times
    bytes1 := sh.CanonicalBytes()
    bytes2 := sh.CanonicalBytes()
    bytes3 := sh.CanonicalBytes()

    // All must be identical
    assert.Equal(t, bytes1, bytes2)
    assert.Equal(t, bytes2, bytes3)
}
```

### Size Tests

```go
func TestCanonicalBytesSize(t *testing.T) {
    sh := createValidSignedHeader()
    bytes := sh.CanonicalBytes()
    
    assert.Equal(t, 3409, len(bytes), "Must be exactly 3409 bytes")
}
```

### Multi-Node Consistency

```bash
# Start 3-node testnet
./scripts/start-3node-dht-testnet.sh

# Extract canonical hashes
for node in node1 node2 node3; do
  grep "CANONICAL SIGNATURE" logs/$node/beacon/beacon.log | head -5
done
```

**Expected output**:

```
node1: canonical_hash: 0xabcd1234... (slot 0, validator 0)
node2: canonical_hash: 0xabcd1234... (slot 0, validator 0)  ✅ MATCH
node3: canonical_hash: 0xabcd1234... (slot 0, validator 0)  ✅ MATCH
```

## Performance

| Operation            | Time     | Notes                  |
| -------------------- | -------- | ---------------------- |
| **CanonicalBytes()** | 2.5 μs   | 1000x faster than JSON |
| **Hash()**           | 8 μs     | SHA3-256 on 3441 bytes |
| **Sign()**           | 1.2 ms   | Dilithium Mode 3       |
| **Verify()**         | 0.9 ms   | Dilithium Mode 3       |
| **Total**            | \~2.1 ms | Per block signature    |

**At 12s block time**: 0.02% of time spent on signing/verification.

## Security Properties

### Immutability Guarantees

{% hint style="danger" %}
**CRITICAL**: Never mutate SignedHeader fields after signature generation!
{% endhint %}

```go
// ❌ WRONG - mutating after signature
signedHeader := createSignedHeader()
signature := sign(signedHeader.Hash(chainID))
signedHeader.Slot++ // BUG: Signature now invalid!

// ✅ CORRECT - freeze after signing
signedHeader := createSignedHeader()
signedHeader.validate() // Panic if invalid
signature := sign(signedHeader.Hash(chainID))
// No mutations allowed
```

### Length Validation

```go
func (sh *SignedHeader) CanonicalBytes() []byte {
    if len(sh.VRFProof) != 3293 {
        panic("VRF proof must be exactly 3293 bytes")
    }
    // Fail-fast instead of silent corruption
}
```

### Domain Separation

**Protocol version bump** (future):

```go
// V1 (current)
domain := "QUANTUMWING_BLOCK_V1|quantum-production-1"

// V2 (after protocol upgrade)
domain := "QUANTUMWING_BLOCK_V2|quantum-production-1"
```

Prevents:

* Old validators signing new blocks
* Signature replay across protocol versions

## Production Deployment

### Verification Checklist

Before deploying, ensure:

* [ ] All validators use **same chain ID** (`quantum-production-1`)
* [ ] All validators have **same genesis** hash
* [ ] VRF proofs are **exactly 3293 bytes**
* [ ] Signatures are **exactly 3293 bytes**
* [ ] Canonical hashes **match across nodes** for same slot
* [ ] No JSON signing code remains

### Monitoring

```bash
# Check canonical signature generation
grep "CANONICAL SIGNATURE GENERATED" logs/validators/*.log

# Expected output:
# validator-0: slot: 0, signature_len: 3293, canonical_hash: 0xabcd...
# validator-1: slot: 4, signature_len: 3293, canonical_hash: 0x1234...
```

### Troubleshooting

**Problem**: "Signature verification failed"

**Possible causes**:

1. Different chain IDs between nodes
2. VRF proof length != 3293 bytes
3. JSON signing instead of canonical
4. Mutated SignedHeader after signing

**Solution**:

```bash
# 1. Check chain ID in all logs
grep "chain_id" logs/beacon/*.log

# 2. Check VRF proof length
grep "VRF proof length" logs/validators/*.log

# 3. Verify canonical bytes hex dump (first 64 chars)
grep "canonical_hex" logs/validators/*.log
```

## References

* **Implementation**: `blockchain/types/signed_header.go`
* **Tests**: `blockchain/types/signed_header_test.go`
* **Dilithium Spec**: [NIST FIPS 204](https://csrc.nist.gov/pubs/fips/204/final)
* **SHA3 Spec**: [NIST FIPS 202](https://csrc.nist.gov/pubs/fips/202/final)

## Next Steps

<table data-view="cards"><thead><tr><th></th><th></th><th data-hidden data-card-target data-type="content-ref"></th></tr></thead><tbody><tr><td><strong>📝 Transaction Format</strong></td><td>How transactions use canonical encoding</td><td><a href="../transactions/format">format</a></td></tr><tr><td><strong>🔐 Quantum-Safe Crypto</strong></td><td>Dilithium Mode 3 signing</td><td><a href="../cryptography/quantum-safe-crypto">quantum-safe-crypto</a></td></tr><tr><td><strong>🧪 Testing Guide</strong></td><td>Verify canonical encoding in your network</td><td><a href="https://github.com/dolfrin/QuantumWing/blob/master/docs/development/testing.md">https://github.com/dolfrin/QuantumWing/blob/master/docs/development/testing.md</a></td></tr></tbody></table>