vault-link/frontend/deterministic-tests

Deterministic Testing Framework

A framework for defining and running deterministic tests for VaultLink sync operations. Unlike the fuzz testing approach, these tests execute exact sequences of operations to verify specific conflict resolution scenarios.

Overview

The deterministic testing framework allows you to:

• Define exact sequences of client operations in TypeScript
• Control both client and server processes (pause/resume)
• Test specific conflict scenarios (write/write, rename/create, etc.)
• Verify that the system resolves conflicts consistently

Architecture

┌─────────────────────────────────────────────┐
│          Test Definition (TypeScript)        │
│   - Declare steps sequentially              │
│   - Specify client operations               │
│   - Add assertions                          │
└──────────────┬──────────────────────────────┘
               │
               v
┌─────────────────────────────────────────────┐
│            Test Runner                       │
│   - Initializes clients                     │
│   - Executes steps in order                 │
│   - Manages server lifecycle                │
└──────────────┬──────────────────────────────┘
               │
               ├─→ DeterministicAgent (per client)
               │    └─→ SyncClient
               │
               └─→ ServerControl
                    └─→ sync_server process

Test Definition Format

Tests are defined using the TestDefinition interface:

interface TestDefinition {
  name: string;
  description?: string;
  clients: number;
  steps: TestStep[];
}

Available Steps

File Operations

{ type: "create", client: 0, path: "file.md", content: "hello" }
{ type: "update", client: 0, path: "file.md", content: "world" }
{ type: "rename", client: 0, oldPath: "A.md", newPath: "B.md" }
{ type: "delete", client: 0, path: "file.md" }

Sync Control

{ type: "sync", client: 0 }        // Wait for specific client
{ type: "sync" }                   // Wait for all clients
{ type: "barrier" }                // Wait for all pending ops
{ type: "disable-sync", client: 0 }
{ type: "enable-sync", client: 0 }

Server Control

{ type: "pause-server" }           // Pause server process
{ type: "resume-server" }          // Resume server process
{ type: "wait", duration: 500 }    // Wait N milliseconds

Assertions

{ type: "assert-content", client: 0, path: "file.md", content: "hello" }
{ type: "assert-exists", client: 0, path: "file.md" }
{ type: "assert-not-exists", client: 0, path: "file.md" }
{ type: "assert-consistent" }      // All clients have same state

Example Tests

Write/Write Conflict

Two clients create the same file with different content:

export const writeWriteConflictTest: TestDefinition = {
  name: "Write/Write Conflict",
  clients: 2,
  steps: [
    { type: "disable-sync", client: 0 },
    { type: "disable-sync", client: 1 },
    { type: "create", client: 0, path: "A.md", content: "hello" },
    { type: "create", client: 1, path: "A.md", content: "world" },
    { type: "enable-sync", client: 0 },
    { type: "enable-sync", client: 1 },
    { type: "barrier" },
    { type: "wait", duration: 500 },
    { type: "barrier" },
    { type: "assert-consistent" }
  ]
};

Rename/Create Conflict

Client 1 renames A→B while Client 0 creates B:

export const renameCreateConflictTest: TestDefinition = {
  name: "Rename-Create Conflict",
  clients: 2,
  steps: [
    { type: "create", client: 0, path: "A.md", content: "hi" },
    { type: "sync", client: 0 },
    { type: "sync", client: 1 },
    { type: "rename", client: 1, oldPath: "A.md", newPath: "B.md" },
    { type: "sync", client: 1 },
    { type: "disable-sync", client: 0 },
    { type: "create", client: 0, path: "B.md", content: "hi" },
    { type: "enable-sync", client: 0 },
    { type: "barrier" },
    { type: "wait", duration: 500 },
    { type: "barrier" },
    { type: "assert-consistent" }
  ]
};

Running Tests

Build and Run

# From frontend/deterministic-tests
npm run test

Run Specific Test

npm run test -- --test write-write-conflict

List Available Tests

npm run test -- --list

Advanced Options

# Use custom server binary
npm run test -- --server /path/to/sync_server

# Use custom config
npm run test -- --config /path/to/config.yml

# Don't manage server (assume it's already running)
npm run test -- --no-manage-server

Creating New Tests

1. Create a new test file in src/tests/:

// my-test.test.ts
import type { TestDefinition } from "../test-definition";

export const myTest: TestDefinition = {
  name: "My Test",
  description: "What this test verifies",
  clients: 2,
  steps: [
    // Your test steps here
  ]
};

2. Register the test in src/cli.ts:

import { myTest } from "./tests/my-test.test";

const TESTS: Record<string, TestDefinition> = {
  // ... existing tests
  "my-test": myTest
};

3. Build and run:

npm run test -- --test my-test

Key Concepts

Synchronization Points

Use explicit sync barriers to ensure operations complete:

• { type: "sync", client: 0 } - Wait for client 0 to finish pending ops
• { type: "barrier" } - Wait for all clients to finish
• { type: "wait", duration: 500 } - Wait for propagation

Offline Testing

Disable sync to simulate offline edits:

{ type: "disable-sync", client: 0 },
{ type: "create", client: 0, path: "file.md", content: "offline edit" },
{ type: "enable-sync", client: 0 },  // Sync when back online

Server Control

Pause the server to test reconnection:

{ type: "pause-server" },
{ type: "create", client: 0, path: "file.md", content: "while paused" },
{ type: "resume-server" },
{ type: "barrier" }

Assertions

Always end tests with consistency checks:

{
  type: "assert-consistent";
} // Verify all clients converged

Troubleshooting

Server Won't Start

• Ensure server is built: cd sync-server && cargo build
• Check config file exists: sync-server/config-e2e.yml
• Verify port 3000 is available

Test Hangs

• Increase wait durations for slow systems
• Add more { type: "barrier" } steps
• Check server logs for errors

Assertion Failures

• Add { type: "wait", duration: 1000 } before assertions
• Check if conflict resolution is working as expected
• Review test steps for logic errors

Comparison to Fuzz Tests

Aspect	Fuzz Tests	Deterministic Tests
Operations	Random	Explicit sequence
Reproducibility	Difficult	Perfect
Coverage	Broad	Targeted
Debugging	Hard	Easy
Use Case	Find edge cases	Verify specific scenarios

Use both approaches:

• Fuzz tests for discovering unexpected issues
• Deterministic tests for verifying specific fixes