diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 09bc48dc..00000000 --- a/CLAUDE.md +++ /dev/null @@ -1,592 +0,0 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## Project Overview - -VaultLink is a self-hosted Obsidian plugin for real-time collaborative file syncing. The project consists of a Rust-based sync server and a TypeScript frontend with four main components: an Obsidian plugin, a sync client library, a test client, and a standalone CLI client. - -## Architecture - -### Core Components - -- **sync-server/**: Rust-based WebSocket server with SQLite database for document versioning and real-time synchronization -- **frontend/sync-client/**: TypeScript library providing core sync functionality, WebSocket management, and file operations -- **frontend/obsidian-plugin/**: Obsidian plugin that integrates the sync client with Obsidian's API -- **frontend/test-client/**: CLI testing tool for simulating multiple concurrent users -- **frontend/local-client-cli/**: Standalone CLI for VaultLink sync client -- **frontend/history-ui/**: Svelte 5 web UI for browsing vault history, viewing diffs, and restoring versions - -### Key Technologies - -- **Backend**: Rust with Axum framework, SQLite with SQLx, WebSockets for real-time sync -- **Frontend**: TypeScript, Webpack for bundling, Node.js native test runner -- **History UI**: Svelte 5 with runes, Vite for bundling, embedded in server binary via `rust-embed` -- **Sync Algorithm**: Uses reconcile-text library for operational transformation - -### Architectural Patterns - -**Server Architecture:** - -- `AppState`: Central state container holding `Database`, `Cursors`, and `Broadcasts` -- `Database`: SQLite-backed document versioning with SQLx for compile-time query verification -- `Broadcasts`: WebSocket broadcast system for real-time updates to connected clients -- `Cursors`: Tracks user cursor positions across documents with background cleanup task - -**Client Architecture (Serial Event Queue Model):** - -- `SyncClient`: Main entry point, orchestrates all sync operations -- `SyncService`: HTTP API client for CRUD operations on documents -- `WebSocketManager`: Manages WebSocket connection and real-time updates -- `Syncer`: Coordinates file synchronisation via a serial drain loop over a `SyncEventQueue` -- `SyncEventQueue`: Intent queue that coalesces events and tracks path→documentId mappings -- `CursorTracker`: Manages local and remote cursor positions -- `Database`: Client-side document metadata cache (persisted via `PersistenceProvider`) -- `FileOperations`: Abstraction layer for filesystem operations (3-way merge on write) - -**Dual-Bundle Strategy:** -The sync-client builds two separate bundles: - -- `sync-client.web.js`: Browser-compatible UMD bundle (excludes `ws` package) -- `sync-client.node.js`: Node.js CommonJS bundle with WebSocket support - -**History UI Architecture:** - -The history UI (`frontend/history-ui/`) is a standalone Svelte 5 SPA that provides read-only vault history browsing. It communicates with the server via the same REST API used by sync clients, plus three additional endpoints: - -- `GET /vaults/:vault_id/documents/:document_id/versions` — all versions of a document (without content) -- `GET /vaults/:vault_id/history?limit=&before_update_id=` — paginated vault-wide version history (cursor-based) -- `POST /vaults/:vault_id/documents/:document_id/restore` — restore a document to a historical version (creates a new version with old content) - -Server-side implementation: -- Database methods: `get_document_versions()` and `get_vault_history()` in `database.rs`, plus a `VaultHistoryRow` helper struct for `sqlx::query_as!` -- Handlers: `fetch_document_versions.rs`, `fetch_vault_history.rs`, `restore_document_version.rs` -- Response type: `VaultHistoryResponse { versions, hasMore }` in `responses.rs` -- SPA serving: `rust-embed` embeds `frontend/history-ui/dist/` into the binary; `index.rs` serves the SPA at `/` and assets at `/assets/*` - -Client-side component hierarchy: -- `App.svelte` — session restore, routing -- `Login.svelte` — vault name + token auth via `/ping` -- `Dashboard.svelte` — main layout: file tree sidebar, activity feed, time-travel slider -- `DocumentDetail.svelte` — version timeline, content preview, diff view, restore -- `DiffView.svelte` — unified diff with LCS algorithm -- `FileTree.svelte` — recursive tree built from flat `relativePath` values -- `ActivityFeed.svelte` — git-log-style feed with action pills (created/updated/renamed/deleted/restored) -- `TimeSlider.svelte` — scrubs through `vaultUpdateId` range, reconstructs vault state at any point - -State is managed with Svelte 5 runes (`$state`, `$derived`, `$effect`) in `lib/stores.svelte.ts`. Auth is stored in `sessionStorage`. The API client (`lib/api.ts`) sets `Authorization: Bearer` and `device-id: history-ui` headers on all requests. - -## Development Commands - -### Initial Setup - -**Node.js (requires version 25):** - -```bash -curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash -nvm install 25 -nvm use 25 -nvm alias default 25 # Optional: set as system default -``` - -**Rust:** - -```bash -curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh -cargo install sqlx-cli cargo-machete cargo-edit cargo-insta -``` - -**Frontend:** - -```bash -cd frontend -npm install -``` - -### Server Development - -```bash -cd sync-server -cargo run config-e2e.yml # Start development server -cargo test --verbose # Run all Rust tests -cargo test # Run specific test -cargo clippy --all-targets --all-features # Lint Rust code -cargo clippy --all-targets --all-features --fix --allow-dirty --allow-staged # Auto-fix clippy warnings -cargo fmt --all -- --check # Check Rust formatting -cargo fmt --all # Auto-format Rust code -cargo machete --with-metadata # Detect unused dependencies -``` - -### Frontend Development - -```bash -cd frontend -npm run dev # Start development mode (watches sync-client and obsidian-plugin) -npm run build # Build all workspaces -npm run build -w sync-client # Build specific workspace -npm run test # Run all tests across all workspaces -npm run test -w sync-client # Run tests for specific workspace -npm run lint # Lint and format TypeScript code with ESLint + Prettier -``` - -### History UI Development - -```bash -cd frontend -npm run dev -w history-ui # Start Vite dev server (localhost:5173, proxies API to localhost:3000) -npm run build -w history-ui # Build for production (output: frontend/history-ui/dist/) -``` - -The history UI is a Svelte 5 SPA embedded in the server binary via `rust-embed`. The build flow is: - -1. `npm run build -w history-ui` produces `frontend/history-ui/dist/` -2. The Rust server embeds these files at compile time (`sync-server/src/server/index.rs`) -3. The server serves `index.html` at `GET /` and static assets at `GET /assets/*` -4. If the dist directory doesn't exist at Rust compile time, `build.rs` creates a placeholder - -During development, run the Vite dev server separately and use its proxy to forward API calls to the running sync server. - -### Database Operations - -```bash -cd sync-server -# Create/reset database for development -rm -rf db.sqlite* -sqlx database create --database-url sqlite://db.sqlite3 -sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3 -cargo sqlx prepare --workspace - -# Add new migration -sqlx migrate add --source src/app_state/database/migrations -sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3 -``` - -### Project Scripts - -- `scripts/check.sh`: Full CI check (builds, lints, tests both server and frontend). **Run before pushing.** -- `scripts/check.sh --fix`: Same as above but auto-fixes linting and formatting issues -- `scripts/e2e.sh`: End-to-end testing (e.g., `scripts/e2e.sh 8` for 8 concurrent clients) -- `scripts/clean-up.sh`: Clean logs and database files -- `scripts/bump-version.sh patch`: Publish new version (options: patch, minor, major) -- `scripts/update-api-types.sh`: Update TypeScript bindings from Rust types (uses ts-rs) - -## Code Structure - -### Workspace Configuration - -The frontend uses npm workspaces with five packages: - -- `sync-client`: Core synchronization logic (builds dual bundles for web and Node.js) -- `obsidian-plugin`: Obsidian-specific integration -- `test-client`: Testing utilities for E2E tests -- `local-client-cli`: Standalone CLI for VaultLink sync client -- `history-ui`: Svelte 5 SPA for vault history browsing (built with Vite, embedded in server binary) - -### Type Generation and API Updates - -Rust structs generate TypeScript types via ts-rs crate: - -1. Rust structs annotated with `#[derive(TS)]` export to `sync-server/bindings/` -2. Run `scripts/update-api-types.sh` to copy bindings to `frontend/sync-client/src/services/types/` -3. Frontend imports these types for type-safe API communication - -### Important Implementation Details - -**SQLx Compile-Time Verification:** - -- SQLx verifies SQL queries at compile time against the database schema -- Run `cargo sqlx prepare --workspace` after schema changes to update `.sqlx/` directory -- CI builds require prepared query metadata to avoid needing a live database - -## Testing - -### Running Tests - -**Server:** - -```bash -cargo test --verbose # All tests -cargo test # Specific test -``` - -**Frontend:** - -```bash -npm run test # All workspaces -npm run test -w sync-client # Specific workspace -``` - -**E2E:** - -```bash -scripts/e2e.sh 8 # 8 concurrent clients -scripts/clean-up.sh # Clean up after tests -``` - -### Test Structure - -- **Rust**: Unit tests alongside source files, uses `cargo-insta` for snapshot testing -- **TypeScript**: `.test.ts` files using Node.js native test runner (not Jest) -- **E2E**: Uses `test-client` to simulate multiple concurrent users with random operations -- **Deterministic**: Step-by-step sync scenario tests in `frontend/deterministic-tests/` - -### Deterministic Tests (`frontend/deterministic-tests/`) - -Controlled, step-by-step sync scenario tests that exercise specific edge cases. Each test defines a sequence of operations (create, update, rename, delete, enable/disable sync, pause/resume server) and asserts convergence across multiple agents. - -**Running:** - -```bash -cd frontend/deterministic-tests -npx webpack --config webpack.config.js # Build (required after changes) -node dist/cli.js # Run all tests -node dist/cli.js --filter=write-write # Run tests matching a name/key -``` - -Requires the server binary at `sync-server/target/release/sync_server` and `sync-server/config-e2e.yml`. The harness starts/stops servers automatically. - -**Architecture:** - -- `DeterministicAgent` extends `InMemoryFileSystem` — wraps a real `SyncClient` with an in-memory filesystem -- `TestRunner` executes `TestStep[]` sequentially, manages agent lifecycle -- `ServerControl` manages server processes (start/stop/SIGSTOP/SIGCONT) -- Tests that use `pause-server`/`resume-server` get dedicated server instances; regular tests share one -- Each test gets a unique vault name (UUID) for isolation - -**Writing Tests — Step Types:** - -```typescript -{ type: "create", client: 0, path: "A.md", content: "hello" } -{ type: "update", client: 0, path: "A.md", content: "updated" } -{ type: "rename", client: 0, oldPath: "A.md", newPath: "B.md" } -{ type: "delete", client: 0, path: "A.md" } -{ type: "enable-sync", client: 0 } // Connects WS, triggers reconciliation -{ type: "disable-sync", client: 0 } // Disconnects WS -{ type: "sync", client: 0 } // Wait for specific client to settle -{ type: "sync" } // Wait for ALL clients to settle -{ type: "barrier" } // Wait for convergence + check consistency -{ type: "pause-server" } // SIGSTOP the server process -{ type: "resume-server" } // SIGCONT + wait for readiness -{ type: "assert-consistent", verify?: (state: AssertableState) => void } -``` - -**Critical Rules When Writing Tests:** - -1. **Agents start with sync DISABLED.** Do not `disable-sync` on an agent that hasn't been `enable-sync`'d — it's already off. - -2. **Do not put `{ type: "sync" }` before `{ type: "barrier" }`.** The barrier already calls `waitAllAgentsSettled()` (2 rounds of `waitForSync` on all agents). Adding a `sync` before it is pure redundancy. Use targeted `{ type: "sync", client: N }` only when you need a specific client to finish before another client acts. - -3. **`enable-sync` blocks until WebSocket connects.** If the server is paused (SIGSTOP), `enable-sync` will hang for 10 seconds then fail. Never `enable-sync` while the server is paused. Tests that need to stall in-flight requests should enable sync FIRST, then pause the server. - -4. **File operations while sync is disabled are queued.** When `createFile` is called on the agent, `enqueueSync(syncLocallyCreatedFile)` fires immediately but the fetch is disabled. The `scheduleSyncForOfflineChanges` reconciliation scans the filesystem and re-enqueues all pending changes on the next `enable-sync`. - -5. **`barrier` retries for up to 60 seconds.** It calls `waitAllAgentsSettled`, checks consistency, and if clients disagree, sleeps 500ms and retries. Tests that need more settling time should add targeted `sync` steps before the barrier (e.g., `{ type: "sync", client: 0 }` to ensure client 0's operations complete first). - -6. **No comments in test files.** The test name/description and step types are self-documenting. Keep test files comment-free. - -7. **Keep tests minimal.** Each test should reproduce exactly one edge case with the fewest steps possible. Don't add `assert-consistent` after `barrier` unless it has a `verify` callback (barrier already checks consistency). Always use inline arrow functions for `verify` callbacks rather than separate named functions. - -8. **Treat sync as a black box in test names/descriptions.** Don't reference internal implementation details (VFS, coalescing, idempotency keys, reconciliation, parentVersionId, etc.). Describe the observable scenario and expected outcome from the user's perspective. - -**Test Patterns for Common Edge Cases:** - -*Two clients create at same path (offline):* -```typescript -steps: [ - { type: "create", client: 0, path: "A.md", content: "hello" }, - { type: "create", client: 1, path: "A.md", content: "world" }, - { type: "enable-sync", client: 0 }, - { type: "enable-sync", client: 1 }, - { type: "barrier" }, - { type: "assert-consistent", verify: verifyMergedContent } -] -``` - -*Client edits while other client is offline:* -```typescript -steps: [ - { type: "create", client: 0, path: "A.md", content: "original" }, - { type: "enable-sync", client: 0 }, - { type: "enable-sync", client: 1 }, - { type: "barrier" }, - // Client 1 goes offline, client 0 edits - { type: "disable-sync", client: 1 }, - { type: "update", client: 0, path: "A.md", content: "edited" }, - { type: "sync", client: 0 }, - // Client 1 reconnects - { type: "enable-sync", client: 1 }, - { type: "barrier" }, - { type: "assert-consistent" } -] -``` - -*Testing behavior during server pause (stalled HTTP requests):* -```typescript -steps: [ - // Setup FIRST — both clients must be online before pausing - { type: "enable-sync", client: 0 }, - { type: "enable-sync", client: 1 }, - { type: "barrier" }, - // NOW pause — in-flight requests from subsequent operations will stall - { type: "pause-server" }, - { type: "create", client: 0, path: "A.md", content: "hello" }, - { type: "resume-server" }, - { type: "barrier" }, - { type: "assert-consistent" } -] -``` - -**Verify Functions and `AssertableState`:** - -The `verify` callback on `assert-consistent` receives an `AssertableState` object (defined in `utils/assertable-state.ts`) with chainable assertion methods: - -```typescript -state.assertFileCount(2) // exact file count -state.assertFileExists("A.md") // file must exist -state.assertFileNotExists("old.md") // file must not exist -state.assertContent("A.md", "hello") // exact content match -state.assertContains("A.md", "hello", "world") // all substrings present -state.assertContainsAny("A.md", "hello", "world") // at least one substring -state.assertAnyFileContains("content-a") // substring in any file -state.assertSubstringCount("A.md", "hello", 1) // occurrence count -state.assertContentInAtMostOneFile("original") // no duplicate content -state.ifFileExists("A.md", (s) => ...) // conditional assertion -state.getContent("A.md") // raw content access -``` - -All methods return `this` for chaining. The object also exposes `files` and `clientFiles` for custom logic. - -For conflict-resolution tests where the outcome is genuinely ambiguous (delete vs update, rename ordering), use `ifFileExists`. For merges where both sides MUST be preserved, use `assertContains`. When the empty-parent merge (invariant #15) is involved, word boundaries may be garbled — check for fragments, not exact substrings. - -```typescript -function verify(state: AssertableState): void { - state.ifFileExists("A.md", (s) => s.assertContent("A.md", "expected content")); -} - -function verify(state: AssertableState): void { - state.assertContains("A.md", "edit from 0", "edit from 1"); -} -``` - -**Adding a New Test:** - -1. Create `frontend/deterministic-tests/src/tests/your-test-name.test.ts` -2. Export a `TestDefinition` with `clients` and `steps` (the test name is derived from the registry key) -3. Import and register in `test-registry.ts` -4. Build with `npx webpack --config webpack.config.js` -5. Run with `node dist/cli.js --filter=your-test-name` - -**Known Limitations:** - -- Cannot test VFS.move failures — the in-memory filesystem never fails -- Cannot `enable-sync` while the server is paused — the WebSocket connection will time out -- The empty-parent 3-way merge (used for smart creates) can produce garbled word boundaries — check for fragments, not exact substrings -- The test harness can hang during shared server cleanup when transitioning to server-pause tests - -## Code Style and Formatting - -### Rust - -- Extensive Clippy lints (see `Cargo.toml`) -- Pedantic linting rules enabled -- Forbids unsafe code -- Uses `rustfmt.toml` for formatting configuration (4 spaces, Unix line endings) -- Run `cargo fmt --all` to format - -### TypeScript - -- **Prettier**: 4-space indentation, no trailing commas, LF line endings -- **YAML/Markdown override**: 2-space indentation (via prettier config) -- **ESLint**: Strict rules with unused imports detection -- Configuration in `frontend/package.json` -- Run `npm run lint` to format and fix issues - -### Svelte (History UI) - -- Uses Svelte 5 runes syntax (`$state`, `$derived`, `$effect`, `$props`) -- Vite as bundler with `@sveltejs/vite-plugin-svelte` -- Excluded from the main ESLint config (Svelte files need different linting); `history-ui/**` is in the eslint ignores list -- CSS is component-scoped via Svelte's `