Compare commits

...
Sign in to create a new pull request.

10 commits

Author SHA1 Message Date
42c9d55489 split: sync-engine rewrite (sync-operations + sync-client.ts)
Replace the single unrestricted-syncer.ts with a two-loop architecture:
- syncer.ts drains the FIFO wire queue (HTTP + WS handlers).
- reconciler.ts moves files to make localPath match remoteRelativePath
  (topo-sorted move graph, in-memory cycle resolution with crash-safe
  swap markers).
- sync-event-queue.ts holds the byDocId / byLocalPath indexes and the
  pending-create promise chain.
- offline-change-detector.ts, expected-fs-events.ts, types.ts, and a
  rewritten cursor-tracker.ts / file-change-notifier.ts round it out.
Plus sync-client.ts wiring, tracing/sync-history.ts updates, index.ts
re-exports, and sync-client tsconfig/webpack/package.json.
2026-05-08 21:37:26 +01:00
0fda95ff8e split: sync-client file-operations + persistence
Rewrite file-operations and safe-filesystem-operations (and their tests),
update filesystem-operations. Drop persistence/database.ts (in-memory
record store moved into sync-event-queue). Update persistence/settings.ts.
2026-05-08 21:36:54 +01:00
45b86cffe4 split: sync-client services layer
Add build-vault-url helper. Rewrite fetch-controller and websocket-manager
(plus their tests). Update server-config and sync-service to consume the
new error types and the regenerated API types from previous chunks.
2026-05-08 21:36:41 +01:00
9d99a4ac23 split: sync-client utils and errors reorganization
Move error classes from services/ and file-operations/ into a new errors/
directory (authentication-error, server-version-mismatch-error,
sync-reset-error, file-not-found-error), plus add file-already-exists-error
and http-client-error. Update consts.ts and utils/* (await-all,
create-client-id, hash, rate-limit, find-matching-file). Replace
data-structures (locks, min-covered, event-listeners, fix-sized-cache) and
add debugging utilities (in-memory-file-system, log-to-console,
slow-web-socket-factory). Removes utils/create-promise.ts.
2026-05-08 21:36:29 +01:00
f7beb31d8f split: regenerated TS API type bindings
Auto-generated TS types regenerated from Rust ts-rs derives, mirrored into
frontend/sync-client/src/services/types/ and frontend/history-ui/src/lib/types/.
Adds ListVaultsResponse, VaultHistoryResponse, VaultInfo and updates several
existing types; removes DeleteDocumentVersion and UpdateDocumentVersion.
2026-05-08 21:36:13 +01:00
042233c4d7 split: server websocket + cursors
src/server/websocket.rs handshake/catch-up rewrite, app_state/cursors.rs,
app_state/websocket/{broadcasts,models,utils}.rs.
2026-05-08 21:35:52 +01:00
4ba439b874 split: server REST endpoints + rate limiting
server.rs router rewrite, auth.rs, device_id_header.rs, requests.rs,
responses.rs, plus per-endpoint changes: create/update/delete_document,
fetch_document_version{,_content,s}, fetch_latest_documents, index.rs.
Adds: fetch_vault_history, list_vaults, rate_limit (new files).
2026-05-08 21:35:41 +01:00
2d5edc6ec5 split: server database (app_state, migrations, models)
src/app_state.rs, src/app_state/database.rs (large schema/query rewrite),
two new migrations (add_idempotency_key, add_creation_vault_update_id),
and src/app_state/database/models.rs.
2026-05-08 21:35:30 +01:00
a9ce09b59d split: server foundation (Cargo, config, errors, utils, main)
Cargo.{toml,lock} bumps, build.rs, config-e2e.yml, rust-toolchain.toml,
src/config/* (database/logging/server/user configs), src/consts.rs,
src/errors.rs, src/main.rs, and src/utils/* (dedup_paths,
find_first_available_path, rotating_file_writer, sanitize_path).
2026-05-08 21:35:18 +01:00
70f97c4b16 split: CI workflows, scripts, root tooling, and docs
Some checks failed
Check / build (pull_request) Has been cancelled
E2E tests / build (pull_request) Has been cancelled
Publish CLI / publish-docker (pull_request) Has been cancelled
Publish server Docker image / publish-docker (pull_request) Has been cancelled
Forgejo workflows (new), GitHub workflow tweaks, .gitignore/.vscode, root
package-lock, rustfmt.toml, scripts/* updates, docs/ updates including
data-flow / authentication / server-setup, CLAUDE.md and README updates.
2026-05-08 21:35:07 +01:00
156 changed files with 12570 additions and 6287 deletions

View file

@ -0,0 +1,35 @@
name: Check
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
workflow_dispatch:
env:
CARGO_TERM_COLOR: always
RUSTFLAGS: "-Dwarnings"
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup Node.js environment
uses: actions/setup-node@v4
with:
node-version: "25.x"
- name: Setup Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: "1.92.0"
components: clippy, rustfmt
- name: Lint & test
run: scripts/check.sh

View file

@ -0,0 +1,38 @@
name: Deploy Documentation
on:
push:
branches:
- main
paths:
- "docs/**"
- ".forgejo/workflows/deploy-docs.yml"
workflow_dispatch:
concurrency:
group: pages
cancel-in-progress: false
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup Node.js environment
uses: actions/setup-node@v4
with:
node-version: "25.x"
- name: Build docs
run: scripts/build-docs.sh
- name: Upload artifact
uses: actions/upload-artifact@v4
with:
name: docs
path: docs/.vitepress/dist

View file

@ -0,0 +1,71 @@
name: E2E tests
on:
push:
branches: ["main"]
pull_request:
branches: ["main"]
schedule:
- cron: "0 * * * *"
workflow_dispatch:
concurrency:
group: e2e-tests
cancel-in-progress: false
env:
RUSTFLAGS: "-Dwarnings"
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup Node.js environment
uses: actions/setup-node@v4
with:
node-version: "25.x"
- name: Setup Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: "1.92.0"
components: clippy, rustfmt
- name: Setup rust
run: |
which sqlx || cargo install sqlx-cli
cd sync-server
sqlx database create --database-url sqlite://db.sqlite3
sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3
- name: E2E tests
run: |
cd sync-server
cargo run config-e2e.yml --color never &
SERVER_PID=$!
cd ..
scripts/e2e.sh 8
EXIT_CODE=$?
kill $SERVER_PID 2>/dev/null || true
wait $SERVER_PID 2>/dev/null || true
exit $EXIT_CODE
- name: Upload e2e logs
if: always()
uses: actions/upload-artifact@v4
with:
name: e2e-logs
path: logs/
retention-days: 30
- name: Cleanup
if: always()
run: scripts/clean-up.sh

View file

@ -0,0 +1,51 @@
name: Publish CLI
on:
push:
branches: ["main"]
tags: ["*"]
pull_request:
branches: ["main"]
jobs:
publish-docker:
runs-on: ubuntu-docker
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Extract registry hostname
id: registry
run: echo "host=$(echo '${{ github.server_url }}' | sed 's|https\?://||')" >> "$GITHUB_OUTPUT"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log into container registry
uses: docker/login-action@v3
with:
registry: ${{ steps.registry.outputs.host }}
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ steps.registry.outputs.host }}/${{ github.repository }}-cli
- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@v5
with:
context: frontend
file: frontend/local-client-cli/Dockerfile
platforms: linux/amd64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=registry,ref=${{ steps.registry.outputs.host }}/${{ github.repository }}-cli:buildcache
cache-to: type=registry,ref=${{ steps.registry.outputs.host }}/${{ github.repository }}-cli:buildcache,mode=max

View file

@ -0,0 +1,71 @@
name: Publish Obsidian plugin
on:
push:
tags: ["*"]
env:
CARGO_TERM_COLOR: always
jobs:
publish-plugin:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Setup Node.js environment
uses: actions/setup-node@v4
with:
node-version: "25.x"
- name: Build plugin
run: |
cd frontend
npm ci
npm run build
- name: Setup Rust toolchain
uses: dtolnay/rust-toolchain@stable
with:
toolchain: "1.92.0"
components: clippy, rustfmt
- name: Install cross-compilation tools
run: |
apt update
apt install -y gcc-aarch64-linux-gnu musl-tools gcc-mingw-w64-x86-64 jq
- name: Build Linux and Windows binaries
run: ./scripts/build-sync-server-binaries.sh
- name: Create release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SERVER_URL: ${{ github.server_url }}
REPO: ${{ github.repository }}
run: |
tag="${GITHUB_REF#refs/tags/}"
mkdir -p release
cp frontend/obsidian-plugin/dist/* release/
cp sync-server/artifacts/sync-server-* release/
# Create draft release via Forgejo API
RELEASE_ID=$(curl -s -X POST \
"${SERVER_URL}/api/v1/repos/${REPO}/releases" \
-H "Authorization: token ${GITHUB_TOKEN}" \
-H "Content-Type: application/json" \
-d "{\"tag_name\": \"${tag}\", \"name\": \"${tag}\", \"draft\": true}" \
| jq -r '.id')
# Upload release assets
for file in release/*; do
filename=$(basename "$file")
curl -s -X POST \
"${SERVER_URL}/api/v1/repos/${REPO}/releases/${RELEASE_ID}/assets?name=${filename}" \
-H "Authorization: token ${GITHUB_TOKEN}" \
-F "attachment=@${file}"
done

View file

@ -0,0 +1,51 @@
name: Publish server Docker image
on:
push:
branches: ["main"]
tags: ["*"]
pull_request:
branches: ["main"]
jobs:
publish-docker:
runs-on: ubuntu-docker
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Extract registry hostname
id: registry
run: echo "host=$(echo '${{ github.server_url }}' | sed 's|https\?://||')" >> "$GITHUB_OUTPUT"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Log into container registry
if: github.ref_type == 'tag'
uses: docker/login-action@v3
with:
registry: ${{ steps.registry.outputs.host }}
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Extract Docker metadata
id: meta
uses: docker/metadata-action@v5
with:
images: ${{ steps.registry.outputs.host }}/${{ github.repository }}
- name: Build and push Docker image
id: build-and-push
uses: docker/build-push-action@v5
with:
context: sync-server
platforms: linux/amd64,linux/arm64
push: ${{ github.ref_type == 'tag' }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
cache-from: type=registry,ref=${{ steps.registry.outputs.host }}/${{ github.repository }}:buildcache
cache-to: type=registry,ref=${{ steps.registry.outputs.host }}/${{ github.repository }}:buildcache,mode=max

View file

@ -23,13 +23,13 @@ jobs:
- name: Setup Node.js environment - name: Setup Node.js environment
uses: actions/setup-node@v4.2.0 uses: actions/setup-node@v4.2.0
with: with:
node-version: "22.x" node-version: "25.x"
check-latest: true check-latest: true
- name: Setup Rust toolchain - name: Setup Rust toolchain
uses: dtolnay/rust-toolchain@stable uses: dtolnay/rust-toolchain@stable
with: with:
toolchain: "1.89.0" toolchain: "1.92.0"
components: clippy, rustfmt components: clippy, rustfmt
- name: Lint & test - name: Lint & test

View file

@ -5,8 +5,8 @@ on:
branches: branches:
- main - main
paths: paths:
- 'docs/**' - "docs/**"
- '.github/workflows/deploy-docs.yml' - ".github/workflows/deploy-docs.yml"
workflow_dispatch: workflow_dispatch:
permissions: permissions:
@ -28,12 +28,11 @@ jobs:
with: with:
fetch-depth: 0 fetch-depth: 0
- name: Setup Node - name: Setup Node.js environment
uses: actions/setup-node@v4 uses: actions/setup-node@v4.2.0
with: with:
node-version: 22 node-version: "25.x"
cache: npm check-latest: true
cache-dependency-path: docs/package-lock.json
- name: Setup Pages - name: Setup Pages
uses: actions/configure-pages@v4 uses: actions/configure-pages@v4

View file

@ -6,7 +6,7 @@ on:
pull_request: pull_request:
branches: ["main"] branches: ["main"]
schedule: schedule:
- cron: '0 * * * *' - cron: "0 * * * *"
workflow_dispatch: workflow_dispatch:
concurrency: concurrency:
@ -28,13 +28,13 @@ jobs:
- name: Setup Node.js environment - name: Setup Node.js environment
uses: actions/setup-node@v4.2.0 uses: actions/setup-node@v4.2.0
with: with:
node-version: "22.x" node-version: "25.x"
check-latest: true check-latest: true
- name: Setup Rust toolchain - name: Setup Rust toolchain
uses: dtolnay/rust-toolchain@stable uses: dtolnay/rust-toolchain@stable
with: with:
toolchain: "1.89.0" toolchain: "1.92.0"
components: clippy, rustfmt components: clippy, rustfmt
- name: Setup rust - name: Setup rust

View file

@ -19,7 +19,7 @@ jobs:
- name: Setup Node.js environment - name: Setup Node.js environment
uses: actions/setup-node@v4.2.0 uses: actions/setup-node@v4.2.0
with: with:
node-version: "22.x" node-version: "25.x"
check-latest: true check-latest: true
- name: Build plugin - name: Build plugin
@ -31,7 +31,7 @@ jobs:
- name: Setup Rust toolchain - name: Setup Rust toolchain
uses: dtolnay/rust-toolchain@stable uses: dtolnay/rust-toolchain@stable
with: with:
toolchain: "1.89.0" toolchain: "1.92.0"
components: clippy, rustfmt components: clippy, rustfmt
- name: Install cross-compilation tools - name: Install cross-compilation tools

9
.gitignore vendored
View file

@ -7,15 +7,18 @@ node_modules
# Frontend build folders # Frontend build folders
frontend/*/dist frontend/*/dist
sync-server/db.sqlite3*
sync-server/databases
# Rust build folders # Rust build folders
sync-server/target sync-server/target
sync-server/artifacts sync-server/artifacts
sync-server/bindings/*.ts sync-server/bindings/*.ts
# build folders
sync-server/db.sqlite3*
**/databases
*.log *.log
*.sqlx *.sqlx
target target
.task

View file

@ -5,6 +5,6 @@
"**/dist": true, "**/dist": true,
"**/node_modules": true, "**/node_modules": true,
"**/.sqlx": true, "**/.sqlx": true,
"**/target": true, "**/target": true
}, }
} }

195
CLAUDE.md
View file

@ -2,109 +2,154 @@
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview ## Project shape
VaultLink is a self-hosted Obsidian plugin for real-time collaborative file syncing. The project consists of a Rust-based sync server and a TypeScript frontend with three main components: an Obsidian plugin, a sync client library, and a test client. VaultLink is a self-hosted Obsidian file-sync system. Two halves of one repo:
## Architecture - `sync-server/` — Rust (axum + sqlx/SQLite). Source of truth for vault state, broadcasts changes via WebSocket.
- `frontend/` — npm workspaces. The sync engine (`sync-client`) is consumed by an Obsidian plugin, a standalone CLI, a fuzz E2E harness, a scripted determinism harness, and a history UI.
### Core Components The HTTP/WS API types are generated from Rust (`ts-rs`) and mirrored into the TS workspaces. **Never hand-edit files in `frontend/sync-client/src/services/types/` or `frontend/history-ui/src/lib/types/`** — run `scripts/update-api-types.sh` after changing anything Serde-derived in the server.
- **sync-server/**: Rust-based WebSocket server with SQLite database for document versioning and real-time synchronization ### Frontend workspaces
- **frontend/sync-client/**: TypeScript library providing core sync functionality, WebSocket management, and file operations
- **frontend/obsidian-plugin/**: Obsidian plugin that integrates the sync client with Obsidian's API
- **frontend/test-client/**: CLI testing tool for the sync functionality
### Key Technologies - `sync-client` — the sync engine; published to consumers via `dist/`. All other TS workspaces depend on it via `file:../sync-client`.
- `obsidian-plugin` — Obsidian plugin built from `sync-client`.
- `local-client-cli` — same engine wrapped as a standalone CLI.
- `history-ui` — vault-history web UI.
- `test-client` — fuzz E2E harness (random ops across N processes).
- `deterministic-tests` — scripted multi-client tests with an in-memory FS, run against a real server.
- **Backend**: Rust with Axum framework, SQLite with SQLx, WebSockets for real-time sync ## Common commands
- **Frontend**: TypeScript, Webpack for bundling, Jest for testing
- **Sync Algorithm**: Uses reconcile-text library for operational transformation
## Development Commands Pre-push hygiene (formats, lints, runs tests, requires clean git state):
### Server Development ```sh
```bash scripts/check.sh --fix
cd sync-server
cargo run config-e2e.yml # Start development server
cargo test --verbose # Run Rust tests
cargo clippy --all-targets --all-features # Lint Rust code
cargo clippy --all-targets --all-features --fix --allow-dirty --allow-staged # Auto-fix clippy warnings
cargo fmt --all -- --check # Check Rust formatting
cargo fmt --all # Auto-format Rust code
cargo machete --with-metadata # Detect unused dependencies
``` ```
### Frontend Development Run the fuzz E2E (N parallel processes):
```bash
```sh
scripts/e2e.sh 12
# Logs land in logs/log_<i>.log. Clean with scripts/clean-up.sh
```
Run deterministic tests (require a release-built server in `sync-server/target/release/sync_server` — they spawn it themselves):
```sh
cd sync-server && cargo build --release && cd ..
cd frontend cd frontend
npm run dev # Start development mode (watches sync-client and obsidian-plugin) npm run build -w sync-client -w deterministic-tests
npm run build # Build all workspaces node deterministic-tests/dist/cli.js # all
npm run test # Run all tests node deterministic-tests/dist/cli.js --filter=rename # subset
npm run lint # Lint and format TypeScript code node deterministic-tests/dist/cli.js --filter=… -j 4 # cap parallelism
``` ```
### Database Setup (Development) Run a single sync-client unit test by file:
```bash
```sh
cd frontend/sync-client && npx tsx --test 'src/**/sync-event-queue.test.ts'
```
Server: dev runs from `sync-server/` against `config-e2e.yml`:
```sh
cd sync-server
cargo run config-e2e.yml # dev
cargo build --release # used by both e2e harnesses
cargo test # unit + ts-rs binding export tests
```
Frontend dev (sync-client + obsidian-plugin watch in parallel):
```sh
cd frontend && npm install && npm run dev
```
Regenerate TS bindings from Rust types (touches `frontend/{sync-client,history-ui}/src/.../types/`):
```sh
scripts/update-api-types.sh
```
## SQLite / sqlx
The server uses `sqlx::query!` macros that need a prepared `.sqlx` cache to compile offline. Touching any SQL means regenerating it:
```sh
cd sync-server cd sync-server
sqlx database create --database-url sqlite://db.sqlite3 sqlx database create --database-url sqlite://db.sqlite3
sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3 sqlx migrate run --source src/app_state/database/migrations --database-url sqlite://db.sqlite3
cargo sqlx prepare --workspace cargo sqlx prepare --workspace
``` ```
### Initial Setup New migrations: `sqlx migrate add --source src/app_state/database/migrations <name>`.
```bash
# Install required cargo tools ## Sync engine architecture
cargo install sqlx-cli cargo-machete cargo-edit
Read `frontend/sync-client/src/sync-operations/` to follow the sync engine; the rest of `sync-client` is plumbing (filesystem ops, persistence, services, telemetry).
The engine is **two independent loops with separate invariants**:
- **Wire loop** (`syncer.ts`) — drains the single-consumer FIFO queue. HTTP and WS handlers update record fields (`remoteRelativePath`, `parentVersionId`, `remoteHash`) and write content to the file at `record.localPath`. They never move files for path placement.
- **Path reconciler** (`reconciler.ts`) — runs after every drained event. Best-effort pass that moves files to make `localPath === remoteRelativePath`. The move graph is topologically sorted; cycles are resolved by reading every file in the cycle into memory and writing each back to its new slot (no tmp files). Records with pending local events are skipped on each pass — the reconciler operates only on settled records. Failures (slot occupied by an untracked file, etc.) are silent skips; the next pass retries.
**`SyncEventQueue`** (`sync-event-queue.ts`) holds:
- `byDocId: Map<DocumentId, DocumentRecord>` — primary record store.
- `byLocalPath: Map<RelativePath, DocumentRecord>` — derived index for path lookups, maintained at every mutation point.
- `events: SyncEvent[]` — pending wire ops in FIFO drain order.
```ts
DocumentRecord = {
documentId,
parentVersionId,
remoteHash?,
remoteRelativePath,
localPath: RelativePath | undefined
}
``` ```
### Scripts `localPath === undefined` means the doc has no local file yet — typically a remote create whose target slot was occupied at receive time; the reconciler will fetch and place when the slot frees (the bytes wait in `pendingPlacementContent`).
- `scripts/check.sh`: Full CI check (builds, lints, tests both server and frontend)
- `scripts/check.sh --fix`: Same as above but auto-fixes linting and formatting issues
- `scripts/e2e.sh`: End-to-end testing
- `scripts/clean-up.sh`: Clean logs and database files
- `scripts/bump-version.sh patch`: Publish new version
- `scripts/update-api-types.sh`: Update TypeScript bindings from Rust types
## Code Structure Local FS events from the watcher update `localPath` synchronously at enqueue time via `setLocalPath` / `upsertRecord`. The wire loop never updates it for path placement; only the reconciler does. A user rename onto a tracked slot enqueues a `LocalDelete` for the displaced doc (the OS rename clobbered its content) and clears that doc's `localPath`.
### Workspace Configuration **Pending creates** use a `Promise<DocumentId>` chain to serialize dependent ops (`LocalUpdate`, `LocalDelete`) behind the still-in-flight `LocalCreate`. `resolveCreate` resolves the promise once the server returns a docId, and `replacePendingDocumentId` swaps the resolved id across already-queued events. `findLatestCreateForPath` is the lookup the watcher uses to attach dependents; `updatePendingCreatePath` rewrites a pending create's `event.path` in place when the user renames the file before its create has acked.
The frontend uses npm workspaces with four packages:
- `sync-client`: Core synchronization logic
- `obsidian-plugin`: Obsidian-specific integration
- `test-client`: Testing utilities
- `local-client-cli`: Standalone CLI for VaultLink sync client
### Type Generation **Watermark.** `lastSeenUpdateId` uses a `MinCovered` (a contiguous-prefix tracker over a stream of integers): we only advance the published min when the next consecutive id has been processed, so out-of-order RemoteChange ids don't fool the WebSocket handshake into requesting a too-recent catch-up.
Rust structs generate TypeScript types via ts-rs crate, stored in `sync-server/bindings/` and used by frontend packages.
### Key Files **Server catch-up.** The server's WS handshake replays events newer than the client's `last_seen_vault_update_id` from the `latest_document_versions` view (one row per doc, the latest). On those replayed rows `is_new_file` means _new to this client_ (`creation_vault_update_id > last_seen_vault_update_id`), not "this row is the doc's first version" — necessary because the catch-up only carries the latest version; if a doc was created and updated past the watermark, the client never sees its create otherwise.
- `sync-server/src/`: Rust server implementation with WebSocket handlers
- `frontend/sync-client/src/sync-client.ts`: Main sync client entry point
- `frontend/obsidian-plugin/src/vault-link-plugin.ts`: Main Obsidian plugin class
- `frontend/sync-client/src/services/sync-service.ts`: Core synchronization logic
## Testing ## Edge-case patterns the sync engine has to survive
### Running Tests The two-loop split defuses most of the old race catalogue (slot-collision stashes, conflict-uuid divergence, `MoveOnConflict.NEW`/`EXISTING` policy choices) by separating wire transport from path placement. What's left:
- Server: `cargo test --verbose`
- Frontend: `npm run test` (runs Jest across all workspaces)
- E2E: `scripts/e2e.sh`
### Test Structure **Pending-create docId is a `Promise`, not a string, until the create acks.** Any `LocalUpdate` / `LocalDelete` queued behind a still-in-flight `LocalCreate` carries the create's `resolvers.promise` as its `documentId`. `replacePendingDocumentId` swaps the resolved id across queued events when the create resolves; `===` comparisons against the resolved string elsewhere will silently fail until that swap runs. Anything that walks `events[]` looking for a docId match must either run after the swap or be tolerant of `Promise`-typed ids.
- Rust: Unit tests alongside source files
- TypeScript: `.test.ts` files using Jest
- E2E: Uses test-client to simulate multiple concurrent users
## Code Style **`processCreate` reads `event.path` live, not `event.originalPath`.** The watcher rewrites `event.path` in place via `updatePendingCreatePath` when the user renames a pending-create file. `originalPath` was removed from `LocalCreate` events specifically because reading it would send the stale pre-rename path to the server.
### Rust **`record.localPath` mutates in place across awaits.** When the watcher renames a doc while a drain handler is awaiting an HTTP roundtrip, the queue mutates the in-flight event's record so subsequent reads see the new path. Snapshotting `record.localPath` into a local at function entry and using it after an `await` reads/writes a now-vacated slot. Read `record.localPath` live; only snapshot for the deliberate "did it change while I was awaiting" comparison.
- Uses extensive Clippy lints (see Cargo.toml)
- Follows pedantic linting rules
- Forbids unsafe code
- Uses cargo fmt with default settings
### TypeScript **Reconciler-defer is the wire-loop's contract with the reconciler.** The reconciler skips records where `hasPendingLocalEventsForDocumentId` returns true. Wire-loop handlers can therefore freely write `remoteRelativePath` to whatever the server returned — even if it disagrees with `localPath` — knowing the reconciler won't move the file out from under a queued user rename.
- Prettier configuration: 4-space tabs, trailing commas removed, LF line endings
- ESLint with unused imports plugin **Watermark advancement is load-bearing both ways.** Branches that _skip_ a remote event without advancing `lastSeenUpdateId` create permanent gaps that re-deliver forever. Branches that _advance_ without applying the content lose data: the server has no further event to re-deliver, the catch-up only carries the latest version, and any state in between is gone. Don't advance unless the event was actually applied (or deliberately discarded after weighing both halves).
- Consistent across all three frontend packages
**`isNewFile` semantics differ between catch-up and real-time.** On WS handshake replay it means _new to this client_ (`creation_vault_update_id > last_seen_vault_update_id`); on real-time broadcasts it means _this version is the create_ (`creation_vault_update_id == vault_update_id`). A handler that decides based on one interpretation will be wrong on the other channel; reasoning about fetch-and-treat-as-new vs. ignore needs to know which channel delivered the event.
**Pause / disable-sync mid-flight** is the one race the new model doesn't structurally fix. An HTTP that committed server-side but whose response was discarded leaves the server holding a doc the client has no record of. Resume → offline scan → server-side dedupe handles it (the server merges the duplicate create into the existing doc), but if the merge produces a deconflict, the client picks up an extra file. Out of scope for the two-loop split.
**Cycle reconciliation uses in-memory content swap.** When the move graph contains a cycle, the reconciler reads every file in the cycle into memory and writes each back to its new slot, with no tmp files. A write-ahead marker at `.vaultlink/swap-<uuid>.json` lists each leg; on startup the reconciler reads the marker, hashes each `from` to determine which legs ran, and replays the rest. The `.vaultlink/**` glob is hard-coded as an internal ignore pattern so swap markers don't get sync'd.
## Two complementary E2E harnesses
- **`test-client` (fuzz):** random ops across N parallel processes for many minutes. Used by `scripts/e2e.sh`. Catches bugs nobody thought to write a test for, but reproductions are noisy.
- **`deterministic-tests`:** scripted scenarios with an in-memory FS pinned to a real server. Used to _capture_ a fuzz-discovered bug as a minimal repro before fixing it. See `frontend/deterministic-tests/README.md` for the step grammar (`pause-server`, `pause-websocket`, `barrier`, `assert-consistent`, etc.).
When a fuzz failure surfaces, the workflow is: root-cause from logs → write a deterministic test that fails on the bug → fix → confirm both the deterministic test and `e2e.sh` pass.
## Style
- TS: 4-space indent, no tabs, LF, prettier (`trailingComma: "none"`). YAML/MD use 2-space indent.
- Rust: `rustfmt.toml` enforces 4-space spaces, LF.
- Lint: ESLint for TS, Clippy for Rust, `cargo machete` for unused deps. All wired into `scripts/check.sh`.

View file

@ -8,12 +8,12 @@
## Develop ## Develop
### Install [nvm](https://github.com/nvm-sh/nvm) ### Set up Node.JS 25 with [nvm](https://github.com/nvm-sh/nvm)
- `curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash` - `curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash`
- `nvm install 22` - `nvm install 25`
- `nvm use 22` - `nvm use 25`
- Optionally set the system-wide default: `nvm alias default 22` - Optionally, set the system-wide default: `nvm alias default 25`
### Set up Rust ### Set up Rust

View file

@ -2,12 +2,7 @@
"version": "0.2", "version": "0.2",
"language": "en-GB", "language": "en-GB",
"dictionaries": ["en-gb"], "dictionaries": ["en-gb"],
"ignorePaths": [ "ignorePaths": ["node_modules", ".vitepress/dist", ".vitepress/cache", "package-lock.json"],
"node_modules",
".vitepress/dist",
".vitepress/cache",
"package-lock.json"
],
"words": [ "words": [
"VaultLink", "VaultLink",
"Obsidian", "Obsidian",

View file

@ -53,7 +53,7 @@ Central authority for synchronisation. Rust + Axum framework.
**Technology**: **Technology**:
- **Language**: Rust 1.89+ - **Language**: Rust 1.92+
- **Framework**: Axum (async web framework) - **Framework**: Axum (async web framework)
- **Database**: SQLite with SQLx - **Database**: SQLite with SQLx
- **Protocol**: WebSockets for real-time communication - **Protocol**: WebSockets for real-time communication

View file

@ -75,7 +75,7 @@ chmod +x sync_server-linux-x86_64
### Build from Source ### Build from Source
Requirements: Rust 1.89.0+, SQLite development headers, SQLx CLI Requirements: Rust 1.92.0+, SQLite development headers, SQLx CLI
```bash ```bash
# Clone the repository # Clone the repository

View file

@ -0,0 +1,8 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { DocumentWithCursors } from "./DocumentWithCursors";
export type ClientCursors = {
userName: string;
deviceId: string;
documentsWithCursors: Array<DocumentWithCursors>;
};

View file

@ -1,7 +1,7 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export interface UpdateDocumentVersion { export type CreateDocumentVersion = {
parent_version_id: bigint;
relative_path: string; relative_path: string;
content: number[]; last_seen_vault_update_id: number;
} content: Array<number>;
};

View file

@ -0,0 +1,6 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { DocumentWithCursors } from "./DocumentWithCursors";
export type CursorPositionFromClient = {
documentsWithCursors: Array<DocumentWithCursors>;
};

View file

@ -0,0 +1,4 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { ClientCursors } from "./ClientCursors";
export type CursorPositionFromServer = { clients: Array<ClientCursors> };

View file

@ -1,5 +1,3 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export interface DeleteDocumentVersion { export type CursorSpan = { start: number; end: number };
relativePath: string;
}

View file

@ -0,0 +1,10 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { DocumentVersion } from "./DocumentVersion";
import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent";
/**
* Response to a create/update document request.
*/
export type DocumentUpdateResponse =
| ({ type: "FastForwardUpdate" } & DocumentVersionWithoutContent)
| ({ type: "MergingUpdate" } & DocumentVersion);

View file

@ -0,0 +1,12 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export type DocumentVersion = {
vaultUpdateId: number;
documentId: string;
relativePath: string;
updatedDate: string;
contentBase64: string;
isDeleted: boolean;
userId: string;
deviceId: string;
};

View file

@ -0,0 +1,16 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export type DocumentVersionWithoutContent = {
vaultUpdateId: number;
documentId: string;
relativePath: string;
updatedDate: string;
isDeleted: boolean;
userId: string;
deviceId: string;
contentSize: number;
/**
* True iff this is the first version of the document
*/
isNewFile: boolean;
};

View file

@ -0,0 +1,9 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { CursorSpan } from "./CursorSpan";
export type DocumentWithCursors = {
vaultUpdateId: number | null;
documentId: string;
relativePath: string;
cursors: Array<CursorSpan>;
};

View file

@ -0,0 +1,13 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent";
/**
* Response to a fetch latest documents request.
*/
export type FetchLatestDocumentsResponse = {
latestDocuments: Array<DocumentVersionWithoutContent>;
/**
* The update ID of the latest document in the response.
*/
lastUpdateId: bigint;
};

View file

@ -0,0 +1,11 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { VaultInfo } from "./VaultInfo";
/**
* Response to listing vaults accessible to the authenticated user.
*/
export type ListVaultsResponse = {
vaults: Array<VaultInfo>;
hasMore: boolean;
userName: string;
};

View file

@ -0,0 +1,25 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Response to a ping request.
*/
export type PingResponse = {
/**
* Semantic version of the server.
*/
serverVersion: string;
/**
* Whether the client is authenticated based on the sent Authorization
* header.
*/
isAuthenticated: boolean;
/**
* List of file extensions that are allowed to be merged.
*/
mergeableFileExtensions: Array<string>;
/**
* API version ensuring backwards & forwards compatibility between the client
* and server.
*/
supportedApiVersion: number;
};

View file

@ -0,0 +1,7 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export type SerializedError = {
errorType: string;
message: string;
causes: Array<string>;
};

View file

@ -0,0 +1,7 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export type UpdateTextDocumentVersion = {
parentVersionId: number;
relativePath: string | null;
content: Array<number | string>;
};

View file

@ -0,0 +1,10 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent";
/**
* Response to a vault history request (paginated).
*/
export type VaultHistoryResponse = {
versions: Array<DocumentVersionWithoutContent>;
hasMore: boolean;
};

View file

@ -0,0 +1,10 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Summary of a single vault returned by the list-vaults endpoint.
*/
export type VaultInfo = {
name: string;
documentCount: number;
createdAt: string | null;
};

View file

@ -0,0 +1,7 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { CursorPositionFromClient } from "./CursorPositionFromClient";
import type { WebSocketHandshake } from "./WebSocketHandshake";
export type WebSocketClientMessage =
| ({ type: "handshake" } & WebSocketHandshake)
| ({ type: "cursorPositions" } & CursorPositionFromClient);

View file

@ -0,0 +1,7 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export type WebSocketHandshake = {
token: string;
deviceId: string;
lastSeenVaultUpdateId: number | null;
};

View file

@ -0,0 +1,7 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { CursorPositionFromServer } from "./CursorPositionFromServer";
import type { WebSocketVaultUpdate } from "./WebSocketVaultUpdate";
export type WebSocketServerMessage =
| ({ type: "vaultUpdate" } & WebSocketVaultUpdate)
| ({ type: "cursorPositions" } & CursorPositionFromServer);

View file

@ -0,0 +1,4 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent";
export type WebSocketVaultUpdate = { document: DocumentVersionWithoutContent };

View file

@ -14,19 +14,17 @@
}, },
"devDependencies": { "devDependencies": {
"byte-base64": "^1.1.0", "byte-base64": "^1.1.0",
"minimatch": "^10.0.1", "minimatch": "^10.1.1",
"p-queue": "^8.1.0", "p-queue": "^9.0.1",
"reconcile-text": "^0.8.0", "reconcile-text": "^0.8.0",
"uuid": "^13.0.0", "@types/node": "^25.0.2",
"@types/node": "^24.8.1", "ts-loader": "^9.5.4",
"ts-loader": "^9.5.2",
"tslib": "2.8.1", "tslib": "2.8.1",
"tsx": "^4.20.6", "tsx": "^4.21.0",
"typescript": "5.8.3", "typescript": "5.9.3",
"webpack": "^5.99.9", "webpack": "^5.103.0",
"webpack-cli": "^6.0.1", "webpack-cli": "^6.0.1",
"webpack-merge": "^6.0.1", "webpack-merge": "^6.0.1",
"@sentry/browser": "^10.8.0", "@sentry/browser": "^10.30.0"
"ws": "^8.18.3"
} }
} }

View file

@ -1,6 +1,6 @@
export const TIMEOUT_FOR_MERGING_HISTORY_ENTRIES_IN_SECONDS = 60; export const TIMEOUT_FOR_MERGING_HISTORY_ENTRIES_IN_SECONDS = 60;
export const DIFF_CACHE_SIZE_MB = 2;
export const MAX_LOG_MESSAGE_COUNT = 100000; export const MAX_LOG_MESSAGE_COUNT = 100000;
export const MAX_HISTORY_ENTRY_COUNT = 5000; export const MAX_HISTORY_ENTRY_COUNT = 5000;
export const SUPPORTED_API_VERSION = 2; export const SUPPORTED_API_VERSION = 3;
export const WEBSOCKET_DISCONNECT_TIMEOUT_IN_S = 10; export const WEBSOCKET_DISCONNECT_TIMEOUT_IN_SECONDS = 10;
export const WEBSOCKET_CONNECTION_TIMEOUT_IN_SECONDS = 10;

View file

@ -0,0 +1,9 @@
export class FileAlreadyExistsError extends Error {
public constructor(
message: string,
public readonly filePath: string
) {
super(message);
this.name = "FileAlreadyExistsError";
}
}

View file

@ -0,0 +1,9 @@
export class HttpClientError extends Error {
public constructor(
public readonly statusCode: number,
message: string
) {
super(message);
this.name = "HttpClientError";
}
}

View file

@ -1,15 +1,14 @@
import { describe, it } from "node:test"; import { describe, it } from "node:test";
import type { import assert from "node:assert/strict";
Database, import type { RelativePath } from "../sync-operations/types";
DocumentRecord,
RelativePath
} from "../persistence/database";
import { FileOperations } from "./file-operations"; import { FileOperations } from "./file-operations";
import { Logger } from "../tracing/logger"; import { Logger } from "../tracing/logger";
import { assertSetContainsExactly } from "../utils/assert-set-contains-exactly"; import { assertSetContainsExactly } from "../utils/assert-set-contains-exactly";
import type { FileSystemOperations } from "./filesystem-operations"; import type { FileSystemOperations } from "./filesystem-operations";
import type { TextWithCursors } from "reconcile-text"; import type { TextWithCursors } from "reconcile-text";
import type { ServerConfig, ServerConfigData } from "../services/server-config"; import type { ServerConfig, ServerConfigData } from "../services/server-config";
import { ExpectedFsEvents } from "../sync-operations/expected-fs-events";
import { FileAlreadyExistsError } from "../errors/file-already-exists-error";
class MockServerConfig implements Pick<ServerConfig, "getConfig"> { class MockServerConfig implements Pick<ServerConfig, "getConfig"> {
public async getConfig(): Promise<ServerConfigData> { public async getConfig(): Promise<ServerConfigData> {
@ -21,29 +20,13 @@ class MockServerConfig implements Pick<ServerConfig, "getConfig"> {
} }
} }
class MockDatabase implements Partial<Database> {
public getLatestDocumentByRelativePath(
_find: RelativePath
): DocumentRecord | undefined {
// no-op
return undefined;
}
public move(
_oldRelativePath: RelativePath,
_newRelativePath: RelativePath
): void {
// no-op
}
}
class FakeFileSystemOperations implements FileSystemOperations { class FakeFileSystemOperations implements FileSystemOperations {
public readonly names = new Set<string>(); public readonly names = new Set<string>();
public async listFilesRecursively( public async listFilesRecursively(
_root: RelativePath | undefined _root: RelativePath | undefined
): Promise<RelativePath[]> { ): Promise<RelativePath[]> {
return ["file.md"]; return Array.from(this.names);
} }
public async read(_path: RelativePath): Promise<Uint8Array> { public async read(_path: RelativePath): Promise<Uint8Array> {
throw new Error("Method not implemented."); throw new Error("Method not implemented.");
@ -63,17 +46,14 @@ class FakeFileSystemOperations implements FileSystemOperations {
public async getFileSize(_path: RelativePath): Promise<number> { public async getFileSize(_path: RelativePath): Promise<number> {
throw new Error("Method not implemented."); throw new Error("Method not implemented.");
} }
public async getModificationTime(_path: RelativePath): Promise<Date> {
throw new Error("Method not implemented.");
}
public async exists(path: RelativePath): Promise<boolean> { public async exists(path: RelativePath): Promise<boolean> {
return this.names.has(path); return this.names.has(path);
} }
public async createDirectory(_path: RelativePath): Promise<void> { public async createDirectory(_path: RelativePath): Promise<void> {
// this is called but irrelevant for this mock // no-op for the in-memory fake; we only track files
} }
public async delete(_path: RelativePath): Promise<void> { public async delete(path: RelativePath): Promise<void> {
throw new Error("Method not implemented."); this.names.delete(path);
} }
public async rename( public async rename(
oldPath: RelativePath, oldPath: RelativePath,
@ -84,152 +64,92 @@ class FakeFileSystemOperations implements FileSystemOperations {
} }
} }
function makeOps(): {
fs: FakeFileSystemOperations;
ops: FileOperations;
} {
const fs = new FakeFileSystemOperations();
const ops = new FileOperations(
new Logger(),
fs,
new MockServerConfig() as ServerConfig, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
new ExpectedFsEvents()
);
return { fs, ops };
}
describe("File operations", () => { describe("File operations", () => {
it("should deconflict renames", async () => { it("create writes the file at the requested path", async () => {
const fileSystemOperations = new FakeFileSystemOperations(); const { fs, ops } = makeOps();
const fileOperations = new FileOperations(
new Logger(),
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
);
await fileOperations.create("a", new Uint8Array()); const result = await ops.create("a", new Uint8Array());
assertSetContainsExactly(fileSystemOperations.names, "a");
await fileOperations.move("a", "b");
assertSetContainsExactly(fileSystemOperations.names, "b");
await fileOperations.create("c", new Uint8Array()); assertSetContainsExactly(fs.names, "a");
assertSetContainsExactly(fileSystemOperations.names, "b", "c"); assert.equal(result.actualPath, "a");
await fileOperations.move("c", "b");
assertSetContainsExactly(fileSystemOperations.names, "b", "b (1)");
await fileOperations.create("c", new Uint8Array());
await fileOperations.move("c", "b");
assertSetContainsExactly(
fileSystemOperations.names,
"b",
"b (1)",
"b (2)"
);
}); });
it("should deconflict renames with file extension", async () => { it("create throws FileAlreadyExistsError when the path is occupied", async () => {
const fileSystemOperations = new FakeFileSystemOperations(); const { fs, ops } = makeOps();
const fileOperations = new FileOperations(
new Logger(), await ops.create("note.md", new Uint8Array());
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion await assert.rejects(
fileSystemOperations, ops.create("note.md", new Uint8Array()),
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion FileAlreadyExistsError
); );
await fileOperations.create("b.md", new Uint8Array()); // The original file is left intact and no other entries appeared.
await fileOperations.create("c.md", new Uint8Array()); assertSetContainsExactly(fs.names, "note.md");
await fileOperations.move("c.md", "b.md");
assertSetContainsExactly(
fileSystemOperations.names,
"b.md",
"b (1).md"
);
await fileOperations.create("d.md", new Uint8Array());
await fileOperations.move("d.md", "b.md");
assertSetContainsExactly(
fileSystemOperations.names,
"b.md",
"b (1).md",
"b (2).md"
);
await fileOperations.create("file-23.md", new Uint8Array());
await fileOperations.create("file-23 (1).md", new Uint8Array());
await fileOperations.move("file-23.md", "file-23 (1).md");
assertSetContainsExactly(
fileSystemOperations.names,
"b.md",
"b (1).md",
"b (2).md",
"file-23 (1).md",
"file-23 (2).md"
);
}); });
it("should deconflict renames with paths", async () => { it("move to an empty target just renames the file", async () => {
const fileSystemOperations = new FakeFileSystemOperations(); const { fs, ops } = makeOps();
const fileOperations = new FileOperations(
new Logger(),
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
);
await fileOperations.create("a/b.c/d", new Uint8Array()); await ops.create("a", new Uint8Array());
await fileOperations.create("a/b.c/e", new Uint8Array()); assertSetContainsExactly(fs.names, "a");
await fileOperations.move("a/b.c/d", "a/b.c/e");
assertSetContainsExactly( const result = await ops.move("a", "b");
fileSystemOperations.names, assertSetContainsExactly(fs.names, "b");
"a/b.c/e", assert.equal(result.actualPath, "b");
"a/b.c/e (1)"
);
}); });
it("should continue deconfliction from existing number in filename", async () => { it("move with same source and target is a no-op", async () => {
const fileSystemOperations = new FakeFileSystemOperations(); const { fs, ops } = makeOps();
const fileOperations = new FileOperations(
new Logger(),
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
);
await fileOperations.create("document (5).md", new Uint8Array()); await ops.create("a", new Uint8Array());
await fileOperations.create("other.md", new Uint8Array()); const result = await ops.move("a", "a");
await fileOperations.move("other.md", "document (5).md"); assertSetContainsExactly(fs.names, "a");
assertSetContainsExactly( assert.equal(result.actualPath, "a");
fileSystemOperations.names,
"document (5).md",
"document (6).md"
);
await fileOperations.create("another.md", new Uint8Array());
await fileOperations.move("another.md", "document (5).md");
assertSetContainsExactly(
fileSystemOperations.names,
"document (5).md",
"document (6).md",
"document (7).md"
);
}); });
it("should handle dotfiles correctly", async () => { it("move throws FileAlreadyExistsError when the target is occupied", async () => {
const fileSystemOperations = new FakeFileSystemOperations(); const { fs, ops } = makeOps();
const fileOperations = new FileOperations(
new Logger(), await ops.create("source.md", new Uint8Array());
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion await ops.create("dest.md", new Uint8Array());
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion await assert.rejects(
ops.move("source.md", "dest.md"),
FileAlreadyExistsError
); );
await fileOperations.create(".gitignore", new Uint8Array()); // Both files are left intact — no displacement happens.
await fileOperations.create("temp", new Uint8Array()); assertSetContainsExactly(fs.names, "source.md", "dest.md");
await fileOperations.move("temp", ".gitignore"); });
assertSetContainsExactly(
fileSystemOperations.names,
".gitignore",
".gitignore (1)"
);
await fileOperations.create(".config.json", new Uint8Array()); it("create works for nested paths (parent-directory creation)", async () => {
await fileOperations.create("temp2", new Uint8Array()); const { fs, ops } = makeOps();
await fileOperations.move("temp2", ".config.json");
assertSetContainsExactly( await ops.create("a/b.c/d", new Uint8Array());
fileSystemOperations.names, assertSetContainsExactly(fs.names, "a/b.c/d");
".gitignore", });
".gitignore (1)",
".config.json", it("move works for nested target paths (parent-directory creation)", async () => {
".config (1).json" const { fs, ops } = makeOps();
);
await ops.create("source", new Uint8Array());
await ops.move("source", "a/b.c/dest");
assertSetContainsExactly(fs.names, "a/b.c/dest");
}); });
}); });

View file

@ -1,28 +1,40 @@
import type { Logger } from "../tracing/logger"; import type { Logger } from "../tracing/logger";
import type { FileSystemOperations } from "./filesystem-operations"; import type { FileSystemOperations } from "./filesystem-operations";
import type { Database, RelativePath } from "../persistence/database"; import type { RelativePath } from "../sync-operations/types";
import { SafeFileSystemOperations } from "./safe-filesystem-operations"; import { SafeFileSystemOperations } from "./safe-filesystem-operations";
import type { TextWithCursors } from "reconcile-text"; import type { TextWithCursors } from "reconcile-text";
import { reconcile } from "reconcile-text"; import { reconcile } from "reconcile-text";
import { isFileTypeMergable } from "../utils/is-file-type-mergable"; import { isFileTypeMergable } from "../utils/is-file-type-mergable";
import { isBinary } from "../utils/is-binary"; import { isBinary } from "../utils/is-binary";
import type { ServerConfig } from "../services/server-config"; import type { ServerConfig } from "../services/server-config";
import { FileNotFoundError } from "../errors/file-not-found-error";
import { FileAlreadyExistsError } from "../errors/file-already-exists-error";
import type { ExpectedFsEvents } from "../sync-operations/expected-fs-events";
/**
* Outcome of a `move`/`create`. `actualPath` is where the file ended up;
* with the conflict-path machinery removed it is always equal to the
* requested path. The shape is preserved so callers don't all need to
* change.
*/
export interface FileOpResult {
actualPath: RelativePath;
}
export class FileOperations { export class FileOperations {
private static readonly PARENTHESES_REGEX = / \((?<count>\d+)\)$/;
private readonly fs: SafeFileSystemOperations; private readonly fs: SafeFileSystemOperations;
public constructor( public constructor(
private readonly logger: Logger, private readonly logger: Logger,
private readonly database: Database,
fs: FileSystemOperations, fs: FileSystemOperations,
private readonly serverConfig: ServerConfig, private readonly serverConfig: ServerConfig,
private readonly expectedFsEvents: ExpectedFsEvents,
private readonly nativeLineEndings = "\n" private readonly nativeLineEndings = "\n"
) { ) {
this.fs = new SafeFileSystemOperations(fs, logger); this.fs = new SafeFileSystemOperations(fs, logger);
} }
private static getParentDirAndFile( private static getParentDirAndFileName(
path: RelativePath path: RelativePath
): [RelativePath, RelativePath] { ): [RelativePath, RelativePath] {
const pathParts = path.split("/"); const pathParts = path.split("/");
@ -47,33 +59,32 @@ export class FileOperations {
/** /**
* Create a file at the specified path. * Create a file at the specified path.
* *
* If a file with the same name already exists, it is moved before creating the new one. * Throws `FileAlreadyExistsError` if a file already lives at `path`.
* Parent directories are created if necessary. * Parent directories are created if necessary. The reconciler is the
* only caller that places files now and pre-checks for conflicts;
* the throw guards against a TOCTOU race rather than being a normal
* code path.
*/ */
public async create( public async create(
path: RelativePath, path: RelativePath,
newContent: Uint8Array newContent: Uint8Array
): Promise<void> { ): Promise<FileOpResult> {
await this.ensureClearPath(path);
return this.fs.write(path, this.toNativeLineEndings(newContent));
}
public async ensureClearPath(path: RelativePath): Promise<void> {
if (await this.fs.exists(path)) { if (await this.fs.exists(path)) {
const deconflictedPath = await this.deconflictPath(path); throw new FileAlreadyExistsError(
try { `Refusing to create '${path}': file already exists`,
this.logger.debug( path
`Didn't expect ${path} to exist, deconflicting by moving it to '${deconflictedPath}'`
); );
this.database.move(path, deconflictedPath);
await this.fs.rename(path, deconflictedPath, true);
} finally {
this.fs.unlock(deconflictedPath);
} }
} else {
await this.createParentDirectories(path); await this.createParentDirectories(path);
this.expectedFsEvents.expectCreate(path);
try {
await this.fs.write(path, this.toNativeLineEndings(newContent));
} catch (e) {
this.expectedFsEvents.unexpectCreate(path);
throw e;
} }
return { actualPath: path };
} }
/** /**
@ -94,10 +105,18 @@ export class FileOperations {
return; return;
} }
// Single-source the expectation registration: register exactly once
// per call, and unexpect from the catch if the underlying fs op
// throws (FileNotFoundError or otherwise). The previous shape
// registered inside each branch and let the catch swallow
// FileNotFoundError, leaking the expectation into the map.
this.expectedFsEvents.expectUpdate(path);
try {
if ( if (
!isFileTypeMergable( !isFileTypeMergable(
path, path,
(await this.serverConfig.getConfig()).mergeableFileExtensions (await this.serverConfig.getConfig())
.mergeableFileExtensions
) || ) ||
isBinary(expectedContent) || isBinary(expectedContent) ||
isBinary(newContent) isBinary(newContent)
@ -113,8 +132,22 @@ export class FileOperations {
return; return;
} }
const expectedText = new TextDecoder().decode(expectedContent); // this comes from a previous read which must only have \n line endings let expectedText = "";
const newText = new TextDecoder().decode(newContent); // this comes from the server which stores text with \n line endings let newText = "";
try {
expectedText = new TextDecoder("utf-8", { fatal: true }).decode(
expectedContent
); // this comes from a previous read which must only have \n line endings
newText = new TextDecoder("utf-8", { fatal: true }).decode(
newContent
); // this comes from the server which stores text with \n line endings
} catch (decodeError) {
this.logger.warn(
`3-way merge aborted for ${path}: one of expected/new is not valid UTF-8 (${decodeError}); falling back to overwrite`
);
await this.fs.write(path, this.toNativeLineEndings(newContent));
return;
}
await this.fs.atomicUpdateText( await this.fs.atomicUpdateText(
path, path,
@ -141,11 +174,27 @@ export class FileOperations {
}; };
} }
); );
} catch (e) {
this.expectedFsEvents.unexpectUpdate(path);
if (e instanceof FileNotFoundError) {
this.logger.debug(
`File ${path} disappeared during write; not recreating`
);
return;
}
throw e;
}
} }
public async delete(path: RelativePath): Promise<void> { public async delete(path: RelativePath): Promise<void> {
if (await this.exists(path)) { if (await this.exists(path)) {
this.expectedFsEvents.expectDelete(path);
try {
await this.fs.delete(path); await this.fs.delete(path);
} catch (e) {
this.expectedFsEvents.unexpectDelete(path);
throw e;
}
await this.deletingEmptyParentDirectoriesOfDeletedFile(path); await this.deletingEmptyParentDirectoriesOfDeletedFile(path);
} else { } else {
this.logger.debug(`No need to delete '${path}', it doesn't exist`); this.logger.debug(`No need to delete '${path}', it doesn't exist`);
@ -160,23 +209,39 @@ export class FileOperations {
return this.fs.exists(path); return this.fs.exists(path);
} }
/**
* Move the file at `oldPath` to `newPath`.
*
* Throws `FileAlreadyExistsError` if a file already lives at `newPath`
* (and `oldPath !== newPath`). The reconciler is the only caller that
* relocates tracked records and pre-checks for conflicts; the throw
* guards against a TOCTOU race.
*/
public async move( public async move(
oldPath: RelativePath, oldPath: RelativePath,
newPath: RelativePath newPath: RelativePath
): Promise<void> { ): Promise<FileOpResult> {
if (oldPath === newPath) { if (oldPath === newPath) {
return; return { actualPath: oldPath };
} }
await this.ensureClearPath(newPath); if (await this.fs.exists(newPath)) {
throw new FileAlreadyExistsError(
`Refusing to move '${oldPath}' onto '${newPath}': target already exists`,
newPath
);
}
await this.createParentDirectories(newPath);
this.database.move(oldPath, newPath); this.expectedFsEvents.expectRename(oldPath, newPath);
try {
await this.fs.rename(oldPath, newPath); await this.fs.rename(oldPath, newPath);
await this.deletingEmptyParentDirectoriesOfDeletedFile(oldPath); } catch (e) {
this.expectedFsEvents.unexpectRename(oldPath, newPath);
throw e;
} }
await this.deletingEmptyParentDirectoriesOfDeletedFile(oldPath);
public reset(): void { return { actualPath: newPath };
this.fs.reset();
} }
private async deletingEmptyParentDirectoriesOfDeletedFile( private async deletingEmptyParentDirectoriesOfDeletedFile(
@ -185,7 +250,7 @@ export class FileOperations {
let directory = path; let directory = path;
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
while (true) { while (true) {
[directory] = FileOperations.getParentDirAndFile(directory); [directory] = FileOperations.getParentDirAndFileName(directory);
if (directory.length === 0) { if (directory.length === 0) {
break; break;
} }
@ -237,55 +302,4 @@ export class FileOperations {
} }
} }
} }
/**
* Deconflicts the given path by appending (1), (2), etc. before the file extension until a non-existent path is found.
* The returned path has a lock acquired on it; it must be released by the caller when no longer needed.
*
* @param path The starting path to deconflict
* @returns a non-existent path with a lock acquired on it
*/
private async deconflictPath(path: RelativePath): Promise<RelativePath> {
// eslint-disable-next-line prefer-const
let [directory, fileName] = FileOperations.getParentDirAndFile(path);
if (directory) {
directory += "/";
}
const nameParts = fileName.split(".");
// Handle dotfiles: ".gitignore" should have no extension, ".config.json" should have ".json"
const isDotfile = fileName.startsWith(".") && nameParts[0] === "";
const extension =
nameParts.length > 1 && !(isDotfile && nameParts.length === 2)
? "." + nameParts[nameParts.length - 1]
: "";
let stem = extension ? nameParts.slice(0, -1).join(".") : fileName;
let currentCount = Number.parseInt(
FileOperations.PARENTHESES_REGEX.exec(stem)?.groups?.count ?? "0"
);
stem = stem.replace(FileOperations.PARENTHESES_REGEX, "");
let newName = path;
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
while (true) {
currentCount++;
newName = `${directory}${stem} (${currentCount})${extension}`;
// Avoid multiple deconflictPath calls returning the same path
if (this.fs.tryLock(newName)) {
const newDocument =
this.database.getLatestDocumentByRelativePath(newName);
if (
newDocument?.isDeleted === false || // the document might have been confirmed by the server at a new path but haven't yet moved there locally
(await this.fs.exists(newName, true))
) {
this.fs.unlock(newName);
} else {
return newName;
}
}
}
}
} }

View file

@ -1,4 +1,4 @@
import type { RelativePath } from "../persistence/database"; import type { RelativePath } from "../sync-operations/types";
import type { TextWithCursors } from "reconcile-text"; import type { TextWithCursors } from "reconcile-text";

View file

@ -1,24 +1,18 @@
import type { RelativePath } from "../persistence/database"; import type { RelativePath } from "../sync-operations/types";
import type { FileSystemOperations } from "./filesystem-operations"; import type { FileSystemOperations } from "./filesystem-operations";
import type { Logger } from "../tracing/logger"; import type { Logger } from "../tracing/logger";
import { Locks } from "../utils/data-structures/locks"; import { FileNotFoundError } from "../errors/file-not-found-error";
import { FileNotFoundError } from "./file-not-found-error";
import type { TextWithCursors } from "reconcile-text"; import type { TextWithCursors } from "reconcile-text";
/** /**
* Decorates `FileSystemOperations` to replace errors with `FileNotFoundError` * Decorates `FileSystemOperations` to replace errors with `FileNotFoundError`
* if the accessed file doesn't exist. It also ensures that there's at most a * if the accessed file doesn't exist.
* single request in-flight for any one file through the use of locks.
*/ */
export class SafeFileSystemOperations implements FileSystemOperations { export class SafeFileSystemOperations implements FileSystemOperations {
private readonly locks: Locks<RelativePath>;
public constructor( public constructor(
private readonly fs: FileSystemOperations, private readonly fs: FileSystemOperations,
private readonly logger: Logger private readonly logger: Logger
) { ) {}
this.locks = new Locks(logger);
}
public async listFilesRecursively( public async listFilesRecursively(
root: RelativePath | undefined root: RelativePath | undefined
@ -31,19 +25,12 @@ export class SafeFileSystemOperations implements FileSystemOperations {
public async read(path: RelativePath): Promise<Uint8Array> { public async read(path: RelativePath): Promise<Uint8Array> {
this.logger.debug(`Reading file '${path}'`); this.logger.debug(`Reading file '${path}'`);
return this.safeOperation( return this.safeOperation(path, async () => this.fs.read(path), "read");
path,
async () =>
this.locks.withLock(path, async () => this.fs.read(path)),
"read"
);
} }
public async write(path: RelativePath, content: Uint8Array): Promise<void> { public async write(path: RelativePath, content: Uint8Array): Promise<void> {
this.logger.debug(`Writing to file '${path}'`); this.logger.debug(`Writing to file '${path}'`);
return this.locks.withLock(path, async () => return this.fs.write(path, content);
this.fs.write(path, content)
);
} }
public async atomicUpdateText( public async atomicUpdateText(
@ -53,10 +40,7 @@ export class SafeFileSystemOperations implements FileSystemOperations {
this.logger.debug(`Atomically updating file '${path}'`); this.logger.debug(`Atomically updating file '${path}'`);
return this.safeOperation( return this.safeOperation(
path, path,
async () => async () => this.fs.atomicUpdateText(path, updater),
this.locks.withLock(path, async () =>
this.fs.atomicUpdateText(path, updater)
),
"atomicUpdateText" "atomicUpdateText"
); );
} }
@ -65,75 +49,38 @@ export class SafeFileSystemOperations implements FileSystemOperations {
// Logging this would be too noisy // Logging this would be too noisy
return this.safeOperation( return this.safeOperation(
path, path,
async () => async () => this.fs.getFileSize(path),
this.locks.withLock(path, async () =>
this.fs.getFileSize(path)
),
"getFileSize" "getFileSize"
); );
} }
public async exists( public async exists(path: RelativePath): Promise<boolean> {
path: RelativePath,
skipLock = false
): Promise<boolean> {
this.logger.debug(`Checking if file '${path}' exists`); this.logger.debug(`Checking if file '${path}' exists`);
if (skipLock) {
return this.fs.exists(path); return this.fs.exists(path);
} else {
return this.locks.withLock(path, async () => this.fs.exists(path));
}
} }
public async createDirectory(path: RelativePath): Promise<void> { public async createDirectory(path: RelativePath): Promise<void> {
this.logger.debug(`Creating directory '${path}'`); this.logger.debug(`Creating directory '${path}'`);
return this.locks.withLock(path, async () => return this.fs.createDirectory(path);
this.fs.createDirectory(path)
);
} }
public async delete(path: RelativePath): Promise<void> { public async delete(path: RelativePath): Promise<void> {
this.logger.debug(`Deleting file '${path}'`); this.logger.debug(`Deleting file '${path}'`);
return this.locks.withLock(path, async () => this.fs.delete(path)); return this.fs.delete(path);
} }
public async rename( public async rename(
oldPath: RelativePath, oldPath: RelativePath,
newPath: RelativePath, newPath: RelativePath
skipLock = false
): Promise<void> { ): Promise<void> {
this.logger.debug(`Renaming file '${oldPath}' to '${newPath}'`); this.logger.debug(`Renaming file '${oldPath}' to '${newPath}'`);
return this.safeOperation( return this.safeOperation(
oldPath, oldPath,
async () => { async () => this.fs.rename(oldPath, newPath),
if (skipLock) {
return this.fs.rename(oldPath, newPath);
} else {
return this.locks.withLock([oldPath, newPath], async () =>
this.fs.rename(oldPath, newPath)
);
}
},
"rename" "rename"
); );
} }
public tryLock(path: RelativePath): boolean {
return this.locks.tryLock(path);
}
public async waitForLock(path: RelativePath): Promise<void> {
return this.locks.waitForLock(path);
}
public unlock(path: RelativePath): void {
this.locks.unlock(path);
}
public reset(): void {
this.locks.reset();
}
/** /**
* Decorate an operation to ensure that the file exists before running it. * Decorate an operation to ensure that the file exists before running it.
* If the operation fails, it will check if the file still exists and throw * If the operation fails, it will check if the file still exists and throw
@ -154,9 +101,6 @@ export class SafeFileSystemOperations implements FileSystemOperations {
try { try {
return await operation(); return await operation();
} catch (error) { } catch (error) {
// Without locking the file, this isn't atomic, however, it's good enough in practice.
// This will only break if the file exists, gets deleted and then immediately
// recreated while `operation` is running.
if (await this.fs.exists(path)) { if (await this.fs.exists(path)) {
throw error; throw error;
} else { } else {

View file

@ -2,6 +2,7 @@ import { awaitAll } from "./utils/await-all";
import { logToConsole } from "./utils/debugging/log-to-console"; import { logToConsole } from "./utils/debugging/log-to-console";
import { slowFetchFactory } from "./utils/debugging/slow-fetch-factory"; import { slowFetchFactory } from "./utils/debugging/slow-fetch-factory";
import { slowWebSocketFactory } from "./utils/debugging/slow-web-socket-factory"; import { slowWebSocketFactory } from "./utils/debugging/slow-web-socket-factory";
import { InMemoryFileSystem } from "./utils/debugging/in-memory-file-system";
import { getRandomColor } from "./utils/get-random-color"; import { getRandomColor } from "./utils/get-random-color";
import { lineAndColumnToPosition } from "./utils/line-and-column-to-position"; import { lineAndColumnToPosition } from "./utils/line-and-column-to-position";
import { positionToLineAndColumn } from "./utils/position-to-line-and-column"; import { positionToLineAndColumn } from "./utils/position-to-line-and-column";
@ -21,14 +22,19 @@ export {
export { Logger, LogLevel, LogLine } from "./tracing/logger"; export { Logger, LogLevel, LogLine } from "./tracing/logger";
export { type SyncSettings, DEFAULT_SETTINGS } from "./persistence/settings"; export { type SyncSettings, DEFAULT_SETTINGS } from "./persistence/settings";
export { rateLimit } from "./utils/rate-limit"; export { rateLimit } from "./utils/rate-limit";
export type { RelativePath, StoredDatabase } from "./persistence/database"; export type {
RelativePath,
StoredSyncState as StoredDatabase,
DocumentRecord
} from "./sync-operations/types";
export type { FileSystemOperations } from "./file-operations/filesystem-operations"; export type { FileSystemOperations } from "./file-operations/filesystem-operations";
export type { PersistenceProvider } from "./persistence/persistence"; export type { PersistenceProvider } from "./persistence/persistence";
export type { CursorSpan } from "./services/types/CursorSpan"; export type { CursorSpan } from "./services/types/CursorSpan";
export type { ClientCursors } from "./services/types/ClientCursors"; export type { ClientCursors } from "./services/types/ClientCursors";
export type { NetworkConnectionStatus } from "./types/network-connection-status"; export type { NetworkConnectionStatus } from "./types/network-connection-status";
export type { ServerVersionMismatchError } from "./services/server-version-mismatch-error"; export type { ServerVersionMismatchError } from "./errors/server-version-mismatch-error";
export type { AuthenticationError } from "./services/authentication-error"; export type { AuthenticationError } from "./errors/authentication-error";
export { SyncResetError } from "./errors/sync-reset-error";
export type { MaybeOutdatedClientCursors } from "./types/maybe-outdated-client-cursors"; export type { MaybeOutdatedClientCursors } from "./types/maybe-outdated-client-cursors";
export { DocumentSyncStatus } from "./types/document-sync-status"; export { DocumentSyncStatus } from "./types/document-sync-status";
export { SyncClient } from "./sync-client"; export { SyncClient } from "./sync-client";
@ -37,7 +43,8 @@ export type { TextWithCursors, CursorPosition } from "reconcile-text";
export const debugging = { export const debugging = {
slowFetchFactory, slowFetchFactory,
slowWebSocketFactory, slowWebSocketFactory,
logToConsole logToConsole,
InMemoryFileSystem
}; };
export const utils = { export const utils = {

View file

@ -1,374 +0,0 @@
import type { Logger } from "../tracing/logger";
import { EMPTY_HASH } from "../utils/hash";
import { CoveredValues } from "../utils/data-structures/min-covered";
import { awaitAll } from "../utils/await-all";
import { removeFromArray } from "../utils/remove-from-array";
export type VaultUpdateId = number;
export type DocumentId = string;
export type RelativePath = string;
export interface DocumentMetadata {
parentVersionId: VaultUpdateId;
hash: string;
remoteRelativePath?: RelativePath;
}
export interface StoredDocumentMetadata {
relativePath: RelativePath;
documentId: DocumentId;
parentVersionId: VaultUpdateId;
remoteRelativePath?: RelativePath;
hash: string;
}
export interface StoredDatabase {
documents: StoredDocumentMetadata[];
lastSeenUpdateId: VaultUpdateId | undefined;
hasInitialSyncCompleted: boolean;
}
/**
* Represents a document in the database.
*
* It is mutable and its content should always represent the latest
* state of the document on disk based on the update events we have seen.
*/
export interface DocumentRecord {
relativePath: RelativePath;
documentId: DocumentId;
metadata: DocumentMetadata | undefined;
isDeleted: boolean;
updates: Promise<unknown>[];
parallelVersion: number;
}
export class Database {
private documents: DocumentRecord[];
private lastSeenUpdateIds: CoveredValues;
private hasInitialSyncCompleted: boolean;
public constructor(
private readonly logger: Logger,
initialState: Partial<StoredDatabase> | undefined,
private readonly saveData: (data: StoredDatabase) => Promise<void>
) {
initialState ??= {};
this.documents =
initialState.documents?.map(
({ relativePath, documentId, ...metadata }) => ({
relativePath,
documentId,
metadata,
isDeleted: false,
updates: [],
parallelVersion: 0
})
) ?? [];
this.ensureConsistency();
this.logger.debug(`Loaded ${this.documents.length} documents`);
const { lastSeenUpdateId } = initialState;
this.logger.debug(`Loaded last seen update id: ${lastSeenUpdateId}`);
this.lastSeenUpdateIds = new CoveredValues(
Math.max(0, lastSeenUpdateId ?? 0) // the first updateId will be 1 which is the first integer after -1
);
this.documents.forEach((doc) => {
this.lastSeenUpdateIds.add(doc.metadata?.parentVersionId);
});
this.hasInitialSyncCompleted =
initialState.hasInitialSyncCompleted ?? false;
this.logger.debug(
`Loaded hasInitialSyncCompleted: ${this.hasInitialSyncCompleted}`
);
}
public get length(): number {
return this.documents.length;
}
public get resolvedDocuments(): DocumentRecord[] {
const paths = new Map<string, DocumentRecord[]>();
this.documents
// eslint-disable-next-line no-restricted-syntax -- Type narrowing, not removing a specific item
.filter(({ metadata }) => metadata !== undefined)
.forEach((record) =>
paths.set(record.relativePath, [
record,
...(paths.get(record.relativePath) ?? [])
])
);
return Array.from(paths.values()).map((records) => {
records.sort(
(a, b) => b.parallelVersion - a.parallelVersion // descending
);
if (
records.length > 1 &&
records.some((current, i) =>
i === 0
? false
: records[i - 1].parallelVersion ===
current.parallelVersion
)
) {
throw new Error(
`Multiple documents with the same parallel version and path at ${records[0].relativePath}`
);
}
return records[0];
});
}
public updateDocumentMetadata(
metadata: {
parentVersionId: VaultUpdateId;
hash: string;
remoteRelativePath: RelativePath;
},
toUpdate: DocumentRecord
): void {
if (!this.documents.includes(toUpdate)) {
throw new Error("Document not found in database");
}
toUpdate.metadata = metadata;
this.saveInTheBackground();
}
public removeDocumentPromise(promise: Promise<unknown>): void {
const entry = this.documents.find(({ updates }) =>
updates.includes(promise)
);
if (entry === undefined) {
// This method should be idempotent and tolerant of
// stragglers calling it after the databse has been reset.
return;
}
removeFromArray(entry.updates, promise);
// No need to save as Promises don't get serialized
}
public removeDocument(find: DocumentRecord): void {
removeFromArray(this.documents, find);
this.saveInTheBackground();
}
public getLatestDocumentByRelativePath(
find: RelativePath
): DocumentRecord | undefined {
const candidates = this.documents.filter(
({ relativePath }) => relativePath === find
);
candidates.sort((a, b) => b.parallelVersion - a.parallelVersion); // descending
return candidates[0];
}
public async getResolvedDocumentByRelativePath(
relativePath: RelativePath,
promise: Promise<unknown>
): Promise<DocumentRecord> {
const entry = this.getLatestDocumentByRelativePath(relativePath);
if (entry === undefined) {
throw new Error(
`Document not found by relative path: ${relativePath}, ${JSON.stringify(
this.documents,
null,
2
)}`
);
}
const currentPromises = entry.updates;
entry.updates = [...currentPromises, promise];
await awaitAll(currentPromises);
return entry;
}
public createNewPendingDocument(
documentId: DocumentId,
relativePath: RelativePath,
promise: Promise<unknown>
): DocumentRecord {
this.logger.debug(
`Creating new pending document: ${relativePath} (${documentId})`
);
const previousEntry =
this.getLatestDocumentByRelativePath(relativePath);
const entry = {
relativePath,
documentId,
metadata: undefined,
isDeleted: false,
updates: [promise],
parallelVersion:
previousEntry?.parallelVersion === undefined
? 0
: previousEntry.parallelVersion + 1
};
this.documents.push(entry);
this.saveInTheBackground();
return entry;
}
public createNewEmptyDocument(
documentId: DocumentId,
parentVersionId: VaultUpdateId,
relativePath: RelativePath
): DocumentRecord {
const entry = {
relativePath,
documentId,
metadata: {
parentVersionId,
hash: EMPTY_HASH,
remoteRelativePath: relativePath
},
isDeleted: false,
updates: [],
parallelVersion: 0
};
this.documents.push(entry);
this.saveInTheBackground();
return entry;
}
public getDocumentByDocumentId(
find: DocumentId
): DocumentRecord | undefined {
return this.documents.find(({ documentId }) => documentId === find);
}
public move(
oldRelativePath: RelativePath,
newRelativePath: RelativePath
): void {
const oldDocument =
this.getLatestDocumentByRelativePath(oldRelativePath);
if (oldDocument === undefined) {
return;
}
const newDocument =
this.getLatestDocumentByRelativePath(newRelativePath);
if (newDocument?.isDeleted === false) {
throw new Error(
`Document already exists at new location: ${newRelativePath}`
);
}
oldDocument.relativePath = newRelativePath;
// We're in a strange state where the target of the move has just got deleted,
// however, its metadata might already have a bunch of updates queued up for
// the document at the new location. We need to keep these updates.
oldDocument.parallelVersion =
newDocument !== undefined ? newDocument.parallelVersion + 1 : 0;
this.saveInTheBackground();
}
public delete(relativePath: RelativePath): void {
const candidate = this.getLatestDocumentByRelativePath(relativePath);
if (candidate === undefined) {
throw new Error(
`Document not found by relative path: ${relativePath}`
);
}
candidate.isDeleted = true;
}
public getHasInitialSyncCompleted(): boolean {
return this.hasInitialSyncCompleted;
}
public setHasInitialSyncCompleted(value: boolean): void {
this.hasInitialSyncCompleted = value;
this.saveInTheBackground();
}
public getLastSeenUpdateId(): VaultUpdateId {
return this.lastSeenUpdateIds.min;
}
public addSeenUpdateId(value: number): void {
const previousMin = this.lastSeenUpdateIds.min;
this.lastSeenUpdateIds.add(value);
if (previousMin !== this.lastSeenUpdateIds.min) {
this.saveInTheBackground();
}
}
public setLastSeenUpdateId(value: number): void {
this.lastSeenUpdateIds.min = value;
this.saveInTheBackground();
}
public reset(): void {
this.documents = [];
this.lastSeenUpdateIds = new CoveredValues(
0 // the first updateId will be 1 which is the first integer after -1
);
this.hasInitialSyncCompleted = false;
this.saveInTheBackground();
}
public async save(): Promise<void> {
return this.saveData({
documents: this.resolvedDocuments.map(
({ relativePath, documentId, metadata }) => ({
documentId,
relativePath,
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
...metadata! // `resolvedDocuments` only returns docs with metadata set
})
),
lastSeenUpdateId: this.lastSeenUpdateIds.min,
hasInitialSyncCompleted: this.hasInitialSyncCompleted
});
}
private ensureConsistency(): void {
const idToPath = new Map<string, string[]>();
this.resolvedDocuments.forEach(({ relativePath, documentId }) => {
idToPath.set(documentId, [
...(idToPath.get(documentId) ?? []),
relativePath
]);
});
const duplicates = Array.from(idToPath.entries())
.filter(([_, paths]) => paths.length > 1)
.map(([id, paths]) => `${id} (${paths.join(", ")})`);
if (duplicates.length > 0) {
throw new Error(
"Document IDs are not unique, found duplicates: " +
duplicates.join("; ")
);
}
}
private saveInTheBackground(): void {
this.ensureConsistency();
void this.save().catch((error: unknown) => {
this.logger.error(`Error saving data: ${error}`);
});
}
}

View file

@ -6,7 +6,6 @@ export interface SyncSettings {
remoteUri: string; remoteUri: string;
token: string; token: string;
vaultName: string; vaultName: string;
syncConcurrency: number;
isSyncEnabled: boolean; isSyncEnabled: boolean;
maxFileSizeMB: number; maxFileSizeMB: number;
ignorePatterns: string[]; ignorePatterns: string[];
@ -14,22 +13,19 @@ export interface SyncSettings {
diffCacheSizeMB: number; diffCacheSizeMB: number;
enableTelemetry: boolean; enableTelemetry: boolean;
networkRetryIntervalMs: number; networkRetryIntervalMs: number;
minimumSaveIntervalMs: number;
} }
export const DEFAULT_SETTINGS: SyncSettings = { export const DEFAULT_SETTINGS: SyncSettings = {
remoteUri: "", remoteUri: "",
token: "", token: "",
vaultName: "default", vaultName: "default",
syncConcurrency: 1,
isSyncEnabled: false, isSyncEnabled: false,
maxFileSizeMB: 10, maxFileSizeMB: 10,
ignorePatterns: [], ignorePatterns: [],
webSocketRetryIntervalMs: 3500, webSocketRetryIntervalMs: 3500,
diffCacheSizeMB: 4, diffCacheSizeMB: 4,
enableTelemetry: false, enableTelemetry: false,
networkRetryIntervalMs: 1000, networkRetryIntervalMs: 1000
minimumSaveIntervalMs: 1000
}; };
export class Settings { export class Settings {
@ -38,7 +34,7 @@ export class Settings {
>(); >();
private settings: SyncSettings; private settings: SyncSettings;
private readonly lock: Lock = new Lock(); private readonly lock: Lock;
public constructor( public constructor(
private readonly logger: Logger, private readonly logger: Logger,
@ -50,6 +46,8 @@ export class Settings {
...(initialState ?? {}) ...(initialState ?? {})
}; };
this.lock = new Lock(Settings.name, this.logger);
this.logger.debug( this.logger.debug(
`Loaded settings: ${JSON.stringify(this.settings, null, 2)}` `Loaded settings: ${JSON.stringify(this.settings, null, 2)}`
); );

View file

@ -0,0 +1,8 @@
import type { Settings } from "../persistence/settings";
export function buildVaultUrl(settings: Settings, path: string): string {
const { vaultName, remoteUri } = settings.getSettings();
const remoteUriWithoutTrailingSlash = remoteUri.replace(/\/+$/, "");
const encodedVaultName = encodeURIComponent(vaultName.trim());
return `${remoteUriWithoutTrailingSlash}/vaults/${encodedVaultName}${path}`;
}

View file

@ -3,7 +3,7 @@ import { describe, it, mock, beforeEach, afterEach } from "node:test";
import assert from "node:assert"; import assert from "node:assert";
import { FetchController } from "./fetch-controller"; import { FetchController } from "./fetch-controller";
import { Logger } from "../tracing/logger"; import { Logger } from "../tracing/logger";
import { SyncResetError } from "./sync-reset-error"; import { SyncResetError } from "../errors/sync-reset-error";
import { sleep } from "../utils/sleep"; import { sleep } from "../utils/sleep";
describe("FetchController", () => { describe("FetchController", () => {

View file

@ -1,6 +1,5 @@
import type { Logger } from "../tracing/logger"; import type { Logger } from "../tracing/logger";
import { createPromise } from "../utils/create-promise"; import { SyncResetError } from "../errors/sync-reset-error";
import { SyncResetError } from "./sync-reset-error";
/** /**
* Offers a resettable fetch implementation that waits until syncing is enabled * Offers a resettable fetch implementation that waits until syncing is enabled
@ -13,15 +12,18 @@ export class FetchController {
// Promise resolves on the next state change: sync enabled/disabled or reset started/ended // Promise resolves on the next state change: sync enabled/disabled or reset started/ended
private until: Promise<symbol>; private until: Promise<symbol>;
private resolveUntil: (result: symbol) => unknown; private resolveUntil: (value: symbol | PromiseLike<symbol>) => void;
private rejectUntil: (reason: unknown) => unknown; private rejectUntil: (reason?: unknown) => void;
public constructor( public constructor(
private _canFetch: boolean, private _canFetch: boolean,
private readonly logger: Logger private readonly logger: Logger
) { ) {
[this.until, this.resolveUntil, this.rejectUntil] = ({
createPromise<symbol>(); promise: this.until,
resolve: this.resolveUntil,
reject: this.rejectUntil
} = Promise.withResolvers<symbol>());
} }
/** /**
@ -42,8 +44,11 @@ export class FetchController {
if (!this.isResetting) { if (!this.isResetting) {
const previousResolve = this.resolveUntil; const previousResolve = this.resolveUntil;
[this.until, this.resolveUntil, this.rejectUntil] = ({
createPromise<symbol>(); promise: this.until,
resolve: this.resolveUntil,
reject: this.rejectUntil
} = Promise.withResolvers<symbol>());
previousResolve(FetchController.UNTIL_RESOLUTION); previousResolve(FetchController.UNTIL_RESOLUTION);
} }
} }
@ -81,7 +86,11 @@ export class FetchController {
} }
this.isResetting = false; this.isResetting = false;
[this.until, this.resolveUntil, this.rejectUntil] = createPromise(); ({
promise: this.until,
resolve: this.resolveUntil,
reject: this.rejectUntil
} = Promise.withResolvers<symbol>());
} }
/** /**

View file

@ -1,6 +1,7 @@
import { SUPPORTED_API_VERSION } from "../consts"; import { SUPPORTED_API_VERSION } from "../consts";
import { AuthenticationError } from "./authentication-error"; import { AuthenticationError } from "../errors/authentication-error";
import { ServerVersionMismatchError } from "./server-version-mismatch-error"; import { ServerVersionMismatchError } from "../errors/server-version-mismatch-error";
import type { Settings } from "../persistence/settings";
import type { SyncService } from "./sync-service"; import type { SyncService } from "./sync-service";
import type { PingResponse } from "./types/PingResponse"; import type { PingResponse } from "./types/PingResponse";
@ -14,7 +15,20 @@ export class ServerConfig {
private response: Promise<PingResponse> | undefined; private response: Promise<PingResponse> | undefined;
private config: ServerConfigData | undefined; private config: ServerConfigData | undefined;
public constructor(private readonly syncService: SyncService) {} public constructor(
private readonly syncService: SyncService,
settings: Settings
) {
settings.onSettingsChanged.add((newSettings, oldSettings) => {
if (
newSettings.token !== oldSettings.token ||
newSettings.vaultName !== oldSettings.vaultName ||
newSettings.remoteUri !== oldSettings.remoteUri
) {
this.reset();
}
});
}
private static validateConfig(config: ServerConfigData): void { private static validateConfig(config: ServerConfigData): void {
if (config.supportedApiVersion !== SUPPORTED_API_VERSION) { if (config.supportedApiVersion !== SUPPORTED_API_VERSION) {
@ -34,11 +48,6 @@ export class ServerConfig {
} }
} }
// warm the cache
public async initialize(): Promise<void> {
await this.getConfig();
}
public async checkConnection(forceUpdate = false): Promise<{ public async checkConnection(forceUpdate = false): Promise<{
isSuccessful: boolean; isSuccessful: boolean;
message: string; message: string;
@ -46,7 +55,7 @@ export class ServerConfig {
try { try {
let { response } = this; let { response } = this;
if (!response || forceUpdate) { if (!response || forceUpdate) {
response = this.response = this.syncService.ping(); response = this.startPing();
} }
const result: PingResponse = await response; // it must be defined, otherwise we would have thrown above const result: PingResponse = await response; // it must be defined, otherwise we would have thrown above
@ -73,7 +82,7 @@ export class ServerConfig {
public async getConfig(): Promise<ServerConfigData> { public async getConfig(): Promise<ServerConfigData> {
if (!this.config) { if (!this.config) {
this.response ??= this.syncService.ping(); this.response ??= this.startPing();
this.config = await this.response; this.config = await this.response;
} }
@ -86,4 +95,15 @@ export class ServerConfig {
this.response = undefined; this.response = undefined;
this.config = undefined; this.config = undefined;
} }
private async startPing(): Promise<PingResponse> {
const pending = this.syncService.ping().catch((e: unknown) => {
if (this.response === pending) {
this.response = undefined;
}
throw e;
});
this.response = pending;
return pending;
}
} }

View file

@ -2,25 +2,27 @@ import type {
DocumentId, DocumentId,
RelativePath, RelativePath,
VaultUpdateId VaultUpdateId
} from "../persistence/database"; } from "../sync-operations/types";
import type { Logger } from "../tracing/logger"; import type { Logger } from "../tracing/logger";
import type { Settings } from "../persistence/settings"; import type { Settings } from "../persistence/settings";
import type { FetchController } from "./fetch-controller"; import type { FetchController } from "./fetch-controller";
import { sleep } from "../utils/sleep"; import { sleep } from "../utils/sleep";
import { SyncResetError } from "./sync-reset-error"; import { SyncResetError } from "../errors/sync-reset-error";
import { HttpClientError } from "../errors/http-client-error";
import type { SerializedError } from "./types/SerializedError"; import type { SerializedError } from "./types/SerializedError";
import type { DocumentVersionWithoutContent } from "./types/DocumentVersionWithoutContent"; import type { DocumentVersionWithoutContent } from "./types/DocumentVersionWithoutContent";
import type { DocumentUpdateResponse } from "./types/DocumentUpdateResponse"; import type { DocumentUpdateResponse } from "./types/DocumentUpdateResponse";
import type { DocumentVersion } from "./types/DocumentVersion"; import type { DocumentVersion } from "./types/DocumentVersion";
import type { FetchLatestDocumentsResponse } from "./types/FetchLatestDocumentsResponse"; import type { FetchLatestDocumentsResponse } from "./types/FetchLatestDocumentsResponse";
import type { PingResponse } from "./types/PingResponse"; import type { PingResponse } from "./types/PingResponse";
import type { DeleteDocumentVersion } from "./types/DeleteDocumentVersion";
import type { UpdateTextDocumentVersion } from "./types/UpdateTextDocumentVersion"; import type { UpdateTextDocumentVersion } from "./types/UpdateTextDocumentVersion";
import { buildVaultUrl } from "./build-vault-url";
export class SyncService { export class SyncService {
private readonly client: typeof globalThis.fetch; private readonly client: typeof globalThis.fetch;
private readonly pingClient: typeof globalThis.fetch; private readonly pingClient: typeof globalThis.fetch;
private isStopped = false;
public constructor( public constructor(
private readonly deviceId: string, private readonly deviceId: string,
@ -65,28 +67,68 @@ export class SyncService {
return result; return result;
} }
private static async throwIfNotOk(
response: Response,
operation: string
): Promise<void> {
if (response.ok) {
return;
}
const message = `Failed to ${operation}: ${await SyncService.errorFromResponse(response)}`;
// 429 is the only 4xx the server uses for *transient* contention
// (`WriteBusyError` → HTTP 429). Every other 4xx means the request
// is permanently rejected and shouldn't be retried.
if (response.status === 429) {
throw new Error(message);
}
if (response.status >= 400 && response.status < 500) {
throw new HttpClientError(response.status, message);
}
throw new Error(message);
}
/**
* Signal that the service is shutting down so any in-flight
* `retryForever` exits at its next iteration instead of looping
* indefinitely after the rest of the client has stopped. Idempotent.
*/
public stop(): void {
this.isStopped = true;
}
/**
* Re-enable the service after a `stop()`. Used when the client pauses
* and resumes syncing within the same lifecycle (e.g. user toggles
* sync off and on).
*/
public resume(): void {
this.isStopped = false;
}
public async create({ public async create({
documentId,
relativePath, relativePath,
lastSeenVaultUpdateId,
contentBytes contentBytes
}: { }: {
documentId?: DocumentId;
relativePath: RelativePath; relativePath: RelativePath;
lastSeenVaultUpdateId: VaultUpdateId;
contentBytes: Uint8Array; contentBytes: Uint8Array;
}): Promise<DocumentVersionWithoutContent> { }): Promise<DocumentUpdateResponse> {
return this.retryForever(async () => { return this.retryForever(async () => {
const formData = new FormData(); const formData = new FormData();
if (documentId !== undefined) {
formData.append("document_id", documentId);
}
formData.append("relative_path", relativePath); formData.append("relative_path", relativePath);
formData.append(
"last_seen_vault_update_id",
lastSeenVaultUpdateId.toString()
);
formData.append( formData.append(
"content", "content",
new Blob([new Uint8Array(contentBytes)]) new Blob([new Uint8Array(contentBytes)])
); );
this.logger.debug( this.logger.debug(
`Creating document with id ${documentId} and relative path ${relativePath}` `Creating document with relative path ${relativePath}`
); );
const response = await this.client(this.getUrl("/documents"), { const response = await this.client(this.getUrl("/documents"), {
@ -95,16 +137,10 @@ export class SyncService {
headers: this.getDefaultHeaders() headers: this.getDefaultHeaders()
}); });
if (!response.ok) { await SyncService.throwIfNotOk(response, "create document");
throw new Error(
`Failed to create document: ${await SyncService.errorFromResponse(
response
)}`
);
}
const result: DocumentVersionWithoutContent = const result: DocumentUpdateResponse =
(await response.json()) as DocumentVersionWithoutContent; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion (await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
this.logger.debug(`Created document ${JSON.stringify(result)}`); this.logger.debug(`Created document ${JSON.stringify(result)}`);
@ -120,17 +156,17 @@ export class SyncService {
}: { }: {
parentVersionId: VaultUpdateId; parentVersionId: VaultUpdateId;
documentId: DocumentId; documentId: DocumentId;
relativePath: RelativePath; relativePath: RelativePath | undefined;
content: (number | string)[]; content: (number | string)[];
}): Promise<DocumentUpdateResponse> { }): Promise<DocumentUpdateResponse> {
return this.retryForever(async () => { return this.retryForever(async () => {
this.logger.debug( this.logger.debug(
`Updating text document ${documentId} with parent version ${parentVersionId} and relative path ${relativePath}, content [${content.join(", ")}]` `Updating text document ${documentId} with parent version ${parentVersionId} and relative path ${relativePath ?? "<unchanged>"}, content [${content.join(", ")}]`
); );
const request: UpdateTextDocumentVersion = { const request: UpdateTextDocumentVersion = {
parentVersionId, parentVersionId,
relativePath, relativePath: relativePath ?? null,
content content
}; };
@ -143,13 +179,7 @@ export class SyncService {
} }
); );
if (!response.ok) { await SyncService.throwIfNotOk(response, "update document");
throw new Error(
`Failed to update document: ${await SyncService.errorFromResponse(
response
)}`
);
}
const result: DocumentUpdateResponse = const result: DocumentUpdateResponse =
(await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion (await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
@ -172,16 +202,18 @@ export class SyncService {
}: { }: {
parentVersionId: VaultUpdateId; parentVersionId: VaultUpdateId;
documentId: DocumentId; documentId: DocumentId;
relativePath: RelativePath; relativePath: RelativePath | undefined;
contentBytes: Uint8Array; contentBytes: Uint8Array;
}): Promise<DocumentUpdateResponse> { }): Promise<DocumentUpdateResponse> {
return this.retryForever(async () => { return this.retryForever(async () => {
this.logger.debug( this.logger.debug(
`Updating binary document ${documentId} with parent version ${parentVersionId} and relative path ${relativePath}` `Updating binary document ${documentId} with parent version ${parentVersionId} and relative path ${relativePath ?? "<unchanged>"}`
); );
const formData = new FormData(); const formData = new FormData();
formData.append("parent_version_id", parentVersionId.toString()); formData.append("parent_version_id", parentVersionId.toString());
if (relativePath !== undefined) {
formData.append("relative_path", relativePath); formData.append("relative_path", relativePath);
}
formData.append( formData.append(
"content", "content",
new Blob([new Uint8Array(contentBytes)]) new Blob([new Uint8Array(contentBytes)])
@ -196,13 +228,7 @@ export class SyncService {
} }
); );
if (!response.ok) { await SyncService.throwIfNotOk(response, "update document");
throw new Error(
`Failed to update document: ${await SyncService.errorFromResponse(
response
)}`
);
}
const result: DocumentUpdateResponse = const result: DocumentUpdateResponse =
(await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion (await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
@ -218,44 +244,29 @@ export class SyncService {
} }
public async delete({ public async delete({
documentId, documentId
relativePath
}: { }: {
documentId: DocumentId; documentId: DocumentId;
relativePath: RelativePath;
}): Promise<DocumentVersionWithoutContent> { }): Promise<DocumentVersionWithoutContent> {
return this.retryForever(async () => { return this.retryForever(async () => {
const request: DeleteDocumentVersion = { this.logger.debug(`Delete document with id ${documentId}`);
relativePath
};
this.logger.debug(
`Delete document with id ${documentId} and relative path ${relativePath}`
);
// The server identifies the document by its URL path; no body
// is needed. Sending one was a leftover of an earlier shape.
const response = await this.client( const response = await this.client(
this.getUrl(`/documents/${documentId}`), this.getUrl(`/documents/${documentId}`),
{ {
method: "DELETE", method: "DELETE",
body: JSON.stringify(request), headers: this.getDefaultHeaders()
headers: this.getDefaultHeaders({ type: "json" })
} }
); );
if (!response.ok) { await SyncService.throwIfNotOk(response, "delete document");
throw new Error(
`Failed to delete document: ${await SyncService.errorFromResponse(
response
)}`
);
}
const result: DocumentVersionWithoutContent = const result: DocumentVersionWithoutContent =
(await response.json()) as DocumentVersionWithoutContent; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion (await response.json()) as DocumentVersionWithoutContent; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
this.logger.debug( this.logger.debug(`Deleted document with id ${documentId}`);
`Deleted document ${relativePath} with id ${documentId}`
);
return result; return result;
}); });
@ -276,13 +287,7 @@ export class SyncService {
} }
); );
if (!response.ok) { await SyncService.throwIfNotOk(response, "get document");
throw new Error(
`Failed to get document: ${await SyncService.errorFromResponse(
response
)}`
);
}
const result: DocumentVersion = const result: DocumentVersion =
(await response.json()) as DocumentVersion; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion (await response.json()) as DocumentVersion; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
@ -314,13 +319,10 @@ export class SyncService {
} }
); );
if (!response.ok) { await SyncService.throwIfNotOk(
throw new Error( response,
`Failed to get document: ${await SyncService.errorFromResponse( "get document version content"
response
)}`
); );
}
const result = await response.bytes(); const result = await response.bytes();
this.logger.debug( this.logger.debug(
@ -341,19 +343,13 @@ export class SyncService {
const url = new URL(this.getUrl("/documents")); const url = new URL(this.getUrl("/documents"));
if (since !== undefined) { if (since !== undefined) {
url.searchParams.append("since", since.toString()); url.searchParams.append("since_update_id", since.toString());
} }
const response = await this.client(url.toString(), { const response = await this.client(url.toString(), {
headers: this.getDefaultHeaders() headers: this.getDefaultHeaders()
}); });
if (!response.ok) { await SyncService.throwIfNotOk(response, "get documents");
throw new Error(
`Failed to get documents: ${await SyncService.errorFromResponse(
response
)}`
);
}
const result: FetchLatestDocumentsResponse = const result: FetchLatestDocumentsResponse =
(await response.json()) as FetchLatestDocumentsResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion (await response.json()) as FetchLatestDocumentsResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
@ -390,10 +386,7 @@ export class SyncService {
} }
private getUrl(path: string): string { private getUrl(path: string): string {
const { vaultName, remoteUri } = this.settings.getSettings(); return buildVaultUrl(this.settings, path);
const remoteUriWithoutTrailingSlash = remoteUri.replace(/\/+$/, "");
const encodedVaultName = encodeURIComponent(vaultName.trim());
return `${remoteUriWithoutTrailingSlash}/vaults/${encodedVaultName}${path}`;
} }
private getDefaultHeaders( private getDefaultHeaders(
@ -414,13 +407,17 @@ export class SyncService {
private async retryForever<T>(fn: () => Promise<T>): Promise<T> { private async retryForever<T>(fn: () => Promise<T>): Promise<T> {
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition // eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
while (true) { while (true) {
this.throwIfStopped();
try { try {
return await fn(); return await fn();
} catch (e) { } catch (e) {
// We must not retry errors coming from reset if (
if (e instanceof SyncResetError) { e instanceof SyncResetError ||
e instanceof HttpClientError
) {
throw e; throw e;
} }
this.throwIfStopped();
const retryInterval = const retryInterval =
this.settings.getSettings().networkRetryIntervalMs; this.settings.getSettings().networkRetryIntervalMs;
@ -431,4 +428,10 @@ export class SyncService {
} }
} }
} }
private throwIfStopped(): void {
if (this.isStopped) {
throw new SyncResetError();
}
}
} }

View file

@ -1,13 +1,7 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually. // This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export interface CreateDocumentVersion { export interface CreateDocumentVersion {
/**
* The client can decide the document id (if it wishes to) in order
* to help with syncing. If the client does not provide a document id,
* the server will generate one. If the client provides a document id
* it must not already exist in the database.
*/
document_id: string | null;
relative_path: string; relative_path: string;
last_seen_vault_update_id: number;
content: number[]; content: number[];
} }

View file

@ -3,7 +3,7 @@ import type { DocumentVersion } from "./DocumentVersion";
import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent"; import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent";
/** /**
* Response to an update document request. * Response to a create/update document request.
*/ */
export type DocumentUpdateResponse = export type DocumentUpdateResponse =
| ({ type: "FastForwardUpdate" } & DocumentVersionWithoutContent) | ({ type: "FastForwardUpdate" } & DocumentVersionWithoutContent)

View file

@ -9,4 +9,8 @@ export interface DocumentVersionWithoutContent {
userId: string; userId: string;
deviceId: string; deviceId: string;
contentSize: number; contentSize: number;
/**
* True iff this is the first version of the document
*/
isNewFile: boolean;
} }

View file

@ -2,8 +2,8 @@
import type { CursorSpan } from "./CursorSpan"; import type { CursorSpan } from "./CursorSpan";
export interface DocumentWithCursors { export interface DocumentWithCursors {
vault_update_id: number | null; vaultUpdateId: number | null;
document_id: string; documentId: string;
relative_path: string; relativePath: string;
cursors: CursorSpan[]; cursors: CursorSpan[];
} }

View file

@ -0,0 +1,11 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { VaultInfo } from "./VaultInfo";
/**
* Response to listing vaults accessible to the authenticated user.
*/
export interface ListVaultsResponse {
vaults: VaultInfo[];
hasMore: boolean;
userName: string;
}

View file

@ -2,6 +2,6 @@
export interface UpdateTextDocumentVersion { export interface UpdateTextDocumentVersion {
parentVersionId: number; parentVersionId: number;
relativePath: string; relativePath: string | null;
content: (number | string)[]; content: (number | string)[];
} }

View file

@ -0,0 +1,10 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent";
/**
* Response to a vault history request (paginated).
*/
export interface VaultHistoryResponse {
versions: DocumentVersionWithoutContent[];
hasMore: boolean;
}

View file

@ -0,0 +1,10 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
/**
* Summary of a single vault returned by the list-vaults endpoint.
*/
export interface VaultInfo {
name: string;
documentCount: number;
createdAt: string | null;
}

View file

@ -2,6 +2,5 @@
import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent"; import type { DocumentVersionWithoutContent } from "./DocumentVersionWithoutContent";
export interface WebSocketVaultUpdate { export interface WebSocketVaultUpdate {
documents: DocumentVersionWithoutContent[]; document: DocumentVersionWithoutContent;
isInitialSync: boolean;
} }

View file

@ -4,8 +4,7 @@ import assert from "node:assert";
import { WebSocketManager } from "./websocket-manager"; import { WebSocketManager } from "./websocket-manager";
import type { Logger } from "../tracing/logger"; import type { Logger } from "../tracing/logger";
import type { Settings } from "../persistence/settings"; import type { Settings } from "../persistence/settings";
// eslint-disable-next-line @typescript-eslint/no-require-imports import { awaitAll } from "../utils/await-all";
const WebSocket = require("ws") as typeof globalThis.WebSocket;
class MockCloseEvent extends Event { class MockCloseEvent extends Event {
public code: number; public code: number;
@ -91,10 +90,8 @@ function createMockFn<T extends (...args: unknown[]) => unknown>(
describe("WebSocketManager", () => { describe("WebSocketManager", () => {
let mockLogger: Logger = undefined as unknown as Logger; let mockLogger: Logger = undefined as unknown as Logger;
let mockSettings: Settings = undefined as unknown as Settings; let mockSettings: Settings = undefined as unknown as Settings;
let deviceId = "test-device-123";
beforeEach(() => { beforeEach(() => {
deviceId = "test-device-123";
const noop = (): void => { const noop = (): void => {
// Intentionally empty for mock // Intentionally empty for mock
}; };
@ -116,7 +113,6 @@ describe("WebSocketManager", () => {
it("cleans up promises after message handling", async () => { it("cleans up promises after message handling", async () => {
const manager = new WebSocketManager( const manager = new WebSocketManager(
deviceId,
mockLogger, mockLogger,
mockSettings, mockSettings,
MockWebSocket as unknown as typeof WebSocket MockWebSocket as unknown as typeof WebSocket
@ -146,7 +142,6 @@ describe("WebSocketManager", () => {
it("cleans up cursor position promises", async () => { it("cleans up cursor position promises", async () => {
const manager = new WebSocketManager( const manager = new WebSocketManager(
deviceId,
mockLogger, mockLogger,
mockSettings, mockSettings,
MockWebSocket as unknown as typeof WebSocket MockWebSocket as unknown as typeof WebSocket
@ -176,7 +171,6 @@ describe("WebSocketManager", () => {
it("logs handshake send errors", async () => { it("logs handshake send errors", async () => {
const manager = new WebSocketManager( const manager = new WebSocketManager(
deviceId,
mockLogger, mockLogger,
mockSettings, mockSettings,
MockWebSocket as unknown as typeof WebSocket MockWebSocket as unknown as typeof WebSocket
@ -205,7 +199,6 @@ describe("WebSocketManager", () => {
it("completes stop with timeout protection", async () => { it("completes stop with timeout protection", async () => {
const manager = new WebSocketManager( const manager = new WebSocketManager(
deviceId,
mockLogger, mockLogger,
mockSettings, mockSettings,
MockWebSocket as unknown as typeof WebSocket MockWebSocket as unknown as typeof WebSocket
@ -220,7 +213,6 @@ describe("WebSocketManager", () => {
it("clears old handlers on reconnection", async () => { it("clears old handlers on reconnection", async () => {
const manager = new WebSocketManager( const manager = new WebSocketManager(
deviceId,
mockLogger, mockLogger,
mockSettings, mockSettings,
MockWebSocket as unknown as typeof WebSocket MockWebSocket as unknown as typeof WebSocket
@ -255,9 +247,68 @@ describe("WebSocketManager", () => {
await manager.stop(); await manager.stop();
}); });
it("handles concurrent stop() calls without stranding either caller", async () => {
// Real WebSocket.close() doesn't fire onclose synchronously, and the
// socket stays reachable across the close handshake. Model that
// here so the manager's `while (isWebSocketConnected)` loop is
// actually awaiting when the second stop() races in. Static OPEN
// is required because the manager compares readyState against
// `factory.OPEN`.
class AsyncCloseWebSocket extends MockWebSocket {
public static readonly OPEN = WebSocket.OPEN;
public override close(code?: number, reason?: string): void {
if (
this.readyState === WebSocket.CLOSED ||
(this as { _closing?: boolean })._closing === true
) {
return;
}
(this as { _closing?: boolean })._closing = true;
setTimeout(() => {
this.readyState = WebSocket.CLOSED;
this.onclose?.(
new MockCloseEvent("close", {
code: code ?? 1000,
reason: reason ?? ""
})
);
}, 5);
}
}
const manager = new WebSocketManager(
mockLogger,
mockSettings,
AsyncCloseWebSocket as unknown as typeof WebSocket
);
manager.start();
await new Promise((resolve) => setTimeout(resolve, 50));
const start = Date.now();
// Two concurrent stops mimic destroy() racing onSettingsChange.
await awaitAll([manager.stop(), manager.stop()]);
const elapsed = Date.now() - start;
// Both should resolve via the normal close path; if the second call
// had clobbered the first's resolver, the first would have been
// stranded until the 10s disconnect timeout.
assert.ok(
elapsed < 1000,
`concurrent stop() took ${elapsed}ms — expected fast resolution`
);
const errorCalls = (mockLogger.error as unknown as { calls: unknown[] })
.calls;
assert.strictEqual(
errorCalls.length,
0,
"no timeout-recovery error should be logged"
);
});
it("tracks message handling promises", async () => { it("tracks message handling promises", async () => {
const manager = new WebSocketManager( const manager = new WebSocketManager(
deviceId,
mockLogger, mockLogger,
mockSettings, mockSettings,
MockWebSocket as unknown as typeof WebSocket MockWebSocket as unknown as typeof WebSocket

View file

@ -4,12 +4,15 @@ import type { WebSocketServerMessage } from "./types/WebSocketServerMessage";
import type { WebSocketClientMessage } from "./types/WebSocketClientMessage"; import type { WebSocketClientMessage } from "./types/WebSocketClientMessage";
import type { CursorPositionFromClient } from "./types/CursorPositionFromClient"; import type { CursorPositionFromClient } from "./types/CursorPositionFromClient";
import type { ClientCursors } from "./types/ClientCursors"; import type { ClientCursors } from "./types/ClientCursors";
import { createPromise } from "../utils/create-promise";
import type { WebSocketVaultUpdate } from "./types/WebSocketVaultUpdate"; import type { WebSocketVaultUpdate } from "./types/WebSocketVaultUpdate";
import { WEBSOCKET_DISCONNECT_TIMEOUT_IN_S } from "../consts"; import {
WEBSOCKET_DISCONNECT_TIMEOUT_IN_SECONDS,
WEBSOCKET_CONNECTION_TIMEOUT_IN_SECONDS
} from "../consts";
import { removeFromArray } from "../utils/remove-from-array"; import { removeFromArray } from "../utils/remove-from-array";
import { EventListeners } from "../utils/data-structures/event-listeners"; import { EventListeners } from "../utils/data-structures/event-listeners";
import { awaitAll } from "../utils/await-all"; import { awaitAll } from "../utils/await-all";
import { buildVaultUrl } from "./build-vault-url";
export class WebSocketManager { export class WebSocketManager {
public readonly onWebSocketStatusChanged = new EventListeners< public readonly onWebSocketStatusChanged = new EventListeners<
@ -26,32 +29,22 @@ export class WebSocketManager {
private isStopped = true; private isStopped = true;
private resolveDisconnectingPromise: null | (() => unknown) = null; private resolveDisconnectingPromise: null | (() => unknown) = null;
private stopPromise: Promise<void> | null = null;
private reconnectTimeoutId: ReturnType<typeof setTimeout> | undefined; private reconnectTimeoutId: ReturnType<typeof setTimeout> | undefined;
private connectionTimeoutId: ReturnType<typeof setTimeout> | undefined;
private readonly outstandingPromises: Promise<unknown>[] = []; private readonly outstandingPromises: Promise<unknown>[] = [];
private webSocket: WebSocket | undefined; private webSocket: WebSocket | undefined;
private readonly webSocketFactoryImplementation: typeof globalThis.WebSocket;
public constructor( public constructor(
private readonly deviceId: string,
private readonly logger: Logger, private readonly logger: Logger,
private readonly settings: Settings, private readonly settings: Settings,
webSocketImplementation?: typeof globalThis.WebSocket private readonly webSocketFactoryImplementation: typeof globalThis.WebSocket = WebSocket
) { ) {}
if (webSocketImplementation) {
this.webSocketFactoryImplementation = webSocketImplementation; public get hasOutstandingWork(): boolean {
} else { return this.outstandingPromises.length > 0;
if (
typeof globalThis !== "undefined" &&
typeof globalThis.WebSocket === "undefined"
) {
// eslint-disable-next-line
this.webSocketFactoryImplementation = require("ws"); // polyfill for WebSocket in Node.js
} else {
this.webSocketFactoryImplementation = WebSocket;
}
}
} }
public get isWebSocketConnected(): boolean { public get isWebSocketConnected(): boolean {
@ -67,49 +60,14 @@ export class WebSocketManager {
} }
public async stop(): Promise<void> { public async stop(): Promise<void> {
const [promise, resolve] = createPromise(); // Concurrent callers (e.g. destroy() and onSettingsChange) must share
this.resolveDisconnectingPromise = resolve; // the same disconnect; otherwise the second call would overwrite
// resolveDisconnectingPromise and strand the first caller's await
this.isStopped = true; // until the timeout rejects.
this.stopPromise ??= this.performStop().finally(() => {
if (this.reconnectTimeoutId !== undefined) { this.stopPromise = null;
clearTimeout(this.reconnectTimeoutId);
this.reconnectTimeoutId = undefined;
}
this.webSocket?.close(1000, "WebSocketManager has been stopped");
// eslint-disable-next-line @typescript-eslint/init-declarations
let timeoutId: ReturnType<typeof setTimeout> | undefined;
const timeoutPromise = new Promise<void>((_, reject) => {
timeoutId = setTimeout(() => {
reject(
new Error(
`Timeout waiting for WebSocket to close after ${WEBSOCKET_DISCONNECT_TIMEOUT_IN_S} seconds`
)
);
}, WEBSOCKET_DISCONNECT_TIMEOUT_IN_S * 1000);
}); });
await this.stopPromise;
try {
while (this.isWebSocketConnected) {
await Promise.race([promise, timeoutPromise]);
}
} catch (error) {
this.logger.error(
`Error while waiting for WebSocket to close: ${String(error)}`
);
// Force cleanup even if close didn't work
this.resolveDisconnectingPromise();
this.resolveDisconnectingPromise = null;
} finally {
// Clear timeout to prevent unhandled rejection
if (timeoutId !== undefined) {
clearTimeout(timeoutId);
}
}
await this.waitUntilFinished();
} }
public async waitUntilFinished(): Promise<void> { public async waitUntilFinished(): Promise<void> {
@ -162,6 +120,59 @@ export class WebSocketManager {
} }
} }
private async performStop(): Promise<void> {
const { promise, resolve } = Promise.withResolvers<undefined>();
this.resolveDisconnectingPromise = (): void => {
resolve(undefined);
};
this.isStopped = true;
if (this.reconnectTimeoutId !== undefined) {
clearTimeout(this.reconnectTimeoutId);
this.reconnectTimeoutId = undefined;
}
if (this.connectionTimeoutId !== undefined) {
clearTimeout(this.connectionTimeoutId);
this.connectionTimeoutId = undefined;
}
this.webSocket?.close(1000, "WebSocketManager has been stopped");
// eslint-disable-next-line @typescript-eslint/init-declarations
let timeoutId: ReturnType<typeof setTimeout> | undefined;
const timeoutPromise = new Promise<void>((_, reject) => {
timeoutId = setTimeout(() => {
reject(
new Error(
`Timeout waiting for WebSocket to close after ${WEBSOCKET_DISCONNECT_TIMEOUT_IN_SECONDS} seconds`
)
);
}, WEBSOCKET_DISCONNECT_TIMEOUT_IN_SECONDS * 1000);
});
try {
while (this.isWebSocketConnected) {
await Promise.race([promise, timeoutPromise]);
}
} catch (error) {
this.logger.error(
`Error while waiting for WebSocket to close: ${String(error)}`
);
// Force cleanup even if close didn't work
this.resolveDisconnectingPromise();
this.resolveDisconnectingPromise = null;
} finally {
// Clear timeout to prevent unhandled rejection
if (timeoutId !== undefined) {
clearTimeout(timeoutId);
}
}
await this.waitUntilFinished();
}
private initializeWebSocket(): void { private initializeWebSocket(): void {
// Clean up old WebSocket handlers to prevent race conditions // Clean up old WebSocket handlers to prevent race conditions
if (this.webSocket) { if (this.webSocket) {
@ -171,26 +182,55 @@ export class WebSocketManager {
this.webSocket.onclose = null; this.webSocket.onclose = null;
this.webSocket.onmessage = null; this.webSocket.onmessage = null;
this.webSocket.onerror = null; this.webSocket.onerror = null;
this.webSocket.close(); this.webSocket.close(
1000,
"Closing previous WebSocket connection"
);
} catch (e) { } catch (e) {
this.logger.error( this.logger.error(
`Failed to close previous WebSocket connection: ${e}` `Failed to close previous WebSocket connection: ${e}`
); );
} }
// Abandon any outstanding handler promises from the previous
// connection. They'll still resolve in the background, but we
// no longer want `waitUntilFinished` / `stop` to block on
// post-reconnect state — and we definitely don't want their
// results applied against a now-stale socket.
this.outstandingPromises.length = 0;
} }
const wsUri = new URL(this.settings.getSettings().remoteUri); // Build the WS URL through the same vault-URL helper the HTTP client
wsUri.protocol = wsUri.protocol === "https" ? "wss" : "ws"; // uses so vault-name encoding, trailing-slash stripping, and any path
wsUri.pathname = `/vaults/${this.settings.getSettings().vaultName}/ws`; // prefix in `remoteUri` stay in sync between transports.
const wsUri = new URL(buildVaultUrl(this.settings, "/ws"));
wsUri.protocol = wsUri.protocol.startsWith("https") ? "wss" : "ws";
this.logger.info(`Connecting to WebSocket at ${wsUri.toString()}`); this.logger.info(`Connecting to WebSocket at ${wsUri.toString()}`);
this.webSocket = new this.webSocketFactoryImplementation(wsUri); const ws = new this.webSocketFactoryImplementation(wsUri);
this.webSocket = ws;
// Set connection timeout to handle cases where server is down and the WebSocket connection won't open.
// The callback closes the *captured* `ws` rather than `this.webSocket` so a delayed timeout cannot
// accidentally close a freshly-constructed replacement socket. (Closing the already-closed `ws` is a no-op.)
this.connectionTimeoutId = setTimeout(() => {
this.connectionTimeoutId = undefined;
this.logger.warn(
`WebSocket connection timeout after ${WEBSOCKET_CONNECTION_TIMEOUT_IN_SECONDS} seconds`
);
// Force close to trigger onclose handler which will schedule reconnection
ws.close(1000, "Connection timeout");
}, WEBSOCKET_CONNECTION_TIMEOUT_IN_SECONDS * 1000);
ws.onopen = (): void => {
if (this.connectionTimeoutId !== undefined) {
clearTimeout(this.connectionTimeoutId);
this.connectionTimeoutId = undefined;
}
this.webSocket.onopen = (): void => {
// Check if we've been stopped while connecting // Check if we've been stopped while connecting
if (this.isStopped) { if (this.isStopped) {
this.webSocket?.close( ws.close(
1000, 1000,
"WebSocketManager was stopped during connection" "WebSocketManager was stopped during connection"
); );
@ -200,7 +240,7 @@ export class WebSocketManager {
this.onWebSocketStatusChanged.trigger(true); this.onWebSocketStatusChanged.trigger(true);
}; };
this.webSocket.onmessage = (event): void => { ws.onmessage = (event): void => {
try { try {
// eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion // eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
const message = JSON.parse( const message = JSON.parse(
@ -231,7 +271,18 @@ export class WebSocketManager {
} }
}; };
this.webSocket.onclose = (event): void => { ws.onerror = (error): void => {
this.logger.warn(
`WebSocket error occurred: ${error instanceof ErrorEvent ? error.message : "Unknown error"}`
);
};
ws.onclose = (event): void => {
if (this.connectionTimeoutId !== undefined) {
clearTimeout(this.connectionTimeoutId);
this.connectionTimeoutId = undefined;
}
this.logger.warn( this.logger.warn(
`WebSocket closed with code ${event.code} (${event.reason == "" ? "unknown reason" : event.reason})` `WebSocket closed with code ${event.code} (${event.reason == "" ? "unknown reason" : event.reason})`
); );
@ -241,10 +292,13 @@ export class WebSocketManager {
this.resolveDisconnectingPromise?.(); this.resolveDisconnectingPromise?.();
this.resolveDisconnectingPromise = null; this.resolveDisconnectingPromise = null;
} else { } else {
const delay =
this.settings.getSettings().webSocketRetryIntervalMs;
this.logger.info(`Reconnecting to WebSocket in ${delay}ms...`);
this.reconnectTimeoutId = setTimeout(() => { this.reconnectTimeoutId = setTimeout(() => {
this.reconnectTimeoutId = undefined; this.reconnectTimeoutId = undefined;
this.initializeWebSocket(); this.initializeWebSocket();
}, this.settings.getSettings().webSocketRetryIntervalMs); }, delay);
} }
}; };
} }
@ -252,19 +306,19 @@ export class WebSocketManager {
private async handleWebSocketMessage( private async handleWebSocketMessage(
message: WebSocketServerMessage message: WebSocketServerMessage
): Promise<void> { ): Promise<void> {
if (message.type === "vaultUpdate") { switch (message.type) {
case "vaultUpdate":
await this.onRemoteVaultUpdateReceived.triggerAsync(message); await this.onRemoteVaultUpdateReceived.triggerAsync(message);
return;
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition case "cursorPositions":
} else if (message.type === "cursorPositions") {
this.logger.debug( this.logger.debug(
`Received cursor positions for ${JSON.stringify(message.clients)}` `Received cursor positions for ${JSON.stringify(message.clients)}`
); );
await this.onRemoteCursorsUpdateReceived.triggerAsync( await this.onRemoteCursorsUpdateReceived.triggerAsync(
message.clients message.clients
); );
} else { return;
default:
this.logger.warn( this.logger.warn(
`Received unknown message type: ${JSON.stringify(message)}` `Received unknown message type: ${JSON.stringify(message)}`
); );

View file

@ -2,8 +2,12 @@ import type { PersistenceProvider } from "./persistence/persistence";
import type { HistoryEntry, HistoryStats } from "./tracing/sync-history"; import type { HistoryEntry, HistoryStats } from "./tracing/sync-history";
import { SyncHistory } from "./tracing/sync-history"; import { SyncHistory } from "./tracing/sync-history";
import { Logger, LogLevel, LogLine } from "./tracing/logger"; import { Logger, LogLevel, LogLine } from "./tracing/logger";
import type { RelativePath, StoredDatabase } from "./persistence/database"; import type {
import { Database } from "./persistence/database"; DocumentId,
RelativePath,
StoredSyncState
} from "./sync-operations/types";
import { SyncEventQueue } from "./sync-operations/sync-event-queue";
import * as Sentry from "@sentry/browser"; import * as Sentry from "@sentry/browser";
import type { SyncSettings } from "./persistence/settings"; import type { SyncSettings } from "./persistence/settings";
import { DEFAULT_SETTINGS, Settings } from "./persistence/settings"; import { DEFAULT_SETTINGS, Settings } from "./persistence/settings";
@ -12,7 +16,6 @@ import { Syncer } from "./sync-operations/syncer";
import type { FileSystemOperations } from "./file-operations/filesystem-operations"; import type { FileSystemOperations } from "./file-operations/filesystem-operations";
import { FileOperations } from "./file-operations/file-operations"; import { FileOperations } from "./file-operations/file-operations";
import { FetchController } from "./services/fetch-controller"; import { FetchController } from "./services/fetch-controller";
import { UnrestrictedSyncer } from "./sync-operations/unrestricted-syncer";
import { rateLimit } from "./utils/rate-limit"; import { rateLimit } from "./utils/rate-limit";
import type { NetworkConnectionStatus } from "./types/network-connection-status"; import type { NetworkConnectionStatus } from "./types/network-connection-status";
import { DocumentSyncStatus } from "./types/document-sync-status"; import { DocumentSyncStatus } from "./types/document-sync-status";
@ -24,42 +27,46 @@ import type { MaybeOutdatedClientCursors } from "./types/maybe-outdated-client-c
import { FileChangeNotifier } from "./sync-operations/file-change-notifier"; import { FileChangeNotifier } from "./sync-operations/file-change-notifier";
import { FixedSizeDocumentCache } from "./utils/data-structures/fix-sized-cache"; import { FixedSizeDocumentCache } from "./utils/data-structures/fix-sized-cache";
import { setUpTelemetry } from "./utils/set-up-telemetry"; import { setUpTelemetry } from "./utils/set-up-telemetry";
import { DIFF_CACHE_SIZE_MB } from "./consts";
import { ServerConfig } from "./services/server-config"; import { ServerConfig } from "./services/server-config";
import type { EventListeners } from "./utils/data-structures/event-listeners"; import type { EventListeners } from "./utils/data-structures/event-listeners";
import { Lock } from "./utils/data-structures/locks";
import { ExpectedFsEvents } from "./sync-operations/expected-fs-events";
export class SyncClient { export class SyncClient {
private hasStartedOfflineSync = false;
private hasFinishedOfflineSync = false; private hasFinishedOfflineSync = false;
private hasStarted = false; private hasStarted = false;
private hasBeenDestroyed = false; private hasBeenDestroyed = false;
private unloadTelemetry?: () => void; private unloadTelemetry?: () => void;
private isDestroying = false; private isDestroying = false;
private readonly eventUnsubscribers: (() => void)[] = []; private readonly eventUnsubscribers: (() => void)[] = [];
private readonly settingsChangeLock = new Lock(
"SyncClient.onSettingsChange"
);
private constructor( private constructor(
public readonly logger: Logger,
private readonly history: SyncHistory, private readonly history: SyncHistory,
private readonly settings: Settings, private readonly settings: Settings,
private readonly database: Database, private readonly syncEventQueue: SyncEventQueue,
private readonly syncer: Syncer, private readonly syncer: Syncer,
private readonly webSocketManager: WebSocketManager, private readonly webSocketManager: WebSocketManager,
public readonly logger: Logger,
private readonly fetchController: FetchController, private readonly fetchController: FetchController,
private readonly cursorTracker: CursorTracker, private readonly cursorTracker: CursorTracker,
private readonly fileChangeNotifier: FileChangeNotifier, private readonly fileChangeNotifier: FileChangeNotifier,
private readonly contentCache: FixedSizeDocumentCache, private readonly contentCache: FixedSizeDocumentCache,
private readonly fileOperations: FileOperations,
private readonly serverConfig: ServerConfig, private readonly serverConfig: ServerConfig,
private readonly syncService: SyncService,
private readonly expectedFsEvents: ExpectedFsEvents,
private readonly persistence: PersistenceProvider< private readonly persistence: PersistenceProvider<
Partial<{ Partial<{
settings: Partial<SyncSettings>; settings: Partial<SyncSettings>;
database: Partial<StoredDatabase>; database: Partial<StoredSyncState>;
}> }>
> >
) {} ) {}
public get documentCount(): number { public get syncedDocumentCount(): number {
return this.database.length; return this.syncEventQueue.syncedDocumentCount;
} }
public get isWebSocketConnected(): boolean { public get isWebSocketConnected(): boolean {
@ -73,6 +80,27 @@ export class SyncClient {
return this.history.onHistoryUpdated; return this.history.onHistoryUpdated;
} }
/**
* Fires whenever a tracked document's local file moves on disk
* watcher-driven user renames, post-create deconflicts placed by
* the reconciler, lost-rename replays in offline scan, slot
* displacements when another record claims a path. Both
* `oldPath` and `newPath` may be `undefined` (placement-pending
* state). Useful for callers that mirror disk-side path state
* e.g. test harnesses tracking which paths are safe to mutate
* and need a signal beyond the user-facing history.
*/
public get onDocumentPathChanged(): EventListeners<
(
documentId: DocumentId,
oldPath: RelativePath | undefined,
newPath: RelativePath | undefined
) => unknown
> {
this.checkIfDestroyed("onDocumentPathChanged getter");
return this.syncEventQueue.onDocumentPathChanged;
}
public get onSettingsChanged(): EventListeners< public get onSettingsChanged(): EventListeners<
(newSettings: SyncSettings, oldSettings: SyncSettings) => unknown (newSettings: SyncSettings, oldSettings: SyncSettings) => unknown
> { > {
@ -101,6 +129,13 @@ export class SyncClient {
return this.cursorTracker.onRemoteCursorsUpdated; return this.cursorTracker.onRemoteCursorsUpdated;
} }
public get hasPendingWork(): boolean {
return (
this.syncEventQueue.pendingUpdateCount > 0 ||
this.webSocketManager.hasOutstandingWork
);
}
public static async create({ public static async create({
fs, fs,
persistence, persistence,
@ -112,7 +147,8 @@ export class SyncClient {
persistence: PersistenceProvider< persistence: PersistenceProvider<
Partial<{ Partial<{
settings: Partial<SyncSettings>; settings: Partial<SyncSettings>;
database: Partial<StoredDatabase>; database: Partial<StoredSyncState>;
deviceId: string;
}> }>
>; >;
fetch?: typeof globalThis.fetch; fetch?: typeof globalThis.fetch;
@ -121,39 +157,46 @@ export class SyncClient {
}): Promise<SyncClient> { }): Promise<SyncClient> {
const logger = new Logger(); const logger = new Logger();
const deviceId = createClientId();
logger.info(`Creating SyncClient with client id ${deviceId}`);
const history = new SyncHistory(logger); const history = new SyncHistory(logger);
let state = (await persistence.load()) ?? { let state = (await persistence.load()) ?? {
settings: undefined, settings: undefined,
database: undefined database: undefined,
deviceId: undefined
}; };
// Persist deviceId across destroy + init so the server's
// lost-create dedup (which scopes by device_id) can recognise
// a retry as belonging to the same client. Without this,
// every fresh `SyncClient` after a destroy would generate a
// new deviceId, the server-side query would miss, and the
// pending-but-lost create would deconflict instead of
// binding to the doc its content was already absorbed into.
let deviceId = state.deviceId;
if (deviceId === undefined) {
deviceId = createClientId();
state = { ...state, deviceId };
await persistence.save(state);
}
logger.info(`Creating SyncClient with client id ${deviceId}`);
const settings = new Settings( const settings = new Settings(
logger, logger,
state.settings, state.settings,
async (data): Promise<void> => { async (data): Promise<void> => {
state = { ...state, settings: data }; state = { ...state, settings: data };
// we're not rate-limiting settings saves as (1) we need to initialise the settings to know the rate limit
// and (2) settings changes are infrequent enough that rate-limiting is not necessary
await persistence.save(state); await persistence.save(state);
} }
); );
const rateLimitedSave = rateLimit( const syncEventQueue = new SyncEventQueue(
persistence.save, settings,
() => settings.getSettings().minimumSaveIntervalMs
);
const database = new Database(
logger, logger,
state.database, state.database,
async (data): Promise<void> => { async (data): Promise<void> => {
state = { ...state, database: data }; state = { ...state, database: data };
await rateLimitedSave(state); await persistence.save(state);
} }
); );
@ -170,32 +213,23 @@ export class SyncClient {
fetch fetch
); );
const serverConfig = new ServerConfig(syncService); const serverConfig = new ServerConfig(syncService, settings);
const expectedFsEvents = new ExpectedFsEvents();
const fileOperations = new FileOperations( const fileOperations = new FileOperations(
logger, logger,
database,
fs, fs,
serverConfig, serverConfig,
expectedFsEvents,
nativeLineEndings nativeLineEndings
); );
const contentCache = new FixedSizeDocumentCache( const contentCache = new FixedSizeDocumentCache(
1024 * 1024 * DIFF_CACHE_SIZE_MB 1024 * 1024 * settings.getSettings().diffCacheSizeMB
);
const unrestrictedSyncer = new UnrestrictedSyncer(
logger,
database,
settings,
syncService,
fileOperations,
history,
contentCache,
serverConfig
); );
const webSocketManager = new WebSocketManager( const webSocketManager = new WebSocketManager(
deviceId,
logger, logger,
settings, settings,
webSocket webSocket
@ -204,34 +238,38 @@ export class SyncClient {
const syncer = new Syncer( const syncer = new Syncer(
deviceId, deviceId,
logger, logger,
database,
settings, settings,
syncService,
webSocketManager, webSocketManager,
fileOperations, fileOperations,
unrestrictedSyncer syncService,
history,
contentCache,
serverConfig,
syncEventQueue
); );
const fileChangeNotifier = new FileChangeNotifier(); const fileChangeNotifier = new FileChangeNotifier();
const cursorTracker = new CursorTracker( const cursorTracker = new CursorTracker(
database, logger,
syncEventQueue,
webSocketManager, webSocketManager,
fileOperations, fileOperations,
fileChangeNotifier fileChangeNotifier
); );
const client = new SyncClient( const client = new SyncClient(
logger,
history, history,
settings, settings,
database, syncEventQueue,
syncer, syncer,
webSocketManager, webSocketManager,
logger,
fetchController, fetchController,
cursorTracker, cursorTracker,
fileChangeNotifier, fileChangeNotifier,
contentCache, contentCache,
fileOperations,
serverConfig, serverConfig,
syncService,
expectedFsEvents,
persistence persistence
); );
@ -321,7 +359,7 @@ export class SyncClient {
/** /**
* Wait for the in-flight operations to finish, reset all tracking, * Wait for the in-flight operations to finish, reset all tracking,
* and the local database but retain the settings. * and the local state but retain the settings.
* The SyncClient can be used again after calling this method. * The SyncClient can be used again after calling this method.
*/ */
public async reset(): Promise<void> { public async reset(): Promise<void> {
@ -332,17 +370,17 @@ export class SyncClient {
); );
await this.pause(); await this.pause();
// clear all local state
this.logger.info("Resetting SyncClient's local state"); this.logger.info("Resetting SyncClient's local state");
this.database.reset(); await this.syncEventQueue.clearAllState();
await this.database.save(); // ensure the new database reads as empty await this.syncEventQueue.save();
this.resetInMemoryState(); this.resetInMemoryState();
this.hasStartedOfflineSync = false;
this.hasFinishedOfflineSync = false; this.hasFinishedOfflineSync = false;
this.serverConfig.reset(); this.serverConfig.reset();
if (this.settings.getSettings().isSyncEnabled) {
await this.startSyncing(); await this.startSyncing();
} }
}
public getSettings(): SyncSettings { public getSettings(): SyncSettings {
return this.settings.getSettings(); return this.settings.getSettings();
@ -363,40 +401,48 @@ export class SyncClient {
await this.settings.setSettings(value); await this.settings.setSettings(value);
} }
public async syncLocallyCreatedFile( public syncLocallyCreatedFile(relativePath: RelativePath): void {
relativePath: RelativePath
): Promise<void> {
this.checkIfDestroyed("syncLocallyCreatedFile"); this.checkIfDestroyed("syncLocallyCreatedFile");
this.fileChangeNotifier.notifyOfFileChange(relativePath); this.fileChangeNotifier.notifyOfFileChange(relativePath); // this is for updating cursors
return this.syncer.syncLocallyCreatedFile(relativePath); if (this.expectedFsEvents.matchCreate(relativePath)) {
return;
} }
public async syncLocallyDeletedFile( this.syncer.syncLocallyCreatedFile(relativePath);
relativePath: RelativePath
): Promise<void> {
this.checkIfDestroyed("syncLocallyDeletedFile");
this.fileChangeNotifier.notifyOfFileChange(relativePath);
return this.syncer.syncLocallyDeletedFile(relativePath);
} }
public async syncLocallyUpdatedFile({ public syncLocallyUpdatedFile({
oldPath, oldPath,
relativePath relativePath
}: { }: {
oldPath?: RelativePath; oldPath?: RelativePath;
relativePath: RelativePath; relativePath: RelativePath;
}): Promise<void> { }): void {
this.checkIfDestroyed("syncLocallyUpdatedFile"); this.checkIfDestroyed("syncLocallyUpdatedFile");
this.fileChangeNotifier.notifyOfFileChange(relativePath); this.fileChangeNotifier.notifyOfFileChange(relativePath); // this is for updating cursors
return this.syncer.syncLocallyUpdatedFile({ if (this.expectedFsEvents.matchUpdate(relativePath, oldPath)) {
return;
}
this.syncer.syncLocallyUpdatedFile({
oldPath, oldPath,
relativePath relativePath
}); });
} }
public syncLocallyDeletedFile(relativePath: RelativePath): void {
this.checkIfDestroyed("syncLocallyDeletedFile");
this.fileChangeNotifier.notifyOfFileChange(relativePath); // this is for updating cursors
if (this.expectedFsEvents.matchDelete(relativePath)) {
return;
}
this.syncer.syncLocallyDeletedFile(relativePath);
}
public getDocumentSyncingStatus( public getDocumentSyncingStatus(
relativePath: RelativePath relativePath: RelativePath
): DocumentSyncStatus { ): DocumentSyncStatus {
@ -406,16 +452,11 @@ export class SyncClient {
return DocumentSyncStatus.SYNCING_IS_DISABLED; return DocumentSyncStatus.SYNCING_IS_DISABLED;
} }
if (!this.syncer.isFirstSyncComplete || !this.hasFinishedOfflineSync) { if (!this.hasFinishedOfflineSync) {
return DocumentSyncStatus.SYNCING; return DocumentSyncStatus.SYNCING;
} }
const document = return this.syncEventQueue.hasPendingEventsForPath(relativePath)
this.database.getLatestDocumentByRelativePath(relativePath);
if (document === undefined) {
return DocumentSyncStatus.SYNCING;
}
return document.updates.length > 0
? DocumentSyncStatus.SYNCING ? DocumentSyncStatus.SYNCING
: DocumentSyncStatus.UP_TO_DATE; : DocumentSyncStatus.UP_TO_DATE;
} }
@ -429,10 +470,8 @@ export class SyncClient {
} }
public async waitUntilFinished(): Promise<void> { public async waitUntilFinished(): Promise<void> {
this.checkIfDestroyed("waitUntilIdle"); this.checkIfDestroyed("waitUntilFinished");
await this.syncer.waitUntilFinished(); await this.waitUntilFinishedInternal();
await this.webSocketManager.waitUntilFinished();
await this.database.save(); // flush all changes to disk
} }
/** /**
@ -440,9 +479,11 @@ export class SyncClient {
* After calling this method, the SyncClient cannot be used again. * After calling this method, the SyncClient cannot be used again.
*/ */
public async destroy(): Promise<void> { public async destroy(): Promise<void> {
this.checkIfDestroyed("destroy"); if (this.hasBeenDestroyed) {
throw new Error(
// Prevent concurrent destroy calls "SyncClient has been destroyed and can no longer be used; called from destroy"
);
}
if (this.isDestroying) { if (this.isDestroying) {
this.logger.warn( this.logger.warn(
"destroy() called while already destroying, ignoring" "destroy() called while already destroying, ignoring"
@ -451,14 +492,16 @@ export class SyncClient {
} }
this.isDestroying = true; this.isDestroying = true;
// cancel everything that's in progress // Run cleanup in `finally` so a thrown pause() — or anything else
// mid-shutdown — still leaves the client in the disposed state
// instead of bricked with subscribers/telemetry hanging on.
try {
await this.pause(); await this.pause();
} finally {
this.hasBeenDestroyed = true; this.hasBeenDestroyed = true;
this.resetInMemoryState(); this.resetInMemoryState();
// Clean up event listeners to prevent memory leaks
this.eventUnsubscribers.forEach((unsubscribe) => { this.eventUnsubscribers.forEach((unsubscribe) => {
unsubscribe(); unsubscribe();
}); });
@ -468,35 +511,73 @@ export class SyncClient {
this.unloadTelemetry?.(); this.unloadTelemetry?.();
} }
}
/**
* The actual drain separated from `waitUntilFinished` so internal
* shutdown paths (`pause` / `destroy`) can wait for in-flight work
* without tripping the public `checkIfDestroyed` guard, which exists
* only to keep external callers from continuing to use a disposed
* client.
*
* Loops because a WebSocket message handler completing is what enqueues
* a `RemoteChange` into the syncer; if we awaited the syncer first and
* the WS handler second, a message arriving mid-wait would leave a fresh
* drain pending while `save()` ran. Each iteration waits for both, then
* re-checks; we exit only once both report idle in the same pass.
*/
private async waitUntilFinishedInternal(): Promise<void> {
while (
this.webSocketManager.hasOutstandingWork ||
this.syncer.hasPendingWork
) {
await this.webSocketManager.waitUntilFinished();
await this.syncer.waitUntilFinished();
}
await this.syncEventQueue.save();
}
private async startSyncing(): Promise<void> { private async startSyncing(): Promise<void> {
this.checkIfDestroyed("startSyncing"); this.checkIfDestroyed("startSyncing");
this.fetchController.finishReset(); this.fetchController.finishReset();
// Undo any earlier `pause()` stop so retryForever keeps retrying.
this.syncService.resume();
await this.serverConfig.initialize(); await this.serverConfig.getConfig();
this.webSocketManager.start();
if (!this.hasStartedOfflineSync) {
this.hasStartedOfflineSync = true;
await this.syncer.scheduleSyncForOfflineChanges(); await this.syncer.scheduleSyncForOfflineChanges();
} this.syncer.resumeDraining();
this.webSocketManager.start();
this.hasFinishedOfflineSync = true; this.hasFinishedOfflineSync = true;
} }
private async pause(): Promise<void> { private async pause(): Promise<void> {
this.hasFinishedOfflineSync = false;
this.syncer.pauseDraining();
this.fetchController.startReset(); this.fetchController.startReset();
// Signal the service so any `retryForever` loop exits at its next
// iteration instead of continuing to retry a network request while
// the rest of the client is winding down.
this.syncService.stop();
await this.webSocketManager.stop(); await this.webSocketManager.stop();
await this.waitUntilFinished(); await this.waitUntilFinishedInternal();
// Clear the offline-scan gate so a subsequent `startSyncing()`
// re-runs the scan; otherwise any local changes made while sync was
// paused (offline edits, deletes, renames) wouldn't be detected, and
// an incoming remote update would silently overwrite them.
this.syncer.clearOfflineScanGate();
// Drop any expected fs events that were registered but never matched
// (e.g. an op aborted by SyncResetError). Otherwise a real user edit
// at the same path after re-enable would be swallowed.
this.expectedFsEvents.clear();
} }
private resetInMemoryState(): void { private resetInMemoryState(): void {
this.history.reset(); this.history.reset();
this.contentCache.reset(); this.contentCache.reset();
// don't reset the logger
this.cursorTracker.reset(); this.cursorTracker.reset();
this.syncer.reset(); this.syncer.reset();
this.fileOperations.reset();
} }
private async onSettingsChange( private async onSettingsChange(
@ -505,14 +586,26 @@ export class SyncClient {
): Promise<void> { ): Promise<void> {
this.checkIfDestroyed("onSettingsChange"); this.checkIfDestroyed("onSettingsChange");
if ( // Serialize listener invocations so back-to-back settings updates
newSettings.vaultName !== oldSettings.vaultName || // can't run reset()/pause()/startSyncing() concurrently.
newSettings.remoteUri !== oldSettings.remoteUri await this.settingsChangeLock.withLock(async () => {
) { // The lock is FIFO, so by the time we run the client may have
await this.reset(); // been destroyed in a queued invocation ahead of us.
if (this.hasBeenDestroyed) {
return;
} }
if (newSettings.isSyncEnabled !== oldSettings.isSyncEnabled) { const connectionChanged =
newSettings.vaultName !== oldSettings.vaultName ||
newSettings.remoteUri !== oldSettings.remoteUri;
if (connectionChanged) {
// reset() pauses, clears state, then starts iff isSyncEnabled
// — so any concurrent isSyncEnabled change is already applied.
await this.reset();
} else if (
newSettings.isSyncEnabled !== oldSettings.isSyncEnabled
) {
if (newSettings.isSyncEnabled) { if (newSettings.isSyncEnabled) {
await this.startSyncing(); await this.startSyncing();
} else { } else {
@ -521,7 +614,9 @@ export class SyncClient {
} }
if (newSettings.diffCacheSizeMB !== oldSettings.diffCacheSizeMB) { if (newSettings.diffCacheSizeMB !== oldSettings.diffCacheSizeMB) {
this.contentCache.resize(newSettings.diffCacheSizeMB * 1024 * 1024); this.contentCache.resize(
newSettings.diffCacheSizeMB * 1024 * 1024
);
} }
if (newSettings.enableTelemetry !== oldSettings.enableTelemetry) { if (newSettings.enableTelemetry !== oldSettings.enableTelemetry) {
@ -531,10 +626,15 @@ export class SyncClient {
this.unloadTelemetry?.(); this.unloadTelemetry?.();
} }
} }
});
} }
private checkIfDestroyed(origin: string): void { private checkIfDestroyed(origin: string): void {
if (this.hasBeenDestroyed) { // Reject new public-API entries the moment destroy() is called,
// not after `pause()` returns. Otherwise an external caller could
// pass the guard and start mutating state while destroy() is
// tearing down the websocket / clearing caches.
if (this.hasBeenDestroyed || this.isDestroying) {
throw new Error( throw new Error(
`SyncClient has been destroyed and can no longer be used; called from ${origin}` `SyncClient has been destroyed and can no longer be used; called from ${origin}`
); );

View file

@ -1,5 +1,6 @@
import type { FileOperations } from "../file-operations/file-operations"; import type { FileOperations } from "../file-operations/file-operations";
import type { Database, RelativePath } from "../persistence/database"; import type { RelativePath } from "./types";
import type { SyncEventQueue } from "./sync-event-queue";
import type { ClientCursors } from "../services/types/ClientCursors"; import type { ClientCursors } from "../services/types/ClientCursors";
import type { CursorSpan } from "../services/types/CursorSpan"; import type { CursorSpan } from "../services/types/CursorSpan";
import type { DocumentWithCursors } from "../services/types/DocumentWithCursors"; import type { DocumentWithCursors } from "../services/types/DocumentWithCursors";
@ -10,6 +11,7 @@ import { hash } from "../utils/hash";
import type { FileChangeNotifier } from "./file-change-notifier"; import type { FileChangeNotifier } from "./file-change-notifier";
import { Lock } from "../utils/data-structures/locks"; import { Lock } from "../utils/data-structures/locks";
import { EventListeners } from "../utils/data-structures/event-listeners"; import { EventListeners } from "../utils/data-structures/event-listeners";
import type { Logger } from "../tracing/logger";
// Cursor positions are updated separately from documents. However, a given cursor position is only // Cursor positions are updated separately from documents. However, a given cursor position is only
// valid within a certain version of the document it belongs to. This class tracks previous and the latest // valid within a certain version of the document it belongs to. This class tracks previous and the latest
@ -22,22 +24,29 @@ export class CursorTracker {
(cursors: MaybeOutdatedClientCursors[]) => unknown (cursors: MaybeOutdatedClientCursors[]) => unknown
>(); >();
private readonly updateLock = new Lock(); private readonly updateLock: Lock;
private knownRemoteCursors: (ClientCursors & { private knownRemoteCursors: (ClientCursors & {
upToDateness: DocumentUpToDateness; upToDateness: DocumentUpToDateness;
})[] = []; })[] = [];
private lastLocalCursorState: DocumentWithCursors[] = []; // Cache the previously sent state as a JSON string rather than as the
private lastLocalCursorStateWithoutDirtyDocuments: DocumentWithCursors[] = // array. We mutate `documentsWithCursors` in-place after the cache check
[]; // (setting `vaultUpdateId = null` for dirty docs); storing the array would
// alias and the next call's equality check would compare against
// post-mutation state.
private lastLocalCursorStateJson = "[]";
private lastLocalCursorStateWithoutDirtyDocumentsJson = "[]";
public constructor( public constructor(
private readonly database: Database, logger: Logger,
private readonly queue: SyncEventQueue,
private readonly webSocketManager: WebSocketManager, private readonly webSocketManager: WebSocketManager,
private readonly fileOperations: FileOperations, private readonly fileOperations: FileOperations,
private readonly fileChangeNotifier: FileChangeNotifier private readonly fileChangeNotifier: FileChangeNotifier
) { ) {
this.updateLock = new Lock(CursorTracker.name, logger);
this.webSocketManager.onRemoteCursorsUpdateReceived.add( this.webSocketManager.onRemoteCursorsUpdateReceived.add(
async (clientCursors) => { async (clientCursors) => {
await this.updateLock.withLock(async () => { await this.updateLock.withLock(async () => {
@ -53,7 +62,7 @@ export class CursorTracker {
for (const cursor of clientCursors.filter((client) => for (const cursor of clientCursors.filter((client) =>
client.documentsWithCursors.every( client.documentsWithCursors.every(
(doc) => doc.vault_update_id != null (doc) => doc.vaultUpdateId != null
) )
)) { )) {
updatedKnownRemoteCursors.push({ updatedKnownRemoteCursors.push({
@ -77,14 +86,20 @@ export class CursorTracker {
for (const clientCursor of this.knownRemoteCursors) { for (const clientCursor of this.knownRemoteCursors) {
if ( if (
clientCursor.documentsWithCursors.some( clientCursor.documentsWithCursors.some(
(document) => (document) => document.relativePath === relativePath
document.relative_path === relativePath
) )
) { ) {
clientCursor.upToDateness = clientCursor.upToDateness =
await this.getDocumentsUpToDateness(clientCursor); await this.getDocumentsUpToDateness(clientCursor);
} }
} }
// Drop the local-cursor send-cache so the next call re-reads
// the file. The first cache key is the editor's input, which
// doesn't change when the file content does — without this,
// a remote update flipping the file from dirty back to clean
// would never re-send the cursor with a fresh `vaultUpdateId`.
this.lastLocalCursorStateJson = "";
this.lastLocalCursorStateWithoutDirtyDocumentsJson = "";
}) })
); );
} }
@ -95,26 +110,24 @@ export class CursorTracker {
public async sendLocalCursorsToServer( public async sendLocalCursorsToServer(
documentToCursors: Record<RelativePath, CursorSpan[]> documentToCursors: Record<RelativePath, CursorSpan[]>
): Promise<void> { ): Promise<void> {
// Serialise concurrent senders so they don't interleave on the
// disk reads + state mutations and emit out-of-order cursor messages.
await this.updateLock.withLock(async () => {
const documentsWithCursors: DocumentWithCursors[] = []; const documentsWithCursors: DocumentWithCursors[] = [];
for (const [relativePath, cursors] of Object.entries( for (const [relativePath, cursors] of Object.entries(
documentToCursors documentToCursors
)) { )) {
const record = const record = this.queue.getRecordByLocalPath(relativePath);
this.database.getLatestDocumentByRelativePath(relativePath);
if (!record) { if (!record) {
continue; // Let's wait for the file to be created before sending cursors continue; // Let's wait for the file to be created before sending cursors
} }
if (!record.metadata) {
continue; // this is a new document, no need to sync the cursors
}
documentsWithCursors.push({ documentsWithCursors.push({
relative_path: relativePath, relativePath: relativePath,
document_id: record.documentId, documentId: record.documentId,
vault_update_id: record.metadata.parentVersionId, vaultUpdateId: record.parentVersionId,
cursors: cursors.map(({ start, end }) => ({ cursors: cursors.map(({ start, end }) => ({
start: Math.min(start, end), start: Math.min(start, end),
end: Math.max(start, end) end: Math.max(start, end)
@ -122,43 +135,42 @@ export class CursorTracker {
}); });
} }
if ( const beforeJson = JSON.stringify(documentsWithCursors);
JSON.stringify(this.lastLocalCursorState) === if (this.lastLocalCursorStateJson === beforeJson) {
JSON.stringify(documentsWithCursors)
) {
// Caching step to avoid reading the edited files all the time // Caching step to avoid reading the edited files all the time
return; return;
} }
this.lastLocalCursorState = documentsWithCursors; this.lastLocalCursorStateJson = beforeJson;
for (const doc of documentsWithCursors) { for (const doc of documentsWithCursors) {
const readContent = await this.fileOperations.read( const readContent = await this.fileOperations.read(
doc.relative_path doc.relativePath
); );
const record = this.database.getLatestDocumentByRelativePath( const record = this.queue.getRecordByLocalPath(
doc.relative_path doc.relativePath
); );
if (record?.metadata?.hash !== hash(readContent)) { if (record?.remoteHash !== (await hash(readContent))) {
doc.vault_update_id = null; doc.vaultUpdateId = null;
} }
} }
const afterJson = JSON.stringify(documentsWithCursors);
if ( if (
JSON.stringify(this.lastLocalCursorStateWithoutDirtyDocuments) === this.lastLocalCursorStateWithoutDirtyDocumentsJson === afterJson
JSON.stringify(documentsWithCursors)
) { ) {
return; return;
} }
this.lastLocalCursorStateWithoutDirtyDocuments = documentsWithCursors; this.lastLocalCursorStateWithoutDirtyDocumentsJson = afterJson;
this.webSocketManager.updateLocalCursors({ documentsWithCursors }); this.webSocketManager.updateLocalCursors({ documentsWithCursors });
});
} }
public reset(): void { public reset(): void {
this.knownRemoteCursors = []; this.knownRemoteCursors = [];
this.lastLocalCursorState = []; this.lastLocalCursorStateJson = "[]";
this.lastLocalCursorStateWithoutDirtyDocuments = []; this.lastLocalCursorStateWithoutDirtyDocumentsJson = "[]";
this.updateLock.reset(); this.updateLock.reset();
} }
@ -223,35 +235,28 @@ export class CursorTracker {
private async getDocumentUpToDateness( private async getDocumentUpToDateness(
document: DocumentWithCursors document: DocumentWithCursors
): Promise<DocumentUpToDateness> { ): Promise<DocumentUpToDateness> {
const record = this.database.getLatestDocumentByRelativePath( const record = this.queue.getRecordByLocalPath(document.relativePath);
document.relative_path
);
if (!record) { if (!record) {
// the document of the cursor must be from the future // the document of the cursor must be from the future
return DocumentUpToDateness.Later; return DocumentUpToDateness.Later;
} }
if ( if (record.parentVersionId < (document.vaultUpdateId ?? 0)) {
(record.metadata?.parentVersionId ?? 0) <
(document.vault_update_id ?? 0)
) {
return DocumentUpToDateness.Later; return DocumentUpToDateness.Later;
} else if ( } else if ((document.vaultUpdateId ?? 0) < record.parentVersionId) {
(document.vault_update_id ?? 0) <
(record.metadata?.parentVersionId ?? 0)
) {
// the document of the cursor must be from the past // the document of the cursor must be from the past
return DocumentUpToDateness.Prior; return DocumentUpToDateness.Prior;
} }
const currentContent = await this.fileOperations.read( const currentContent = await this.fileOperations.read(
document.relative_path document.relativePath
); );
return this.database.getLatestDocumentByRelativePath( const currentRecord = this.queue.getRecordByLocalPath(
document.relative_path document.relativePath
)?.metadata?.hash === hash(currentContent) );
return currentRecord?.remoteHash === (await hash(currentContent))
? DocumentUpToDateness.UpToDate ? DocumentUpToDateness.UpToDate
: DocumentUpToDateness.Prior; : DocumentUpToDateness.Prior;
} }

View file

@ -0,0 +1,138 @@
import type { RelativePath } from "./types";
/**
* Counter-based registry of filesystem events the syncer is about to
* cause. The syncer's own writes/renames/deletes go through
* `FileOperations`, which calls into the host filesystem; the host then
* fires watcher events that come back through `SyncClient.syncLocallyXxx`.
* Without filtering, those echo events would be re-uploaded to the server
* and broadcast back, producing an unbounded loop.
*
* The fix: every fs call in `FileOperations` registers the event it is
* about to provoke; the matching `syncLocallyXxx` handler consumes it.
* User-initiated edits never register, so they pass through unchanged.
*
* Counts are per (kind, path) so back-to-back syncer ops on the same path
* (e.g. apply remote update then re-apply during convergence) match
* one-for-one. If the watcher never fires for a registered op (e.g. the
* fs throws before notifying), the entry is left behind; `clear()` is
* called on pause/destroy to drop those before they collide with a real
* user event later.
*/
export class ExpectedFsEvents {
private readonly creates = new Map<RelativePath, number>();
private readonly updates = new Map<RelativePath, number>();
private readonly deletes = new Map<RelativePath, number>();
// Renames are keyed by `JSON.stringify({oldPath, newPath})` so the
// delimiter cannot occur inside either path.
private readonly renames = new Map<RelativePath, number>();
private static renameKey(
oldPath: RelativePath,
newPath: RelativePath
): string {
return JSON.stringify({ oldPath, newPath });
}
public expectCreate(path: RelativePath): void {
this.bump(this.creates, path);
}
public expectUpdate(path: RelativePath): void {
this.bump(this.updates, path);
}
public expectDelete(path: RelativePath): void {
this.bump(this.deletes, path);
}
public expectRename(oldPath: RelativePath, newPath: RelativePath): void {
this.bump(this.renames, ExpectedFsEvents.renameKey(oldPath, newPath));
}
/**
* Cancel a previously-registered expectation when the fs op that registered
* it failed before any watcher event could fire. Without this, a leaked
* expectation silently swallows the next genuine user event at the same
* path (or, for renames, the same `oldPath → newPath` pair).
*
* Floored at zero: if the watcher *did* fire (op partially completed) and
* already consumed the entry, the unexpect is a no-op. The fallback is
* acceptable at worst we re-upload a real edit we'd otherwise filter.
*/
public unexpectCreate(path: RelativePath): void {
this.decrement(this.creates, path);
}
public unexpectUpdate(path: RelativePath): void {
this.decrement(this.updates, path);
}
public unexpectDelete(path: RelativePath): void {
this.decrement(this.deletes, path);
}
public unexpectRename(oldPath: RelativePath, newPath: RelativePath): void {
this.decrement(
this.renames,
ExpectedFsEvents.renameKey(oldPath, newPath)
);
}
public matchCreate(path: RelativePath): boolean {
return this.consume(this.creates, path);
}
public matchUpdate(
path: RelativePath,
oldPath: RelativePath | undefined
): boolean {
if (oldPath !== undefined) {
return this.consume(
this.renames,
ExpectedFsEvents.renameKey(oldPath, path)
);
}
return this.consume(this.updates, path);
}
public matchDelete(path: RelativePath): boolean {
return this.consume(this.deletes, path);
}
public clear(): void {
this.creates.clear();
this.updates.clear();
this.deletes.clear();
this.renames.clear();
}
private bump(map: Map<RelativePath, number>, key: RelativePath): void {
map.set(key, (map.get(key) ?? 0) + 1);
}
private consume(
map: Map<RelativePath, number>,
key: RelativePath
): boolean {
const count = map.get(key) ?? 0;
if (count === 0) {
return false;
}
if (count === 1) {
map.delete(key);
} else {
map.set(key, count - 1);
}
return true;
}
private decrement(map: Map<RelativePath, number>, key: RelativePath): void {
const count = map.get(key) ?? 0;
if (count <= 1) {
map.delete(key);
} else {
map.set(key, count - 1);
}
}
}

View file

@ -1,4 +1,4 @@
import type { RelativePath } from "../persistence/database"; import type { RelativePath } from "./types";
import { EventListeners } from "../utils/data-structures/event-listeners"; import { EventListeners } from "../utils/data-structures/event-listeners";
export class FileChangeNotifier { export class FileChangeNotifier {

View file

@ -0,0 +1,185 @@
import { describe, it } from "node:test";
import assert from "node:assert";
import { Logger } from "../tracing/logger";
import { Settings } from "../persistence/settings";
import { STORED_STATE_SCHEMA_VERSION, SyncEventQueue } from "./sync-event-queue";
import { scheduleOfflineChanges } from "./offline-change-detector";
import type { FileOperations } from "../file-operations/file-operations";
import type { RelativePath } from "./types";
const makeQueue = async (): Promise<SyncEventQueue> => {
const logger = new Logger();
const settings = new Settings(logger, {}, async () => {
/* no-op */
});
return new SyncEventQueue(
settings,
logger,
{ schemaVersion: STORED_STATE_SCHEMA_VERSION },
async () => {
/* no-op */
}
);
};
const makeOperations = (
files: Record<string, Uint8Array>
): FileOperations => {
return {
listFilesRecursively: async () => Object.keys(files),
read: async (path: RelativePath) => {
const data = files[path];
if (data === undefined) {
throw new Error(`File not found: ${path}`);
}
return data;
}
} as unknown as FileOperations;
};
describe("scheduleOfflineChanges", () => {
it("does not bind a local file to a placement-pending record whose remoteRelativePath was persisted before the doc moved on the server", async () => {
// The bug: persisted byDocId can carry a placement-pending record
// whose `remoteRelativePath` was saved before the doc was moved
// server-side. After restart, offline-scan running before WS
// catch-up would bind an unrelated local file at that stale path
// to the moved doc and push the user's content as an update —
// silently corrupting the moved doc and stranding the local file.
const queue = await makeQueue();
// Stale placement-pending record: server has moved this doc
// away from "stale-X.md" since this snapshot was saved.
await queue.upsertRecord({
documentId: "MOVED-DOC",
parentVersionId: 5,
remoteRelativePath: "stale-X.md" as RelativePath,
remoteHash: "hash-from-old-state",
localPath: undefined
});
// User has an unrelated local file at the stale path.
const operations = makeOperations({
"stale-X.md": new TextEncoder().encode(
"user's unrelated local content"
)
});
const enqueued: { kind: string; path: string }[] = [];
await scheduleOfflineChanges(
new Logger(),
operations,
queue,
(path) => enqueued.push({ kind: "create", path }),
(args) => enqueued.push({ kind: "update", path: args.relativePath }),
(path) => enqueued.push({ kind: "delete", path })
);
// The local file must become a fresh CREATE — never a hostile
// UPDATE on the moved doc.
assert.deepStrictEqual(enqueued, [
{ kind: "create", path: "stale-X.md" }
]);
// The placement-pending record must remain placement-pending —
// its localPath must not have been bound to the unrelated user
// file. The reconciler will place it correctly once WS catch-up
// updates `remoteRelativePath` to the doc's current location.
const record = queue.getDocumentByDocumentId("MOVED-DOC");
assert.notStrictEqual(record, undefined);
assert.strictEqual(record?.localPath, undefined);
});
it("schedules an update for a local file that matches a settled record's localPath", async () => {
const queue = await makeQueue();
await queue.upsertRecord({
documentId: "SETTLED-DOC",
parentVersionId: 2,
remoteRelativePath: "doc.md" as RelativePath,
remoteHash: "hash",
localPath: "doc.md" as RelativePath
});
const operations = makeOperations({
"doc.md": new TextEncoder().encode("content")
});
const enqueued: { kind: string; path: string }[] = [];
await scheduleOfflineChanges(
new Logger(),
operations,
queue,
(path) => enqueued.push({ kind: "create", path }),
(args) => enqueued.push({ kind: "update", path: args.relativePath }),
(path) => enqueued.push({ kind: "delete", path })
);
assert.deepStrictEqual(enqueued, [
{ kind: "update", path: "doc.md" }
]);
});
it("schedules a delete for a settled record whose local file is missing", async () => {
const queue = await makeQueue();
await queue.upsertRecord({
documentId: "VANISHED-DOC",
parentVersionId: 4,
remoteRelativePath: "gone.md" as RelativePath,
remoteHash: "hash",
localPath: "gone.md" as RelativePath
});
const operations = makeOperations({});
const enqueued: { kind: string; path: string }[] = [];
await scheduleOfflineChanges(
new Logger(),
operations,
queue,
(path) => enqueued.push({ kind: "create", path }),
(args) => enqueued.push({ kind: "update", path: args.relativePath }),
(path) => enqueued.push({ kind: "delete", path })
);
assert.deepStrictEqual(enqueued, [
{ kind: "delete", path: "gone.md" }
]);
});
it("detects an offline rename when an untracked file matches a deleted record's content hash", async () => {
const queue = await makeQueue();
const content = new TextEncoder().encode("body");
const contentHash = await (await import("../utils/hash")).hash(content);
await queue.upsertRecord({
documentId: "DOC-1",
parentVersionId: 5,
remoteRelativePath: "old.md" as RelativePath,
remoteHash: contentHash,
localPath: "old.md" as RelativePath
});
const operations = makeOperations({ "new.md": content });
const enqueued: {
kind: string;
path: string;
oldPath?: string;
}[] = [];
await scheduleOfflineChanges(
new Logger(),
operations,
queue,
(path) => enqueued.push({ kind: "create", path }),
(args) =>
enqueued.push({
kind: "update",
path: args.relativePath,
oldPath: args.oldPath
}),
(path) => enqueued.push({ kind: "delete", path })
);
assert.deepStrictEqual(enqueued, [
{ kind: "update", path: "new.md", oldPath: "old.md" }
]);
});
});

View file

@ -0,0 +1,188 @@
import type { DocumentRecord, RelativePath } from "./types";
import type { Logger } from "../tracing/logger";
import { hash } from "../utils/hash";
import type { FileOperations } from "../file-operations/file-operations";
import { findMatchingFile } from "../utils/find-matching-file";
import type { SyncEventQueue } from "./sync-event-queue";
import { removeFromArray } from "../utils/remove-from-array";
import { FileNotFoundError } from "../errors/file-not-found-error";
/**
* Scans the local filesystem and the document database to determine
* which files were created, updated, moved, or deleted while the
* client was offline, then enqueues the appropriate sync events.
*
* Placement-pending records (`localPath === undefined`) are deliberately
* NOT bound to local files at the same `remoteRelativePath` here. The
* persisted byDocId snapshot can be stale a doc's server-side path
* may have changed since the last save, so binding by stored path would
* fold an unrelated user file into a moved doc and silently corrupt it.
* Local files at those paths fall through to the LocalCreate flow below;
* the server's create_document handler dedupes by path+freshness when
* the doc really is at that path, and otherwise creates a new doc that
* the reconciler places correctly once catch-up updates the stale
* record's `remoteRelativePath`.
*/
export async function scheduleOfflineChanges(
logger: Logger,
operations: FileOperations,
queue: SyncEventQueue,
enqueueCreate: (path: RelativePath) => void,
enqueueUpdate: (args: {
oldPath?: RelativePath;
relativePath: RelativePath;
}) => void,
enqueueDelete: (path: RelativePath) => void
): Promise<void> {
const allLocalFiles = new Set(await operations.listFilesRecursively());
logger.info(`Scheduling sync for ${allLocalFiles.size} local files`);
// `allSettledDocuments()` skips records with `localPath === undefined`
// — those have no local file by definition and don't participate in
// the disk-vs-record diff. The reconciler will place them on its
// next pass.
const allDocuments = queue.allSettledDocuments();
// A doc is "possibly deleted" only if it has no local file. Including
// docs that still exist locally would queue a spurious delete alongside
// the update below.
const locallyPossiblyDeletedFiles: DocumentRecord[] = [];
for (const record of allDocuments.values()) {
// `localPath` is guaranteed non-undefined for entries in
// `allSettledDocuments()`, but narrow explicitly for the type
// checker (and so a future change to that helper doesn't
// silently break this loop).
if (
record.localPath !== undefined &&
!allLocalFiles.has(record.localPath)
) {
locallyPossiblyDeletedFiles.push(record);
}
}
const locallyPossibleCreatedFiles: RelativePath[] = [];
const syncedLocalFiles: RelativePath[] = [];
for (const localFile of allLocalFiles) {
if (allDocuments.has(localFile)) {
syncedLocalFiles.push(localFile);
} else if (queue.hasPendingCreateForPath(localFile)) {
// A LocalCreate for this path is still in flight (no
// record yet — its docId is a Promise). Re-enqueueing
// would fire a second HTTP create that the server then
// deconflicts to a sibling path, leaving the same bytes
// in two docs. Skip; the in-flight create owns this slot.
continue;
} else {
locallyPossibleCreatedFiles.push(localFile);
}
}
const renamedPaths = new Set<RelativePath>();
// Track paths that were in `allLocalFiles` at scan-start but have
// since disappeared. The scan awaits between `listFilesRecursively`
// and each `read`, so a concurrent delete (slow file events, real
// user activity) can vacate a slot mid-scan. Throwing would abort
// the whole scan; nothing to sync for a file that's already gone.
const disappearedPaths = new Set<RelativePath>();
for (const path of locallyPossibleCreatedFiles) {
let content: Uint8Array;
try {
content = await operations.read(path);
} catch (e) {
if (e instanceof FileNotFoundError) {
logger.debug(
`File ${path} disappeared before offline-scan could read it; skipping`
);
disappearedPaths.add(path);
continue;
}
throw e;
}
const contentHash = await hash(content);
const matchingDeletedFile = await findMatchingFile(
contentHash,
locallyPossiblyDeletedFiles
);
if (matchingDeletedFile !== undefined) {
// localPath is guaranteed defined for records in
// locallyPossiblyDeletedFiles (we filtered above).
const oldPath = matchingDeletedFile.localPath;
if (oldPath === undefined) {
continue;
}
logger.debug(
`File ${path} might have been moved from ${oldPath} while offline, scheduling sync to move it`
);
enqueueUpdate({
oldPath,
relativePath: path
});
removeFromArray(locallyPossiblyDeletedFiles, matchingDeletedFile);
renamedPaths.add(path);
}
}
for (const path of locallyPossibleCreatedFiles) {
if (renamedPaths.has(path) || disappearedPaths.has(path)) {
continue;
}
logger.info(
`File ${path} was created while offline, scheduling sync to create it`
);
enqueueCreate(path);
}
for (const item of locallyPossiblyDeletedFiles) {
if (item.localPath === undefined) {
continue;
}
logger.info(
`File ${item.localPath} was deleted while offline, scheduling sync to delete it`
);
enqueueDelete(item.localPath);
}
for (const path of syncedLocalFiles) {
const record = allDocuments.get(path);
if (
record !== undefined &&
record.localPath !== undefined &&
record.localPath !== record.remoteRelativePath &&
!allLocalFiles.has(record.remoteRelativePath) &&
queue.byLocalPath.get(record.remoteRelativePath) === undefined
) {
// Lost local-rename recovery. The record's `localPath`
// (where the user has the file now) and
// `remoteRelativePath` (where the server still thinks it
// lives) disagree, which means a queued user-rename's
// LocalUpdate never reached the server before the queue
// was wiped (typically a sync reset). Without this
// branch the next `enqueueUpdate({ relativePath: path })`
// is a content-only update — server keeps the doc at the
// old path, the user's file at the new path orphans, and
// other clients never see the rename. Replay the rename
// by restoring the OLD localPath so the queue's enqueue
// can find the record by `oldPath`, then enqueueUpdate
// moves it back to the new path with `isUserRename`.
// Only fires when the old slot is genuinely empty
// (neither on disk nor claimed by another tracked
// record) — otherwise the rename target is occupied and
// we'd be confusing the byLocalPath index.
const oldPath = record.remoteRelativePath;
const newPath = record.localPath;
logger.info(
`Lost local rename detected: doc ${record.documentId} at ${oldPath} (server) vs ${newPath} (local); replaying rename to server`
);
await queue.setLocalPath(record.documentId, oldPath);
enqueueUpdate({ oldPath, relativePath: newPath });
continue;
}
logger.info(
`File ${path} may have been updated while offline, scheduling sync to update it`
);
enqueueUpdate({ relativePath: path });
}
}

View file

@ -0,0 +1,69 @@
import { describe, it } from "node:test";
import assert from "node:assert";
import { Logger, LogLevel } from "../tracing/logger";
import { Settings } from "../persistence/settings";
import { STORED_STATE_SCHEMA_VERSION, SyncEventQueue } from "./sync-event-queue";
import { Reconciler } from "./reconciler";
import { SyncResetError } from "../errors/sync-reset-error";
import type { FileOperations } from "../file-operations/file-operations";
import type { SyncService } from "../services/sync-service";
import type { RelativePath } from "./types";
describe("Reconciler", () => {
it("does not emit an error when placement fetch is interrupted by reset", async () => {
const logger = new Logger();
const settings = new Settings(logger, {}, async () => {
/* no-op */
});
const queue = new SyncEventQueue(
settings,
logger,
{ schemaVersion: STORED_STATE_SCHEMA_VERSION },
async () => {
/* no-op */
}
);
await queue.upsertRecord({
documentId: "DOC-1",
parentVersionId: 1,
remoteHash: "hash",
remoteRelativePath: "remote.md" as RelativePath,
localPath: undefined
});
const operations = {
exists: async () => false,
create: async () => {
assert.fail("reset-interrupted placement should not write");
}
} as unknown as FileOperations;
const syncService = {
getDocumentVersionContent: async () => {
throw new SyncResetError();
}
} as unknown as SyncService;
const reconciler = new Reconciler(
logger,
operations,
syncService,
queue,
new Map()
);
await reconciler.run();
assert.deepStrictEqual(logger.getMessages(LogLevel.ERROR), []);
assert.ok(
logger
.getMessages(LogLevel.INFO)
.some((line) =>
line.message.includes(
"content fetch for DOC-1 interrupted by sync reset"
)
)
);
});
});

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,907 @@
import { describe, it } from "node:test";
import assert from "node:assert";
import {
STORED_STATE_SCHEMA_VERSION,
SyncEventQueue
} from "./sync-event-queue";
import { Settings } from "../persistence/settings";
import { Logger } from "../tracing/logger";
import type { DocumentVersionWithoutContent } from "../services/types/DocumentVersionWithoutContent";
import { SyncEventType } from "./types";
import type { DocumentRecord, RelativePath, StoredSyncState } from "./types";
interface QueueHarness {
queue: SyncEventQueue;
settings: Settings;
saveCalls: StoredSyncState[];
}
function createHarness(
options: {
ignorePatterns?: string[];
initialState?: Partial<StoredSyncState>;
omitSchemaVersion?: boolean;
} = {}
): QueueHarness {
const logger = new Logger();
const settings = new Settings(
logger,
{ ignorePatterns: options.ignorePatterns ?? [] },
async () => {
/* no-op */
}
);
const saveCalls: StoredSyncState[] = [];
const initialState: Partial<StoredSyncState> | undefined =
options.initialState === undefined && options.omitSchemaVersion !== true
? { schemaVersion: STORED_STATE_SCHEMA_VERSION }
: options.initialState;
const queue = new SyncEventQueue(
settings,
logger,
initialState,
async (data) => {
saveCalls.push(data);
}
);
return { queue, settings, saveCalls };
}
function createQueue(ignorePatterns: string[] = []): SyncEventQueue {
return createHarness({ ignorePatterns }).queue;
}
function fakeRemoteVersion(
documentId: string,
overrides: Partial<DocumentVersionWithoutContent> = {}
): DocumentVersionWithoutContent {
return {
vaultUpdateId: 1,
documentId,
relativePath: `${documentId}.md`,
updatedDate: "2026-01-01",
isDeleted: false,
userId: "user",
deviceId: "device",
contentSize: 100,
isNewFile: true,
...overrides
};
}
function fakeRecord(
documentId: string,
overrides: Partial<DocumentRecord> = {}
): DocumentRecord {
const path = `${documentId.toLowerCase()}.md`;
return {
documentId,
parentVersionId: 1,
remoteHash: `hash-${documentId}`,
remoteRelativePath: path,
localPath: path,
...overrides
};
}
describe("SyncEventQueue", () => {
it("returns enqueued events in FIFO order with no coalescing", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "b.md" });
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "c.md" });
await queue.enqueue({ type: SyncEventType.LocalDelete, path: "a.md" });
const first = await queue.next();
assert.strictEqual(first?.type, SyncEventType.LocalCreate);
const second = await queue.next();
assert.strictEqual(second?.type, SyncEventType.LocalCreate);
const third = await queue.next();
assert.strictEqual(third?.type, SyncEventType.LocalDelete);
assert.strictEqual(third.documentId, "A");
assert.strictEqual(await queue.next(), undefined);
});
it("create events are returned FIFO", async () => {
const queue = createQueue();
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "a.md" });
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "b.md" });
const first = await queue.next();
assert.strictEqual(first?.type, SyncEventType.LocalCreate);
assert.strictEqual(first.path, "a.md");
const second = await queue.next();
assert.strictEqual(second?.type, SyncEventType.LocalCreate);
assert.strictEqual(second.path, "b.md");
});
it("delete resolves documentId from path", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.enqueue({ type: SyncEventType.LocalDelete, path: "a.md" });
const event = await queue.next();
assert.strictEqual(event?.type, SyncEventType.LocalDelete);
assert.strictEqual(event.documentId, "A");
});
it("delete for unknown path is silently ignored", async () => {
const queue = createQueue();
await queue.enqueue({
type: SyncEventType.LocalDelete,
path: "unknown.md"
});
assert.strictEqual(queue.pendingUpdateCount, 0);
});
it("delete clears the localPath of the affected record", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.enqueue({ type: SyncEventType.LocalDelete, path: "a.md" });
const record = queue.getDocumentByDocumentId("A");
assert.ok(record !== undefined);
assert.strictEqual(record.localPath, undefined);
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath),
undefined
);
});
it("document store CRUD operations work correctly", async () => {
const queue = createQueue();
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath),
undefined
);
assert.strictEqual(queue.syncedDocumentCount, 0);
await queue.upsertRecord(fakeRecord("A"));
assert.strictEqual(queue.syncedDocumentCount, 1);
const settled = queue.getRecordByLocalPath("a.md" as RelativePath);
assert.strictEqual(settled?.documentId, "A");
assert.strictEqual(settled.localPath, "a.md");
assert.strictEqual(settled.remoteRelativePath, "a.md");
const found = queue.getDocumentByDocumentId("A");
assert.strictEqual(found?.localPath, "a.md");
assert.strictEqual(found.documentId, "A");
await queue.removeDocumentById("A");
assert.strictEqual(queue.syncedDocumentCount, 0);
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath),
undefined
);
assert.strictEqual(queue.getDocumentByDocumentId("A"), undefined);
});
it("LocalUpdate with oldPath moves the document on disk", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.enqueue({
type: SyncEventType.LocalUpdate,
path: "b.md",
oldPath: "a.md"
});
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath),
undefined
);
const moved = queue.getRecordByLocalPath("b.md" as RelativePath);
assert.strictEqual(moved?.documentId, "A");
assert.strictEqual(moved.localPath, "b.md");
// The doc's remoteRelativePath is owned by the wire loop, not the
// watcher path — a local rename does not move the server-side path.
assert.strictEqual(moved.remoteRelativePath, "a.md");
});
it("LocalUpdate rename onto a tracked slot enqueues a delete for the displaced doc", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.upsertRecord(fakeRecord("B"));
// User renames a.md onto b.md, clobbering b.md on disk.
await queue.enqueue({
type: SyncEventType.LocalUpdate,
path: "b.md",
oldPath: "a.md"
});
// Doc A now lives at b.md.
const aRecord = queue.getDocumentByDocumentId("A");
assert.strictEqual(aRecord?.localPath, "b.md");
const slot = queue.getRecordByLocalPath("b.md" as RelativePath);
assert.strictEqual(slot?.documentId, "A");
// Doc B has no local file anymore (its bytes were overwritten).
const bRecord = queue.getDocumentByDocumentId("B");
assert.strictEqual(bRecord?.localPath, undefined);
// Two events should be queued: the LocalDelete for B, then the
// LocalUpdate for A (push order in `enqueue`).
assert.strictEqual(queue.pendingUpdateCount, 2);
const first = await queue.next();
assert.strictEqual(first?.type, SyncEventType.LocalDelete);
assert.strictEqual(first.documentId, "B");
assert.strictEqual(first.path, "b.md");
const second = await queue.next();
assert.strictEqual(second?.type, SyncEventType.LocalUpdate);
assert.strictEqual(second.documentId, "A");
assert.strictEqual(second.path, "b.md");
assert.strictEqual(second.isUserRename, true);
});
it("settled record owns a path over a stale pending create", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A", { localPath: "b.md" }));
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "b.md" });
await queue.enqueue({
type: SyncEventType.LocalUpdate,
path: "c.md",
oldPath: "b.md"
});
const aRecord = queue.getDocumentByDocumentId("A");
assert.strictEqual(aRecord?.localPath, "c.md");
assert.strictEqual(
queue.getRecordByLocalPath("b.md" as RelativePath),
undefined
);
assert.strictEqual(
queue.getRecordByLocalPath("c.md" as RelativePath)?.documentId,
"A"
);
const create = await queue.next();
assert.strictEqual(create?.type, SyncEventType.LocalCreate);
assert.strictEqual(create.path, "b.md");
const update = await queue.next();
assert.strictEqual(update?.type, SyncEventType.LocalUpdate);
assert.strictEqual(update.documentId, "A");
assert.strictEqual(update.path, "c.md");
});
it("byLocalPath stays consistent across upsertRecord, setLocalPath, and rename", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
assert.strictEqual(queue.byLocalPath.size, 1);
assert.strictEqual(
queue.byLocalPath.get("a.md" as RelativePath)?.documentId,
"A"
);
// upsertRecord on an existing record with a non-undefined
// localPath does NOT rewrite localPath. The watcher path and the
// reconciler are the only authorities on localPath of an
// already-placed record; letting the wire loop re-key here would
// race a user rename that landed during an HTTP roundtrip.
await queue.upsertRecord(
fakeRecord("A", { localPath: "renamed.md" as RelativePath })
);
assert.strictEqual(queue.byLocalPath.size, 1);
assert.strictEqual(
queue.byLocalPath.get("a.md" as RelativePath)?.documentId,
"A"
);
assert.strictEqual(
queue.byLocalPath.get("renamed.md" as RelativePath),
undefined
);
assert.strictEqual(queue.getDocumentByDocumentId("A")?.localPath, "a.md");
// setLocalPath does re-key — it's the explicit path-mutation API.
await queue.setLocalPath("A", "later.md" as RelativePath);
assert.strictEqual(queue.byLocalPath.size, 1);
assert.strictEqual(
queue.byLocalPath.get("a.md" as RelativePath),
undefined
);
assert.strictEqual(
queue.byLocalPath.get("later.md" as RelativePath)?.documentId,
"A"
);
// setLocalPath to undefined should drop the entry.
await queue.setLocalPath("A", undefined);
assert.strictEqual(queue.byLocalPath.size, 0);
assert.strictEqual(
queue.byLocalPath.get("later.md" as RelativePath),
undefined
);
// The record is still tracked by docId.
assert.strictEqual(
queue.getDocumentByDocumentId("A")?.localPath,
undefined
);
});
it("upsertRecord installs localPath only when the existing record has none (placement-pending → placed)", async () => {
const queue = createQueue();
// Same-docId-collapse shape: a placement-pending record (created
// earlier by a remote-create handler when the slot was occupied)
// gets resolved by a LocalCreate that returns the same docId.
// The watcher hasn't touched localPath since the record is
// placement-pending, so installing the now-known path is correct.
await queue.upsertRecord(fakeRecord("A", { localPath: undefined }));
assert.strictEqual(queue.byLocalPath.size, 0);
await queue.upsertRecord(
fakeRecord("A", { localPath: "fresh.md" as RelativePath })
);
assert.strictEqual(queue.byLocalPath.size, 1);
assert.strictEqual(
queue.byLocalPath.get("fresh.md" as RelativePath)?.documentId,
"A"
);
assert.strictEqual(
queue.getDocumentByDocumentId("A")?.localPath,
"fresh.md"
);
});
it("upsertRecord ignores stale localPath from the wire loop after a watcher rename", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
// Watcher renames a.md -> renamed.md while the wire loop is
// mid-roundtrip. The wire loop captured an earlier snapshot of
// localPath and now tries to write it back through upsertRecord.
await queue.enqueue({
type: SyncEventType.LocalUpdate,
path: "renamed.md",
oldPath: "a.md"
});
assert.strictEqual(
queue.getDocumentByDocumentId("A")?.localPath,
"renamed.md"
);
await queue.upsertRecord(
fakeRecord("A", {
parentVersionId: 2,
remoteRelativePath: "a.md",
remoteHash: "hash-A-v2",
localPath: "a.md" as RelativePath
})
);
// The watcher's rename wins: localPath stays at renamed.md.
const record = queue.getDocumentByDocumentId("A");
assert.strictEqual(record?.localPath, "renamed.md");
assert.strictEqual(record.parentVersionId, 2);
assert.strictEqual(record.remoteRelativePath, "a.md");
assert.strictEqual(record.remoteHash, "hash-A-v2");
assert.strictEqual(
queue.byLocalPath.get("renamed.md" as RelativePath)?.documentId,
"A"
);
assert.strictEqual(
queue.byLocalPath.get("a.md" as RelativePath),
undefined
);
});
it("create can be re-enqueued after being dequeued", async () => {
const queue = createQueue();
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "a.md" });
await queue.next();
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "a.md" });
assert.strictEqual(queue.pendingUpdateCount, 1);
});
it("silently ignores create events matching ignore patterns", async () => {
const queue = createQueue(["*.tmp", ".hidden/**"]);
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: "scratch.tmp"
});
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: ".hidden/secret.md"
});
assert.strictEqual(queue.pendingUpdateCount, 0);
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: "notes-new.md"
});
assert.strictEqual(queue.pendingUpdateCount, 1);
await queue.enqueue({
type: SyncEventType.RemoteChange,
remoteVersion: fakeRemoteVersion("N")
});
assert.strictEqual(queue.pendingUpdateCount, 2);
});
it("addInternalIgnorePattern hides paths from enqueue and survives settings reload", async () => {
const harness = createHarness({ ignorePatterns: ["*.tmp"] });
const { queue, settings } = harness;
queue.addInternalIgnorePattern(".vaultlink/**");
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: ".vaultlink/swap"
});
assert.strictEqual(queue.pendingUpdateCount, 0);
// User-pattern matching still works alongside the internal pattern.
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: "scratch.tmp"
});
assert.strictEqual(queue.pendingUpdateCount, 0);
// Settings reload must not forget the internal pattern.
await settings.setSettings({ ignorePatterns: ["*.bak"] });
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: ".vaultlink/another"
});
assert.strictEqual(queue.pendingUpdateCount, 0);
// The new user pattern took effect.
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: "old.bak"
});
assert.strictEqual(queue.pendingUpdateCount, 0);
// And paths outside both pattern sets still pass through.
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: "notes.md"
});
assert.strictEqual(queue.pendingUpdateCount, 1);
});
it("clearPending removes events but keeps documents", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "b.md" });
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "c.md" });
assert.strictEqual(queue.pendingUpdateCount, 2);
queue.clearPending();
assert.strictEqual(queue.pendingUpdateCount, 0);
assert.strictEqual(queue.syncedDocumentCount, 1);
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath)?.documentId,
"A"
);
});
it("allSettledDocuments returns all tracked documents that have a localPath", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.upsertRecord(fakeRecord("B"));
// A doc with no local file (e.g. a remote create whose slot was
// occupied) should not appear in the localPath-keyed view.
await queue.upsertRecord(fakeRecord("C", { localPath: undefined }));
const docs = queue.allSettledDocuments();
assert.strictEqual(docs.size, 2);
const paths = Array.from(docs.keys()).sort();
assert.deepStrictEqual(paths, ["a.md", "b.md"]);
});
it("loads initial state from persistence", () => {
const harness = createHarness({
initialState: {
schemaVersion: STORED_STATE_SCHEMA_VERSION,
documents: [
fakeRecord("A", { parentVersionId: 5 }),
fakeRecord("B", { parentVersionId: 3 })
],
lastSeenUpdateId: 4
}
});
const { queue } = harness;
assert.strictEqual(queue.syncedDocumentCount, 2);
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath)?.documentId,
"A"
);
assert.strictEqual(
queue.getRecordByLocalPath("b.md" as RelativePath)?.documentId,
"B"
);
assert.strictEqual(queue.lastSeenUpdateId, 4);
});
it("constructor with mismatched schema version wipes state and saves the new version", () => {
const harness = createHarness({
initialState: {
schemaVersion: 0,
documents: [fakeRecord("A"), fakeRecord("B")],
lastSeenUpdateId: 7
}
});
// Persisted documents and watermark were discarded.
assert.strictEqual(harness.queue.syncedDocumentCount, 0);
assert.strictEqual(harness.queue.lastSeenUpdateId, 0);
// The constructor scheduled a save (don't await — fire-and-forget),
// but we synchronously enqueued it so it should have landed by now.
// The recorded save uses the current schema version.
assert.ok(harness.saveCalls.length >= 1);
const last = harness.saveCalls[harness.saveCalls.length - 1];
assert.strictEqual(last.schemaVersion, STORED_STATE_SCHEMA_VERSION);
assert.deepStrictEqual(last.documents, []);
assert.strictEqual(last.lastSeenUpdateId, 0);
});
it("constructor with missing schema version also wipes state", () => {
const harness = createHarness({
initialState: {
documents: [fakeRecord("A")],
lastSeenUpdateId: 3
}
});
assert.strictEqual(harness.queue.syncedDocumentCount, 0);
assert.strictEqual(harness.queue.lastSeenUpdateId, 0);
assert.ok(harness.saveCalls.length >= 1);
assert.strictEqual(
harness.saveCalls[harness.saveCalls.length - 1].schemaVersion,
STORED_STATE_SCHEMA_VERSION
);
});
it("resolveCreate settles the document and resolves the create promise", async () => {
const queue = createQueue();
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "a.md" });
const event = await queue.next(); // dequeue the create
assert.ok(event?.type === SyncEventType.LocalCreate);
const createPromise = event.resolvers.promise;
await queue.resolveCreate(
event,
fakeRecord("DOC-1", {
parentVersionId: 5,
localPath: "a.md" as RelativePath,
remoteRelativePath: "a.md" as RelativePath
})
);
// Document is now settled
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath)?.documentId,
"DOC-1"
);
// Promise was resolved
assert.strictEqual(await createPromise, "DOC-1");
});
it("delete collapses a pending create that has not started processing", async () => {
const queue = createQueue();
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "a.md" });
const create = queue.peekFront();
assert.ok(create?.type === SyncEventType.LocalCreate);
await queue.enqueue({ type: SyncEventType.LocalDelete, path: "a.md" });
assert.strictEqual(queue.pendingUpdateCount, 0);
assert.strictEqual(await queue.next(), undefined);
await assert.rejects(create.resolvers.promise, /cancelled/);
});
it("resolveCreate does not claim a localPath after an in-flight pending create was deleted", async () => {
const queue = createQueue();
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "a.md" });
const create = queue.peekFront();
assert.ok(create?.type === SyncEventType.LocalCreate);
create.isProcessing = true;
await queue.enqueue({ type: SyncEventType.LocalDelete, path: "a.md" });
await queue.resolveCreate(
create,
fakeRecord("DOC-1", {
localPath: "a.md" as RelativePath,
remoteRelativePath: "a.md" as RelativePath
})
);
assert.strictEqual(
queue.getDocumentByDocumentId("DOC-1")?.localPath,
undefined
);
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath),
undefined
);
const deleteEvent = await queue.next();
assert.strictEqual(deleteEvent?.type, SyncEventType.LocalDelete);
assert.strictEqual(deleteEvent.documentId, "DOC-1");
});
it("resolveCreate only clears localPath for a pending delete of that path", async () => {
const queue = createQueue();
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: "old.md"
});
const create = queue.peekFront();
assert.ok(create?.type === SyncEventType.LocalCreate);
create.isProcessing = true;
await queue.enqueue({
type: SyncEventType.LocalDelete,
path: "old.md"
});
await queue.resolveCreate(
create,
fakeRecord("DOC-1", {
localPath: "new.md" as RelativePath,
remoteRelativePath: "new.md" as RelativePath
})
);
assert.strictEqual(
queue.getDocumentByDocumentId("DOC-1")?.localPath,
"new.md"
);
assert.strictEqual(
queue.getRecordByLocalPath("new.md" as RelativePath)?.documentId,
"DOC-1"
);
const deleteEvent = await queue.next();
assert.strictEqual(deleteEvent?.type, SyncEventType.LocalDelete);
assert.strictEqual(deleteEvent.documentId, "DOC-1");
assert.strictEqual(deleteEvent.path, "old.md");
});
it("pending create owns a same-path delete over a stale deleting record", async () => {
const queue = createQueue();
await queue.upsertRecord(
fakeRecord("OLD", { localPath: "a.md" as RelativePath })
);
queue.markServerDeletePending("OLD");
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "a.md" });
const create = queue.peekFront();
assert.ok(create?.type === SyncEventType.LocalCreate);
create.isProcessing = true;
await queue.enqueue({ type: SyncEventType.LocalDelete, path: "a.md" });
assert.strictEqual(
queue.getDocumentByDocumentId("OLD")?.localPath,
undefined
);
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath),
undefined
);
const createEvent = await queue.next();
assert.strictEqual(createEvent, create);
const deleteEvent = await queue.next();
assert.strictEqual(deleteEvent?.type, SyncEventType.LocalDelete);
assert.strictEqual(deleteEvent.documentId, create.resolvers.promise);
});
it("rename of a queued create drains same-path deletes first", async () => {
const queue = createQueue();
await queue.upsertRecord(
fakeRecord("OLD", { localPath: "target.md" as RelativePath })
);
await queue.enqueue({
type: SyncEventType.LocalCreate,
path: "source.md"
});
const create = queue.peekFront();
assert.ok(create?.type === SyncEventType.LocalCreate);
await queue.enqueue({
type: SyncEventType.LocalDelete,
path: "target.md"
});
await queue.enqueue({
type: SyncEventType.LocalUpdate,
oldPath: "source.md",
path: "target.md"
});
const deleteEvent = await queue.next();
assert.strictEqual(deleteEvent?.type, SyncEventType.LocalDelete);
assert.strictEqual(deleteEvent.documentId, "OLD");
assert.strictEqual(deleteEvent.path, "target.md");
const createEvent = await queue.next();
assert.strictEqual(createEvent, create);
assert.strictEqual(createEvent.path, "target.md");
const updateEvent = await queue.next();
assert.strictEqual(updateEvent?.type, SyncEventType.LocalUpdate);
assert.strictEqual(updateEvent.documentId, create.resolvers.promise);
assert.strictEqual(updateEvent.path, "target.md");
});
it("findLatestCreateForPath returns the pending create", async () => {
const queue = createQueue();
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "a.md" });
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "b.md" });
const found = queue.findLatestCreateForPath("a.md" as RelativePath);
assert.ok(found !== undefined);
assert.strictEqual(found.path, "a.md");
const missing = queue.findLatestCreateForPath("c.md" as RelativePath);
assert.strictEqual(missing, undefined);
});
it("hasPendingEventsForPath reflects pending events", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
assert.strictEqual(
queue.hasPendingEventsForPath("a.md" as RelativePath),
false
);
await queue.enqueue({ type: SyncEventType.LocalDelete, path: "a.md" });
// After a delete the localPath is cleared; an unknown path is treated
// as "must be pending creation", so this still returns true.
assert.strictEqual(
queue.hasPendingEventsForPath("a.md" as RelativePath),
true
);
});
it("setLocalPath displaces a previous holder of the same path", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.upsertRecord(
fakeRecord("B", { localPath: "b.md" as RelativePath })
);
// Move B onto a.md — the slot already held by A. The invariant
// requires A's localPath to be cleared (placement-pending),
// and byLocalPath["a.md"] === B.
await queue.setLocalPath("B", "a.md" as RelativePath);
const a = queue.getDocumentByDocumentId("A");
const b = queue.getDocumentByDocumentId("B");
assert.strictEqual(a?.localPath, undefined);
assert.strictEqual(b?.localPath, "a.md");
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath)?.documentId,
"B"
);
// B's old slot is now empty — nothing else moved into it.
assert.strictEqual(
queue.getRecordByLocalPath("b.md" as RelativePath),
undefined
);
});
it("upsertRecord displaces a previous holder of the same path", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
// A new record (different docId) claims a.md. The prior holder
// (A) must be displaced — its localPath cleared, and
// byLocalPath["a.md"] now points at the new record.
await queue.upsertRecord(
fakeRecord("B", { localPath: "a.md" as RelativePath })
);
const a = queue.getDocumentByDocumentId("A");
const b = queue.getDocumentByDocumentId("B");
assert.strictEqual(a?.localPath, undefined);
assert.strictEqual(b?.localPath, "a.md");
assert.strictEqual(
queue.getRecordByLocalPath("a.md" as RelativePath)?.documentId,
"B"
);
});
it("the localPath/byLocalPath invariant holds across rename + recreate cycles", async () => {
// Construct the exact same-path create cycle that produces the
// bug-D race: docA at P, then docB created at P (via
// upsertRecord), and finally a setLocalPath that would move a
// third doc onto P. The invariant must hold at every step:
// exactly one record has localPath===P at any given time, and
// byLocalPath.get(P) returns it.
const queue = createQueue();
const path = "p.md" as RelativePath;
await queue.upsertRecord(
fakeRecord("A", { localPath: path, remoteRelativePath: path })
);
// Sanity: A holds the slot.
assert.strictEqual(queue.getRecordByLocalPath(path)?.documentId, "A");
assert.strictEqual(queue.getDocumentByDocumentId("A")?.localPath, path);
// docB created at P via upsertRecord (e.g. a remote create
// that races A's local file onto the same slot). A must be
// displaced.
await queue.upsertRecord(
fakeRecord("B", { localPath: path, remoteRelativePath: path })
);
assert.strictEqual(
queue.getDocumentByDocumentId("A")?.localPath,
undefined
);
assert.strictEqual(queue.getDocumentByDocumentId("B")?.localPath, path);
assert.strictEqual(queue.getRecordByLocalPath(path)?.documentId, "B");
// Now setLocalPath moves a third doc C onto P. B must in turn
// be displaced; the invariant still holds.
await queue.upsertRecord(
fakeRecord("C", { localPath: "c.md" as RelativePath })
);
await queue.setLocalPath("C", path);
assert.strictEqual(
queue.getDocumentByDocumentId("B")?.localPath,
undefined
);
assert.strictEqual(queue.getDocumentByDocumentId("C")?.localPath, path);
assert.strictEqual(queue.getRecordByLocalPath(path)?.documentId, "C");
// Across the whole cycle exactly one record holds the slot.
const holders = Array.from(queue.allRecords()).filter(
(r) => r.localPath === path
);
assert.strictEqual(holders.length, 1);
assert.strictEqual(holders[0].documentId, "C");
});
it("clearAllState clears everything", async () => {
const queue = createQueue();
await queue.upsertRecord(fakeRecord("A"));
await queue.enqueue({ type: SyncEventType.LocalCreate, path: "b.md" });
await queue.clearAllState();
assert.strictEqual(queue.syncedDocumentCount, 0);
assert.strictEqual(queue.pendingUpdateCount, 0);
assert.strictEqual(queue.byLocalPath.size, 0);
});
});

File diff suppressed because it is too large Load diff

File diff suppressed because it is too large Load diff

View file

@ -0,0 +1,74 @@
import type { DocumentVersionWithoutContent } from "../services/types/DocumentVersionWithoutContent";
export type VaultUpdateId = number;
export type DocumentId = string;
export type RelativePath = string;
export interface DocumentRecord {
documentId: DocumentId;
parentVersionId: VaultUpdateId;
// Hash of the last server version this client has observed for the doc.
// `undefined` means we have a record but haven't actually seen content
// yet — typically a remote-create whose target slot was occupied at
// receive time, where we deliberately defer the fetch to the reconciler.
// Consumers should treat undefined as "no comparison possible" (the
// fast-skip in `processLocalUpdate` falls through to a real upload).
remoteHash: string | undefined;
remoteRelativePath: RelativePath;
// Where the doc's file currently lives on disk. `undefined` means the doc
// has no local file yet — happens for a remote create whose
// `remoteRelativePath` slot was occupied at receive time. The reconciler
// will place the file once the slot frees, fetching content from the
// server on demand.
localPath: RelativePath | undefined;
}
export interface StoredSyncState {
schemaVersion: number;
documents: DocumentRecord[] | undefined;
lastSeenUpdateId: VaultUpdateId | undefined;
}
export enum SyncEventType {
LocalCreate = "local-create",
LocalUpdate = "local-update", // includes both content and path changes
LocalDelete = "local-delete",
RemoteChange = "remote-change" // includes every type of create/update/delete coming from the server
}
export type FileSyncEvent =
| { type: SyncEventType.LocalCreate; path: RelativePath }
| {
type: SyncEventType.LocalUpdate;
path: RelativePath;
oldPath?: RelativePath; // oldPath is undefined for content changes
}
| { type: SyncEventType.LocalDelete; path: RelativePath }
| {
type: SyncEventType.RemoteChange;
remoteVersion: DocumentVersionWithoutContent;
};
export type SyncEvent =
| {
type: SyncEventType.LocalCreate;
path: RelativePath; // current path on disk; mutated in place by `updatePendingCreatePath` when the user renames mid-flight
isProcessing: boolean; // true once the wire loop has started this create; deletes after that must wait for the server ack
resolvers: PromiseWithResolvers<DocumentId>;
}
| {
type: SyncEventType.LocalUpdate;
documentId: DocumentId | Promise<DocumentId>; // if it's a promise, the promise is fulfilled once the document's create event is processed
path: RelativePath; // current path on disk
originalPath: RelativePath; // original path on disk when the event was queued
isUserRename: boolean; // true iff this event was queued because the user renamed the file
}
| {
type: SyncEventType.LocalDelete;
documentId: DocumentId | Promise<DocumentId>; // if it's a promise, the promise is fulfilled once the document's create event is processed
path: RelativePath; // only used for showing on the UI
}
| {
type: SyncEventType.RemoteChange;
remoteVersion: DocumentVersionWithoutContent;
};

View file

@ -1,596 +0,0 @@
import type {
Database,
DocumentRecord,
RelativePath
} from "../persistence/database";
import { diff } from "reconcile-text";
import type { SyncService } from "../services/sync-service";
import type { Logger } from "../tracing/logger";
import type {
CommonHistoryEntry,
SyncCreateDetails,
SyncDeleteDetails,
SyncDetails,
SyncHistory,
SyncMovedDetails,
SyncUpdateDetails
} from "../tracing/sync-history";
import { SyncStatus, SyncType } from "../tracing/sync-history";
import { EMPTY_HASH, hash } from "../utils/hash";
import { base64ToBytes } from "byte-base64";
import type { Settings } from "../persistence/settings";
import type { FileOperations } from "../file-operations/file-operations";
import { createPromise } from "../utils/create-promise";
import { FileNotFoundError } from "../file-operations/file-not-found-error";
import { SyncResetError } from "../services/sync-reset-error";
import { globsToRegexes } from "../utils/globs-to-regexes";
import type { DocumentVersion } from "../services/types/DocumentVersion";
import type { DocumentUpdateResponse } from "../services/types/DocumentUpdateResponse";
import type { DocumentVersionWithoutContent } from "../services/types/DocumentVersionWithoutContent";
import type { FixedSizeDocumentCache } from "../utils/data-structures/fix-sized-cache";
import { isFileTypeMergable } from "../utils/is-file-type-mergable";
import { isBinary } from "../utils/is-binary";
import type { ServerConfig } from "../services/server-config";
export class UnrestrictedSyncer {
private ignorePatterns: RegExp[];
public constructor(
private readonly logger: Logger,
private readonly database: Database,
private readonly settings: Settings,
private readonly syncService: SyncService,
private readonly operations: FileOperations,
private readonly history: SyncHistory,
private readonly contentCache: FixedSizeDocumentCache,
private readonly serverConfig: ServerConfig
) {
this.ignorePatterns = globsToRegexes(
this.settings.getSettings().ignorePatterns,
this.logger
);
this.settings.onSettingsChanged.add((newSettings) => {
this.ignorePatterns = globsToRegexes(
newSettings.ignorePatterns,
this.logger
);
});
}
public async unrestrictedSyncLocallyCreatedFile(
document: DocumentRecord
): Promise<void> {
const updateDetails: SyncCreateDetails = {
type: SyncType.CREATE,
relativePath: document.relativePath
};
return this.executeSync(updateDetails, async () => {
const originalRelativePath = document.relativePath;
if (document.isDeleted) {
this.logger.debug(
`Document ${originalRelativePath} has been already deleted, no need to create it`
);
return;
}
const contentBytes =
await this.operations.read(originalRelativePath); // this can throw FileNotFoundError
const contentHash = hash(contentBytes);
const response = await this.syncService.create({
documentId: document.documentId,
relativePath: originalRelativePath,
contentBytes
});
// In case a document with the same name (but different ID) had existed remotely that we haven't known about
if (response.relativePath != originalRelativePath) {
this.logger.debug(
`Document ${originalRelativePath} has been created remotely at a different path: ${response.relativePath}, moving it locally`
);
await this.operations.move(
document.relativePath,
response.relativePath
); // this can throw FileNotFoundError
}
this.database.updateDocumentMetadata(
{
parentVersionId: response.vaultUpdateId,
hash: contentHash,
remoteRelativePath: response.relativePath
},
document
);
this.database.addSeenUpdateId(response.vaultUpdateId);
await this.updateCache(
response.vaultUpdateId,
contentBytes,
response.relativePath
);
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: updateDetails,
message: `Successfully uploaded locally created file`
});
});
}
public async unrestrictedSyncLocallyDeletedFile(
document: DocumentRecord
): Promise<void> {
const updateDetails: SyncDeleteDetails = {
type: SyncType.DELETE,
relativePath: document.relativePath
};
await this.executeSync(updateDetails, async () => {
const response = await this.syncService.delete({
documentId: document.documentId,
relativePath: document.relativePath
});
this.database.updateDocumentMetadata(
{
parentVersionId: response.vaultUpdateId,
hash: EMPTY_HASH,
remoteRelativePath: document.relativePath
},
document
);
this.database.addSeenUpdateId(response.vaultUpdateId);
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: updateDetails,
message: `Successfully deleted locally deleted file on the server`,
author: response.userId
});
});
}
public async unrestrictedSyncLocallyUpdatedFile({
oldPath,
document,
// We use the same code path for both local and remote updates. We need to force the update
// if there are no local changes but we know that the remote version is newer.
force = false
}: {
oldPath?: RelativePath;
force?: boolean;
document: DocumentRecord;
}): Promise<void> {
const updateDetails: SyncUpdateDetails | SyncMovedDetails =
oldPath !== undefined
? {
type: SyncType.MOVE,
relativePath: document.relativePath,
movedFrom: oldPath
}
: {
type: SyncType.UPDATE,
relativePath: document.relativePath
};
await this.executeSync(updateDetails, async () => {
const originalRelativePath = document.relativePath;
if (document.isDeleted || document.metadata === undefined) {
this.logger.debug(
`Document ${document.relativePath} has been already deleted, no need to update it`
);
return;
}
const contentBytes = await this.operations.read(
document.relativePath
); // this can throw FileNotFoundError
let contentHash = hash(contentBytes);
const areThereLocalChanges = !(
document.metadata.hash === contentHash && oldPath === undefined
);
let response: DocumentVersion | DocumentUpdateResponse | undefined =
undefined;
if (areThereLocalChanges) {
const isText =
!isBinary(contentBytes) &&
isFileTypeMergable(
document.relativePath,
(await this.serverConfig.getConfig())
.mergeableFileExtensions
);
const cachedVersion = this.contentCache.get(
document.metadata.parentVersionId
);
response =
isText && cachedVersion !== undefined
? await this.syncService.putText({
documentId: document.documentId,
parentVersionId:
document.metadata.parentVersionId,
relativePath: document.relativePath,
content: diff(
new TextDecoder().decode(cachedVersion),
new TextDecoder().decode(contentBytes)
)
})
: await this.syncService.putBinary({
documentId: document.documentId,
parentVersionId:
document.metadata.parentVersionId,
relativePath: document.relativePath,
contentBytes
});
} else {
if (!force) {
this.logger.debug(
`File hash of ${document.relativePath} matches with last synced version and the path hasn't changed; no need to sync`
);
return;
}
response = await this.syncService.get({
documentId: document.documentId
});
}
// `document` is mutable and reflects the latest state in the local database
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
if (document.isDeleted) {
this.logger.info(
`Document ${document.relativePath} has been deleted before we could finish updating it`
);
this.database.addSeenUpdateId(response.vaultUpdateId);
return;
}
if (
// `Syncer` creates fake local document metadata for all remote docs with invalid hashes. The parent IDs will likely match
// the latest versions so we still need to update the local versions to turn the fakes into real metadata.
document.metadata.parentVersionId > response.vaultUpdateId
) {
this.logger.debug(
`Document ${document.relativePath} is already more up to date than the fetched version`
);
this.database.addSeenUpdateId(response.vaultUpdateId); // in case the previous `vaultUpdateId` update hasn't made it through
return;
}
if (response.isDeleted) {
return this.applyRemoteDeleteLocally(document, response);
}
let actualPath = document.relativePath;
if (response.relativePath != originalRelativePath) {
actualPath = response.relativePath;
// Make sure to update the remote relative path to avoid uploading
// the file as a result of this filesystem event.
document.metadata.remoteRelativePath = response.relativePath;
await this.operations.move(
document.relativePath,
response.relativePath
); // this can throw FileNotFoundError
}
if (!("type" in response) || response.type === "MergingUpdate") {
const responseBytes = base64ToBytes(response.contentBase64);
contentHash = hash(responseBytes);
this.database.updateDocumentMetadata(
{
parentVersionId: response.vaultUpdateId,
hash: contentHash,
remoteRelativePath: response.relativePath
},
document
);
await this.operations.write(
actualPath,
contentBytes,
responseBytes
);
await this.updateCache(
response.vaultUpdateId,
responseBytes,
actualPath
);
if (!force) {
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: updateDetails,
message: `The file we updated had been updated remotely, so we downloaded the merged version`
});
}
} else {
this.database.updateDocumentMetadata(
{
parentVersionId: response.vaultUpdateId,
hash: contentHash,
remoteRelativePath: response.relativePath
},
document
);
await this.updateCache(
response.vaultUpdateId,
contentBytes,
actualPath
);
}
this.database.addSeenUpdateId(response.vaultUpdateId);
const actualUpdateDetails: SyncUpdateDetails | SyncMovedDetails =
oldPath !== undefined ||
response.relativePath != originalRelativePath
? {
type: SyncType.MOVE,
relativePath: response.relativePath,
movedFrom: originalRelativePath
}
: {
type: SyncType.UPDATE,
relativePath: response.relativePath
};
if (areThereLocalChanges) {
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: actualUpdateDetails,
message: `Successfully uploaded locally updated file to the server`,
author: response.userId
});
} else {
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: actualUpdateDetails,
message: `Successfully downloaded remotely updated file from the server`,
author: response.userId,
timestamp: new Date(response.updatedDate)
});
}
});
}
public async unrestrictedSyncRemotelyUpdatedFile(
remoteVersion: DocumentVersionWithoutContent,
document?: DocumentRecord
): Promise<void> {
const updateDetails: SyncCreateDetails = {
type: SyncType.CREATE,
relativePath: remoteVersion.relativePath
};
await this.executeSync(updateDetails, async () => {
if (document?.metadata !== undefined) {
// If the file exists locally, let's pretend the user has updated it
// and deal with remote update/deletion within `unrestrictedSyncLocallyUpdatedFile`
if (
document.metadata.parentVersionId >=
remoteVersion.vaultUpdateId
) {
this.logger.debug(
`Document ${remoteVersion.relativePath} is already at least as up to date as the fetched version`
);
return;
}
return this.unrestrictedSyncLocallyUpdatedFile({
document,
force: true
});
} else if (remoteVersion.isDeleted) {
// Either the document hasn't made it to us before and therefore we don't need to delete it,
// or we already have it, in which case the preceeding if would've dealt with it
this.logger.debug(
`Document ${remoteVersion.relativePath} has been deleted remotely, no need to sync`
);
return;
}
// Don't download oversized files
const historyEntryForSkippedOversizedFile =
this.getHistoryEntryForSkippedOversizedFile(
remoteVersion.contentSize,
remoteVersion.relativePath
);
if (historyEntryForSkippedOversizedFile !== undefined) {
this.history.addHistoryEntry(
historyEntryForSkippedOversizedFile
);
return;
}
const contentBytes =
await this.syncService.getDocumentVersionContent({
documentId: remoteVersion.documentId,
vaultUpdateId: remoteVersion.vaultUpdateId
});
// We're trying to create an entirely new document that didn't exist locally
document = this.database.getDocumentByDocumentId(
remoteVersion.documentId
);
// It can happen that a concurrent sync operation has already created the document, so we can bail here
if (document !== undefined) {
this.logger.debug(
`Document ${remoteVersion.relativePath} has already been created locally, no need to create it again`
);
return;
}
await this.operations.ensureClearPath(remoteVersion.relativePath);
const [promise, resolve] = createPromise();
this.database.updateDocumentMetadata(
{
parentVersionId: remoteVersion.vaultUpdateId,
hash: hash(contentBytes),
remoteRelativePath: remoteVersion.relativePath
},
this.database.createNewPendingDocument(
remoteVersion.documentId,
remoteVersion.relativePath,
promise
)
);
await this.operations.create(
remoteVersion.relativePath,
contentBytes
);
await this.updateCache(
remoteVersion.vaultUpdateId,
contentBytes,
remoteVersion.relativePath
);
resolve();
this.database.removeDocumentPromise(promise);
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: updateDetails,
message: `Successfully downloaded remote file which hadn't existed locally`,
author: remoteVersion.userId,
timestamp: new Date(remoteVersion.updatedDate)
});
});
}
public async executeSync<T>(
details: SyncDetails,
fn: () => Promise<T>
): Promise<T | undefined> {
for (const pattern of this.ignorePatterns) {
if (pattern.test(details.relativePath)) {
this.logger.debug(
`File '${details.relativePath}' is ignored by the ignore pattern: ${pattern}`
);
return; // bail without SKIPPED status because we were told to ignore this file and we shouldn't clutter up the history
}
}
try {
// Only check the size of files which already exist locally.
if (await this.operations.exists(details.relativePath)) {
const sizeInBytes = await this.operations.getFileSize(
details.relativePath
);
const historyEntryForSkippedOversizedFile =
this.getHistoryEntryForSkippedOversizedFile(
sizeInBytes,
details.relativePath
);
if (historyEntryForSkippedOversizedFile !== undefined) {
this.history.addHistoryEntry(
historyEntryForSkippedOversizedFile
);
return;
}
}
return await fn();
} catch (e) {
if (e instanceof FileNotFoundError) {
// A subsequent sync operation must have been creating to deal with this
this.logger.info(
`Skiping file '${details.relativePath}' because it no longer exists when trying to ${details.type.toLocaleLowerCase()} it`
);
return;
}
if (e instanceof SyncResetError) {
this.logger.info(
`Interrupting sync operation because of a reset`
);
return;
} else {
this.history.addHistoryEntry({
status: SyncStatus.ERROR,
details,
message: `Failed to sync file '${details.relativePath}' because of ${e} when trying to ${details.type.toLocaleLowerCase()} it`
});
throw e;
}
}
}
private getHistoryEntryForSkippedOversizedFile(
sizeInBytes: number,
relativePath: RelativePath
): CommonHistoryEntry | undefined {
const sizeInMB = Math.round(sizeInBytes / 1024 / 1024);
const { maxFileSizeMB } = this.settings.getSettings();
if (sizeInMB > maxFileSizeMB) {
return {
status: SyncStatus.SKIPPED,
details: {
type: SyncType.SKIPPED,
relativePath
},
message: `File size of ${sizeInMB} MB exceeds the maximum file size limit of ${
maxFileSizeMB
} MB`
};
}
}
private async updateCache(
updateId: number,
contentBytes: Uint8Array,
filePath: RelativePath
): Promise<void> {
if (
isFileTypeMergable(
filePath,
(await this.serverConfig.getConfig()).mergeableFileExtensions
) &&
!isBinary(contentBytes)
) {
this.contentCache.put(updateId, contentBytes);
}
}
private async applyRemoteDeleteLocally(
document: DocumentRecord,
response: DocumentVersion | DocumentUpdateResponse
): Promise<void> {
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: {
type: SyncType.DELETE,
relativePath: document.relativePath
},
message: "File has been deleted remotely, so we deleted it locally",
author: response.userId,
timestamp: new Date(response.updatedDate)
});
this.database.delete(document.relativePath);
this.database.updateDocumentMetadata(
{
parentVersionId: response.vaultUpdateId,
hash: EMPTY_HASH,
remoteRelativePath: response.relativePath
},
document
);
await this.operations.delete(document.relativePath);
this.database.addSeenUpdateId(response.vaultUpdateId);
}
}

View file

@ -2,7 +2,7 @@ import {
MAX_HISTORY_ENTRY_COUNT, MAX_HISTORY_ENTRY_COUNT,
TIMEOUT_FOR_MERGING_HISTORY_ENTRIES_IN_SECONDS TIMEOUT_FOR_MERGING_HISTORY_ENTRIES_IN_SECONDS
} from "../consts"; } from "../consts";
import type { RelativePath } from "../persistence/database"; import type { RelativePath } from "../sync-operations/types";
import type { Logger } from "./logger"; import type { Logger } from "./logger";
import { removeFromArray } from "../utils/remove-from-array"; import { removeFromArray } from "../utils/remove-from-array";
import { EventListeners } from "../utils/data-structures/event-listeners"; import { EventListeners } from "../utils/data-structures/event-listeners";
@ -28,7 +28,7 @@ export interface SyncDeleteDetails {
relativePath: RelativePath; relativePath: RelativePath;
} }
export interface SyncSkippedDetails { interface SyncSkippedDetails {
type: SyncType.SKIPPED; type: SyncType.SKIPPED;
relativePath: RelativePath; relativePath: RelativePath;
} }
@ -40,12 +40,15 @@ export type SyncDetails =
| SyncMovedDetails | SyncMovedDetails
| SyncSkippedDetails; | SyncSkippedDetails;
export interface CommonHistoryEntry { export interface HistoryEntry {
status: SyncStatus; status: SyncStatus;
message: string; message: string;
details: SyncDetails; details: SyncDetails;
timestamp: Date;
// `author` is the server-side user id and only exists for entries that
// round-tripped through the server. Local-only entries (e.g. SKIPPED)
// legitimately have no author.
author?: string; author?: string;
timestamp?: Date;
} }
export enum SyncType { export enum SyncType {
@ -62,8 +65,6 @@ export enum SyncStatus {
SKIPPED = "SKIPPED" SKIPPED = "SKIPPED"
} }
export type HistoryEntry = CommonHistoryEntry & { timestamp: Date };
export interface HistoryStats { export interface HistoryStats {
success: number; success: number;
error: number; error: number;
@ -93,25 +94,20 @@ export class SyncHistory {
* *
* If the entry list is too long, the oldest entry will be removed. * If the entry list is too long, the oldest entry will be removed.
*/ */
public addHistoryEntry(entry: CommonHistoryEntry): void { public addHistoryEntry(entry: HistoryEntry): void {
const historyEntry = { const candidate = this.findSimilarRecentUpdateEntry(entry);
...entry,
timestamp: entry.timestamp ?? new Date()
};
const candidate = this.findSimilarRecentUpdateEntry(historyEntry);
if (candidate !== undefined) { if (candidate !== undefined) {
removeFromArray(this._entries, candidate); removeFromArray(this._entries, candidate);
} }
// Insert the entry at the beginning // Insert the entry at the beginning
this._entries.unshift(historyEntry); this._entries.unshift(entry);
if (this._entries.length > MAX_HISTORY_ENTRY_COUNT) { if (this._entries.length > MAX_HISTORY_ENTRY_COUNT) {
this._entries.pop(); this._entries.pop();
} }
this.updateSuccessCount(historyEntry); this.updateSuccessCount(entry);
} }
public reset(): void { public reset(): void {

View file

@ -9,7 +9,7 @@ type ResolvedTuple<T extends readonly unknown[]> = {
export const awaitAll = async <T extends readonly unknown[]>( export const awaitAll = async <T extends readonly unknown[]>(
promises: PromiseTuple<T> promises: PromiseTuple<T>
): Promise<ResolvedTuple<T>> => { ): Promise<ResolvedTuple<T>> => {
// eslint-disable-next-line no-restricted-properties // eslint-disable-next-line no-restricted-properties, @typescript-eslint/await-thenable
const result = await Promise.allSettled(promises); const result = await Promise.allSettled(promises);
for (const res of result) { for (const res of result) {
if (res.status === "rejected") { if (res.status === "rejected") {

View file

@ -1,5 +1,3 @@
import { v4 as uuidv4 } from "uuid";
export function createClientId(): string { export function createClientId(): string {
// @ts-expect-error, injected by webpack // @ts-expect-error, injected by webpack
const packageVersion = __CURRENT_VERSION__; // eslint-disable-line const packageVersion = __CURRENT_VERSION__; // eslint-disable-line
@ -11,5 +9,5 @@ export function createClientId(): string {
? process.platform ? process.platform
: "unknown"; : "unknown";
return `vault-link/${packageVersion} (${uuidv4()}; ${platform})`; return `vault-link/${packageVersion} (${Math.round(Math.random() * 1e10)}; ${platform})`;
} }

View file

@ -1,25 +0,0 @@
type ResolveFunction<T> = undefined extends T
? (value?: T) => unknown
: (value: T) => unknown;
/**
* A type-safe utility function to create a Promise with resolve and reject functions.
* @returns A tuple containing a Promise, a resolve function, and a reject function.
*/
export function createPromise<T = unknown>(): [
Promise<T>,
ResolveFunction<T>,
(error: unknown) => unknown
] {
let resolve: undefined | ResolveFunction<T> = undefined;
let reject: undefined | ((error: unknown) => unknown) = undefined;
const creationPromise = new Promise<T>(
(resolve_, reject_) =>
// eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
((resolve = resolve_ as ResolveFunction<T>), (reject = reject_))
);
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
return [creationPromise, resolve!, reject!];
}

View file

@ -40,9 +40,14 @@ export class EventListeners<TListener extends (...args: any[]) => any> {
* @param args The arguments to pass to each listener * @param args The arguments to pass to each listener
*/ */
public trigger(...args: Parameters<TListener>): void { public trigger(...args: Parameters<TListener>): void {
this.listeners.forEach((listener) => { const snapshot = this.listeners.slice();
for (const listener of snapshot) {
// allow removing listeners during the trigger loop
if (!this.listeners.includes(listener)) {
continue;
}
listener(...args); listener(...args);
}); }
} }
/** /**
@ -53,16 +58,19 @@ export class EventListeners<TListener extends (...args: any[]) => any> {
* @param args The arguments to pass to each listener * @param args The arguments to pass to each listener
*/ */
public async triggerAsync(...args: Parameters<TListener>): Promise<void> { public async triggerAsync(...args: Parameters<TListener>): Promise<void> {
await awaitAll( const snapshot = this.listeners.slice();
this.listeners const promises: Promise<unknown>[] = [];
.map((listener) => { for (const listener of snapshot) {
// eslint-disable-next-line @typescript-eslint/no-unsafe-return if (!this.listeners.includes(listener)) {
return listener(...args); continue;
}) }
.filter((result): result is Promise<unknown> => { // eslint-disable-next-line @typescript-eslint/no-unsafe-assignment
return result instanceof Promise; const result = listener(...args);
}) if (result instanceof Promise) {
); promises.push(result);
}
}
await awaitAll(promises);
} }
public clear(): void { public clear(): void {

View file

@ -1,6 +1,6 @@
// Implements an in-memory fixed-size cache for document contents, // Implements an in-memory fixed-size cache for document contents,
import type { VaultUpdateId } from "../../persistence/database"; import type { VaultUpdateId } from "../../sync-operations/types";
// Doubly-linked list node for O(1) LRU operations // Doubly-linked list node for O(1) LRU operations
class LRUNode { class LRUNode {

View file

@ -1,22 +1,24 @@
import { describe, it, beforeEach } from "node:test"; import { describe, it, beforeEach } from "node:test";
import assert from "node:assert"; import assert from "node:assert";
import { Logger } from "../../tracing/logger"; import { Logger } from "../../tracing/logger";
import type { RelativePath } from "../../persistence/database"; import type { RelativePath } from "../../sync-operations/types";
import { Locks } from "./locks"; import { Locks } from "./locks";
import { awaitAll } from "../await-all"; import { awaitAll } from "../await-all";
import { sleep } from "../sleep"; import { sleep } from "../sleep";
import { SyncResetError } from "../../services/sync-reset-error"; import { SyncResetError } from "../../errors/sync-reset-error";
describe("withLock", () => { describe("withLock", () => {
const testPath: RelativePath = "test/document/path"; const testPath: RelativePath = "test/document/path";
const testPath2: RelativePath = "test/document/path2"; const testPath2: RelativePath = "test/document/path2";
const testPath3: RelativePath = "test/document/path3";
const logger = new Logger(); const logger = new Logger();
// eslint-disable-next-line @typescript-eslint/init-declarations // eslint-disable-next-line @typescript-eslint/init-declarations
let locks: Locks<RelativePath>; let locks: Locks<RelativePath>;
beforeEach(() => { beforeEach(() => {
locks = new Locks<RelativePath>(logger); locks = new Locks<RelativePath>("locks-test", logger);
}); });
it("should execute function with single key lock", async () => { it("should execute function with single key lock", async () => {
@ -56,22 +58,32 @@ describe("withLock", () => {
it("should sort multiple keys to prevent deadlocks", async () => { it("should sort multiple keys to prevent deadlocks", async () => {
const executionOrder: string[] = []; const executionOrder: string[] = [];
// Start two concurrent operations with keys in different orders await locks.waitForLock(testPath);
const promise1 = locks.withLock([testPath2, testPath], async () => {
const promise = awaitAll([
locks.withLock([testPath2, testPath3, testPath], async () => {
executionOrder.push("operation1-start"); executionOrder.push("operation1-start");
await sleep(50);
executionOrder.push("operation1-end"); executionOrder.push("operation1-end");
return "result1"; return "result1";
}); }),
const promise2 = locks.withLock([testPath, testPath2], async () => { locks.withLock([testPath3, testPath, testPath2], async () => {
executionOrder.push("operation2-start"); executionOrder.push("operation2-start");
await sleep(50);
executionOrder.push("operation2-end"); executionOrder.push("operation2-end");
return "result2"; return "result2";
}); })
]);
const [result1, result2] = await awaitAll([promise1, promise2]); locks.unlock(testPath);
const [result1, result2] = await Promise.race([
promise,
new Promise<never>((_, reject) => {
setTimeout(() => {
reject(new Error("Deadlock detected"));
}, 1000);
})
]);
assert.strictEqual(result1, "result1"); assert.strictEqual(result1, "result1");
assert.strictEqual(result2, "result2"); assert.strictEqual(result2, "result2");
@ -234,13 +246,14 @@ describe("withLock", () => {
describe("reset", () => { describe("reset", () => {
const testPath: RelativePath = "test/document/path"; const testPath: RelativePath = "test/document/path";
const testPath2: RelativePath = "test/document/path2";
const logger = new Logger(); const logger = new Logger();
// eslint-disable-next-line @typescript-eslint/init-declarations // eslint-disable-next-line @typescript-eslint/init-declarations
let locks: Locks<RelativePath>; let locks: Locks<RelativePath>;
beforeEach(() => { beforeEach(() => {
locks = new Locks<RelativePath>(logger); locks = new Locks<RelativePath>("locks-test", logger);
}); });
it("should reject pending waiters with SyncResetError while running operation completes", async () => { it("should reject pending waiters with SyncResetError while running operation completes", async () => {
@ -289,4 +302,38 @@ describe("reset", () => {
const result = await locks.withLock(testPath, () => "success"); const result = await locks.withLock(testPath, () => "success");
assert.strictEqual(result, "success"); assert.strictEqual(result, "success");
}); });
it("should release partially acquired locks when reset interrupts multi-key acquisition", async () => {
// Hold testPath2 so multi-key acquisition will block on it
await locks.waitForLock(testPath2);
// Start multi-key lock that will acquire testPath first, then block on testPath2
const multiKeyPromise = locks.withLock(
[testPath, testPath2],
async () => "multi"
);
void multiKeyPromise.catch(() => {}); // eslint-disable-line @typescript-eslint/no-empty-function
// Wait for the multi-key operation to acquire testPath and start waiting on testPath2
await sleep(10);
// Reset should reject the waiting operation
locks.reset();
await assert.rejects(multiKeyPromise, (err: Error) => {
assert.ok(err instanceof SyncResetError);
return true;
});
// The key that was already acquired (testPath) should now be released
// This would hang/timeout if the lock was leaked
const result = await Promise.race([
locks.withLock(testPath, () => "success"),
sleep(100).then(() => {
throw new Error("Lock was not released - deadlock detected");
})
]);
assert.strictEqual(result, "success");
});
}); });

View file

@ -1,6 +1,5 @@
import { SyncResetError } from "../../services/sync-reset-error"; import { SyncResetError } from "../../errors/sync-reset-error";
import type { Logger } from "../../tracing/logger"; import type { Logger } from "../../tracing/logger";
import { awaitAll } from "../await-all";
/** /**
* Manages exclusive locks on items to prevent concurrent modifications. * Manages exclusive locks on items to prevent concurrent modifications.
@ -8,17 +7,23 @@ import { awaitAll } from "../await-all";
* *
* @template T The type of the key used for locking * @template T The type of the key used for locking
*/ */
/** Waiter entry with callbacks */
interface WaiterEntry {
resolve: () => unknown;
reject: (err: unknown) => unknown;
}
export class Locks<T> { export class Locks<T> {
/** Currently locked keys */ /** Currently locked keys */
private readonly locked = new Set<T>(); private readonly locked = new Set<T>();
/** Queue of resolve functions waiting for each key */ /** Queue of waiters for each key */
private readonly waiters = new Map< private readonly waiters = new Map<T, WaiterEntry[]>();
T,
[() => unknown, (err: unknown) => unknown][]
>();
public constructor(private readonly logger?: Logger) {} public constructor(
private readonly name: string,
private readonly logger?: Logger
) {}
/** /**
* Executes a function while holding exclusive locks on one or more keys. * Executes a function while holding exclusive locks on one or more keys.
@ -59,12 +64,17 @@ export class Locks<T> {
const uniqueKeys = Array.from(new Set(keys)); const uniqueKeys = Array.from(new Set(keys));
uniqueKeys.sort((a, b) => String(a).localeCompare(String(b))); // Ensure consistent order to prevent deadlocks uniqueKeys.sort((a, b) => String(a).localeCompare(String(b))); // Ensure consistent order to prevent deadlocks
await awaitAll(uniqueKeys.map(async (key) => this.waitForLock(key))); const lockedKeys = [];
try { try {
for (const key of uniqueKeys) {
// Must acquire locks in-order (not concurrently) to prevent deadlocks
await this.waitForLock(key);
lockedKeys.push(key);
}
return await fn(); return await fn();
} finally { } finally {
uniqueKeys.forEach((key) => { lockedKeys.forEach((key) => {
this.unlock(key); this.unlock(key);
}); });
} }
@ -74,7 +84,7 @@ export class Locks<T> {
// Resolve all waiting promises before clearing to prevent deadlock // Resolve all waiting promises before clearing to prevent deadlock
// Any operation waiting for a lock will be granted access immediately // Any operation waiting for a lock will be granted access immediately
for (const waiting of this.waiters.values()) { for (const waiting of this.waiters.values()) {
for (const [_, reject] of waiting) { for (const { reject } of waiting) {
reject(new SyncResetError()); reject(new SyncResetError());
} }
} }
@ -82,6 +92,10 @@ export class Locks<T> {
this.waiters.clear(); this.waiters.clear();
} }
public isLocked(key: T): boolean {
return this.locked.has(key);
}
/** /**
* Attempts to acquire a lock immediately without waiting. * Attempts to acquire a lock immediately without waiting.
* Must call `unlock()` if successful. * Must call `unlock()` if successful.
@ -111,7 +125,7 @@ export class Locks<T> {
return Promise.resolve(); return Promise.resolve();
} }
this.logger?.debug(`Waiting for lock on ${key}`); this.logger?.debug(`Waiting for lock '${this.name}' on '${key}'`);
return new Promise((resolve, reject) => { return new Promise((resolve, reject) => {
// DefaultDict behavior // DefaultDict behavior
@ -121,7 +135,10 @@ export class Locks<T> {
this.waiters.set(key, waiting); this.waiters.set(key, waiting);
} }
waiting.push([resolve, reject]); waiting.push({
resolve,
reject
});
}); });
} }
@ -134,15 +151,20 @@ export class Locks<T> {
*/ */
public unlock(key: T): void { public unlock(key: T): void {
if (!this.locked.has(key)) { if (!this.locked.has(key)) {
this.logger?.debug(
`Attempted to unlock '${this.name}' on '${key}' which is not locked`
);
return; return;
} }
// Remove first waiter to ensure FIFO order this.logger?.debug(`Releasing lock '${this.name}' on '${key}'`);
const [resolveNextWaiting, _] = this.waiters.get(key)?.shift() ?? [];
if (resolveNextWaiting) { // Remove first waiter to ensure FIFO order
this.logger?.debug(`Granted lock on ${key}`); const nextWaiter = this.waiters.get(key)?.shift();
resolveNextWaiting();
if (nextWaiter) {
this.logger?.debug(`Granted lock '${this.name}' on '${key}'`);
nextWaiter.resolve();
} else { } else {
this.locked.delete(key); this.locked.delete(key);
} }
@ -152,8 +174,8 @@ export class Locks<T> {
export class Lock { export class Lock {
private readonly locks: Locks<boolean>; private readonly locks: Locks<boolean>;
public constructor(logger?: Logger) { public constructor(name: string, logger?: Logger) {
this.locks = new Locks(logger); this.locks = new Locks(name, logger);
} }
public async withLock<R>(fn: () => R | Promise<R>): Promise<R> { public async withLock<R>(fn: () => R | Promise<R>): Promise<R> {

View file

@ -1,15 +1,15 @@
import { describe, it } from "node:test"; import { describe, it } from "node:test";
import assert from "node:assert"; import assert from "node:assert";
import { CoveredValues } from "./min-covered"; import { MinCovered } from "./min-covered";
describe("CoveredValues", () => { describe("MinCovered", () => {
it("should initialize with the given min value", () => { it("should initialize with the given min value", () => {
const covered = new CoveredValues(5); const covered = new MinCovered(5);
assert.strictEqual(covered.min, 5); assert.strictEqual(covered.min, 5);
}); });
it("should add values greater than min", () => { it("should add values greater than min", () => {
const covered = new CoveredValues(0); const covered = new MinCovered(0);
covered.add(3); covered.add(3);
assert.strictEqual(covered.min, 0); assert.strictEqual(covered.min, 0);
covered.add(1); covered.add(1);
@ -21,7 +21,7 @@ describe("CoveredValues", () => {
}); });
it("should ignore duplicate values", () => { it("should ignore duplicate values", () => {
const covered = new CoveredValues(0); const covered = new MinCovered(0);
covered.add(3); covered.add(3);
covered.add(3); covered.add(3);
covered.add(3); covered.add(3);
@ -32,7 +32,7 @@ describe("CoveredValues", () => {
}); });
it("should handle multiple consecutive values", () => { it("should handle multiple consecutive values", () => {
const covered = new CoveredValues(132); const covered = new MinCovered(132);
for (let i = 250; i > 132; i--) { for (let i = 250; i > 132; i--) {
assert.strictEqual(covered.min, 132); assert.strictEqual(covered.min, 132);
covered.add(i); covered.add(i);
@ -41,36 +41,32 @@ describe("CoveredValues", () => {
}); });
it("should handle adding values lower than current min", () => { it("should handle adding values lower than current min", () => {
const covered = new CoveredValues(5); const covered = new MinCovered(5);
covered.add(3); covered.add(3);
assert.strictEqual(covered.min, 5); assert.strictEqual(covered.min, 5);
covered.add(6); covered.add(6);
assert.strictEqual(covered.min, 6); assert.strictEqual(covered.min, 6);
}); });
it("should auto-advance when setting min value", () => { it("should auto-advance when adding the value that fills the next gap", () => {
const covered = new CoveredValues(5); const covered = new MinCovered(5);
covered.add(7); covered.add(7);
covered.add(8); covered.add(8);
covered.add(9); covered.add(9);
assert.strictEqual(covered.min, 5); assert.strictEqual(covered.min, 5);
// Setting min to 6 should auto-advance through 7, 8, 9 // Adding 6 fills the gap and auto-advances through 7, 8, 9
covered.min = 6; covered.add(6);
assert.strictEqual(covered.min, 9); assert.strictEqual(covered.min, 9);
covered.add(10); covered.add(10);
assert.strictEqual(covered.min, 10); assert.strictEqual(covered.min, 10);
}); });
it("should handle setting min value with no consecutive values", () => { it("should rewind when reset is called explicitly", () => {
const covered = new CoveredValues(5); const covered = new MinCovered(5);
covered.add(10); covered.add(7);
covered.add(15); covered.reset(3);
assert.strictEqual(covered.min, 5); assert.strictEqual(covered.min, 3);
// Setting min to 8 should not auto-advance (no consecutive values) covered.add(4);
covered.min = 8; assert.strictEqual(covered.min, 4);
assert.strictEqual(covered.min, 8);
// Add 9 to trigger auto-advance to 10
covered.add(9);
assert.strictEqual(covered.min, 10);
}); });
}); });

View file

@ -7,13 +7,13 @@
* *
* @example * @example
* ```typescript * ```typescript
* const covered = new CoveredValues(0); * const covered = new MinCovered(0);
* covered.add(2); // seenValues = [2], min = 0 * covered.add(2); // seenValues = [2], min = 0
* covered.add(1); // seenValues = [], min = 2 * covered.add(1); // seenValues = [], min = 2
* covered.min; // returns 2 * covered.min; // returns 2
* ``` * ```
*/ */
export class CoveredValues { export class MinCovered {
private seenValues: number[] = []; private seenValues: number[] = [];
public constructor(private minValue: number) {} public constructor(private minValue: number) {}
@ -22,12 +22,6 @@ export class CoveredValues {
return this.minValue; return this.minValue;
} }
public set min(value: number) {
this.minValue = Math.max(value, this.minValue);
this.seenValues = this.seenValues.filter((v) => v > this.minValue);
this.advanceMinWhilePossible();
}
public add(value: number | undefined): void { public add(value: number | undefined): void {
if (value === undefined || value < this.minValue) { if (value === undefined || value < this.minValue) {
return; return;
@ -49,6 +43,11 @@ export class CoveredValues {
this.advanceMinWhilePossible(); this.advanceMinWhilePossible();
} }
public reset(minValue?: number): void {
this.minValue = minValue ?? 0;
this.seenValues = [];
}
private advanceMinWhilePossible(): void { private advanceMinWhilePossible(): void {
while ( while (
this.seenValues.length > 0 && this.seenValues.length > 0 &&

View file

@ -0,0 +1,69 @@
import type { RelativePath } from "../../sync-operations/types";
import type { TextWithCursors } from "reconcile-text";
import type { FileSystemOperations } from "../../file-operations/filesystem-operations";
export class InMemoryFileSystem implements FileSystemOperations {
protected readonly files = new Map<string, Uint8Array>();
public async listFilesRecursively(
_root: RelativePath | undefined = undefined // we don't use multi-level paths during tests
): Promise<RelativePath[]> {
return Array.from(this.files.keys());
}
public async read(path: RelativePath): Promise<Uint8Array> {
const file = this.files.get(path);
if (!file) {
throw new Error(`File ${path} does not exist`);
}
return file;
}
public async write(path: RelativePath, content: Uint8Array): Promise<void> {
this.files.set(path, content);
}
public async atomicUpdateText(
path: RelativePath,
updater: (current: TextWithCursors) => TextWithCursors
): Promise<string> {
const file = this.files.get(path);
if (!file) {
throw new Error(`File ${path} does not exist`);
}
const currentContent = new TextDecoder().decode(file);
const newContent = updater({ text: currentContent, cursors: [] }).text;
this.files.set(path, new TextEncoder().encode(newContent));
return newContent;
}
public async getFileSize(path: RelativePath): Promise<number> {
return (await this.read(path)).length;
}
public async exists(path: RelativePath): Promise<boolean> {
return this.files.has(path);
}
public async createDirectory(_path: RelativePath): Promise<void> {
// This doesn't mean anything in our virtual FS representation
}
public async delete(path: RelativePath): Promise<void> {
this.files.delete(path);
}
public async rename(
oldPath: RelativePath,
newPath: RelativePath
): Promise<void> {
const file = this.files.get(oldPath);
if (!file) {
throw new Error(`File ${oldPath} does not exist`);
}
this.files.set(newPath, file);
if (oldPath !== newPath) {
this.files.delete(oldPath);
}
}
}

View file

@ -1,10 +1,44 @@
import type { SyncClient } from "../../sync-client"; /* eslint-disable no-console */
import type { LogLine } from "../../tracing/logger"; import type { Logger, LogLine } from "../../tracing/logger";
import { LogLevel } from "../../tracing/logger"; import { LogLevel } from "../../tracing/logger";
export function logToConsole(client: SyncClient): void { const COLORS = {
client.logger.onLogEmitted.add((logLine: LogLine) => { reset: "\x1b[0m",
const formatted = `${logLine.timestamp.toISOString()} ${logLine.level} ${logLine.message}`; red: "\x1b[31m",
yellow: "\x1b[33m",
blue: "\x1b[34m",
gray: "\x1b[90m"
};
export function logToConsole(
logger: Logger,
{ useColors = true }: { useColors?: boolean } = {}
): void {
logger.onLogEmitted.add((logLine: LogLine) => {
const timestamp = logLine.timestamp.toISOString();
const { message } = logLine;
let color = "";
let reset = "";
if (useColors) {
({ reset } = COLORS);
switch (logLine.level) {
case LogLevel.ERROR:
color = COLORS.red;
break;
case LogLevel.WARNING:
color = COLORS.yellow;
break;
case LogLevel.INFO:
color = COLORS.blue;
break;
case LogLevel.DEBUG:
color = COLORS.gray;
break;
}
}
const formatted = `${timestamp} ${color}${logLine.level}${reset} ${message}`;
switch (logLine.level) { switch (logLine.level) {
case LogLevel.ERROR: case LogLevel.ERROR:

View file

@ -11,7 +11,7 @@ export function slowWebSocketFactory(
private static readonly RECEIVE_KEY = "websocket-receive"; private static readonly RECEIVE_KEY = "websocket-receive";
private static readonly SEND_KEY = "websocket-send"; private static readonly SEND_KEY = "websocket-send";
private readonly locks = new Locks(logger); private readonly locks = new Locks(FlakyWebSocket.name, logger);
public set onopen(callback: ((event: Event) => void) | null) { public set onopen(callback: ((event: Event) => void) | null) {
super.onopen = async (event: Event): Promise<void> => { super.onopen = async (event: Event): Promise<void> => {

View file

@ -1,14 +1,17 @@
import type { DocumentRecord } from "../persistence/database"; import type { DocumentRecord } from "../sync-operations/types";
import { EMPTY_HASH } from "./hash"; import { EMPTY_HASH } from "./hash";
// TODO: make this smarter so that offline files can be renamed & edited at the same time // TODO: make this smarter so that offline files can be renamed & edited at the same time
export function findMatchingFile( export async function findMatchingFile(
contentHash: string, contentHash: string,
candidates: DocumentRecord[] candidates: DocumentRecord[]
): DocumentRecord | undefined { ): Promise<DocumentRecord | undefined> {
if (contentHash === EMPTY_HASH) { if (contentHash === (await EMPTY_HASH)) {
return undefined; return undefined;
} }
return candidates.find(({ metadata }) => metadata?.hash === contentHash); return candidates.find(
(record) =>
record.remoteHash !== undefined && record.remoteHash === contentHash
);
} }

View file

@ -1,12 +1,14 @@
// https://stackoverflow.com/questions/7616461/generate-a-hash-from-string-in-javascript export async function hash(content: Uint8Array): Promise<string> {
export function hash(content: Uint8Array): string { // Re-wrap into a fresh Uint8Array<ArrayBuffer> so SubtleCrypto's
let result = 0; // BufferSource overload accepts it without an unsafe type assertion.
// eslint-disable-next-line @typescript-eslint/prefer-for-of // The lib types require an ArrayBuffer-backed view; the source may
for (let i = 0; i < content.length; i++) { // be backed by SharedArrayBuffer in some runtimes.
result = (result << 5) - result + content[i]; const buffer = new ArrayBuffer(content.byteLength);
result |= 0; // Convert to 32bit integer new Uint8Array(buffer).set(content);
} const digest = await crypto.subtle.digest("SHA-256", buffer);
return Math.abs(result).toString(16).padStart(8, "0"); const bytes = new Uint8Array(digest);
return Array.from(bytes, (b) => b.toString(16).padStart(2, "0")).join("");
} }
export const EMPTY_HASH = hash(new Uint8Array(0)); // SHA-256 of empty content, computed once at import time
export const EMPTY_HASH: Promise<string> = hash(new Uint8Array());

View file

@ -1,4 +1,4 @@
import { createPromise } from "./create-promise"; import { awaitAll } from "./await-all";
import { sleep } from "./sleep"; import { sleep } from "./sleep";
/** /**
@ -45,18 +45,16 @@ export function rateLimit<
newArgs = undefined; newArgs = undefined;
} }
const [promise, resolve] = createPromise(); // `running` must signal both "minimum interval has elapsed" *and*
running = promise; // "fn() has finished" — otherwise an `fn` that takes longer than
sleep( // the interval would let a queued waiter fire a concurrent `fn`
const interval =
typeof minIntervalMs === "function" typeof minIntervalMs === "function"
? minIntervalMs() ? minIntervalMs()
: minIntervalMs : minIntervalMs;
) const fnPromise = fn(...args);
.then(resolve) running = awaitAll([fnPromise.catch(() => undefined), sleep(interval)]);
.catch(() => { return fnPromise;
// sleep cannot fail
});
return fn(...args);
}; };
return decoratedFn; return decoratedFn;

View file

@ -12,7 +12,5 @@
"declaration": true, "declaration": true,
"declarationDir": "./dist/types" "declarationDir": "./dist/types"
}, },
"exclude": [ "exclude": ["./dist"]
"./dist"
]
} }

View file

@ -49,11 +49,6 @@ module.exports = [
type: "umd" type: "umd"
}, },
globalObject: "this" globalObject: "this"
},
resolve: {
fallback: {
ws: false // Exclude `ws` from the browser bundle
}
} }
}), }),
merge(common, { merge(common, {
@ -62,10 +57,6 @@ module.exports = [
path: path.resolve(__dirname, "dist"), path: path.resolve(__dirname, "dist"),
filename: "sync-client.node.js", filename: "sync-client.node.js",
libraryTarget: "commonjs2" libraryTarget: "commonjs2"
},
externals: {
bufferutil: "bufferutil",
"utf-8-validate": "utf-8-validate" // required for ws: https://github.com/websockets/ws/issues/2245#issuecomment-2250318733
} }
}) })
]; ];

6
package-lock.json generated Normal file
View file

@ -0,0 +1,6 @@
{
"name": "vault-link",
"lockfileVersion": 3,
"requires": true,
"packages": {}
}

Some files were not shown because too many files have changed in this diff Show more