andras/vault-link
1
0
Fork 0
Code Issues Pull requests 18 Projects Releases 55 Packages Wiki Activity Actions 1

ai

Browse source
This commit is contained in:
Andras Schmelczer 2026-03-21 12:47:39 +00:00
parent 8f2f5e4fa9
commit a20264bcaf
112 changed files with 12567 additions and 2694 deletions
Download patch file Download diff file Expand all files Collapse all files

197
frontend/sync-client/ARCHITECTURE.md Normal file
View file

@ -0,0 +1,197 @@
# Sync Client Architecture
## Overview
The sync client synchronizes Obsidian vault files between clients via a central server. It handles offline edits, concurrent multi-client changes, crash recovery, and real-time updates via WebSocket.
## Architecture Layers
```
SyncClient (public API — unchanged)
│
├── Syncer (event router + reconciliation orchestrator)
│ │
│ ├── SyncEventQueue (per-document coalescing FIFO)
│ │ │
│ │ └── executor callback → sync-actions functions
│ │
│ └── VirtualFilesystem (document identity + state tracking)
│
├── WebSocketManager (connection, message serialization)
├── FileOperations (filesystem abstraction, 3-way merge on write)
├── SyncService (HTTP client for server REST API)
├── CursorTracker (collaborative cursor positions)
└── ContentCache (LRU cache for diff computation)
```
## Key Design Decisions
### 1. Sequential Processing
All sync operations run one at a time. No concurrent sync operations, no locks, no deadlock prevention, no generation counters. The server uses SQLite which serializes writes anyway, so client-side parallelism provided no real benefit while creating enormous complexity.
The only concurrency that remains is between the sync queue and the WebSocket message handler, but both funnel events into the same sequential queue.
### 2. Virtual Filesystem (VFS)
Replaces the old `Database` class. Documents have explicit states as a discriminated union:
```typescript
type VirtualDocument =
| PendingDocument // Created locally, server doesn't know yet
| TrackedDocument // Synced with server
| DeletedLocallyDocument // Deleted locally, server not yet notified
```
Three internal indexes replace the old flat array + `parallelVersion` system:
- `pathIndex` — at most one live document per path
- `documentIdIndex` — all documents with a server-assigned ID
- `idempotencyKeyIndex` — pending documents only
No more inferring state from `metadata === undefined` + `isDeleted` + `parentVersionId === 0`. The state field is the discriminator.
### 3. Per-Document Event Coalescing
Events from file watchers and WebSocket broadcasts are grouped by document identity and coalesced:
- 10 rapid edits → 1 sync operation (content read at execution time)
- create then delete → noop (file never reached the server)
- move A→B then B→C → move A→C
This replaces the old opaque-closure FIFO where every event was independent.
### 4. Server Protocol Unchanged
The server still does 3-way merging via `reconcile-text`. Response types (`FastForwardUpdate`, `MergingUpdate`) are unchanged. The client content cache remains for diff computation (needed for mobile bandwidth). Idempotency keys remain for crash-safe creates. No server changes were made.
## Module Descriptions
### `persistence/vfs.ts` — Virtual Filesystem
Tracks document identity across creates, moves, deletes. Provides:
- State transitions: `createPending()`, `confirmCreate()`, `assignDocumentId()`, `updateTracked()`, `deleteLocally()`, `confirmDelete()`
- Queries: `getByPath()`, `getByDocumentId()`, `getByIdempotencyKey()`
- Disk reconciliation: `reconcileWithDisk()` returns a pure result comparing VFS state against filesystem
- Persistence: serializes to `StoredDatabase` format (backward compatible)
### `sync-operations/sync-events.ts` — Event Types + Coalescing
Pure functions with no side effects. Defines `SyncEvent` (6 types from file watchers and WebSocket) and `CoalescedAction` (8 possible merged actions). The `coalesce()` function implements a 48-entry transition table.
### `sync-operations/sync-event-queue.ts` — Event Queue
Per-document coalescing FIFO. Maps each event to a document key (documentId for tracked docs, `path:<relativePath>` for pending docs). Processes one document at a time via an injected executor. Supports key migration when pending docs receive a documentId.
On reset (WebSocket disconnect): remote events are cleared (server replays on reconnect), local events are preserved (unsynced user actions).
### `sync-operations/sync-actions.ts` — Sync Action Implementations
Extracted from the old `unrestricted-syncer.ts`. Each function takes explicit dependencies (`SyncDeps`) and a VFS document:
- `executeSyncCreate()` — POST to server with idempotencyKey, handle response
- `executeSyncUpdate()` — compute diff from cache, PUT to server
- `executeSyncDelete()` — DELETE on server, confirm in VFS
- `executeRemoteUpdate()` — download content, write to disk, update VFS
- `applyServerResponse()` — handle MergingUpdate/FastForwardUpdate, path changes, idempotent returns
### `sync-operations/syncer.ts` — Orchestrator
Thin layer that:
- Converts file change events → `SyncEvent` objects → enqueue
- Converts WebSocket broadcasts → `SyncEvent` objects → enqueue
- Sets up the executor that dispatches `CoalescedAction` → sync-actions functions
- Runs offline reconciliation: resolve idempotency keys → scan filesystem → enqueue results
- Manages the `scheduleSyncForOfflineChanges` lifecycle
## Offline Reconciliation Algorithm
Runs on startup and WebSocket reconnect:
1. **Resolve idempotency keys** — call server for pending creates whose responses were lost
2. **Clean up orphans** — remove pending docs whose files no longer exist
3. **Scan filesystem** — `vfs.reconcileWithDisk()` compares VFS state vs actual files
4. **Apply moves** — update VFS for detected file moves (content hash matching)
5. **Enqueue events** in order:
- Interrupted deletes (VFS says deleted-locally, file gone, server not notified)
- Moves (detected via hash matching)
- Updates (file content changed)
- Creates (new files with no VFS entry)
- Delete candidates (VFS entry but file missing, not matched as a move)
Creates run before delete candidates so the server can merge creates with existing documents (preserving documentIds).
## Document Lifecycle
```
[File created locally]
→ VFS: createPending(path) → PendingDocument
→ Queue: enqueue local-create
→ Action: POST /documents with idempotencyKey
→ Server: returns FastForwardUpdate or MergingUpdate
→ VFS: confirmCreate() → TrackedDocument
[File edited locally]
→ Queue: enqueue local-update
→ Action: compute diff, PUT /documents/:id/text
→ Server: returns FastForwardUpdate or MergingUpdate
→ VFS: updateTracked()
[File deleted locally]
→ VFS: deleteLocally() → DeletedLocallyDocument (or removed if pending)
→ Queue: enqueue local-delete
→ Action: DELETE /documents/:id
→ VFS: confirmDelete() → removed
[Remote update via WebSocket]
→ Queue: enqueue remote-update
→ Action: fetch content, write to disk
→ VFS: updateTracked()
[Crash during create → restart]
→ VFS loads PendingDocument from disk (idempotencyKey preserved)
→ resolveIdempotencyKeys() maps key → documentId
→ VFS: assignDocumentId() → TrackedDocument with serverVersion=0
→ Queue: enqueue local-create (retry)
→ Server: returns existing document (idempotent)
→ VFS: updateTracked() with real serverVersion
```
## Invariants
1. **All state mutations go through the sequential queue.** No document state can change while a sync operation is running. File-change handlers and WebSocket handlers only enqueue events.
2. **Content cache stores server content after merges.** The cache is used for diff computation: `diff(cached_server_content, new_local_content)`. The server applies diffs against its content at `parentVersionId`.
3. **Idempotency keys survive crashes.** VFS persists pending documents with their keys. On restart, `resolveIdempotencyKeys` maps keys to documentIds. The key is preserved on TrackedDocument when `serverVersion === 0` so retry creates remain idempotent.
4. **Write file before updating metadata.** If the write fails, metadata still points to the old version. On recovery, the stale `serverVersion` triggers a re-fetch from server.
5. **Local events survive reset.** When the WebSocket disconnects, remote events are cleared (server replays on reconnect) but local events are preserved in the queue as unsynced user actions.
6. **Creates run before delete candidates** in the reconciliation ordering. A create may adopt a "deleted" document's identity via server-side merge.
## What Was Removed
- **PQueue** — configurable concurrency queue (replaced by sequential event queue)
- **Locks** — per-document multi-key locks with alphabetical ordering
- **Generation counters** — `resetGeneration` for stale operation detection
- **`containsDocument` guards** — 11 guards after async operations for concurrent-delete protection
- **`parentVersionIdForUpdate` snapshots** — mutable reference protection
- **`parallelVersion`** — collision tracking for multiple docs at same path
- **`UnrestrictedSyncer`** — 1,169-line class with nested if/else (replaced by sync-actions.ts with explicit dispatch)
- **`Database` class usage** — replaced by VFS everywhere (class still exists for type exports)
## Files
```
persistence/
vfs.ts 779 lines — Virtual Filesystem
database.ts 535 lines — Type definitions only (StoredDatabase, RelativePath, etc.)
sync-operations/
syncer.ts 615 lines — Orchestrator
sync-actions.ts 1229 lines — Action implementations
sync-event-queue.ts 242 lines — Per-document coalescing queue
sync-events.ts 297 lines — Event types + coalescing logic
unrestricted-syncer.ts 1169 lines — DEAD CODE (not imported, to be deleted)
cursor-tracker.ts 273 lines — Cursor position tracking
```

12
frontend/sync-client/package.json
View file

@ -13,18 +13,22 @@
"test": "tsx --test 'src/**/*.test.ts'"
},
"devDependencies": {
"@sentry/browser": "^10.30.0",
"@types/murmurhash3js-revisited": "^3.0.3",
"@types/node": "^25.0.2",
"byte-base64": "^1.1.0",
"minimatch": "^10.1.1",
"p-queue": "^9.0.1",
"reconcile-text": "^0.8.0",
"@types/node": "^25.0.2",
"reconcile-text": "^0.11.0",
"ts-loader": "^9.5.4",
"tslib": "2.8.1",
"tsx": "^4.21.0",
"typescript": "5.9.3",
"webpack": "^5.103.0",
"webpack-cli": "^6.0.1",
"webpack-merge": "^6.0.1",
"@sentry/browser": "^10.30.0"
"webpack-merge": "^6.0.1"
},
"dependencies": {
"murmurhash3js-revisited": "^3.0.0"
}
}

2
frontend/sync-client/src/consts.ts
View file

@ -5,3 +5,5 @@ export const MAX_HISTORY_ENTRY_COUNT = 5000;
export const SUPPORTED_API_VERSION = 3;
export const WEBSOCKET_DISCONNECT_TIMEOUT_IN_SECONDS = 10;
export const WEBSOCKET_CONNECTION_TIMEOUT_IN_SECONDS = 10;
export const WEBSOCKET_HEARTBEAT_INTERVAL_MS = 30_000;
export const WEBSOCKET_HEARTBEAT_TIMEOUT_MS = 90_000;

28
frontend/sync-client/src/file-operations/file-operations.test.ts
View file

@ -1,9 +1,6 @@
import { describe, it } from "node:test";
import type {
Database,
DocumentRecord,
RelativePath
} from "../persistence/database";
import type { RelativePath } from "../persistence/database";
import type { VirtualFilesystem } from "../persistence/vfs";
import { FileOperations } from "./file-operations";
import { Logger } from "../tracing/logger";
import { assertSetContainsExactly } from "../utils/assert-set-contains-exactly";
@ -21,17 +18,14 @@ class MockServerConfig implements Pick<ServerConfig, "getConfig"> {
}
}
class MockDatabase implements Partial<Database> {
public getLatestDocumentByRelativePath(
_target: RelativePath
): DocumentRecord | undefined {
// no-op
class MockVfs implements Partial<VirtualFilesystem> {
public getByPath(_path: string): undefined {
return undefined;
}
public move(
_oldRelativePath: RelativePath,
_newRelativePath: RelativePath
_oldPath: string,
_newPath: string
): void {
// no-op
}
@ -89,7 +83,7 @@ describe("File operations", () => {
const fileSystemOperations = new FakeFileSystemOperations();
const fileOperations = new FileOperations(
new Logger(),
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
);
@ -119,7 +113,7 @@ describe("File operations", () => {
const fileSystemOperations = new FakeFileSystemOperations();
const fileOperations = new FileOperations(
new Logger(),
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
);
@ -159,7 +153,7 @@ describe("File operations", () => {
const fileSystemOperations = new FakeFileSystemOperations();
const fileOperations = new FileOperations(
new Logger(),
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
);
@ -178,7 +172,7 @@ describe("File operations", () => {
const fileSystemOperations = new FakeFileSystemOperations();
const fileOperations = new FileOperations(
new Logger(),
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
);
@ -207,7 +201,7 @@ describe("File operations", () => {
const fileSystemOperations = new FakeFileSystemOperations();
const fileOperations = new FileOperations(
new Logger(),
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
fileSystemOperations,
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
);

226
frontend/sync-client/src/file-operations/file-operations.ts
View file

@ -1,12 +1,15 @@
import type { Logger } from "../tracing/logger";
import type { FileSystemOperations } from "./filesystem-operations";
import type { Database, RelativePath } from "../persistence/database";
import type { RelativePath } from "../persistence/database";
import type { VirtualFilesystem } from "../persistence/vfs";
import { SafeFileSystemOperations } from "./safe-filesystem-operations";
import type { TextWithCursors } from "reconcile-text";
import { reconcile } from "reconcile-text";
import { isFileTypeMergable } from "../utils/is-file-type-mergable";
import { isBinary } from "../utils/is-binary";
import { decodeText, normalizeToUtf8 } from "../utils/decode-text";
import type { ServerConfig } from "../services/server-config";
import { validateRelativePath } from "../utils/validate-relative-path";
export class FileOperations {
private static readonly PARENTHESES_REGEX = / \((?<count>\d+)\)$/;
@ -14,7 +17,7 @@ export class FileOperations {
public constructor(
private readonly logger: Logger,
private readonly database: Database,
private readonly vfs: VirtualFilesystem,
fs: FileSystemOperations,
private readonly serverConfig: ServerConfig,
private readonly nativeLineEndings = "\n"
@ -41,7 +44,8 @@ export class FileOperations {
}
public async read(path: RelativePath): Promise<Uint8Array> {
return this.fromNativeLineEndings(await this.fs.read(path));
const raw = await this.fs.read(path);
return this.fromNativeLineEndings(normalizeToUtf8(raw));
}
/**
@ -54,25 +58,96 @@ export class FileOperations {
path: RelativePath,
newContent: Uint8Array
): Promise<void> {
validateRelativePath(path);
await this.ensureClearPath(path);
return this.fs.write(path, this.toNativeLineEndings(newContent));
}
public async ensureClearPath(path: RelativePath): Promise<void> {
if (await this.fs.exists(path)) {
public async ensureClearPath(path: RelativePath): Promise<RelativePath | undefined> {
validateRelativePath(path);
// Acquire the lock on `path` first, then check existence inside the
// lock. The previous code checked exists() before locking, which
// created a TOCTOU race: two concurrent calls could both see the
// file as existing, but the second one would try to rename a file
// that was already moved by the first.
await this.fs.waitForLock(path);
try {
return await this.ensureClearPathLocked(path);
} finally {
this.fs.unlock(path);
}
}
/**
* Internal implementation of `ensureClearPath` that assumes the caller
* already holds the file-level lock on `path`. This allows callers like
* `move()` to keep the lock held across both the clear and the subsequent
* rename, closing the race window where another operation could create a
* file at `path` between the two steps.
*/
private async ensureClearPathLocked(
path: RelativePath
): Promise<RelativePath | undefined> {
if (await this.fs.exists(path, true)) {
const deconflictedPath = await this.deconflictPath(path);
try {
this.logger.debug(
`Didn't expect ${path} to exist, deconflicting by moving it to '${deconflictedPath}'`
);
this.database.move(path, deconflictedPath);
// deconflictedPath is already locked via tryLock in
// deconflictPath(), so we pass skipLock=true to the
// rename to avoid deadlocking on the destination lock.
await this.fs.rename(path, deconflictedPath, true);
try {
this.vfs.move(path, deconflictedPath);
// Tell the sync system this displacement is system-initiated
// (not a user rename) by setting remoteRelativePath to the
// deconflicted path. This makes the check in
// syncLocallyUpdatedFile (remoteRelativePath === relativePath)
// pass, preventing the displacement from being uploaded as a
// rename to the server. Without this, the rename event from
// fs.rename() triggers an update with the deconflicted path,
// the server deconflicts further, and an infinite cascade
// ensues. The force:true content-match shortcut ensures that
// when the server eventually broadcasts the document's real
// path, the client just updates metadata without moving the
// file back.
const displacedDoc =
this.vfs.getByPath(deconflictedPath);
if (
displacedDoc?.state === "tracked" &&
displacedDoc.remoteRelativePath !== undefined
) {
displacedDoc.remoteRelativePath =
deconflictedPath;
}
} catch (e) {
// vfs.move() failed (e.g., a non-deleted document
// already exists at deconflictedPath). Revert the
// filesystem rename to keep file and VFS
// consistent. If the revert also fails, log it —
// scheduleSyncForOfflineChanges will reconcile.
this.logger.warn(
`vfs.move(${path}, ${deconflictedPath}) failed in ensureClearPath: ${e}, reverting filesystem rename`
);
try {
await this.fs.rename(deconflictedPath, path, true);
} catch (revertError) {
this.logger.warn(
`Failed to revert filesystem rename from ${deconflictedPath} to ${path}: ${revertError}`
);
}
throw e;
}
} finally {
this.fs.unlock(deconflictedPath);
}
return deconflictedPath;
} else {
await this.createParentDirectories(path);
return undefined;
}
}
@ -87,6 +162,7 @@ export class FileOperations {
expectedContent: Uint8Array,
newContent: Uint8Array
): Promise<void> {
validateRelativePath(path);
if (!(await this.fs.exists(path))) {
this.logger.debug(
`The caller assumed ${path} exists, but it no longer, so we wont recreate it`
@ -113,8 +189,10 @@ export class FileOperations {
return;
}
const expectedText = new TextDecoder().decode(expectedContent); // this comes from a previous read which must only have \n line endings
const newText = new TextDecoder().decode(newContent); // this comes from the server which stores text with \n line endings
const expectedText = (decodeText(expectedContent) ?? "").normalize(
"NFC"
); // this comes from a previous read which must only have \n line endings
const newText = (decodeText(newContent) ?? "").normalize("NFC"); // this comes from the server which stores text with \n line endings
await this.fs.atomicUpdateText(
path,
@ -123,12 +201,29 @@ export class FileOperations {
`Performing a 3-way merge for ${path} with the expected content`
);
text = text.replaceAll(this.nativeLineEndings, "\n");
const merged = reconcile(
expectedText,
{ text, cursors },
newText
);
text = text
.replaceAll(this.nativeLineEndings, "\n")
.normalize("NFC");
let merged: TextWithCursors;
try {
merged = reconcile(
expectedText,
{ text, cursors },
newText,
"Markdown"
);
} catch {
// 3-way merge failed (e.g., content was fully replaced
// by another agent). Save the local content as a conflict
// file before overwriting with the server's content, so
// the user's edits are never silently lost.
this.logger.info(
`3-way merge failed for ${path}, saving local content as conflict file and using server content`
);
this.saveConflictFile(path, text);
merged = { text: newText, cursors: [] };
}
const resultText = merged.text.replaceAll(
"\n",
@ -144,6 +239,7 @@ export class FileOperations {
}
public async delete(path: RelativePath): Promise<void> {
validateRelativePath(path);
if (await this.exists(path)) {
await this.fs.delete(path);
await this.deletingEmptyParentDirectoriesOfDeletedFile(path);
@ -164,13 +260,46 @@ export class FileOperations {
oldPath: RelativePath,
newPath: RelativePath
): Promise<void> {
validateRelativePath(oldPath);
validateRelativePath(newPath);
if (oldPath === newPath) {
return;
}
await this.ensureClearPath(newPath);
this.database.move(oldPath, newPath);
await this.fs.rename(oldPath, newPath);
// Hold the newPath lock across both ensureClearPath and rename.
// Without this, another operation could create a file at newPath
// between ensureClearPath releasing the lock and rename acquiring
// it, causing the rename to silently overwrite the new file.
await this.fs.waitForLock(newPath);
try {
await this.ensureClearPathLocked(newPath);
// skipLock=true because we already hold the newPath lock.
// The oldPath lock is not needed; sync operations run
// sequentially so no concurrent operation can race on paths.
await this.fs.rename(oldPath, newPath, true);
} finally {
this.fs.unlock(newPath);
}
try {
this.vfs.move(oldPath, newPath);
} catch (e) {
// vfs.move() failed (e.g., a non-deleted document already
// exists at newPath). Revert the filesystem rename to keep the
// file and VFS consistent. If the revert also fails, log
// it — scheduleSyncForOfflineChanges will reconcile on the
// next cycle.
this.logger.warn(
`vfs.move(${oldPath}, ${newPath}) failed: ${e}, reverting filesystem rename`
);
try {
await this.fs.rename(newPath, oldPath);
} catch (revertError) {
this.logger.warn(
`Failed to revert filesystem rename from ${newPath} to ${oldPath}: ${revertError}`
);
}
throw e;
}
await this.deletingEmptyParentDirectoriesOfDeletedFile(oldPath);
}
@ -204,25 +333,60 @@ export class FileOperations {
}
private fromNativeLineEndings(content: Uint8Array): Uint8Array {
if (isBinary(content)) {
const text = decodeText(content);
if (text === undefined) {
return content;
}
const decoder = new TextDecoder("utf-8");
let text = decoder.decode(content);
text = text.replaceAll(this.nativeLineEndings, "\n");
return new TextEncoder().encode(text);
const normalized = text.replaceAll(this.nativeLineEndings, "\n");
return new TextEncoder().encode(normalized);
}
private toNativeLineEndings(content: Uint8Array): Uint8Array {
if (isBinary(content)) {
const text = decodeText(content);
if (text === undefined) {
return content;
}
const decoder = new TextDecoder("utf-8");
let text = decoder.decode(content);
text = text.replaceAll("\n", this.nativeLineEndings);
return new TextEncoder().encode(text);
const normalized = text.replaceAll("\n", this.nativeLineEndings);
return new TextEncoder().encode(normalized);
}
/**
* Save the local content of a file as a conflict file when 3-way merge
* fails, so the user's edits are never silently lost. The conflict file
* is created at a deconflicted path (e.g., "file (conflict 1).md").
*
* This is fire-and-forget — errors are logged but do not prevent the
* caller from proceeding with the server's content.
*/
private saveConflictFile(
path: RelativePath,
localContent: string
): void {
const contentBytes = new TextEncoder().encode(
localContent.replaceAll("\n", this.nativeLineEndings)
);
// Fire-and-forget: we don't want a failed conflict-save to prevent
// the server content from being written.
void (async () => {
try {
const conflictPath =
await this.deconflictPath(path);
try {
await this.fs.write(conflictPath, contentBytes);
this.logger.info(
`Saved local content as conflict file: ${conflictPath}`
);
} finally {
this.fs.unlock(conflictPath);
}
} catch (e) {
this.logger.warn(
`Failed to save conflict file for ${path}: ${e}`
);
}
})();
}
private async createParentDirectories(path: string): Promise<void> {
@ -275,10 +439,12 @@ export class FileOperations {
// Avoid multiple deconflictPath calls returning the same path
if (this.fs.tryLock(newName)) {
const newDocument =
this.database.getLatestDocumentByRelativePath(newName);
// getByPath only returns live docs (pending/tracked), not
// deleted-locally ones, so a non-undefined result means
// the path is occupied.
const existingDoc = this.vfs.getByPath(newName);
if (
newDocument?.isDeleted === false || // the document might have been confirmed by the server at a new path but haven't yet moved there locally
existingDoc !== undefined || // the document might have been confirmed by the server at a new path but haven't yet moved there locally
(await this.fs.exists(newName, true))
) {
this.fs.unlock(newName);

385
frontend/sync-client/src/persistence/database.ts
View file

@ -1,26 +1,14 @@
import type { Logger } from "../tracing/logger";
import { EMPTY_HASH } from "../utils/hash";
import { CoveredValues } from "../utils/data-structures/min-covered";
import { awaitAll } from "../utils/await-all";
import { removeFromArray } from "../utils/remove-from-array";
export type VaultUpdateId = number;
export type DocumentId = string;
export type RelativePath = string;
export interface DocumentMetadata {
documentId: DocumentId;
parentVersionId: VaultUpdateId;
hash: string;
remoteRelativePath?: RelativePath;
}
export interface StoredDocumentMetadata {
relativePath: RelativePath;
documentId: DocumentId;
parentVersionId: VaultUpdateId;
remoteRelativePath?: RelativePath;
hash: string;
isDeleted?: boolean;
}
export interface StoredPendingDocument {
@ -34,374 +22,3 @@ export interface StoredDatabase {
pendingDocuments?: StoredPendingDocument[];
lastSeenUpdateId: VaultUpdateId | undefined;
}
/**
* Represents a document in the database.
*
* It is mutable and its content should always represent the latest
* state of the document on disk based on the update events we have seen.
*/
export interface DocumentRecord {
relativePath: RelativePath;
metadata: DocumentMetadata | undefined;
isDeleted: boolean;
parallelVersion: number;
/** The path when this pending document was first created locally.
* Survives renames so we can match it against server responses
* when a create request succeeded but the response was lost. */
originalCreationPath?: RelativePath;
idempotencyKey?: string;
}
export class Database {
private documents: DocumentRecord[];
private lastSeenUpdateIds: CoveredValues;
public constructor(
private readonly logger: Logger,
initialState: Partial<StoredDatabase> | undefined,
private readonly saveData: (data: StoredDatabase) => Promise<void>
) {
initialState ??= {};
const validDocuments = (initialState.documents ?? []).filter(
(doc) =>
this.validateStoredField(doc, "relativePath", "string") &&
this.validateStoredField(doc, "documentId", "string") &&
this.validateStoredField(doc, "parentVersionId", "number")
);
this.documents = validDocuments.map(
({ relativePath, ...metadata }) => ({
relativePath,
metadata,
isDeleted: false,
parallelVersion: 0
})
);
const validPendingDocuments = (
initialState.pendingDocuments ?? []
).filter(
(doc) =>
this.validateStoredField(doc, "relativePath", "string") &&
this.validateStoredField(doc, "idempotencyKey", "string")
);
for (const pending of validPendingDocuments) {
const existing = this.getLatestDocumentByRelativePath(
pending.relativePath
);
this.documents.push({
relativePath: pending.relativePath,
metadata: undefined,
isDeleted: false,
parallelVersion:
existing !== undefined
? existing.parallelVersion + 1
: 0,
originalCreationPath: pending.originalCreationPath,
idempotencyKey: pending.idempotencyKey
});
}
this.ensureConsistency();
this.logger.debug(`Loaded ${this.documents.length} documents`);
const { lastSeenUpdateId } = initialState;
this.logger.debug(`Loaded last seen update id: ${lastSeenUpdateId}`);
this.lastSeenUpdateIds = new CoveredValues(
Math.max(0, lastSeenUpdateId ?? 0) // the first updateId will be 1 which is the first integer after -1
);
this.documents.forEach((doc) => {
this.lastSeenUpdateIds.add(doc.metadata?.parentVersionId);
});
}
private validateStoredField(
doc: object,
field: string,
expectedType: "string" | "number"
): boolean {
const value = (doc as Record<string, unknown>)[field];
if (
typeof value !== expectedType ||
(expectedType === "string" && !value) ||
(expectedType === "number" && isNaN(value as number))
) {
this.logger.warn(
`Skipping stored document with invalid ${field}: ${JSON.stringify(doc)}`
);
return false;
}
return true;
}
public get length(): number {
return this.documents.length;
}
public get resolvedDocuments(): DocumentRecord[] {
const paths = new Map<string, DocumentRecord[]>();
this.documents
// eslint-disable-next-line no-restricted-syntax -- Type narrowing, not removing a specific item
.filter(({ metadata }) => metadata !== undefined)
.forEach((record) =>
paths.set(record.relativePath, [
record,
...(paths.get(record.relativePath) ?? [])
])
);
return Array.from(paths.values()).map((records) => {
records.sort(
(a, b) => b.parallelVersion - a.parallelVersion // descending
);
if (
records.length > 1 &&
records.some((current, i) =>
i === 0
? false
: records[i - 1].parallelVersion ===
current.parallelVersion
)
) {
throw new Error(
`Multiple documents with the same parallel version and path at ${records[0].relativePath}`
);
}
return records[0];
});
}
public get pendingDocuments(): DocumentRecord[] {
return this.documents.filter(
(doc) => doc.metadata === undefined && !doc.isDeleted
);
}
public updateDocumentMetadata(
metadata: {
documentId: DocumentId;
parentVersionId: VaultUpdateId;
hash: string;
remoteRelativePath: RelativePath;
},
target: DocumentRecord
): void {
if (!this.documents.includes(target)) {
throw new Error("Document not found in database");
}
this.logger.debug(
`Updating document metadata for ${target.relativePath} from ${JSON.stringify(
target.metadata,
null,
2
)} to ${JSON.stringify(metadata, null, 2)}`
);
target.metadata = metadata;
this.saveInTheBackground();
}
public getLatestDocumentByRelativePath(
target: RelativePath
): DocumentRecord | undefined {
const candidates = this.documents.filter(
({ relativePath }) => relativePath === target
);
candidates.sort((a, b) => b.parallelVersion - a.parallelVersion); // descending
return candidates[0];
}
public createNewPendingDocument(
relativePath: RelativePath
): DocumentRecord {
this.logger.debug(`Creating new pending document: ${relativePath}`);
const previousEntry =
this.getLatestDocumentByRelativePath(relativePath);
const entry: DocumentRecord = {
relativePath,
metadata: undefined,
isDeleted: false,
parallelVersion:
previousEntry?.parallelVersion === undefined
? 0
: previousEntry.parallelVersion + 1,
originalCreationPath: relativePath,
idempotencyKey: crypto.randomUUID()
};
this.documents.push(entry);
// Save without consistency check — pending docs can't violate
// the documentId uniqueness invariant since they have no metadata.
void this.save().catch((error: unknown) => {
this.logger.error(`Error saving data: ${error}`);
});
return entry;
}
public getDocumentByDocumentId(
target: DocumentId
): DocumentRecord | undefined {
return this.documents.find(
({ metadata }) => metadata?.documentId === target
);
}
public move(
oldRelativePath: RelativePath,
newRelativePath: RelativePath
): void {
const oldDocument =
this.getLatestDocumentByRelativePath(oldRelativePath);
if (oldDocument === undefined) {
return;
}
const newDocument =
this.getLatestDocumentByRelativePath(newRelativePath);
if (newDocument?.isDeleted === false) {
throw new Error(
`Document already exists at new location: ${newRelativePath}`
);
}
oldDocument.relativePath = newRelativePath;
// We might be in a strange state where the target of the move has just got deleted,
// however, its metadata might already have a bunch of updates queued up for
// the document at the new location. We need to keep these updates.
oldDocument.parallelVersion =
newDocument !== undefined ? newDocument.parallelVersion + 1 : 0;
this.saveInTheBackground();
}
public delete(relativePath: RelativePath): void {
const candidate = this.getLatestDocumentByRelativePath(relativePath);
if (candidate === undefined) {
return;
}
candidate.isDeleted = true;
}
public removeDocument(target: DocumentRecord): void {
removeFromArray(this.documents, target);
this.saveInTheBackground();
}
public containsDocument(target: DocumentRecord): boolean {
return this.documents.includes(target);
}
public getLastSeenUpdateId(): VaultUpdateId {
return this.lastSeenUpdateIds.min;
}
public addSeenUpdateId(value: number): void {
const previousMin = this.lastSeenUpdateIds.min;
this.lastSeenUpdateIds.add(value);
if (previousMin !== this.lastSeenUpdateIds.min) {
this.saveInTheBackground();
}
}
public setLastSeenUpdateId(value: number): void {
this.lastSeenUpdateIds.min = value;
this.saveInTheBackground();
}
public reset(): void {
this.documents = [];
this.lastSeenUpdateIds = new CoveredValues(
0 // the first updateId will be 1 which is the first integer after -1
);
this.saveInTheBackground();
}
public async save(): Promise<void> {
return this.saveData({
documents: this.resolvedDocuments.map(
({ relativePath, metadata }) => ({
relativePath,
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
...metadata! // filtered to only docs with metadata set
})
),
pendingDocuments: this.pendingDocuments.map(
({ relativePath, idempotencyKey, originalCreationPath }) => ({
relativePath,
idempotencyKey: idempotencyKey!,
originalCreationPath: originalCreationPath!
})
),
lastSeenUpdateId: this.lastSeenUpdateIds.min
});
}
private ensureConsistency(): void {
// Check for duplicate documentIds across ALL documents with metadata,
// not just the deduplicated resolvedDocuments view. A duplicate on a
// lower-parallelVersion record would otherwise go undetected.
const allWithMetadata = this.documents
// eslint-disable-next-line no-restricted-syntax -- Type narrowing, not removing a specific item
.filter((d) => d.metadata !== undefined);
const documentIdSet = new Set<string>();
for (const doc of allWithMetadata) {
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
const docId = doc.metadata!.documentId;
if (documentIdSet.has(docId)) {
throw new Error(
`Duplicate documentId ${docId} found in database`
);
}
documentIdSet.add(docId);
}
// Also check the deduplicated view for path-level invariants
const idToPath = new Map<string, string[]>();
this.resolvedDocuments.forEach(({ relativePath, metadata }) => {
if (metadata === undefined) {
return;
}
idToPath.set(metadata.documentId, [
...(idToPath.get(metadata.documentId) ?? []),
relativePath
]);
});
const duplicates = Array.from(idToPath.entries())
.filter(([_, paths]) => paths.length > 1)
.map(([id, paths]) => {
let details = "";
for (const path of paths) {
const doc = this.getLatestDocumentByRelativePath(path);
details += `\n- ${JSON.stringify(doc, null, 2)}`;
}
return `${id} (${paths.join(", ")}): ${details}`;
});
if (duplicates.length > 0) {
throw new Error(
"Document IDs are not unique, found duplicates: " +
duplicates.join("; ")
);
}
}
private saveInTheBackground(): void {
this.ensureConsistency();
void this.save().catch((error: unknown) => {
this.logger.error(`Error saving data: ${error}`);
});
}
}

2
frontend/sync-client/src/persistence/settings.ts
View file

@ -6,7 +6,6 @@ export interface SyncSettings {
remoteUri: string;
token: string;
vaultName: string;
syncConcurrency: number;
isSyncEnabled: boolean;
maxFileSizeMB: number;
ignorePatterns: string[];
@ -21,7 +20,6 @@ export const DEFAULT_SETTINGS: SyncSettings = {
remoteUri: "",
token: "",
vaultName: "default",
syncConcurrency: 1,
isSyncEnabled: false,
maxFileSizeMB: 10,
ignorePatterns: [],

820
frontend/sync-client/src/persistence/vfs.ts Normal file
View file

@ -0,0 +1,820 @@
import type { Logger } from "../tracing/logger";
import { EMPTY_HASH } from "../utils/hash";
import { CoveredValues } from "../utils/data-structures/min-covered";
import type {
StoredDatabase,
StoredDocumentMetadata,
StoredPendingDocument,
VaultUpdateId,
DocumentId,
RelativePath
} from "./database";
// ---------------------------------------------------------------------------
// Document state types (discriminated union)
// ---------------------------------------------------------------------------
export interface PendingDocument {
readonly state: "pending";
relativePath: string;
readonly idempotencyKey: string;
readonly originalCreationPath: string;
}
export interface TrackedDocument {
readonly state: "tracked";
relativePath: string;
documentId: string;
serverVersion: number;
localHash: string;
remoteRelativePath: string;
idempotencyKey?: string;
}
export interface DeletedLocallyDocument {
readonly state: "deleted-locally";
relativePath: string;
readonly documentId: string;
readonly serverVersion: number;
readonly remoteRelativePath: string;
}
export type VirtualDocument =
| PendingDocument
| TrackedDocument
| DeletedLocallyDocument;
// ---------------------------------------------------------------------------
// Reconciliation result
// ---------------------------------------------------------------------------
export interface ReconciliationResult {
newFiles: string[];
modifiedFiles: { path: string; documentId: string }[];
missingFiles: VirtualDocument[];
movedFiles: { document: TrackedDocument; newPath: string }[];
}
// ---------------------------------------------------------------------------
// VirtualFilesystem
// ---------------------------------------------------------------------------
export class VirtualFilesystem {
/** One live document per path (pending or tracked, NOT deleted-locally). */
private readonly pathIndex = new Map<string, VirtualDocument>();
/** All documents that have a documentId (tracked + deleted-locally). */
private readonly documentIdIndex = new Map<string, VirtualDocument>();
/** Pending documents by idempotency key. */
private readonly idempotencyKeyIndex = new Map<string, PendingDocument>();
private lastSeenUpdateIds: CoveredValues;
private pendingSave: Promise<void> = Promise.resolve();
public constructor(
private readonly logger: Logger,
initialState: Partial<StoredDatabase> | undefined,
private readonly saveData: (data: StoredDatabase) => Promise<void>
) {
const state: Partial<StoredDatabase> = initialState ?? {};
const validDocuments = (state.documents ?? []).filter(
(doc) =>
this.validateStoredField(doc, "relativePath", "string") &&
this.validateStoredField(doc, "documentId", "string") &&
this.validateStoredField(doc, "parentVersionId", "number")
);
for (const stored of validDocuments) {
if (stored.isDeleted === true) {
const doc: DeletedLocallyDocument = {
state: "deleted-locally",
relativePath: stored.relativePath,
documentId: stored.documentId,
serverVersion: stored.parentVersionId,
remoteRelativePath:
stored.remoteRelativePath ?? stored.relativePath
};
// deleted-locally docs go into documentIdIndex only
this.documentIdIndex.set(doc.documentId, doc);
} else {
const doc: TrackedDocument = {
state: "tracked",
relativePath: stored.relativePath,
documentId: stored.documentId,
serverVersion: stored.parentVersionId,
localHash: stored.hash,
remoteRelativePath:
stored.remoteRelativePath ?? stored.relativePath
};
// If two stored documents have the same path, last one wins
// (matches old behavior where highest parallelVersion wins)
this.pathIndex.set(doc.relativePath, doc);
this.documentIdIndex.set(doc.documentId, doc);
}
}
const validPendingDocuments = (state.pendingDocuments ?? []).filter(
(doc) =>
this.validateStoredField(doc, "relativePath", "string") &&
this.validateStoredField(doc, "idempotencyKey", "string")
);
for (const stored of validPendingDocuments) {
// If a live doc already exists at this path, skip the pending one
// only if the live doc is tracked (has metadata). If a pending doc
// already exists, skip duplicates.
const existing = this.pathIndex.get(stored.relativePath);
if (existing?.state === "pending") {
this.logger.debug(
`Skipping duplicate pending document at ${stored.relativePath}`
);
continue;
}
const doc: PendingDocument = {
state: "pending",
relativePath: stored.relativePath,
idempotencyKey: stored.idempotencyKey,
originalCreationPath:
stored.originalCreationPath ?? stored.relativePath
};
// A pending doc at a path where a tracked doc exists: the pending
// doc takes precedence in pathIndex (mirrors old behavior where
// pending has higher parallelVersion).
this.pathIndex.set(doc.relativePath, doc);
this.idempotencyKeyIndex.set(doc.idempotencyKey, doc);
}
this.ensureConsistency();
const totalDocs =
this.pathIndex.size + this.deletedLocallyDocuments().length;
this.logger.debug(`Loaded ${totalDocs} documents`);
const { lastSeenUpdateId } = state;
this.logger.debug(`Loaded last seen update id: ${lastSeenUpdateId}`);
this.lastSeenUpdateIds = new CoveredValues(
Math.max(0, lastSeenUpdateId ?? 0)
);
// Seed CoveredValues with known server versions
for (const doc of this.documentIdIndex.values()) {
if (doc.state === "tracked") {
this.lastSeenUpdateIds.add(doc.serverVersion);
} else if (doc.state === "deleted-locally") {
this.lastSeenUpdateIds.add(doc.serverVersion);
}
}
}
// -----------------------------------------------------------------------
// Validation helper
// -----------------------------------------------------------------------
private validateStoredField(
doc: object,
field: string,
expectedType: "string" | "number"
): boolean {
const value = (doc as Record<string, unknown>)[field];
if (
typeof value !== expectedType ||
(expectedType === "string" && !value) ||
(expectedType === "number" && isNaN(value as number))
) {
this.logger.warn(
`Skipping stored document with invalid ${field}: ${JSON.stringify(doc)}`
);
return false;
}
return true;
}
// -----------------------------------------------------------------------
// Queries
// -----------------------------------------------------------------------
public getByPath(path: string): VirtualDocument | undefined {
return this.pathIndex.get(path);
}
public getByDocumentId(id: string): VirtualDocument | undefined {
return this.documentIdIndex.get(id);
}
public getByIdempotencyKey(key: string): PendingDocument | undefined {
return this.idempotencyKeyIndex.get(key);
}
public trackedDocuments(): TrackedDocument[] {
const result: TrackedDocument[] = [];
for (const doc of this.pathIndex.values()) {
if (doc.state === "tracked") {
result.push(doc);
}
}
return result;
}
public pendingDocuments(): PendingDocument[] {
const result: PendingDocument[] = [];
for (const doc of this.pathIndex.values()) {
if (doc.state === "pending") {
result.push(doc);
}
}
return result;
}
public deletedLocallyDocuments(): DeletedLocallyDocument[] {
const result: DeletedLocallyDocument[] = [];
for (const doc of this.documentIdIndex.values()) {
if (doc.state === "deleted-locally") {
result.push(doc);
}
}
return result;
}
/** All live documents (pending + tracked) that occupy a path. */
public allLiveDocuments(): VirtualDocument[] {
return Array.from(this.pathIndex.values());
}
/** Total number of documents across all indexes (live + deleted-locally). */
public get length(): number {
// pathIndex has live docs (pending + tracked).
// documentIdIndex has tracked + deleted-locally.
// Tracked docs appear in both, so count:
// pending (pathIndex only) + tracked (both) + deleted-locally (documentIdIndex only)
// = pathIndex.size + deletedLocally count
let deletedCount = 0;
for (const doc of this.documentIdIndex.values()) {
if (doc.state === "deleted-locally") {
deletedCount++;
}
}
return this.pathIndex.size + deletedCount;
}
public contains(doc: VirtualDocument): boolean {
switch (doc.state) {
case "pending":
return this.idempotencyKeyIndex.get(doc.idempotencyKey) === doc;
case "tracked":
return this.documentIdIndex.get(doc.documentId) === doc;
case "deleted-locally":
return this.documentIdIndex.get(doc.documentId) === doc;
}
}
// -----------------------------------------------------------------------
// Update ID tracking
// -----------------------------------------------------------------------
public getLastSeenUpdateId(): number {
return this.lastSeenUpdateIds.min;
}
public addSeenUpdateId(value: number): void {
const previousMin = this.lastSeenUpdateIds.min;
this.lastSeenUpdateIds.add(value);
if (previousMin !== this.lastSeenUpdateIds.min) {
this.saveInTheBackground();
}
}
public setLastSeenUpdateId(value: number): void {
this.lastSeenUpdateIds.min = value;
this.saveInTheBackground();
}
// -----------------------------------------------------------------------
// Mutations
// -----------------------------------------------------------------------
/**
* Create a pending document at the given path. If a pending document
* already exists at the path, return it (idempotent). Generates a new
* idempotency key via `crypto.randomUUID()`.
*
* Awaits save() so the idempotency key is persisted before any HTTP
* request is sent.
*/
public async createPending(path: string): Promise<PendingDocument> {
this.logger.debug(`Creating new pending document: ${path}`);
const existing = this.pathIndex.get(path);
if (existing?.state === "pending") {
this.logger.debug(
`Pending document already exists at ${path}, reusing it`
);
return existing;
}
const doc: PendingDocument = {
state: "pending",
relativePath: path,
idempotencyKey: crypto.randomUUID(),
originalCreationPath: path
};
this.pathIndex.set(path, doc);
this.idempotencyKeyIndex.set(doc.idempotencyKey, doc);
// Awaited so the idempotency key is persisted before any HTTP
// request is sent — a crash before save would lose the key.
await this.save();
return doc;
}
/**
* Confirm a pending create: transition from pending to tracked.
* Removes the pending doc and inserts a tracked doc with full metadata.
*/
public confirmCreate(
idempotencyKey: string,
documentId: DocumentId,
serverVersion: VaultUpdateId,
localHash: string,
remoteRelativePath: RelativePath
): TrackedDocument {
const pending = this.idempotencyKeyIndex.get(idempotencyKey);
if (pending === undefined) {
// The pending doc was already promoted to tracked by
// assignDocumentId (resolveIdempotencyKeys) or a previous
// confirmCreate call. Find the tracked doc and update it.
// Try by documentId first, then by scanning for the key.
let existing = this.documentIdIndex.get(documentId);
if (existing?.state !== "tracked") {
// The server may have assigned a different documentId
// (e.g., merge). Scan all tracked docs for the key.
for (const doc of this.documentIdIndex.values()) {
if (doc.state === "tracked" && doc.idempotencyKey === idempotencyKey) {
existing = doc;
break;
}
}
}
if (existing?.state === "tracked") {
// If the server assigned a different documentId than what
// assignDocumentId set, update the index.
if (existing.documentId !== documentId) {
this.documentIdIndex.delete(existing.documentId);
existing.documentId = documentId;
this.documentIdIndex.set(documentId, existing);
}
existing.serverVersion = serverVersion;
existing.localHash = localHash;
existing.remoteRelativePath = remoteRelativePath;
existing.idempotencyKey = undefined;
this.lastSeenUpdateIds.add(serverVersion);
this.saveInTheBackground();
return existing;
}
// Truly not found — nothing to update
throw new Error(
`No pending document with idempotency key ${idempotencyKey}`
);
}
const tracked: TrackedDocument = {
state: "tracked",
relativePath: pending.relativePath,
documentId,
serverVersion,
localHash,
remoteRelativePath
};
// Remove pending from indexes
this.idempotencyKeyIndex.delete(idempotencyKey);
// Update pathIndex (pending -> tracked at same path)
this.pathIndex.set(tracked.relativePath, tracked);
// Add to documentIdIndex
this.documentIdIndex.set(tracked.documentId, tracked);
this.lastSeenUpdateIds.add(serverVersion);
this.saveInTheBackground();
return tracked;
}
/**
* Assign a documentId to a pending document (used by resolveIdempotencyKeys).
* Sets serverVersion = 0 as a placeholder — the sync path must treat
* serverVersion === 0 as needing a create retry.
*
* Returns the new TrackedDocument, or undefined if the key is not found.
*/
public assignDocumentId(
idempotencyKey: string,
documentId: DocumentId
): TrackedDocument | undefined {
const pending = this.idempotencyKeyIndex.get(idempotencyKey);
if (pending === undefined) {
return undefined;
}
const tracked: TrackedDocument = {
state: "tracked",
relativePath: pending.relativePath,
documentId,
serverVersion: 0,
localHash: "",
remoteRelativePath: pending.relativePath,
idempotencyKey: pending.idempotencyKey
};
// Remove pending from indexes
this.idempotencyKeyIndex.delete(idempotencyKey);
// Update pathIndex
this.pathIndex.set(tracked.relativePath, tracked);
// Add to documentIdIndex
this.documentIdIndex.set(tracked.documentId, tracked);
this.saveInTheBackground();
return tracked;
}
/**
* Update an existing tracked document's metadata.
*/
public updateTracked(
documentId: DocumentId,
serverVersion: VaultUpdateId,
localHash: string,
remoteRelativePath: RelativePath
): void {
const doc = this.documentIdIndex.get(documentId);
if (doc?.state !== "tracked") {
throw new Error(
`Tracked document with id ${documentId} not found`
);
}
doc.serverVersion = serverVersion;
doc.localHash = localHash;
doc.remoteRelativePath = remoteRelativePath;
this.lastSeenUpdateIds.add(serverVersion);
this.saveInTheBackground();
}
/**
* Move a document from one path to another. Throws if the target path
* is occupied by a live document.
*/
public move(oldPath: string, newPath: string): void {
const doc = this.pathIndex.get(oldPath);
if (doc === undefined) {
return;
}
// If another document occupies the target path, it was likely
// orphaned by an earlier displacement that wasn't reconciled.
// Remove it from the path index — reconcileWithDisk will
// re-discover the file if it still exists on disk.
const existingAtNew = this.pathIndex.get(newPath);
if (existingAtNew !== undefined && existingAtNew !== doc) {
this.pathIndex.delete(newPath);
}
// Remove from old path
this.pathIndex.delete(oldPath);
// Update the document's relativePath
doc.relativePath = newPath;
// Insert at new path
this.pathIndex.set(newPath, doc);
this.saveInTheBackground();
}
/**
* Mark a document as deleted locally.
* - Pending: remove entirely (no server-side state to track).
* - Tracked: transition to deleted-locally (keep in documentIdIndex).
*/
public deleteLocally(path: string): void {
const doc = this.pathIndex.get(path);
if (doc === undefined) {
return;
}
// Remove from pathIndex in all cases
this.pathIndex.delete(path);
if (doc.state === "pending") {
// Remove from idempotencyKeyIndex too
this.idempotencyKeyIndex.delete(doc.idempotencyKey);
} else if (doc.state === "tracked") {
// Transition to deleted-locally
const deleted: DeletedLocallyDocument = {
state: "deleted-locally",
relativePath: doc.relativePath,
documentId: doc.documentId,
serverVersion: doc.serverVersion,
remoteRelativePath: doc.remoteRelativePath
};
// Replace in documentIdIndex
this.documentIdIndex.set(deleted.documentId, deleted);
}
this.saveInTheBackground();
}
/**
* Confirm a server-side delete: remove the document entirely.
*/
public confirmDelete(documentId: DocumentId): void {
const doc = this.documentIdIndex.get(documentId);
if (doc === undefined) {
return;
}
this.documentIdIndex.delete(documentId);
// Also remove from pathIndex if present (tracked docs are in both)
if (doc.state === "tracked") {
const atPath = this.pathIndex.get(doc.relativePath);
if (atPath === doc) {
this.pathIndex.delete(doc.relativePath);
}
}
this.saveInTheBackground();
}
/**
* Remove a document from all indexes entirely.
*/
public remove(doc: VirtualDocument): void {
switch (doc.state) {
case "pending": {
this.idempotencyKeyIndex.delete(doc.idempotencyKey);
const atPath = this.pathIndex.get(doc.relativePath);
if (atPath === doc) {
this.pathIndex.delete(doc.relativePath);
}
break;
}
case "tracked": {
this.documentIdIndex.delete(doc.documentId);
const atPath = this.pathIndex.get(doc.relativePath);
if (atPath === doc) {
this.pathIndex.delete(doc.relativePath);
}
break;
}
case "deleted-locally": {
this.documentIdIndex.delete(doc.documentId);
break;
}
}
this.saveInTheBackground();
}
/**
* Ensure no other document has the given documentId. If a different
* document already holds it, remove that document and return it (so
* the caller can do optional file-level cleanup). Returns undefined
* if no conflict exists.
*/
public ensureUniqueDocumentId(
documentId: DocumentId,
keeper: VirtualDocument
): VirtualDocument | undefined {
const existing = this.documentIdIndex.get(documentId);
if (existing !== undefined && existing !== keeper) {
this.remove(existing);
return existing;
}
return undefined;
}
// -----------------------------------------------------------------------
// Persistence
// -----------------------------------------------------------------------
public async save(): Promise<void> {
const data = this.snapshotForSave();
const previousSave = this.pendingSave;
const thisSave = (async () => {
await previousSave.catch(() => {});
await this.saveData(data);
})();
this.pendingSave = thisSave.catch(() => {});
return thisSave;
}
public saveInTheBackground(): void {
this.ensureConsistency();
void this.save().catch((error: unknown) => {
this.logger.error(`Error saving data: ${error}`);
});
}
public reset(): void {
this.pathIndex.clear();
this.documentIdIndex.clear();
this.idempotencyKeyIndex.clear();
this.lastSeenUpdateIds = new CoveredValues(0);
this.saveInTheBackground();
}
/**
* Serialize to StoredDatabase format for backward compatibility.
*/
private snapshotForSave(): StoredDatabase {
const documents: StoredDocumentMetadata[] = [];
// Tracked documents
for (const doc of this.pathIndex.values()) {
if (doc.state === "tracked") {
documents.push({
relativePath: doc.relativePath,
documentId: doc.documentId,
parentVersionId: doc.serverVersion,
hash: doc.localHash,
remoteRelativePath: doc.remoteRelativePath
});
}
}
// Deleted-locally documents (with isDeleted flag)
for (const doc of this.documentIdIndex.values()) {
if (doc.state === "deleted-locally") {
documents.push({
relativePath: doc.relativePath,
documentId: doc.documentId,
parentVersionId: doc.serverVersion,
hash: "",
isDeleted: true,
remoteRelativePath: doc.remoteRelativePath
});
}
}
// Pending documents
const pendingDocuments: StoredPendingDocument[] = [];
for (const doc of this.idempotencyKeyIndex.values()) {
pendingDocuments.push({
relativePath: doc.relativePath,
idempotencyKey: doc.idempotencyKey,
originalCreationPath: doc.originalCreationPath
});
}
return {
documents,
pendingDocuments,
lastSeenUpdateId: this.lastSeenUpdateIds.min
};
}
// -----------------------------------------------------------------------
// Consistency check
// -----------------------------------------------------------------------
private ensureConsistency(): void {
// Check that documentIdIndex has no duplicates (by construction it
// shouldn't, since it's a Map keyed by documentId). But verify that
// pathIndex entries with documentIds are consistent.
const seenDocIds = new Set<string>();
for (const doc of this.pathIndex.values()) {
if (doc.state === "tracked") {
if (seenDocIds.has(doc.documentId)) {
throw new Error(
`Duplicate documentId ${doc.documentId} found in VFS pathIndex`
);
}
seenDocIds.add(doc.documentId);
}
}
for (const doc of this.documentIdIndex.values()) {
if (doc.state === "deleted-locally") {
if (seenDocIds.has(doc.documentId)) {
throw new Error(
`Duplicate documentId ${doc.documentId} found across live and deleted documents`
);
}
seenDocIds.add(doc.documentId);
}
}
}
// -----------------------------------------------------------------------
// Disk reconciliation
// -----------------------------------------------------------------------
/**
* Compare VFS entries against files on disk and produce a pure result
* describing what changed. Does NOT mutate the VFS.
*
* @param diskFiles - List of relative paths that currently exist on disk.
* @param readAndHash - Callback to read a file and return its hash, or
* undefined if the file cannot be read.
*/
public async reconcileWithDisk(
diskFiles: string[],
readAndHash: (path: string) => Promise<string | undefined>
): Promise<ReconciliationResult> {
const diskSet = new Set(diskFiles);
const newFiles: string[] = [];
const modifiedFiles: { path: string; documentId: string }[] = [];
const missingFiles: VirtualDocument[] = [];
const movedFiles: { document: TrackedDocument; newPath: string }[] = [];
// Collect missing tracked/pending docs (file not on disk)
const missingTracked: TrackedDocument[] = [];
for (const doc of this.pathIndex.values()) {
if (!diskSet.has(doc.relativePath)) {
if (doc.state === "tracked") {
missingTracked.push(doc);
}
missingFiles.push(doc);
}
}
// For each disk file, classify it
for (const path of diskFiles) {
const doc = this.pathIndex.get(path);
if (doc === undefined) {
// File on disk, not in VFS — could be new or a move
newFiles.push(path);
} else if (doc.state === "tracked") {
// Check if content changed
const fileHash = await readAndHash(path);
if (
fileHash !== undefined &&
fileHash !== doc.localHash
) {
modifiedFiles.push({
path,
documentId: doc.documentId
});
}
}
// If pending, nothing to reconcile — it's already pending
}
// Attempt move detection: for each new file, try to match against
// a missing tracked doc by content hash
if (missingTracked.length > 0 && newFiles.length > 0) {
const remainingNew: string[] = [];
for (const path of newFiles) {
const fileHash = await readAndHash(path);
if (fileHash === undefined || fileHash === EMPTY_HASH) {
remainingNew.push(path);
continue;
}
// Find a single unique match among missing tracked docs
const matches = missingTracked.filter(
(doc) => doc.localHash === fileHash
);
if (matches.length === 1) {
const match = matches[0];
movedFiles.push({ document: match, newPath: path });
// Remove from missingTracked so it can't match again
const idx = missingTracked.indexOf(match);
if (idx !== -1) {
missingTracked.splice(idx, 1);
}
// Remove from missingFiles too
const missingIdx = missingFiles.indexOf(match);
if (missingIdx !== -1) {
missingFiles.splice(missingIdx, 1);
}
} else {
remainingNew.push(path);
}
}
// Replace newFiles with the remaining unmatched ones
newFiles.length = 0;
newFiles.push(...remainingNew);
}
return { newFiles, modifiedFiles, missingFiles, movedFiles };
}
}

26
frontend/sync-client/src/services/fetch-controller.ts
View file

@ -6,6 +6,8 @@ import { SyncResetError } from "../errors/sync-reset-error";
* Offers a resettable fetch implementation that waits until syncing is enabled
* and aborts outstanding requests when a reset is started.
*/
const HTTP_REQUEST_TIMEOUT_MS = 30_000;
export class FetchController {
private static readonly UNTIL_RESOLUTION = Symbol();
@ -81,7 +83,17 @@ export class FetchController {
}
this.isResetting = false;
[this.until, this.resolveUntil, this.rejectUntil] = createPromise();
// Capture the old resolve before creating a fresh promise, then
// resolve the old one — exactly the same pattern the canFetch
// setter uses. This wakes up any fetches that entered the
// while-loop between startReset and finishReset so they re-check
// the condition. Without this, a canFetch change that occurred
// during the reset (setter skips resolution while isResetting is
// true) would leave fetches blocking on an unresolved promise.
const previousResolve = this.resolveUntil;
[this.until, this.resolveUntil, this.rejectUntil] =
createPromise<symbol>();
previousResolve(FetchController.UNTIL_RESOLUTION);
}
/**
@ -117,7 +129,17 @@ export class FetchController {
? input.clone()
: input;
const fetchPromise = fetch(_input, init);
const combinedSignal = init?.signal
? AbortSignal.any([
AbortSignal.timeout(HTTP_REQUEST_TIMEOUT_MS),
init.signal
])
: AbortSignal.timeout(HTTP_REQUEST_TIMEOUT_MS);
const fetchPromise = fetch(_input, {
...init,
signal: combinedSignal
});
// We only want to catch rejections from `this.until`
let result: symbol | Response | undefined = undefined;

97
frontend/sync-client/src/services/sync-service.ts
View file

@ -49,13 +49,44 @@ export class SyncService {
.get("Content-Type")
?.includes("application/json") == true
) {
const result: SerializedError =
(await response.json()) as SerializedError; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
return SyncService.formatError(result);
try {
const result: SerializedError =
(await response.json()) as SerializedError; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
return SyncService.formatError(result);
} catch {
return `HTTP ${response.status}: ${response.statusText} (failed to parse error response body)`;
}
}
return `HTTP ${response.status}: ${response.statusText}`;
}
/**
* Safely parse JSON from a response body. If parsing fails (e.g., malformed
* JSON from the server), throws an HttpClientError with status 0 so that
* retryForever does not retry indefinitely.
*/
private static async parseJsonResponse<T>(
response: Response
): Promise<T> {
try {
return (await response.json()) as T; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
} catch (error) {
// Timeout and abort errors are transient — let them propagate
// so retryForever can retry. Only wrap genuine parse failures
// (malformed JSON) as HttpClientError to prevent infinite retries.
if (
error instanceof Error &&
(error.name === "TimeoutError" || error.name === "AbortError")
) {
throw error;
}
throw new HttpClientError(
0,
`Failed to parse JSON response: ${error}`
);
}
}
private static formatError(error: SerializedError): string {
let result = error.message;
if (error.causes.length > 0) {
@ -117,7 +148,9 @@ export class SyncService {
}
const result: DocumentUpdateResponse =
(await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
await SyncService.parseJsonResponse<DocumentUpdateResponse>(
response
);
this.logger.debug(`Created document ${JSON.stringify(result)}`);
@ -164,7 +197,9 @@ export class SyncService {
}
const result: DocumentUpdateResponse =
(await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
await SyncService.parseJsonResponse<DocumentUpdateResponse>(
response
);
this.logger.debug(
`Updated document ${JSON.stringify(result)} with id ${result.documentId
@ -215,7 +250,9 @@ export class SyncService {
}
const result: DocumentUpdateResponse =
(await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
await SyncService.parseJsonResponse<DocumentUpdateResponse>(
response
);
this.logger.debug(
`Updated document ${JSON.stringify(result)} with id ${result.documentId
@ -234,9 +271,7 @@ export class SyncService {
relativePath: RelativePath;
}): Promise<DocumentVersionWithoutContent> {
return this.retryForever(async () => {
const request: DeleteDocumentVersion = {
relativePath
};
const request: DeleteDocumentVersion = {};
this.logger.debug(
`Delete document with id ${documentId} and relative path ${relativePath}`
@ -259,7 +294,9 @@ export class SyncService {
}
const result: DocumentVersionWithoutContent =
(await response.json()) as DocumentVersionWithoutContent; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
await SyncService.parseJsonResponse<DocumentVersionWithoutContent>(
response
);
this.logger.debug(
`Deleted document ${relativePath} with id ${documentId}`
@ -292,7 +329,9 @@ export class SyncService {
}
const result: DocumentVersion =
(await response.json()) as DocumentVersion; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
await SyncService.parseJsonResponse<DocumentVersion>(
response
);
this.logger.debug(`Got document ${JSON.stringify(result)}`);
@ -361,7 +400,9 @@ export class SyncService {
}
const result: FetchLatestDocumentsResponse =
(await response.json()) as FetchLatestDocumentsResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
await SyncService.parseJsonResponse<FetchLatestDocumentsResponse>(
response
);
this.logger.debug(
`Got ${result.latestDocuments.length} document metadata`
@ -389,15 +430,16 @@ export class SyncService {
);
if (!response.ok) {
throw new Error(
`Failed to resolve idempotency keys: ${await SyncService.errorFromResponse(
response
)}`
await SyncService.throwHttpError(
response,
"Failed to resolve idempotency keys"
);
}
const result: { resolved: Record<string, string> } =
(await response.json()) as { resolved: Record<string, string> }; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
const result =
await SyncService.parseJsonResponse<{
resolved: Record<string, string>;
}>(response);
const resolved = new Map<string, string>(
Object.entries(result.resolved)
@ -425,7 +467,8 @@ export class SyncService {
);
}
const result: PingResponse = (await response.json()) as PingResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
const result: PingResponse =
await SyncService.parseJsonResponse<PingResponse>(response);
this.logger.debug(
`Pinged server, got response: ${JSON.stringify(result)}`
@ -457,6 +500,7 @@ export class SyncService {
}
private async retryForever<T>(fn: () => Promise<T>): Promise<T> {
let attempt = 0;
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
while (true) {
try {
@ -473,12 +517,19 @@ export class SyncService {
throw e;
}
const retryInterval =
attempt++;
const baseDelay =
this.settings.getSettings().networkRetryIntervalMs;
this.logger.error(
`Failed network call (${e}), retrying in ${retryInterval}ms`
const exponentialDelay = Math.min(
baseDelay * Math.pow(2, Math.min(attempt - 1, 5)),
30000
);
await sleep(retryInterval);
const jitter = Math.random() * exponentialDelay * 0.5;
const delay = exponentialDelay + jitter;
this.logger.error(
`Failed network call (${e}), retrying in ${Math.round(delay)}ms (attempt ${attempt})`
);
await sleep(delay);
}
}
}

2
frontend/sync-client/src/services/types/DeleteDocumentVersion.ts
View file

@ -1,3 +1,3 @@
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
export interface DeleteDocumentVersion { relativePath: string, }
export type DeleteDocumentVersion = Record<string, never>;

2
frontend/sync-client/src/services/types/WebSocketClientMessage.ts
View file

@ -2,4 +2,4 @@
import type { CursorPositionFromClient } from "./CursorPositionFromClient";
import type { WebSocketHandshake } from "./WebSocketHandshake";
export type WebSocketClientMessage = { "type": "handshake" } & WebSocketHandshake | { "type": "cursorPositions" } & CursorPositionFromClient;
export type WebSocketClientMessage = { "type": "handshake" } & WebSocketHandshake | { "type": "cursorPositions" } & CursorPositionFromClient | { "type": "ping" };

92
frontend/sync-client/src/services/websocket-manager.ts
View file

@ -34,6 +34,14 @@ export class WebSocketManager {
private readonly outstandingPromises: Promise<unknown>[] = [];
/**
* Chains WebSocket message processing so only one message is handled
* at a time. Without this, a burst of messages would create many
* concurrent sync operations (each calling scheduleSyncForOfflineChanges
* and processing documents in parallel).
*/
private messageProcessingChain: Promise<void> = Promise.resolve();
private webSocket: WebSocket | undefined;
public constructor(
@ -102,6 +110,11 @@ export class WebSocketManager {
}
}
// Wait for any already-enqueued message handlers to finish.
// The isStopped guard in onmessage prevents NEW messages from
// being enqueued, but handlers that were chained before stop()
// set the flag may still be in flight.
await this.messageProcessingChain;
await this.waitUntilFinished();
}
@ -216,29 +229,75 @@ export class WebSocketManager {
};
this.webSocket.onmessage = (event): void => {
// Discard messages received after stop() has been called.
// Without this guard, messages arriving between close()
// and the onclose event would be enqueued into
// messageProcessingChain and execute after stop() returns.
if (this.isStopped) {
return;
}
try {
// eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
const message = JSON.parse(
event.data
) as WebSocketServerMessage;
// Track the message handling promise
const messageHandlingPromise = this.handleWebSocketMessage(
message
)
// Cursor updates are pure reads (update an in-memory map) —
// handle immediately without blocking behind vault update
// processing. This avoids cursor latency during large syncs.
if (message.type === "cursorPositions") {
this.logger.debug(
`Received cursor positions for ${JSON.stringify(message.clients)}`
);
const cursorPromise =
this.onRemoteCursorsUpdateReceived
.triggerAsync(message.clients)
.catch((error: unknown) => {
this.logger.error(
`Error handling cursor update: ${String(error)}`
);
});
// Track for waitUntilFinished / hasOutstandingWork
this.outstandingPromises.push(cursorPromise);
void cursorPromise.finally(() => {
removeFromArray(
this.outstandingPromises,
cursorPromise
);
});
return;
}
// Vault updates require serialization: each waits for the
// previous one to finish. This provides back-pressure so a
// burst of WebSocket messages doesn't create unbounded
// concurrent sync operations.
//
// Read-reassign safety: we read messageProcessingChain,
// chain a .then() onto it, and assign the resulting promise
// back. This is safe because JavaScript is single-threaded:
// no other code can run between the read and the assignment.
// The next onmessage invocation will see the updated chain
// and append after this handler, preserving FIFO order.
this.messageProcessingChain = this.messageProcessingChain
.then(async () => this.handleWebSocketMessage(message))
.catch((error: unknown) => {
this.logger.error(
`Error handling WebSocket message: ${String(error)}`
);
})
.finally(() => {
removeFromArray(
this.outstandingPromises,
messageHandlingPromise
);
});
void this.outstandingPromises.push(messageHandlingPromise); // ignore the returned promise
const messageHandlingPromise = this.messageProcessingChain;
// Track the promise for waitUntilFinished / hasOutstandingWork
this.outstandingPromises.push(messageHandlingPromise);
void messageHandlingPromise.finally(() => {
removeFromArray(
this.outstandingPromises,
messageHandlingPromise
);
});
} catch (error) {
this.logger.error(
`Error parsing WebSocket message: ${String(error)}`
@ -283,17 +342,8 @@ export class WebSocketManager {
): Promise<void> {
if (message.type === "vaultUpdate") {
await this.onRemoteVaultUpdateReceived.triggerAsync(message);
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
} else if (message.type === "cursorPositions") {
this.logger.debug(
`Received cursor positions for ${JSON.stringify(message.clients)}`
);
await this.onRemoteCursorsUpdateReceived.triggerAsync(
message.clients
);
} else {
// Cursor messages are handled inline in onmessage (not chained)
this.logger.warn(
`Received unknown message type: ${JSON.stringify(message)}`
);

104
frontend/sync-client/src/sync-client.ts
View file

@ -3,7 +3,6 @@ import type { HistoryEntry, HistoryStats } from "./tracing/sync-history";
import { SyncHistory } from "./tracing/sync-history";
import { Logger, LogLevel, LogLine } from "./tracing/logger";
import type { RelativePath, StoredDatabase } from "./persistence/database";
import { Database } from "./persistence/database";
import * as Sentry from "@sentry/browser";
import type { SyncSettings } from "./persistence/settings";
import { DEFAULT_SETTINGS, Settings } from "./persistence/settings";
@ -12,7 +11,8 @@ import { Syncer } from "./sync-operations/syncer";
import type { FileSystemOperations } from "./file-operations/filesystem-operations";
import { FileOperations } from "./file-operations/file-operations";
import { FetchController } from "./services/fetch-controller";
import { UnrestrictedSyncer } from "./sync-operations/unrestricted-syncer";
import { VirtualFilesystem } from "./persistence/vfs";
import type { SyncDeps } from "./sync-operations/sync-actions";
import { rateLimit } from "./utils/rate-limit";
import type { NetworkConnectionStatus } from "./types/network-connection-status";
import { DocumentSyncStatus } from "./types/document-sync-status";
@ -25,7 +25,6 @@ import type { MaybeOutdatedClientCursors } from "./types/maybe-outdated-client-c
import { FileChangeNotifier } from "./sync-operations/file-change-notifier";
import { FixedSizeDocumentCache } from "./utils/data-structures/fix-sized-cache";
import { setUpTelemetry } from "./utils/set-up-telemetry";
import { DIFF_CACHE_SIZE_MB } from "./consts";
import { ServerConfig } from "./services/server-config";
import type { EventListeners } from "./utils/data-structures/event-listeners";
@ -41,7 +40,7 @@ export class SyncClient {
public readonly logger: Logger,
private readonly history: SyncHistory,
private readonly settings: Settings,
private readonly database: Database,
private readonly vfs: VirtualFilesystem,
private readonly syncer: Syncer,
private readonly webSocketManager: WebSocketManager,
private readonly fetchController: FetchController,
@ -59,7 +58,7 @@ export class SyncClient {
) { }
public get documentCount(): number {
return this.database.length;
return this.vfs.length;
}
public get isWebSocketConnected(): boolean {
@ -148,7 +147,7 @@ export class SyncClient {
() => settings.getSettings().minimumSaveIntervalMs
);
const database = new Database(
const vfs = new VirtualFilesystem(
logger,
state.database,
async (data): Promise<void> => {
@ -174,25 +173,26 @@ export class SyncClient {
const fileOperations = new FileOperations(
logger,
database,
vfs,
fs,
serverConfig,
nativeLineEndings
);
const contentCache = new FixedSizeDocumentCache(
1024 * 1024 * DIFF_CACHE_SIZE_MB
1024 * 1024 * settings.getSettings().diffCacheSizeMB
);
const unrestrictedSyncer = new UnrestrictedSyncer(
const syncDeps: SyncDeps = {
logger,
database,
settings,
vfs,
syncService,
fileOperations,
operations: fileOperations,
history,
contentCache,
serverConfig
);
serverConfig,
settings
};
const webSocketManager = new WebSocketManager(
logger,
@ -203,17 +203,17 @@ export class SyncClient {
const syncer = new Syncer(
deviceId,
logger,
database,
vfs,
settings,
webSocketManager,
fileOperations,
unrestrictedSyncer
syncDeps
);
const fileChangeNotifier = new FileChangeNotifier();
const cursorTracker = new CursorTracker(
logger,
database,
vfs,
webSocketManager,
fileOperations,
fileChangeNotifier
@ -222,7 +222,7 @@ export class SyncClient {
logger,
history,
settings,
database,
vfs,
syncer,
webSocketManager,
fetchController,
@ -333,8 +333,8 @@ export class SyncClient {
// clear all local state
this.logger.info("Resetting SyncClient's local state");
this.database.reset();
await this.database.save(); // ensure the new database reads as empty
this.vfs.reset();
await this.vfs.save(); // ensure the new database reads as empty
this.resetInMemoryState();
this.hasFinishedOfflineSync = false;
this.serverConfig.reset();
@ -433,14 +433,34 @@ export class SyncClient {
while (true) {
iteration++;
this.logger.info(`waitUntilFinished: iteration ${iteration}`);
await this.webSocketManager.waitUntilFinished();
await this.syncer.waitUntilFinished();
await this.webSocketManager.waitUntilFinished();
// Check if anything new arrived while we were waiting
if (!this.webSocketManager.hasOutstandingWork()) {
if (
!this.webSocketManager.hasOutstandingWork() &&
!this.syncer.hasOutstandingWork()
) {
break;
}
}
await this.database.save(); // flush all changes to disk
// Run a final filesystem scan to catch any operations that were
// silently dropped (e.g., due to mutable document references
// pointing to a moved path after concurrent renames).
await this.syncer.runFinalConsistencyCheck();
// Wait for any work produced by the final scan
while (true) {
await this.syncer.waitUntilFinished();
await this.webSocketManager.waitUntilFinished();
if (
!this.webSocketManager.hasOutstandingWork() &&
!this.syncer.hasOutstandingWork()
) {
break;
}
}
await this.vfs.save(); // flush all changes to disk
}
/**
@ -467,6 +487,8 @@ export class SyncClient {
this.resetInMemoryState();
// Clean up event listeners to prevent memory leaks
this.syncer.destroy();
this.cursorTracker.destroy();
this.eventUnsubscribers.forEach((unsubscribe) => {
unsubscribe();
});
@ -491,10 +513,16 @@ export class SyncClient {
}
/**
* Hard pause: aborts all in-flight HTTP operations via FetchController reset.
* Used when the SyncClient is being destroyed or fully reset (connection
* settings changed). This is the nuclear option — every outstanding fetch
* is rejected with SyncResetError so the queue drains immediately.
* Pause syncing: aborts all in-flight HTTP operations via FetchController
* reset, stops the WebSocket, and waits for the sync queue to drain.
*
* Used by both destroy/reset (connection settings changed) and when the
* user toggles sync off. In both cases, `fetchController.startReset()` is
* needed because the settings-change listener may have already set
* `canFetch = false`, which would cause controlled fetches to block
* indefinitely. The WebSocket close handler triggers `syncer.reset()`
* automatically via the `onWebSocketStatusChanged` listener, so an
* explicit `syncer.reset()` call is not needed here.
*/
private async pause(): Promise<void> {
this.hasFinishedOfflineSync = false;
@ -504,29 +532,15 @@ export class SyncClient {
await this.waitUntilFinished();
} catch (e) {
// SyncResetError is expected here — we just called startReset()
// which rejects in-flight fetches. Only re-throw non-reset errors
// (after ensuring the FetchController is left in a usable state).
this.fetchController.finishReset();
// which rejects in-flight fetches. Only re-throw non-reset errors.
if (!(e instanceof SyncResetError)) {
throw e;
}
} finally {
this.fetchController.finishReset();
}
}
/**
* Soft pause: stops the WebSocket and clears the sync queue, but lets
* in-flight HTTP operations complete naturally. Used when the user toggles
* sync off — we don't want to abort creates/updates that are mid-flight
* because they'd just be re-queued on re-enable, potentially leading to
* an infinite retry loop with flaky connections.
*/
private async softPause(): Promise<void> {
this.hasFinishedOfflineSync = false;
await this.webSocketManager.stop();
this.syncer.reset();
await this.waitUntilFinished();
}
private resetInMemoryState(): void {
this.history.reset();
this.contentCache.reset();
@ -553,7 +567,7 @@ export class SyncClient {
if (newSettings.isSyncEnabled) {
await this.startSyncing();
} else {
await this.softPause();
await this.pause();
}
}

151
frontend/sync-client/src/sync-operations/cursor-tracker.ts
View file

@ -1,5 +1,6 @@
import type { FileOperations } from "../file-operations/file-operations";
import type { Database, RelativePath } from "../persistence/database";
import type { RelativePath } from "../persistence/database";
import type { VirtualFilesystem } from "../persistence/vfs";
import type { ClientCursors } from "../services/types/ClientCursors";
import type { CursorSpan } from "../services/types/CursorSpan";
import type { DocumentWithCursors } from "../services/types/DocumentWithCursors";
@ -24,6 +25,7 @@ export class CursorTracker {
>();
private readonly updateLock: Lock;
private readonly eventUnsubscribers: (() => void)[] = [];
private knownRemoteCursors: (ClientCursors & {
upToDateness: DocumentUpToDateness;
@ -35,61 +37,68 @@ export class CursorTracker {
public constructor(
private readonly logger: Logger,
private readonly database: Database,
private readonly vfs: VirtualFilesystem,
private readonly webSocketManager: WebSocketManager,
private readonly fileOperations: FileOperations,
private readonly fileChangeNotifier: FileChangeNotifier
) {
this.updateLock = new Lock(CursorTracker.name, logger);
this.webSocketManager.onRemoteCursorsUpdateReceived.add(
async (clientCursors) => {
await this.updateLock.withLock(async () => {
// The latest message will contain all active clients, so we can delete the ones
// from the local list which are no longer active.
const allIds = new Set(
clientCursors.map((c) => c.deviceId)
);
const updatedKnownRemoteCursors =
this.knownRemoteCursors.filter((c) =>
allIds.has(c.deviceId)
this.eventUnsubscribers.push(
this.webSocketManager.onRemoteCursorsUpdateReceived.add(
async (clientCursors) => {
await this.updateLock.withLock(async () => {
// The latest message will contain all active clients, so we can delete the ones
// from the local list which are no longer active.
const allIds = new Set(
clientCursors.map((c) => c.deviceId)
);
const updatedKnownRemoteCursors =
this.knownRemoteCursors.filter((c) =>
allIds.has(c.deviceId)
);
for (const cursor of clientCursors.filter((client) =>
client.documentsWithCursors.every(
(doc) => doc.vault_update_id != null
)
)) {
updatedKnownRemoteCursors.push({
...cursor,
upToDateness:
await this.getDocumentsUpToDateness(cursor)
});
}
for (const cursor of clientCursors.filter((client) =>
client.documentsWithCursors.every(
(doc) => doc.vault_update_id != null
)
)) {
updatedKnownRemoteCursors.push({
...cursor,
upToDateness:
await this.getDocumentsUpToDateness(cursor)
});
}
this.knownRemoteCursors = updatedKnownRemoteCursors;
});
this.knownRemoteCursors = updatedKnownRemoteCursors;
});
this.onRemoteCursorsUpdated.trigger(
this.getRelevantAndPruneKnownClientCursors()
);
}
this.onRemoteCursorsUpdated.trigger(
this.getRelevantAndPruneKnownClientCursors()
);
}
)
);
this.fileChangeNotifier.onFileChanged.add(async (relativePath) =>
this.updateLock.withLock(async () => {
for (const clientCursor of this.knownRemoteCursors) {
if (
clientCursor.documentsWithCursors.some(
(document) =>
document.relative_path === relativePath
)
) {
clientCursor.upToDateness =
await this.getDocumentsUpToDateness(clientCursor);
}
}
})
this.eventUnsubscribers.push(
this.fileChangeNotifier.onFileChanged.add(
async (relativePath) =>
this.updateLock.withLock(async () => {
for (const clientCursor of this.knownRemoteCursors) {
if (
clientCursor.documentsWithCursors.some(
(document) =>
document.relative_path === relativePath
)
) {
clientCursor.upToDateness =
await this.getDocumentsUpToDateness(
clientCursor
);
}
}
})
)
);
}
@ -104,21 +113,20 @@ export class CursorTracker {
for (const [relativePath, cursors] of Object.entries(
documentToCursors
)) {
const record =
this.database.getLatestDocumentByRelativePath(relativePath);
const doc = this.vfs.getByPath(relativePath);
if (!record) {
if (!doc) {
continue; // Let's wait for the file to be created before sending cursors
}
if (!record.metadata) {
continue; // this is a new document, no need to sync the cursors
if (doc.state !== "tracked") {
continue; // this is a pending document, no need to sync the cursors
}
documentsWithCursors.push({
relative_path: relativePath,
document_id: record.metadata.documentId,
vault_update_id: record.metadata.parentVersionId,
document_id: doc.documentId,
vault_update_id: doc.serverVersion,
cursors: cursors.map(({ start, end }) => ({
start: Math.min(start, end),
end: Math.max(start, end)
@ -139,10 +147,10 @@ export class CursorTracker {
const readContent = await this.fileOperations.read(
doc.relative_path
);
const record = this.database.getLatestDocumentByRelativePath(
doc.relative_path
);
if (record?.metadata?.hash !== hash(readContent)) {
const vfsDoc = this.vfs.getByPath(doc.relative_path);
const storedHash =
vfsDoc?.state === "tracked" ? vfsDoc.localHash : undefined;
if (storedHash !== hash(readContent)) {
doc.vault_update_id = null;
}
}
@ -166,6 +174,12 @@ export class CursorTracker {
this.updateLock.reset();
}
public destroy(): void {
for (const unsubscribe of this.eventUnsubscribers) {
unsubscribe();
}
}
private getRelevantAndPruneKnownClientCursors(): MaybeOutdatedClientCursors[] {
const result: MaybeOutdatedClientCursors[] = [];
const included = new Set<string>();
@ -227,24 +241,19 @@ export class CursorTracker {
private async getDocumentUpToDateness(
document: DocumentWithCursors
): Promise<DocumentUpToDateness> {
const record = this.database.getLatestDocumentByRelativePath(
document.relative_path
);
const vfsDoc = this.vfs.getByPath(document.relative_path);
if (!record) {
if (!vfsDoc) {
// the document of the cursor must be from the future
return DocumentUpToDateness.Later;
}
if (
(record.metadata?.parentVersionId ?? 0) <
(document.vault_update_id ?? 0)
) {
const serverVersion =
vfsDoc.state === "tracked" ? vfsDoc.serverVersion : 0;
if (serverVersion < (document.vault_update_id ?? 0)) {
return DocumentUpToDateness.Later;
} else if (
(document.vault_update_id ?? 0) <
(record.metadata?.parentVersionId ?? 0)
) {
} else if ((document.vault_update_id ?? 0) < serverVersion) {
// the document of the cursor must be from the past
return DocumentUpToDateness.Prior;
}
@ -253,9 +262,11 @@ export class CursorTracker {
document.relative_path
);
return this.database.getLatestDocumentByRelativePath(
document.relative_path
)?.metadata?.hash === hash(currentContent)
const freshDoc = this.vfs.getByPath(document.relative_path);
const storedHash =
freshDoc?.state === "tracked" ? freshDoc.localHash : undefined;
return storedHash === hash(currentContent)
? DocumentUpToDateness.UpToDate
: DocumentUpToDateness.Prior;
}

1182
frontend/sync-client/src/sync-operations/sync-actions.ts Normal file
View file

File diff suppressed because it is too large Load diff

268
frontend/sync-client/src/sync-operations/sync-event-queue.ts Normal file
View file

@ -0,0 +1,268 @@
import type { Logger } from "../tracing/logger";
import type { VirtualFilesystem } from "../persistence/vfs";
import type { SyncEvent, CoalescedAction } from "./sync-events";
import { coalesce, eventToInitialAction } from "./sync-events";
import { SyncResetError } from "../errors/sync-reset-error";
import { EventListeners } from "../utils/data-structures/event-listeners";
// A document key is either a documentId (for tracked docs) or "path:<relativePath>" (for pending docs)
type DocumentKey = string;
export class SyncEventQueue {
private readonly documentStates = new Map<DocumentKey, CoalescedAction>();
private readonly processingOrder: DocumentKey[] = [];
private currentlyProcessing: DocumentKey | null = null;
private currentOperation: Promise<void> | null = null;
private readonly idleWaiters: (() => void)[] = [];
private isResetting = false;
private isPaused = false;
public readonly onRemainingOperationsCountChanged = new EventListeners<
(count: number) => unknown
>();
// The executor is injected by the Syncer — it processes one CoalescedAction for one document
private executor:
| ((key: DocumentKey, action: CoalescedAction) => Promise<void>)
| undefined;
constructor(
private readonly logger: Logger,
private readonly vfs: VirtualFilesystem
) {}
public setExecutor(
executor: (key: DocumentKey, action: CoalescedAction) => Promise<void>
): void {
this.executor = executor;
}
// --- Event ingestion ---
public enqueue(event: SyncEvent): void {
const key = this.resolveKey(event);
const existing = this.documentStates.get(key);
if (existing === undefined || existing.action === "noop") {
this.documentStates.set(key, eventToInitialAction(event));
this.addToProcessingOrder(key);
} else {
const newAction = coalesce(existing, event);
if (newAction.action === "noop") {
this.documentStates.delete(key);
this.removeFromProcessingOrder(key);
} else {
this.documentStates.set(key, newAction);
// If the key isn't in processingOrder (was being processed), add it back
if (
!this.processingOrder.includes(key) &&
this.currentlyProcessing !== key
) {
this.addToProcessingOrder(key);
}
}
}
this.triggerCountChanged();
this.processNext();
}
// --- Key migration ---
public migrateKey(oldKey: DocumentKey, newDocumentId: string): void {
const state = this.documentStates.get(oldKey);
if (state === undefined) return;
this.documentStates.delete(oldKey);
this.removeFromProcessingOrder(oldKey);
const existingNew = this.documentStates.get(newDocumentId);
if (existingNew !== undefined) {
// Merge: coalesce the old state into the new key's state.
// This is unusual but can happen during key resolution races.
// Keep the existing state at the new key (it's more recent).
} else {
this.documentStates.set(newDocumentId, state);
this.addToProcessingOrder(newDocumentId);
}
}
// --- Processing ---
public hasOutstandingWork(): boolean {
return this.documentStates.size > 0 || this.currentOperation !== null;
}
public hasPendingEventsFor(key: string): boolean {
return (
this.documentStates.has(key) ||
this.documentStates.has("path:" + key) ||
this.currentlyProcessing === key ||
this.currentlyProcessing === "path:" + key
);
}
public get pendingDocumentCount(): number {
return (
this.documentStates.size +
(this.currentOperation !== null ? 1 : 0)
);
}
public async waitForIdle(): Promise<void> {
// When paused, consider the queue idle if no operation is running.
// Queued events exist but are intentionally held until resume().
if (this.currentOperation === null && (this.isPaused || this.documentStates.size === 0)) {
return;
}
return new Promise<void>((resolve) => {
this.idleWaiters.push(resolve);
});
}
// --- Reset ---
public reset(): void {
this.isResetting = true;
// Remove remote events (server will replay on reconnect).
// Preserve local events (unsynced user actions).
for (const [key, state] of this.documentStates.entries()) {
if (
state.action === "remote-update" ||
state.action === "remote-delete"
) {
this.documentStates.delete(key);
this.removeFromProcessingOrder(key);
}
}
this.idleWaiters.length = 0;
}
public clearResetting(): void {
this.isResetting = false;
}
/** Pause processing. Events can still be enqueued but won't be executed. */
public pause(): void {
this.isPaused = true;
}
/** Resume processing. Immediately processes any queued events. */
public resume(): void {
this.isPaused = false;
this.processNext();
}
public destroy(): void {
this.documentStates.clear();
this.processingOrder.length = 0;
this.currentlyProcessing = null;
this.idleWaiters.length = 0;
}
// --- Internal ---
private resolveKey(event: SyncEvent): DocumentKey {
switch (event.type) {
case "remote-update":
case "remote-delete":
return event.version.documentId;
case "local-create":
return "path:" + event.path;
case "local-update":
case "local-delete": {
const doc = this.vfs.getByPath(event.path);
if (doc !== undefined && doc.state !== "pending") {
return doc.documentId;
}
return "path:" + event.path;
}
case "local-move": {
const doc =
this.vfs.getByPath(event.toPath) ??
this.vfs.getByPath(event.fromPath);
if (doc !== undefined && doc.state !== "pending") {
return doc.documentId;
}
return "path:" + event.fromPath;
}
}
}
private processNext(): void {
if (this.currentOperation !== null) return;
if (this.isPaused) {
// Even when paused, resolve idle waiters since no operation is
// running. This is needed because internalReconcile() pauses the
// queue then calls waitForIdle() — if a previously-started
// operation finishes while paused, idle waiters must be notified.
if (this.idleWaiters.length > 0) {
const waiters = this.idleWaiters.splice(0);
for (const w of waiters) w();
}
return;
}
while (this.processingOrder.length > 0) {
const key = this.processingOrder.shift()!;
const action = this.documentStates.get(key);
if (action === undefined || action.action === "noop") {
this.documentStates.delete(key);
continue;
}
this.currentlyProcessing = key;
this.documentStates.delete(key);
this.currentOperation = (async () => {
try {
if (this.isResetting) throw new SyncResetError();
if (this.executor === undefined) {
throw new Error("No executor set");
}
await this.executor(key, action);
} catch (e) {
if (!(e instanceof SyncResetError)) {
this.logger.info(
`Sync operation for ${key} failed, will retry: ${e}`
);
}
} finally {
this.currentlyProcessing = null;
this.currentOperation = null;
this.triggerCountChanged();
this.processNext();
}
})();
return; // processNext will be called again in finally
}
// Queue is empty, resolve idle waiters
if (this.currentOperation === null) {
const waiters = this.idleWaiters.splice(0);
for (const w of waiters) w();
this.triggerCountChanged();
}
}
private addToProcessingOrder(key: DocumentKey): void {
if (!this.processingOrder.includes(key)) {
this.processingOrder.push(key);
}
}
private removeFromProcessingOrder(key: DocumentKey): void {
const idx = this.processingOrder.indexOf(key);
if (idx !== -1) this.processingOrder.splice(idx, 1);
}
private triggerCountChanged(): void {
this.onRemainingOperationsCountChanged.trigger(
this.pendingDocumentCount
);
}
}

301
frontend/sync-client/src/sync-operations/sync-events.ts Normal file
View file

@ -0,0 +1,301 @@
import type { DocumentVersionWithoutContent } from "../services/types/DocumentVersionWithoutContent";
// ---------------------------------------------------------------------------
// Raw sync events — emitted by file watchers and WebSocket handlers
// ---------------------------------------------------------------------------
export type SyncEvent =
| { type: "local-create"; path: string }
| { type: "local-update"; path: string }
| { type: "local-delete"; path: string }
| { type: "local-move"; fromPath: string; toPath: string }
| { type: "remote-update"; version: DocumentVersionWithoutContent }
| { type: "remote-delete"; version: DocumentVersionWithoutContent };
// ---------------------------------------------------------------------------
// Coalesced actions — the result of merging multiple events on the same key
// ---------------------------------------------------------------------------
export type CoalescedAction =
| { action: "create"; path: string }
| { action: "update"; path: string }
| { action: "delete"; path: string }
| { action: "move"; fromPath: string; toPath: string }
| { action: "move-and-update"; fromPath: string; toPath: string }
| { action: "remote-update"; version: DocumentVersionWithoutContent }
| { action: "remote-delete"; version: DocumentVersionWithoutContent }
| { action: "noop" };
/**
* Convert a single SyncEvent to its initial CoalescedAction.
*/
export function eventToInitialAction(event: SyncEvent): CoalescedAction {
switch (event.type) {
case "local-create":
return { action: "create", path: event.path };
case "local-update":
return { action: "update", path: event.path };
case "local-delete":
return { action: "delete", path: event.path };
case "local-move":
return {
action: "move",
fromPath: event.fromPath,
toPath: event.toPath
};
case "remote-update":
return { action: "remote-update", version: event.version };
case "remote-delete":
return { action: "remote-delete", version: event.version };
}
}
/**
* Coalesce a new SyncEvent into an existing CoalescedAction.
*
* This implements the full transition table for combining sequential events
* that target the same logical document. The goal is to reduce multiple
* events into a single action that captures the net effect.
*
* Transition table (current action x new event -> result):
*
* | Current \ New Event | local-create | local-update | local-delete | local-move(to) | remote-update | remote-delete |
* |---------------------|-------------|-------------|-------------|----------------|---------------|---------------|
* | create | create | create | noop | create(to) | create | noop |
* | update | update | update | delete | move-and-update| remote-update | delete |
* | delete | create | update | delete | move | remote-update | delete |
* | move | move | move-and-upd| delete | move(orig,to) | move | delete |
* | move-and-update | move-and-upd| move-and-upd| delete | m-a-u(orig,to) | move-and-upd | delete |
* | remote-update | create | remote-upd | remote-del | remote-upd | remote-upd | remote-del |
* | remote-delete | create | remote-del | remote-del | remote-del | remote-upd | remote-del |
* | noop | create | update | delete | move | remote-update | remote-delete |
*/
export function coalesce(
current: CoalescedAction,
event: SyncEvent
): CoalescedAction {
switch (current.action) {
case "create":
return coalesceFromCreate(current, event);
case "update":
return coalesceFromUpdate(current, event);
case "delete":
return coalesceFromDelete(event);
case "move":
return coalesceFromMove(current, event);
case "move-and-update":
return coalesceFromMoveAndUpdate(current, event);
case "remote-update":
return coalesceFromRemoteUpdate(current, event);
case "remote-delete":
return coalesceFromRemoteDelete(current, event);
case "noop":
return eventToInitialAction(event);
}
}
function coalesceFromCreate(
current: { action: "create"; path: string },
event: SyncEvent
): CoalescedAction {
switch (event.type) {
case "local-create":
// create + create = still create (idempotent)
return current;
case "local-update":
// create + update = still create (content will be read at sync time)
return current;
case "local-delete":
// create + delete = noop (file never reached server)
return { action: "noop" };
case "local-move":
// create + move = create at new path
return { action: "create", path: event.toPath };
case "remote-update":
// create + remote-update = still create (local create takes precedence)
return current;
case "remote-delete":
// create + remote-delete = noop
return { action: "noop" };
}
}
function coalesceFromUpdate(
current: { action: "update"; path: string },
event: SyncEvent
): CoalescedAction {
switch (event.type) {
case "local-create":
// update + create = update (file was already tracked)
return current;
case "local-update":
// update + update = update
return current;
case "local-delete":
// update + delete = delete
return { action: "delete", path: current.path };
case "local-move":
// update + move = move-and-update
return {
action: "move-and-update",
fromPath: event.fromPath,
toPath: event.toPath
};
case "remote-update":
// update + remote-update = remote-update (forces server fetch
// so remote changes are applied even when there are no local edits)
return { action: "remote-update", version: event.version };
case "remote-delete":
// update + remote-delete = delete
return { action: "delete", path: current.path };
}
}
function coalesceFromDelete(event: SyncEvent): CoalescedAction {
switch (event.type) {
case "local-create":
// delete + create = create (file re-created)
return { action: "create", path: event.path };
case "local-update":
// delete + update = update (file re-appeared with changes)
return { action: "update", path: event.path };
case "local-delete":
// delete + delete = delete (idempotent)
return { action: "delete", path: event.path };
case "local-move":
// delete + move = move (the original delete is superseded)
return {
action: "move",
fromPath: event.fromPath,
toPath: event.toPath
};
case "remote-update":
// delete + remote-update = remote-update (server has new version)
return { action: "remote-update", version: event.version };
case "remote-delete":
// delete + remote-delete = delete
return { action: "delete", path: event.version.relativePath };
}
}
function coalesceFromMove(
current: { action: "move"; fromPath: string; toPath: string },
event: SyncEvent
): CoalescedAction {
switch (event.type) {
case "local-create":
// move + create = move (file already at destination)
return current;
case "local-update":
// move + update = move-and-update
return {
action: "move-and-update",
fromPath: current.fromPath,
toPath: current.toPath
};
case "local-delete":
// move + delete = delete (from original path)
return { action: "delete", path: current.fromPath };
case "local-move":
// move(A->B) + move(B->C) = move(A->C)
return {
action: "move",
fromPath: current.fromPath,
toPath: event.toPath
};
case "remote-update":
// move + remote-update = move (local move takes precedence)
return current;
case "remote-delete":
// move + remote-delete = delete
return { action: "delete", path: current.fromPath };
}
}
function coalesceFromMoveAndUpdate(
current: { action: "move-and-update"; fromPath: string; toPath: string },
event: SyncEvent
): CoalescedAction {
switch (event.type) {
case "local-create":
// move-and-update + create = move-and-update
return current;
case "local-update":
// move-and-update + update = move-and-update
return current;
case "local-delete":
// move-and-update + delete = delete (from original path)
return { action: "delete", path: current.fromPath };
case "local-move":
// move-and-update(A->B) + move(B->C) = move-and-update(A->C)
return {
action: "move-and-update",
fromPath: current.fromPath,
toPath: event.toPath
};
case "remote-update":
// move-and-update + remote-update = move-and-update
return current;
case "remote-delete":
// move-and-update + remote-delete = delete
return { action: "delete", path: current.fromPath };
}
}
function coalesceFromRemoteUpdate(
current: { action: "remote-update"; version: DocumentVersionWithoutContent },
event: SyncEvent
): CoalescedAction {
switch (event.type) {
case "local-create":
// remote-update + create = create (local create wins — will be
// sent to server, which will merge or deconflict)
return { action: "create", path: event.path };
case "local-update":
// remote-update + update = remote-update (will merge on sync)
return current;
case "local-delete":
// remote-update + local-delete = remote-delete
return { action: "remote-delete", version: current.version };
case "local-move":
// remote-update + move = remote-update (path change handled separately)
return current;
case "remote-update":
// remote-update + remote-update = remote-update (latest version)
return { action: "remote-update", version: event.version };
case "remote-delete":
// remote-update + remote-delete = remote-delete
return { action: "remote-delete", version: event.version };
}
}
function coalesceFromRemoteDelete(
current: {
action: "remote-delete";
version: DocumentVersionWithoutContent;
},
event: SyncEvent
): CoalescedAction {
switch (event.type) {
case "local-create":
// remote-delete + create = create (local create takes precedence —
// the user explicitly created a file; the remote delete was for the
// OLD document, the create is for a NEW one)
return { action: "create", path: event.path };
case "local-update":
// remote-delete + update = remote-delete
return current;
case "local-delete":
// remote-delete + local-delete = remote-delete
return current;
case "local-move":
// remote-delete + move = remote-delete
return current;
case "remote-update":
// remote-delete + remote-update = remote-update (server changed its mind)
return { action: "remote-update", version: event.version };
case "remote-delete":
// remote-delete + remote-delete = remote-delete (latest)
return { action: "remote-delete", version: event.version };
}
}

949
frontend/sync-client/src/sync-operations/syncer.ts
View file

File diff suppressed because it is too large Load diff

826
frontend/sync-client/src/sync-operations/unrestricted-syncer.ts
View file

@ -1,826 +0,0 @@
import type {
Database,
DocumentRecord,
RelativePath
} from "../persistence/database";
import { diff } from "reconcile-text";
import type { SyncService } from "../services/sync-service";
import type { Logger } from "../tracing/logger";
import type {
CommonHistoryEntry,
SyncCreateDetails,
SyncDeleteDetails,
SyncDetails,
SyncHistory,
SyncMovedDetails,
SyncUpdateDetails
} from "../tracing/sync-history";
import { SyncStatus, SyncType } from "../tracing/sync-history";
import { EMPTY_HASH, hash } from "../utils/hash";
import { base64ToBytes } from "byte-base64";
import type { Settings } from "../persistence/settings";
import type { FileOperations } from "../file-operations/file-operations";
import { FileNotFoundError } from "../errors/file-not-found-error";
import { SyncResetError } from "../errors/sync-reset-error";
import { globsToRegexes } from "../utils/globs-to-regexes";
import type { DocumentVersion } from "../services/types/DocumentVersion";
import type { DocumentUpdateResponse } from "../services/types/DocumentUpdateResponse";
import type { DocumentVersionWithoutContent } from "../services/types/DocumentVersionWithoutContent";
import type { FixedSizeDocumentCache } from "../utils/data-structures/fix-sized-cache";
import { isFileTypeMergable } from "../utils/is-file-type-mergable";
import { isBinary } from "../utils/is-binary";
import type { ServerConfig } from "../services/server-config";
export class UnrestrictedSyncer {
private ignorePatterns: RegExp[];
public constructor(
private readonly logger: Logger,
private readonly database: Database,
private readonly settings: Settings,
private readonly syncService: SyncService,
private readonly operations: FileOperations,
private readonly history: SyncHistory,
private readonly contentCache: FixedSizeDocumentCache,
private readonly serverConfig: ServerConfig
) {
this.ignorePatterns = globsToRegexes(
this.settings.getSettings().ignorePatterns,
this.logger
);
this.settings.onSettingsChanged.add((newSettings) => {
this.ignorePatterns = globsToRegexes(
newSettings.ignorePatterns,
this.logger
);
});
}
public async resolveIdempotencyKeys(): Promise<void> {
const pendingDocs = this.database.pendingDocuments;
if (pendingDocs.length === 0) {
return;
}
const keys = pendingDocs
.map((d) => d.idempotencyKey)
// eslint-disable-next-line no-restricted-syntax -- Type narrowing, not removing a specific item
.filter((k): k is string => k !== undefined);
if (keys.length === 0) {
return;
}
this.logger.debug(
`Resolving ${keys.length} pending idempotency keys`
);
const resolved =
await this.syncService.resolveIdempotencyKeys(keys);
for (const doc of pendingDocs) {
if (
doc.idempotencyKey !== undefined &&
resolved.has(doc.idempotencyKey)
) {
// Check if document was removed by a concurrent operation
// (e.g., a delete) between the snapshot and now
if (!this.database.containsDocument(doc)) {
this.logger.info(
`Pending doc at ${doc.relativePath} was removed during key resolution, skipping`
);
continue;
}
const documentId = resolved.get(doc.idempotencyKey)!; // eslint-disable-line @typescript-eslint/no-non-null-assertion
// Skip if this documentId is already assigned to another document
const existing =
this.database.getDocumentByDocumentId(documentId);
if (existing !== undefined) {
this.logger.debug(
`Document ${documentId} already exists at ${existing.relativePath}, removing stale pending doc at ${doc.relativePath}`
);
this.database.removeDocument(doc);
continue;
}
this.logger.info(
`Resolved idempotency key ${doc.idempotencyKey} to document ${documentId} for ${doc.relativePath}`
);
this.database.updateDocumentMetadata(
{
documentId,
parentVersionId: 0,
hash: "",
remoteRelativePath: doc.relativePath
},
doc
);
}
}
}
public async unrestrictedSyncLocallyCreatedOrUpdatedFile({
oldPath,
// We use the same code path for both local and remote updates. We need to force the update
// if there are no local changes but we know that the remote version is newer.
force = false,
document
}: {
oldPath?: RelativePath;
force?: boolean;
document: DocumentRecord;
}): Promise<void> {
const updateDetails:
| SyncCreateDetails
| SyncUpdateDetails
| SyncMovedDetails =
document.metadata === undefined
? {
type: SyncType.CREATE,
relativePath: document.relativePath
}
: oldPath !== undefined
? {
type: SyncType.MOVE,
relativePath: document.relativePath,
movedFrom: oldPath
}
: {
type: SyncType.UPDATE,
relativePath: document.relativePath
};
await this.executeSync(updateDetails, async () => {
const originalRelativePath = document.relativePath;
if (document.isDeleted) {
this.logger.debug(
`Document ${document.relativePath} has been already deleted, no need to update it`
);
return;
}
const contentBytes = await this.operations.read(
document.relativePath
); // this can throw FileNotFoundError
const contentHash = hash(contentBytes);
let response: DocumentVersion | DocumentUpdateResponse | undefined =
undefined;
if (
document.metadata === undefined ||
document.metadata.parentVersionId === 0
) {
// parentVersionId === 0 occurs when resolveIdempotencyKeys
// assigned a documentId but hasn't synced yet. Treat as a
// create — the server will recognise the idempotency key
// and return the existing document.
response = await this.syncService.create({
relativePath: originalRelativePath,
contentBytes,
idempotencyKey: document.idempotencyKey
});
await this.handleMaybeMergingResponse({
document,
response,
contentHash,
originalRelativePath,
originalContentBytes: contentBytes,
isCreate: true
});
} else {
const areThereLocalChanges =
document.metadata.hash !== contentHash ||
oldPath !== undefined;
if (areThereLocalChanges) {
const isText =
!isBinary(contentBytes) &&
isFileTypeMergable(
document.relativePath,
(await this.serverConfig.getConfig())
.mergeableFileExtensions
);
// Snapshot parentVersionId atomically with the cache
// lookup. document.metadata is a mutable shared
// reference — a concurrent operation could update
// parentVersionId between the cache lookup and the
// putText call, causing a diff/version mismatch.
const parentVersionIdForUpdate =
document.metadata.parentVersionId;
const cachedVersion = this.contentCache.get(
parentVersionIdForUpdate
);
response =
isText && cachedVersion !== undefined
? await this.syncService.putText({
documentId: document.metadata.documentId,
parentVersionId: parentVersionIdForUpdate,
relativePath: document.relativePath,
content: diff(
new TextDecoder().decode(cachedVersion),
new TextDecoder().decode(contentBytes)
)
})
: await this.syncService.putBinary({
documentId: document.metadata.documentId,
parentVersionId: parentVersionIdForUpdate,
relativePath: document.relativePath,
contentBytes
});
} else {
if (!force) {
this.logger.debug(
`File hash of ${document.relativePath} matches with last synced version and the path hasn't changed; no need to sync`
);
return;
}
// we use this code path (force == true) to sync remotely updated files which have no local changes
response = await this.syncService.get({
documentId: document.metadata.documentId
});
}
await this.handleMaybeMergingResponse({
document,
response,
contentHash,
originalRelativePath,
originalContentBytes: contentBytes
});
}
if (!("type" in response) || response.type === "MergingUpdate") {
if (!force) {
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: updateDetails,
message: `The file we updated had been updated remotely, so we downloaded the merged version`
});
return;
}
}
const actualUpdateDetails: SyncUpdateDetails | SyncMovedDetails =
oldPath !== undefined ||
response.relativePath != originalRelativePath
? {
type: SyncType.MOVE,
relativePath: response.relativePath,
movedFrom: originalRelativePath
}
: {
type: SyncType.UPDATE,
relativePath: response.relativePath
};
if (!response.isDeleted) {
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: actualUpdateDetails,
message: `Successfully downloaded remotely updated file from the server`,
author: response.userId,
timestamp: new Date(response.updatedDate)
});
} else {
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: {
type: SyncType.DELETE,
relativePath: document.relativePath
},
message:
"Successfully deleted file which had been deleted remotely",
author: response.userId,
timestamp: new Date(response.updatedDate)
});
}
});
}
public async unrestrictedSyncLocallyDeletedFile(
document: DocumentRecord
): Promise<void> {
const updateDetails: SyncDeleteDetails = {
type: SyncType.DELETE,
relativePath: document.relativePath
};
await this.executeSync(updateDetails, async () => {
if (document.metadata === undefined) {
this.logger.debug(
`Document ${document.relativePath} has never been synced, no need to delete it remotely`
);
return;
}
const response = await this.syncService.delete({
documentId: document.metadata.documentId,
relativePath: document.relativePath
});
// A concurrent merge operation may have removed this document from the
// database while we were waiting for the delete response. In that case,
// the merge already handled the state transition and we should not
// update metadata (which would fail anyway since the document is gone).
if (!this.database.containsDocument(document)) {
this.logger.debug(
`Document ${document.relativePath} was removed from database by a concurrent operation, skipping metadata update after delete`
);
this.database.addSeenUpdateId(response.vaultUpdateId);
return;
}
this.database.updateDocumentMetadata(
{
documentId: response.documentId,
parentVersionId: response.vaultUpdateId,
hash: EMPTY_HASH,
remoteRelativePath: document.relativePath
},
document
);
this.database.addSeenUpdateId(response.vaultUpdateId);
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: updateDetails,
message: `Successfully deleted locally deleted file on the server`,
author: response.userId
});
});
}
public async unrestrictedSyncRemotelyUpdatedFile(
remoteVersion: DocumentVersionWithoutContent,
document?: DocumentRecord
): Promise<void> {
const updateDetails: SyncCreateDetails = {
type: SyncType.CREATE,
relativePath: remoteVersion.relativePath
};
await this.executeSync(updateDetails, async () => {
if (document?.metadata !== undefined) {
// If the file exists locally, let's pretend the user has updated it
// and deal with remote update/deletion within `unrestrictedSyncLocallyUpdatedFile`
if (
document.metadata.parentVersionId >=
remoteVersion.vaultUpdateId
) {
this.logger.debug(
`Document ${document.relativePath} is already at least as up-to-date as the fetched version`
);
return;
}
return this.unrestrictedSyncLocallyCreatedOrUpdatedFile({
document,
force: true
});
} else if (remoteVersion.isDeleted) {
// Either the document hasn't made it to us before and therefore we don't need to delete it,
// or we already have it, in which case the preceeding if would've dealt with it
this.logger.debug(
`Document ${remoteVersion.relativePath} has been deleted remotely, no need to sync`
);
return;
}
// Don't download oversized files
const historyEntryForSkippedOversizedFile =
this.getHistoryEntryForSkippedOversizedFile(
remoteVersion.contentSize,
remoteVersion.relativePath
);
if (historyEntryForSkippedOversizedFile !== undefined) {
this.history.addHistoryEntry(
historyEntryForSkippedOversizedFile
);
return;
}
const contentBytes =
await this.syncService.getDocumentVersionContent({
documentId: remoteVersion.documentId,
vaultUpdateId: remoteVersion.vaultUpdateId
});
// We're trying to create an entirely new document that didn't exist locally
document = this.database.getDocumentByDocumentId(
remoteVersion.documentId
);
// It can happen that a concurrent sync operation has already created the document, so we can bail here
if (document !== undefined) {
this.logger.debug(
`Document ${remoteVersion.relativePath} has already been created locally, no need to create it again`
);
return;
}
await this.operations.ensureClearPath(remoteVersion.relativePath);
this.database.updateDocumentMetadata(
{
documentId: remoteVersion.documentId,
parentVersionId: remoteVersion.vaultUpdateId,
hash: hash(contentBytes),
remoteRelativePath: remoteVersion.relativePath
},
this.database.createNewPendingDocument(
remoteVersion.relativePath
)
);
await this.operations.create(
remoteVersion.relativePath,
contentBytes
);
await this.updateCache(
remoteVersion.vaultUpdateId,
contentBytes,
remoteVersion.relativePath
);
this.history.addHistoryEntry({
status: SyncStatus.SUCCESS,
details: updateDetails,
message: `Successfully downloaded remote file which hadn't existed locally`,
author: remoteVersion.userId,
timestamp: new Date(remoteVersion.updatedDate)
});
});
}
private async executeSync<T>(
details: SyncDetails,
fn: () => Promise<T>
): Promise<T | undefined> {
if (!this.settings.getSettings().isSyncEnabled) {
this.logger.info(
`Skipping sync operation for file '${details.relativePath}' because sync is disabled`
);
return;
}
for (const pattern of this.ignorePatterns) {
if (pattern.test(details.relativePath)) {
this.logger.debug(
`File '${details.relativePath}' is ignored by the ignore pattern: ${pattern}`
);
return; // bail without SKIPPED status because we were told to ignore this file and we shouldn't clutter up the history
}
}
try {
// Only check the size of files which already exist locally.
if (await this.operations.exists(details.relativePath)) {
const sizeInBytes = await this.operations.getFileSize(
details.relativePath
);
const historyEntryForSkippedOversizedFile =
this.getHistoryEntryForSkippedOversizedFile(
sizeInBytes,
details.relativePath
);
if (historyEntryForSkippedOversizedFile !== undefined) {
this.history.addHistoryEntry(
historyEntryForSkippedOversizedFile
);
return;
}
}
return await fn();
} catch (e) {
if (e instanceof FileNotFoundError) {
// A subsequent sync operation must have been creating to deal with this
this.logger.info(
`Skiping file '${details.relativePath}' because it no longer exists when trying to ${details.type.toLocaleLowerCase()} it`
);
return;
}
if (e instanceof SyncResetError) {
this.logger.info(
`Interrupting sync operation because of a reset`
);
return;
} else {
this.history.addHistoryEntry({
status: SyncStatus.ERROR,
details,
message: `Failed to sync file '${details.relativePath}' because of ${e} when trying to ${details.type.toLocaleLowerCase()} it`
});
throw e;
}
}
}
private async handleMaybeMergingResponse({
document,
response,
contentHash,
originalRelativePath,
originalContentBytes,
isCreate
}: {
document: DocumentRecord;
response: DocumentVersion | DocumentUpdateResponse;
contentHash: string;
originalRelativePath: string;
originalContentBytes: Uint8Array;
isCreate?: boolean;
}): Promise<void> {
// `document` is mutable and reflects the latest state in the local database
if (document.isDeleted) {
this.logger.info(
`Document ${document.relativePath} has been deleted before we could finish updating it`
);
// Assign metadata so the pending delete can inform the server
if (document.metadata === undefined) {
const existingWithSameId =
this.database.getDocumentByDocumentId(
response.documentId
);
if (
existingWithSameId !== undefined &&
existingWithSameId !== document
) {
// Another doc already has this documentId — the server
// knows about it. Just remove this stale pending doc.
this.database.removeDocument(document);
} else {
this.database.updateDocumentMetadata(
{
documentId: response.documentId,
parentVersionId: response.vaultUpdateId,
hash: contentHash,
remoteRelativePath: response.relativePath
},
document
);
}
}
this.database.addSeenUpdateId(response.vaultUpdateId);
return;
}
if (
(document.metadata?.parentVersionId ?? 0) > response.vaultUpdateId
) {
this.logger.debug(
`Document ${document.relativePath} is already more up to date than the fetched version`
);
this.database.addSeenUpdateId(response.vaultUpdateId); // in case the previous `vaultUpdateId` update hasn't made it through
return;
}
if (response.isDeleted) {
return this.applyRemoteDeleteLocally(document, response);
}
let actualPath = document.relativePath;
let existingContentBytes: Uint8Array | undefined;
if (isCreate) {
// We have a file locally that got moved by another client to the same path as the one we're trying to create.
// The server returns a merging update for the document ID that already exists locally (but at another path).
// We have to merge these two documents by extending the provenance of the existing document and deleting
// the old document that the new document already contains the content for.
const existingDocument = this.database.getDocumentByDocumentId(
response.documentId
);
// If existingDocument === document, then a previous sync operation already
// assigned this documentId to our document. We don't need to merge - just
// continue to update the metadata below.
if (existingDocument !== undefined && existingDocument !== document) {
this.logger.info(
`Merging existing document ${existingDocument.relativePath} into ${document.relativePath
} after concurrent move & creation`
);
if (!existingDocument.isDeleted) {
this.database.delete(existingDocument.relativePath); // make sure syncLocallyDeletedFile doesn't actually schedule deleting the new file
try {
existingContentBytes = await this.operations.read(
existingDocument.relativePath
);
} catch (e) {
if (e instanceof FileNotFoundError) {
return;
}
throw e;
}
this.database.removeDocument(existingDocument);
await this.operations.delete(existingDocument.relativePath);
} else {
this.database.removeDocument(existingDocument);
}
}
}
// A document's documentId should never change once assigned. If the response has a
// different documentId than what the document already has, it means the file was
// renamed during the sync operation and the response is for a different document.
// We should bail out and let subsequent sync operations fix the state.
if (
document.metadata?.documentId !== undefined &&
document.metadata.documentId !== response.documentId
) {
this.logger.info(
`Document ${document.relativePath} already has documentId ${document.metadata.documentId}, ` +
`but response has documentId ${response.documentId}. Ignoring response to prevent documentId corruption.`
);
this.database.addSeenUpdateId(response.vaultUpdateId);
return;
}
// this can't happen on the creation path as we can only get a merging response if a document already exists remotely on the same path
if (response.relativePath != originalRelativePath) {
actualPath = response.relativePath;
// Make sure to update the remote relative path to avoid uploading
// the file as a result of this filesystem event.
if (document.metadata !== undefined) {
document.metadata.remoteRelativePath = response.relativePath;
}
await this.operations.move(
document.relativePath,
response.relativePath
); // this can throw FileNotFoundError
}
if (!("type" in response) || response.type === "MergingUpdate") {
const responseBytes = base64ToBytes(response.contentBase64);
// Write file BEFORE updating metadata so that if the write fails,
// metadata doesn't point to a version whose content was never written.
await this.operations.write(
actualPath,
originalContentBytes,
responseBytes
);
if (existingContentBytes !== undefined) {
// the merge case is only always for text files, so don't mind that we have to provide a byte array here
await this.operations.write(
actualPath,
new Uint8Array(0),
existingContentBytes
);
}
// Re-read and re-hash after write because the 3-way merge in
// operations.write() may produce content different from responseBytes.
const actualContent = await this.operations.read(actualPath);
const actualHash = hash(actualContent);
// The document may have been removed by a concurrent operation
// (e.g., a delete) during the awaited file write/read above.
// The file is safely on disk; recovery will re-detect it.
if (!this.database.containsDocument(document)) {
this.logger.info(
`Document ${document.relativePath} was removed during sync, skipping metadata update`
);
return;
}
this.database.updateDocumentMetadata(
{
documentId: response.documentId,
parentVersionId: response.vaultUpdateId,
hash: actualHash,
remoteRelativePath: response.relativePath
},
document
);
// Cache the SERVER's content (responseBytes), not the local
// content (actualContent). The cache is used to compute diffs
// for subsequent updates: diff(cached, newFileContent). The
// server applies this diff against its content at
// parentVersionId, which is responseBytes. Using actualContent
// would produce diffs that don't match the server's state.
await this.updateCache(
response.vaultUpdateId,
responseBytes,
actualPath
);
} else {
// FastForwardUpdate — the server accepted our content as-is,
// UNLESS this was an idempotent create return (the server
// returned the original version, whose content may differ from
// what we sent). Detect this by comparing contentSize.
const serverContentMatchesLocal =
!("contentSize" in response) ||
response.contentSize === originalContentBytes.length;
if (serverContentMatchesLocal) {
this.database.updateDocumentMetadata(
{
documentId: response.documentId,
parentVersionId: response.vaultUpdateId,
hash: contentHash,
remoteRelativePath: response.relativePath
},
document
);
await this.updateCache(
response.vaultUpdateId,
originalContentBytes,
actualPath
);
} else {
// The server returned a stale idempotent version. Fetch
// the actual content so the cache stays consistent, then
// the hash mismatch will trigger a follow-up update sync.
const serverContent =
await this.syncService.getDocumentVersionContent({
documentId: response.documentId,
vaultUpdateId: response.vaultUpdateId
});
this.database.updateDocumentMetadata(
{
documentId: response.documentId,
parentVersionId: response.vaultUpdateId,
hash: hash(serverContent),
remoteRelativePath: response.relativePath
},
document
);
await this.updateCache(
response.vaultUpdateId,
serverContent,
actualPath
);
}
}
this.database.addSeenUpdateId(response.vaultUpdateId);
}
private getHistoryEntryForSkippedOversizedFile(
sizeInBytes: number,
relativePath: RelativePath
): CommonHistoryEntry | undefined {
const { maxFileSizeMB } = this.settings.getSettings();
const maxFileSizeBytes = maxFileSizeMB * 1024 * 1024;
if (sizeInBytes > maxFileSizeBytes) {
const sizeInMB = (sizeInBytes / 1024 / 1024).toFixed(1);
return {
status: SyncStatus.SKIPPED,
details: {
type: SyncType.SKIPPED,
relativePath
},
message: `File size of ${sizeInMB} MB exceeds the maximum file size limit of ${maxFileSizeMB
} MB`
};
}
}
private async updateCache(
updateId: number,
contentBytes: Uint8Array,
filePath: RelativePath
): Promise<void> {
if (
isFileTypeMergable(
filePath,
(await this.serverConfig.getConfig()).mergeableFileExtensions
) &&
!isBinary(contentBytes)
) {
this.contentCache.put(updateId, contentBytes);
}
}
private async applyRemoteDeleteLocally(
document: DocumentRecord,
response: DocumentVersion | DocumentUpdateResponse
): Promise<void> {
this.database.delete(document.relativePath);
this.database.updateDocumentMetadata(
{
documentId: response.documentId,
parentVersionId: response.vaultUpdateId,
hash: EMPTY_HASH,
remoteRelativePath: response.relativePath
},
document
);
await this.operations.delete(document.relativePath);
this.database.addSeenUpdateId(response.vaultUpdateId);
}
}

46
frontend/sync-client/src/utils/decode-text.ts Normal file
View file

@ -0,0 +1,46 @@
/**
* Transcode UTF-16 content to UTF-8. Detects UTF-16 LE/BE by BOM.
* Non-UTF-16 content (valid UTF-8 or binary) is returned as-is.
*
* Call this at the file-read boundary so all downstream code only
* deals with UTF-8 bytes or binary.
*/
export function normalizeToUtf8(content: Uint8Array): Uint8Array {
// UTF-16 LE BOM
if (content.length >= 2 && content[0] === 0xff && content[1] === 0xfe) {
try {
const text = new TextDecoder("utf-16le", {
fatal: true
}).decode(content);
return new TextEncoder().encode(text);
} catch {
return content;
}
}
// UTF-16 BE BOM
if (content.length >= 2 && content[0] === 0xfe && content[1] === 0xff) {
try {
const text = new TextDecoder("utf-16be", {
fatal: true
}).decode(content);
return new TextEncoder().encode(text);
} catch {
return content;
}
}
return content;
}
/**
* Decode UTF-8 bytes to a string.
* Returns `undefined` if the content is not valid UTF-8.
*/
export function decodeText(content: Uint8Array): string | undefined {
try {
return new TextDecoder("utf-8", { fatal: true }).decode(content);
} catch {
return undefined;
}
}

14
frontend/sync-client/src/utils/find-matching-file.ts
View file

@ -1,14 +0,0 @@
import type { DocumentRecord } from "../persistence/database";
import { EMPTY_HASH } from "./hash";
// TODO: make this smarter so that offline files can be renamed & edited at the same time
export function findMatchingFile(
contentHash: string,
candidates: DocumentRecord[]
): DocumentRecord | undefined {
if (contentHash === EMPTY_HASH) {
return undefined;
}
return candidates.find(({ metadata }) => metadata?.hash === contentHash);
}

38
frontend/sync-client/src/utils/hash.ts
View file

@ -1,12 +1,34 @@
// https://stackoverflow.com/questions/7616461/generate-a-hash-from-string-in-javascript
export function hash(content: Uint8Array): string {
let result = 0;
// eslint-disable-next-line @typescript-eslint/prefer-for-of
for (let i = 0; i < content.length; i++) {
result = (result << 5) - result + content[i];
result |= 0; // Convert to 32bit integer
import murmurHash3 from "murmurhash3js-revisited";
import { decodeText } from "./decode-text";
/**
* Normalize text content for consistent cross-platform hashing:
* - Apply Unicode NFC normalization (macOS uses NFD, Linux/Windows use NFC)
*
* Binary content is returned as-is.
*/
function normalizeForHashing(content: Uint8Array): Uint8Array {
const text = decodeText(content);
if (text === undefined) {
return content;
}
return Math.abs(result).toString(16).padStart(8, "0");
const normalized = text.normalize("NFC");
return new TextEncoder().encode(normalized);
}
/**
* MurmurHash3 x64 128-bit hash. Produces a 32-character hex string.
*
* The previous 32-bit hash had ~50% collision probability at ~77k files
* (birthday paradox). At 128 bits, collisions are effectively impossible.
*
* Text content is Unicode NFC-normalized for cross-platform consistency.
* Binary content is hashed as-is.
*/
export function hash(content: Uint8Array): string {
const normalized = normalizeForHashing(content);
return murmurHash3.x64.hash128(normalized);
}
export const EMPTY_HASH = hash(new Uint8Array(0));

23
frontend/sync-client/src/utils/is-binary.ts
View file

@ -1,16 +1,11 @@
// Text is unlikely to contain null bytes, so we can use that to distinguish binary files.
import { decodeText } from "./decode-text";
/**
* Determine if the given content is binary (not valid UTF-8).
*
* Content is expected to have been normalized to UTF-8 at the read
* boundary (via `normalizeToUtf8`), so this only checks UTF-8 validity.
*/
export function isBinary(content: Uint8Array): boolean {
for (const byte of content) {
if (byte === 0) {
return true;
}
}
try {
new TextDecoder("utf-8", { fatal: true }).decode(content);
} catch {
return true;
}
return false;
return decodeText(content) === undefined;
}

81
frontend/sync-client/src/utils/validate-relative-path.test.ts Normal file
View file

@ -0,0 +1,81 @@
import { describe, it } from "node:test";
import assert from "node:assert";
import { validateRelativePath } from "./validate-relative-path";
describe("validateRelativePath", () => {
it("accepts normal relative paths", () => {
assert.doesNotThrow(() => { validateRelativePath("file.md"); });
assert.doesNotThrow(() => { validateRelativePath("folder/file.md"); });
assert.doesNotThrow(
() => { validateRelativePath("deeply/nested/folder/file.md"); }
);
assert.doesNotThrow(() => { validateRelativePath("file with spaces.md"); });
assert.doesNotThrow(() => { validateRelativePath(".hidden-file"); });
assert.doesNotThrow(() => { validateRelativePath("folder/.hidden"); });
});
it("accepts paths with single dots", () => {
assert.doesNotThrow(() => { validateRelativePath("./file.md"); });
assert.doesNotThrow(() => { validateRelativePath("folder/./file.md"); });
});
it("rejects empty paths", () => {
assert.throws(() => { validateRelativePath(""); }, /must not be empty/);
});
it("rejects paths with .. components", () => {
assert.throws(
() => { validateRelativePath("../file.md"); },
/must not contain '\.\.'/
);
assert.throws(
() => { validateRelativePath("folder/../file.md"); },
/must not contain '\.\.'/
);
assert.throws(
() => { validateRelativePath("folder/../../etc/passwd"); },
/must not contain '\.\.'/
);
assert.throws(
() => { validateRelativePath(".."); },
/must not contain '\.\.'/
);
});
it("does not reject paths containing .. as part of a filename", () => {
assert.doesNotThrow(
() => { validateRelativePath("file..name.md"); }
);
assert.doesNotThrow(
() => { validateRelativePath("folder/file..bak"); }
);
});
it("rejects absolute paths starting with /", () => {
assert.throws(
() => { validateRelativePath("/etc/passwd"); },
/must be relative/
);
});
it("rejects absolute paths starting with \\", () => {
assert.throws(
() => { validateRelativePath("\\Windows\\System32"); },
/must be relative/
);
});
it("rejects paths containing backslashes", () => {
assert.throws(
() => { validateRelativePath("folder\\file.md"); },
/must use forward slashes/
);
});
it("rejects paths with null bytes", () => {
assert.throws(
() => { validateRelativePath("file\0.md"); },
/null byte/
);
});
});

46
frontend/sync-client/src/utils/validate-relative-path.ts Normal file
View file

@ -0,0 +1,46 @@
import type { RelativePath } from "../persistence/database";
/**
* Validates that a relative path is safe and cannot escape the vault root.
*
* Rejects paths that:
* - Are empty
* - Start with `/` or `\` (absolute paths)
* - Contain `..` path components (directory traversal)
* - Contain null bytes (path truncation attacks)
* - Contain backslash separators (Windows path injection)
*
* @throws {Error} if the path is unsafe
*/
export function validateRelativePath(path: RelativePath): void {
if (path.length === 0) {
throw new Error("Path must not be empty");
}
if (path.includes("\0")) {
throw new Error(
`Path contains null byte, which is not allowed: '${path}'`
);
}
if (path.startsWith("/") || path.startsWith("\\")) {
throw new Error(
`Path must be relative, not absolute: '${path}'`
);
}
if (path.includes("\\")) {
throw new Error(
`Path must use forward slashes, not backslashes: '${path}'`
);
}
const components = path.split("/");
for (const component of components) {
if (component === "..") {
throw new Error(
`Path must not contain '..' components: '${path}'`
);
}
}
}