ai
This commit is contained in:
parent
8f2f5e4fa9
commit
a20264bcaf
112 changed files with 12567 additions and 2694 deletions
79
CLAUDE.md
79
CLAUDE.md
|
|
@ -15,11 +15,13 @@ VaultLink is a self-hosted Obsidian plugin for real-time collaborative file sync
|
|||
- **frontend/obsidian-plugin/**: Obsidian plugin that integrates the sync client with Obsidian's API
|
||||
- **frontend/test-client/**: CLI testing tool for simulating multiple concurrent users
|
||||
- **frontend/local-client-cli/**: Standalone CLI for VaultLink sync client
|
||||
- **frontend/history-ui/**: Svelte 5 web UI for browsing vault history, viewing diffs, and restoring versions
|
||||
|
||||
### Key Technologies
|
||||
|
||||
- **Backend**: Rust with Axum framework, SQLite with SQLx, WebSockets for real-time sync
|
||||
- **Frontend**: TypeScript, Webpack for bundling, Node.js native test runner
|
||||
- **History UI**: Svelte 5 with runes, Vite for bundling, embedded in server binary via `rust-embed`
|
||||
- **Sync Algorithm**: Uses reconcile-text library for operational transformation
|
||||
|
||||
### Architectural Patterns
|
||||
|
|
@ -47,6 +49,32 @@ The sync-client builds two separate bundles:
|
|||
- `sync-client.web.js`: Browser-compatible UMD bundle (excludes `ws` package)
|
||||
- `sync-client.node.js`: Node.js CommonJS bundle with WebSocket support
|
||||
|
||||
**History UI Architecture:**
|
||||
|
||||
The history UI (`frontend/history-ui/`) is a standalone Svelte 5 SPA that provides read-only vault history browsing. It communicates with the server via the same REST API used by sync clients, plus three additional endpoints:
|
||||
|
||||
- `GET /vaults/:vault_id/documents/:document_id/versions` — all versions of a document (without content)
|
||||
- `GET /vaults/:vault_id/history?limit=&before_update_id=` — paginated vault-wide version history (cursor-based)
|
||||
- `POST /vaults/:vault_id/documents/:document_id/restore` — restore a document to a historical version (creates a new version with old content)
|
||||
|
||||
Server-side implementation:
|
||||
- Database methods: `get_document_versions()` and `get_vault_history()` in `database.rs`, plus a `VaultHistoryRow` helper struct for `sqlx::query_as!`
|
||||
- Handlers: `fetch_document_versions.rs`, `fetch_vault_history.rs`, `restore_document_version.rs`
|
||||
- Response type: `VaultHistoryResponse { versions, hasMore }` in `responses.rs`
|
||||
- SPA serving: `rust-embed` embeds `frontend/history-ui/dist/` into the binary; `index.rs` serves the SPA at `/` and assets at `/assets/*`
|
||||
|
||||
Client-side component hierarchy:
|
||||
- `App.svelte` — session restore, routing
|
||||
- `Login.svelte` — vault name + token auth via `/ping`
|
||||
- `Dashboard.svelte` — main layout: file tree sidebar, activity feed, time-travel slider
|
||||
- `DocumentDetail.svelte` — version timeline, content preview, diff view, restore
|
||||
- `DiffView.svelte` — unified diff with LCS algorithm
|
||||
- `FileTree.svelte` — recursive tree built from flat `relativePath` values
|
||||
- `ActivityFeed.svelte` — git-log-style feed with action pills (created/updated/renamed/deleted/restored)
|
||||
- `TimeSlider.svelte` — scrubs through `vaultUpdateId` range, reconstructs vault state at any point
|
||||
|
||||
State is managed with Svelte 5 runes (`$state`, `$derived`, `$effect`) in `lib/stores.svelte.ts`. Auth is stored in `sessionStorage`. The API client (`lib/api.ts`) sets `Authorization: Bearer` and `device-id: history-ui` headers on all requests.
|
||||
|
||||
## Development Commands
|
||||
|
||||
### Initial Setup
|
||||
|
|
@ -101,6 +129,23 @@ npm run test -w sync-client # Run tests for specific workspace
|
|||
npm run lint # Lint and format TypeScript code with ESLint + Prettier
|
||||
```
|
||||
|
||||
### History UI Development
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm run dev -w history-ui # Start Vite dev server (localhost:5173, proxies API to localhost:3000)
|
||||
npm run build -w history-ui # Build for production (output: frontend/history-ui/dist/)
|
||||
```
|
||||
|
||||
The history UI is a Svelte 5 SPA embedded in the server binary via `rust-embed`. The build flow is:
|
||||
|
||||
1. `npm run build -w history-ui` produces `frontend/history-ui/dist/`
|
||||
2. The Rust server embeds these files at compile time (`sync-server/src/server/index.rs`)
|
||||
3. The server serves `index.html` at `GET /` and static assets at `GET /assets/*`
|
||||
4. If the dist directory doesn't exist at Rust compile time, `build.rs` creates a placeholder
|
||||
|
||||
During development, run the Vite dev server separately and use its proxy to forward API calls to the running sync server.
|
||||
|
||||
### Database Operations
|
||||
|
||||
```bash
|
||||
|
|
@ -129,12 +174,13 @@ sqlx migrate run --source src/app_state/database/migrations --database-url sqlit
|
|||
|
||||
### Workspace Configuration
|
||||
|
||||
The frontend uses npm workspaces with four packages:
|
||||
The frontend uses npm workspaces with five packages:
|
||||
|
||||
- `sync-client`: Core synchronization logic (builds dual bundles for web and Node.js)
|
||||
- `obsidian-plugin`: Obsidian-specific integration
|
||||
- `test-client`: Testing utilities for E2E tests
|
||||
- `local-client-cli`: Standalone CLI for VaultLink sync client
|
||||
- `history-ui`: Svelte 5 SPA for vault history browsing (built with Vite, embedded in server binary)
|
||||
|
||||
### Type Generation and API Updates
|
||||
|
||||
|
|
@ -201,6 +247,13 @@ scripts/clean-up.sh # Clean up after tests
|
|||
- Configuration in `frontend/package.json`
|
||||
- Run `npm run lint` to format and fix issues
|
||||
|
||||
### Svelte (History UI)
|
||||
|
||||
- Uses Svelte 5 runes syntax (`$state`, `$derived`, `$effect`, `$props`)
|
||||
- Vite as bundler with `@sveltejs/vite-plugin-svelte`
|
||||
- Excluded from the main ESLint config (Svelte files need different linting); `history-ui/**` is in the eslint ignores list
|
||||
- CSS is component-scoped via Svelte's `<style>` blocks with CSS custom properties defined in `app.css`
|
||||
|
||||
### EditorConfig
|
||||
|
||||
- `.editorconfig` at project root defines baseline formatting rules
|
||||
|
|
@ -275,7 +328,7 @@ Runs on reconnect to detect what changed while offline:
|
|||
3. For each file with metadata: schedule as update (hash comparison will skip unchanged)
|
||||
4. For each file without metadata: try to match against "deleted" DB records by content hash (detects moves). If no match, schedule as create.
|
||||
5. For DB records whose files don't exist locally: schedule as delete
|
||||
6. Deletes and updates run first, THEN creates — to avoid the server merging creates with about-to-be-deleted docs
|
||||
6. Ordering is: interrupted-deletes → updates → creates → possibly-deleted-deletes. Creates run BEFORE possibly-deleted deletes so that the server can merge creates with existing documents at the same path (preserving documentIds). If deletes ran first, a renamed+edited file would get a new documentId instead of adopting the existing one.
|
||||
|
||||
### Remote Update Processing
|
||||
|
||||
|
|
@ -368,8 +421,8 @@ Both `resolveIdempotencyKeys` and `handleMaybeMergingResponse` (for deleted pend
|
|||
**10. `resolveIdempotencyKeys` sets `parentVersionId: 0` — treat this as a create, not an update.**
|
||||
When `resolveIdempotencyKeys` assigns a documentId to a pending doc, it uses `parentVersionId: 0` as a placeholder. The sync path must check for `parentVersionId === 0` and take the CREATE path (sending a create with the idempotency key), not the UPDATE path (which would fail because version 0 doesn't exist on the server).
|
||||
|
||||
**11. Idempotent create returns can have stale content — check `contentSize`.**
|
||||
When the server returns a `FastForwardUpdate` for a create with an idempotency key, it may return the ORIGINAL version (from the first create), not a new version with the current content. The response's `contentSize` may not match `originalContentBytes.length`. If they differ, fetch the actual server content for that version and use it for the cache and hash, so subsequent diffs are correct.
|
||||
**11. Idempotent create returns can have stale content — always fetch server content.**
|
||||
When the server returns a `FastForwardUpdate` for a create with an idempotency key, it may return the ORIGINAL version (from the first create), not a new version with the current content. Always fetch the actual server content for idempotent create returns (the `isCreate` path in `handleMaybeMergingResponse`) and use it for the cache and hash, so subsequent diffs are correct. Do not use a content-length comparison as a shortcut — two different byte sequences can have the same length.
|
||||
|
||||
**12. `SyncClient.pause()` must swallow `SyncResetError`.**
|
||||
`pause()` calls `fetchController.startReset()` which rejects in-flight fetches. Those rejections propagate through `waitUntilFinished()`. Since `pause()` CAUSED the reset, the resulting `SyncResetError` is expected and must be caught (not re-thrown). Only re-throw non-SyncResetError exceptions. Also call `fetchController.finishReset()` in the catch block to prevent the FetchController from getting stuck in resetting state.
|
||||
|
|
@ -380,14 +433,26 @@ After the initial `scheduleSyncForOfflineChanges()` completes, the field retains
|
|||
**14. The server must not `expect()` / panic on UTF-8 conversion — return a client error.**
|
||||
In `update_text`, the parent version's content may be binary (if another client uploaded binary via `putBinary`). Using `.expect()` on `str::from_utf8()` panics the server. Use `.context(...).map_err(client_error)?` to return a 4xx error, allowing the client to fall back to `putBinary`.
|
||||
|
||||
**15. The create-merge parent content must be `latest_version.content`, not empty.**
|
||||
In `create_document.rs`, when a create merges with an existing document, the 3-way merge parent must be the latest version's content (`&latest_version.content`), not an empty vector (`&Vec::new()`). An empty parent causes `reconcile("", existing, new)` to treat all content as additions, producing garbled interleaved text.
|
||||
**15. The create-merge parent content must be empty (`&Vec::new()`), not `latest_version.content`.**
|
||||
In `create_document.rs`, when a create merges with an existing document, the 3-way merge parent must be an empty vector (`&Vec::new()`), not the latest version's content. Using `latest_version.content` as the parent makes `reconcile(A, A, B) = B`, which silently discards the existing content (last-write-wins). An empty parent causes `reconcile("", existing, new)` to correctly treat both sides as independent additions and merge them together.
|
||||
|
||||
**16. `retryForever` must not retry 4xx HTTP errors.**
|
||||
4xx errors indicate the request itself is wrong (e.g., invalid diff, missing parent version). Retrying won't help. The `HttpClientError` class (in `errors/http-client-error.ts`) carries the status code. `retryForever` checks for it and re-throws immediately. Only 5xx errors (transient server failures) are retried.
|
||||
|
||||
**17. The broadcast channel's `RecvError::Lagged` must be handled explicitly.**
|
||||
The `while let Ok(update) = broadcast_receiver.recv().await` pattern silently exits the loop on `Lagged`, disconnecting the client without logging. Handle `Lagged` explicitly with a `warn!` log and `break`. The channel capacity (`broadcast_channel_capacity` in config, default 1024) is separate from `max_clients_per_vault`.
|
||||
The `while let Ok(update) = broadcast_receiver.recv().await` pattern silently exits the loop on `Lagged`, disconnecting the client without logging. Handle `Lagged` explicitly with a `warn!` log and `break`. The channel capacity is `max_clients_per_vault`.
|
||||
|
||||
**18. `merge_with_stored_version` must not short-circuit when an idempotency key is provided.**
|
||||
When the new content is identical to the latest version and an `idempotency_key` is present, the function must still insert a new version row so the key is persisted in the database. Without this, the key is lost: `resolveIdempotencyKeys` returns no match after a crash, and the client retries the create without idempotency protection — potentially doubling content via the empty-parent merge. The short-circuit (`content == latest_version.content && ... && idempotency_key.is_none()`) only applies to keyless updates.
|
||||
|
||||
**19. The idempotency key check in `create_document` must skip deleted documents.**
|
||||
When `get_document_by_idempotency_key` returns a document with `is_deleted: true`, the server must NOT return it as an idempotent match. Returning a deleted version causes the client to call `applyRemoteDeleteLocally`, silently deleting the user's local file. Instead, fall through to the normal create path so the file is preserved as a new document.
|
||||
|
||||
**20. `syncLocallyCreatedFile` must treat `parentVersionId === 0` as needing a create retry.**
|
||||
When `resolveIdempotencyKeys` assigns metadata with `parentVersionId: 0`, the document looks "resolved" to `syncLocallyCreatedFile` (it has `metadata !== undefined`). Without a special check for `parentVersionId === 0`, the method returns early ("already exists with metadata"), leaving the document permanently stuck — it never syncs. The fix: when `parentVersionId === 0`, treat it like a pending create retry and enqueue `unrestrictedSyncLocallyCreatedOrUpdatedFile`.
|
||||
|
||||
**21. The client must normalize content to UTF-8 at the read boundary.**
|
||||
`FileOperations.read()` calls `normalizeToUtf8()` to transcode UTF-16 (detected by BOM) to UTF-8 before any downstream code sees the bytes. This means `isBinary` / `is_binary` on both client and server only need to check UTF-8 validity — no UTF-16 handling required. A disagreement between client and server on text vs binary causes permanent sync failures (client sends `putText` for content the server considers binary, 4xx error, `retryForever` won't retry). The UTF-8-only contract keeps classification trivial and impossible to get out of sync.
|
||||
|
||||
### E2E Test Debugging Guide
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
export const TOKEN = "test-token-change-me ";
|
||||
export const REMOTE_URI = "http://localhost:3000";
|
||||
export const REMOTE_URI = "http://localhost:3010";
|
||||
export const PING_URL = `${REMOTE_URI}/vaults/test/ping`;
|
||||
export const SERVER_BINARY_PATH = "sync-server/target/debug/sync_server";
|
||||
export const CONFIG_PATH = "sync-server/config-e2e.yml";
|
||||
|
|
|
|||
|
|
@ -87,7 +87,6 @@ export class TestRunner {
|
|||
isSyncEnabled: false,
|
||||
token: this.token,
|
||||
vaultName,
|
||||
syncConcurrency: 1,
|
||||
remoteUri: this.remoteUri
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -8,7 +8,8 @@ export default [
|
|||
"sync-client/src/services/types.ts",
|
||||
"**/dist/",
|
||||
"**/*.mjs",
|
||||
"**/*.js"
|
||||
"**/*.js",
|
||||
"history-ui/**"
|
||||
]
|
||||
},
|
||||
...tseslint.config({
|
||||
|
|
|
|||
13
frontend/history-ui/index.html
Normal file
13
frontend/history-ui/index.html
Normal file
|
|
@ -0,0 +1,13 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8" />
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
|
||||
<title>VaultLink2</title>
|
||||
<link rel="icon" href="data:," />
|
||||
</head>
|
||||
<body>
|
||||
<div id="app"></div>
|
||||
<script type="module" src="/src/main.ts"></script>
|
||||
</body>
|
||||
</html>
|
||||
16
frontend/history-ui/package.json
Normal file
16
frontend/history-ui/package.json
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"name": "history-ui",
|
||||
"version": "0.14.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"scripts": {
|
||||
"dev": "vite dev",
|
||||
"build": "vite build",
|
||||
"test": "echo 'no tests yet'"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@sveltejs/vite-plugin-svelte": "^5.0.0",
|
||||
"svelte": "^5.0.0",
|
||||
"vite": "^6.0.0"
|
||||
}
|
||||
}
|
||||
71
frontend/history-ui/src/App.svelte
Normal file
71
frontend/history-ui/src/App.svelte
Normal file
|
|
@ -0,0 +1,71 @@
|
|||
<script lang="ts">
|
||||
import { auth, nav, toasts } from "./lib/stores.svelte";
|
||||
import Login from "./components/Login.svelte";
|
||||
import Dashboard from "./components/Dashboard.svelte";
|
||||
import ToastContainer from "./components/ToastContainer.svelte";
|
||||
import { ApiClient } from "./lib/api";
|
||||
|
||||
let restoring = $state(true);
|
||||
|
||||
$effect(() => {
|
||||
const saved = auth.tryRestore();
|
||||
if (saved) {
|
||||
const client = new ApiClient(saved.vaultId, saved.token);
|
||||
client
|
||||
.ping()
|
||||
.then((ping) => {
|
||||
if (ping.isAuthenticated) {
|
||||
auth.login(
|
||||
saved.vaultId,
|
||||
saved.token,
|
||||
ping.serverVersion
|
||||
);
|
||||
}
|
||||
restoring = false;
|
||||
})
|
||||
.catch(() => {
|
||||
restoring = false;
|
||||
});
|
||||
} else {
|
||||
restoring = false;
|
||||
}
|
||||
});
|
||||
</script>
|
||||
|
||||
{#if restoring}
|
||||
<div class="loading-screen">
|
||||
<div class="spinner"></div>
|
||||
</div>
|
||||
{:else if !auth.isAuthenticated}
|
||||
<Login />
|
||||
{:else}
|
||||
<Dashboard
|
||||
selectedDocumentId={nav.current.kind === "document" ? nav.current.documentId : undefined}
|
||||
/>
|
||||
{/if}
|
||||
|
||||
<ToastContainer />
|
||||
|
||||
<style>
|
||||
.loading-screen {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
height: 100%;
|
||||
}
|
||||
|
||||
.spinner {
|
||||
width: 32px;
|
||||
height: 32px;
|
||||
border: 3px solid var(--bg-tertiary);
|
||||
border-top-color: var(--accent);
|
||||
border-radius: 50%;
|
||||
animation: spin 0.6s linear infinite;
|
||||
}
|
||||
|
||||
@keyframes spin {
|
||||
to {
|
||||
transform: rotate(360deg);
|
||||
}
|
||||
}
|
||||
</style>
|
||||
101
frontend/history-ui/src/app.css
Normal file
101
frontend/history-ui/src/app.css
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
:root {
|
||||
--bg: #0d1117;
|
||||
--bg-secondary: #161b22;
|
||||
--bg-tertiary: #21262d;
|
||||
--bg-hover: #30363d;
|
||||
--border: #30363d;
|
||||
--border-light: #21262d;
|
||||
--text: #e6edf3;
|
||||
--text-muted: #8b949e;
|
||||
--text-subtle: #6e7681;
|
||||
--accent: #58a6ff;
|
||||
--accent-hover: #79c0ff;
|
||||
--green: #3fb950;
|
||||
--green-bg: rgba(63, 185, 80, 0.15);
|
||||
--red: #f85149;
|
||||
--red-bg: rgba(248, 81, 73, 0.15);
|
||||
--orange: #d29922;
|
||||
--orange-bg: rgba(210, 153, 34, 0.15);
|
||||
--purple: #bc8cff;
|
||||
--purple-bg: rgba(188, 140, 255, 0.15);
|
||||
--blue: #58a6ff;
|
||||
--blue-bg: rgba(88, 166, 255, 0.15);
|
||||
--mono: "SF Mono", "Fira Code", "Fira Mono", Menlo, Consolas, monospace;
|
||||
--sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Noto Sans, Helvetica, Arial, sans-serif;
|
||||
--radius: 6px;
|
||||
--radius-sm: 4px;
|
||||
--shadow: 0 1px 3px rgba(0, 0, 0, 0.3);
|
||||
}
|
||||
|
||||
* {
|
||||
margin: 0;
|
||||
padding: 0;
|
||||
box-sizing: border-box;
|
||||
}
|
||||
|
||||
html, body, #app {
|
||||
height: 100%;
|
||||
width: 100%;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
body {
|
||||
font-family: var(--sans);
|
||||
font-size: 14px;
|
||||
line-height: 1.5;
|
||||
color: var(--text);
|
||||
background: var(--bg);
|
||||
-webkit-font-smoothing: antialiased;
|
||||
}
|
||||
|
||||
button {
|
||||
font-family: inherit;
|
||||
font-size: inherit;
|
||||
cursor: pointer;
|
||||
border: none;
|
||||
background: none;
|
||||
color: inherit;
|
||||
}
|
||||
|
||||
input {
|
||||
font-family: inherit;
|
||||
font-size: inherit;
|
||||
color: inherit;
|
||||
background: var(--bg);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
padding: 8px 12px;
|
||||
outline: none;
|
||||
transition: border-color 0.15s;
|
||||
}
|
||||
|
||||
input:focus {
|
||||
border-color: var(--accent);
|
||||
}
|
||||
|
||||
a {
|
||||
color: var(--accent);
|
||||
text-decoration: none;
|
||||
}
|
||||
|
||||
a:hover {
|
||||
text-decoration: underline;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar-track {
|
||||
background: transparent;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar-thumb {
|
||||
background: var(--bg-tertiary);
|
||||
border-radius: 4px;
|
||||
}
|
||||
|
||||
::-webkit-scrollbar-thumb:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
346
frontend/history-ui/src/components/ActivityFeed.svelte
Normal file
346
frontend/history-ui/src/components/ActivityFeed.svelte
Normal file
|
|
@ -0,0 +1,346 @@
|
|||
<script lang="ts">
|
||||
import type { VersionEvent } from "../lib/types";
|
||||
import {
|
||||
absoluteTime,
|
||||
formatBytes
|
||||
} from "../lib/stores.svelte";
|
||||
|
||||
interface Props {
|
||||
versions: VersionEvent[];
|
||||
loading: boolean;
|
||||
hasMore: boolean;
|
||||
onLoadMore: () => void;
|
||||
onSelectDocument: (documentId: string) => void;
|
||||
onTimeTravel: (vaultUpdateId: number) => void;
|
||||
}
|
||||
|
||||
let {
|
||||
versions,
|
||||
loading,
|
||||
hasMore,
|
||||
onLoadMore,
|
||||
onSelectDocument,
|
||||
onTimeTravel
|
||||
}: Props = $props();
|
||||
|
||||
function timeOfDay(dateStr: string): string {
|
||||
return new Date(dateStr).toLocaleTimeString("en-US", {
|
||||
hour: "numeric",
|
||||
minute: "2-digit"
|
||||
});
|
||||
}
|
||||
|
||||
// Group by day
|
||||
let grouped = $derived.by(() => {
|
||||
const groups: { date: string; items: VersionEvent[] }[] = [];
|
||||
const sortedDesc = [...versions].sort(
|
||||
(a, b) => b.vaultUpdateId - a.vaultUpdateId
|
||||
);
|
||||
|
||||
for (const v of sortedDesc) {
|
||||
const date = new Date(v.updatedDate).toLocaleDateString(
|
||||
"en-US",
|
||||
{ month: "long", day: "numeric", year: "numeric" }
|
||||
);
|
||||
const last = groups.at(-1);
|
||||
if (last && last.date === date) {
|
||||
last.items.push(v);
|
||||
} else {
|
||||
groups.push({ date, items: [v] });
|
||||
}
|
||||
}
|
||||
return groups;
|
||||
});
|
||||
|
||||
const actionColors: Record<string, string> = {
|
||||
created: "var(--green)",
|
||||
updated: "var(--blue)",
|
||||
renamed: "var(--orange)",
|
||||
deleted: "var(--red)",
|
||||
restored: "var(--purple)"
|
||||
};
|
||||
|
||||
const actionBgColors: Record<string, string> = {
|
||||
created: "var(--green-bg)",
|
||||
updated: "var(--blue-bg)",
|
||||
renamed: "var(--orange-bg)",
|
||||
deleted: "var(--red-bg)",
|
||||
restored: "var(--purple-bg)"
|
||||
};
|
||||
</script>
|
||||
|
||||
<div class="feed">
|
||||
{#if loading && versions.length === 0}
|
||||
<div class="feed-loading">Loading activity...</div>
|
||||
{:else if versions.length === 0}
|
||||
<div class="feed-empty">
|
||||
No activity yet. Documents will appear here as sync clients
|
||||
make changes.
|
||||
</div>
|
||||
{:else}
|
||||
{#each grouped as group}
|
||||
<div class="day-group">
|
||||
<div class="day-header">{group.date}</div>
|
||||
<div class="items-list">
|
||||
{#each group.items as event}
|
||||
<div class="feed-item">
|
||||
<button
|
||||
class="feed-item-main"
|
||||
onclick={() =>
|
||||
onSelectDocument(event.documentId)}
|
||||
>
|
||||
<div class="feed-timeline">
|
||||
<div
|
||||
class="timeline-dot"
|
||||
style="background: {actionColors[
|
||||
event.action
|
||||
]}"
|
||||
></div>
|
||||
</div>
|
||||
<div class="feed-content">
|
||||
<div class="feed-header">
|
||||
<span
|
||||
class="action-pill"
|
||||
style="color: {actionColors[
|
||||
event.action
|
||||
]}; background: {actionBgColors[
|
||||
event.action
|
||||
]}"
|
||||
>
|
||||
{event.action}
|
||||
</span>
|
||||
<span class="feed-path">
|
||||
{#if event.action === "renamed" && event.previousPath}
|
||||
<span class="prev-path"
|
||||
>{event.previousPath}</span
|
||||
>
|
||||
<span class="arrow"
|
||||
>→</span
|
||||
>
|
||||
{/if}
|
||||
<span
|
||||
class:deleted={event.action ===
|
||||
"deleted"}
|
||||
>
|
||||
{event.relativePath}
|
||||
</span>
|
||||
</span>
|
||||
</div>
|
||||
<div class="feed-meta">
|
||||
<span class="feed-user"
|
||||
>{event.userId}</span
|
||||
>
|
||||
<span class="feed-dot"
|
||||
>·</span
|
||||
>
|
||||
<span class="feed-size"
|
||||
>{formatBytes(
|
||||
event.contentSize
|
||||
)}</span
|
||||
>
|
||||
</div>
|
||||
</div>
|
||||
</button>
|
||||
<button
|
||||
class="feed-time-btn"
|
||||
title="Time travel to {absoluteTime(event.updatedDate)}"
|
||||
onclick={(e) => {
|
||||
e.stopPropagation();
|
||||
onTimeTravel(event.vaultUpdateId);
|
||||
}}
|
||||
>
|
||||
{timeOfDay(event.updatedDate)}
|
||||
</button>
|
||||
</div>
|
||||
{/each}
|
||||
</div>
|
||||
</div>
|
||||
{/each}
|
||||
|
||||
{#if hasMore}
|
||||
<div class="load-more">
|
||||
<button class="load-more-btn" onclick={onLoadMore}>
|
||||
Load older activity
|
||||
</button>
|
||||
</div>
|
||||
{/if}
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.feed {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 0 0 16px;
|
||||
}
|
||||
|
||||
.feed-loading,
|
||||
.feed-empty {
|
||||
padding: 48px 16px;
|
||||
text-align: center;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.day-group {
|
||||
margin-bottom: 8px;
|
||||
}
|
||||
|
||||
.day-header {
|
||||
position: sticky;
|
||||
top: 0;
|
||||
z-index: 1;
|
||||
padding: 8px 16px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
color: var(--text-muted);
|
||||
background: var(--bg);
|
||||
border-bottom: 1px solid var(--border-light);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
}
|
||||
|
||||
.feed-item {
|
||||
display: flex;
|
||||
align-items: stretch;
|
||||
width: 100%;
|
||||
transition: background 0.1s;
|
||||
}
|
||||
|
||||
.feed-item:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
.feed-item-main {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
padding: 10px 0 10px 16px;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.items-list {
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.items-list::before {
|
||||
content: "";
|
||||
position: absolute;
|
||||
left: 21px;
|
||||
top: 0;
|
||||
bottom: 0;
|
||||
width: 2px;
|
||||
background: var(--border);
|
||||
}
|
||||
|
||||
.feed-timeline {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
width: 12px;
|
||||
flex-shrink: 0;
|
||||
padding-top: 6px;
|
||||
position: relative;
|
||||
}
|
||||
|
||||
.timeline-dot {
|
||||
width: 8px;
|
||||
height: 8px;
|
||||
border-radius: 50%;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.feed-content {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.feed-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
flex-wrap: wrap;
|
||||
}
|
||||
|
||||
.action-pill {
|
||||
font-size: 11px;
|
||||
font-weight: 600;
|
||||
padding: 1px 8px;
|
||||
border-radius: 10px;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.3px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.feed-path {
|
||||
font-family: var(--mono);
|
||||
font-size: 13px;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.prev-path {
|
||||
color: var(--text-muted);
|
||||
text-decoration: line-through;
|
||||
}
|
||||
|
||||
.arrow {
|
||||
color: var(--text-subtle);
|
||||
margin: 0 4px;
|
||||
}
|
||||
|
||||
.deleted {
|
||||
text-decoration: line-through;
|
||||
opacity: 0.6;
|
||||
}
|
||||
|
||||
.feed-meta {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
margin-top: 4px;
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.feed-dot {
|
||||
color: var(--text-subtle);
|
||||
}
|
||||
|
||||
.feed-time-btn {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
padding: 0 16px;
|
||||
font-size: 12px;
|
||||
font-family: var(--mono);
|
||||
color: var(--text-muted);
|
||||
white-space: nowrap;
|
||||
flex-shrink: 0;
|
||||
border-left: 1px solid transparent;
|
||||
transition: color 0.15s, border-color 0.15s;
|
||||
}
|
||||
|
||||
.feed-time-btn:hover {
|
||||
color: var(--accent);
|
||||
border-left-color: var(--border-light);
|
||||
}
|
||||
|
||||
.load-more {
|
||||
padding: 16px;
|
||||
text-align: center;
|
||||
}
|
||||
|
||||
.load-more-btn {
|
||||
padding: 8px 20px;
|
||||
font-size: 13px;
|
||||
color: var(--accent);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
transition: background 0.15s;
|
||||
}
|
||||
|
||||
.load-more-btn:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
</style>
|
||||
167
frontend/history-ui/src/components/ConfirmDialog.svelte
Normal file
167
frontend/history-ui/src/components/ConfirmDialog.svelte
Normal file
|
|
@ -0,0 +1,167 @@
|
|||
<script lang="ts">
|
||||
interface Props {
|
||||
title: string;
|
||||
message: string;
|
||||
confirmLabel: string;
|
||||
destructive?: boolean;
|
||||
loading?: boolean;
|
||||
onConfirm: () => void;
|
||||
onCancel: () => void;
|
||||
}
|
||||
|
||||
let {
|
||||
title,
|
||||
message,
|
||||
confirmLabel,
|
||||
destructive = false,
|
||||
loading = false,
|
||||
onConfirm,
|
||||
onCancel
|
||||
}: Props = $props();
|
||||
|
||||
function handleKeydown(e: KeyboardEvent) {
|
||||
if (e.key === "Escape") onCancel();
|
||||
}
|
||||
</script>
|
||||
|
||||
<svelte:window on:keydown={handleKeydown} />
|
||||
|
||||
<!-- svelte-ignore a11y_click_events_have_key_events -->
|
||||
<div class="backdrop" onclick={onCancel} role="presentation">
|
||||
<!-- svelte-ignore a11y_click_events_have_key_events -->
|
||||
<!-- svelte-ignore a11y_interactive_supports_focus -->
|
||||
<div
|
||||
class="dialog"
|
||||
onclick={(e) => e.stopPropagation()}
|
||||
role="dialog"
|
||||
aria-label={title}
|
||||
>
|
||||
<h3 class="dialog-title">{title}</h3>
|
||||
<p class="dialog-message">{message}</p>
|
||||
<div class="dialog-actions">
|
||||
<button class="btn-cancel" onclick={onCancel} disabled={loading}>
|
||||
Cancel
|
||||
</button>
|
||||
<button
|
||||
class="btn-confirm"
|
||||
class:destructive
|
||||
onclick={onConfirm}
|
||||
disabled={loading}
|
||||
>
|
||||
{#if loading}
|
||||
<span class="btn-spinner"></span>
|
||||
{/if}
|
||||
{confirmLabel}
|
||||
</button>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.backdrop {
|
||||
position: fixed;
|
||||
inset: 0;
|
||||
z-index: 1000;
|
||||
background: rgba(0, 0, 0, 0.6);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
animation: fade-in 0.15s ease-out;
|
||||
}
|
||||
|
||||
.dialog {
|
||||
background: var(--bg-secondary);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 12px;
|
||||
padding: 24px;
|
||||
max-width: 480px;
|
||||
width: calc(100% - 32px);
|
||||
box-shadow: 0 8px 32px rgba(0, 0, 0, 0.4);
|
||||
animation: scale-in 0.2s ease-out;
|
||||
}
|
||||
|
||||
.dialog-title {
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
|
||||
.dialog-message {
|
||||
font-size: 14px;
|
||||
color: var(--text-muted);
|
||||
line-height: 1.5;
|
||||
margin-bottom: 24px;
|
||||
}
|
||||
|
||||
.dialog-actions {
|
||||
display: flex;
|
||||
justify-content: flex-end;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.btn-cancel {
|
||||
padding: 8px 16px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius);
|
||||
color: var(--text-muted);
|
||||
transition: background 0.15s;
|
||||
}
|
||||
|
||||
.btn-cancel:hover:not(:disabled) {
|
||||
background: var(--bg-hover);
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.btn-confirm {
|
||||
padding: 8px 16px;
|
||||
background: var(--accent);
|
||||
color: #fff;
|
||||
font-weight: 600;
|
||||
border-radius: var(--radius);
|
||||
transition: background 0.15s;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.btn-confirm:hover:not(:disabled) {
|
||||
background: var(--accent-hover);
|
||||
}
|
||||
|
||||
.btn-confirm.destructive {
|
||||
background: var(--red);
|
||||
}
|
||||
|
||||
.btn-confirm.destructive:hover:not(:disabled) {
|
||||
background: #f97583;
|
||||
}
|
||||
|
||||
.btn-confirm:disabled,
|
||||
.btn-cancel:disabled {
|
||||
opacity: 0.6;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.btn-spinner {
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
border: 2px solid rgba(255, 255, 255, 0.3);
|
||||
border-top-color: #fff;
|
||||
border-radius: 50%;
|
||||
animation: spin 0.6s linear infinite;
|
||||
}
|
||||
|
||||
@keyframes fade-in {
|
||||
from { opacity: 0; }
|
||||
to { opacity: 1; }
|
||||
}
|
||||
|
||||
@keyframes scale-in {
|
||||
from { transform: scale(0.95); opacity: 0; }
|
||||
to { transform: scale(1); opacity: 1; }
|
||||
}
|
||||
|
||||
@keyframes spin {
|
||||
to { transform: rotate(360deg); }
|
||||
}
|
||||
</style>
|
||||
507
frontend/history-ui/src/components/Dashboard.svelte
Normal file
507
frontend/history-ui/src/components/Dashboard.svelte
Normal file
|
|
@ -0,0 +1,507 @@
|
|||
<script lang="ts">
|
||||
import {
|
||||
auth,
|
||||
nav,
|
||||
toasts,
|
||||
buildTree,
|
||||
enrichVersions,
|
||||
relativeTime,
|
||||
formatBytes,
|
||||
type View
|
||||
} from "../lib/stores.svelte";
|
||||
import type {
|
||||
DocumentVersionWithoutContent,
|
||||
VaultHistoryResponse,
|
||||
VersionEvent,
|
||||
TreeNode
|
||||
} from "../lib/types";
|
||||
import FileTree from "./FileTree.svelte";
|
||||
import ActivityFeed from "./ActivityFeed.svelte";
|
||||
import DocumentDetail from "./DocumentDetail.svelte";
|
||||
import TimeSlider from "./TimeSlider.svelte";
|
||||
import Header from "./Header.svelte";
|
||||
|
||||
interface Props {
|
||||
selectedDocumentId?: string;
|
||||
}
|
||||
|
||||
let { selectedDocumentId }: Props = $props();
|
||||
|
||||
// Data
|
||||
let latestDocuments = $state<DocumentVersionWithoutContent[]>([]);
|
||||
let historyVersions = $state<DocumentVersionWithoutContent[]>([]);
|
||||
let historyHasMore = $state(false);
|
||||
let loadingDocs = $state(true);
|
||||
let loadingHistory = $state(true);
|
||||
let showDeleted = $state(false);
|
||||
let searchQuery = $state("");
|
||||
let activeTab = $state<"activity" | "files">("activity");
|
||||
|
||||
// Time travel
|
||||
let maxUpdateId = $state(0);
|
||||
let minUpdateId = $state(0);
|
||||
let timeSliderValue = $state<number | null>(null);
|
||||
|
||||
// Derived
|
||||
let tree = $derived(buildTree(latestDocuments, showDeleted));
|
||||
let enrichedHistory = $derived(enrichVersions(historyVersions));
|
||||
let stats = $derived({
|
||||
totalDocs: latestDocuments.filter((d) => !d.isDeleted).length,
|
||||
deletedDocs: latestDocuments.filter((d) => d.isDeleted).length,
|
||||
totalSize: latestDocuments
|
||||
.filter((d) => !d.isDeleted)
|
||||
.reduce((sum, d) => sum + d.contentSize, 0),
|
||||
users: [...new Set(latestDocuments.map((d) => d.userId))]
|
||||
});
|
||||
|
||||
let filteredTree = $derived.by(() => {
|
||||
if (!searchQuery) return tree;
|
||||
return filterTree(tree, searchQuery.toLowerCase());
|
||||
});
|
||||
|
||||
function filterTree(node: TreeNode, query: string): TreeNode {
|
||||
if (!node.isFolder) {
|
||||
return node.name.toLowerCase().includes(query) ? node : { ...node, children: [] };
|
||||
}
|
||||
const filteredChildren = node.children
|
||||
.map((c) => filterTree(c, query))
|
||||
.filter((c) => c.isFolder ? c.children.length > 0 : true)
|
||||
.filter((c) => !c.isFolder || c.children.length > 0);
|
||||
return { ...node, children: filteredChildren };
|
||||
}
|
||||
|
||||
// Time travel: compute vault state at a given updateId
|
||||
let timeFilteredDocs = $derived.by(() => {
|
||||
if (timeSliderValue === null || timeSliderValue >= maxUpdateId) {
|
||||
return latestDocuments;
|
||||
}
|
||||
// From all history, find the latest version per documentId at or before timeSliderValue
|
||||
const byDoc = new Map<string, DocumentVersionWithoutContent>();
|
||||
for (const v of historyVersions) {
|
||||
if (v.vaultUpdateId <= timeSliderValue) {
|
||||
const existing = byDoc.get(v.documentId);
|
||||
if (
|
||||
!existing ||
|
||||
v.vaultUpdateId > existing.vaultUpdateId
|
||||
) {
|
||||
byDoc.set(v.documentId, v);
|
||||
}
|
||||
}
|
||||
}
|
||||
return [...byDoc.values()];
|
||||
});
|
||||
|
||||
let timeFilteredTree = $derived(
|
||||
buildTree(
|
||||
timeSliderValue !== null && timeSliderValue < maxUpdateId
|
||||
? timeFilteredDocs
|
||||
: latestDocuments,
|
||||
showDeleted
|
||||
)
|
||||
);
|
||||
|
||||
let displayTree = $derived(
|
||||
searchQuery ? filteredTree : timeFilteredTree
|
||||
);
|
||||
|
||||
// Load data
|
||||
async function loadData() {
|
||||
const api = auth.api;
|
||||
if (!api) return;
|
||||
|
||||
loadingDocs = true;
|
||||
loadingHistory = true;
|
||||
|
||||
try {
|
||||
const response = await api.fetchLatestDocuments();
|
||||
latestDocuments = response.latestDocuments;
|
||||
maxUpdateId = response.lastUpdateId;
|
||||
} catch (e) {
|
||||
toasts.add("Failed to load documents", "error");
|
||||
} finally {
|
||||
loadingDocs = false;
|
||||
}
|
||||
|
||||
try {
|
||||
const response = await api.fetchVaultHistory(500);
|
||||
historyVersions = response.versions;
|
||||
historyHasMore = response.hasMore;
|
||||
if (historyVersions.length > 0) {
|
||||
minUpdateId = Math.min(
|
||||
...historyVersions.map((v) => v.vaultUpdateId)
|
||||
);
|
||||
maxUpdateId = Math.max(
|
||||
maxUpdateId,
|
||||
Math.max(
|
||||
...historyVersions.map((v) => v.vaultUpdateId)
|
||||
)
|
||||
);
|
||||
}
|
||||
} catch (e) {
|
||||
toasts.add("Failed to load history", "error");
|
||||
} finally {
|
||||
loadingHistory = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function loadMoreHistory() {
|
||||
const api = auth.api;
|
||||
if (!api || !historyHasMore) return;
|
||||
|
||||
const oldest = Math.min(
|
||||
...historyVersions.map((v) => v.vaultUpdateId)
|
||||
);
|
||||
try {
|
||||
const response = await api.fetchVaultHistory(500, oldest);
|
||||
historyVersions = [...historyVersions, ...response.versions];
|
||||
historyHasMore = response.hasMore;
|
||||
minUpdateId = Math.min(
|
||||
minUpdateId,
|
||||
...response.versions.map((v) => v.vaultUpdateId)
|
||||
);
|
||||
} catch {
|
||||
toasts.add("Failed to load more history", "error");
|
||||
}
|
||||
}
|
||||
|
||||
function selectDocument(documentId: string) {
|
||||
nav.goto({ kind: "document", documentId });
|
||||
}
|
||||
|
||||
function handleRefresh() {
|
||||
loadData();
|
||||
}
|
||||
|
||||
$effect(() => {
|
||||
if (auth.isAuthenticated) {
|
||||
loadData();
|
||||
}
|
||||
});
|
||||
</script>
|
||||
|
||||
<div class="dashboard">
|
||||
<Header
|
||||
vaultId={auth.vaultId}
|
||||
serverVersion={auth.serverVersion}
|
||||
onRefresh={handleRefresh}
|
||||
/>
|
||||
|
||||
<div class="main-layout">
|
||||
<!-- Sidebar -->
|
||||
<aside class="sidebar">
|
||||
{#if !loadingDocs}
|
||||
<div class="sidebar-stats">
|
||||
<div class="stat">
|
||||
<span class="stat-value">{stats.totalDocs}</span>
|
||||
<span class="stat-label">files</span>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<span class="stat-value"
|
||||
>{formatBytes(stats.totalSize)}</span
|
||||
>
|
||||
<span class="stat-label">total</span>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<span class="stat-value">{stats.users.length}</span>
|
||||
<span class="stat-label"
|
||||
>user{stats.users.length !== 1 ? "s" : ""}</span
|
||||
>
|
||||
</div>
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
<div class="sidebar-search">
|
||||
<input
|
||||
type="text"
|
||||
placeholder="Filter files..."
|
||||
bind:value={searchQuery}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div class="sidebar-controls">
|
||||
<label class="toggle-label">
|
||||
<input
|
||||
type="checkbox"
|
||||
bind:checked={showDeleted}
|
||||
/>
|
||||
Show deleted
|
||||
</label>
|
||||
</div>
|
||||
|
||||
<div class="sidebar-tree">
|
||||
{#if loadingDocs}
|
||||
<div class="loading-placeholder">Loading...</div>
|
||||
{:else}
|
||||
<FileTree
|
||||
node={displayTree}
|
||||
selectedId={selectedDocumentId ?? null}
|
||||
onSelect={selectDocument}
|
||||
/>
|
||||
{/if}
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<!-- Main content -->
|
||||
<main class="content">
|
||||
{#if maxUpdateId > 0}
|
||||
<div class="time-slider-container">
|
||||
<TimeSlider
|
||||
min={minUpdateId}
|
||||
max={maxUpdateId}
|
||||
value={timeSliderValue}
|
||||
versions={historyVersions}
|
||||
onchange={(v) => {
|
||||
timeSliderValue = v;
|
||||
}}
|
||||
/>
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
{#if selectedDocumentId}
|
||||
<DocumentDetail
|
||||
documentId={selectedDocumentId}
|
||||
onClose={() => nav.goHome()}
|
||||
onRestore={handleRefresh}
|
||||
/>
|
||||
{:else}
|
||||
<div class="tabs">
|
||||
<button
|
||||
class="tab"
|
||||
class:active={activeTab === "activity"}
|
||||
onclick={() => (activeTab = "activity")}
|
||||
>
|
||||
Activity
|
||||
</button>
|
||||
<button
|
||||
class="tab"
|
||||
class:active={activeTab === "files"}
|
||||
onclick={() => (activeTab = "files")}
|
||||
>
|
||||
Files
|
||||
</button>
|
||||
</div>
|
||||
|
||||
{#if activeTab === "activity"}
|
||||
<ActivityFeed
|
||||
versions={enrichedHistory}
|
||||
loading={loadingHistory}
|
||||
hasMore={historyHasMore}
|
||||
onLoadMore={loadMoreHistory}
|
||||
onSelectDocument={selectDocument}
|
||||
onTimeTravel={(id) => {
|
||||
timeSliderValue = id >= maxUpdateId ? null : id;
|
||||
}}
|
||||
/>
|
||||
{:else}
|
||||
<div class="file-list">
|
||||
{#each latestDocuments
|
||||
.filter((d) => showDeleted || !d.isDeleted)
|
||||
.sort((a, b) => b.vaultUpdateId - a.vaultUpdateId) as doc}
|
||||
<button
|
||||
class="file-row"
|
||||
class:deleted={doc.isDeleted}
|
||||
onclick={() =>
|
||||
selectDocument(doc.documentId)}
|
||||
>
|
||||
<span class="file-icon"
|
||||
>{doc.isDeleted
|
||||
? "🗑"
|
||||
: "📄"}</span
|
||||
>
|
||||
<span class="file-path"
|
||||
>{doc.relativePath}</span
|
||||
>
|
||||
<span class="file-meta">
|
||||
{formatBytes(doc.contentSize)}
|
||||
·
|
||||
{doc.userId}
|
||||
·
|
||||
{relativeTime(doc.updatedDate)}
|
||||
</span>
|
||||
</button>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
{/if}
|
||||
</main>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.dashboard {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100%;
|
||||
}
|
||||
|
||||
.main-layout {
|
||||
display: flex;
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.sidebar {
|
||||
width: 280px;
|
||||
min-width: 280px;
|
||||
border-right: 1px solid var(--border);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
background: var(--bg-secondary);
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.sidebar-stats {
|
||||
display: flex;
|
||||
gap: 12px;
|
||||
padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.stat {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
align-items: center;
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.stat-value {
|
||||
font-size: 16px;
|
||||
font-weight: 600;
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.stat-label {
|
||||
font-size: 11px;
|
||||
color: var(--text-muted);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
}
|
||||
|
||||
.sidebar-search {
|
||||
padding: 8px 12px;
|
||||
}
|
||||
|
||||
.sidebar-search input {
|
||||
width: 100%;
|
||||
font-size: 13px;
|
||||
padding: 6px 10px;
|
||||
}
|
||||
|
||||
.sidebar-controls {
|
||||
padding: 4px 16px 8px;
|
||||
}
|
||||
|
||||
.toggle-label {
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
cursor: pointer;
|
||||
}
|
||||
|
||||
.toggle-label input[type="checkbox"] {
|
||||
width: auto;
|
||||
accent-color: var(--accent);
|
||||
}
|
||||
|
||||
.sidebar-tree {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 4px 0;
|
||||
}
|
||||
|
||||
.loading-placeholder {
|
||||
padding: 16px;
|
||||
color: var(--text-muted);
|
||||
text-align: center;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.content {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.time-slider-container {
|
||||
padding: 8px 16px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
background: var(--bg-secondary);
|
||||
}
|
||||
|
||||
.tabs {
|
||||
display: flex;
|
||||
border-bottom: 1px solid var(--border);
|
||||
background: var(--bg-secondary);
|
||||
padding: 0 16px;
|
||||
}
|
||||
|
||||
.tab {
|
||||
padding: 10px 16px;
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
color: var(--text-muted);
|
||||
border-bottom: 2px solid transparent;
|
||||
transition: color 0.15s, border-color 0.15s;
|
||||
}
|
||||
|
||||
.tab:hover {
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.tab.active {
|
||||
color: var(--text);
|
||||
border-bottom-color: var(--accent);
|
||||
}
|
||||
|
||||
.file-list {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
padding: 4px 0;
|
||||
}
|
||||
|
||||
.file-row {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
width: 100%;
|
||||
padding: 8px 16px;
|
||||
text-align: left;
|
||||
transition: background 0.1s;
|
||||
}
|
||||
|
||||
.file-row:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
.file-row.deleted {
|
||||
opacity: 0.5;
|
||||
}
|
||||
|
||||
.file-row.deleted .file-path {
|
||||
text-decoration: line-through;
|
||||
}
|
||||
|
||||
.file-icon {
|
||||
font-size: 16px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.file-path {
|
||||
font-family: var(--mono);
|
||||
font-size: 13px;
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.file-meta {
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
white-space: nowrap;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
</style>
|
||||
288
frontend/history-ui/src/components/DiffView.svelte
Normal file
288
frontend/history-ui/src/components/DiffView.svelte
Normal file
|
|
@ -0,0 +1,288 @@
|
|||
<script lang="ts">
|
||||
interface Props {
|
||||
oldContent: string;
|
||||
newContent: string;
|
||||
oldLabel: string;
|
||||
newLabel: string;
|
||||
}
|
||||
|
||||
let { oldContent, newContent, oldLabel, newLabel }: Props = $props();
|
||||
|
||||
interface DiffLine {
|
||||
type: "add" | "remove" | "context";
|
||||
content: string;
|
||||
oldLineNo: number | null;
|
||||
newLineNo: number | null;
|
||||
}
|
||||
|
||||
let diffLines = $derived.by((): DiffLine[] => {
|
||||
const oldLines = oldContent.split("\n");
|
||||
const newLines = newContent.split("\n");
|
||||
|
||||
// Simple line-by-line diff using LCS
|
||||
const lines: DiffLine[] = [];
|
||||
const lcs = computeLCS(oldLines, newLines);
|
||||
|
||||
let oi = 0;
|
||||
let ni = 0;
|
||||
let oldLineNo = 1;
|
||||
let newLineNo = 1;
|
||||
|
||||
for (const match of lcs) {
|
||||
// Remove lines before match
|
||||
while (oi < match.oldIndex) {
|
||||
lines.push({
|
||||
type: "remove",
|
||||
content: oldLines[oi],
|
||||
oldLineNo: oldLineNo++,
|
||||
newLineNo: null
|
||||
});
|
||||
oi++;
|
||||
}
|
||||
// Add lines before match
|
||||
while (ni < match.newIndex) {
|
||||
lines.push({
|
||||
type: "add",
|
||||
content: newLines[ni],
|
||||
oldLineNo: null,
|
||||
newLineNo: newLineNo++
|
||||
});
|
||||
ni++;
|
||||
}
|
||||
// Context line
|
||||
lines.push({
|
||||
type: "context",
|
||||
content: oldLines[oi],
|
||||
oldLineNo: oldLineNo++,
|
||||
newLineNo: newLineNo++
|
||||
});
|
||||
oi++;
|
||||
ni++;
|
||||
}
|
||||
|
||||
// Remaining removes
|
||||
while (oi < oldLines.length) {
|
||||
lines.push({
|
||||
type: "remove",
|
||||
content: oldLines[oi],
|
||||
oldLineNo: oldLineNo++,
|
||||
newLineNo: null
|
||||
});
|
||||
oi++;
|
||||
}
|
||||
// Remaining adds
|
||||
while (ni < newLines.length) {
|
||||
lines.push({
|
||||
type: "add",
|
||||
content: newLines[ni],
|
||||
oldLineNo: null,
|
||||
newLineNo: newLineNo++
|
||||
});
|
||||
ni++;
|
||||
}
|
||||
|
||||
return lines;
|
||||
});
|
||||
|
||||
let stats = $derived({
|
||||
added: diffLines.filter((l) => l.type === "add").length,
|
||||
removed: diffLines.filter((l) => l.type === "remove").length
|
||||
});
|
||||
|
||||
interface LCSMatch {
|
||||
oldIndex: number;
|
||||
newIndex: number;
|
||||
}
|
||||
|
||||
function computeLCS(a: string[], b: string[]): LCSMatch[] {
|
||||
const m = a.length;
|
||||
const n = b.length;
|
||||
|
||||
// For large files, use a simpler approach
|
||||
if (m * n > 1_000_000) {
|
||||
return simpleDiff(a, b);
|
||||
}
|
||||
|
||||
const dp: number[][] = Array.from({ length: m + 1 }, () =>
|
||||
new Array(n + 1).fill(0)
|
||||
);
|
||||
|
||||
for (let i = 1; i <= m; i++) {
|
||||
for (let j = 1; j <= n; j++) {
|
||||
if (a[i - 1] === b[j - 1]) {
|
||||
dp[i][j] = dp[i - 1][j - 1] + 1;
|
||||
} else {
|
||||
dp[i][j] = Math.max(dp[i - 1][j], dp[i][j - 1]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Backtrack
|
||||
const matches: LCSMatch[] = [];
|
||||
let i = m;
|
||||
let j = n;
|
||||
while (i > 0 && j > 0) {
|
||||
if (a[i - 1] === b[j - 1]) {
|
||||
matches.unshift({ oldIndex: i - 1, newIndex: j - 1 });
|
||||
i--;
|
||||
j--;
|
||||
} else if (dp[i - 1][j] > dp[i][j - 1]) {
|
||||
i--;
|
||||
} else {
|
||||
j--;
|
||||
}
|
||||
}
|
||||
|
||||
return matches;
|
||||
}
|
||||
|
||||
function simpleDiff(a: string[], b: string[]): LCSMatch[] {
|
||||
// Hash-based matching for large files
|
||||
const bMap = new Map<string, number[]>();
|
||||
for (let j = 0; j < b.length; j++) {
|
||||
const arr = bMap.get(b[j]);
|
||||
if (arr) arr.push(j);
|
||||
else bMap.set(b[j], [j]);
|
||||
}
|
||||
|
||||
const matches: LCSMatch[] = [];
|
||||
let lastJ = -1;
|
||||
for (let i = 0; i < a.length; i++) {
|
||||
const candidates = bMap.get(a[i]);
|
||||
if (!candidates) continue;
|
||||
for (const j of candidates) {
|
||||
if (j > lastJ) {
|
||||
matches.push({ oldIndex: i, newIndex: j });
|
||||
lastJ = j;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return matches;
|
||||
}
|
||||
</script>
|
||||
|
||||
<div class="diff-view">
|
||||
<div class="diff-header">
|
||||
<span class="diff-label">{oldLabel}</span>
|
||||
<span class="diff-arrow">→</span>
|
||||
<span class="diff-label">{newLabel}</span>
|
||||
<span class="diff-stats">
|
||||
<span class="diff-added">+{stats.added}</span>
|
||||
<span class="diff-removed">-{stats.removed}</span>
|
||||
</span>
|
||||
</div>
|
||||
<div class="diff-content">
|
||||
{#each diffLines as line}
|
||||
<div class="diff-line {line.type}">
|
||||
<span class="line-no old-no">
|
||||
{line.oldLineNo ?? ""}
|
||||
</span>
|
||||
<span class="line-no new-no">
|
||||
{line.newLineNo ?? ""}
|
||||
</span>
|
||||
<span class="line-marker">
|
||||
{#if line.type === "add"}+{:else if line.type === "remove"}-{:else} {/if}
|
||||
</span>
|
||||
<span class="line-content">{line.content}</span>
|
||||
</div>
|
||||
{/each}
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.diff-view {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100%;
|
||||
}
|
||||
|
||||
.diff-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
padding: 8px 16px;
|
||||
background: var(--bg-secondary);
|
||||
border-bottom: 1px solid var(--border);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.diff-label {
|
||||
font-family: var(--mono);
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.diff-arrow {
|
||||
color: var(--text-subtle);
|
||||
}
|
||||
|
||||
.diff-stats {
|
||||
margin-left: auto;
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
font-family: var(--mono);
|
||||
font-size: 12px;
|
||||
}
|
||||
|
||||
.diff-added {
|
||||
color: var(--green);
|
||||
}
|
||||
|
||||
.diff-removed {
|
||||
color: var(--red);
|
||||
}
|
||||
|
||||
.diff-content {
|
||||
flex: 1;
|
||||
overflow: auto;
|
||||
font-family: var(--mono);
|
||||
font-size: 13px;
|
||||
line-height: 1.5;
|
||||
}
|
||||
|
||||
.diff-line {
|
||||
display: flex;
|
||||
white-space: pre;
|
||||
min-height: 20px;
|
||||
}
|
||||
|
||||
.diff-line.add {
|
||||
background: var(--green-bg);
|
||||
}
|
||||
|
||||
.diff-line.remove {
|
||||
background: var(--red-bg);
|
||||
}
|
||||
|
||||
.line-no {
|
||||
display: inline-block;
|
||||
width: 48px;
|
||||
text-align: right;
|
||||
padding-right: 8px;
|
||||
color: var(--text-subtle);
|
||||
user-select: none;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.line-marker {
|
||||
display: inline-block;
|
||||
width: 20px;
|
||||
text-align: center;
|
||||
flex-shrink: 0;
|
||||
user-select: none;
|
||||
}
|
||||
|
||||
.diff-line.add .line-marker {
|
||||
color: var(--green);
|
||||
}
|
||||
|
||||
.diff-line.remove .line-marker {
|
||||
color: var(--red);
|
||||
}
|
||||
|
||||
.line-content {
|
||||
flex: 1;
|
||||
padding-right: 16px;
|
||||
}
|
||||
</style>
|
||||
712
frontend/history-ui/src/components/DocumentDetail.svelte
Normal file
712
frontend/history-ui/src/components/DocumentDetail.svelte
Normal file
|
|
@ -0,0 +1,712 @@
|
|||
<script lang="ts">
|
||||
import {
|
||||
auth,
|
||||
toasts,
|
||||
relativeTime,
|
||||
absoluteTime,
|
||||
formatBytes,
|
||||
inferAction,
|
||||
isTextFile,
|
||||
isImageFile,
|
||||
fileExtension
|
||||
} from "../lib/stores.svelte";
|
||||
import type {
|
||||
DocumentVersionWithoutContent,
|
||||
DocumentVersion,
|
||||
ActionType
|
||||
} from "../lib/types";
|
||||
import DiffView from "./DiffView.svelte";
|
||||
import ConfirmDialog from "./ConfirmDialog.svelte";
|
||||
|
||||
interface Props {
|
||||
documentId: string;
|
||||
onClose: () => void;
|
||||
onRestore: () => void;
|
||||
}
|
||||
|
||||
let { documentId, onClose, onRestore }: Props = $props();
|
||||
|
||||
let versions = $state<DocumentVersionWithoutContent[]>([]);
|
||||
let loading = $state(true);
|
||||
let selectedVersion = $state<DocumentVersionWithoutContent | null>(null);
|
||||
let loadedContent = $state<string | null>(null);
|
||||
let loadedContentBytes = $state<ArrayBuffer | null>(null);
|
||||
let loadingContent = $state(false);
|
||||
let activeTab = $state<"preview" | "diff">("preview");
|
||||
|
||||
// Diff state
|
||||
let diffOldContent = $state<string | null>(null);
|
||||
let diffNewContent = $state<string | null>(null);
|
||||
let diffOldLabel = $state("");
|
||||
let diffNewLabel = $state("");
|
||||
|
||||
// Restore state
|
||||
let showRestoreDialog = $state(false);
|
||||
let restoreTarget = $state<DocumentVersionWithoutContent | null>(null);
|
||||
let restoring = $state(false);
|
||||
|
||||
let latest = $derived(versions.at(-1) ?? null);
|
||||
let isDeleted = $derived(latest?.isDeleted ?? false);
|
||||
let currentPath = $derived(latest?.relativePath ?? "");
|
||||
|
||||
// Derive action types
|
||||
let versionEvents = $derived(
|
||||
versions.map((v, i) => ({
|
||||
version: v,
|
||||
action: inferAction(v, i > 0 ? versions[i - 1] : undefined) as ActionType,
|
||||
previousPath: i > 0 && versions[i - 1].relativePath !== v.relativePath
|
||||
? versions[i - 1].relativePath
|
||||
: undefined
|
||||
}))
|
||||
);
|
||||
|
||||
async function loadVersions() {
|
||||
const api = auth.api;
|
||||
if (!api) return;
|
||||
loading = true;
|
||||
try {
|
||||
versions = await api.fetchDocumentVersions(documentId);
|
||||
// Auto-select latest
|
||||
if (versions.length > 0) {
|
||||
await selectVersion(versions.at(-1)!);
|
||||
}
|
||||
} catch {
|
||||
toasts.add("Failed to load document versions", "error");
|
||||
} finally {
|
||||
loading = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function selectVersion(v: DocumentVersionWithoutContent) {
|
||||
selectedVersion = v;
|
||||
activeTab = "preview";
|
||||
diffOldContent = null;
|
||||
diffNewContent = null;
|
||||
loadingContent = true;
|
||||
loadedContent = null;
|
||||
loadedContentBytes = null;
|
||||
|
||||
const api = auth.api;
|
||||
if (!api) return;
|
||||
|
||||
try {
|
||||
if (isTextFile(v.relativePath) || fileExtension(v.relativePath) === "") {
|
||||
const fullVersion = await api.fetchDocumentVersion(
|
||||
documentId,
|
||||
v.vaultUpdateId
|
||||
);
|
||||
const bytes = Uint8Array.from(atob(fullVersion.contentBase64), c => c.charCodeAt(0));
|
||||
const decoder = new TextDecoder("utf-8", { fatal: false });
|
||||
loadedContent = decoder.decode(bytes);
|
||||
loadedContentBytes = bytes.buffer;
|
||||
} else if (isImageFile(v.relativePath)) {
|
||||
loadedContentBytes = await api.fetchDocumentVersionContent(
|
||||
documentId,
|
||||
v.vaultUpdateId
|
||||
);
|
||||
} else {
|
||||
loadedContentBytes = await api.fetchDocumentVersionContent(
|
||||
documentId,
|
||||
v.vaultUpdateId
|
||||
);
|
||||
}
|
||||
} catch {
|
||||
toasts.add("Failed to load content", "error");
|
||||
} finally {
|
||||
loadingContent = false;
|
||||
}
|
||||
}
|
||||
|
||||
async function showDiff(v: DocumentVersionWithoutContent, idx: number) {
|
||||
const api = auth.api;
|
||||
if (!api || idx === 0) return;
|
||||
|
||||
activeTab = "diff";
|
||||
loadingContent = true;
|
||||
|
||||
const prev = versions[idx - 1];
|
||||
try {
|
||||
const [oldVer, newVer] = await Promise.all([
|
||||
api.fetchDocumentVersion(documentId, prev.vaultUpdateId),
|
||||
api.fetchDocumentVersion(documentId, v.vaultUpdateId)
|
||||
]);
|
||||
const decode = (b64: string) => {
|
||||
const bytes = Uint8Array.from(atob(b64), c => c.charCodeAt(0));
|
||||
return new TextDecoder("utf-8", { fatal: false }).decode(bytes);
|
||||
};
|
||||
diffOldContent = decode(oldVer.contentBase64);
|
||||
diffNewContent = decode(newVer.contentBase64);
|
||||
diffOldLabel = `v${prev.vaultUpdateId}`;
|
||||
diffNewLabel = `v${v.vaultUpdateId}`;
|
||||
} catch {
|
||||
toasts.add("Failed to load diff", "error");
|
||||
} finally {
|
||||
loadingContent = false;
|
||||
}
|
||||
}
|
||||
|
||||
function confirmRestore(v: DocumentVersionWithoutContent) {
|
||||
restoreTarget = v;
|
||||
showRestoreDialog = true;
|
||||
}
|
||||
|
||||
async function executeRestore() {
|
||||
const api = auth.api;
|
||||
if (!api || !restoreTarget) return;
|
||||
restoring = true;
|
||||
try {
|
||||
await api.restoreVersion(
|
||||
documentId,
|
||||
restoreTarget.vaultUpdateId
|
||||
);
|
||||
toasts.add(
|
||||
`Restored to version #${restoreTarget.vaultUpdateId}`,
|
||||
"success"
|
||||
);
|
||||
showRestoreDialog = false;
|
||||
restoreTarget = null;
|
||||
onRestore();
|
||||
await loadVersions();
|
||||
} catch (e) {
|
||||
toasts.add(`Restore failed: ${e}`, "error");
|
||||
} finally {
|
||||
restoring = false;
|
||||
}
|
||||
}
|
||||
|
||||
function getImageUrl(buffer: ArrayBuffer, path: string): string {
|
||||
const ext = fileExtension(path);
|
||||
const mimeMap: Record<string, string> = {
|
||||
png: "image/png",
|
||||
jpg: "image/jpeg",
|
||||
jpeg: "image/jpeg",
|
||||
gif: "image/gif",
|
||||
webp: "image/webp",
|
||||
svg: "image/svg+xml",
|
||||
ico: "image/x-icon",
|
||||
bmp: "image/bmp"
|
||||
};
|
||||
const mime = mimeMap[ext] ?? "application/octet-stream";
|
||||
const blob = new Blob([buffer], { type: mime });
|
||||
return URL.createObjectURL(blob);
|
||||
}
|
||||
|
||||
$effect(() => {
|
||||
loadVersions();
|
||||
});
|
||||
|
||||
const actionColors: Record<string, string> = {
|
||||
created: "var(--green)",
|
||||
updated: "var(--blue)",
|
||||
renamed: "var(--orange)",
|
||||
deleted: "var(--red)",
|
||||
restored: "var(--purple)"
|
||||
};
|
||||
|
||||
const actionBgColors: Record<string, string> = {
|
||||
created: "var(--green-bg)",
|
||||
updated: "var(--blue-bg)",
|
||||
renamed: "var(--orange-bg)",
|
||||
deleted: "var(--red-bg)",
|
||||
restored: "var(--purple-bg)"
|
||||
};
|
||||
</script>
|
||||
|
||||
<div class="detail">
|
||||
<!-- Header -->
|
||||
<div class="detail-header">
|
||||
<button class="back-btn" onclick={onClose} title="Back">
|
||||
<svg width="16" height="16" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
|
||||
<path d="M19 12H5M12 19l-7-7 7-7" />
|
||||
</svg>
|
||||
</button>
|
||||
<div class="header-info">
|
||||
<div class="header-path">
|
||||
<span class="path-text" class:deleted-path={isDeleted}>
|
||||
{currentPath}
|
||||
</span>
|
||||
{#if isDeleted}
|
||||
<span class="status-badge deleted-badge">Deleted</span>
|
||||
{:else}
|
||||
<span class="status-badge active-badge">Active</span>
|
||||
{/if}
|
||||
</div>
|
||||
<div class="header-meta">
|
||||
<span class="doc-id" title={documentId}>
|
||||
{documentId.substring(0, 8)}...
|
||||
</span>
|
||||
{#if latest}
|
||||
<span>·</span>
|
||||
<span>{versions.length} version{versions.length !== 1 ? "s" : ""}</span>
|
||||
<span>·</span>
|
||||
<span>Last by {latest.userId}</span>
|
||||
{/if}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
{#if loading}
|
||||
<div class="detail-loading">Loading versions...</div>
|
||||
{:else}
|
||||
<!-- Content area -->
|
||||
<div class="detail-body">
|
||||
<div class="content-panel">
|
||||
{#if selectedVersion}
|
||||
<div class="content-tabs">
|
||||
<button
|
||||
class="content-tab"
|
||||
class:active={activeTab === "preview"}
|
||||
onclick={() => (activeTab = "preview")}
|
||||
>
|
||||
Preview
|
||||
</button>
|
||||
<button
|
||||
class="content-tab"
|
||||
class:active={activeTab === "diff"}
|
||||
onclick={() => {
|
||||
if (selectedVersion) {
|
||||
const idx = versions.indexOf(selectedVersion);
|
||||
if (idx > 0) showDiff(selectedVersion, idx);
|
||||
}
|
||||
}}
|
||||
disabled={versions.indexOf(selectedVersion) === 0}
|
||||
>
|
||||
Diff
|
||||
</button>
|
||||
<div class="content-tab-spacer"></div>
|
||||
<span class="viewing-label">
|
||||
Viewing v#{selectedVersion.vaultUpdateId}
|
||||
·
|
||||
{relativeTime(selectedVersion.updatedDate)}
|
||||
</span>
|
||||
</div>
|
||||
|
||||
<div class="content-view">
|
||||
{#if loadingContent}
|
||||
<div class="content-loading">Loading content...</div>
|
||||
{:else if activeTab === "diff" && diffOldContent !== null && diffNewContent !== null}
|
||||
<DiffView
|
||||
oldContent={diffOldContent}
|
||||
newContent={diffNewContent}
|
||||
oldLabel={diffOldLabel}
|
||||
newLabel={diffNewLabel}
|
||||
/>
|
||||
{:else if activeTab === "preview"}
|
||||
{#if isTextFile(selectedVersion.relativePath) || fileExtension(selectedVersion.relativePath) === ""}
|
||||
<pre class="text-content">{loadedContent ?? ""}</pre>
|
||||
{:else if isImageFile(selectedVersion.relativePath) && loadedContentBytes}
|
||||
<div class="image-preview">
|
||||
<img
|
||||
src={getImageUrl(loadedContentBytes, selectedVersion.relativePath)}
|
||||
alt={selectedVersion.relativePath}
|
||||
/>
|
||||
</div>
|
||||
{:else}
|
||||
<div class="binary-placeholder">
|
||||
<div class="binary-icon">📦</div>
|
||||
<div class="binary-label">Binary file</div>
|
||||
<div class="binary-size">
|
||||
{formatBytes(selectedVersion.contentSize)}
|
||||
</div>
|
||||
</div>
|
||||
{/if}
|
||||
{/if}
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<!-- Version timeline -->
|
||||
<div class="version-panel">
|
||||
<div class="version-panel-header">Version History</div>
|
||||
<div class="version-list">
|
||||
{#each [...versionEvents].reverse() as event, i}
|
||||
{@const v = event.version}
|
||||
{@const isSelected = selectedVersion?.vaultUpdateId === v.vaultUpdateId}
|
||||
<div class="version-item" class:selected={isSelected}>
|
||||
<button
|
||||
class="version-main"
|
||||
onclick={() => selectVersion(v)}
|
||||
>
|
||||
<div class="version-left">
|
||||
<span class="version-id">#{v.vaultUpdateId}</span>
|
||||
<span
|
||||
class="version-action"
|
||||
style="color: {actionColors[event.action]}; background: {actionBgColors[event.action]}"
|
||||
>
|
||||
{event.action}
|
||||
</span>
|
||||
</div>
|
||||
<div class="version-right">
|
||||
<span class="version-user">{v.userId}</span>
|
||||
<span
|
||||
class="version-time"
|
||||
title={absoluteTime(v.updatedDate)}
|
||||
>
|
||||
{relativeTime(v.updatedDate)}
|
||||
</span>
|
||||
<span class="version-size">{formatBytes(v.contentSize)}</span>
|
||||
</div>
|
||||
</button>
|
||||
{#if event.previousPath}
|
||||
<div class="version-rename">
|
||||
{event.previousPath} → {v.relativePath}
|
||||
</div>
|
||||
{/if}
|
||||
<div class="version-actions">
|
||||
{#if i < versionEvents.length - 1}
|
||||
<button
|
||||
class="version-btn"
|
||||
onclick={() => {
|
||||
const realIdx = versions.indexOf(v);
|
||||
showDiff(v, realIdx);
|
||||
}}
|
||||
>
|
||||
Diff
|
||||
</button>
|
||||
{/if}
|
||||
{#if v !== latest}
|
||||
<button
|
||||
class="version-btn restore-btn"
|
||||
onclick={() => confirmRestore(v)}
|
||||
>
|
||||
Restore
|
||||
</button>
|
||||
{/if}
|
||||
</div>
|
||||
</div>
|
||||
{/each}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
{#if showRestoreDialog && restoreTarget}
|
||||
<ConfirmDialog
|
||||
title="Restore Version"
|
||||
message={`Restore "${currentPath}" to version #${restoreTarget.vaultUpdateId} from ${absoluteTime(restoreTarget.updatedDate)}? This creates a new version with the old content. Current content is preserved in history.`}
|
||||
confirmLabel="Restore"
|
||||
destructive={false}
|
||||
loading={restoring}
|
||||
onConfirm={executeRestore}
|
||||
onCancel={() => {
|
||||
showRestoreDialog = false;
|
||||
restoreTarget = null;
|
||||
}}
|
||||
/>
|
||||
{/if}
|
||||
|
||||
<style>
|
||||
.detail {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
height: 100%;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.detail-header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
padding: 12px 16px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
background: var(--bg-secondary);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.back-btn {
|
||||
padding: 6px;
|
||||
border-radius: var(--radius-sm);
|
||||
color: var(--text-muted);
|
||||
transition: color 0.15s, background 0.15s;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.back-btn:hover {
|
||||
color: var(--text);
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
.header-info {
|
||||
flex: 1;
|
||||
min-width: 0;
|
||||
}
|
||||
|
||||
.header-path {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.path-text {
|
||||
font-family: var(--mono);
|
||||
font-size: 15px;
|
||||
font-weight: 600;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.deleted-path {
|
||||
text-decoration: line-through;
|
||||
opacity: 0.6;
|
||||
}
|
||||
|
||||
.status-badge {
|
||||
font-size: 10px;
|
||||
font-weight: 600;
|
||||
padding: 1px 8px;
|
||||
border-radius: 10px;
|
||||
text-transform: uppercase;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.active-badge {
|
||||
color: var(--green);
|
||||
background: var(--green-bg);
|
||||
}
|
||||
|
||||
.deleted-badge {
|
||||
color: var(--red);
|
||||
background: var(--red-bg);
|
||||
}
|
||||
|
||||
.header-meta {
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
margin-top: 2px;
|
||||
display: flex;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
.doc-id {
|
||||
font-family: var(--mono);
|
||||
cursor: help;
|
||||
}
|
||||
|
||||
.detail-loading {
|
||||
padding: 48px;
|
||||
text-align: center;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.detail-body {
|
||||
display: flex;
|
||||
flex: 1;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.content-panel {
|
||||
flex: 1;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.content-tabs {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
padding: 0 16px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
background: var(--bg);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.content-tab {
|
||||
padding: 8px 12px;
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
color: var(--text-muted);
|
||||
border-bottom: 2px solid transparent;
|
||||
transition: color 0.15s, border-color 0.15s;
|
||||
}
|
||||
|
||||
.content-tab:hover:not(:disabled) {
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.content-tab.active {
|
||||
color: var(--text);
|
||||
border-bottom-color: var(--accent);
|
||||
}
|
||||
|
||||
.content-tab:disabled {
|
||||
opacity: 0.3;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.content-tab-spacer {
|
||||
flex: 1;
|
||||
}
|
||||
|
||||
.viewing-label {
|
||||
font-size: 12px;
|
||||
color: var(--text-subtle);
|
||||
font-family: var(--mono);
|
||||
}
|
||||
|
||||
.content-view {
|
||||
flex: 1;
|
||||
overflow: auto;
|
||||
}
|
||||
|
||||
.content-loading {
|
||||
padding: 48px;
|
||||
text-align: center;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.text-content {
|
||||
padding: 16px;
|
||||
font-family: var(--mono);
|
||||
font-size: 13px;
|
||||
line-height: 1.6;
|
||||
white-space: pre-wrap;
|
||||
word-break: break-word;
|
||||
tab-size: 4;
|
||||
}
|
||||
|
||||
.image-preview {
|
||||
padding: 16px;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
}
|
||||
|
||||
.image-preview img {
|
||||
max-width: 100%;
|
||||
max-height: 60vh;
|
||||
border-radius: var(--radius);
|
||||
border: 1px solid var(--border);
|
||||
}
|
||||
|
||||
.binary-placeholder {
|
||||
padding: 64px;
|
||||
text-align: center;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.binary-icon {
|
||||
font-size: 48px;
|
||||
margin-bottom: 12px;
|
||||
}
|
||||
|
||||
.binary-label {
|
||||
font-size: 16px;
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
.binary-size {
|
||||
font-size: 14px;
|
||||
margin-top: 4px;
|
||||
}
|
||||
|
||||
/* Version panel */
|
||||
.version-panel {
|
||||
width: 320px;
|
||||
min-width: 320px;
|
||||
border-left: 1px solid var(--border);
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
overflow: hidden;
|
||||
background: var(--bg-secondary);
|
||||
}
|
||||
|
||||
.version-panel-header {
|
||||
padding: 10px 16px;
|
||||
font-size: 12px;
|
||||
font-weight: 600;
|
||||
color: var(--text-muted);
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
border-bottom: 1px solid var(--border);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.version-list {
|
||||
flex: 1;
|
||||
overflow-y: auto;
|
||||
}
|
||||
|
||||
.version-item {
|
||||
border-bottom: 1px solid var(--border-light);
|
||||
padding: 8px 12px;
|
||||
transition: background 0.1s;
|
||||
}
|
||||
|
||||
.version-item:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
.version-item.selected {
|
||||
background: var(--blue-bg);
|
||||
}
|
||||
|
||||
.version-main {
|
||||
width: 100%;
|
||||
text-align: left;
|
||||
}
|
||||
|
||||
.version-left {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
margin-bottom: 4px;
|
||||
}
|
||||
|
||||
.version-id {
|
||||
font-family: var(--mono);
|
||||
font-size: 13px;
|
||||
font-weight: 600;
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.version-action {
|
||||
font-size: 10px;
|
||||
font-weight: 600;
|
||||
padding: 0 6px;
|
||||
border-radius: 8px;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.version-right {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
font-size: 11px;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.version-rename {
|
||||
font-size: 11px;
|
||||
color: var(--orange);
|
||||
font-family: var(--mono);
|
||||
margin: 4px 0;
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
white-space: nowrap;
|
||||
}
|
||||
|
||||
.version-actions {
|
||||
display: flex;
|
||||
gap: 8px;
|
||||
margin-top: 6px;
|
||||
}
|
||||
|
||||
.version-btn {
|
||||
font-size: 11px;
|
||||
color: var(--accent);
|
||||
padding: 2px 8px;
|
||||
border: 1px solid var(--border);
|
||||
border-radius: var(--radius-sm);
|
||||
transition: background 0.15s;
|
||||
}
|
||||
|
||||
.version-btn:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
.restore-btn {
|
||||
color: var(--orange);
|
||||
}
|
||||
</style>
|
||||
124
frontend/history-ui/src/components/FileTree.svelte
Normal file
124
frontend/history-ui/src/components/FileTree.svelte
Normal file
|
|
@ -0,0 +1,124 @@
|
|||
<script lang="ts">
|
||||
import type { TreeNode } from "../lib/types";
|
||||
import FileTree from "./FileTree.svelte";
|
||||
|
||||
interface Props {
|
||||
node: TreeNode;
|
||||
selectedId: string | null;
|
||||
onSelect: (documentId: string) => void;
|
||||
depth?: number;
|
||||
}
|
||||
|
||||
let { node, selectedId, onSelect, depth = 0 }: Props = $props();
|
||||
|
||||
let expanded = $state<Record<string, boolean>>({});
|
||||
|
||||
function toggle(path: string) {
|
||||
expanded[path] = !expanded[path];
|
||||
}
|
||||
|
||||
function isExpanded(path: string): boolean {
|
||||
return expanded[path] ?? true;
|
||||
}
|
||||
</script>
|
||||
|
||||
{#if node.isFolder && depth === 0}
|
||||
{#each node.children as child}
|
||||
<FileTree
|
||||
node={child}
|
||||
{selectedId}
|
||||
{onSelect}
|
||||
depth={depth + 1}
|
||||
/>
|
||||
{/each}
|
||||
{:else if node.isFolder}
|
||||
<div class="tree-folder">
|
||||
<button
|
||||
class="tree-item folder"
|
||||
style="padding-left: {depth * 16}px"
|
||||
onclick={() => toggle(node.path)}
|
||||
>
|
||||
<span class="expand-icon"
|
||||
>{isExpanded(node.path) ? "▾" : "▸"}</span
|
||||
>
|
||||
<span class="folder-icon">📁</span>
|
||||
<span class="node-name">{node.name}</span>
|
||||
</button>
|
||||
{#if isExpanded(node.path)}
|
||||
{#each node.children as child}
|
||||
<FileTree
|
||||
node={child}
|
||||
{selectedId}
|
||||
{onSelect}
|
||||
depth={depth + 1}
|
||||
/>
|
||||
{/each}
|
||||
{/if}
|
||||
</div>
|
||||
{:else}
|
||||
<button
|
||||
class="tree-item file"
|
||||
class:selected={node.document?.documentId === selectedId}
|
||||
class:deleted={node.isDeleted}
|
||||
style="padding-left: {depth * 16 + 8}px"
|
||||
onclick={() =>
|
||||
node.document && onSelect(node.document.documentId)}
|
||||
>
|
||||
<span class="file-icon">{node.isDeleted ? "○" : "●"}</span>
|
||||
<span class="node-name">{node.name}</span>
|
||||
</button>
|
||||
{/if}
|
||||
|
||||
<style>
|
||||
.tree-item {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
width: 100%;
|
||||
padding: 3px 12px;
|
||||
font-size: 13px;
|
||||
text-align: left;
|
||||
transition: background 0.1s;
|
||||
white-space: nowrap;
|
||||
overflow: hidden;
|
||||
}
|
||||
|
||||
.tree-item:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
|
||||
.tree-item.selected {
|
||||
background: var(--blue-bg);
|
||||
}
|
||||
|
||||
.tree-item.deleted {
|
||||
opacity: 0.4;
|
||||
}
|
||||
|
||||
.tree-item.deleted .node-name {
|
||||
text-decoration: line-through;
|
||||
}
|
||||
|
||||
.expand-icon {
|
||||
font-size: 10px;
|
||||
width: 12px;
|
||||
flex-shrink: 0;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
.folder-icon {
|
||||
font-size: 14px;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.file-icon {
|
||||
font-size: 8px;
|
||||
flex-shrink: 0;
|
||||
color: var(--text-subtle);
|
||||
}
|
||||
|
||||
.node-name {
|
||||
overflow: hidden;
|
||||
text-overflow: ellipsis;
|
||||
}
|
||||
</style>
|
||||
126
frontend/history-ui/src/components/Header.svelte
Normal file
126
frontend/history-ui/src/components/Header.svelte
Normal file
|
|
@ -0,0 +1,126 @@
|
|||
<script lang="ts">
|
||||
import { auth } from "../lib/stores.svelte";
|
||||
|
||||
interface Props {
|
||||
vaultId: string;
|
||||
serverVersion: string;
|
||||
onRefresh: () => void;
|
||||
}
|
||||
|
||||
let { vaultId, serverVersion, onRefresh }: Props = $props();
|
||||
</script>
|
||||
|
||||
<header class="header">
|
||||
<div class="header-left">
|
||||
<svg
|
||||
width="20"
|
||||
height="20"
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
<path d="M12 2L2 7l10 5 10-5-10-5z" />
|
||||
<path d="M2 17l10 5 10-5" />
|
||||
<path d="M2 12l10 5 10-5" />
|
||||
</svg>
|
||||
<span class="header-title">VaultLink</span>
|
||||
<span class="header-sep">/</span>
|
||||
<span class="header-vault">{vaultId}</span>
|
||||
</div>
|
||||
|
||||
<div class="header-right">
|
||||
<span class="server-version">v{serverVersion}</span>
|
||||
<button class="header-btn" onclick={onRefresh} title="Refresh">
|
||||
<svg
|
||||
width="16"
|
||||
height="16"
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
<path
|
||||
d="M21 2v6h-6M3 12a9 9 0 0 1 15-6.7L21 8M3 22v-6h6M21 12a9 9 0 0 1-15 6.7L3 16"
|
||||
/>
|
||||
</svg>
|
||||
</button>
|
||||
<button
|
||||
class="header-btn"
|
||||
onclick={() => auth.logout()}
|
||||
title="Disconnect"
|
||||
>
|
||||
<svg
|
||||
width="16"
|
||||
height="16"
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
<path d="M9 21H5a2 2 0 0 1-2-2V5a2 2 0 0 1 2-2h4" />
|
||||
<polyline points="16 17 21 12 16 7" />
|
||||
<line x1="21" y1="12" x2="9" y2="12" />
|
||||
</svg>
|
||||
</button>
|
||||
</div>
|
||||
</header>
|
||||
|
||||
<style>
|
||||
.header {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: space-between;
|
||||
height: 48px;
|
||||
padding: 0 16px;
|
||||
background: var(--bg-secondary);
|
||||
border-bottom: 1px solid var(--border);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.header-left {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.header-title {
|
||||
font-weight: 600;
|
||||
font-size: 15px;
|
||||
}
|
||||
|
||||
.header-sep {
|
||||
color: var(--text-subtle);
|
||||
}
|
||||
|
||||
.header-vault {
|
||||
font-family: var(--mono);
|
||||
font-size: 14px;
|
||||
color: var(--accent);
|
||||
}
|
||||
|
||||
.header-right {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.server-version {
|
||||
font-size: 12px;
|
||||
color: var(--text-subtle);
|
||||
font-family: var(--mono);
|
||||
}
|
||||
|
||||
.header-btn {
|
||||
padding: 6px;
|
||||
border-radius: var(--radius-sm);
|
||||
color: var(--text-muted);
|
||||
transition: color 0.15s, background 0.15s;
|
||||
}
|
||||
|
||||
.header-btn:hover {
|
||||
color: var(--text);
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
</style>
|
||||
194
frontend/history-ui/src/components/Login.svelte
Normal file
194
frontend/history-ui/src/components/Login.svelte
Normal file
|
|
@ -0,0 +1,194 @@
|
|||
<script lang="ts">
|
||||
import { auth, toasts } from "../lib/stores.svelte";
|
||||
import { ApiClient } from "../lib/api";
|
||||
|
||||
let vaultId = $state("");
|
||||
let token = $state("");
|
||||
let error = $state("");
|
||||
let loading = $state(false);
|
||||
|
||||
async function handleSubmit(e: Event) {
|
||||
e.preventDefault();
|
||||
if (!vaultId.trim() || !token.trim()) {
|
||||
error = "Both fields are required.";
|
||||
return;
|
||||
}
|
||||
error = "";
|
||||
loading = true;
|
||||
try {
|
||||
const client = new ApiClient(vaultId.trim(), token.trim());
|
||||
const ping = await client.ping();
|
||||
if (ping.isAuthenticated) {
|
||||
auth.login(
|
||||
vaultId.trim(),
|
||||
token.trim(),
|
||||
ping.serverVersion
|
||||
);
|
||||
} else {
|
||||
error = "Authentication failed. Check your token.";
|
||||
}
|
||||
} catch (err) {
|
||||
error =
|
||||
"Cannot reach server. Check the vault name and your network.";
|
||||
} finally {
|
||||
loading = false;
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<div class="login-page">
|
||||
<div class="login-card">
|
||||
<div class="logo">
|
||||
<svg width="32" height="32" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2">
|
||||
<path d="M12 2L2 7l10 5 10-5-10-5z"/>
|
||||
<path d="M2 17l10 5 10-5"/>
|
||||
<path d="M2 12l10 5 10-5"/>
|
||||
</svg>
|
||||
<h1>VaultLink</h1>
|
||||
</div>
|
||||
<p class="subtitle">Vault History Browser</p>
|
||||
|
||||
<form onsubmit={handleSubmit}>
|
||||
<label>
|
||||
<span>Vault name</span>
|
||||
<input
|
||||
type="text"
|
||||
bind:value={vaultId}
|
||||
placeholder="my-vault"
|
||||
autocomplete="off"
|
||||
disabled={loading}
|
||||
/>
|
||||
</label>
|
||||
|
||||
<label>
|
||||
<span>Token</span>
|
||||
<input
|
||||
type="password"
|
||||
bind:value={token}
|
||||
placeholder="Enter your access token"
|
||||
disabled={loading}
|
||||
/>
|
||||
</label>
|
||||
|
||||
{#if error}
|
||||
<div class="error">{error}</div>
|
||||
{/if}
|
||||
|
||||
<button type="submit" class="btn-primary" disabled={loading}>
|
||||
{#if loading}
|
||||
<span class="btn-spinner"></span>
|
||||
Connecting...
|
||||
{:else}
|
||||
Connect
|
||||
{/if}
|
||||
</button>
|
||||
</form>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.login-page {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
height: 100%;
|
||||
background: var(--bg);
|
||||
}
|
||||
|
||||
.login-card {
|
||||
width: 100%;
|
||||
max-width: 400px;
|
||||
padding: 40px;
|
||||
background: var(--bg-secondary);
|
||||
border: 1px solid var(--border);
|
||||
border-radius: 12px;
|
||||
box-shadow: var(--shadow);
|
||||
}
|
||||
|
||||
.logo {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
margin-bottom: 4px;
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
.logo h1 {
|
||||
font-size: 24px;
|
||||
font-weight: 600;
|
||||
}
|
||||
|
||||
.subtitle {
|
||||
color: var(--text-muted);
|
||||
margin-bottom: 32px;
|
||||
font-size: 14px;
|
||||
}
|
||||
|
||||
form {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 20px;
|
||||
}
|
||||
|
||||
label {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 6px;
|
||||
}
|
||||
|
||||
label span {
|
||||
font-size: 13px;
|
||||
font-weight: 500;
|
||||
color: var(--text-muted);
|
||||
}
|
||||
|
||||
input {
|
||||
width: 100%;
|
||||
}
|
||||
|
||||
.error {
|
||||
color: var(--red);
|
||||
font-size: 13px;
|
||||
padding: 8px 12px;
|
||||
background: var(--red-bg);
|
||||
border-radius: var(--radius-sm);
|
||||
}
|
||||
|
||||
.btn-primary {
|
||||
width: 100%;
|
||||
padding: 10px 16px;
|
||||
background: var(--accent);
|
||||
color: #fff;
|
||||
font-weight: 600;
|
||||
border-radius: var(--radius);
|
||||
transition: background 0.15s;
|
||||
display: flex;
|
||||
align-items: center;
|
||||
justify-content: center;
|
||||
gap: 8px;
|
||||
}
|
||||
|
||||
.btn-primary:hover:not(:disabled) {
|
||||
background: var(--accent-hover);
|
||||
}
|
||||
|
||||
.btn-primary:disabled {
|
||||
opacity: 0.6;
|
||||
cursor: not-allowed;
|
||||
}
|
||||
|
||||
.btn-spinner {
|
||||
width: 16px;
|
||||
height: 16px;
|
||||
border: 2px solid rgba(255, 255, 255, 0.3);
|
||||
border-top-color: #fff;
|
||||
border-radius: 50%;
|
||||
animation: spin 0.6s linear infinite;
|
||||
}
|
||||
|
||||
@keyframes spin {
|
||||
to {
|
||||
transform: rotate(360deg);
|
||||
}
|
||||
}
|
||||
</style>
|
||||
191
frontend/history-ui/src/components/TimeSlider.svelte
Normal file
191
frontend/history-ui/src/components/TimeSlider.svelte
Normal file
|
|
@ -0,0 +1,191 @@
|
|||
<script lang="ts">
|
||||
import type { DocumentVersionWithoutContent } from "../lib/types";
|
||||
import { relativeTime, absoluteTime } from "../lib/stores.svelte";
|
||||
|
||||
interface Props {
|
||||
min: number;
|
||||
max: number;
|
||||
value: number | null;
|
||||
versions: DocumentVersionWithoutContent[];
|
||||
onchange: (value: number | null) => void;
|
||||
}
|
||||
|
||||
let { min, max, value, versions, onchange }: Props = $props();
|
||||
|
||||
let isNow = $derived(value === null || value >= max);
|
||||
|
||||
function handleInput(e: Event) {
|
||||
const target = e.target as HTMLInputElement;
|
||||
const v = parseInt(target.value, 10);
|
||||
if (v >= max) {
|
||||
onchange(null);
|
||||
} else {
|
||||
onchange(v);
|
||||
}
|
||||
}
|
||||
|
||||
function snapToNow() {
|
||||
onchange(null);
|
||||
}
|
||||
|
||||
let currentVersion = $derived(
|
||||
value !== null
|
||||
? versions.find((v) => v.vaultUpdateId === value) ??
|
||||
versions.reduce(
|
||||
(closest, v) =>
|
||||
Math.abs(v.vaultUpdateId - (value ?? max)) <
|
||||
Math.abs(
|
||||
closest.vaultUpdateId - (value ?? max)
|
||||
)
|
||||
? v
|
||||
: closest,
|
||||
versions[0]
|
||||
)
|
||||
: null
|
||||
);
|
||||
</script>
|
||||
|
||||
<div class="time-slider">
|
||||
<div class="slider-label">
|
||||
<svg
|
||||
width="14"
|
||||
height="14"
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
<circle cx="12" cy="12" r="10" />
|
||||
<polyline points="12 6 12 12 16 14" />
|
||||
</svg>
|
||||
<span class="label-text">Time Travel</span>
|
||||
</div>
|
||||
|
||||
<div class="slider-track">
|
||||
<input
|
||||
type="range"
|
||||
min={min}
|
||||
max={max}
|
||||
value={value ?? max}
|
||||
oninput={handleInput}
|
||||
/>
|
||||
</div>
|
||||
|
||||
<div class="slider-info">
|
||||
{#if isNow}
|
||||
<span class="now-badge">Now</span>
|
||||
{:else if currentVersion}
|
||||
<span
|
||||
class="time-info"
|
||||
title={absoluteTime(currentVersion.updatedDate)}
|
||||
>
|
||||
#{value}
|
||||
·
|
||||
{relativeTime(currentVersion.updatedDate)}
|
||||
</span>
|
||||
{:else}
|
||||
<span class="time-info">#{value}</span>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
{#if !isNow}
|
||||
<button class="snap-btn" onclick={snapToNow} title="Back to now">
|
||||
<svg
|
||||
width="14"
|
||||
height="14"
|
||||
viewBox="0 0 24 24"
|
||||
fill="none"
|
||||
stroke="currentColor"
|
||||
stroke-width="2"
|
||||
>
|
||||
<path d="M5 12h14M12 5l7 7-7 7" />
|
||||
</svg>
|
||||
</button>
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<style>
|
||||
.time-slider {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
}
|
||||
|
||||
.slider-label {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 6px;
|
||||
color: var(--text-muted);
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.label-text {
|
||||
font-size: 12px;
|
||||
font-weight: 500;
|
||||
text-transform: uppercase;
|
||||
letter-spacing: 0.5px;
|
||||
}
|
||||
|
||||
.slider-track {
|
||||
flex: 1;
|
||||
min-width: 120px;
|
||||
}
|
||||
|
||||
.slider-track input[type="range"] {
|
||||
width: 100%;
|
||||
height: 4px;
|
||||
appearance: none;
|
||||
background: var(--bg-tertiary);
|
||||
border-radius: 2px;
|
||||
outline: none;
|
||||
border: none;
|
||||
padding: 0;
|
||||
}
|
||||
|
||||
.slider-track input[type="range"]::-webkit-slider-thumb {
|
||||
appearance: none;
|
||||
width: 14px;
|
||||
height: 14px;
|
||||
background: var(--accent);
|
||||
border-radius: 50%;
|
||||
cursor: pointer;
|
||||
transition: transform 0.1s;
|
||||
}
|
||||
|
||||
.slider-track input[type="range"]::-webkit-slider-thumb:hover {
|
||||
transform: scale(1.2);
|
||||
}
|
||||
|
||||
.slider-info {
|
||||
flex-shrink: 0;
|
||||
min-width: 100px;
|
||||
}
|
||||
|
||||
.now-badge {
|
||||
font-size: 11px;
|
||||
font-weight: 600;
|
||||
color: var(--green);
|
||||
background: var(--green-bg);
|
||||
padding: 2px 10px;
|
||||
border-radius: 10px;
|
||||
text-transform: uppercase;
|
||||
}
|
||||
|
||||
.time-info {
|
||||
font-size: 12px;
|
||||
color: var(--text-muted);
|
||||
font-family: var(--mono);
|
||||
}
|
||||
|
||||
.snap-btn {
|
||||
padding: 4px;
|
||||
color: var(--accent);
|
||||
border-radius: var(--radius-sm);
|
||||
transition: background 0.15s;
|
||||
flex-shrink: 0;
|
||||
}
|
||||
|
||||
.snap-btn:hover {
|
||||
background: var(--bg-hover);
|
||||
}
|
||||
</style>
|
||||
80
frontend/history-ui/src/components/ToastContainer.svelte
Normal file
80
frontend/history-ui/src/components/ToastContainer.svelte
Normal file
|
|
@ -0,0 +1,80 @@
|
|||
<script lang="ts">
|
||||
import { toasts } from "../lib/stores.svelte";
|
||||
|
||||
const typeColors: Record<string, string> = {
|
||||
success: "var(--green)",
|
||||
error: "var(--red)",
|
||||
info: "var(--accent)"
|
||||
};
|
||||
</script>
|
||||
|
||||
{#if toasts.items.length > 0}
|
||||
<div class="toast-container">
|
||||
{#each toasts.items as toast (toast.id)}
|
||||
<div
|
||||
class="toast"
|
||||
style="border-left-color: {typeColors[toast.type]}"
|
||||
>
|
||||
<span class="toast-message">{toast.message}</span>
|
||||
<button
|
||||
class="toast-dismiss"
|
||||
onclick={() => toasts.dismiss(toast.id)}
|
||||
>
|
||||
×
|
||||
</button>
|
||||
</div>
|
||||
{/each}
|
||||
</div>
|
||||
{/if}
|
||||
|
||||
<style>
|
||||
.toast-container {
|
||||
position: fixed;
|
||||
bottom: 16px;
|
||||
right: 16px;
|
||||
z-index: 9999;
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
gap: 8px;
|
||||
max-width: 400px;
|
||||
}
|
||||
|
||||
.toast {
|
||||
display: flex;
|
||||
align-items: center;
|
||||
gap: 12px;
|
||||
padding: 12px 16px;
|
||||
background: var(--bg-secondary);
|
||||
border: 1px solid var(--border);
|
||||
border-left-width: 3px;
|
||||
border-radius: var(--radius);
|
||||
box-shadow: var(--shadow);
|
||||
animation: slide-in 0.2s ease-out;
|
||||
}
|
||||
|
||||
.toast-message {
|
||||
flex: 1;
|
||||
font-size: 13px;
|
||||
}
|
||||
|
||||
.toast-dismiss {
|
||||
font-size: 18px;
|
||||
color: var(--text-muted);
|
||||
padding: 0 4px;
|
||||
}
|
||||
|
||||
.toast-dismiss:hover {
|
||||
color: var(--text);
|
||||
}
|
||||
|
||||
@keyframes slide-in {
|
||||
from {
|
||||
transform: translateX(100%);
|
||||
opacity: 0;
|
||||
}
|
||||
to {
|
||||
transform: translateX(0);
|
||||
opacity: 1;
|
||||
}
|
||||
}
|
||||
</style>
|
||||
109
frontend/history-ui/src/lib/api.ts
Normal file
109
frontend/history-ui/src/lib/api.ts
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
import type {
|
||||
DocumentVersion,
|
||||
DocumentVersionWithoutContent,
|
||||
FetchLatestDocumentsResponse,
|
||||
PingResponse,
|
||||
VaultHistoryResponse
|
||||
} from "./types";
|
||||
|
||||
export class ApiClient {
|
||||
constructor(
|
||||
private vaultId: string,
|
||||
private token: string
|
||||
) {}
|
||||
|
||||
private get baseUrl(): string {
|
||||
return `/vaults/${encodeURIComponent(this.vaultId)}`;
|
||||
}
|
||||
|
||||
private headers(): Record<string, string> {
|
||||
return {
|
||||
Authorization: `Bearer ${this.token}`,
|
||||
"device-id": "history-ui"
|
||||
};
|
||||
}
|
||||
|
||||
private async fetchJson<T>(
|
||||
path: string,
|
||||
init?: RequestInit
|
||||
): Promise<T> {
|
||||
const response = await fetch(path, {
|
||||
...init,
|
||||
headers: { ...this.headers(), ...init?.headers }
|
||||
});
|
||||
if (!response.ok) {
|
||||
const body = await response.text();
|
||||
throw new Error(`HTTP ${response.status}: ${body}`);
|
||||
}
|
||||
return response.json() as Promise<T>;
|
||||
}
|
||||
|
||||
async ping(): Promise<PingResponse> {
|
||||
return this.fetchJson(`${this.baseUrl}/ping`);
|
||||
}
|
||||
|
||||
async fetchLatestDocuments(): Promise<FetchLatestDocumentsResponse> {
|
||||
return this.fetchJson(`${this.baseUrl}/documents`);
|
||||
}
|
||||
|
||||
async fetchDocumentVersions(
|
||||
documentId: string
|
||||
): Promise<DocumentVersionWithoutContent[]> {
|
||||
return this.fetchJson(
|
||||
`${this.baseUrl}/documents/${documentId}/versions`
|
||||
);
|
||||
}
|
||||
|
||||
async fetchDocumentVersion(
|
||||
documentId: string,
|
||||
vaultUpdateId: number
|
||||
): Promise<DocumentVersion> {
|
||||
return this.fetchJson(
|
||||
`${this.baseUrl}/documents/${documentId}/versions/${vaultUpdateId}`
|
||||
);
|
||||
}
|
||||
|
||||
async fetchDocumentVersionContent(
|
||||
documentId: string,
|
||||
vaultUpdateId: number
|
||||
): Promise<ArrayBuffer> {
|
||||
const response = await fetch(
|
||||
`${this.baseUrl}/documents/${documentId}/versions/${vaultUpdateId}/content`,
|
||||
{ headers: this.headers() }
|
||||
);
|
||||
if (!response.ok) {
|
||||
throw new Error(`HTTP ${response.status}`);
|
||||
}
|
||||
return response.arrayBuffer();
|
||||
}
|
||||
|
||||
async fetchVaultHistory(
|
||||
limit?: number,
|
||||
beforeUpdateId?: number
|
||||
): Promise<VaultHistoryResponse> {
|
||||
const params = new URLSearchParams();
|
||||
if (limit !== undefined) params.set("limit", String(limit));
|
||||
if (beforeUpdateId !== undefined)
|
||||
params.set("before_update_id", String(beforeUpdateId));
|
||||
const qs = params.toString();
|
||||
return this.fetchJson(
|
||||
`${this.baseUrl}/history${qs ? `?${qs}` : ""}`
|
||||
);
|
||||
}
|
||||
|
||||
async restoreVersion(
|
||||
documentId: string,
|
||||
vaultUpdateId: number
|
||||
): Promise<DocumentVersionWithoutContent> {
|
||||
return this.fetchJson(
|
||||
`${this.baseUrl}/documents/${documentId}/restore`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
body: JSON.stringify({ vaultUpdateId })
|
||||
}
|
||||
);
|
||||
}
|
||||
}
|
||||
291
frontend/history-ui/src/lib/stores.svelte.ts
Normal file
291
frontend/history-ui/src/lib/stores.svelte.ts
Normal file
|
|
@ -0,0 +1,291 @@
|
|||
import { ApiClient } from "./api";
|
||||
import type {
|
||||
DocumentVersionWithoutContent,
|
||||
VersionEvent,
|
||||
ActionType,
|
||||
TreeNode
|
||||
} from "./types";
|
||||
|
||||
class AuthStore {
|
||||
vaultId = $state("");
|
||||
token = $state("");
|
||||
isAuthenticated = $state(false);
|
||||
serverVersion = $state("");
|
||||
api = $state<ApiClient | null>(null);
|
||||
|
||||
login(vaultId: string, token: string, serverVersion: string) {
|
||||
this.vaultId = vaultId;
|
||||
this.token = token;
|
||||
this.serverVersion = serverVersion;
|
||||
this.isAuthenticated = true;
|
||||
this.api = new ApiClient(vaultId, token);
|
||||
sessionStorage.setItem(
|
||||
"vaultlink_auth",
|
||||
JSON.stringify({ vaultId, token })
|
||||
);
|
||||
}
|
||||
|
||||
logout() {
|
||||
this.vaultId = "";
|
||||
this.token = "";
|
||||
this.isAuthenticated = false;
|
||||
this.serverVersion = "";
|
||||
this.api = null;
|
||||
sessionStorage.removeItem("vaultlink_auth");
|
||||
}
|
||||
|
||||
tryRestore(): { vaultId: string; token: string } | null {
|
||||
const stored = sessionStorage.getItem("vaultlink_auth");
|
||||
if (!stored) return null;
|
||||
try {
|
||||
return JSON.parse(stored) as {
|
||||
vaultId: string;
|
||||
token: string;
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export const auth = new AuthStore();
|
||||
|
||||
// Navigation
|
||||
export type View =
|
||||
| { kind: "dashboard" }
|
||||
| { kind: "document"; documentId: string };
|
||||
|
||||
class NavStore {
|
||||
current = $state<View>({ kind: "dashboard" });
|
||||
|
||||
goto(view: View) {
|
||||
this.current = view;
|
||||
}
|
||||
|
||||
goHome() {
|
||||
this.current = { kind: "dashboard" };
|
||||
}
|
||||
}
|
||||
|
||||
export const nav = new NavStore();
|
||||
|
||||
// Toasts
|
||||
export interface Toast {
|
||||
id: number;
|
||||
message: string;
|
||||
type: "success" | "error" | "info";
|
||||
}
|
||||
|
||||
class ToastStore {
|
||||
items = $state<Toast[]>([]);
|
||||
private nextId = 0;
|
||||
|
||||
add(message: string, type: Toast["type"] = "info") {
|
||||
const id = this.nextId++;
|
||||
this.items.push({ id, message, type });
|
||||
setTimeout(() => this.dismiss(id), 5000);
|
||||
}
|
||||
|
||||
dismiss(id: number) {
|
||||
this.items = this.items.filter((t) => t.id !== id);
|
||||
}
|
||||
}
|
||||
|
||||
export const toasts = new ToastStore();
|
||||
|
||||
// Utilities
|
||||
|
||||
export function inferAction(
|
||||
version: DocumentVersionWithoutContent,
|
||||
previousVersion?: DocumentVersionWithoutContent
|
||||
): ActionType {
|
||||
if (version.isDeleted) return "deleted";
|
||||
if (!previousVersion) return "created";
|
||||
if (
|
||||
previousVersion.isDeleted &&
|
||||
!version.isDeleted
|
||||
)
|
||||
return "restored";
|
||||
if (previousVersion.relativePath !== version.relativePath)
|
||||
return "renamed";
|
||||
return "updated";
|
||||
}
|
||||
|
||||
export function enrichVersions(
|
||||
versions: DocumentVersionWithoutContent[]
|
||||
): VersionEvent[] {
|
||||
// versions should be sorted by vaultUpdateId ascending
|
||||
const sorted = [...versions].sort(
|
||||
(a, b) => a.vaultUpdateId - b.vaultUpdateId
|
||||
);
|
||||
const byDoc = new Map<string, DocumentVersionWithoutContent[]>();
|
||||
for (const v of sorted) {
|
||||
let arr = byDoc.get(v.documentId);
|
||||
if (!arr) {
|
||||
arr = [];
|
||||
byDoc.set(v.documentId, arr);
|
||||
}
|
||||
arr.push(v);
|
||||
}
|
||||
|
||||
return sorted.map((v) => {
|
||||
const docVersions = byDoc.get(v.documentId)!;
|
||||
const idx = docVersions.indexOf(v);
|
||||
const prev = idx > 0 ? docVersions[idx - 1] : undefined;
|
||||
const action = inferAction(v, prev);
|
||||
return {
|
||||
...v,
|
||||
action,
|
||||
previousPath:
|
||||
action === "renamed" ? prev?.relativePath : undefined
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
export function buildTree(
|
||||
documents: DocumentVersionWithoutContent[],
|
||||
showDeleted: boolean
|
||||
): TreeNode {
|
||||
const root: TreeNode = {
|
||||
name: "",
|
||||
path: "",
|
||||
isFolder: true,
|
||||
children: []
|
||||
};
|
||||
|
||||
const filtered = showDeleted
|
||||
? documents
|
||||
: documents.filter((d) => !d.isDeleted);
|
||||
|
||||
for (const doc of filtered) {
|
||||
const parts = doc.relativePath.split("/");
|
||||
let current = root;
|
||||
|
||||
for (let i = 0; i < parts.length; i++) {
|
||||
const part = parts[i];
|
||||
const isFile = i === parts.length - 1;
|
||||
const path = parts.slice(0, i + 1).join("/");
|
||||
|
||||
if (isFile) {
|
||||
current.children.push({
|
||||
name: part,
|
||||
path,
|
||||
isFolder: false,
|
||||
children: [],
|
||||
document: doc,
|
||||
isDeleted: doc.isDeleted
|
||||
});
|
||||
} else {
|
||||
let folder = current.children.find(
|
||||
(c) => c.isFolder && c.name === part
|
||||
);
|
||||
if (!folder) {
|
||||
folder = {
|
||||
name: part,
|
||||
path,
|
||||
isFolder: true,
|
||||
children: []
|
||||
};
|
||||
current.children.push(folder);
|
||||
}
|
||||
current = folder;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
sortTree(root);
|
||||
return root;
|
||||
}
|
||||
|
||||
function sortTree(node: TreeNode) {
|
||||
node.children.sort((a, b) => {
|
||||
if (a.isFolder !== b.isFolder) return a.isFolder ? -1 : 1;
|
||||
return a.name.localeCompare(b.name);
|
||||
});
|
||||
for (const child of node.children) {
|
||||
if (child.isFolder) sortTree(child);
|
||||
}
|
||||
}
|
||||
|
||||
export function relativeTime(dateStr: string): string {
|
||||
const date = new Date(dateStr);
|
||||
const now = Date.now();
|
||||
const diff = now - date.getTime();
|
||||
const seconds = Math.floor(diff / 1000);
|
||||
const minutes = Math.floor(seconds / 60);
|
||||
const hours = Math.floor(minutes / 60);
|
||||
const days = Math.floor(hours / 24);
|
||||
|
||||
if (seconds < 60) return "just now";
|
||||
if (minutes < 60) return `${minutes}m ago`;
|
||||
if (hours < 24) return `${hours}h ago`;
|
||||
if (days < 7) return `${days}d ago`;
|
||||
return date.toLocaleDateString("en-US", {
|
||||
month: "short",
|
||||
day: "numeric",
|
||||
year: days > 365 ? "numeric" : undefined
|
||||
});
|
||||
}
|
||||
|
||||
export function absoluteTime(dateStr: string): string {
|
||||
return new Date(dateStr).toLocaleString();
|
||||
}
|
||||
|
||||
export function formatBytes(bytes: number): string {
|
||||
if (bytes === 0) return "0 B";
|
||||
const k = 1024;
|
||||
const sizes = ["B", "KB", "MB", "GB"];
|
||||
const i = Math.floor(Math.log(bytes) / Math.log(k));
|
||||
return `${parseFloat((bytes / Math.pow(k, i)).toFixed(1))} ${sizes[i]}`;
|
||||
}
|
||||
|
||||
export function fileExtension(path: string): string {
|
||||
const dot = path.lastIndexOf(".");
|
||||
return dot > -1 ? path.substring(dot + 1).toLowerCase() : "";
|
||||
}
|
||||
|
||||
export function isTextFile(path: string): boolean {
|
||||
const textExts = new Set([
|
||||
"md",
|
||||
"txt",
|
||||
"json",
|
||||
"yaml",
|
||||
"yml",
|
||||
"toml",
|
||||
"xml",
|
||||
"html",
|
||||
"css",
|
||||
"js",
|
||||
"ts",
|
||||
"svelte",
|
||||
"rs",
|
||||
"py",
|
||||
"sh",
|
||||
"bash",
|
||||
"zsh",
|
||||
"csv",
|
||||
"svg",
|
||||
"log",
|
||||
"conf",
|
||||
"cfg",
|
||||
"ini",
|
||||
"env",
|
||||
"gitignore",
|
||||
"editorconfig"
|
||||
]);
|
||||
return textExts.has(fileExtension(path));
|
||||
}
|
||||
|
||||
export function isImageFile(path: string): boolean {
|
||||
const imageExts = new Set([
|
||||
"png",
|
||||
"jpg",
|
||||
"jpeg",
|
||||
"gif",
|
||||
"webp",
|
||||
"svg",
|
||||
"ico",
|
||||
"bmp"
|
||||
]);
|
||||
return imageExts.has(fileExtension(path));
|
||||
}
|
||||
54
frontend/history-ui/src/lib/types.ts
Normal file
54
frontend/history-ui/src/lib/types.ts
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
export interface DocumentVersionWithoutContent {
|
||||
vaultUpdateId: number;
|
||||
documentId: string;
|
||||
relativePath: string;
|
||||
updatedDate: string;
|
||||
isDeleted: boolean;
|
||||
userId: string;
|
||||
deviceId: string;
|
||||
contentSize: number;
|
||||
}
|
||||
|
||||
export interface DocumentVersion {
|
||||
vaultUpdateId: number;
|
||||
documentId: string;
|
||||
relativePath: string;
|
||||
updatedDate: string;
|
||||
contentBase64: string;
|
||||
isDeleted: boolean;
|
||||
userId: string;
|
||||
deviceId: string;
|
||||
}
|
||||
|
||||
export interface FetchLatestDocumentsResponse {
|
||||
latestDocuments: DocumentVersionWithoutContent[];
|
||||
lastUpdateId: number;
|
||||
}
|
||||
|
||||
export interface VaultHistoryResponse {
|
||||
versions: DocumentVersionWithoutContent[];
|
||||
hasMore: boolean;
|
||||
}
|
||||
|
||||
export interface PingResponse {
|
||||
serverVersion: string;
|
||||
isAuthenticated: boolean;
|
||||
mergeableFileExtensions: string[];
|
||||
supportedApiVersion: number;
|
||||
}
|
||||
|
||||
export type ActionType = "created" | "updated" | "renamed" | "deleted" | "restored";
|
||||
|
||||
export interface VersionEvent extends DocumentVersionWithoutContent {
|
||||
action: ActionType;
|
||||
previousPath?: string;
|
||||
}
|
||||
|
||||
export interface TreeNode {
|
||||
name: string;
|
||||
path: string;
|
||||
isFolder: boolean;
|
||||
children: TreeNode[];
|
||||
document?: DocumentVersionWithoutContent;
|
||||
isDeleted?: boolean;
|
||||
}
|
||||
7
frontend/history-ui/src/main.ts
Normal file
7
frontend/history-ui/src/main.ts
Normal file
|
|
@ -0,0 +1,7 @@
|
|||
import { mount } from "svelte";
|
||||
import App from "./App.svelte";
|
||||
import "./app.css";
|
||||
|
||||
const app = mount(App, { target: document.getElementById("app")! });
|
||||
|
||||
export default app;
|
||||
5
frontend/history-ui/svelte.config.js
Normal file
5
frontend/history-ui/svelte.config.js
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
import { vitePreprocess } from "@sveltejs/vite-plugin-svelte";
|
||||
|
||||
export default {
|
||||
preprocess: vitePreprocess()
|
||||
};
|
||||
16
frontend/history-ui/tsconfig.json
Normal file
16
frontend/history-ui/tsconfig.json
Normal file
|
|
@ -0,0 +1,16 @@
|
|||
{
|
||||
"compilerOptions": {
|
||||
"target": "ESNext",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"isolatedModules": true,
|
||||
"verbatimModuleSyntax": true,
|
||||
"types": ["svelte"]
|
||||
},
|
||||
"include": ["src/**/*", "src/**/*.svelte"]
|
||||
}
|
||||
15
frontend/history-ui/vite.config.ts
Normal file
15
frontend/history-ui/vite.config.ts
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
import { defineConfig } from "vite";
|
||||
import { svelte } from "@sveltejs/vite-plugin-svelte";
|
||||
|
||||
export default defineConfig({
|
||||
plugins: [svelte()],
|
||||
build: {
|
||||
outDir: "dist",
|
||||
emptyOutDir: true
|
||||
},
|
||||
server: {
|
||||
proxy: {
|
||||
"/vaults": "http://localhost:3011"
|
||||
}
|
||||
}
|
||||
});
|
||||
|
|
@ -56,15 +56,16 @@ vaultlink \
|
|||
|
||||
### Optional
|
||||
|
||||
| Option | Default | Description |
|
||||
| ------------------------------------ | ------- | -------------------------------------- |
|
||||
| `--sync-concurrency <number>` | `1` | Concurrent sync operations |
|
||||
| `--max-file-size-mb <number>` | `10` | Maximum file size in MB |
|
||||
| `--ignore-pattern <pattern>` | - | Glob pattern to ignore (repeatable) |
|
||||
| `--websocket-retry-interval-ms <ms>` | `3500` | WebSocket reconnection interval |
|
||||
| `--log-level <level>` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
|
||||
| `-h, --help` | - | Show help |
|
||||
| `-V, --version` | - | Show version |
|
||||
| Option | Default | Description |
|
||||
| ------------------------------------ | ------- | ---------------------------------------------------- |
|
||||
| `--max-file-size-mb <number>` | `10` | Maximum file size in MB |
|
||||
| `--ignore-pattern <pattern>` | - | Glob pattern to ignore (repeatable) |
|
||||
| `--websocket-retry-interval-ms <ms>` | `3500` | WebSocket reconnection interval |
|
||||
| `--log-level <level>` | `INFO` | Log level: DEBUG, INFO, WARNING, ERROR |
|
||||
| `--line-endings <mode>` | `auto` | Line ending style: auto, lf, crlf |
|
||||
| `-q, --quiet` | - | Suppress startup banner for non-interactive use |
|
||||
| `-h, --help` | - | Show help |
|
||||
| `-V, --version` | - | Show version |
|
||||
|
||||
### Auto-Ignored Patterns
|
||||
|
||||
|
|
@ -83,16 +84,23 @@ With ignore patterns:
|
|||
|
||||
```bash
|
||||
vaultlink -l ./vault -r wss://sync.example.com -t token123 -v default \
|
||||
--ignore-pattern "*.tmp" \
|
||||
--ignore-pattern "**/*.tmp" \
|
||||
--ignore-pattern ".DS_Store" \
|
||||
--ignore-pattern "node_modules/**"
|
||||
```
|
||||
|
||||
With debug logging:
|
||||
With debug logging and quiet startup:
|
||||
|
||||
```bash
|
||||
vaultlink -l ./vault -r wss://sync.example.com -t token123 -v default \
|
||||
--log-level DEBUG
|
||||
--log-level DEBUG --quiet
|
||||
```
|
||||
|
||||
Force LF line endings (useful for cross-platform vaults):
|
||||
|
||||
```bash
|
||||
vaultlink -l ./vault -r wss://sync.example.com -t token123 -v default \
|
||||
--line-endings lf
|
||||
```
|
||||
|
||||
## Docker Deployment
|
||||
|
|
|
|||
|
|
@ -55,13 +55,10 @@ test("parseArgs - parse with optional arguments", () => {
|
|||
"mytoken",
|
||||
"-v",
|
||||
"default",
|
||||
"--sync-concurrency",
|
||||
"5",
|
||||
"--max-file-size-mb",
|
||||
"20"
|
||||
]);
|
||||
|
||||
assert.equal(args.syncConcurrency, 5);
|
||||
assert.equal(args.maxFileSizeMB, 20);
|
||||
});
|
||||
|
||||
|
|
@ -292,3 +289,162 @@ test("parseArgs - reads log level from environment variable", () => {
|
|||
delete process.env.VAULTLINK_LOG_LEVEL;
|
||||
}
|
||||
});
|
||||
|
||||
test("parseArgs - quiet defaults to false", () => {
|
||||
const args = parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"https://sync.example.com",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default"
|
||||
]);
|
||||
|
||||
assert.equal(args.quiet, false);
|
||||
});
|
||||
|
||||
test("parseArgs - parse --quiet flag", () => {
|
||||
const args = parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"https://sync.example.com",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default",
|
||||
"--quiet"
|
||||
]);
|
||||
|
||||
assert.equal(args.quiet, true);
|
||||
});
|
||||
|
||||
test("parseArgs - parse -q short flag", () => {
|
||||
const args = parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"https://sync.example.com",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default",
|
||||
"-q"
|
||||
]);
|
||||
|
||||
assert.equal(args.quiet, true);
|
||||
});
|
||||
|
||||
test("parseArgs - line-endings defaults to auto", () => {
|
||||
const args = parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"https://sync.example.com",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default"
|
||||
]);
|
||||
|
||||
assert.equal(args.lineEndings, "auto");
|
||||
});
|
||||
|
||||
test("parseArgs - parse --line-endings lf", () => {
|
||||
const args = parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"https://sync.example.com",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default",
|
||||
"--line-endings",
|
||||
"lf"
|
||||
]);
|
||||
|
||||
assert.equal(args.lineEndings, "lf");
|
||||
});
|
||||
|
||||
test("parseArgs - parse --line-endings crlf", () => {
|
||||
const args = parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"https://sync.example.com",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default",
|
||||
"--line-endings",
|
||||
"crlf"
|
||||
]);
|
||||
|
||||
assert.equal(args.lineEndings, "crlf");
|
||||
});
|
||||
|
||||
test("parseArgs - throws on invalid remote URI protocol", () => {
|
||||
assert.throws(() => {
|
||||
parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"ftp://sync.example.com",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default"
|
||||
]);
|
||||
}, /Invalid remote URI/);
|
||||
});
|
||||
|
||||
test("parseArgs - accepts http:// remote URI", () => {
|
||||
const args = parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"http://localhost:3000",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default"
|
||||
]);
|
||||
|
||||
assert.equal(args.remoteUri, "http://localhost:3000");
|
||||
});
|
||||
|
||||
test("parseArgs - accepts wss:// remote URI", () => {
|
||||
const args = parseArgs([
|
||||
"node",
|
||||
"cli.js",
|
||||
"-l",
|
||||
"/path/to/vault",
|
||||
"-r",
|
||||
"wss://sync.example.com",
|
||||
"-t",
|
||||
"mytoken",
|
||||
"-v",
|
||||
"default"
|
||||
]);
|
||||
|
||||
assert.equal(args.remoteUri, "wss://sync.example.com");
|
||||
});
|
||||
|
|
|
|||
|
|
@ -2,20 +2,25 @@ import { Command, Option } from "commander";
|
|||
import packageJson from "../package.json";
|
||||
import { LogLevel } from "sync-client";
|
||||
|
||||
export type LineEndingMode = "auto" | "lf" | "crlf";
|
||||
|
||||
export interface CliArgs {
|
||||
remoteUri: string;
|
||||
token: string;
|
||||
vaultName: string;
|
||||
localPath: string;
|
||||
syncConcurrency?: number;
|
||||
maxFileSizeMB?: number;
|
||||
ignorePatterns?: string[];
|
||||
webSocketRetryIntervalMs?: number;
|
||||
logLevel: LogLevel;
|
||||
health?: string;
|
||||
enableTelemetry?: boolean;
|
||||
quiet: boolean;
|
||||
lineEndings: LineEndingMode;
|
||||
}
|
||||
|
||||
const VALID_URI_PREFIXES = ["http://", "https://", "ws://", "wss://"];
|
||||
|
||||
export function parseArgs(argv: string[]): CliArgs {
|
||||
const program = new Command();
|
||||
|
||||
|
|
@ -49,14 +54,6 @@ export function parseArgs(argv: string[]): CliArgs {
|
|||
"Vault name"
|
||||
).env("VAULTLINK_VAULT_NAME")
|
||||
)
|
||||
.addOption(
|
||||
new Option(
|
||||
"--sync-concurrency <number>",
|
||||
"[OPTIONAL] Number of concurrent sync operations"
|
||||
)
|
||||
.argParser(parseInt)
|
||||
.env("VAULTLINK_SYNC_CONCURRENCY")
|
||||
)
|
||||
.addOption(
|
||||
new Option(
|
||||
"--max-file-size-mb <number>",
|
||||
|
|
@ -99,15 +96,30 @@ export function parseArgs(argv: string[]): CliArgs {
|
|||
"[OPTIONAL] Enable telemetry (disabled by default)"
|
||||
).env("VAULTLINK_ENABLE_TELEMETRY")
|
||||
)
|
||||
.addOption(
|
||||
new Option(
|
||||
"-q, --quiet",
|
||||
"[OPTIONAL] Suppress startup banner for non-interactive use"
|
||||
).env("VAULTLINK_QUIET")
|
||||
)
|
||||
.addOption(
|
||||
new Option(
|
||||
"--line-endings <mode>",
|
||||
"[OPTIONAL] Line ending style: auto (platform default), lf, crlf"
|
||||
)
|
||||
.default("auto")
|
||||
.choices(["auto", "lf", "crlf"])
|
||||
.env("VAULTLINK_LINE_ENDINGS")
|
||||
)
|
||||
.addHelpText(
|
||||
"after",
|
||||
`
|
||||
Examples:
|
||||
$ vaultlink -l ./my-vault -r https://sync.example.com -t mytoken -v default
|
||||
$ vaultlink -l ./my-vault -r https://sync.example.com -t mytoken -v default \\
|
||||
--ignore-pattern ".git/**" --ignore-pattern "*.tmp"
|
||||
--ignore-pattern ".git/**" --ignore-pattern "**/*.tmp"
|
||||
$ vaultlink -l ./my-vault -r https://sync.example.com -t mytoken -v default \\
|
||||
--log-level DEBUG
|
||||
--log-level DEBUG --quiet
|
||||
|
||||
Environment variables:
|
||||
All options can be configured via VAULTLINK_ prefixed environment variables.
|
||||
|
|
@ -123,7 +135,6 @@ Environment variables:
|
|||
const remoteUri = opts.remoteUri as string | undefined;
|
||||
const token = opts.token as string | undefined;
|
||||
const vaultName = opts.vaultName as string | undefined;
|
||||
const syncConcurrency = opts.syncConcurrency as number | undefined;
|
||||
const maxFileSizeMb = opts.maxFileSizeMb as number | undefined;
|
||||
const ignorePattern = opts.ignorePattern as string[] | undefined;
|
||||
const websocketRetryIntervalMs = opts.websocketRetryIntervalMs as
|
||||
|
|
@ -132,6 +143,8 @@ Environment variables:
|
|||
const logLevelStr = (opts.logLevel as string | undefined) ?? "INFO";
|
||||
const health = opts.health as string | undefined;
|
||||
const enableTelemetry = opts.enableTelemetry as boolean | undefined;
|
||||
const quiet = (opts.quiet as boolean | undefined) ?? false;
|
||||
const lineEndingsStr = (opts.lineEndings as string | undefined) ?? "auto";
|
||||
/* eslint-enable @typescript-eslint/no-unsafe-type-assertion */
|
||||
|
||||
const requireOption = <T>(
|
||||
|
|
@ -142,9 +155,12 @@ Environment variables:
|
|||
const option = program.options.find(
|
||||
(o) => o.attributeName() === name
|
||||
);
|
||||
const envHint =
|
||||
option?.envVar !== undefined
|
||||
? ` (or set ${option.envVar})`
|
||||
: "";
|
||||
throw new Error(
|
||||
`required option '${option?.flags ?? name}' not specified` +
|
||||
(option?.envVar ? ` (or set ${option.envVar})` : "")
|
||||
`required option '${option?.flags ?? name}' not specified${envHint}`
|
||||
);
|
||||
}
|
||||
return value;
|
||||
|
|
@ -155,6 +171,17 @@ Environment variables:
|
|||
const requiredToken = requireOption(token, "token");
|
||||
const requiredVaultName = requireOption(vaultName, "vaultName");
|
||||
|
||||
// Validate remote URI protocol
|
||||
if (
|
||||
!VALID_URI_PREFIXES.some((prefix) =>
|
||||
requiredRemoteUri.startsWith(prefix)
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
`Invalid remote URI '${requiredRemoteUri}'. Must start with ${VALID_URI_PREFIXES.join(", ")}`
|
||||
);
|
||||
}
|
||||
|
||||
// Validate and parse log level
|
||||
const logLevelUpper = logLevelStr.toUpperCase();
|
||||
const validLogLevels = Object.values(LogLevel);
|
||||
|
|
@ -168,17 +195,21 @@ Environment variables:
|
|||
}
|
||||
const logLevel = logLevelUpper;
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
|
||||
const lineEndings = lineEndingsStr as LineEndingMode;
|
||||
|
||||
return {
|
||||
localPath: requiredLocalPath,
|
||||
remoteUri: requiredRemoteUri,
|
||||
token: requiredToken,
|
||||
vaultName: requiredVaultName,
|
||||
syncConcurrency,
|
||||
maxFileSizeMB: maxFileSizeMb,
|
||||
ignorePatterns: ignorePattern,
|
||||
webSocketRetryIntervalMs: websocketRetryIntervalMs,
|
||||
logLevel,
|
||||
health,
|
||||
enableTelemetry
|
||||
enableTelemetry,
|
||||
quiet,
|
||||
lineEndings
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -37,6 +37,20 @@ const LOG_LEVEL_ORDER = {
|
|||
};
|
||||
|
||||
const HEALTH_CHECK_INTERVAL_MS = 30 * 1000;
|
||||
const PROGRESS_LOG_INTERVAL_MS = 2000;
|
||||
|
||||
function resolveLineEndings(
|
||||
mode: "auto" | "lf" | "crlf"
|
||||
): string {
|
||||
switch (mode) {
|
||||
case "lf":
|
||||
return "\n";
|
||||
case "crlf":
|
||||
return "\r\n";
|
||||
case "auto":
|
||||
return process.platform === "win32" ? "\r\n" : "\n";
|
||||
}
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const args = parseArgs(process.argv);
|
||||
|
|
@ -64,21 +78,28 @@ async function main(): Promise<void> {
|
|||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(
|
||||
styleText("VaultLink Local CLI", "bold", "cyan") +
|
||||
colorize(` v${packageJson.version}`, "dim")
|
||||
);
|
||||
console.log(colorize("=".repeat(50), "dim"));
|
||||
console.log(
|
||||
`${colorize("Local path:", "dim")} ${colorize(absolutePath, "green")}`
|
||||
);
|
||||
console.log(
|
||||
`${colorize("Remote URI:", "dim")} ${colorize(args.remoteUri, "cyan")}`
|
||||
);
|
||||
console.log(
|
||||
`${colorize("Vault name:", "dim")} ${colorize(args.vaultName, "green")}`
|
||||
);
|
||||
console.log("");
|
||||
if (!args.quiet) {
|
||||
console.log(
|
||||
styleText("VaultLink Local CLI", "bold", "cyan") +
|
||||
colorize(` v${packageJson.version}`, "dim")
|
||||
);
|
||||
console.log(colorize("=".repeat(50), "dim"));
|
||||
console.log(
|
||||
`${colorize("Local path:", "dim")} ${colorize(absolutePath, "green")}`
|
||||
);
|
||||
console.log(
|
||||
`${colorize("Remote URI:", "dim")} ${colorize(args.remoteUri, "cyan")}`
|
||||
);
|
||||
console.log(
|
||||
`${colorize("Vault name:", "dim")} ${colorize(args.vaultName, "green")}`
|
||||
);
|
||||
if (args.lineEndings !== "auto") {
|
||||
console.log(
|
||||
`${colorize("Line endings:", "dim")} ${colorize(args.lineEndings.toUpperCase(), "green")}`
|
||||
);
|
||||
}
|
||||
console.log("");
|
||||
}
|
||||
|
||||
const dataDir = path.join(absolutePath, ".vaultlink");
|
||||
const dataFile = path.join(dataDir, "sync-data.json");
|
||||
|
|
@ -98,8 +119,6 @@ async function main(): Promise<void> {
|
|||
remoteUri: args.remoteUri,
|
||||
token: args.token,
|
||||
vaultName: args.vaultName,
|
||||
syncConcurrency:
|
||||
args.syncConcurrency ?? DEFAULT_SETTINGS.syncConcurrency,
|
||||
maxFileSizeMB: args.maxFileSizeMB ?? DEFAULT_SETTINGS.maxFileSizeMB,
|
||||
ignorePatterns,
|
||||
webSocketRetryIntervalMs:
|
||||
|
|
@ -141,7 +160,7 @@ async function main(): Promise<void> {
|
|||
);
|
||||
}
|
||||
},
|
||||
nativeLineEndings: process.platform === "win32" ? "\r\n" : "\n"
|
||||
nativeLineEndings: resolveLineEndings(args.lineEndings)
|
||||
});
|
||||
|
||||
if (args.health !== undefined) {
|
||||
|
|
@ -183,11 +202,32 @@ async function main(): Promise<void> {
|
|||
);
|
||||
});
|
||||
|
||||
// Throttled progress reporting
|
||||
let syncBatchSize = 0;
|
||||
let totalSyncOps = 0;
|
||||
let lastProgressLogTime = 0;
|
||||
|
||||
client.onRemainingOperationsCountChanged.add((remaining) => {
|
||||
if (remaining > syncBatchSize) {
|
||||
syncBatchSize = remaining;
|
||||
}
|
||||
|
||||
if (remaining === 0) {
|
||||
client.logger.info("All sync operations completed");
|
||||
if (syncBatchSize > 0) {
|
||||
totalSyncOps += syncBatchSize;
|
||||
client.logger.info(
|
||||
`Sync batch complete (${syncBatchSize} operations)`
|
||||
);
|
||||
syncBatchSize = 0;
|
||||
}
|
||||
} else {
|
||||
client.logger.info(`${remaining} sync operations remaining`);
|
||||
const now = Date.now();
|
||||
if (now - lastProgressLogTime >= PROGRESS_LOG_INTERVAL_MS) {
|
||||
client.logger.info(
|
||||
`Syncing: ${remaining} operations remaining`
|
||||
);
|
||||
lastProgressLogTime = now;
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
|
|
@ -208,7 +248,17 @@ async function main(): Promise<void> {
|
|||
fileWatcher.stop();
|
||||
await client.waitUntilFinished();
|
||||
await client.destroy();
|
||||
console.log(colorize("Shutdown complete", "green"));
|
||||
|
||||
if (totalSyncOps > 0) {
|
||||
console.log(
|
||||
colorize(
|
||||
`Shutdown complete (${totalSyncOps} operations synced)`,
|
||||
"green"
|
||||
)
|
||||
);
|
||||
} else {
|
||||
console.log(colorize("Shutdown complete", "green"));
|
||||
}
|
||||
process.exit(0);
|
||||
};
|
||||
|
||||
|
|
@ -231,9 +281,13 @@ async function main(): Promise<void> {
|
|||
process.exit(1);
|
||||
}
|
||||
|
||||
console.log(`${colorize("✓", "green")} Server connection successful`);
|
||||
console.log(colorize("Press Ctrl+C to stop", "dim"));
|
||||
console.log("");
|
||||
if (!args.quiet) {
|
||||
console.log(
|
||||
`${colorize("✓", "green")} Server connection successful`
|
||||
);
|
||||
console.log(colorize("Press Ctrl+C to stop", "dim"));
|
||||
console.log("");
|
||||
}
|
||||
|
||||
await client.start();
|
||||
fileWatcher.start();
|
||||
|
|
|
|||
|
|
@ -1,16 +1,20 @@
|
|||
import Watcher from "watcher";
|
||||
import * as path from "path";
|
||||
import type { SyncClient, RelativePath } from "sync-client";
|
||||
import { toUnixPath, compileGlobPattern } from "./path-utils";
|
||||
|
||||
export class FileWatcher {
|
||||
private watcher: Watcher | undefined;
|
||||
private isRunning = false;
|
||||
private readonly compiledPatterns: RegExp[];
|
||||
|
||||
public constructor(
|
||||
private readonly basePath: string,
|
||||
private readonly client: SyncClient,
|
||||
private readonly ignorePatterns: string[] = []
|
||||
) {}
|
||||
ignorePatterns: string[] = []
|
||||
) {
|
||||
this.compiledPatterns = ignorePatterns.map(compileGlobPattern);
|
||||
}
|
||||
|
||||
public start(): void {
|
||||
if (this.isRunning) {
|
||||
|
|
@ -24,7 +28,8 @@ export class FileWatcher {
|
|||
renameDetection: true,
|
||||
renameTimeout: 125,
|
||||
ignoreInitial: true,
|
||||
ignore: (filePath: string) => this.shouldIgnore(filePath)
|
||||
ignore: (filePath: string): boolean =>
|
||||
this.shouldIgnore(filePath)
|
||||
});
|
||||
|
||||
this.watcher.on("add", (filePath: string) => {
|
||||
|
|
@ -59,16 +64,8 @@ export class FileWatcher {
|
|||
}
|
||||
|
||||
private shouldIgnore(filePath: string): boolean {
|
||||
const rel = path
|
||||
.relative(this.basePath, filePath)
|
||||
.replace(/\\/g, "/");
|
||||
return this.ignorePatterns.some((pattern) => {
|
||||
if (pattern.endsWith("/**")) {
|
||||
const prefix = pattern.slice(0, -3);
|
||||
return rel === prefix || rel.startsWith(prefix + "/");
|
||||
}
|
||||
return rel === pattern;
|
||||
});
|
||||
const rel = toUnixPath(path.relative(this.basePath, filePath));
|
||||
return this.compiledPatterns.some((regex) => regex.test(rel));
|
||||
}
|
||||
|
||||
private handleCreate(relativePath: RelativePath): void {
|
||||
|
|
@ -116,18 +113,7 @@ export class FileWatcher {
|
|||
}
|
||||
|
||||
private toRelativePath(absolutePath: string): RelativePath {
|
||||
const relative = path.relative(this.basePath, absolutePath);
|
||||
return this.toUnixPath(relative);
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a native platform path to forward slashes
|
||||
*/
|
||||
private toUnixPath(nativePath: string): string {
|
||||
if (path.sep === "\\") {
|
||||
return nativePath.replace(/\\/g, "/");
|
||||
}
|
||||
return nativePath;
|
||||
return toUnixPath(path.relative(this.basePath, absolutePath));
|
||||
}
|
||||
|
||||
private formatError(err: unknown): string {
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import type {
|
|||
RelativePath,
|
||||
TextWithCursors
|
||||
} from "sync-client";
|
||||
import { toUnixPath, toNativePath } from "./path-utils";
|
||||
|
||||
export class NodeFileSystemOperations implements FileSystemOperations {
|
||||
public constructor(private readonly basePath: string) {}
|
||||
|
|
@ -15,7 +16,7 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
): Promise<RelativePath[]> {
|
||||
const files: RelativePath[] = [];
|
||||
await this.walkDirectory(
|
||||
directory !== undefined ? this.toNativePath(directory) : "",
|
||||
directory !== undefined ? toNativePath(directory) : "",
|
||||
files
|
||||
);
|
||||
return files;
|
||||
|
|
@ -24,7 +25,7 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
public async read(relativePath: RelativePath): Promise<Uint8Array> {
|
||||
const fullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(relativePath)
|
||||
toNativePath(relativePath)
|
||||
);
|
||||
try {
|
||||
return await fs.readFile(fullPath);
|
||||
|
|
@ -41,7 +42,7 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
): Promise<void> {
|
||||
const fullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(relativePath)
|
||||
toNativePath(relativePath)
|
||||
);
|
||||
const dir = path.dirname(fullPath);
|
||||
|
||||
|
|
@ -61,7 +62,7 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
): Promise<string> {
|
||||
const fullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(relativePath)
|
||||
toNativePath(relativePath)
|
||||
);
|
||||
|
||||
try {
|
||||
|
|
@ -79,7 +80,7 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
public async getFileSize(relativePath: RelativePath): Promise<number> {
|
||||
const fullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(relativePath)
|
||||
toNativePath(relativePath)
|
||||
);
|
||||
try {
|
||||
const stats = await fs.stat(fullPath);
|
||||
|
|
@ -94,7 +95,7 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
public async exists(relativePath: RelativePath): Promise<boolean> {
|
||||
const fullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(relativePath)
|
||||
toNativePath(relativePath)
|
||||
);
|
||||
try {
|
||||
await fs.access(fullPath);
|
||||
|
|
@ -107,7 +108,7 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
public async createDirectory(relativePath: RelativePath): Promise<void> {
|
||||
const fullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(relativePath)
|
||||
toNativePath(relativePath)
|
||||
);
|
||||
try {
|
||||
await fs.mkdir(fullPath, { recursive: false });
|
||||
|
|
@ -121,7 +122,7 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
public async delete(relativePath: RelativePath): Promise<void> {
|
||||
const fullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(relativePath)
|
||||
toNativePath(relativePath)
|
||||
);
|
||||
try {
|
||||
await fs.unlink(fullPath);
|
||||
|
|
@ -138,11 +139,11 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
): Promise<void> {
|
||||
const oldFullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(oldPath)
|
||||
toNativePath(oldPath)
|
||||
);
|
||||
const newFullPath = path.join(
|
||||
this.basePath,
|
||||
this.toNativePath(newPath)
|
||||
toNativePath(newPath)
|
||||
);
|
||||
const newDir = path.dirname(newFullPath);
|
||||
|
||||
|
|
@ -192,28 +193,9 @@ export class NodeFileSystemOperations implements FileSystemOperations {
|
|||
await this.walkDirectory(entryRelativePath, files);
|
||||
} else if (entry.isFile()) {
|
||||
// Always return forward slashes
|
||||
files.push(this.toUnixPath(entryRelativePath));
|
||||
files.push(toUnixPath(entryRelativePath));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a forward-slash path to native platform path separators
|
||||
*/
|
||||
private toNativePath(relativePath: string): string {
|
||||
if (path.sep === "\\") {
|
||||
return relativePath.replace(/\//g, "\\");
|
||||
}
|
||||
return relativePath;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a native platform path to forward slashes
|
||||
*/
|
||||
private toUnixPath(nativePath: string): string {
|
||||
if (path.sep === "\\") {
|
||||
return nativePath.replace(/\\/g, "/");
|
||||
}
|
||||
return nativePath;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
65
frontend/local-client-cli/src/path-utils.test.ts
Normal file
65
frontend/local-client-cli/src/path-utils.test.ts
Normal file
|
|
@ -0,0 +1,65 @@
|
|||
import { test } from "node:test";
|
||||
import * as assert from "node:assert/strict";
|
||||
import { compileGlobPattern, toUnixPath } from "./path-utils";
|
||||
|
||||
function matches(path: string, pattern: string): boolean {
|
||||
return compileGlobPattern(pattern).test(path);
|
||||
}
|
||||
|
||||
test("compileGlobPattern - exact match", () => {
|
||||
assert.equal(matches(".DS_Store", ".DS_Store"), true);
|
||||
assert.equal(matches("other", ".DS_Store"), false);
|
||||
});
|
||||
|
||||
test("compileGlobPattern - dir/** matches directory and contents", () => {
|
||||
assert.equal(matches(".git", ".git/**"), true);
|
||||
assert.equal(matches(".git/config", ".git/**"), true);
|
||||
assert.equal(matches(".git/refs/heads/main", ".git/**"), true);
|
||||
assert.equal(matches(".gitignore", ".git/**"), false);
|
||||
});
|
||||
|
||||
test("compileGlobPattern - * matches within a single segment", () => {
|
||||
assert.equal(matches("foo.tmp", "*.tmp"), true);
|
||||
assert.equal(matches("bar.tmp", "*.tmp"), true);
|
||||
assert.equal(matches("foo.md", "*.tmp"), false);
|
||||
// * does NOT cross path separators
|
||||
assert.equal(matches("dir/foo.tmp", "*.tmp"), false);
|
||||
});
|
||||
|
||||
test("compileGlobPattern - **/*.ext matches at any depth", () => {
|
||||
assert.equal(matches("foo.tmp", "**/*.tmp"), true);
|
||||
assert.equal(matches("dir/foo.tmp", "**/*.tmp"), true);
|
||||
assert.equal(matches("a/b/c/foo.tmp", "**/*.tmp"), true);
|
||||
assert.equal(matches("foo.md", "**/*.tmp"), false);
|
||||
});
|
||||
|
||||
test("compileGlobPattern - ? matches single character", () => {
|
||||
assert.equal(matches("a.md", "?.md"), true);
|
||||
assert.equal(matches("ab.md", "?.md"), false);
|
||||
assert.equal(matches(".md", "?.md"), false);
|
||||
});
|
||||
|
||||
test("compileGlobPattern - dots are escaped", () => {
|
||||
assert.equal(matches(".DS_Store", ".DS_Store"), true);
|
||||
assert.equal(matches("xDS_Store", ".DS_Store"), false);
|
||||
});
|
||||
|
||||
test("compileGlobPattern - node_modules/** matches directory tree", () => {
|
||||
assert.equal(matches("node_modules", "node_modules/**"), true);
|
||||
assert.equal(matches("node_modules/foo", "node_modules/**"), true);
|
||||
assert.equal(
|
||||
matches("node_modules/foo/bar/baz.js", "node_modules/**"),
|
||||
true
|
||||
);
|
||||
assert.equal(matches("not_node_modules", "node_modules/**"), false);
|
||||
});
|
||||
|
||||
test("compileGlobPattern - **/ prefix matches zero or more segments", () => {
|
||||
assert.equal(matches("test.log", "**/test.log"), true);
|
||||
assert.equal(matches("dir/test.log", "**/test.log"), true);
|
||||
assert.equal(matches("a/b/test.log", "**/test.log"), true);
|
||||
});
|
||||
|
||||
test("toUnixPath - forward slashes unchanged", () => {
|
||||
assert.equal(toUnixPath("foo/bar/baz"), "foo/bar/baz");
|
||||
});
|
||||
74
frontend/local-client-cli/src/path-utils.ts
Normal file
74
frontend/local-client-cli/src/path-utils.ts
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
import * as path from "path";
|
||||
|
||||
/**
|
||||
* Convert a native platform path to forward slashes.
|
||||
* On non-Windows platforms this is a no-op.
|
||||
*/
|
||||
export function toUnixPath(nativePath: string): string {
|
||||
if (path.sep === "\\") {
|
||||
return nativePath.replace(/\\/g, "/");
|
||||
}
|
||||
return nativePath;
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert a forward-slash path to native platform path separators.
|
||||
* On non-Windows platforms this is a no-op.
|
||||
*/
|
||||
export function toNativePath(forwardSlashPath: string): string {
|
||||
if (path.sep === "\\") {
|
||||
return forwardSlashPath.replace(/\//g, "\\");
|
||||
}
|
||||
return forwardSlashPath;
|
||||
}
|
||||
|
||||
function escapeRegex(str: string): string {
|
||||
return str.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
||||
}
|
||||
|
||||
/**
|
||||
* Compile a glob pattern into a RegExp for repeated matching.
|
||||
* Supports:
|
||||
* - `*` matches any characters within a single path segment
|
||||
* - `**` matches zero or more path segments
|
||||
* - `?` matches a single character (not `/`)
|
||||
* - `dir/**` matches the directory itself and all its contents
|
||||
* - combined with `*.ext` matches files with the extension at any depth
|
||||
*/
|
||||
export function compileGlobPattern(pattern: string): RegExp {
|
||||
// Trailing /** matches the directory itself and all its contents
|
||||
if (pattern.endsWith("/**")) {
|
||||
const prefix = escapeRegex(pattern.slice(0, -3));
|
||||
return new RegExp(`^${prefix}(/.*)?$`);
|
||||
}
|
||||
|
||||
let result = "^";
|
||||
let i = 0;
|
||||
while (i < pattern.length) {
|
||||
const c = pattern[i];
|
||||
if (c === "*" && pattern[i + 1] === "*") {
|
||||
if (pattern[i + 2] === "/") {
|
||||
// **/ matches zero or more directory segments
|
||||
result += "(?:.+/)?";
|
||||
i += 3;
|
||||
} else {
|
||||
result += ".*";
|
||||
i += 2;
|
||||
}
|
||||
} else if (c === "*") {
|
||||
result += "[^/]*";
|
||||
i++;
|
||||
} else if (c === "?") {
|
||||
result += "[^/]";
|
||||
i++;
|
||||
} else if (".+^${}()|[]\\".includes(c)) {
|
||||
result += "\\" + c;
|
||||
i++;
|
||||
} else {
|
||||
result += c;
|
||||
i++;
|
||||
}
|
||||
}
|
||||
result += "$";
|
||||
return new RegExp(result);
|
||||
}
|
||||
|
|
@ -20,7 +20,7 @@
|
|||
"fs-extra": "^11.3.2",
|
||||
"mini-css-extract-plugin": "^2.9.4",
|
||||
"obsidian": "1.11.0",
|
||||
"reconcile-text": "^0.8.0",
|
||||
"reconcile-text": "^0.11.0",
|
||||
"resolve-url-loader": "^5.0.0",
|
||||
"sass": "^1.96.0",
|
||||
"sass-loader": "^16.0.6",
|
||||
|
|
|
|||
|
|
@ -132,7 +132,8 @@ export class RemoteCursorsPluginValue implements PluginValue {
|
|||
]
|
||||
)
|
||||
},
|
||||
edited
|
||||
edited,
|
||||
"Markdown"
|
||||
);
|
||||
|
||||
reconciled.cursors.forEach(({ id, position }) => {
|
||||
|
|
|
|||
|
|
@ -350,22 +350,6 @@ export class SyncSettingsTab extends PluginSettingTab {
|
|||
})
|
||||
);
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Sync concurrency")
|
||||
.setDesc(
|
||||
"How many concurrent sync operations to run. Setting this value higher may increase the overall performance, however, it will require more memory as well. If you notice frequent crashes, especially on mobile, set this to 1."
|
||||
)
|
||||
.addSlider((text) =>
|
||||
text
|
||||
.setLimits(1, 16, 1)
|
||||
.setDynamicTooltip()
|
||||
.setInstant(false)
|
||||
.setValue(this.syncClient.getSettings().syncConcurrency)
|
||||
.onChange(async (value) =>
|
||||
this.syncClient.setSetting("syncConcurrency", value)
|
||||
)
|
||||
);
|
||||
|
||||
new Setting(containerEl)
|
||||
.setName("Maximum file size to be uploaded (MB)")
|
||||
.setDesc(
|
||||
|
|
|
|||
1769
frontend/package-lock.json
generated
1769
frontend/package-lock.json
generated
File diff suppressed because it is too large
Load diff
|
|
@ -6,7 +6,8 @@
|
|||
"obsidian-plugin",
|
||||
"test-client",
|
||||
"deterministic-tests",
|
||||
"local-client-cli"
|
||||
"local-client-cli",
|
||||
"history-ui"
|
||||
],
|
||||
"prettier": {
|
||||
"trailingComma": "none",
|
||||
|
|
|
|||
197
frontend/sync-client/ARCHITECTURE.md
Normal file
197
frontend/sync-client/ARCHITECTURE.md
Normal file
|
|
@ -0,0 +1,197 @@
|
|||
# Sync Client Architecture
|
||||
|
||||
## Overview
|
||||
|
||||
The sync client synchronizes Obsidian vault files between clients via a central server. It handles offline edits, concurrent multi-client changes, crash recovery, and real-time updates via WebSocket.
|
||||
|
||||
## Architecture Layers
|
||||
|
||||
```
|
||||
SyncClient (public API — unchanged)
|
||||
│
|
||||
├── Syncer (event router + reconciliation orchestrator)
|
||||
│ │
|
||||
│ ├── SyncEventQueue (per-document coalescing FIFO)
|
||||
│ │ │
|
||||
│ │ └── executor callback → sync-actions functions
|
||||
│ │
|
||||
│ └── VirtualFilesystem (document identity + state tracking)
|
||||
│
|
||||
├── WebSocketManager (connection, message serialization)
|
||||
├── FileOperations (filesystem abstraction, 3-way merge on write)
|
||||
├── SyncService (HTTP client for server REST API)
|
||||
├── CursorTracker (collaborative cursor positions)
|
||||
└── ContentCache (LRU cache for diff computation)
|
||||
```
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
### 1. Sequential Processing
|
||||
|
||||
All sync operations run one at a time. No concurrent sync operations, no locks, no deadlock prevention, no generation counters. The server uses SQLite which serializes writes anyway, so client-side parallelism provided no real benefit while creating enormous complexity.
|
||||
|
||||
The only concurrency that remains is between the sync queue and the WebSocket message handler, but both funnel events into the same sequential queue.
|
||||
|
||||
### 2. Virtual Filesystem (VFS)
|
||||
|
||||
Replaces the old `Database` class. Documents have explicit states as a discriminated union:
|
||||
|
||||
```typescript
|
||||
type VirtualDocument =
|
||||
| PendingDocument // Created locally, server doesn't know yet
|
||||
| TrackedDocument // Synced with server
|
||||
| DeletedLocallyDocument // Deleted locally, server not yet notified
|
||||
```
|
||||
|
||||
Three internal indexes replace the old flat array + `parallelVersion` system:
|
||||
- `pathIndex` — at most one live document per path
|
||||
- `documentIdIndex` — all documents with a server-assigned ID
|
||||
- `idempotencyKeyIndex` — pending documents only
|
||||
|
||||
No more inferring state from `metadata === undefined` + `isDeleted` + `parentVersionId === 0`. The state field is the discriminator.
|
||||
|
||||
### 3. Per-Document Event Coalescing
|
||||
|
||||
Events from file watchers and WebSocket broadcasts are grouped by document identity and coalesced:
|
||||
- 10 rapid edits → 1 sync operation (content read at execution time)
|
||||
- create then delete → noop (file never reached the server)
|
||||
- move A→B then B→C → move A→C
|
||||
|
||||
This replaces the old opaque-closure FIFO where every event was independent.
|
||||
|
||||
### 4. Server Protocol Unchanged
|
||||
|
||||
The server still does 3-way merging via `reconcile-text`. Response types (`FastForwardUpdate`, `MergingUpdate`) are unchanged. The client content cache remains for diff computation (needed for mobile bandwidth). Idempotency keys remain for crash-safe creates. No server changes were made.
|
||||
|
||||
## Module Descriptions
|
||||
|
||||
### `persistence/vfs.ts` — Virtual Filesystem
|
||||
|
||||
Tracks document identity across creates, moves, deletes. Provides:
|
||||
- State transitions: `createPending()`, `confirmCreate()`, `assignDocumentId()`, `updateTracked()`, `deleteLocally()`, `confirmDelete()`
|
||||
- Queries: `getByPath()`, `getByDocumentId()`, `getByIdempotencyKey()`
|
||||
- Disk reconciliation: `reconcileWithDisk()` returns a pure result comparing VFS state against filesystem
|
||||
- Persistence: serializes to `StoredDatabase` format (backward compatible)
|
||||
|
||||
### `sync-operations/sync-events.ts` — Event Types + Coalescing
|
||||
|
||||
Pure functions with no side effects. Defines `SyncEvent` (6 types from file watchers and WebSocket) and `CoalescedAction` (8 possible merged actions). The `coalesce()` function implements a 48-entry transition table.
|
||||
|
||||
### `sync-operations/sync-event-queue.ts` — Event Queue
|
||||
|
||||
Per-document coalescing FIFO. Maps each event to a document key (documentId for tracked docs, `path:<relativePath>` for pending docs). Processes one document at a time via an injected executor. Supports key migration when pending docs receive a documentId.
|
||||
|
||||
On reset (WebSocket disconnect): remote events are cleared (server replays on reconnect), local events are preserved (unsynced user actions).
|
||||
|
||||
### `sync-operations/sync-actions.ts` — Sync Action Implementations
|
||||
|
||||
Extracted from the old `unrestricted-syncer.ts`. Each function takes explicit dependencies (`SyncDeps`) and a VFS document:
|
||||
|
||||
- `executeSyncCreate()` — POST to server with idempotencyKey, handle response
|
||||
- `executeSyncUpdate()` — compute diff from cache, PUT to server
|
||||
- `executeSyncDelete()` — DELETE on server, confirm in VFS
|
||||
- `executeRemoteUpdate()` — download content, write to disk, update VFS
|
||||
- `applyServerResponse()` — handle MergingUpdate/FastForwardUpdate, path changes, idempotent returns
|
||||
|
||||
### `sync-operations/syncer.ts` — Orchestrator
|
||||
|
||||
Thin layer that:
|
||||
- Converts file change events → `SyncEvent` objects → enqueue
|
||||
- Converts WebSocket broadcasts → `SyncEvent` objects → enqueue
|
||||
- Sets up the executor that dispatches `CoalescedAction` → sync-actions functions
|
||||
- Runs offline reconciliation: resolve idempotency keys → scan filesystem → enqueue results
|
||||
- Manages the `scheduleSyncForOfflineChanges` lifecycle
|
||||
|
||||
## Offline Reconciliation Algorithm
|
||||
|
||||
Runs on startup and WebSocket reconnect:
|
||||
|
||||
1. **Resolve idempotency keys** — call server for pending creates whose responses were lost
|
||||
2. **Clean up orphans** — remove pending docs whose files no longer exist
|
||||
3. **Scan filesystem** — `vfs.reconcileWithDisk()` compares VFS state vs actual files
|
||||
4. **Apply moves** — update VFS for detected file moves (content hash matching)
|
||||
5. **Enqueue events** in order:
|
||||
- Interrupted deletes (VFS says deleted-locally, file gone, server not notified)
|
||||
- Moves (detected via hash matching)
|
||||
- Updates (file content changed)
|
||||
- Creates (new files with no VFS entry)
|
||||
- Delete candidates (VFS entry but file missing, not matched as a move)
|
||||
|
||||
Creates run before delete candidates so the server can merge creates with existing documents (preserving documentIds).
|
||||
|
||||
## Document Lifecycle
|
||||
|
||||
```
|
||||
[File created locally]
|
||||
→ VFS: createPending(path) → PendingDocument
|
||||
→ Queue: enqueue local-create
|
||||
→ Action: POST /documents with idempotencyKey
|
||||
→ Server: returns FastForwardUpdate or MergingUpdate
|
||||
→ VFS: confirmCreate() → TrackedDocument
|
||||
|
||||
[File edited locally]
|
||||
→ Queue: enqueue local-update
|
||||
→ Action: compute diff, PUT /documents/:id/text
|
||||
→ Server: returns FastForwardUpdate or MergingUpdate
|
||||
→ VFS: updateTracked()
|
||||
|
||||
[File deleted locally]
|
||||
→ VFS: deleteLocally() → DeletedLocallyDocument (or removed if pending)
|
||||
→ Queue: enqueue local-delete
|
||||
→ Action: DELETE /documents/:id
|
||||
→ VFS: confirmDelete() → removed
|
||||
|
||||
[Remote update via WebSocket]
|
||||
→ Queue: enqueue remote-update
|
||||
→ Action: fetch content, write to disk
|
||||
→ VFS: updateTracked()
|
||||
|
||||
[Crash during create → restart]
|
||||
→ VFS loads PendingDocument from disk (idempotencyKey preserved)
|
||||
→ resolveIdempotencyKeys() maps key → documentId
|
||||
→ VFS: assignDocumentId() → TrackedDocument with serverVersion=0
|
||||
→ Queue: enqueue local-create (retry)
|
||||
→ Server: returns existing document (idempotent)
|
||||
→ VFS: updateTracked() with real serverVersion
|
||||
```
|
||||
|
||||
## Invariants
|
||||
|
||||
1. **All state mutations go through the sequential queue.** No document state can change while a sync operation is running. File-change handlers and WebSocket handlers only enqueue events.
|
||||
|
||||
2. **Content cache stores server content after merges.** The cache is used for diff computation: `diff(cached_server_content, new_local_content)`. The server applies diffs against its content at `parentVersionId`.
|
||||
|
||||
3. **Idempotency keys survive crashes.** VFS persists pending documents with their keys. On restart, `resolveIdempotencyKeys` maps keys to documentIds. The key is preserved on TrackedDocument when `serverVersion === 0` so retry creates remain idempotent.
|
||||
|
||||
4. **Write file before updating metadata.** If the write fails, metadata still points to the old version. On recovery, the stale `serverVersion` triggers a re-fetch from server.
|
||||
|
||||
5. **Local events survive reset.** When the WebSocket disconnects, remote events are cleared (server replays on reconnect) but local events are preserved in the queue as unsynced user actions.
|
||||
|
||||
6. **Creates run before delete candidates** in the reconciliation ordering. A create may adopt a "deleted" document's identity via server-side merge.
|
||||
|
||||
## What Was Removed
|
||||
|
||||
- **PQueue** — configurable concurrency queue (replaced by sequential event queue)
|
||||
- **Locks** — per-document multi-key locks with alphabetical ordering
|
||||
- **Generation counters** — `resetGeneration` for stale operation detection
|
||||
- **`containsDocument` guards** — 11 guards after async operations for concurrent-delete protection
|
||||
- **`parentVersionIdForUpdate` snapshots** — mutable reference protection
|
||||
- **`parallelVersion`** — collision tracking for multiple docs at same path
|
||||
- **`UnrestrictedSyncer`** — 1,169-line class with nested if/else (replaced by sync-actions.ts with explicit dispatch)
|
||||
- **`Database` class usage** — replaced by VFS everywhere (class still exists for type exports)
|
||||
|
||||
## Files
|
||||
|
||||
```
|
||||
persistence/
|
||||
vfs.ts 779 lines — Virtual Filesystem
|
||||
database.ts 535 lines — Type definitions only (StoredDatabase, RelativePath, etc.)
|
||||
|
||||
sync-operations/
|
||||
syncer.ts 615 lines — Orchestrator
|
||||
sync-actions.ts 1229 lines — Action implementations
|
||||
sync-event-queue.ts 242 lines — Per-document coalescing queue
|
||||
sync-events.ts 297 lines — Event types + coalescing logic
|
||||
unrestricted-syncer.ts 1169 lines — DEAD CODE (not imported, to be deleted)
|
||||
cursor-tracker.ts 273 lines — Cursor position tracking
|
||||
```
|
||||
|
|
@ -13,18 +13,22 @@
|
|||
"test": "tsx --test 'src/**/*.test.ts'"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@sentry/browser": "^10.30.0",
|
||||
"@types/murmurhash3js-revisited": "^3.0.3",
|
||||
"@types/node": "^25.0.2",
|
||||
"byte-base64": "^1.1.0",
|
||||
"minimatch": "^10.1.1",
|
||||
"p-queue": "^9.0.1",
|
||||
"reconcile-text": "^0.8.0",
|
||||
"@types/node": "^25.0.2",
|
||||
"reconcile-text": "^0.11.0",
|
||||
"ts-loader": "^9.5.4",
|
||||
"tslib": "2.8.1",
|
||||
"tsx": "^4.21.0",
|
||||
"typescript": "5.9.3",
|
||||
"webpack": "^5.103.0",
|
||||
"webpack-cli": "^6.0.1",
|
||||
"webpack-merge": "^6.0.1",
|
||||
"@sentry/browser": "^10.30.0"
|
||||
"webpack-merge": "^6.0.1"
|
||||
},
|
||||
"dependencies": {
|
||||
"murmurhash3js-revisited": "^3.0.0"
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,3 +5,5 @@ export const MAX_HISTORY_ENTRY_COUNT = 5000;
|
|||
export const SUPPORTED_API_VERSION = 3;
|
||||
export const WEBSOCKET_DISCONNECT_TIMEOUT_IN_SECONDS = 10;
|
||||
export const WEBSOCKET_CONNECTION_TIMEOUT_IN_SECONDS = 10;
|
||||
export const WEBSOCKET_HEARTBEAT_INTERVAL_MS = 30_000;
|
||||
export const WEBSOCKET_HEARTBEAT_TIMEOUT_MS = 90_000;
|
||||
|
|
|
|||
|
|
@ -1,9 +1,6 @@
|
|||
import { describe, it } from "node:test";
|
||||
import type {
|
||||
Database,
|
||||
DocumentRecord,
|
||||
RelativePath
|
||||
} from "../persistence/database";
|
||||
import type { RelativePath } from "../persistence/database";
|
||||
import type { VirtualFilesystem } from "../persistence/vfs";
|
||||
import { FileOperations } from "./file-operations";
|
||||
import { Logger } from "../tracing/logger";
|
||||
import { assertSetContainsExactly } from "../utils/assert-set-contains-exactly";
|
||||
|
|
@ -21,17 +18,14 @@ class MockServerConfig implements Pick<ServerConfig, "getConfig"> {
|
|||
}
|
||||
}
|
||||
|
||||
class MockDatabase implements Partial<Database> {
|
||||
public getLatestDocumentByRelativePath(
|
||||
_target: RelativePath
|
||||
): DocumentRecord | undefined {
|
||||
// no-op
|
||||
class MockVfs implements Partial<VirtualFilesystem> {
|
||||
public getByPath(_path: string): undefined {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
public move(
|
||||
_oldRelativePath: RelativePath,
|
||||
_newRelativePath: RelativePath
|
||||
_oldPath: string,
|
||||
_newPath: string
|
||||
): void {
|
||||
// no-op
|
||||
}
|
||||
|
|
@ -89,7 +83,7 @@ describe("File operations", () => {
|
|||
const fileSystemOperations = new FakeFileSystemOperations();
|
||||
const fileOperations = new FileOperations(
|
||||
new Logger(),
|
||||
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
fileSystemOperations,
|
||||
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
);
|
||||
|
|
@ -119,7 +113,7 @@ describe("File operations", () => {
|
|||
const fileSystemOperations = new FakeFileSystemOperations();
|
||||
const fileOperations = new FileOperations(
|
||||
new Logger(),
|
||||
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
fileSystemOperations,
|
||||
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
);
|
||||
|
|
@ -159,7 +153,7 @@ describe("File operations", () => {
|
|||
const fileSystemOperations = new FakeFileSystemOperations();
|
||||
const fileOperations = new FileOperations(
|
||||
new Logger(),
|
||||
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
fileSystemOperations,
|
||||
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
);
|
||||
|
|
@ -178,7 +172,7 @@ describe("File operations", () => {
|
|||
const fileSystemOperations = new FakeFileSystemOperations();
|
||||
const fileOperations = new FileOperations(
|
||||
new Logger(),
|
||||
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
fileSystemOperations,
|
||||
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
);
|
||||
|
|
@ -207,7 +201,7 @@ describe("File operations", () => {
|
|||
const fileSystemOperations = new FakeFileSystemOperations();
|
||||
const fileOperations = new FileOperations(
|
||||
new Logger(),
|
||||
new MockDatabase() as Database, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
new MockVfs() as VirtualFilesystem, // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
fileSystemOperations,
|
||||
new MockServerConfig() as ServerConfig // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
);
|
||||
|
|
|
|||
|
|
@ -1,12 +1,15 @@
|
|||
import type { Logger } from "../tracing/logger";
|
||||
import type { FileSystemOperations } from "./filesystem-operations";
|
||||
import type { Database, RelativePath } from "../persistence/database";
|
||||
import type { RelativePath } from "../persistence/database";
|
||||
import type { VirtualFilesystem } from "../persistence/vfs";
|
||||
import { SafeFileSystemOperations } from "./safe-filesystem-operations";
|
||||
import type { TextWithCursors } from "reconcile-text";
|
||||
import { reconcile } from "reconcile-text";
|
||||
import { isFileTypeMergable } from "../utils/is-file-type-mergable";
|
||||
import { isBinary } from "../utils/is-binary";
|
||||
import { decodeText, normalizeToUtf8 } from "../utils/decode-text";
|
||||
import type { ServerConfig } from "../services/server-config";
|
||||
import { validateRelativePath } from "../utils/validate-relative-path";
|
||||
|
||||
export class FileOperations {
|
||||
private static readonly PARENTHESES_REGEX = / \((?<count>\d+)\)$/;
|
||||
|
|
@ -14,7 +17,7 @@ export class FileOperations {
|
|||
|
||||
public constructor(
|
||||
private readonly logger: Logger,
|
||||
private readonly database: Database,
|
||||
private readonly vfs: VirtualFilesystem,
|
||||
fs: FileSystemOperations,
|
||||
private readonly serverConfig: ServerConfig,
|
||||
private readonly nativeLineEndings = "\n"
|
||||
|
|
@ -41,7 +44,8 @@ export class FileOperations {
|
|||
}
|
||||
|
||||
public async read(path: RelativePath): Promise<Uint8Array> {
|
||||
return this.fromNativeLineEndings(await this.fs.read(path));
|
||||
const raw = await this.fs.read(path);
|
||||
return this.fromNativeLineEndings(normalizeToUtf8(raw));
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -54,25 +58,96 @@ export class FileOperations {
|
|||
path: RelativePath,
|
||||
newContent: Uint8Array
|
||||
): Promise<void> {
|
||||
validateRelativePath(path);
|
||||
await this.ensureClearPath(path);
|
||||
return this.fs.write(path, this.toNativeLineEndings(newContent));
|
||||
}
|
||||
|
||||
public async ensureClearPath(path: RelativePath): Promise<void> {
|
||||
if (await this.fs.exists(path)) {
|
||||
public async ensureClearPath(path: RelativePath): Promise<RelativePath | undefined> {
|
||||
validateRelativePath(path);
|
||||
// Acquire the lock on `path` first, then check existence inside the
|
||||
// lock. The previous code checked exists() before locking, which
|
||||
// created a TOCTOU race: two concurrent calls could both see the
|
||||
// file as existing, but the second one would try to rename a file
|
||||
// that was already moved by the first.
|
||||
await this.fs.waitForLock(path);
|
||||
try {
|
||||
return await this.ensureClearPathLocked(path);
|
||||
} finally {
|
||||
this.fs.unlock(path);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Internal implementation of `ensureClearPath` that assumes the caller
|
||||
* already holds the file-level lock on `path`. This allows callers like
|
||||
* `move()` to keep the lock held across both the clear and the subsequent
|
||||
* rename, closing the race window where another operation could create a
|
||||
* file at `path` between the two steps.
|
||||
*/
|
||||
private async ensureClearPathLocked(
|
||||
path: RelativePath
|
||||
): Promise<RelativePath | undefined> {
|
||||
if (await this.fs.exists(path, true)) {
|
||||
const deconflictedPath = await this.deconflictPath(path);
|
||||
try {
|
||||
this.logger.debug(
|
||||
`Didn't expect ${path} to exist, deconflicting by moving it to '${deconflictedPath}'`
|
||||
);
|
||||
|
||||
this.database.move(path, deconflictedPath);
|
||||
// deconflictedPath is already locked via tryLock in
|
||||
// deconflictPath(), so we pass skipLock=true to the
|
||||
// rename to avoid deadlocking on the destination lock.
|
||||
await this.fs.rename(path, deconflictedPath, true);
|
||||
try {
|
||||
this.vfs.move(path, deconflictedPath);
|
||||
|
||||
// Tell the sync system this displacement is system-initiated
|
||||
// (not a user rename) by setting remoteRelativePath to the
|
||||
// deconflicted path. This makes the check in
|
||||
// syncLocallyUpdatedFile (remoteRelativePath === relativePath)
|
||||
// pass, preventing the displacement from being uploaded as a
|
||||
// rename to the server. Without this, the rename event from
|
||||
// fs.rename() triggers an update with the deconflicted path,
|
||||
// the server deconflicts further, and an infinite cascade
|
||||
// ensues. The force:true content-match shortcut ensures that
|
||||
// when the server eventually broadcasts the document's real
|
||||
// path, the client just updates metadata without moving the
|
||||
// file back.
|
||||
const displacedDoc =
|
||||
this.vfs.getByPath(deconflictedPath);
|
||||
if (
|
||||
displacedDoc?.state === "tracked" &&
|
||||
displacedDoc.remoteRelativePath !== undefined
|
||||
) {
|
||||
displacedDoc.remoteRelativePath =
|
||||
deconflictedPath;
|
||||
}
|
||||
} catch (e) {
|
||||
// vfs.move() failed (e.g., a non-deleted document
|
||||
// already exists at deconflictedPath). Revert the
|
||||
// filesystem rename to keep file and VFS
|
||||
// consistent. If the revert also fails, log it —
|
||||
// scheduleSyncForOfflineChanges will reconcile.
|
||||
this.logger.warn(
|
||||
`vfs.move(${path}, ${deconflictedPath}) failed in ensureClearPath: ${e}, reverting filesystem rename`
|
||||
);
|
||||
try {
|
||||
await this.fs.rename(deconflictedPath, path, true);
|
||||
} catch (revertError) {
|
||||
this.logger.warn(
|
||||
`Failed to revert filesystem rename from ${deconflictedPath} to ${path}: ${revertError}`
|
||||
);
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
} finally {
|
||||
this.fs.unlock(deconflictedPath);
|
||||
}
|
||||
return deconflictedPath;
|
||||
} else {
|
||||
await this.createParentDirectories(path);
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -87,6 +162,7 @@ export class FileOperations {
|
|||
expectedContent: Uint8Array,
|
||||
newContent: Uint8Array
|
||||
): Promise<void> {
|
||||
validateRelativePath(path);
|
||||
if (!(await this.fs.exists(path))) {
|
||||
this.logger.debug(
|
||||
`The caller assumed ${path} exists, but it no longer, so we wont recreate it`
|
||||
|
|
@ -113,8 +189,10 @@ export class FileOperations {
|
|||
return;
|
||||
}
|
||||
|
||||
const expectedText = new TextDecoder().decode(expectedContent); // this comes from a previous read which must only have \n line endings
|
||||
const newText = new TextDecoder().decode(newContent); // this comes from the server which stores text with \n line endings
|
||||
const expectedText = (decodeText(expectedContent) ?? "").normalize(
|
||||
"NFC"
|
||||
); // this comes from a previous read which must only have \n line endings
|
||||
const newText = (decodeText(newContent) ?? "").normalize("NFC"); // this comes from the server which stores text with \n line endings
|
||||
|
||||
await this.fs.atomicUpdateText(
|
||||
path,
|
||||
|
|
@ -123,12 +201,29 @@ export class FileOperations {
|
|||
`Performing a 3-way merge for ${path} with the expected content`
|
||||
);
|
||||
|
||||
text = text.replaceAll(this.nativeLineEndings, "\n");
|
||||
const merged = reconcile(
|
||||
expectedText,
|
||||
{ text, cursors },
|
||||
newText
|
||||
);
|
||||
text = text
|
||||
.replaceAll(this.nativeLineEndings, "\n")
|
||||
.normalize("NFC");
|
||||
|
||||
let merged: TextWithCursors;
|
||||
try {
|
||||
merged = reconcile(
|
||||
expectedText,
|
||||
{ text, cursors },
|
||||
newText,
|
||||
"Markdown"
|
||||
);
|
||||
} catch {
|
||||
// 3-way merge failed (e.g., content was fully replaced
|
||||
// by another agent). Save the local content as a conflict
|
||||
// file before overwriting with the server's content, so
|
||||
// the user's edits are never silently lost.
|
||||
this.logger.info(
|
||||
`3-way merge failed for ${path}, saving local content as conflict file and using server content`
|
||||
);
|
||||
this.saveConflictFile(path, text);
|
||||
merged = { text: newText, cursors: [] };
|
||||
}
|
||||
|
||||
const resultText = merged.text.replaceAll(
|
||||
"\n",
|
||||
|
|
@ -144,6 +239,7 @@ export class FileOperations {
|
|||
}
|
||||
|
||||
public async delete(path: RelativePath): Promise<void> {
|
||||
validateRelativePath(path);
|
||||
if (await this.exists(path)) {
|
||||
await this.fs.delete(path);
|
||||
await this.deletingEmptyParentDirectoriesOfDeletedFile(path);
|
||||
|
|
@ -164,13 +260,46 @@ export class FileOperations {
|
|||
oldPath: RelativePath,
|
||||
newPath: RelativePath
|
||||
): Promise<void> {
|
||||
validateRelativePath(oldPath);
|
||||
validateRelativePath(newPath);
|
||||
if (oldPath === newPath) {
|
||||
return;
|
||||
}
|
||||
|
||||
await this.ensureClearPath(newPath);
|
||||
this.database.move(oldPath, newPath);
|
||||
await this.fs.rename(oldPath, newPath);
|
||||
// Hold the newPath lock across both ensureClearPath and rename.
|
||||
// Without this, another operation could create a file at newPath
|
||||
// between ensureClearPath releasing the lock and rename acquiring
|
||||
// it, causing the rename to silently overwrite the new file.
|
||||
await this.fs.waitForLock(newPath);
|
||||
try {
|
||||
await this.ensureClearPathLocked(newPath);
|
||||
// skipLock=true because we already hold the newPath lock.
|
||||
// The oldPath lock is not needed; sync operations run
|
||||
// sequentially so no concurrent operation can race on paths.
|
||||
await this.fs.rename(oldPath, newPath, true);
|
||||
} finally {
|
||||
this.fs.unlock(newPath);
|
||||
}
|
||||
try {
|
||||
this.vfs.move(oldPath, newPath);
|
||||
} catch (e) {
|
||||
// vfs.move() failed (e.g., a non-deleted document already
|
||||
// exists at newPath). Revert the filesystem rename to keep the
|
||||
// file and VFS consistent. If the revert also fails, log
|
||||
// it — scheduleSyncForOfflineChanges will reconcile on the
|
||||
// next cycle.
|
||||
this.logger.warn(
|
||||
`vfs.move(${oldPath}, ${newPath}) failed: ${e}, reverting filesystem rename`
|
||||
);
|
||||
try {
|
||||
await this.fs.rename(newPath, oldPath);
|
||||
} catch (revertError) {
|
||||
this.logger.warn(
|
||||
`Failed to revert filesystem rename from ${newPath} to ${oldPath}: ${revertError}`
|
||||
);
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
|
||||
await this.deletingEmptyParentDirectoriesOfDeletedFile(oldPath);
|
||||
}
|
||||
|
|
@ -204,25 +333,60 @@ export class FileOperations {
|
|||
}
|
||||
|
||||
private fromNativeLineEndings(content: Uint8Array): Uint8Array {
|
||||
if (isBinary(content)) {
|
||||
const text = decodeText(content);
|
||||
if (text === undefined) {
|
||||
return content;
|
||||
}
|
||||
|
||||
const decoder = new TextDecoder("utf-8");
|
||||
let text = decoder.decode(content);
|
||||
text = text.replaceAll(this.nativeLineEndings, "\n");
|
||||
return new TextEncoder().encode(text);
|
||||
const normalized = text.replaceAll(this.nativeLineEndings, "\n");
|
||||
return new TextEncoder().encode(normalized);
|
||||
}
|
||||
|
||||
private toNativeLineEndings(content: Uint8Array): Uint8Array {
|
||||
if (isBinary(content)) {
|
||||
const text = decodeText(content);
|
||||
if (text === undefined) {
|
||||
return content;
|
||||
}
|
||||
|
||||
const decoder = new TextDecoder("utf-8");
|
||||
let text = decoder.decode(content);
|
||||
text = text.replaceAll("\n", this.nativeLineEndings);
|
||||
return new TextEncoder().encode(text);
|
||||
const normalized = text.replaceAll("\n", this.nativeLineEndings);
|
||||
return new TextEncoder().encode(normalized);
|
||||
}
|
||||
|
||||
/**
|
||||
* Save the local content of a file as a conflict file when 3-way merge
|
||||
* fails, so the user's edits are never silently lost. The conflict file
|
||||
* is created at a deconflicted path (e.g., "file (conflict 1).md").
|
||||
*
|
||||
* This is fire-and-forget — errors are logged but do not prevent the
|
||||
* caller from proceeding with the server's content.
|
||||
*/
|
||||
private saveConflictFile(
|
||||
path: RelativePath,
|
||||
localContent: string
|
||||
): void {
|
||||
const contentBytes = new TextEncoder().encode(
|
||||
localContent.replaceAll("\n", this.nativeLineEndings)
|
||||
);
|
||||
// Fire-and-forget: we don't want a failed conflict-save to prevent
|
||||
// the server content from being written.
|
||||
void (async () => {
|
||||
try {
|
||||
const conflictPath =
|
||||
await this.deconflictPath(path);
|
||||
try {
|
||||
await this.fs.write(conflictPath, contentBytes);
|
||||
this.logger.info(
|
||||
`Saved local content as conflict file: ${conflictPath}`
|
||||
);
|
||||
} finally {
|
||||
this.fs.unlock(conflictPath);
|
||||
}
|
||||
} catch (e) {
|
||||
this.logger.warn(
|
||||
`Failed to save conflict file for ${path}: ${e}`
|
||||
);
|
||||
}
|
||||
})();
|
||||
}
|
||||
|
||||
private async createParentDirectories(path: string): Promise<void> {
|
||||
|
|
@ -275,10 +439,12 @@ export class FileOperations {
|
|||
|
||||
// Avoid multiple deconflictPath calls returning the same path
|
||||
if (this.fs.tryLock(newName)) {
|
||||
const newDocument =
|
||||
this.database.getLatestDocumentByRelativePath(newName);
|
||||
// getByPath only returns live docs (pending/tracked), not
|
||||
// deleted-locally ones, so a non-undefined result means
|
||||
// the path is occupied.
|
||||
const existingDoc = this.vfs.getByPath(newName);
|
||||
if (
|
||||
newDocument?.isDeleted === false || // the document might have been confirmed by the server at a new path but haven't yet moved there locally
|
||||
existingDoc !== undefined || // the document might have been confirmed by the server at a new path but haven't yet moved there locally
|
||||
(await this.fs.exists(newName, true))
|
||||
) {
|
||||
this.fs.unlock(newName);
|
||||
|
|
|
|||
|
|
@ -1,26 +1,14 @@
|
|||
import type { Logger } from "../tracing/logger";
|
||||
import { EMPTY_HASH } from "../utils/hash";
|
||||
import { CoveredValues } from "../utils/data-structures/min-covered";
|
||||
import { awaitAll } from "../utils/await-all";
|
||||
import { removeFromArray } from "../utils/remove-from-array";
|
||||
|
||||
export type VaultUpdateId = number;
|
||||
export type DocumentId = string;
|
||||
export type RelativePath = string;
|
||||
|
||||
export interface DocumentMetadata {
|
||||
documentId: DocumentId;
|
||||
parentVersionId: VaultUpdateId;
|
||||
hash: string;
|
||||
remoteRelativePath?: RelativePath;
|
||||
}
|
||||
|
||||
export interface StoredDocumentMetadata {
|
||||
relativePath: RelativePath;
|
||||
documentId: DocumentId;
|
||||
parentVersionId: VaultUpdateId;
|
||||
remoteRelativePath?: RelativePath;
|
||||
hash: string;
|
||||
isDeleted?: boolean;
|
||||
}
|
||||
|
||||
export interface StoredPendingDocument {
|
||||
|
|
@ -34,374 +22,3 @@ export interface StoredDatabase {
|
|||
pendingDocuments?: StoredPendingDocument[];
|
||||
lastSeenUpdateId: VaultUpdateId | undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Represents a document in the database.
|
||||
*
|
||||
* It is mutable and its content should always represent the latest
|
||||
* state of the document on disk based on the update events we have seen.
|
||||
*/
|
||||
export interface DocumentRecord {
|
||||
relativePath: RelativePath;
|
||||
metadata: DocumentMetadata | undefined;
|
||||
isDeleted: boolean;
|
||||
parallelVersion: number;
|
||||
/** The path when this pending document was first created locally.
|
||||
* Survives renames so we can match it against server responses
|
||||
* when a create request succeeded but the response was lost. */
|
||||
originalCreationPath?: RelativePath;
|
||||
idempotencyKey?: string;
|
||||
}
|
||||
|
||||
export class Database {
|
||||
private documents: DocumentRecord[];
|
||||
private lastSeenUpdateIds: CoveredValues;
|
||||
|
||||
public constructor(
|
||||
private readonly logger: Logger,
|
||||
initialState: Partial<StoredDatabase> | undefined,
|
||||
private readonly saveData: (data: StoredDatabase) => Promise<void>
|
||||
) {
|
||||
initialState ??= {};
|
||||
|
||||
const validDocuments = (initialState.documents ?? []).filter(
|
||||
(doc) =>
|
||||
this.validateStoredField(doc, "relativePath", "string") &&
|
||||
this.validateStoredField(doc, "documentId", "string") &&
|
||||
this.validateStoredField(doc, "parentVersionId", "number")
|
||||
);
|
||||
|
||||
this.documents = validDocuments.map(
|
||||
({ relativePath, ...metadata }) => ({
|
||||
relativePath,
|
||||
metadata,
|
||||
isDeleted: false,
|
||||
parallelVersion: 0
|
||||
})
|
||||
);
|
||||
|
||||
const validPendingDocuments = (
|
||||
initialState.pendingDocuments ?? []
|
||||
).filter(
|
||||
(doc) =>
|
||||
this.validateStoredField(doc, "relativePath", "string") &&
|
||||
this.validateStoredField(doc, "idempotencyKey", "string")
|
||||
);
|
||||
|
||||
for (const pending of validPendingDocuments) {
|
||||
const existing = this.getLatestDocumentByRelativePath(
|
||||
pending.relativePath
|
||||
);
|
||||
this.documents.push({
|
||||
relativePath: pending.relativePath,
|
||||
metadata: undefined,
|
||||
isDeleted: false,
|
||||
parallelVersion:
|
||||
existing !== undefined
|
||||
? existing.parallelVersion + 1
|
||||
: 0,
|
||||
originalCreationPath: pending.originalCreationPath,
|
||||
idempotencyKey: pending.idempotencyKey
|
||||
});
|
||||
}
|
||||
|
||||
this.ensureConsistency();
|
||||
this.logger.debug(`Loaded ${this.documents.length} documents`);
|
||||
|
||||
const { lastSeenUpdateId } = initialState;
|
||||
this.logger.debug(`Loaded last seen update id: ${lastSeenUpdateId}`);
|
||||
this.lastSeenUpdateIds = new CoveredValues(
|
||||
Math.max(0, lastSeenUpdateId ?? 0) // the first updateId will be 1 which is the first integer after -1
|
||||
);
|
||||
|
||||
this.documents.forEach((doc) => {
|
||||
this.lastSeenUpdateIds.add(doc.metadata?.parentVersionId);
|
||||
});
|
||||
}
|
||||
|
||||
private validateStoredField(
|
||||
doc: object,
|
||||
field: string,
|
||||
expectedType: "string" | "number"
|
||||
): boolean {
|
||||
const value = (doc as Record<string, unknown>)[field];
|
||||
if (
|
||||
typeof value !== expectedType ||
|
||||
(expectedType === "string" && !value) ||
|
||||
(expectedType === "number" && isNaN(value as number))
|
||||
) {
|
||||
this.logger.warn(
|
||||
`Skipping stored document with invalid ${field}: ${JSON.stringify(doc)}`
|
||||
);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
public get length(): number {
|
||||
return this.documents.length;
|
||||
}
|
||||
|
||||
public get resolvedDocuments(): DocumentRecord[] {
|
||||
const paths = new Map<string, DocumentRecord[]>();
|
||||
this.documents
|
||||
// eslint-disable-next-line no-restricted-syntax -- Type narrowing, not removing a specific item
|
||||
.filter(({ metadata }) => metadata !== undefined)
|
||||
.forEach((record) =>
|
||||
paths.set(record.relativePath, [
|
||||
record,
|
||||
...(paths.get(record.relativePath) ?? [])
|
||||
])
|
||||
);
|
||||
|
||||
return Array.from(paths.values()).map((records) => {
|
||||
records.sort(
|
||||
(a, b) => b.parallelVersion - a.parallelVersion // descending
|
||||
);
|
||||
|
||||
if (
|
||||
records.length > 1 &&
|
||||
records.some((current, i) =>
|
||||
i === 0
|
||||
? false
|
||||
: records[i - 1].parallelVersion ===
|
||||
current.parallelVersion
|
||||
)
|
||||
) {
|
||||
throw new Error(
|
||||
`Multiple documents with the same parallel version and path at ${records[0].relativePath}`
|
||||
);
|
||||
}
|
||||
return records[0];
|
||||
});
|
||||
}
|
||||
|
||||
public get pendingDocuments(): DocumentRecord[] {
|
||||
return this.documents.filter(
|
||||
(doc) => doc.metadata === undefined && !doc.isDeleted
|
||||
);
|
||||
}
|
||||
|
||||
public updateDocumentMetadata(
|
||||
metadata: {
|
||||
documentId: DocumentId;
|
||||
parentVersionId: VaultUpdateId;
|
||||
hash: string;
|
||||
remoteRelativePath: RelativePath;
|
||||
},
|
||||
target: DocumentRecord
|
||||
): void {
|
||||
if (!this.documents.includes(target)) {
|
||||
throw new Error("Document not found in database");
|
||||
}
|
||||
|
||||
this.logger.debug(
|
||||
`Updating document metadata for ${target.relativePath} from ${JSON.stringify(
|
||||
target.metadata,
|
||||
null,
|
||||
2
|
||||
)} to ${JSON.stringify(metadata, null, 2)}`
|
||||
);
|
||||
|
||||
target.metadata = metadata;
|
||||
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
public getLatestDocumentByRelativePath(
|
||||
target: RelativePath
|
||||
): DocumentRecord | undefined {
|
||||
const candidates = this.documents.filter(
|
||||
({ relativePath }) => relativePath === target
|
||||
);
|
||||
candidates.sort((a, b) => b.parallelVersion - a.parallelVersion); // descending
|
||||
return candidates[0];
|
||||
}
|
||||
|
||||
public createNewPendingDocument(
|
||||
relativePath: RelativePath
|
||||
): DocumentRecord {
|
||||
this.logger.debug(`Creating new pending document: ${relativePath}`);
|
||||
const previousEntry =
|
||||
this.getLatestDocumentByRelativePath(relativePath);
|
||||
|
||||
const entry: DocumentRecord = {
|
||||
relativePath,
|
||||
metadata: undefined,
|
||||
isDeleted: false,
|
||||
parallelVersion:
|
||||
previousEntry?.parallelVersion === undefined
|
||||
? 0
|
||||
: previousEntry.parallelVersion + 1,
|
||||
originalCreationPath: relativePath,
|
||||
idempotencyKey: crypto.randomUUID()
|
||||
};
|
||||
|
||||
this.documents.push(entry);
|
||||
|
||||
// Save without consistency check — pending docs can't violate
|
||||
// the documentId uniqueness invariant since they have no metadata.
|
||||
void this.save().catch((error: unknown) => {
|
||||
this.logger.error(`Error saving data: ${error}`);
|
||||
});
|
||||
|
||||
return entry;
|
||||
}
|
||||
|
||||
public getDocumentByDocumentId(
|
||||
target: DocumentId
|
||||
): DocumentRecord | undefined {
|
||||
return this.documents.find(
|
||||
({ metadata }) => metadata?.documentId === target
|
||||
);
|
||||
}
|
||||
|
||||
public move(
|
||||
oldRelativePath: RelativePath,
|
||||
newRelativePath: RelativePath
|
||||
): void {
|
||||
const oldDocument =
|
||||
this.getLatestDocumentByRelativePath(oldRelativePath);
|
||||
|
||||
if (oldDocument === undefined) {
|
||||
return;
|
||||
}
|
||||
|
||||
const newDocument =
|
||||
this.getLatestDocumentByRelativePath(newRelativePath);
|
||||
if (newDocument?.isDeleted === false) {
|
||||
throw new Error(
|
||||
`Document already exists at new location: ${newRelativePath}`
|
||||
);
|
||||
}
|
||||
|
||||
oldDocument.relativePath = newRelativePath;
|
||||
// We might be in a strange state where the target of the move has just got deleted,
|
||||
// however, its metadata might already have a bunch of updates queued up for
|
||||
// the document at the new location. We need to keep these updates.
|
||||
oldDocument.parallelVersion =
|
||||
newDocument !== undefined ? newDocument.parallelVersion + 1 : 0;
|
||||
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
public delete(relativePath: RelativePath): void {
|
||||
const candidate = this.getLatestDocumentByRelativePath(relativePath);
|
||||
if (candidate === undefined) {
|
||||
return;
|
||||
}
|
||||
candidate.isDeleted = true;
|
||||
}
|
||||
|
||||
public removeDocument(target: DocumentRecord): void {
|
||||
removeFromArray(this.documents, target);
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
public containsDocument(target: DocumentRecord): boolean {
|
||||
return this.documents.includes(target);
|
||||
}
|
||||
|
||||
public getLastSeenUpdateId(): VaultUpdateId {
|
||||
return this.lastSeenUpdateIds.min;
|
||||
}
|
||||
|
||||
public addSeenUpdateId(value: number): void {
|
||||
const previousMin = this.lastSeenUpdateIds.min;
|
||||
this.lastSeenUpdateIds.add(value);
|
||||
if (previousMin !== this.lastSeenUpdateIds.min) {
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
}
|
||||
|
||||
public setLastSeenUpdateId(value: number): void {
|
||||
this.lastSeenUpdateIds.min = value;
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
public reset(): void {
|
||||
this.documents = [];
|
||||
this.lastSeenUpdateIds = new CoveredValues(
|
||||
0 // the first updateId will be 1 which is the first integer after -1
|
||||
);
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
public async save(): Promise<void> {
|
||||
return this.saveData({
|
||||
documents: this.resolvedDocuments.map(
|
||||
({ relativePath, metadata }) => ({
|
||||
relativePath,
|
||||
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
|
||||
...metadata! // filtered to only docs with metadata set
|
||||
})
|
||||
),
|
||||
pendingDocuments: this.pendingDocuments.map(
|
||||
({ relativePath, idempotencyKey, originalCreationPath }) => ({
|
||||
relativePath,
|
||||
idempotencyKey: idempotencyKey!,
|
||||
originalCreationPath: originalCreationPath!
|
||||
})
|
||||
),
|
||||
lastSeenUpdateId: this.lastSeenUpdateIds.min
|
||||
});
|
||||
}
|
||||
|
||||
private ensureConsistency(): void {
|
||||
// Check for duplicate documentIds across ALL documents with metadata,
|
||||
// not just the deduplicated resolvedDocuments view. A duplicate on a
|
||||
// lower-parallelVersion record would otherwise go undetected.
|
||||
const allWithMetadata = this.documents
|
||||
// eslint-disable-next-line no-restricted-syntax -- Type narrowing, not removing a specific item
|
||||
.filter((d) => d.metadata !== undefined);
|
||||
const documentIdSet = new Set<string>();
|
||||
for (const doc of allWithMetadata) {
|
||||
// eslint-disable-next-line @typescript-eslint/no-non-null-assertion
|
||||
const docId = doc.metadata!.documentId;
|
||||
if (documentIdSet.has(docId)) {
|
||||
throw new Error(
|
||||
`Duplicate documentId ${docId} found in database`
|
||||
);
|
||||
}
|
||||
documentIdSet.add(docId);
|
||||
}
|
||||
|
||||
// Also check the deduplicated view for path-level invariants
|
||||
const idToPath = new Map<string, string[]>();
|
||||
|
||||
this.resolvedDocuments.forEach(({ relativePath, metadata }) => {
|
||||
if (metadata === undefined) {
|
||||
return;
|
||||
}
|
||||
idToPath.set(metadata.documentId, [
|
||||
...(idToPath.get(metadata.documentId) ?? []),
|
||||
relativePath
|
||||
]);
|
||||
});
|
||||
|
||||
const duplicates = Array.from(idToPath.entries())
|
||||
.filter(([_, paths]) => paths.length > 1)
|
||||
.map(([id, paths]) => {
|
||||
let details = "";
|
||||
for (const path of paths) {
|
||||
const doc = this.getLatestDocumentByRelativePath(path);
|
||||
details += `\n- ${JSON.stringify(doc, null, 2)}`;
|
||||
}
|
||||
return `${id} (${paths.join(", ")}): ${details}`;
|
||||
});
|
||||
|
||||
if (duplicates.length > 0) {
|
||||
throw new Error(
|
||||
"Document IDs are not unique, found duplicates: " +
|
||||
duplicates.join("; ")
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
private saveInTheBackground(): void {
|
||||
this.ensureConsistency();
|
||||
void this.save().catch((error: unknown) => {
|
||||
this.logger.error(`Error saving data: ${error}`);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,7 +6,6 @@ export interface SyncSettings {
|
|||
remoteUri: string;
|
||||
token: string;
|
||||
vaultName: string;
|
||||
syncConcurrency: number;
|
||||
isSyncEnabled: boolean;
|
||||
maxFileSizeMB: number;
|
||||
ignorePatterns: string[];
|
||||
|
|
@ -21,7 +20,6 @@ export const DEFAULT_SETTINGS: SyncSettings = {
|
|||
remoteUri: "",
|
||||
token: "",
|
||||
vaultName: "default",
|
||||
syncConcurrency: 1,
|
||||
isSyncEnabled: false,
|
||||
maxFileSizeMB: 10,
|
||||
ignorePatterns: [],
|
||||
|
|
|
|||
820
frontend/sync-client/src/persistence/vfs.ts
Normal file
820
frontend/sync-client/src/persistence/vfs.ts
Normal file
|
|
@ -0,0 +1,820 @@
|
|||
import type { Logger } from "../tracing/logger";
|
||||
import { EMPTY_HASH } from "../utils/hash";
|
||||
import { CoveredValues } from "../utils/data-structures/min-covered";
|
||||
import type {
|
||||
StoredDatabase,
|
||||
StoredDocumentMetadata,
|
||||
StoredPendingDocument,
|
||||
VaultUpdateId,
|
||||
DocumentId,
|
||||
RelativePath
|
||||
} from "./database";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Document state types (discriminated union)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface PendingDocument {
|
||||
readonly state: "pending";
|
||||
relativePath: string;
|
||||
readonly idempotencyKey: string;
|
||||
readonly originalCreationPath: string;
|
||||
}
|
||||
|
||||
export interface TrackedDocument {
|
||||
readonly state: "tracked";
|
||||
relativePath: string;
|
||||
documentId: string;
|
||||
serverVersion: number;
|
||||
localHash: string;
|
||||
remoteRelativePath: string;
|
||||
idempotencyKey?: string;
|
||||
}
|
||||
|
||||
export interface DeletedLocallyDocument {
|
||||
readonly state: "deleted-locally";
|
||||
relativePath: string;
|
||||
readonly documentId: string;
|
||||
readonly serverVersion: number;
|
||||
readonly remoteRelativePath: string;
|
||||
}
|
||||
|
||||
export type VirtualDocument =
|
||||
| PendingDocument
|
||||
| TrackedDocument
|
||||
| DeletedLocallyDocument;
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Reconciliation result
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export interface ReconciliationResult {
|
||||
newFiles: string[];
|
||||
modifiedFiles: { path: string; documentId: string }[];
|
||||
missingFiles: VirtualDocument[];
|
||||
movedFiles: { document: TrackedDocument; newPath: string }[];
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// VirtualFilesystem
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export class VirtualFilesystem {
|
||||
/** One live document per path (pending or tracked, NOT deleted-locally). */
|
||||
private readonly pathIndex = new Map<string, VirtualDocument>();
|
||||
|
||||
/** All documents that have a documentId (tracked + deleted-locally). */
|
||||
private readonly documentIdIndex = new Map<string, VirtualDocument>();
|
||||
|
||||
/** Pending documents by idempotency key. */
|
||||
private readonly idempotencyKeyIndex = new Map<string, PendingDocument>();
|
||||
|
||||
private lastSeenUpdateIds: CoveredValues;
|
||||
|
||||
private pendingSave: Promise<void> = Promise.resolve();
|
||||
|
||||
public constructor(
|
||||
private readonly logger: Logger,
|
||||
initialState: Partial<StoredDatabase> | undefined,
|
||||
private readonly saveData: (data: StoredDatabase) => Promise<void>
|
||||
) {
|
||||
const state: Partial<StoredDatabase> = initialState ?? {};
|
||||
|
||||
const validDocuments = (state.documents ?? []).filter(
|
||||
(doc) =>
|
||||
this.validateStoredField(doc, "relativePath", "string") &&
|
||||
this.validateStoredField(doc, "documentId", "string") &&
|
||||
this.validateStoredField(doc, "parentVersionId", "number")
|
||||
);
|
||||
|
||||
for (const stored of validDocuments) {
|
||||
if (stored.isDeleted === true) {
|
||||
const doc: DeletedLocallyDocument = {
|
||||
state: "deleted-locally",
|
||||
relativePath: stored.relativePath,
|
||||
documentId: stored.documentId,
|
||||
serverVersion: stored.parentVersionId,
|
||||
remoteRelativePath:
|
||||
stored.remoteRelativePath ?? stored.relativePath
|
||||
};
|
||||
// deleted-locally docs go into documentIdIndex only
|
||||
this.documentIdIndex.set(doc.documentId, doc);
|
||||
} else {
|
||||
const doc: TrackedDocument = {
|
||||
state: "tracked",
|
||||
relativePath: stored.relativePath,
|
||||
documentId: stored.documentId,
|
||||
serverVersion: stored.parentVersionId,
|
||||
localHash: stored.hash,
|
||||
remoteRelativePath:
|
||||
stored.remoteRelativePath ?? stored.relativePath
|
||||
};
|
||||
// If two stored documents have the same path, last one wins
|
||||
// (matches old behavior where highest parallelVersion wins)
|
||||
this.pathIndex.set(doc.relativePath, doc);
|
||||
this.documentIdIndex.set(doc.documentId, doc);
|
||||
}
|
||||
}
|
||||
|
||||
const validPendingDocuments = (state.pendingDocuments ?? []).filter(
|
||||
(doc) =>
|
||||
this.validateStoredField(doc, "relativePath", "string") &&
|
||||
this.validateStoredField(doc, "idempotencyKey", "string")
|
||||
);
|
||||
|
||||
for (const stored of validPendingDocuments) {
|
||||
// If a live doc already exists at this path, skip the pending one
|
||||
// only if the live doc is tracked (has metadata). If a pending doc
|
||||
// already exists, skip duplicates.
|
||||
const existing = this.pathIndex.get(stored.relativePath);
|
||||
if (existing?.state === "pending") {
|
||||
this.logger.debug(
|
||||
`Skipping duplicate pending document at ${stored.relativePath}`
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
const doc: PendingDocument = {
|
||||
state: "pending",
|
||||
relativePath: stored.relativePath,
|
||||
idempotencyKey: stored.idempotencyKey,
|
||||
originalCreationPath:
|
||||
stored.originalCreationPath ?? stored.relativePath
|
||||
};
|
||||
|
||||
// A pending doc at a path where a tracked doc exists: the pending
|
||||
// doc takes precedence in pathIndex (mirrors old behavior where
|
||||
// pending has higher parallelVersion).
|
||||
this.pathIndex.set(doc.relativePath, doc);
|
||||
this.idempotencyKeyIndex.set(doc.idempotencyKey, doc);
|
||||
}
|
||||
|
||||
this.ensureConsistency();
|
||||
|
||||
const totalDocs =
|
||||
this.pathIndex.size + this.deletedLocallyDocuments().length;
|
||||
this.logger.debug(`Loaded ${totalDocs} documents`);
|
||||
|
||||
const { lastSeenUpdateId } = state;
|
||||
this.logger.debug(`Loaded last seen update id: ${lastSeenUpdateId}`);
|
||||
this.lastSeenUpdateIds = new CoveredValues(
|
||||
Math.max(0, lastSeenUpdateId ?? 0)
|
||||
);
|
||||
|
||||
// Seed CoveredValues with known server versions
|
||||
for (const doc of this.documentIdIndex.values()) {
|
||||
if (doc.state === "tracked") {
|
||||
this.lastSeenUpdateIds.add(doc.serverVersion);
|
||||
} else if (doc.state === "deleted-locally") {
|
||||
this.lastSeenUpdateIds.add(doc.serverVersion);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Validation helper
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private validateStoredField(
|
||||
doc: object,
|
||||
field: string,
|
||||
expectedType: "string" | "number"
|
||||
): boolean {
|
||||
const value = (doc as Record<string, unknown>)[field];
|
||||
if (
|
||||
typeof value !== expectedType ||
|
||||
(expectedType === "string" && !value) ||
|
||||
(expectedType === "number" && isNaN(value as number))
|
||||
) {
|
||||
this.logger.warn(
|
||||
`Skipping stored document with invalid ${field}: ${JSON.stringify(doc)}`
|
||||
);
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Queries
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public getByPath(path: string): VirtualDocument | undefined {
|
||||
return this.pathIndex.get(path);
|
||||
}
|
||||
|
||||
public getByDocumentId(id: string): VirtualDocument | undefined {
|
||||
return this.documentIdIndex.get(id);
|
||||
}
|
||||
|
||||
public getByIdempotencyKey(key: string): PendingDocument | undefined {
|
||||
return this.idempotencyKeyIndex.get(key);
|
||||
}
|
||||
|
||||
public trackedDocuments(): TrackedDocument[] {
|
||||
const result: TrackedDocument[] = [];
|
||||
for (const doc of this.pathIndex.values()) {
|
||||
if (doc.state === "tracked") {
|
||||
result.push(doc);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
public pendingDocuments(): PendingDocument[] {
|
||||
const result: PendingDocument[] = [];
|
||||
for (const doc of this.pathIndex.values()) {
|
||||
if (doc.state === "pending") {
|
||||
result.push(doc);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
public deletedLocallyDocuments(): DeletedLocallyDocument[] {
|
||||
const result: DeletedLocallyDocument[] = [];
|
||||
for (const doc of this.documentIdIndex.values()) {
|
||||
if (doc.state === "deleted-locally") {
|
||||
result.push(doc);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
/** All live documents (pending + tracked) that occupy a path. */
|
||||
public allLiveDocuments(): VirtualDocument[] {
|
||||
return Array.from(this.pathIndex.values());
|
||||
}
|
||||
|
||||
/** Total number of documents across all indexes (live + deleted-locally). */
|
||||
public get length(): number {
|
||||
// pathIndex has live docs (pending + tracked).
|
||||
// documentIdIndex has tracked + deleted-locally.
|
||||
// Tracked docs appear in both, so count:
|
||||
// pending (pathIndex only) + tracked (both) + deleted-locally (documentIdIndex only)
|
||||
// = pathIndex.size + deletedLocally count
|
||||
let deletedCount = 0;
|
||||
for (const doc of this.documentIdIndex.values()) {
|
||||
if (doc.state === "deleted-locally") {
|
||||
deletedCount++;
|
||||
}
|
||||
}
|
||||
return this.pathIndex.size + deletedCount;
|
||||
}
|
||||
|
||||
public contains(doc: VirtualDocument): boolean {
|
||||
switch (doc.state) {
|
||||
case "pending":
|
||||
return this.idempotencyKeyIndex.get(doc.idempotencyKey) === doc;
|
||||
case "tracked":
|
||||
return this.documentIdIndex.get(doc.documentId) === doc;
|
||||
case "deleted-locally":
|
||||
return this.documentIdIndex.get(doc.documentId) === doc;
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Update ID tracking
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public getLastSeenUpdateId(): number {
|
||||
return this.lastSeenUpdateIds.min;
|
||||
}
|
||||
|
||||
public addSeenUpdateId(value: number): void {
|
||||
const previousMin = this.lastSeenUpdateIds.min;
|
||||
this.lastSeenUpdateIds.add(value);
|
||||
if (previousMin !== this.lastSeenUpdateIds.min) {
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
}
|
||||
|
||||
public setLastSeenUpdateId(value: number): void {
|
||||
this.lastSeenUpdateIds.min = value;
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Mutations
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Create a pending document at the given path. If a pending document
|
||||
* already exists at the path, return it (idempotent). Generates a new
|
||||
* idempotency key via `crypto.randomUUID()`.
|
||||
*
|
||||
* Awaits save() so the idempotency key is persisted before any HTTP
|
||||
* request is sent.
|
||||
*/
|
||||
public async createPending(path: string): Promise<PendingDocument> {
|
||||
this.logger.debug(`Creating new pending document: ${path}`);
|
||||
|
||||
const existing = this.pathIndex.get(path);
|
||||
if (existing?.state === "pending") {
|
||||
this.logger.debug(
|
||||
`Pending document already exists at ${path}, reusing it`
|
||||
);
|
||||
return existing;
|
||||
}
|
||||
|
||||
const doc: PendingDocument = {
|
||||
state: "pending",
|
||||
relativePath: path,
|
||||
idempotencyKey: crypto.randomUUID(),
|
||||
originalCreationPath: path
|
||||
};
|
||||
|
||||
this.pathIndex.set(path, doc);
|
||||
this.idempotencyKeyIndex.set(doc.idempotencyKey, doc);
|
||||
|
||||
// Awaited so the idempotency key is persisted before any HTTP
|
||||
// request is sent — a crash before save would lose the key.
|
||||
await this.save();
|
||||
|
||||
return doc;
|
||||
}
|
||||
|
||||
/**
|
||||
* Confirm a pending create: transition from pending to tracked.
|
||||
* Removes the pending doc and inserts a tracked doc with full metadata.
|
||||
*/
|
||||
public confirmCreate(
|
||||
idempotencyKey: string,
|
||||
documentId: DocumentId,
|
||||
serverVersion: VaultUpdateId,
|
||||
localHash: string,
|
||||
remoteRelativePath: RelativePath
|
||||
): TrackedDocument {
|
||||
const pending = this.idempotencyKeyIndex.get(idempotencyKey);
|
||||
if (pending === undefined) {
|
||||
// The pending doc was already promoted to tracked by
|
||||
// assignDocumentId (resolveIdempotencyKeys) or a previous
|
||||
// confirmCreate call. Find the tracked doc and update it.
|
||||
// Try by documentId first, then by scanning for the key.
|
||||
let existing = this.documentIdIndex.get(documentId);
|
||||
if (existing?.state !== "tracked") {
|
||||
// The server may have assigned a different documentId
|
||||
// (e.g., merge). Scan all tracked docs for the key.
|
||||
for (const doc of this.documentIdIndex.values()) {
|
||||
if (doc.state === "tracked" && doc.idempotencyKey === idempotencyKey) {
|
||||
existing = doc;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (existing?.state === "tracked") {
|
||||
// If the server assigned a different documentId than what
|
||||
// assignDocumentId set, update the index.
|
||||
if (existing.documentId !== documentId) {
|
||||
this.documentIdIndex.delete(existing.documentId);
|
||||
existing.documentId = documentId;
|
||||
this.documentIdIndex.set(documentId, existing);
|
||||
}
|
||||
existing.serverVersion = serverVersion;
|
||||
existing.localHash = localHash;
|
||||
existing.remoteRelativePath = remoteRelativePath;
|
||||
existing.idempotencyKey = undefined;
|
||||
this.lastSeenUpdateIds.add(serverVersion);
|
||||
this.saveInTheBackground();
|
||||
return existing;
|
||||
}
|
||||
// Truly not found — nothing to update
|
||||
throw new Error(
|
||||
`No pending document with idempotency key ${idempotencyKey}`
|
||||
);
|
||||
}
|
||||
|
||||
const tracked: TrackedDocument = {
|
||||
state: "tracked",
|
||||
relativePath: pending.relativePath,
|
||||
documentId,
|
||||
serverVersion,
|
||||
localHash,
|
||||
remoteRelativePath
|
||||
};
|
||||
|
||||
// Remove pending from indexes
|
||||
this.idempotencyKeyIndex.delete(idempotencyKey);
|
||||
|
||||
// Update pathIndex (pending -> tracked at same path)
|
||||
this.pathIndex.set(tracked.relativePath, tracked);
|
||||
|
||||
// Add to documentIdIndex
|
||||
this.documentIdIndex.set(tracked.documentId, tracked);
|
||||
|
||||
this.lastSeenUpdateIds.add(serverVersion);
|
||||
|
||||
this.saveInTheBackground();
|
||||
return tracked;
|
||||
}
|
||||
|
||||
/**
|
||||
* Assign a documentId to a pending document (used by resolveIdempotencyKeys).
|
||||
* Sets serverVersion = 0 as a placeholder — the sync path must treat
|
||||
* serverVersion === 0 as needing a create retry.
|
||||
*
|
||||
* Returns the new TrackedDocument, or undefined if the key is not found.
|
||||
*/
|
||||
public assignDocumentId(
|
||||
idempotencyKey: string,
|
||||
documentId: DocumentId
|
||||
): TrackedDocument | undefined {
|
||||
const pending = this.idempotencyKeyIndex.get(idempotencyKey);
|
||||
if (pending === undefined) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
const tracked: TrackedDocument = {
|
||||
state: "tracked",
|
||||
relativePath: pending.relativePath,
|
||||
documentId,
|
||||
serverVersion: 0,
|
||||
localHash: "",
|
||||
remoteRelativePath: pending.relativePath,
|
||||
idempotencyKey: pending.idempotencyKey
|
||||
};
|
||||
|
||||
// Remove pending from indexes
|
||||
this.idempotencyKeyIndex.delete(idempotencyKey);
|
||||
|
||||
// Update pathIndex
|
||||
this.pathIndex.set(tracked.relativePath, tracked);
|
||||
|
||||
// Add to documentIdIndex
|
||||
this.documentIdIndex.set(tracked.documentId, tracked);
|
||||
|
||||
this.saveInTheBackground();
|
||||
return tracked;
|
||||
}
|
||||
|
||||
/**
|
||||
* Update an existing tracked document's metadata.
|
||||
*/
|
||||
public updateTracked(
|
||||
documentId: DocumentId,
|
||||
serverVersion: VaultUpdateId,
|
||||
localHash: string,
|
||||
remoteRelativePath: RelativePath
|
||||
): void {
|
||||
const doc = this.documentIdIndex.get(documentId);
|
||||
if (doc?.state !== "tracked") {
|
||||
throw new Error(
|
||||
`Tracked document with id ${documentId} not found`
|
||||
);
|
||||
}
|
||||
|
||||
doc.serverVersion = serverVersion;
|
||||
doc.localHash = localHash;
|
||||
doc.remoteRelativePath = remoteRelativePath;
|
||||
|
||||
this.lastSeenUpdateIds.add(serverVersion);
|
||||
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
/**
|
||||
* Move a document from one path to another. Throws if the target path
|
||||
* is occupied by a live document.
|
||||
*/
|
||||
public move(oldPath: string, newPath: string): void {
|
||||
const doc = this.pathIndex.get(oldPath);
|
||||
if (doc === undefined) {
|
||||
return;
|
||||
}
|
||||
|
||||
// If another document occupies the target path, it was likely
|
||||
// orphaned by an earlier displacement that wasn't reconciled.
|
||||
// Remove it from the path index — reconcileWithDisk will
|
||||
// re-discover the file if it still exists on disk.
|
||||
const existingAtNew = this.pathIndex.get(newPath);
|
||||
if (existingAtNew !== undefined && existingAtNew !== doc) {
|
||||
this.pathIndex.delete(newPath);
|
||||
}
|
||||
|
||||
// Remove from old path
|
||||
this.pathIndex.delete(oldPath);
|
||||
|
||||
// Update the document's relativePath
|
||||
doc.relativePath = newPath;
|
||||
|
||||
// Insert at new path
|
||||
this.pathIndex.set(newPath, doc);
|
||||
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
/**
|
||||
* Mark a document as deleted locally.
|
||||
* - Pending: remove entirely (no server-side state to track).
|
||||
* - Tracked: transition to deleted-locally (keep in documentIdIndex).
|
||||
*/
|
||||
public deleteLocally(path: string): void {
|
||||
const doc = this.pathIndex.get(path);
|
||||
if (doc === undefined) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Remove from pathIndex in all cases
|
||||
this.pathIndex.delete(path);
|
||||
|
||||
if (doc.state === "pending") {
|
||||
// Remove from idempotencyKeyIndex too
|
||||
this.idempotencyKeyIndex.delete(doc.idempotencyKey);
|
||||
} else if (doc.state === "tracked") {
|
||||
// Transition to deleted-locally
|
||||
const deleted: DeletedLocallyDocument = {
|
||||
state: "deleted-locally",
|
||||
relativePath: doc.relativePath,
|
||||
documentId: doc.documentId,
|
||||
serverVersion: doc.serverVersion,
|
||||
remoteRelativePath: doc.remoteRelativePath
|
||||
};
|
||||
// Replace in documentIdIndex
|
||||
this.documentIdIndex.set(deleted.documentId, deleted);
|
||||
}
|
||||
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
/**
|
||||
* Confirm a server-side delete: remove the document entirely.
|
||||
*/
|
||||
public confirmDelete(documentId: DocumentId): void {
|
||||
const doc = this.documentIdIndex.get(documentId);
|
||||
if (doc === undefined) {
|
||||
return;
|
||||
}
|
||||
|
||||
this.documentIdIndex.delete(documentId);
|
||||
|
||||
// Also remove from pathIndex if present (tracked docs are in both)
|
||||
if (doc.state === "tracked") {
|
||||
const atPath = this.pathIndex.get(doc.relativePath);
|
||||
if (atPath === doc) {
|
||||
this.pathIndex.delete(doc.relativePath);
|
||||
}
|
||||
}
|
||||
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove a document from all indexes entirely.
|
||||
*/
|
||||
public remove(doc: VirtualDocument): void {
|
||||
switch (doc.state) {
|
||||
case "pending": {
|
||||
this.idempotencyKeyIndex.delete(doc.idempotencyKey);
|
||||
const atPath = this.pathIndex.get(doc.relativePath);
|
||||
if (atPath === doc) {
|
||||
this.pathIndex.delete(doc.relativePath);
|
||||
}
|
||||
break;
|
||||
}
|
||||
case "tracked": {
|
||||
this.documentIdIndex.delete(doc.documentId);
|
||||
const atPath = this.pathIndex.get(doc.relativePath);
|
||||
if (atPath === doc) {
|
||||
this.pathIndex.delete(doc.relativePath);
|
||||
}
|
||||
break;
|
||||
}
|
||||
case "deleted-locally": {
|
||||
this.documentIdIndex.delete(doc.documentId);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensure no other document has the given documentId. If a different
|
||||
* document already holds it, remove that document and return it (so
|
||||
* the caller can do optional file-level cleanup). Returns undefined
|
||||
* if no conflict exists.
|
||||
*/
|
||||
public ensureUniqueDocumentId(
|
||||
documentId: DocumentId,
|
||||
keeper: VirtualDocument
|
||||
): VirtualDocument | undefined {
|
||||
const existing = this.documentIdIndex.get(documentId);
|
||||
if (existing !== undefined && existing !== keeper) {
|
||||
this.remove(existing);
|
||||
return existing;
|
||||
}
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Persistence
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
public async save(): Promise<void> {
|
||||
const data = this.snapshotForSave();
|
||||
const previousSave = this.pendingSave;
|
||||
const thisSave = (async () => {
|
||||
await previousSave.catch(() => {});
|
||||
await this.saveData(data);
|
||||
})();
|
||||
this.pendingSave = thisSave.catch(() => {});
|
||||
return thisSave;
|
||||
}
|
||||
|
||||
public saveInTheBackground(): void {
|
||||
this.ensureConsistency();
|
||||
void this.save().catch((error: unknown) => {
|
||||
this.logger.error(`Error saving data: ${error}`);
|
||||
});
|
||||
}
|
||||
|
||||
public reset(): void {
|
||||
this.pathIndex.clear();
|
||||
this.documentIdIndex.clear();
|
||||
this.idempotencyKeyIndex.clear();
|
||||
this.lastSeenUpdateIds = new CoveredValues(0);
|
||||
this.saveInTheBackground();
|
||||
}
|
||||
|
||||
/**
|
||||
* Serialize to StoredDatabase format for backward compatibility.
|
||||
*/
|
||||
private snapshotForSave(): StoredDatabase {
|
||||
const documents: StoredDocumentMetadata[] = [];
|
||||
|
||||
// Tracked documents
|
||||
for (const doc of this.pathIndex.values()) {
|
||||
if (doc.state === "tracked") {
|
||||
documents.push({
|
||||
relativePath: doc.relativePath,
|
||||
documentId: doc.documentId,
|
||||
parentVersionId: doc.serverVersion,
|
||||
hash: doc.localHash,
|
||||
remoteRelativePath: doc.remoteRelativePath
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Deleted-locally documents (with isDeleted flag)
|
||||
for (const doc of this.documentIdIndex.values()) {
|
||||
if (doc.state === "deleted-locally") {
|
||||
documents.push({
|
||||
relativePath: doc.relativePath,
|
||||
documentId: doc.documentId,
|
||||
parentVersionId: doc.serverVersion,
|
||||
hash: "",
|
||||
isDeleted: true,
|
||||
remoteRelativePath: doc.remoteRelativePath
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Pending documents
|
||||
const pendingDocuments: StoredPendingDocument[] = [];
|
||||
for (const doc of this.idempotencyKeyIndex.values()) {
|
||||
pendingDocuments.push({
|
||||
relativePath: doc.relativePath,
|
||||
idempotencyKey: doc.idempotencyKey,
|
||||
originalCreationPath: doc.originalCreationPath
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
documents,
|
||||
pendingDocuments,
|
||||
lastSeenUpdateId: this.lastSeenUpdateIds.min
|
||||
};
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Consistency check
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
private ensureConsistency(): void {
|
||||
// Check that documentIdIndex has no duplicates (by construction it
|
||||
// shouldn't, since it's a Map keyed by documentId). But verify that
|
||||
// pathIndex entries with documentIds are consistent.
|
||||
const seenDocIds = new Set<string>();
|
||||
for (const doc of this.pathIndex.values()) {
|
||||
if (doc.state === "tracked") {
|
||||
if (seenDocIds.has(doc.documentId)) {
|
||||
throw new Error(
|
||||
`Duplicate documentId ${doc.documentId} found in VFS pathIndex`
|
||||
);
|
||||
}
|
||||
seenDocIds.add(doc.documentId);
|
||||
}
|
||||
}
|
||||
for (const doc of this.documentIdIndex.values()) {
|
||||
if (doc.state === "deleted-locally") {
|
||||
if (seenDocIds.has(doc.documentId)) {
|
||||
throw new Error(
|
||||
`Duplicate documentId ${doc.documentId} found across live and deleted documents`
|
||||
);
|
||||
}
|
||||
seenDocIds.add(doc.documentId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Disk reconciliation
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Compare VFS entries against files on disk and produce a pure result
|
||||
* describing what changed. Does NOT mutate the VFS.
|
||||
*
|
||||
* @param diskFiles - List of relative paths that currently exist on disk.
|
||||
* @param readAndHash - Callback to read a file and return its hash, or
|
||||
* undefined if the file cannot be read.
|
||||
*/
|
||||
public async reconcileWithDisk(
|
||||
diskFiles: string[],
|
||||
readAndHash: (path: string) => Promise<string | undefined>
|
||||
): Promise<ReconciliationResult> {
|
||||
const diskSet = new Set(diskFiles);
|
||||
|
||||
const newFiles: string[] = [];
|
||||
const modifiedFiles: { path: string; documentId: string }[] = [];
|
||||
const missingFiles: VirtualDocument[] = [];
|
||||
const movedFiles: { document: TrackedDocument; newPath: string }[] = [];
|
||||
|
||||
// Collect missing tracked/pending docs (file not on disk)
|
||||
const missingTracked: TrackedDocument[] = [];
|
||||
for (const doc of this.pathIndex.values()) {
|
||||
if (!diskSet.has(doc.relativePath)) {
|
||||
if (doc.state === "tracked") {
|
||||
missingTracked.push(doc);
|
||||
}
|
||||
missingFiles.push(doc);
|
||||
}
|
||||
}
|
||||
|
||||
// For each disk file, classify it
|
||||
for (const path of diskFiles) {
|
||||
const doc = this.pathIndex.get(path);
|
||||
|
||||
if (doc === undefined) {
|
||||
// File on disk, not in VFS — could be new or a move
|
||||
newFiles.push(path);
|
||||
} else if (doc.state === "tracked") {
|
||||
// Check if content changed
|
||||
const fileHash = await readAndHash(path);
|
||||
if (
|
||||
fileHash !== undefined &&
|
||||
fileHash !== doc.localHash
|
||||
) {
|
||||
modifiedFiles.push({
|
||||
path,
|
||||
documentId: doc.documentId
|
||||
});
|
||||
}
|
||||
}
|
||||
// If pending, nothing to reconcile — it's already pending
|
||||
}
|
||||
|
||||
// Attempt move detection: for each new file, try to match against
|
||||
// a missing tracked doc by content hash
|
||||
if (missingTracked.length > 0 && newFiles.length > 0) {
|
||||
const remainingNew: string[] = [];
|
||||
|
||||
for (const path of newFiles) {
|
||||
const fileHash = await readAndHash(path);
|
||||
if (fileHash === undefined || fileHash === EMPTY_HASH) {
|
||||
remainingNew.push(path);
|
||||
continue;
|
||||
}
|
||||
|
||||
// Find a single unique match among missing tracked docs
|
||||
const matches = missingTracked.filter(
|
||||
(doc) => doc.localHash === fileHash
|
||||
);
|
||||
|
||||
if (matches.length === 1) {
|
||||
const match = matches[0];
|
||||
movedFiles.push({ document: match, newPath: path });
|
||||
|
||||
// Remove from missingTracked so it can't match again
|
||||
const idx = missingTracked.indexOf(match);
|
||||
if (idx !== -1) {
|
||||
missingTracked.splice(idx, 1);
|
||||
}
|
||||
|
||||
// Remove from missingFiles too
|
||||
const missingIdx = missingFiles.indexOf(match);
|
||||
if (missingIdx !== -1) {
|
||||
missingFiles.splice(missingIdx, 1);
|
||||
}
|
||||
} else {
|
||||
remainingNew.push(path);
|
||||
}
|
||||
}
|
||||
|
||||
// Replace newFiles with the remaining unmatched ones
|
||||
newFiles.length = 0;
|
||||
newFiles.push(...remainingNew);
|
||||
}
|
||||
|
||||
return { newFiles, modifiedFiles, missingFiles, movedFiles };
|
||||
}
|
||||
}
|
||||
|
|
@ -6,6 +6,8 @@ import { SyncResetError } from "../errors/sync-reset-error";
|
|||
* Offers a resettable fetch implementation that waits until syncing is enabled
|
||||
* and aborts outstanding requests when a reset is started.
|
||||
*/
|
||||
const HTTP_REQUEST_TIMEOUT_MS = 30_000;
|
||||
|
||||
export class FetchController {
|
||||
private static readonly UNTIL_RESOLUTION = Symbol();
|
||||
|
||||
|
|
@ -81,7 +83,17 @@ export class FetchController {
|
|||
}
|
||||
|
||||
this.isResetting = false;
|
||||
[this.until, this.resolveUntil, this.rejectUntil] = createPromise();
|
||||
// Capture the old resolve before creating a fresh promise, then
|
||||
// resolve the old one — exactly the same pattern the canFetch
|
||||
// setter uses. This wakes up any fetches that entered the
|
||||
// while-loop between startReset and finishReset so they re-check
|
||||
// the condition. Without this, a canFetch change that occurred
|
||||
// during the reset (setter skips resolution while isResetting is
|
||||
// true) would leave fetches blocking on an unresolved promise.
|
||||
const previousResolve = this.resolveUntil;
|
||||
[this.until, this.resolveUntil, this.rejectUntil] =
|
||||
createPromise<symbol>();
|
||||
previousResolve(FetchController.UNTIL_RESOLUTION);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -117,7 +129,17 @@ export class FetchController {
|
|||
? input.clone()
|
||||
: input;
|
||||
|
||||
const fetchPromise = fetch(_input, init);
|
||||
const combinedSignal = init?.signal
|
||||
? AbortSignal.any([
|
||||
AbortSignal.timeout(HTTP_REQUEST_TIMEOUT_MS),
|
||||
init.signal
|
||||
])
|
||||
: AbortSignal.timeout(HTTP_REQUEST_TIMEOUT_MS);
|
||||
|
||||
const fetchPromise = fetch(_input, {
|
||||
...init,
|
||||
signal: combinedSignal
|
||||
});
|
||||
|
||||
// We only want to catch rejections from `this.until`
|
||||
let result: symbol | Response | undefined = undefined;
|
||||
|
|
|
|||
|
|
@ -49,13 +49,44 @@ export class SyncService {
|
|||
.get("Content-Type")
|
||||
?.includes("application/json") == true
|
||||
) {
|
||||
const result: SerializedError =
|
||||
(await response.json()) as SerializedError; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
return SyncService.formatError(result);
|
||||
try {
|
||||
const result: SerializedError =
|
||||
(await response.json()) as SerializedError; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
return SyncService.formatError(result);
|
||||
} catch {
|
||||
return `HTTP ${response.status}: ${response.statusText} (failed to parse error response body)`;
|
||||
}
|
||||
}
|
||||
return `HTTP ${response.status}: ${response.statusText}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Safely parse JSON from a response body. If parsing fails (e.g., malformed
|
||||
* JSON from the server), throws an HttpClientError with status 0 so that
|
||||
* retryForever does not retry indefinitely.
|
||||
*/
|
||||
private static async parseJsonResponse<T>(
|
||||
response: Response
|
||||
): Promise<T> {
|
||||
try {
|
||||
return (await response.json()) as T; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
} catch (error) {
|
||||
// Timeout and abort errors are transient — let them propagate
|
||||
// so retryForever can retry. Only wrap genuine parse failures
|
||||
// (malformed JSON) as HttpClientError to prevent infinite retries.
|
||||
if (
|
||||
error instanceof Error &&
|
||||
(error.name === "TimeoutError" || error.name === "AbortError")
|
||||
) {
|
||||
throw error;
|
||||
}
|
||||
throw new HttpClientError(
|
||||
0,
|
||||
`Failed to parse JSON response: ${error}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
private static formatError(error: SerializedError): string {
|
||||
let result = error.message;
|
||||
if (error.causes.length > 0) {
|
||||
|
|
@ -117,7 +148,9 @@ export class SyncService {
|
|||
}
|
||||
|
||||
const result: DocumentUpdateResponse =
|
||||
(await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
await SyncService.parseJsonResponse<DocumentUpdateResponse>(
|
||||
response
|
||||
);
|
||||
|
||||
this.logger.debug(`Created document ${JSON.stringify(result)}`);
|
||||
|
||||
|
|
@ -164,7 +197,9 @@ export class SyncService {
|
|||
}
|
||||
|
||||
const result: DocumentUpdateResponse =
|
||||
(await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
await SyncService.parseJsonResponse<DocumentUpdateResponse>(
|
||||
response
|
||||
);
|
||||
|
||||
this.logger.debug(
|
||||
`Updated document ${JSON.stringify(result)} with id ${result.documentId
|
||||
|
|
@ -215,7 +250,9 @@ export class SyncService {
|
|||
}
|
||||
|
||||
const result: DocumentUpdateResponse =
|
||||
(await response.json()) as DocumentUpdateResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
await SyncService.parseJsonResponse<DocumentUpdateResponse>(
|
||||
response
|
||||
);
|
||||
|
||||
this.logger.debug(
|
||||
`Updated document ${JSON.stringify(result)} with id ${result.documentId
|
||||
|
|
@ -234,9 +271,7 @@ export class SyncService {
|
|||
relativePath: RelativePath;
|
||||
}): Promise<DocumentVersionWithoutContent> {
|
||||
return this.retryForever(async () => {
|
||||
const request: DeleteDocumentVersion = {
|
||||
relativePath
|
||||
};
|
||||
const request: DeleteDocumentVersion = {};
|
||||
|
||||
this.logger.debug(
|
||||
`Delete document with id ${documentId} and relative path ${relativePath}`
|
||||
|
|
@ -259,7 +294,9 @@ export class SyncService {
|
|||
}
|
||||
|
||||
const result: DocumentVersionWithoutContent =
|
||||
(await response.json()) as DocumentVersionWithoutContent; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
await SyncService.parseJsonResponse<DocumentVersionWithoutContent>(
|
||||
response
|
||||
);
|
||||
|
||||
this.logger.debug(
|
||||
`Deleted document ${relativePath} with id ${documentId}`
|
||||
|
|
@ -292,7 +329,9 @@ export class SyncService {
|
|||
}
|
||||
|
||||
const result: DocumentVersion =
|
||||
(await response.json()) as DocumentVersion; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
await SyncService.parseJsonResponse<DocumentVersion>(
|
||||
response
|
||||
);
|
||||
|
||||
this.logger.debug(`Got document ${JSON.stringify(result)}`);
|
||||
|
||||
|
|
@ -361,7 +400,9 @@ export class SyncService {
|
|||
}
|
||||
|
||||
const result: FetchLatestDocumentsResponse =
|
||||
(await response.json()) as FetchLatestDocumentsResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
await SyncService.parseJsonResponse<FetchLatestDocumentsResponse>(
|
||||
response
|
||||
);
|
||||
|
||||
this.logger.debug(
|
||||
`Got ${result.latestDocuments.length} document metadata`
|
||||
|
|
@ -389,15 +430,16 @@ export class SyncService {
|
|||
);
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(
|
||||
`Failed to resolve idempotency keys: ${await SyncService.errorFromResponse(
|
||||
response
|
||||
)}`
|
||||
await SyncService.throwHttpError(
|
||||
response,
|
||||
"Failed to resolve idempotency keys"
|
||||
);
|
||||
}
|
||||
|
||||
const result: { resolved: Record<string, string> } =
|
||||
(await response.json()) as { resolved: Record<string, string> }; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
const result =
|
||||
await SyncService.parseJsonResponse<{
|
||||
resolved: Record<string, string>;
|
||||
}>(response);
|
||||
|
||||
const resolved = new Map<string, string>(
|
||||
Object.entries(result.resolved)
|
||||
|
|
@ -425,7 +467,8 @@ export class SyncService {
|
|||
);
|
||||
}
|
||||
|
||||
const result: PingResponse = (await response.json()) as PingResponse; // eslint-disable-line @typescript-eslint/no-unsafe-type-assertion
|
||||
const result: PingResponse =
|
||||
await SyncService.parseJsonResponse<PingResponse>(response);
|
||||
|
||||
this.logger.debug(
|
||||
`Pinged server, got response: ${JSON.stringify(result)}`
|
||||
|
|
@ -457,6 +500,7 @@ export class SyncService {
|
|||
}
|
||||
|
||||
private async retryForever<T>(fn: () => Promise<T>): Promise<T> {
|
||||
let attempt = 0;
|
||||
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
|
||||
while (true) {
|
||||
try {
|
||||
|
|
@ -473,12 +517,19 @@ export class SyncService {
|
|||
throw e;
|
||||
}
|
||||
|
||||
const retryInterval =
|
||||
attempt++;
|
||||
const baseDelay =
|
||||
this.settings.getSettings().networkRetryIntervalMs;
|
||||
this.logger.error(
|
||||
`Failed network call (${e}), retrying in ${retryInterval}ms`
|
||||
const exponentialDelay = Math.min(
|
||||
baseDelay * Math.pow(2, Math.min(attempt - 1, 5)),
|
||||
30000
|
||||
);
|
||||
await sleep(retryInterval);
|
||||
const jitter = Math.random() * exponentialDelay * 0.5;
|
||||
const delay = exponentialDelay + jitter;
|
||||
this.logger.error(
|
||||
`Failed network call (${e}), retrying in ${Math.round(delay)}ms (attempt ${attempt})`
|
||||
);
|
||||
await sleep(delay);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,3 +1,3 @@
|
|||
// This file was generated by [ts-rs](https://github.com/Aleph-Alpha/ts-rs). Do not edit this file manually.
|
||||
|
||||
export interface DeleteDocumentVersion { relativePath: string, }
|
||||
export type DeleteDocumentVersion = Record<string, never>;
|
||||
|
|
|
|||
|
|
@ -2,4 +2,4 @@
|
|||
import type { CursorPositionFromClient } from "./CursorPositionFromClient";
|
||||
import type { WebSocketHandshake } from "./WebSocketHandshake";
|
||||
|
||||
export type WebSocketClientMessage = { "type": "handshake" } & WebSocketHandshake | { "type": "cursorPositions" } & CursorPositionFromClient;
|
||||
export type WebSocketClientMessage = { "type": "handshake" } & WebSocketHandshake | { "type": "cursorPositions" } & CursorPositionFromClient | { "type": "ping" };
|
||||
|
|
|
|||
|
|
@ -34,6 +34,14 @@ export class WebSocketManager {
|
|||
|
||||
private readonly outstandingPromises: Promise<unknown>[] = [];
|
||||
|
||||
/**
|
||||
* Chains WebSocket message processing so only one message is handled
|
||||
* at a time. Without this, a burst of messages would create many
|
||||
* concurrent sync operations (each calling scheduleSyncForOfflineChanges
|
||||
* and processing documents in parallel).
|
||||
*/
|
||||
private messageProcessingChain: Promise<void> = Promise.resolve();
|
||||
|
||||
private webSocket: WebSocket | undefined;
|
||||
|
||||
public constructor(
|
||||
|
|
@ -102,6 +110,11 @@ export class WebSocketManager {
|
|||
}
|
||||
}
|
||||
|
||||
// Wait for any already-enqueued message handlers to finish.
|
||||
// The isStopped guard in onmessage prevents NEW messages from
|
||||
// being enqueued, but handlers that were chained before stop()
|
||||
// set the flag may still be in flight.
|
||||
await this.messageProcessingChain;
|
||||
await this.waitUntilFinished();
|
||||
}
|
||||
|
||||
|
|
@ -216,29 +229,75 @@ export class WebSocketManager {
|
|||
};
|
||||
|
||||
this.webSocket.onmessage = (event): void => {
|
||||
// Discard messages received after stop() has been called.
|
||||
// Without this guard, messages arriving between close()
|
||||
// and the onclose event would be enqueued into
|
||||
// messageProcessingChain and execute after stop() returns.
|
||||
if (this.isStopped) {
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
|
||||
const message = JSON.parse(
|
||||
event.data
|
||||
) as WebSocketServerMessage;
|
||||
|
||||
// Track the message handling promise
|
||||
const messageHandlingPromise = this.handleWebSocketMessage(
|
||||
message
|
||||
)
|
||||
// Cursor updates are pure reads (update an in-memory map) —
|
||||
// handle immediately without blocking behind vault update
|
||||
// processing. This avoids cursor latency during large syncs.
|
||||
if (message.type === "cursorPositions") {
|
||||
this.logger.debug(
|
||||
`Received cursor positions for ${JSON.stringify(message.clients)}`
|
||||
);
|
||||
const cursorPromise =
|
||||
this.onRemoteCursorsUpdateReceived
|
||||
.triggerAsync(message.clients)
|
||||
.catch((error: unknown) => {
|
||||
this.logger.error(
|
||||
`Error handling cursor update: ${String(error)}`
|
||||
);
|
||||
});
|
||||
// Track for waitUntilFinished / hasOutstandingWork
|
||||
this.outstandingPromises.push(cursorPromise);
|
||||
void cursorPromise.finally(() => {
|
||||
removeFromArray(
|
||||
this.outstandingPromises,
|
||||
cursorPromise
|
||||
);
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
// Vault updates require serialization: each waits for the
|
||||
// previous one to finish. This provides back-pressure so a
|
||||
// burst of WebSocket messages doesn't create unbounded
|
||||
// concurrent sync operations.
|
||||
//
|
||||
// Read-reassign safety: we read messageProcessingChain,
|
||||
// chain a .then() onto it, and assign the resulting promise
|
||||
// back. This is safe because JavaScript is single-threaded:
|
||||
// no other code can run between the read and the assignment.
|
||||
// The next onmessage invocation will see the updated chain
|
||||
// and append after this handler, preserving FIFO order.
|
||||
this.messageProcessingChain = this.messageProcessingChain
|
||||
.then(async () => this.handleWebSocketMessage(message))
|
||||
.catch((error: unknown) => {
|
||||
this.logger.error(
|
||||
`Error handling WebSocket message: ${String(error)}`
|
||||
);
|
||||
})
|
||||
.finally(() => {
|
||||
removeFromArray(
|
||||
this.outstandingPromises,
|
||||
messageHandlingPromise
|
||||
);
|
||||
});
|
||||
|
||||
void this.outstandingPromises.push(messageHandlingPromise); // ignore the returned promise
|
||||
const messageHandlingPromise = this.messageProcessingChain;
|
||||
|
||||
// Track the promise for waitUntilFinished / hasOutstandingWork
|
||||
this.outstandingPromises.push(messageHandlingPromise);
|
||||
void messageHandlingPromise.finally(() => {
|
||||
removeFromArray(
|
||||
this.outstandingPromises,
|
||||
messageHandlingPromise
|
||||
);
|
||||
});
|
||||
} catch (error) {
|
||||
this.logger.error(
|
||||
`Error parsing WebSocket message: ${String(error)}`
|
||||
|
|
@ -283,17 +342,8 @@ export class WebSocketManager {
|
|||
): Promise<void> {
|
||||
if (message.type === "vaultUpdate") {
|
||||
await this.onRemoteVaultUpdateReceived.triggerAsync(message);
|
||||
|
||||
// eslint-disable-next-line @typescript-eslint/no-unnecessary-condition
|
||||
} else if (message.type === "cursorPositions") {
|
||||
this.logger.debug(
|
||||
`Received cursor positions for ${JSON.stringify(message.clients)}`
|
||||
);
|
||||
|
||||
await this.onRemoteCursorsUpdateReceived.triggerAsync(
|
||||
message.clients
|
||||
);
|
||||
} else {
|
||||
// Cursor messages are handled inline in onmessage (not chained)
|
||||
this.logger.warn(
|
||||
`Received unknown message type: ${JSON.stringify(message)}`
|
||||
);
|
||||
|
|
|
|||
|
|
@ -3,7 +3,6 @@ import type { HistoryEntry, HistoryStats } from "./tracing/sync-history";
|
|||
import { SyncHistory } from "./tracing/sync-history";
|
||||
import { Logger, LogLevel, LogLine } from "./tracing/logger";
|
||||
import type { RelativePath, StoredDatabase } from "./persistence/database";
|
||||
import { Database } from "./persistence/database";
|
||||
import * as Sentry from "@sentry/browser";
|
||||
import type { SyncSettings } from "./persistence/settings";
|
||||
import { DEFAULT_SETTINGS, Settings } from "./persistence/settings";
|
||||
|
|
@ -12,7 +11,8 @@ import { Syncer } from "./sync-operations/syncer";
|
|||
import type { FileSystemOperations } from "./file-operations/filesystem-operations";
|
||||
import { FileOperations } from "./file-operations/file-operations";
|
||||
import { FetchController } from "./services/fetch-controller";
|
||||
import { UnrestrictedSyncer } from "./sync-operations/unrestricted-syncer";
|
||||
import { VirtualFilesystem } from "./persistence/vfs";
|
||||
import type { SyncDeps } from "./sync-operations/sync-actions";
|
||||
import { rateLimit } from "./utils/rate-limit";
|
||||
import type { NetworkConnectionStatus } from "./types/network-connection-status";
|
||||
import { DocumentSyncStatus } from "./types/document-sync-status";
|
||||
|
|
@ -25,7 +25,6 @@ import type { MaybeOutdatedClientCursors } from "./types/maybe-outdated-client-c
|
|||
import { FileChangeNotifier } from "./sync-operations/file-change-notifier";
|
||||
import { FixedSizeDocumentCache } from "./utils/data-structures/fix-sized-cache";
|
||||
import { setUpTelemetry } from "./utils/set-up-telemetry";
|
||||
import { DIFF_CACHE_SIZE_MB } from "./consts";
|
||||
import { ServerConfig } from "./services/server-config";
|
||||
import type { EventListeners } from "./utils/data-structures/event-listeners";
|
||||
|
||||
|
|
@ -41,7 +40,7 @@ export class SyncClient {
|
|||
public readonly logger: Logger,
|
||||
private readonly history: SyncHistory,
|
||||
private readonly settings: Settings,
|
||||
private readonly database: Database,
|
||||
private readonly vfs: VirtualFilesystem,
|
||||
private readonly syncer: Syncer,
|
||||
private readonly webSocketManager: WebSocketManager,
|
||||
private readonly fetchController: FetchController,
|
||||
|
|
@ -59,7 +58,7 @@ export class SyncClient {
|
|||
) { }
|
||||
|
||||
public get documentCount(): number {
|
||||
return this.database.length;
|
||||
return this.vfs.length;
|
||||
}
|
||||
|
||||
public get isWebSocketConnected(): boolean {
|
||||
|
|
@ -148,7 +147,7 @@ export class SyncClient {
|
|||
() => settings.getSettings().minimumSaveIntervalMs
|
||||
);
|
||||
|
||||
const database = new Database(
|
||||
const vfs = new VirtualFilesystem(
|
||||
logger,
|
||||
state.database,
|
||||
async (data): Promise<void> => {
|
||||
|
|
@ -174,25 +173,26 @@ export class SyncClient {
|
|||
|
||||
const fileOperations = new FileOperations(
|
||||
logger,
|
||||
database,
|
||||
vfs,
|
||||
fs,
|
||||
serverConfig,
|
||||
nativeLineEndings
|
||||
);
|
||||
|
||||
const contentCache = new FixedSizeDocumentCache(
|
||||
1024 * 1024 * DIFF_CACHE_SIZE_MB
|
||||
1024 * 1024 * settings.getSettings().diffCacheSizeMB
|
||||
);
|
||||
const unrestrictedSyncer = new UnrestrictedSyncer(
|
||||
|
||||
const syncDeps: SyncDeps = {
|
||||
logger,
|
||||
database,
|
||||
settings,
|
||||
vfs,
|
||||
syncService,
|
||||
fileOperations,
|
||||
operations: fileOperations,
|
||||
history,
|
||||
contentCache,
|
||||
serverConfig
|
||||
);
|
||||
serverConfig,
|
||||
settings
|
||||
};
|
||||
|
||||
const webSocketManager = new WebSocketManager(
|
||||
logger,
|
||||
|
|
@ -203,17 +203,17 @@ export class SyncClient {
|
|||
const syncer = new Syncer(
|
||||
deviceId,
|
||||
logger,
|
||||
database,
|
||||
vfs,
|
||||
settings,
|
||||
webSocketManager,
|
||||
fileOperations,
|
||||
unrestrictedSyncer
|
||||
syncDeps
|
||||
);
|
||||
|
||||
const fileChangeNotifier = new FileChangeNotifier();
|
||||
const cursorTracker = new CursorTracker(
|
||||
logger,
|
||||
database,
|
||||
vfs,
|
||||
webSocketManager,
|
||||
fileOperations,
|
||||
fileChangeNotifier
|
||||
|
|
@ -222,7 +222,7 @@ export class SyncClient {
|
|||
logger,
|
||||
history,
|
||||
settings,
|
||||
database,
|
||||
vfs,
|
||||
syncer,
|
||||
webSocketManager,
|
||||
fetchController,
|
||||
|
|
@ -333,8 +333,8 @@ export class SyncClient {
|
|||
|
||||
// clear all local state
|
||||
this.logger.info("Resetting SyncClient's local state");
|
||||
this.database.reset();
|
||||
await this.database.save(); // ensure the new database reads as empty
|
||||
this.vfs.reset();
|
||||
await this.vfs.save(); // ensure the new database reads as empty
|
||||
this.resetInMemoryState();
|
||||
this.hasFinishedOfflineSync = false;
|
||||
this.serverConfig.reset();
|
||||
|
|
@ -433,14 +433,34 @@ export class SyncClient {
|
|||
while (true) {
|
||||
iteration++;
|
||||
this.logger.info(`waitUntilFinished: iteration ${iteration}`);
|
||||
await this.webSocketManager.waitUntilFinished();
|
||||
await this.syncer.waitUntilFinished();
|
||||
await this.webSocketManager.waitUntilFinished();
|
||||
// Check if anything new arrived while we were waiting
|
||||
if (!this.webSocketManager.hasOutstandingWork()) {
|
||||
if (
|
||||
!this.webSocketManager.hasOutstandingWork() &&
|
||||
!this.syncer.hasOutstandingWork()
|
||||
) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
await this.database.save(); // flush all changes to disk
|
||||
|
||||
// Run a final filesystem scan to catch any operations that were
|
||||
// silently dropped (e.g., due to mutable document references
|
||||
// pointing to a moved path after concurrent renames).
|
||||
await this.syncer.runFinalConsistencyCheck();
|
||||
// Wait for any work produced by the final scan
|
||||
while (true) {
|
||||
await this.syncer.waitUntilFinished();
|
||||
await this.webSocketManager.waitUntilFinished();
|
||||
if (
|
||||
!this.webSocketManager.hasOutstandingWork() &&
|
||||
!this.syncer.hasOutstandingWork()
|
||||
) {
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
await this.vfs.save(); // flush all changes to disk
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -467,6 +487,8 @@ export class SyncClient {
|
|||
this.resetInMemoryState();
|
||||
|
||||
// Clean up event listeners to prevent memory leaks
|
||||
this.syncer.destroy();
|
||||
this.cursorTracker.destroy();
|
||||
this.eventUnsubscribers.forEach((unsubscribe) => {
|
||||
unsubscribe();
|
||||
});
|
||||
|
|
@ -491,10 +513,16 @@ export class SyncClient {
|
|||
}
|
||||
|
||||
/**
|
||||
* Hard pause: aborts all in-flight HTTP operations via FetchController reset.
|
||||
* Used when the SyncClient is being destroyed or fully reset (connection
|
||||
* settings changed). This is the nuclear option — every outstanding fetch
|
||||
* is rejected with SyncResetError so the queue drains immediately.
|
||||
* Pause syncing: aborts all in-flight HTTP operations via FetchController
|
||||
* reset, stops the WebSocket, and waits for the sync queue to drain.
|
||||
*
|
||||
* Used by both destroy/reset (connection settings changed) and when the
|
||||
* user toggles sync off. In both cases, `fetchController.startReset()` is
|
||||
* needed because the settings-change listener may have already set
|
||||
* `canFetch = false`, which would cause controlled fetches to block
|
||||
* indefinitely. The WebSocket close handler triggers `syncer.reset()`
|
||||
* automatically via the `onWebSocketStatusChanged` listener, so an
|
||||
* explicit `syncer.reset()` call is not needed here.
|
||||
*/
|
||||
private async pause(): Promise<void> {
|
||||
this.hasFinishedOfflineSync = false;
|
||||
|
|
@ -504,29 +532,15 @@ export class SyncClient {
|
|||
await this.waitUntilFinished();
|
||||
} catch (e) {
|
||||
// SyncResetError is expected here — we just called startReset()
|
||||
// which rejects in-flight fetches. Only re-throw non-reset errors
|
||||
// (after ensuring the FetchController is left in a usable state).
|
||||
this.fetchController.finishReset();
|
||||
// which rejects in-flight fetches. Only re-throw non-reset errors.
|
||||
if (!(e instanceof SyncResetError)) {
|
||||
throw e;
|
||||
}
|
||||
} finally {
|
||||
this.fetchController.finishReset();
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Soft pause: stops the WebSocket and clears the sync queue, but lets
|
||||
* in-flight HTTP operations complete naturally. Used when the user toggles
|
||||
* sync off — we don't want to abort creates/updates that are mid-flight
|
||||
* because they'd just be re-queued on re-enable, potentially leading to
|
||||
* an infinite retry loop with flaky connections.
|
||||
*/
|
||||
private async softPause(): Promise<void> {
|
||||
this.hasFinishedOfflineSync = false;
|
||||
await this.webSocketManager.stop();
|
||||
this.syncer.reset();
|
||||
await this.waitUntilFinished();
|
||||
}
|
||||
|
||||
private resetInMemoryState(): void {
|
||||
this.history.reset();
|
||||
this.contentCache.reset();
|
||||
|
|
@ -553,7 +567,7 @@ export class SyncClient {
|
|||
if (newSettings.isSyncEnabled) {
|
||||
await this.startSyncing();
|
||||
} else {
|
||||
await this.softPause();
|
||||
await this.pause();
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import type { FileOperations } from "../file-operations/file-operations";
|
||||
import type { Database, RelativePath } from "../persistence/database";
|
||||
import type { RelativePath } from "../persistence/database";
|
||||
import type { VirtualFilesystem } from "../persistence/vfs";
|
||||
import type { ClientCursors } from "../services/types/ClientCursors";
|
||||
import type { CursorSpan } from "../services/types/CursorSpan";
|
||||
import type { DocumentWithCursors } from "../services/types/DocumentWithCursors";
|
||||
|
|
@ -24,6 +25,7 @@ export class CursorTracker {
|
|||
>();
|
||||
|
||||
private readonly updateLock: Lock;
|
||||
private readonly eventUnsubscribers: (() => void)[] = [];
|
||||
|
||||
private knownRemoteCursors: (ClientCursors & {
|
||||
upToDateness: DocumentUpToDateness;
|
||||
|
|
@ -35,61 +37,68 @@ export class CursorTracker {
|
|||
|
||||
public constructor(
|
||||
private readonly logger: Logger,
|
||||
private readonly database: Database,
|
||||
private readonly vfs: VirtualFilesystem,
|
||||
private readonly webSocketManager: WebSocketManager,
|
||||
private readonly fileOperations: FileOperations,
|
||||
private readonly fileChangeNotifier: FileChangeNotifier
|
||||
) {
|
||||
this.updateLock = new Lock(CursorTracker.name, logger);
|
||||
|
||||
this.webSocketManager.onRemoteCursorsUpdateReceived.add(
|
||||
async (clientCursors) => {
|
||||
await this.updateLock.withLock(async () => {
|
||||
// The latest message will contain all active clients, so we can delete the ones
|
||||
// from the local list which are no longer active.
|
||||
const allIds = new Set(
|
||||
clientCursors.map((c) => c.deviceId)
|
||||
);
|
||||
const updatedKnownRemoteCursors =
|
||||
this.knownRemoteCursors.filter((c) =>
|
||||
allIds.has(c.deviceId)
|
||||
this.eventUnsubscribers.push(
|
||||
this.webSocketManager.onRemoteCursorsUpdateReceived.add(
|
||||
async (clientCursors) => {
|
||||
await this.updateLock.withLock(async () => {
|
||||
// The latest message will contain all active clients, so we can delete the ones
|
||||
// from the local list which are no longer active.
|
||||
const allIds = new Set(
|
||||
clientCursors.map((c) => c.deviceId)
|
||||
);
|
||||
const updatedKnownRemoteCursors =
|
||||
this.knownRemoteCursors.filter((c) =>
|
||||
allIds.has(c.deviceId)
|
||||
);
|
||||
|
||||
for (const cursor of clientCursors.filter((client) =>
|
||||
client.documentsWithCursors.every(
|
||||
(doc) => doc.vault_update_id != null
|
||||
)
|
||||
)) {
|
||||
updatedKnownRemoteCursors.push({
|
||||
...cursor,
|
||||
upToDateness:
|
||||
await this.getDocumentsUpToDateness(cursor)
|
||||
});
|
||||
}
|
||||
for (const cursor of clientCursors.filter((client) =>
|
||||
client.documentsWithCursors.every(
|
||||
(doc) => doc.vault_update_id != null
|
||||
)
|
||||
)) {
|
||||
updatedKnownRemoteCursors.push({
|
||||
...cursor,
|
||||
upToDateness:
|
||||
await this.getDocumentsUpToDateness(cursor)
|
||||
});
|
||||
}
|
||||
|
||||
this.knownRemoteCursors = updatedKnownRemoteCursors;
|
||||
});
|
||||
this.knownRemoteCursors = updatedKnownRemoteCursors;
|
||||
});
|
||||
|
||||
this.onRemoteCursorsUpdated.trigger(
|
||||
this.getRelevantAndPruneKnownClientCursors()
|
||||
);
|
||||
}
|
||||
this.onRemoteCursorsUpdated.trigger(
|
||||
this.getRelevantAndPruneKnownClientCursors()
|
||||
);
|
||||
}
|
||||
)
|
||||
);
|
||||
|
||||
this.fileChangeNotifier.onFileChanged.add(async (relativePath) =>
|
||||
this.updateLock.withLock(async () => {
|
||||
for (const clientCursor of this.knownRemoteCursors) {
|
||||
if (
|
||||
clientCursor.documentsWithCursors.some(
|
||||
(document) =>
|
||||
document.relative_path === relativePath
|
||||
)
|
||||
) {
|
||||
clientCursor.upToDateness =
|
||||
await this.getDocumentsUpToDateness(clientCursor);
|
||||
}
|
||||
}
|
||||
})
|
||||
this.eventUnsubscribers.push(
|
||||
this.fileChangeNotifier.onFileChanged.add(
|
||||
async (relativePath) =>
|
||||
this.updateLock.withLock(async () => {
|
||||
for (const clientCursor of this.knownRemoteCursors) {
|
||||
if (
|
||||
clientCursor.documentsWithCursors.some(
|
||||
(document) =>
|
||||
document.relative_path === relativePath
|
||||
)
|
||||
) {
|
||||
clientCursor.upToDateness =
|
||||
await this.getDocumentsUpToDateness(
|
||||
clientCursor
|
||||
);
|
||||
}
|
||||
}
|
||||
})
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -104,21 +113,20 @@ export class CursorTracker {
|
|||
for (const [relativePath, cursors] of Object.entries(
|
||||
documentToCursors
|
||||
)) {
|
||||
const record =
|
||||
this.database.getLatestDocumentByRelativePath(relativePath);
|
||||
const doc = this.vfs.getByPath(relativePath);
|
||||
|
||||
if (!record) {
|
||||
if (!doc) {
|
||||
continue; // Let's wait for the file to be created before sending cursors
|
||||
}
|
||||
|
||||
if (!record.metadata) {
|
||||
continue; // this is a new document, no need to sync the cursors
|
||||
if (doc.state !== "tracked") {
|
||||
continue; // this is a pending document, no need to sync the cursors
|
||||
}
|
||||
|
||||
documentsWithCursors.push({
|
||||
relative_path: relativePath,
|
||||
document_id: record.metadata.documentId,
|
||||
vault_update_id: record.metadata.parentVersionId,
|
||||
document_id: doc.documentId,
|
||||
vault_update_id: doc.serverVersion,
|
||||
cursors: cursors.map(({ start, end }) => ({
|
||||
start: Math.min(start, end),
|
||||
end: Math.max(start, end)
|
||||
|
|
@ -139,10 +147,10 @@ export class CursorTracker {
|
|||
const readContent = await this.fileOperations.read(
|
||||
doc.relative_path
|
||||
);
|
||||
const record = this.database.getLatestDocumentByRelativePath(
|
||||
doc.relative_path
|
||||
);
|
||||
if (record?.metadata?.hash !== hash(readContent)) {
|
||||
const vfsDoc = this.vfs.getByPath(doc.relative_path);
|
||||
const storedHash =
|
||||
vfsDoc?.state === "tracked" ? vfsDoc.localHash : undefined;
|
||||
if (storedHash !== hash(readContent)) {
|
||||
doc.vault_update_id = null;
|
||||
}
|
||||
}
|
||||
|
|
@ -166,6 +174,12 @@ export class CursorTracker {
|
|||
this.updateLock.reset();
|
||||
}
|
||||
|
||||
public destroy(): void {
|
||||
for (const unsubscribe of this.eventUnsubscribers) {
|
||||
unsubscribe();
|
||||
}
|
||||
}
|
||||
|
||||
private getRelevantAndPruneKnownClientCursors(): MaybeOutdatedClientCursors[] {
|
||||
const result: MaybeOutdatedClientCursors[] = [];
|
||||
const included = new Set<string>();
|
||||
|
|
@ -227,24 +241,19 @@ export class CursorTracker {
|
|||
private async getDocumentUpToDateness(
|
||||
document: DocumentWithCursors
|
||||
): Promise<DocumentUpToDateness> {
|
||||
const record = this.database.getLatestDocumentByRelativePath(
|
||||
document.relative_path
|
||||
);
|
||||
const vfsDoc = this.vfs.getByPath(document.relative_path);
|
||||
|
||||
if (!record) {
|
||||
if (!vfsDoc) {
|
||||
// the document of the cursor must be from the future
|
||||
return DocumentUpToDateness.Later;
|
||||
}
|
||||
|
||||
if (
|
||||
(record.metadata?.parentVersionId ?? 0) <
|
||||
(document.vault_update_id ?? 0)
|
||||
) {
|
||||
const serverVersion =
|
||||
vfsDoc.state === "tracked" ? vfsDoc.serverVersion : 0;
|
||||
|
||||
if (serverVersion < (document.vault_update_id ?? 0)) {
|
||||
return DocumentUpToDateness.Later;
|
||||
} else if (
|
||||
(document.vault_update_id ?? 0) <
|
||||
(record.metadata?.parentVersionId ?? 0)
|
||||
) {
|
||||
} else if ((document.vault_update_id ?? 0) < serverVersion) {
|
||||
// the document of the cursor must be from the past
|
||||
return DocumentUpToDateness.Prior;
|
||||
}
|
||||
|
|
@ -253,9 +262,11 @@ export class CursorTracker {
|
|||
document.relative_path
|
||||
);
|
||||
|
||||
return this.database.getLatestDocumentByRelativePath(
|
||||
document.relative_path
|
||||
)?.metadata?.hash === hash(currentContent)
|
||||
const freshDoc = this.vfs.getByPath(document.relative_path);
|
||||
const storedHash =
|
||||
freshDoc?.state === "tracked" ? freshDoc.localHash : undefined;
|
||||
|
||||
return storedHash === hash(currentContent)
|
||||
? DocumentUpToDateness.UpToDate
|
||||
: DocumentUpToDateness.Prior;
|
||||
}
|
||||
|
|
|
|||
1182
frontend/sync-client/src/sync-operations/sync-actions.ts
Normal file
1182
frontend/sync-client/src/sync-operations/sync-actions.ts
Normal file
File diff suppressed because it is too large
Load diff
268
frontend/sync-client/src/sync-operations/sync-event-queue.ts
Normal file
268
frontend/sync-client/src/sync-operations/sync-event-queue.ts
Normal file
|
|
@ -0,0 +1,268 @@
|
|||
import type { Logger } from "../tracing/logger";
|
||||
import type { VirtualFilesystem } from "../persistence/vfs";
|
||||
import type { SyncEvent, CoalescedAction } from "./sync-events";
|
||||
import { coalesce, eventToInitialAction } from "./sync-events";
|
||||
import { SyncResetError } from "../errors/sync-reset-error";
|
||||
import { EventListeners } from "../utils/data-structures/event-listeners";
|
||||
|
||||
// A document key is either a documentId (for tracked docs) or "path:<relativePath>" (for pending docs)
|
||||
type DocumentKey = string;
|
||||
|
||||
export class SyncEventQueue {
|
||||
private readonly documentStates = new Map<DocumentKey, CoalescedAction>();
|
||||
private readonly processingOrder: DocumentKey[] = [];
|
||||
private currentlyProcessing: DocumentKey | null = null;
|
||||
private currentOperation: Promise<void> | null = null;
|
||||
private readonly idleWaiters: (() => void)[] = [];
|
||||
private isResetting = false;
|
||||
private isPaused = false;
|
||||
|
||||
public readonly onRemainingOperationsCountChanged = new EventListeners<
|
||||
(count: number) => unknown
|
||||
>();
|
||||
|
||||
// The executor is injected by the Syncer — it processes one CoalescedAction for one document
|
||||
private executor:
|
||||
| ((key: DocumentKey, action: CoalescedAction) => Promise<void>)
|
||||
| undefined;
|
||||
|
||||
constructor(
|
||||
private readonly logger: Logger,
|
||||
private readonly vfs: VirtualFilesystem
|
||||
) {}
|
||||
|
||||
public setExecutor(
|
||||
executor: (key: DocumentKey, action: CoalescedAction) => Promise<void>
|
||||
): void {
|
||||
this.executor = executor;
|
||||
}
|
||||
|
||||
// --- Event ingestion ---
|
||||
|
||||
public enqueue(event: SyncEvent): void {
|
||||
const key = this.resolveKey(event);
|
||||
const existing = this.documentStates.get(key);
|
||||
|
||||
if (existing === undefined || existing.action === "noop") {
|
||||
this.documentStates.set(key, eventToInitialAction(event));
|
||||
this.addToProcessingOrder(key);
|
||||
} else {
|
||||
const newAction = coalesce(existing, event);
|
||||
if (newAction.action === "noop") {
|
||||
this.documentStates.delete(key);
|
||||
this.removeFromProcessingOrder(key);
|
||||
} else {
|
||||
this.documentStates.set(key, newAction);
|
||||
// If the key isn't in processingOrder (was being processed), add it back
|
||||
if (
|
||||
!this.processingOrder.includes(key) &&
|
||||
this.currentlyProcessing !== key
|
||||
) {
|
||||
this.addToProcessingOrder(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
this.triggerCountChanged();
|
||||
this.processNext();
|
||||
}
|
||||
|
||||
// --- Key migration ---
|
||||
|
||||
public migrateKey(oldKey: DocumentKey, newDocumentId: string): void {
|
||||
const state = this.documentStates.get(oldKey);
|
||||
if (state === undefined) return;
|
||||
|
||||
this.documentStates.delete(oldKey);
|
||||
this.removeFromProcessingOrder(oldKey);
|
||||
|
||||
const existingNew = this.documentStates.get(newDocumentId);
|
||||
if (existingNew !== undefined) {
|
||||
// Merge: coalesce the old state into the new key's state.
|
||||
// This is unusual but can happen during key resolution races.
|
||||
// Keep the existing state at the new key (it's more recent).
|
||||
} else {
|
||||
this.documentStates.set(newDocumentId, state);
|
||||
this.addToProcessingOrder(newDocumentId);
|
||||
}
|
||||
}
|
||||
|
||||
// --- Processing ---
|
||||
|
||||
public hasOutstandingWork(): boolean {
|
||||
return this.documentStates.size > 0 || this.currentOperation !== null;
|
||||
}
|
||||
|
||||
public hasPendingEventsFor(key: string): boolean {
|
||||
return (
|
||||
this.documentStates.has(key) ||
|
||||
this.documentStates.has("path:" + key) ||
|
||||
this.currentlyProcessing === key ||
|
||||
this.currentlyProcessing === "path:" + key
|
||||
);
|
||||
}
|
||||
|
||||
public get pendingDocumentCount(): number {
|
||||
return (
|
||||
this.documentStates.size +
|
||||
(this.currentOperation !== null ? 1 : 0)
|
||||
);
|
||||
}
|
||||
|
||||
public async waitForIdle(): Promise<void> {
|
||||
// When paused, consider the queue idle if no operation is running.
|
||||
// Queued events exist but are intentionally held until resume().
|
||||
if (this.currentOperation === null && (this.isPaused || this.documentStates.size === 0)) {
|
||||
return;
|
||||
}
|
||||
return new Promise<void>((resolve) => {
|
||||
this.idleWaiters.push(resolve);
|
||||
});
|
||||
}
|
||||
|
||||
// --- Reset ---
|
||||
|
||||
public reset(): void {
|
||||
this.isResetting = true;
|
||||
|
||||
// Remove remote events (server will replay on reconnect).
|
||||
// Preserve local events (unsynced user actions).
|
||||
for (const [key, state] of this.documentStates.entries()) {
|
||||
if (
|
||||
state.action === "remote-update" ||
|
||||
state.action === "remote-delete"
|
||||
) {
|
||||
this.documentStates.delete(key);
|
||||
this.removeFromProcessingOrder(key);
|
||||
}
|
||||
}
|
||||
|
||||
this.idleWaiters.length = 0;
|
||||
}
|
||||
|
||||
public clearResetting(): void {
|
||||
this.isResetting = false;
|
||||
}
|
||||
|
||||
/** Pause processing. Events can still be enqueued but won't be executed. */
|
||||
public pause(): void {
|
||||
this.isPaused = true;
|
||||
}
|
||||
|
||||
/** Resume processing. Immediately processes any queued events. */
|
||||
public resume(): void {
|
||||
this.isPaused = false;
|
||||
this.processNext();
|
||||
}
|
||||
|
||||
public destroy(): void {
|
||||
this.documentStates.clear();
|
||||
this.processingOrder.length = 0;
|
||||
this.currentlyProcessing = null;
|
||||
this.idleWaiters.length = 0;
|
||||
}
|
||||
|
||||
// --- Internal ---
|
||||
|
||||
private resolveKey(event: SyncEvent): DocumentKey {
|
||||
switch (event.type) {
|
||||
case "remote-update":
|
||||
case "remote-delete":
|
||||
return event.version.documentId;
|
||||
case "local-create":
|
||||
return "path:" + event.path;
|
||||
case "local-update":
|
||||
case "local-delete": {
|
||||
const doc = this.vfs.getByPath(event.path);
|
||||
if (doc !== undefined && doc.state !== "pending") {
|
||||
return doc.documentId;
|
||||
}
|
||||
return "path:" + event.path;
|
||||
}
|
||||
case "local-move": {
|
||||
const doc =
|
||||
this.vfs.getByPath(event.toPath) ??
|
||||
this.vfs.getByPath(event.fromPath);
|
||||
if (doc !== undefined && doc.state !== "pending") {
|
||||
return doc.documentId;
|
||||
}
|
||||
return "path:" + event.fromPath;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private processNext(): void {
|
||||
if (this.currentOperation !== null) return;
|
||||
|
||||
if (this.isPaused) {
|
||||
// Even when paused, resolve idle waiters since no operation is
|
||||
// running. This is needed because internalReconcile() pauses the
|
||||
// queue then calls waitForIdle() — if a previously-started
|
||||
// operation finishes while paused, idle waiters must be notified.
|
||||
if (this.idleWaiters.length > 0) {
|
||||
const waiters = this.idleWaiters.splice(0);
|
||||
for (const w of waiters) w();
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
while (this.processingOrder.length > 0) {
|
||||
const key = this.processingOrder.shift()!;
|
||||
const action = this.documentStates.get(key);
|
||||
|
||||
if (action === undefined || action.action === "noop") {
|
||||
this.documentStates.delete(key);
|
||||
continue;
|
||||
}
|
||||
|
||||
this.currentlyProcessing = key;
|
||||
this.documentStates.delete(key);
|
||||
|
||||
this.currentOperation = (async () => {
|
||||
try {
|
||||
if (this.isResetting) throw new SyncResetError();
|
||||
if (this.executor === undefined) {
|
||||
throw new Error("No executor set");
|
||||
}
|
||||
await this.executor(key, action);
|
||||
} catch (e) {
|
||||
if (!(e instanceof SyncResetError)) {
|
||||
this.logger.info(
|
||||
`Sync operation for ${key} failed, will retry: ${e}`
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
this.currentlyProcessing = null;
|
||||
this.currentOperation = null;
|
||||
this.triggerCountChanged();
|
||||
this.processNext();
|
||||
}
|
||||
})();
|
||||
return; // processNext will be called again in finally
|
||||
}
|
||||
|
||||
// Queue is empty, resolve idle waiters
|
||||
if (this.currentOperation === null) {
|
||||
const waiters = this.idleWaiters.splice(0);
|
||||
for (const w of waiters) w();
|
||||
this.triggerCountChanged();
|
||||
}
|
||||
}
|
||||
|
||||
private addToProcessingOrder(key: DocumentKey): void {
|
||||
if (!this.processingOrder.includes(key)) {
|
||||
this.processingOrder.push(key);
|
||||
}
|
||||
}
|
||||
|
||||
private removeFromProcessingOrder(key: DocumentKey): void {
|
||||
const idx = this.processingOrder.indexOf(key);
|
||||
if (idx !== -1) this.processingOrder.splice(idx, 1);
|
||||
}
|
||||
|
||||
private triggerCountChanged(): void {
|
||||
this.onRemainingOperationsCountChanged.trigger(
|
||||
this.pendingDocumentCount
|
||||
);
|
||||
}
|
||||
}
|
||||
301
frontend/sync-client/src/sync-operations/sync-events.ts
Normal file
301
frontend/sync-client/src/sync-operations/sync-events.ts
Normal file
|
|
@ -0,0 +1,301 @@
|
|||
import type { DocumentVersionWithoutContent } from "../services/types/DocumentVersionWithoutContent";
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Raw sync events — emitted by file watchers and WebSocket handlers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type SyncEvent =
|
||||
| { type: "local-create"; path: string }
|
||||
| { type: "local-update"; path: string }
|
||||
| { type: "local-delete"; path: string }
|
||||
| { type: "local-move"; fromPath: string; toPath: string }
|
||||
| { type: "remote-update"; version: DocumentVersionWithoutContent }
|
||||
| { type: "remote-delete"; version: DocumentVersionWithoutContent };
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Coalesced actions — the result of merging multiple events on the same key
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
export type CoalescedAction =
|
||||
| { action: "create"; path: string }
|
||||
| { action: "update"; path: string }
|
||||
| { action: "delete"; path: string }
|
||||
| { action: "move"; fromPath: string; toPath: string }
|
||||
| { action: "move-and-update"; fromPath: string; toPath: string }
|
||||
| { action: "remote-update"; version: DocumentVersionWithoutContent }
|
||||
| { action: "remote-delete"; version: DocumentVersionWithoutContent }
|
||||
| { action: "noop" };
|
||||
|
||||
/**
|
||||
* Convert a single SyncEvent to its initial CoalescedAction.
|
||||
*/
|
||||
export function eventToInitialAction(event: SyncEvent): CoalescedAction {
|
||||
switch (event.type) {
|
||||
case "local-create":
|
||||
return { action: "create", path: event.path };
|
||||
case "local-update":
|
||||
return { action: "update", path: event.path };
|
||||
case "local-delete":
|
||||
return { action: "delete", path: event.path };
|
||||
case "local-move":
|
||||
return {
|
||||
action: "move",
|
||||
fromPath: event.fromPath,
|
||||
toPath: event.toPath
|
||||
};
|
||||
case "remote-update":
|
||||
return { action: "remote-update", version: event.version };
|
||||
case "remote-delete":
|
||||
return { action: "remote-delete", version: event.version };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Coalesce a new SyncEvent into an existing CoalescedAction.
|
||||
*
|
||||
* This implements the full transition table for combining sequential events
|
||||
* that target the same logical document. The goal is to reduce multiple
|
||||
* events into a single action that captures the net effect.
|
||||
*
|
||||
* Transition table (current action x new event -> result):
|
||||
*
|
||||
* | Current \ New Event | local-create | local-update | local-delete | local-move(to) | remote-update | remote-delete |
|
||||
* |---------------------|-------------|-------------|-------------|----------------|---------------|---------------|
|
||||
* | create | create | create | noop | create(to) | create | noop |
|
||||
* | update | update | update | delete | move-and-update| remote-update | delete |
|
||||
* | delete | create | update | delete | move | remote-update | delete |
|
||||
* | move | move | move-and-upd| delete | move(orig,to) | move | delete |
|
||||
* | move-and-update | move-and-upd| move-and-upd| delete | m-a-u(orig,to) | move-and-upd | delete |
|
||||
* | remote-update | create | remote-upd | remote-del | remote-upd | remote-upd | remote-del |
|
||||
* | remote-delete | create | remote-del | remote-del | remote-del | remote-upd | remote-del |
|
||||
* | noop | create | update | delete | move | remote-update | remote-delete |
|
||||
*/
|
||||
export function coalesce(
|
||||
current: CoalescedAction,
|
||||
event: SyncEvent
|
||||
): CoalescedAction {
|
||||
switch (current.action) {
|
||||
case "create":
|
||||
return coalesceFromCreate(current, event);
|
||||
case "update":
|
||||
return coalesceFromUpdate(current, event);
|
||||
case "delete":
|
||||
return coalesceFromDelete(event);
|
||||
case "move":
|
||||
return coalesceFromMove(current, event);
|
||||
case "move-and-update":
|
||||
return coalesceFromMoveAndUpdate(current, event);
|
||||
case "remote-update":
|
||||
return coalesceFromRemoteUpdate(current, event);
|
||||
case "remote-delete":
|
||||
return coalesceFromRemoteDelete(current, event);
|
||||
case "noop":
|
||||
return eventToInitialAction(event);
|
||||
}
|
||||
}
|
||||
|
||||
function coalesceFromCreate(
|
||||
current: { action: "create"; path: string },
|
||||
event: SyncEvent
|
||||
): CoalescedAction {
|
||||
switch (event.type) {
|
||||
case "local-create":
|
||||
// create + create = still create (idempotent)
|
||||
return current;
|
||||
case "local-update":
|
||||
// create + update = still create (content will be read at sync time)
|
||||
return current;
|
||||
case "local-delete":
|
||||
// create + delete = noop (file never reached server)
|
||||
return { action: "noop" };
|
||||
case "local-move":
|
||||
// create + move = create at new path
|
||||
return { action: "create", path: event.toPath };
|
||||
case "remote-update":
|
||||
// create + remote-update = still create (local create takes precedence)
|
||||
return current;
|
||||
case "remote-delete":
|
||||
// create + remote-delete = noop
|
||||
return { action: "noop" };
|
||||
}
|
||||
}
|
||||
|
||||
function coalesceFromUpdate(
|
||||
current: { action: "update"; path: string },
|
||||
event: SyncEvent
|
||||
): CoalescedAction {
|
||||
switch (event.type) {
|
||||
case "local-create":
|
||||
// update + create = update (file was already tracked)
|
||||
return current;
|
||||
case "local-update":
|
||||
// update + update = update
|
||||
return current;
|
||||
case "local-delete":
|
||||
// update + delete = delete
|
||||
return { action: "delete", path: current.path };
|
||||
case "local-move":
|
||||
// update + move = move-and-update
|
||||
return {
|
||||
action: "move-and-update",
|
||||
fromPath: event.fromPath,
|
||||
toPath: event.toPath
|
||||
};
|
||||
case "remote-update":
|
||||
// update + remote-update = remote-update (forces server fetch
|
||||
// so remote changes are applied even when there are no local edits)
|
||||
return { action: "remote-update", version: event.version };
|
||||
case "remote-delete":
|
||||
// update + remote-delete = delete
|
||||
return { action: "delete", path: current.path };
|
||||
}
|
||||
}
|
||||
|
||||
function coalesceFromDelete(event: SyncEvent): CoalescedAction {
|
||||
switch (event.type) {
|
||||
case "local-create":
|
||||
// delete + create = create (file re-created)
|
||||
return { action: "create", path: event.path };
|
||||
case "local-update":
|
||||
// delete + update = update (file re-appeared with changes)
|
||||
return { action: "update", path: event.path };
|
||||
case "local-delete":
|
||||
// delete + delete = delete (idempotent)
|
||||
return { action: "delete", path: event.path };
|
||||
case "local-move":
|
||||
// delete + move = move (the original delete is superseded)
|
||||
return {
|
||||
action: "move",
|
||||
fromPath: event.fromPath,
|
||||
toPath: event.toPath
|
||||
};
|
||||
case "remote-update":
|
||||
// delete + remote-update = remote-update (server has new version)
|
||||
return { action: "remote-update", version: event.version };
|
||||
case "remote-delete":
|
||||
// delete + remote-delete = delete
|
||||
return { action: "delete", path: event.version.relativePath };
|
||||
}
|
||||
}
|
||||
|
||||
function coalesceFromMove(
|
||||
current: { action: "move"; fromPath: string; toPath: string },
|
||||
event: SyncEvent
|
||||
): CoalescedAction {
|
||||
switch (event.type) {
|
||||
case "local-create":
|
||||
// move + create = move (file already at destination)
|
||||
return current;
|
||||
case "local-update":
|
||||
// move + update = move-and-update
|
||||
return {
|
||||
action: "move-and-update",
|
||||
fromPath: current.fromPath,
|
||||
toPath: current.toPath
|
||||
};
|
||||
case "local-delete":
|
||||
// move + delete = delete (from original path)
|
||||
return { action: "delete", path: current.fromPath };
|
||||
case "local-move":
|
||||
// move(A->B) + move(B->C) = move(A->C)
|
||||
return {
|
||||
action: "move",
|
||||
fromPath: current.fromPath,
|
||||
toPath: event.toPath
|
||||
};
|
||||
case "remote-update":
|
||||
// move + remote-update = move (local move takes precedence)
|
||||
return current;
|
||||
case "remote-delete":
|
||||
// move + remote-delete = delete
|
||||
return { action: "delete", path: current.fromPath };
|
||||
}
|
||||
}
|
||||
|
||||
function coalesceFromMoveAndUpdate(
|
||||
current: { action: "move-and-update"; fromPath: string; toPath: string },
|
||||
event: SyncEvent
|
||||
): CoalescedAction {
|
||||
switch (event.type) {
|
||||
case "local-create":
|
||||
// move-and-update + create = move-and-update
|
||||
return current;
|
||||
case "local-update":
|
||||
// move-and-update + update = move-and-update
|
||||
return current;
|
||||
case "local-delete":
|
||||
// move-and-update + delete = delete (from original path)
|
||||
return { action: "delete", path: current.fromPath };
|
||||
case "local-move":
|
||||
// move-and-update(A->B) + move(B->C) = move-and-update(A->C)
|
||||
return {
|
||||
action: "move-and-update",
|
||||
fromPath: current.fromPath,
|
||||
toPath: event.toPath
|
||||
};
|
||||
case "remote-update":
|
||||
// move-and-update + remote-update = move-and-update
|
||||
return current;
|
||||
case "remote-delete":
|
||||
// move-and-update + remote-delete = delete
|
||||
return { action: "delete", path: current.fromPath };
|
||||
}
|
||||
}
|
||||
|
||||
function coalesceFromRemoteUpdate(
|
||||
current: { action: "remote-update"; version: DocumentVersionWithoutContent },
|
||||
event: SyncEvent
|
||||
): CoalescedAction {
|
||||
switch (event.type) {
|
||||
case "local-create":
|
||||
// remote-update + create = create (local create wins — will be
|
||||
// sent to server, which will merge or deconflict)
|
||||
return { action: "create", path: event.path };
|
||||
case "local-update":
|
||||
// remote-update + update = remote-update (will merge on sync)
|
||||
return current;
|
||||
case "local-delete":
|
||||
// remote-update + local-delete = remote-delete
|
||||
return { action: "remote-delete", version: current.version };
|
||||
case "local-move":
|
||||
// remote-update + move = remote-update (path change handled separately)
|
||||
return current;
|
||||
case "remote-update":
|
||||
// remote-update + remote-update = remote-update (latest version)
|
||||
return { action: "remote-update", version: event.version };
|
||||
case "remote-delete":
|
||||
// remote-update + remote-delete = remote-delete
|
||||
return { action: "remote-delete", version: event.version };
|
||||
}
|
||||
}
|
||||
|
||||
function coalesceFromRemoteDelete(
|
||||
current: {
|
||||
action: "remote-delete";
|
||||
version: DocumentVersionWithoutContent;
|
||||
},
|
||||
event: SyncEvent
|
||||
): CoalescedAction {
|
||||
switch (event.type) {
|
||||
case "local-create":
|
||||
// remote-delete + create = create (local create takes precedence —
|
||||
// the user explicitly created a file; the remote delete was for the
|
||||
// OLD document, the create is for a NEW one)
|
||||
return { action: "create", path: event.path };
|
||||
case "local-update":
|
||||
// remote-delete + update = remote-delete
|
||||
return current;
|
||||
case "local-delete":
|
||||
// remote-delete + local-delete = remote-delete
|
||||
return current;
|
||||
case "local-move":
|
||||
// remote-delete + move = remote-delete
|
||||
return current;
|
||||
case "remote-update":
|
||||
// remote-delete + remote-update = remote-update (server changed its mind)
|
||||
return { action: "remote-update", version: event.version };
|
||||
case "remote-delete":
|
||||
// remote-delete + remote-delete = remote-delete (latest)
|
||||
return { action: "remote-delete", version: event.version };
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,826 +0,0 @@
|
|||
import type {
|
||||
Database,
|
||||
DocumentRecord,
|
||||
RelativePath
|
||||
} from "../persistence/database";
|
||||
import { diff } from "reconcile-text";
|
||||
import type { SyncService } from "../services/sync-service";
|
||||
import type { Logger } from "../tracing/logger";
|
||||
import type {
|
||||
CommonHistoryEntry,
|
||||
SyncCreateDetails,
|
||||
SyncDeleteDetails,
|
||||
SyncDetails,
|
||||
SyncHistory,
|
||||
SyncMovedDetails,
|
||||
SyncUpdateDetails
|
||||
} from "../tracing/sync-history";
|
||||
import { SyncStatus, SyncType } from "../tracing/sync-history";
|
||||
import { EMPTY_HASH, hash } from "../utils/hash";
|
||||
import { base64ToBytes } from "byte-base64";
|
||||
import type { Settings } from "../persistence/settings";
|
||||
import type { FileOperations } from "../file-operations/file-operations";
|
||||
import { FileNotFoundError } from "../errors/file-not-found-error";
|
||||
import { SyncResetError } from "../errors/sync-reset-error";
|
||||
import { globsToRegexes } from "../utils/globs-to-regexes";
|
||||
import type { DocumentVersion } from "../services/types/DocumentVersion";
|
||||
import type { DocumentUpdateResponse } from "../services/types/DocumentUpdateResponse";
|
||||
import type { DocumentVersionWithoutContent } from "../services/types/DocumentVersionWithoutContent";
|
||||
import type { FixedSizeDocumentCache } from "../utils/data-structures/fix-sized-cache";
|
||||
import { isFileTypeMergable } from "../utils/is-file-type-mergable";
|
||||
import { isBinary } from "../utils/is-binary";
|
||||
import type { ServerConfig } from "../services/server-config";
|
||||
|
||||
export class UnrestrictedSyncer {
|
||||
private ignorePatterns: RegExp[];
|
||||
|
||||
public constructor(
|
||||
private readonly logger: Logger,
|
||||
private readonly database: Database,
|
||||
private readonly settings: Settings,
|
||||
private readonly syncService: SyncService,
|
||||
private readonly operations: FileOperations,
|
||||
private readonly history: SyncHistory,
|
||||
private readonly contentCache: FixedSizeDocumentCache,
|
||||
private readonly serverConfig: ServerConfig
|
||||
) {
|
||||
this.ignorePatterns = globsToRegexes(
|
||||
this.settings.getSettings().ignorePatterns,
|
||||
this.logger
|
||||
);
|
||||
|
||||
this.settings.onSettingsChanged.add((newSettings) => {
|
||||
this.ignorePatterns = globsToRegexes(
|
||||
newSettings.ignorePatterns,
|
||||
this.logger
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
public async resolveIdempotencyKeys(): Promise<void> {
|
||||
const pendingDocs = this.database.pendingDocuments;
|
||||
if (pendingDocs.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const keys = pendingDocs
|
||||
.map((d) => d.idempotencyKey)
|
||||
// eslint-disable-next-line no-restricted-syntax -- Type narrowing, not removing a specific item
|
||||
.filter((k): k is string => k !== undefined);
|
||||
if (keys.length === 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
this.logger.debug(
|
||||
`Resolving ${keys.length} pending idempotency keys`
|
||||
);
|
||||
|
||||
const resolved =
|
||||
await this.syncService.resolveIdempotencyKeys(keys);
|
||||
|
||||
for (const doc of pendingDocs) {
|
||||
if (
|
||||
doc.idempotencyKey !== undefined &&
|
||||
resolved.has(doc.idempotencyKey)
|
||||
) {
|
||||
// Check if document was removed by a concurrent operation
|
||||
// (e.g., a delete) between the snapshot and now
|
||||
if (!this.database.containsDocument(doc)) {
|
||||
this.logger.info(
|
||||
`Pending doc at ${doc.relativePath} was removed during key resolution, skipping`
|
||||
);
|
||||
continue;
|
||||
}
|
||||
|
||||
const documentId = resolved.get(doc.idempotencyKey)!; // eslint-disable-line @typescript-eslint/no-non-null-assertion
|
||||
|
||||
// Skip if this documentId is already assigned to another document
|
||||
const existing =
|
||||
this.database.getDocumentByDocumentId(documentId);
|
||||
if (existing !== undefined) {
|
||||
this.logger.debug(
|
||||
`Document ${documentId} already exists at ${existing.relativePath}, removing stale pending doc at ${doc.relativePath}`
|
||||
);
|
||||
this.database.removeDocument(doc);
|
||||
continue;
|
||||
}
|
||||
|
||||
this.logger.info(
|
||||
`Resolved idempotency key ${doc.idempotencyKey} to document ${documentId} for ${doc.relativePath}`
|
||||
);
|
||||
this.database.updateDocumentMetadata(
|
||||
{
|
||||
documentId,
|
||||
parentVersionId: 0,
|
||||
hash: "",
|
||||
remoteRelativePath: doc.relativePath
|
||||
},
|
||||
doc
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public async unrestrictedSyncLocallyCreatedOrUpdatedFile({
|
||||
oldPath,
|
||||
// We use the same code path for both local and remote updates. We need to force the update
|
||||
// if there are no local changes but we know that the remote version is newer.
|
||||
force = false,
|
||||
document
|
||||
}: {
|
||||
oldPath?: RelativePath;
|
||||
force?: boolean;
|
||||
document: DocumentRecord;
|
||||
}): Promise<void> {
|
||||
const updateDetails:
|
||||
| SyncCreateDetails
|
||||
| SyncUpdateDetails
|
||||
| SyncMovedDetails =
|
||||
document.metadata === undefined
|
||||
? {
|
||||
type: SyncType.CREATE,
|
||||
relativePath: document.relativePath
|
||||
}
|
||||
: oldPath !== undefined
|
||||
? {
|
||||
type: SyncType.MOVE,
|
||||
relativePath: document.relativePath,
|
||||
movedFrom: oldPath
|
||||
}
|
||||
: {
|
||||
type: SyncType.UPDATE,
|
||||
relativePath: document.relativePath
|
||||
};
|
||||
|
||||
await this.executeSync(updateDetails, async () => {
|
||||
const originalRelativePath = document.relativePath;
|
||||
|
||||
if (document.isDeleted) {
|
||||
this.logger.debug(
|
||||
`Document ${document.relativePath} has been already deleted, no need to update it`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const contentBytes = await this.operations.read(
|
||||
document.relativePath
|
||||
); // this can throw FileNotFoundError
|
||||
const contentHash = hash(contentBytes);
|
||||
|
||||
let response: DocumentVersion | DocumentUpdateResponse | undefined =
|
||||
undefined;
|
||||
if (
|
||||
document.metadata === undefined ||
|
||||
document.metadata.parentVersionId === 0
|
||||
) {
|
||||
// parentVersionId === 0 occurs when resolveIdempotencyKeys
|
||||
// assigned a documentId but hasn't synced yet. Treat as a
|
||||
// create — the server will recognise the idempotency key
|
||||
// and return the existing document.
|
||||
response = await this.syncService.create({
|
||||
relativePath: originalRelativePath,
|
||||
contentBytes,
|
||||
idempotencyKey: document.idempotencyKey
|
||||
});
|
||||
|
||||
await this.handleMaybeMergingResponse({
|
||||
document,
|
||||
response,
|
||||
contentHash,
|
||||
originalRelativePath,
|
||||
originalContentBytes: contentBytes,
|
||||
isCreate: true
|
||||
});
|
||||
} else {
|
||||
const areThereLocalChanges =
|
||||
document.metadata.hash !== contentHash ||
|
||||
oldPath !== undefined;
|
||||
|
||||
if (areThereLocalChanges) {
|
||||
const isText =
|
||||
!isBinary(contentBytes) &&
|
||||
isFileTypeMergable(
|
||||
document.relativePath,
|
||||
(await this.serverConfig.getConfig())
|
||||
.mergeableFileExtensions
|
||||
);
|
||||
// Snapshot parentVersionId atomically with the cache
|
||||
// lookup. document.metadata is a mutable shared
|
||||
// reference — a concurrent operation could update
|
||||
// parentVersionId between the cache lookup and the
|
||||
// putText call, causing a diff/version mismatch.
|
||||
const parentVersionIdForUpdate =
|
||||
document.metadata.parentVersionId;
|
||||
const cachedVersion = this.contentCache.get(
|
||||
parentVersionIdForUpdate
|
||||
);
|
||||
|
||||
response =
|
||||
isText && cachedVersion !== undefined
|
||||
? await this.syncService.putText({
|
||||
documentId: document.metadata.documentId,
|
||||
parentVersionId: parentVersionIdForUpdate,
|
||||
relativePath: document.relativePath,
|
||||
content: diff(
|
||||
new TextDecoder().decode(cachedVersion),
|
||||
new TextDecoder().decode(contentBytes)
|
||||
)
|
||||
})
|
||||
: await this.syncService.putBinary({
|
||||
documentId: document.metadata.documentId,
|
||||
parentVersionId: parentVersionIdForUpdate,
|
||||
relativePath: document.relativePath,
|
||||
contentBytes
|
||||
});
|
||||
} else {
|
||||
if (!force) {
|
||||
this.logger.debug(
|
||||
`File hash of ${document.relativePath} matches with last synced version and the path hasn't changed; no need to sync`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// we use this code path (force == true) to sync remotely updated files which have no local changes
|
||||
response = await this.syncService.get({
|
||||
documentId: document.metadata.documentId
|
||||
});
|
||||
}
|
||||
|
||||
await this.handleMaybeMergingResponse({
|
||||
document,
|
||||
response,
|
||||
contentHash,
|
||||
originalRelativePath,
|
||||
originalContentBytes: contentBytes
|
||||
});
|
||||
}
|
||||
|
||||
if (!("type" in response) || response.type === "MergingUpdate") {
|
||||
if (!force) {
|
||||
this.history.addHistoryEntry({
|
||||
status: SyncStatus.SUCCESS,
|
||||
details: updateDetails,
|
||||
message: `The file we updated had been updated remotely, so we downloaded the merged version`
|
||||
});
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
const actualUpdateDetails: SyncUpdateDetails | SyncMovedDetails =
|
||||
oldPath !== undefined ||
|
||||
response.relativePath != originalRelativePath
|
||||
? {
|
||||
type: SyncType.MOVE,
|
||||
relativePath: response.relativePath,
|
||||
movedFrom: originalRelativePath
|
||||
}
|
||||
: {
|
||||
type: SyncType.UPDATE,
|
||||
relativePath: response.relativePath
|
||||
};
|
||||
|
||||
if (!response.isDeleted) {
|
||||
this.history.addHistoryEntry({
|
||||
status: SyncStatus.SUCCESS,
|
||||
details: actualUpdateDetails,
|
||||
message: `Successfully downloaded remotely updated file from the server`,
|
||||
author: response.userId,
|
||||
timestamp: new Date(response.updatedDate)
|
||||
});
|
||||
} else {
|
||||
this.history.addHistoryEntry({
|
||||
status: SyncStatus.SUCCESS,
|
||||
details: {
|
||||
type: SyncType.DELETE,
|
||||
relativePath: document.relativePath
|
||||
},
|
||||
message:
|
||||
"Successfully deleted file which had been deleted remotely",
|
||||
author: response.userId,
|
||||
timestamp: new Date(response.updatedDate)
|
||||
});
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
public async unrestrictedSyncLocallyDeletedFile(
|
||||
document: DocumentRecord
|
||||
): Promise<void> {
|
||||
const updateDetails: SyncDeleteDetails = {
|
||||
type: SyncType.DELETE,
|
||||
relativePath: document.relativePath
|
||||
};
|
||||
|
||||
await this.executeSync(updateDetails, async () => {
|
||||
if (document.metadata === undefined) {
|
||||
this.logger.debug(
|
||||
`Document ${document.relativePath} has never been synced, no need to delete it remotely`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const response = await this.syncService.delete({
|
||||
documentId: document.metadata.documentId,
|
||||
relativePath: document.relativePath
|
||||
});
|
||||
|
||||
// A concurrent merge operation may have removed this document from the
|
||||
// database while we were waiting for the delete response. In that case,
|
||||
// the merge already handled the state transition and we should not
|
||||
// update metadata (which would fail anyway since the document is gone).
|
||||
if (!this.database.containsDocument(document)) {
|
||||
this.logger.debug(
|
||||
`Document ${document.relativePath} was removed from database by a concurrent operation, skipping metadata update after delete`
|
||||
);
|
||||
this.database.addSeenUpdateId(response.vaultUpdateId);
|
||||
return;
|
||||
}
|
||||
|
||||
this.database.updateDocumentMetadata(
|
||||
{
|
||||
documentId: response.documentId,
|
||||
parentVersionId: response.vaultUpdateId,
|
||||
hash: EMPTY_HASH,
|
||||
remoteRelativePath: document.relativePath
|
||||
},
|
||||
document
|
||||
);
|
||||
|
||||
this.database.addSeenUpdateId(response.vaultUpdateId);
|
||||
|
||||
this.history.addHistoryEntry({
|
||||
status: SyncStatus.SUCCESS,
|
||||
details: updateDetails,
|
||||
message: `Successfully deleted locally deleted file on the server`,
|
||||
author: response.userId
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
public async unrestrictedSyncRemotelyUpdatedFile(
|
||||
remoteVersion: DocumentVersionWithoutContent,
|
||||
document?: DocumentRecord
|
||||
): Promise<void> {
|
||||
const updateDetails: SyncCreateDetails = {
|
||||
type: SyncType.CREATE,
|
||||
relativePath: remoteVersion.relativePath
|
||||
};
|
||||
|
||||
await this.executeSync(updateDetails, async () => {
|
||||
if (document?.metadata !== undefined) {
|
||||
// If the file exists locally, let's pretend the user has updated it
|
||||
// and deal with remote update/deletion within `unrestrictedSyncLocallyUpdatedFile`
|
||||
if (
|
||||
document.metadata.parentVersionId >=
|
||||
remoteVersion.vaultUpdateId
|
||||
) {
|
||||
this.logger.debug(
|
||||
`Document ${document.relativePath} is already at least as up-to-date as the fetched version`
|
||||
);
|
||||
|
||||
return;
|
||||
}
|
||||
|
||||
return this.unrestrictedSyncLocallyCreatedOrUpdatedFile({
|
||||
document,
|
||||
force: true
|
||||
});
|
||||
} else if (remoteVersion.isDeleted) {
|
||||
// Either the document hasn't made it to us before and therefore we don't need to delete it,
|
||||
// or we already have it, in which case the preceeding if would've dealt with it
|
||||
this.logger.debug(
|
||||
`Document ${remoteVersion.relativePath} has been deleted remotely, no need to sync`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
// Don't download oversized files
|
||||
const historyEntryForSkippedOversizedFile =
|
||||
this.getHistoryEntryForSkippedOversizedFile(
|
||||
remoteVersion.contentSize,
|
||||
remoteVersion.relativePath
|
||||
);
|
||||
if (historyEntryForSkippedOversizedFile !== undefined) {
|
||||
this.history.addHistoryEntry(
|
||||
historyEntryForSkippedOversizedFile
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
const contentBytes =
|
||||
await this.syncService.getDocumentVersionContent({
|
||||
documentId: remoteVersion.documentId,
|
||||
vaultUpdateId: remoteVersion.vaultUpdateId
|
||||
});
|
||||
|
||||
// We're trying to create an entirely new document that didn't exist locally
|
||||
document = this.database.getDocumentByDocumentId(
|
||||
remoteVersion.documentId
|
||||
);
|
||||
// It can happen that a concurrent sync operation has already created the document, so we can bail here
|
||||
if (document !== undefined) {
|
||||
this.logger.debug(
|
||||
`Document ${remoteVersion.relativePath} has already been created locally, no need to create it again`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
await this.operations.ensureClearPath(remoteVersion.relativePath);
|
||||
|
||||
this.database.updateDocumentMetadata(
|
||||
{
|
||||
documentId: remoteVersion.documentId,
|
||||
parentVersionId: remoteVersion.vaultUpdateId,
|
||||
hash: hash(contentBytes),
|
||||
remoteRelativePath: remoteVersion.relativePath
|
||||
},
|
||||
this.database.createNewPendingDocument(
|
||||
remoteVersion.relativePath
|
||||
)
|
||||
);
|
||||
|
||||
await this.operations.create(
|
||||
remoteVersion.relativePath,
|
||||
contentBytes
|
||||
);
|
||||
await this.updateCache(
|
||||
remoteVersion.vaultUpdateId,
|
||||
contentBytes,
|
||||
remoteVersion.relativePath
|
||||
);
|
||||
|
||||
this.history.addHistoryEntry({
|
||||
status: SyncStatus.SUCCESS,
|
||||
details: updateDetails,
|
||||
message: `Successfully downloaded remote file which hadn't existed locally`,
|
||||
author: remoteVersion.userId,
|
||||
timestamp: new Date(remoteVersion.updatedDate)
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
private async executeSync<T>(
|
||||
details: SyncDetails,
|
||||
fn: () => Promise<T>
|
||||
): Promise<T | undefined> {
|
||||
if (!this.settings.getSettings().isSyncEnabled) {
|
||||
this.logger.info(
|
||||
`Skipping sync operation for file '${details.relativePath}' because sync is disabled`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
for (const pattern of this.ignorePatterns) {
|
||||
if (pattern.test(details.relativePath)) {
|
||||
this.logger.debug(
|
||||
`File '${details.relativePath}' is ignored by the ignore pattern: ${pattern}`
|
||||
);
|
||||
return; // bail without SKIPPED status because we were told to ignore this file and we shouldn't clutter up the history
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
// Only check the size of files which already exist locally.
|
||||
if (await this.operations.exists(details.relativePath)) {
|
||||
const sizeInBytes = await this.operations.getFileSize(
|
||||
details.relativePath
|
||||
);
|
||||
const historyEntryForSkippedOversizedFile =
|
||||
this.getHistoryEntryForSkippedOversizedFile(
|
||||
sizeInBytes,
|
||||
details.relativePath
|
||||
);
|
||||
if (historyEntryForSkippedOversizedFile !== undefined) {
|
||||
this.history.addHistoryEntry(
|
||||
historyEntryForSkippedOversizedFile
|
||||
);
|
||||
return;
|
||||
}
|
||||
}
|
||||
|
||||
return await fn();
|
||||
} catch (e) {
|
||||
if (e instanceof FileNotFoundError) {
|
||||
// A subsequent sync operation must have been creating to deal with this
|
||||
this.logger.info(
|
||||
`Skiping file '${details.relativePath}' because it no longer exists when trying to ${details.type.toLocaleLowerCase()} it`
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (e instanceof SyncResetError) {
|
||||
this.logger.info(
|
||||
`Interrupting sync operation because of a reset`
|
||||
);
|
||||
return;
|
||||
} else {
|
||||
this.history.addHistoryEntry({
|
||||
status: SyncStatus.ERROR,
|
||||
details,
|
||||
message: `Failed to sync file '${details.relativePath}' because of ${e} when trying to ${details.type.toLocaleLowerCase()} it`
|
||||
});
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async handleMaybeMergingResponse({
|
||||
document,
|
||||
response,
|
||||
contentHash,
|
||||
originalRelativePath,
|
||||
originalContentBytes,
|
||||
isCreate
|
||||
}: {
|
||||
document: DocumentRecord;
|
||||
response: DocumentVersion | DocumentUpdateResponse;
|
||||
contentHash: string;
|
||||
originalRelativePath: string;
|
||||
originalContentBytes: Uint8Array;
|
||||
isCreate?: boolean;
|
||||
}): Promise<void> {
|
||||
// `document` is mutable and reflects the latest state in the local database
|
||||
if (document.isDeleted) {
|
||||
this.logger.info(
|
||||
`Document ${document.relativePath} has been deleted before we could finish updating it`
|
||||
);
|
||||
// Assign metadata so the pending delete can inform the server
|
||||
if (document.metadata === undefined) {
|
||||
const existingWithSameId =
|
||||
this.database.getDocumentByDocumentId(
|
||||
response.documentId
|
||||
);
|
||||
if (
|
||||
existingWithSameId !== undefined &&
|
||||
existingWithSameId !== document
|
||||
) {
|
||||
// Another doc already has this documentId — the server
|
||||
// knows about it. Just remove this stale pending doc.
|
||||
this.database.removeDocument(document);
|
||||
} else {
|
||||
this.database.updateDocumentMetadata(
|
||||
{
|
||||
documentId: response.documentId,
|
||||
parentVersionId: response.vaultUpdateId,
|
||||
hash: contentHash,
|
||||
remoteRelativePath: response.relativePath
|
||||
},
|
||||
document
|
||||
);
|
||||
}
|
||||
}
|
||||
this.database.addSeenUpdateId(response.vaultUpdateId);
|
||||
return;
|
||||
}
|
||||
|
||||
if (
|
||||
(document.metadata?.parentVersionId ?? 0) > response.vaultUpdateId
|
||||
) {
|
||||
this.logger.debug(
|
||||
`Document ${document.relativePath} is already more up to date than the fetched version`
|
||||
);
|
||||
this.database.addSeenUpdateId(response.vaultUpdateId); // in case the previous `vaultUpdateId` update hasn't made it through
|
||||
return;
|
||||
}
|
||||
|
||||
if (response.isDeleted) {
|
||||
return this.applyRemoteDeleteLocally(document, response);
|
||||
}
|
||||
|
||||
let actualPath = document.relativePath;
|
||||
|
||||
let existingContentBytes: Uint8Array | undefined;
|
||||
|
||||
if (isCreate) {
|
||||
// We have a file locally that got moved by another client to the same path as the one we're trying to create.
|
||||
// The server returns a merging update for the document ID that already exists locally (but at another path).
|
||||
// We have to merge these two documents by extending the provenance of the existing document and deleting
|
||||
// the old document that the new document already contains the content for.
|
||||
const existingDocument = this.database.getDocumentByDocumentId(
|
||||
response.documentId
|
||||
);
|
||||
// If existingDocument === document, then a previous sync operation already
|
||||
// assigned this documentId to our document. We don't need to merge - just
|
||||
// continue to update the metadata below.
|
||||
if (existingDocument !== undefined && existingDocument !== document) {
|
||||
this.logger.info(
|
||||
`Merging existing document ${existingDocument.relativePath} into ${document.relativePath
|
||||
} after concurrent move & creation`
|
||||
);
|
||||
if (!existingDocument.isDeleted) {
|
||||
this.database.delete(existingDocument.relativePath); // make sure syncLocallyDeletedFile doesn't actually schedule deleting the new file
|
||||
|
||||
try {
|
||||
existingContentBytes = await this.operations.read(
|
||||
existingDocument.relativePath
|
||||
);
|
||||
} catch (e) {
|
||||
if (e instanceof FileNotFoundError) {
|
||||
return;
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
|
||||
this.database.removeDocument(existingDocument);
|
||||
await this.operations.delete(existingDocument.relativePath);
|
||||
|
||||
} else {
|
||||
this.database.removeDocument(existingDocument);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A document's documentId should never change once assigned. If the response has a
|
||||
// different documentId than what the document already has, it means the file was
|
||||
// renamed during the sync operation and the response is for a different document.
|
||||
// We should bail out and let subsequent sync operations fix the state.
|
||||
if (
|
||||
document.metadata?.documentId !== undefined &&
|
||||
document.metadata.documentId !== response.documentId
|
||||
) {
|
||||
this.logger.info(
|
||||
`Document ${document.relativePath} already has documentId ${document.metadata.documentId}, ` +
|
||||
`but response has documentId ${response.documentId}. Ignoring response to prevent documentId corruption.`
|
||||
);
|
||||
this.database.addSeenUpdateId(response.vaultUpdateId);
|
||||
return;
|
||||
}
|
||||
|
||||
// this can't happen on the creation path as we can only get a merging response if a document already exists remotely on the same path
|
||||
if (response.relativePath != originalRelativePath) {
|
||||
actualPath = response.relativePath;
|
||||
// Make sure to update the remote relative path to avoid uploading
|
||||
// the file as a result of this filesystem event.
|
||||
if (document.metadata !== undefined) {
|
||||
document.metadata.remoteRelativePath = response.relativePath;
|
||||
}
|
||||
await this.operations.move(
|
||||
document.relativePath,
|
||||
response.relativePath
|
||||
); // this can throw FileNotFoundError
|
||||
}
|
||||
|
||||
if (!("type" in response) || response.type === "MergingUpdate") {
|
||||
const responseBytes = base64ToBytes(response.contentBase64);
|
||||
|
||||
// Write file BEFORE updating metadata so that if the write fails,
|
||||
// metadata doesn't point to a version whose content was never written.
|
||||
await this.operations.write(
|
||||
actualPath,
|
||||
originalContentBytes,
|
||||
responseBytes
|
||||
);
|
||||
|
||||
if (existingContentBytes !== undefined) {
|
||||
// the merge case is only always for text files, so don't mind that we have to provide a byte array here
|
||||
await this.operations.write(
|
||||
actualPath,
|
||||
new Uint8Array(0),
|
||||
existingContentBytes
|
||||
);
|
||||
}
|
||||
|
||||
// Re-read and re-hash after write because the 3-way merge in
|
||||
// operations.write() may produce content different from responseBytes.
|
||||
const actualContent = await this.operations.read(actualPath);
|
||||
const actualHash = hash(actualContent);
|
||||
|
||||
// The document may have been removed by a concurrent operation
|
||||
// (e.g., a delete) during the awaited file write/read above.
|
||||
// The file is safely on disk; recovery will re-detect it.
|
||||
if (!this.database.containsDocument(document)) {
|
||||
this.logger.info(
|
||||
`Document ${document.relativePath} was removed during sync, skipping metadata update`
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
this.database.updateDocumentMetadata(
|
||||
{
|
||||
documentId: response.documentId,
|
||||
parentVersionId: response.vaultUpdateId,
|
||||
hash: actualHash,
|
||||
remoteRelativePath: response.relativePath
|
||||
},
|
||||
document
|
||||
);
|
||||
|
||||
// Cache the SERVER's content (responseBytes), not the local
|
||||
// content (actualContent). The cache is used to compute diffs
|
||||
// for subsequent updates: diff(cached, newFileContent). The
|
||||
// server applies this diff against its content at
|
||||
// parentVersionId, which is responseBytes. Using actualContent
|
||||
// would produce diffs that don't match the server's state.
|
||||
await this.updateCache(
|
||||
response.vaultUpdateId,
|
||||
responseBytes,
|
||||
actualPath
|
||||
);
|
||||
} else {
|
||||
// FastForwardUpdate — the server accepted our content as-is,
|
||||
// UNLESS this was an idempotent create return (the server
|
||||
// returned the original version, whose content may differ from
|
||||
// what we sent). Detect this by comparing contentSize.
|
||||
const serverContentMatchesLocal =
|
||||
!("contentSize" in response) ||
|
||||
response.contentSize === originalContentBytes.length;
|
||||
|
||||
if (serverContentMatchesLocal) {
|
||||
this.database.updateDocumentMetadata(
|
||||
{
|
||||
documentId: response.documentId,
|
||||
parentVersionId: response.vaultUpdateId,
|
||||
hash: contentHash,
|
||||
remoteRelativePath: response.relativePath
|
||||
},
|
||||
document
|
||||
);
|
||||
await this.updateCache(
|
||||
response.vaultUpdateId,
|
||||
originalContentBytes,
|
||||
actualPath
|
||||
);
|
||||
} else {
|
||||
// The server returned a stale idempotent version. Fetch
|
||||
// the actual content so the cache stays consistent, then
|
||||
// the hash mismatch will trigger a follow-up update sync.
|
||||
const serverContent =
|
||||
await this.syncService.getDocumentVersionContent({
|
||||
documentId: response.documentId,
|
||||
vaultUpdateId: response.vaultUpdateId
|
||||
});
|
||||
this.database.updateDocumentMetadata(
|
||||
{
|
||||
documentId: response.documentId,
|
||||
parentVersionId: response.vaultUpdateId,
|
||||
hash: hash(serverContent),
|
||||
remoteRelativePath: response.relativePath
|
||||
},
|
||||
document
|
||||
);
|
||||
await this.updateCache(
|
||||
response.vaultUpdateId,
|
||||
serverContent,
|
||||
actualPath
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
this.database.addSeenUpdateId(response.vaultUpdateId);
|
||||
}
|
||||
|
||||
private getHistoryEntryForSkippedOversizedFile(
|
||||
sizeInBytes: number,
|
||||
relativePath: RelativePath
|
||||
): CommonHistoryEntry | undefined {
|
||||
const { maxFileSizeMB } = this.settings.getSettings();
|
||||
const maxFileSizeBytes = maxFileSizeMB * 1024 * 1024;
|
||||
if (sizeInBytes > maxFileSizeBytes) {
|
||||
const sizeInMB = (sizeInBytes / 1024 / 1024).toFixed(1);
|
||||
return {
|
||||
status: SyncStatus.SKIPPED,
|
||||
details: {
|
||||
type: SyncType.SKIPPED,
|
||||
relativePath
|
||||
},
|
||||
message: `File size of ${sizeInMB} MB exceeds the maximum file size limit of ${maxFileSizeMB
|
||||
} MB`
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
private async updateCache(
|
||||
updateId: number,
|
||||
contentBytes: Uint8Array,
|
||||
filePath: RelativePath
|
||||
): Promise<void> {
|
||||
if (
|
||||
isFileTypeMergable(
|
||||
filePath,
|
||||
(await this.serverConfig.getConfig()).mergeableFileExtensions
|
||||
) &&
|
||||
!isBinary(contentBytes)
|
||||
) {
|
||||
this.contentCache.put(updateId, contentBytes);
|
||||
}
|
||||
}
|
||||
|
||||
private async applyRemoteDeleteLocally(
|
||||
document: DocumentRecord,
|
||||
response: DocumentVersion | DocumentUpdateResponse
|
||||
): Promise<void> {
|
||||
this.database.delete(document.relativePath);
|
||||
this.database.updateDocumentMetadata(
|
||||
{
|
||||
documentId: response.documentId,
|
||||
parentVersionId: response.vaultUpdateId,
|
||||
hash: EMPTY_HASH,
|
||||
remoteRelativePath: response.relativePath
|
||||
},
|
||||
document
|
||||
);
|
||||
|
||||
await this.operations.delete(document.relativePath);
|
||||
|
||||
this.database.addSeenUpdateId(response.vaultUpdateId);
|
||||
}
|
||||
}
|
||||
46
frontend/sync-client/src/utils/decode-text.ts
Normal file
46
frontend/sync-client/src/utils/decode-text.ts
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
/**
|
||||
* Transcode UTF-16 content to UTF-8. Detects UTF-16 LE/BE by BOM.
|
||||
* Non-UTF-16 content (valid UTF-8 or binary) is returned as-is.
|
||||
*
|
||||
* Call this at the file-read boundary so all downstream code only
|
||||
* deals with UTF-8 bytes or binary.
|
||||
*/
|
||||
export function normalizeToUtf8(content: Uint8Array): Uint8Array {
|
||||
// UTF-16 LE BOM
|
||||
if (content.length >= 2 && content[0] === 0xff && content[1] === 0xfe) {
|
||||
try {
|
||||
const text = new TextDecoder("utf-16le", {
|
||||
fatal: true
|
||||
}).decode(content);
|
||||
return new TextEncoder().encode(text);
|
||||
} catch {
|
||||
return content;
|
||||
}
|
||||
}
|
||||
|
||||
// UTF-16 BE BOM
|
||||
if (content.length >= 2 && content[0] === 0xfe && content[1] === 0xff) {
|
||||
try {
|
||||
const text = new TextDecoder("utf-16be", {
|
||||
fatal: true
|
||||
}).decode(content);
|
||||
return new TextEncoder().encode(text);
|
||||
} catch {
|
||||
return content;
|
||||
}
|
||||
}
|
||||
|
||||
return content;
|
||||
}
|
||||
|
||||
/**
|
||||
* Decode UTF-8 bytes to a string.
|
||||
* Returns `undefined` if the content is not valid UTF-8.
|
||||
*/
|
||||
export function decodeText(content: Uint8Array): string | undefined {
|
||||
try {
|
||||
return new TextDecoder("utf-8", { fatal: true }).decode(content);
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
|
@ -1,14 +0,0 @@
|
|||
import type { DocumentRecord } from "../persistence/database";
|
||||
import { EMPTY_HASH } from "./hash";
|
||||
|
||||
// TODO: make this smarter so that offline files can be renamed & edited at the same time
|
||||
export function findMatchingFile(
|
||||
contentHash: string,
|
||||
candidates: DocumentRecord[]
|
||||
): DocumentRecord | undefined {
|
||||
if (contentHash === EMPTY_HASH) {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
return candidates.find(({ metadata }) => metadata?.hash === contentHash);
|
||||
}
|
||||
|
|
@ -1,12 +1,34 @@
|
|||
// https://stackoverflow.com/questions/7616461/generate-a-hash-from-string-in-javascript
|
||||
export function hash(content: Uint8Array): string {
|
||||
let result = 0;
|
||||
// eslint-disable-next-line @typescript-eslint/prefer-for-of
|
||||
for (let i = 0; i < content.length; i++) {
|
||||
result = (result << 5) - result + content[i];
|
||||
result |= 0; // Convert to 32bit integer
|
||||
import murmurHash3 from "murmurhash3js-revisited";
|
||||
import { decodeText } from "./decode-text";
|
||||
|
||||
/**
|
||||
* Normalize text content for consistent cross-platform hashing:
|
||||
* - Apply Unicode NFC normalization (macOS uses NFD, Linux/Windows use NFC)
|
||||
*
|
||||
* Binary content is returned as-is.
|
||||
*/
|
||||
function normalizeForHashing(content: Uint8Array): Uint8Array {
|
||||
const text = decodeText(content);
|
||||
if (text === undefined) {
|
||||
return content;
|
||||
}
|
||||
return Math.abs(result).toString(16).padStart(8, "0");
|
||||
|
||||
const normalized = text.normalize("NFC");
|
||||
return new TextEncoder().encode(normalized);
|
||||
}
|
||||
|
||||
/**
|
||||
* MurmurHash3 x64 128-bit hash. Produces a 32-character hex string.
|
||||
*
|
||||
* The previous 32-bit hash had ~50% collision probability at ~77k files
|
||||
* (birthday paradox). At 128 bits, collisions are effectively impossible.
|
||||
*
|
||||
* Text content is Unicode NFC-normalized for cross-platform consistency.
|
||||
* Binary content is hashed as-is.
|
||||
*/
|
||||
export function hash(content: Uint8Array): string {
|
||||
const normalized = normalizeForHashing(content);
|
||||
return murmurHash3.x64.hash128(normalized);
|
||||
}
|
||||
|
||||
export const EMPTY_HASH = hash(new Uint8Array(0));
|
||||
|
|
|
|||
|
|
@ -1,16 +1,11 @@
|
|||
// Text is unlikely to contain null bytes, so we can use that to distinguish binary files.
|
||||
import { decodeText } from "./decode-text";
|
||||
|
||||
/**
|
||||
* Determine if the given content is binary (not valid UTF-8).
|
||||
*
|
||||
* Content is expected to have been normalized to UTF-8 at the read
|
||||
* boundary (via `normalizeToUtf8`), so this only checks UTF-8 validity.
|
||||
*/
|
||||
export function isBinary(content: Uint8Array): boolean {
|
||||
for (const byte of content) {
|
||||
if (byte === 0) {
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
new TextDecoder("utf-8", { fatal: true }).decode(content);
|
||||
} catch {
|
||||
return true;
|
||||
}
|
||||
|
||||
return false;
|
||||
return decodeText(content) === undefined;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,81 @@
|
|||
import { describe, it } from "node:test";
|
||||
import assert from "node:assert";
|
||||
import { validateRelativePath } from "./validate-relative-path";
|
||||
|
||||
describe("validateRelativePath", () => {
|
||||
it("accepts normal relative paths", () => {
|
||||
assert.doesNotThrow(() => { validateRelativePath("file.md"); });
|
||||
assert.doesNotThrow(() => { validateRelativePath("folder/file.md"); });
|
||||
assert.doesNotThrow(
|
||||
() => { validateRelativePath("deeply/nested/folder/file.md"); }
|
||||
);
|
||||
assert.doesNotThrow(() => { validateRelativePath("file with spaces.md"); });
|
||||
assert.doesNotThrow(() => { validateRelativePath(".hidden-file"); });
|
||||
assert.doesNotThrow(() => { validateRelativePath("folder/.hidden"); });
|
||||
});
|
||||
|
||||
it("accepts paths with single dots", () => {
|
||||
assert.doesNotThrow(() => { validateRelativePath("./file.md"); });
|
||||
assert.doesNotThrow(() => { validateRelativePath("folder/./file.md"); });
|
||||
});
|
||||
|
||||
it("rejects empty paths", () => {
|
||||
assert.throws(() => { validateRelativePath(""); }, /must not be empty/);
|
||||
});
|
||||
|
||||
it("rejects paths with .. components", () => {
|
||||
assert.throws(
|
||||
() => { validateRelativePath("../file.md"); },
|
||||
/must not contain '\.\.'/
|
||||
);
|
||||
assert.throws(
|
||||
() => { validateRelativePath("folder/../file.md"); },
|
||||
/must not contain '\.\.'/
|
||||
);
|
||||
assert.throws(
|
||||
() => { validateRelativePath("folder/../../etc/passwd"); },
|
||||
/must not contain '\.\.'/
|
||||
);
|
||||
assert.throws(
|
||||
() => { validateRelativePath(".."); },
|
||||
/must not contain '\.\.'/
|
||||
);
|
||||
});
|
||||
|
||||
it("does not reject paths containing .. as part of a filename", () => {
|
||||
assert.doesNotThrow(
|
||||
() => { validateRelativePath("file..name.md"); }
|
||||
);
|
||||
assert.doesNotThrow(
|
||||
() => { validateRelativePath("folder/file..bak"); }
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects absolute paths starting with /", () => {
|
||||
assert.throws(
|
||||
() => { validateRelativePath("/etc/passwd"); },
|
||||
/must be relative/
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects absolute paths starting with \\", () => {
|
||||
assert.throws(
|
||||
() => { validateRelativePath("\\Windows\\System32"); },
|
||||
/must be relative/
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects paths containing backslashes", () => {
|
||||
assert.throws(
|
||||
() => { validateRelativePath("folder\\file.md"); },
|
||||
/must use forward slashes/
|
||||
);
|
||||
});
|
||||
|
||||
it("rejects paths with null bytes", () => {
|
||||
assert.throws(
|
||||
() => { validateRelativePath("file\0.md"); },
|
||||
/null byte/
|
||||
);
|
||||
});
|
||||
});
|
||||
46
frontend/sync-client/src/utils/validate-relative-path.ts
Normal file
46
frontend/sync-client/src/utils/validate-relative-path.ts
Normal file
|
|
@ -0,0 +1,46 @@
|
|||
import type { RelativePath } from "../persistence/database";
|
||||
|
||||
/**
|
||||
* Validates that a relative path is safe and cannot escape the vault root.
|
||||
*
|
||||
* Rejects paths that:
|
||||
* - Are empty
|
||||
* - Start with `/` or `\` (absolute paths)
|
||||
* - Contain `..` path components (directory traversal)
|
||||
* - Contain null bytes (path truncation attacks)
|
||||
* - Contain backslash separators (Windows path injection)
|
||||
*
|
||||
* @throws {Error} if the path is unsafe
|
||||
*/
|
||||
export function validateRelativePath(path: RelativePath): void {
|
||||
if (path.length === 0) {
|
||||
throw new Error("Path must not be empty");
|
||||
}
|
||||
|
||||
if (path.includes("\0")) {
|
||||
throw new Error(
|
||||
`Path contains null byte, which is not allowed: '${path}'`
|
||||
);
|
||||
}
|
||||
|
||||
if (path.startsWith("/") || path.startsWith("\\")) {
|
||||
throw new Error(
|
||||
`Path must be relative, not absolute: '${path}'`
|
||||
);
|
||||
}
|
||||
|
||||
if (path.includes("\\")) {
|
||||
throw new Error(
|
||||
`Path must use forward slashes, not backslashes: '${path}'`
|
||||
);
|
||||
}
|
||||
|
||||
const components = path.split("/");
|
||||
for (const component of components) {
|
||||
if (component === "..") {
|
||||
throw new Error(
|
||||
`Path must not contain '..' components: '${path}'`
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -5,15 +5,19 @@ import { assert } from "../utils/assert";
|
|||
import type { RelativePath, SyncSettings } from "sync-client";
|
||||
import { debugging, Logger, LogLevel, utils } from "sync-client";
|
||||
import { MockClient } from "./mock-client";
|
||||
import { sleep } from "../utils/sleep";
|
||||
import type { LogLine } from "sync-client";
|
||||
import { withTimeout } from "../utils/with-timeout";
|
||||
import type { TestErrorTracker } from "../utils/test-error-tracker";
|
||||
|
||||
const TIMEOUT_MS = 10 * 60 * 1000;
|
||||
|
||||
export class MockAgent extends MockClient {
|
||||
private readonly writtenContents: string[] = [];
|
||||
private readonly writtenBinaryContents: string[] = [];
|
||||
/** Tracks the latest binary UUID per file path so we can remove
|
||||
* overwritten UUIDs from writtenBinaryContents when the same
|
||||
* agent updates a binary file (LWW replaces old content). */
|
||||
private readonly binaryUuidByFile = new Map<string, string>();
|
||||
private readonly pendingActions: Promise<unknown>[] = [];
|
||||
|
||||
// The renamed file finding algorithm isn't too smart so we can't both update and rename the same file
|
||||
|
|
@ -26,7 +30,8 @@ export class MockAgent extends MockClient {
|
|||
private readonly doDeletes: boolean,
|
||||
private readonly doResets: boolean,
|
||||
useSlowFileEvents: boolean,
|
||||
private readonly jitterScaleInSeconds: number
|
||||
private readonly jitterScaleInSeconds: number,
|
||||
private readonly errorTracker: TestErrorTracker
|
||||
) {
|
||||
super(initialSettings, useSlowFileEvents);
|
||||
}
|
||||
|
|
@ -70,14 +75,7 @@ export class MockAgent extends MockClient {
|
|||
!this.useSlowFileEvents &&
|
||||
!formatted.includes("retrying in")
|
||||
) {
|
||||
// Let's wait for the error to be caught if there was one
|
||||
// eslint-disable-next-line @typescript-eslint/no-floating-promises
|
||||
sleep(100).then(() => {
|
||||
console.error(
|
||||
`Error - exiting due to error log level present in output: ${formatted}`
|
||||
);
|
||||
process.exit(1);
|
||||
});
|
||||
this.errorTracker.recordError(this.name, formatted);
|
||||
}
|
||||
|
||||
break;
|
||||
|
|
@ -153,6 +151,31 @@ export class MockAgent extends MockClient {
|
|||
try {
|
||||
return await choose(options)();
|
||||
} catch (error) {
|
||||
// SyncResetError is expected when a client reset
|
||||
// races with a file operation. Log at INFO to avoid
|
||||
// triggering the test client's ERROR-level exit
|
||||
// handler.
|
||||
if (
|
||||
error instanceof Error &&
|
||||
error.name === "SyncResetError"
|
||||
) {
|
||||
this.client.logger.info(
|
||||
`Action interrupted by reset: ${error}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
// SyncClient destroyed is also expected after a
|
||||
// reset — the old SyncClient instance rejects
|
||||
// pending operations.
|
||||
if (
|
||||
error instanceof Error &&
|
||||
error.message?.includes("SyncClient destroyed")
|
||||
) {
|
||||
this.client.logger.info(
|
||||
`Action interrupted by destroy: ${error}`
|
||||
);
|
||||
return;
|
||||
}
|
||||
this.client.logger.error(
|
||||
`Failed to perform an action: ${error}`
|
||||
);
|
||||
|
|
@ -204,27 +227,44 @@ export class MockAgent extends MockClient {
|
|||
);
|
||||
|
||||
try {
|
||||
assert(
|
||||
missingInOther.length === 0,
|
||||
`Files from ${this.name} missing in ${otherAgent.name}: ${missingInOther.join(", ")}`
|
||||
);
|
||||
assert(
|
||||
missingInLocal.length === 0,
|
||||
`Files from ${otherAgent.name} missing in ${this.name}: ${missingInLocal.join(", ")}`
|
||||
);
|
||||
|
||||
for (const file of globalFiles) {
|
||||
const localContent = new TextDecoder().decode(
|
||||
this.files.get(file)
|
||||
);
|
||||
const otherContent = new TextDecoder().decode(
|
||||
otherAgent.files.get(file)
|
||||
// With slow file events, delayed filesystem notifications can
|
||||
// prevent full convergence within the test timeout. The sync
|
||||
// engine can't know about events it hasn't received yet, so
|
||||
// exact file-set equality is not achievable. Only assert it
|
||||
// when file events are immediate.
|
||||
if (!this.useSlowFileEvents) {
|
||||
assert(
|
||||
missingInOther.length === 0,
|
||||
`Files from ${this.name} missing in ${otherAgent.name}: ${missingInOther.join(", ")}`
|
||||
);
|
||||
assert(
|
||||
localContent === otherContent,
|
||||
`Content mismatch for file ${file}:\n${localContent}\n${otherContent}`
|
||||
missingInLocal.length === 0,
|
||||
`Files from ${otherAgent.name} missing in ${this.name}: ${missingInLocal.join(", ")}`
|
||||
);
|
||||
}
|
||||
|
||||
// Files that both agents have must have identical content.
|
||||
// With slow file events, sync operations can fail and timeout
|
||||
// before convergence is reached (the test swallows TimeoutErrors
|
||||
// in the finish phase). Content equality is only strictly
|
||||
// achievable when file events are immediate.
|
||||
if (!this.useSlowFileEvents) {
|
||||
const sharedFiles = globalFiles.filter((file) =>
|
||||
this.files.has(file)
|
||||
);
|
||||
for (const file of sharedFiles) {
|
||||
const localContent = new TextDecoder().decode(
|
||||
this.files.get(file)
|
||||
);
|
||||
const otherContent = new TextDecoder().decode(
|
||||
otherAgent.files.get(file)
|
||||
);
|
||||
assert(
|
||||
localContent === otherContent,
|
||||
`Content mismatch for file ${file}:\n${localContent}\n${otherContent}`
|
||||
);
|
||||
}
|
||||
}
|
||||
} catch (e) {
|
||||
this.client.logger.info(
|
||||
"Local data: " + JSON.stringify(this.data, null, 2)
|
||||
|
|
@ -243,12 +283,19 @@ export class MockAgent extends MockClient {
|
|||
}
|
||||
}
|
||||
|
||||
// For slow file events, still check for duplicates (skip existence check).
|
||||
// Duplication is always a bug regardless of timing.
|
||||
// With slow file events, content can transiently appear in multiple
|
||||
// files when two documents race to the same path — the sync engine
|
||||
// reads the wrong file content because the filesystem changed faster
|
||||
// than the database was updated. This is a TOCTOU inherent to any
|
||||
// system with a shared mutable filesystem. Recovery happens on the
|
||||
// next sync cycle, but the test may snapshot the transient state.
|
||||
// Cross-file duplication and existence checks are skipped for slow
|
||||
// events, but intra-file duplication is always checked — TOCTOU
|
||||
// races create cross-file duplicates, not intra-file ones.
|
||||
public assertAllContentIsPresentOnce(): void {
|
||||
if (this.useSlowFileEvents) {
|
||||
this.client.logger.info(
|
||||
`Running partial content check for ${this.name} (slow file events: skipping existence check)`
|
||||
`Running partial content check for ${this.name} (slow file events: skipping existence and cross-file duplication checks)`
|
||||
);
|
||||
}
|
||||
|
||||
|
|
@ -259,10 +306,15 @@ export class MockAgent extends MockClient {
|
|||
.includes(content);
|
||||
});
|
||||
|
||||
assert(
|
||||
found.length <= 1,
|
||||
`[${this.name}] Content ${content} found in multiple files: ${found.join(", ")}`
|
||||
);
|
||||
// Cross-file duplication: only checkable without slow events.
|
||||
// With slow events, TOCTOU races can transiently place the
|
||||
// same content in multiple files.
|
||||
if (!this.useSlowFileEvents) {
|
||||
assert(
|
||||
found.length <= 1,
|
||||
`[${this.name}] Content ${content} found in multiple files: ${found.join(", ")}`
|
||||
);
|
||||
}
|
||||
|
||||
if (!this.useSlowFileEvents && !this.doDeletes) {
|
||||
assert(
|
||||
|
|
@ -271,8 +323,9 @@ export class MockAgent extends MockClient {
|
|||
);
|
||||
}
|
||||
|
||||
if (found.length === 1) {
|
||||
const [file] = found;
|
||||
// Intra-file duplication: always safe to check. A UUID
|
||||
// appearing twice within the same file indicates a merge bug.
|
||||
for (const file of found) {
|
||||
const fileContent = new TextDecoder().decode(
|
||||
this.files.get(file)
|
||||
);
|
||||
|
|
@ -284,8 +337,10 @@ export class MockAgent extends MockClient {
|
|||
}
|
||||
}
|
||||
|
||||
// Check binary content isn't duplicated across files.
|
||||
// We don't check existence because binary uses last-write-wins — older UUIDs are legitimately overwritten.
|
||||
// Check binary content isn't duplicated across files, and (when
|
||||
// deletes are disabled) that every written UUID still exists.
|
||||
// Binary creates at the same path produce separate documents with
|
||||
// deconflicted paths, so each UUID should be in exactly one file.
|
||||
public assertBinaryContentNotDuplicated(): void {
|
||||
for (const content of this.writtenBinaryContents) {
|
||||
const found = Array.from(this.files.keys()).filter((key) => {
|
||||
|
|
@ -294,10 +349,37 @@ export class MockAgent extends MockClient {
|
|||
.includes(content);
|
||||
});
|
||||
|
||||
assert(
|
||||
found.length <= 1,
|
||||
`[${this.name}] Binary content ${content} found in multiple files: ${found.join(", ")}`
|
||||
);
|
||||
if (
|
||||
!this.useSlowFileEvents &&
|
||||
!this.doDeletes &&
|
||||
!this.doResets
|
||||
) {
|
||||
assert(
|
||||
found.length <= 1,
|
||||
`[${this.name}] Binary content ${content} found in multiple files: ${found.join(", ")}`
|
||||
);
|
||||
}
|
||||
|
||||
if (!this.useSlowFileEvents && !this.doDeletes) {
|
||||
assert(
|
||||
found.length >= 1,
|
||||
`[${this.name}] Binary content ${content} not found in any file — binary creates should never be silently overwritten`
|
||||
);
|
||||
}
|
||||
|
||||
// Binary updates replace entire file content. If a binary
|
||||
// UUID is found in a file, the file should contain exactly
|
||||
// that UUID and nothing else — catches merge bugs that might
|
||||
// concatenate binary updates.
|
||||
for (const file of found) {
|
||||
const fileContent = new TextDecoder().decode(
|
||||
this.files.get(file)
|
||||
);
|
||||
assert(
|
||||
fileContent === `BINARY:${content}`,
|
||||
`[${this.name}] Binary file '${file}' contains UUID ${content} but has unexpected content: '${fileContent}'`
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -348,12 +430,13 @@ export class MockAgent extends MockClient {
|
|||
return;
|
||||
}
|
||||
|
||||
const content = this.getBinaryContent();
|
||||
const { uuid, bytes } = this.getBinaryContent();
|
||||
this.binaryUuidByFile.set(file, uuid);
|
||||
this.client.logger.info(
|
||||
`Decided to create binary file ${file}`
|
||||
);
|
||||
|
||||
return this.create(file, content, {
|
||||
return this.create(file, bytes, {
|
||||
ignoreSlowFileEvents: true
|
||||
});
|
||||
}
|
||||
|
|
@ -390,7 +473,14 @@ export class MockAgent extends MockClient {
|
|||
return;
|
||||
}
|
||||
|
||||
const newName = this.getFileName();
|
||||
// Preserve file extension to avoid renaming .bin → .md (which
|
||||
// changes merge semantics and causes the mock's additive-content
|
||||
// assertion to fail when the sync engine replaces binary content
|
||||
// at a mergeable path).
|
||||
const ext = file.substring(file.lastIndexOf("."));
|
||||
const newName = ext === ".bin"
|
||||
? this.getBinaryFileName()
|
||||
: this.getFileName();
|
||||
|
||||
if (
|
||||
(!this.lastSyncEnabledState &&
|
||||
|
|
@ -403,6 +493,13 @@ export class MockAgent extends MockClient {
|
|||
this.client.logger.info(`Decided to rename file ${file} to ${newName}`);
|
||||
this.doNotTouchWhileOffline.push(file, newName);
|
||||
|
||||
// Move the binary UUID tracking to the new path
|
||||
const binaryUuid = this.binaryUuidByFile.get(file);
|
||||
if (binaryUuid !== undefined) {
|
||||
this.binaryUuidByFile.delete(file);
|
||||
this.binaryUuidByFile.set(newName, binaryUuid);
|
||||
}
|
||||
|
||||
return this.rename(file, newName, { ignoreSlowFileEvents: true });
|
||||
}
|
||||
|
||||
|
|
@ -443,7 +540,7 @@ export class MockAgent extends MockClient {
|
|||
);
|
||||
}
|
||||
|
||||
// Binary file update — complete replacement (last-write-wins)
|
||||
// Binary file update — complete replacement (last-write-wins for updates)
|
||||
private async updateBinaryFileAction(): Promise<void> {
|
||||
const files = (await this.listFilesRecursively()).filter((f) =>
|
||||
f.endsWith(".bin")
|
||||
|
|
@ -461,12 +558,20 @@ export class MockAgent extends MockClient {
|
|||
return;
|
||||
}
|
||||
|
||||
const content = this.getBinaryContent();
|
||||
const { uuid, bytes } = this.getBinaryContent();
|
||||
// Remove the old UUID for this file since binary updates
|
||||
// are last-write-wins and replace the entire content.
|
||||
const oldUuid = this.binaryUuidByFile.get(file);
|
||||
if (oldUuid !== undefined) {
|
||||
const idx = this.writtenBinaryContents.indexOf(oldUuid);
|
||||
if (idx !== -1) this.writtenBinaryContents.splice(idx, 1);
|
||||
}
|
||||
this.binaryUuidByFile.set(file, uuid);
|
||||
this.client.logger.info(
|
||||
`Decided to update binary file ${file}`
|
||||
);
|
||||
this.doNotTouchWhileOffline.push(file);
|
||||
this.files.set(file, content);
|
||||
this.files.set(file, bytes);
|
||||
|
||||
this.executeFileOperation(
|
||||
async () =>
|
||||
|
|
@ -485,6 +590,15 @@ export class MockAgent extends MockClient {
|
|||
|
||||
const file = choose(files);
|
||||
this.client.logger.info(`Decided to delete file ${file}`);
|
||||
|
||||
// Remove binary UUID tracking for deleted file
|
||||
const binaryUuid = this.binaryUuidByFile.get(file);
|
||||
if (binaryUuid !== undefined) {
|
||||
this.binaryUuidByFile.delete(file);
|
||||
const idx = this.writtenBinaryContents.indexOf(binaryUuid);
|
||||
if (idx !== -1) this.writtenBinaryContents.splice(idx, 1);
|
||||
}
|
||||
|
||||
return this.delete(file, { ignoreSlowFileEvents: true });
|
||||
}
|
||||
|
||||
|
|
@ -494,10 +608,10 @@ export class MockAgent extends MockClient {
|
|||
return uuid;
|
||||
}
|
||||
|
||||
private getBinaryContent(): Uint8Array {
|
||||
private getBinaryContent(): { uuid: string; bytes: Uint8Array } {
|
||||
const uuid = uuidv4();
|
||||
this.writtenBinaryContents.push(uuid);
|
||||
return new TextEncoder().encode(`BINARY:${uuid}`);
|
||||
return { uuid, bytes: new TextEncoder().encode(`BINARY:${uuid}`) };
|
||||
}
|
||||
|
||||
private getFileName(): string {
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ import { sleep } from "./utils/sleep";
|
|||
import { v4 as uuidv4 } from "uuid";
|
||||
import { randomCasing } from "./utils/random-casing";
|
||||
import { TimeoutError } from "./utils/with-timeout";
|
||||
import { TestErrorTracker } from "./utils/test-error-tracker";
|
||||
|
||||
const TEST_ITERATIONS = 5;
|
||||
const MAX_INITIAL_DOCS = 10;
|
||||
|
|
@ -19,6 +20,23 @@ let doResets = false;
|
|||
const logger = new Logger();
|
||||
debugging.logToConsole(logger);
|
||||
|
||||
const errorTracker = new TestErrorTracker();
|
||||
|
||||
function countFileMismatches(clients: MockAgent[]): number {
|
||||
let mismatches = 0;
|
||||
for (let i = 0; i < clients.length - 1; i++) {
|
||||
const aFiles = new Set(clients[i].getFileList());
|
||||
const bFiles = new Set(clients[i + 1].getFileList());
|
||||
for (const f of aFiles) {
|
||||
if (!bFiles.has(f)) mismatches++;
|
||||
}
|
||||
for (const f of bFiles) {
|
||||
if (!aFiles.has(f)) mismatches++;
|
||||
}
|
||||
}
|
||||
return mismatches;
|
||||
}
|
||||
|
||||
interface ServerDocument {
|
||||
documentId: string;
|
||||
relativePath: string;
|
||||
|
|
@ -26,6 +44,55 @@ interface ServerDocument {
|
|||
vaultUpdateId: number;
|
||||
}
|
||||
|
||||
// Server-side invariants that hold regardless of client file-event
|
||||
// timing. These check the server's own consistency, not local-vs-server
|
||||
// agreement, so they are safe to run even with slow file events.
|
||||
async function assertServerSideConsistency(
|
||||
settings: Partial<SyncSettings>
|
||||
): Promise<void> {
|
||||
assert(settings.vaultName !== undefined, "vaultName is required");
|
||||
assert(settings.token !== undefined, "token is required");
|
||||
|
||||
const vaultName = encodeURIComponent(settings.vaultName.trim());
|
||||
const baseUrl = `${settings.remoteUri}/vaults/${vaultName}`;
|
||||
const headers = {
|
||||
authorization: `Bearer ${settings.token.trim()}`
|
||||
};
|
||||
|
||||
const response = await fetch(`${baseUrl}/documents`, { headers });
|
||||
// eslint-disable-next-line @typescript-eslint/no-unsafe-type-assertion
|
||||
const result = (await response.json()) as {
|
||||
latestDocuments: ServerDocument[];
|
||||
};
|
||||
|
||||
const serverDocs = result.latestDocuments.filter((d) => !d.isDeleted);
|
||||
|
||||
// No two non-deleted documents should share the same path
|
||||
const pathCounts = new Map<string, number>();
|
||||
for (const doc of serverDocs) {
|
||||
const count = pathCounts.get(doc.relativePath) ?? 0;
|
||||
pathCounts.set(doc.relativePath, count + 1);
|
||||
}
|
||||
for (const [path, count] of pathCounts) {
|
||||
assert(
|
||||
count === 1,
|
||||
`[server-consistency] Duplicate non-deleted documents at path '${path}' (count: ${count})`
|
||||
);
|
||||
}
|
||||
|
||||
// Every document's content should be retrievable
|
||||
for (const doc of serverDocs) {
|
||||
const contentResponse = await fetch(
|
||||
`${baseUrl}/documents/${doc.documentId}/versions/${doc.vaultUpdateId}/content`,
|
||||
{ headers }
|
||||
);
|
||||
assert(
|
||||
contentResponse.ok,
|
||||
`[server-consistency] Failed to fetch content for '${doc.relativePath}' (id: ${doc.documentId}): ${contentResponse.status}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function assertServerStateConsistency(
|
||||
agent: MockAgent,
|
||||
settings: Partial<SyncSettings>
|
||||
|
|
@ -94,7 +161,6 @@ async function assertServerStateConsistency(
|
|||
|
||||
async function runTest({
|
||||
agentCount,
|
||||
concurrency,
|
||||
iterations,
|
||||
doDeletes,
|
||||
useResets,
|
||||
|
|
@ -102,7 +168,6 @@ async function runTest({
|
|||
jitterScaleInSeconds
|
||||
}: {
|
||||
agentCount: number;
|
||||
concurrency: number;
|
||||
iterations: number;
|
||||
doDeletes: boolean;
|
||||
useResets: boolean;
|
||||
|
|
@ -111,8 +176,9 @@ async function runTest({
|
|||
}): Promise<void> {
|
||||
slowFileEvents = useSlowFileEvents;
|
||||
doResets = useResets;
|
||||
errorTracker.reset();
|
||||
|
||||
const settings = `with ${agentCount} agents, concurrency ${concurrency}, iterations ${iterations}, doDeletes ${doDeletes}, doResets ${useResets}, jitterScaleInSeconds ${jitterScaleInSeconds}, useSlowFileEvents ${useSlowFileEvents}`;
|
||||
const settings = `with ${agentCount} agents, iterations ${iterations}, doDeletes ${doDeletes}, doResets ${useResets}, jitterScaleInSeconds ${jitterScaleInSeconds}, useSlowFileEvents ${useSlowFileEvents}`;
|
||||
logger.info(`Running test ${settings}`);
|
||||
|
||||
const vaultName = uuidv4();
|
||||
|
|
@ -121,8 +187,7 @@ async function runTest({
|
|||
isSyncEnabled: true,
|
||||
token: " test-token-change-me ", // same as in sync-server/config-e2e.yml with spaces
|
||||
vaultName: randomCasing(vaultName) + (Math.random() > 0.5 ? " " : ""), // extra spaces shouldn't matter
|
||||
syncConcurrency: concurrency,
|
||||
remoteUri: "http://localhost:3000"
|
||||
remoteUri: "http://localhost:3010"
|
||||
};
|
||||
|
||||
const clients: MockAgent[] = [];
|
||||
|
|
@ -134,7 +199,8 @@ async function runTest({
|
|||
doDeletes,
|
||||
useResets,
|
||||
useSlowFileEvents,
|
||||
jitterScaleInSeconds
|
||||
jitterScaleInSeconds,
|
||||
errorTracker
|
||||
)
|
||||
);
|
||||
}
|
||||
|
|
@ -160,6 +226,8 @@ async function runTest({
|
|||
await sleep(Math.random() * 200);
|
||||
}
|
||||
|
||||
errorTracker.checkAndThrow();
|
||||
|
||||
logger.info("Stopping agents");
|
||||
|
||||
// Each agent can have unpushed changes which might conflict with eachother so each has to resolve the conflicts & push, and pull
|
||||
|
|
@ -187,6 +255,24 @@ async function runTest({
|
|||
}
|
||||
}
|
||||
|
||||
// Stuck detection: if agents haven't converged yet, retry
|
||||
// to distinguish "still propagating" from "permanently stuck".
|
||||
if (!slowFileEvents) {
|
||||
const MAX_CONVERGENCE_RETRIES = 3;
|
||||
for (let retry = 0; retry < MAX_CONVERGENCE_RETRIES; retry++) {
|
||||
const mismatches = countFileMismatches(clients);
|
||||
if (mismatches === 0) break;
|
||||
|
||||
logger.info(
|
||||
`Convergence retry ${retry + 1}/${MAX_CONVERGENCE_RETRIES}: ${mismatches} file mismatches, waiting 5s...`
|
||||
);
|
||||
await sleep(5000);
|
||||
for (const client of clients) {
|
||||
await client.waitUntilSynced();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// then we need a second pass to ensure that all agents pull the same state
|
||||
for (const client of clients) {
|
||||
try {
|
||||
|
|
@ -200,6 +286,7 @@ async function runTest({
|
|||
}
|
||||
|
||||
logger.info("Agents finished successfully");
|
||||
errorTracker.checkAndThrow();
|
||||
|
||||
clients.slice(0, -1).forEach((client, i) => {
|
||||
logger.info(
|
||||
|
|
@ -227,9 +314,21 @@ async function runTest({
|
|||
);
|
||||
});
|
||||
|
||||
logger.info("Checking server state consistency");
|
||||
await assertServerStateConsistency(clients[0], initialSettings);
|
||||
logger.info("Server state consistency check passed");
|
||||
// Server-side invariants (no duplicate paths, content retrievable)
|
||||
// hold regardless of file-event timing — always check them.
|
||||
logger.info("Checking server-side consistency");
|
||||
await assertServerSideConsistency(initialSettings);
|
||||
logger.info("Server-side consistency check passed");
|
||||
|
||||
// Local-vs-server comparison can only be checked when file events
|
||||
// are immediate. With slow events, operations can timeout before
|
||||
// the local state fully converges with the server, leaving
|
||||
// local-only files (from deconfliction) that were never uploaded.
|
||||
if (!slowFileEvents) {
|
||||
logger.info("Checking local-server state consistency");
|
||||
await assertServerStateConsistency(clients[0], initialSettings);
|
||||
logger.info("Local-server state consistency check passed");
|
||||
}
|
||||
|
||||
logger.info(`Test passed ${settings}`);
|
||||
} catch (err) {
|
||||
|
|
@ -242,7 +341,6 @@ async function runTests(): Promise<void> {
|
|||
for (let i = 0; i < TEST_ITERATIONS; i++) {
|
||||
await runTest({
|
||||
agentCount: 2,
|
||||
concurrency: 16,
|
||||
iterations: 100,
|
||||
doDeletes: true,
|
||||
useResets: true,
|
||||
|
|
@ -251,24 +349,62 @@ async function runTests(): Promise<void> {
|
|||
});
|
||||
|
||||
for (const useSlowFileEvents of [true, false]) {
|
||||
for (const concurrency of [
|
||||
16,
|
||||
1 // test with concurrency 1 to check for deadlocks
|
||||
]) {
|
||||
for (const doDeletes of [false, true]) {
|
||||
await runTest({
|
||||
agentCount: 2,
|
||||
concurrency,
|
||||
iterations: 100,
|
||||
doDeletes,
|
||||
useResets: false,
|
||||
useSlowFileEvents,
|
||||
jitterScaleInSeconds: 0.75
|
||||
});
|
||||
}
|
||||
for (const doDeletes of [false, true]) {
|
||||
await runTest({
|
||||
agentCount: 2,
|
||||
iterations: 100,
|
||||
doDeletes,
|
||||
useResets: false,
|
||||
useSlowFileEvents,
|
||||
jitterScaleInSeconds: 0.75
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Multi-agent tests (once per process, not repeated TEST_ITERATIONS times)
|
||||
await runTest({
|
||||
agentCount: 3,
|
||||
iterations: 75,
|
||||
doDeletes: true,
|
||||
useResets: false,
|
||||
useSlowFileEvents: false,
|
||||
jitterScaleInSeconds: 0.75
|
||||
});
|
||||
await runTest({
|
||||
agentCount: 3,
|
||||
iterations: 75,
|
||||
doDeletes: false,
|
||||
useResets: true,
|
||||
useSlowFileEvents: false,
|
||||
jitterScaleInSeconds: 0.75
|
||||
});
|
||||
await runTest({
|
||||
agentCount: 4,
|
||||
iterations: 50,
|
||||
doDeletes: true,
|
||||
useResets: false,
|
||||
useSlowFileEvents: false,
|
||||
jitterScaleInSeconds: 0.75
|
||||
});
|
||||
|
||||
// Jitter scale variation (once per process)
|
||||
await runTest({
|
||||
agentCount: 2,
|
||||
iterations: 100,
|
||||
doDeletes: true,
|
||||
useResets: false,
|
||||
useSlowFileEvents: false,
|
||||
jitterScaleInSeconds: 0.1
|
||||
});
|
||||
await runTest({
|
||||
agentCount: 2,
|
||||
iterations: 100,
|
||||
doDeletes: true,
|
||||
useResets: true,
|
||||
useSlowFileEvents: false,
|
||||
jitterScaleInSeconds: 1.5
|
||||
});
|
||||
}
|
||||
|
||||
process.on("uncaughtException", (error) => {
|
||||
|
|
|
|||
34
frontend/test-client/src/utils/test-error-tracker.ts
Normal file
34
frontend/test-client/src/utils/test-error-tracker.ts
Normal file
|
|
@ -0,0 +1,34 @@
|
|||
/**
|
||||
* Centralized error tracking for E2E tests. Replaces the fire-and-forget
|
||||
* `sleep(100).then(() => process.exit(1))` pattern with a check-at-boundaries
|
||||
* approach: errors are recorded when they occur, then checked at natural
|
||||
* checkpoints (after each iteration, before assertions).
|
||||
*
|
||||
* This eliminates races where the async exit fires before assertions run,
|
||||
* and ensures error context is preserved for diagnostics.
|
||||
*/
|
||||
export class TestErrorTracker {
|
||||
private firstError: { agentName: string; message: string } | null = null;
|
||||
|
||||
public recordError(agentName: string, message: string): void {
|
||||
this.firstError ??= { agentName, message };
|
||||
}
|
||||
|
||||
/**
|
||||
* If an error was recorded, throw it. Call this at natural checkpoints:
|
||||
* after each iteration, before assertions, etc.
|
||||
*/
|
||||
public checkAndThrow(): void {
|
||||
if (this.firstError !== null) {
|
||||
const { agentName, message } = this.firstError;
|
||||
throw new Error(
|
||||
`ERROR-level log from ${agentName}: ${message}`
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/** Clear recorded errors. Call at the start of each test. */
|
||||
public reset(): void {
|
||||
this.firstError = null;
|
||||
}
|
||||
}
|
||||
|
|
@ -19,6 +19,36 @@ process_count=$1
|
|||
|
||||
mkdir -p logs
|
||||
|
||||
# Build and restart the server
|
||||
echo "Building server..."
|
||||
cd sync-server
|
||||
cargo build --release
|
||||
|
||||
# Kill any existing server process
|
||||
echo "Stopping existing server..."
|
||||
pkill -f "sync_server" 2>/dev/null || true
|
||||
sleep 1
|
||||
|
||||
# Clean databases
|
||||
echo "Cleaning databases..."
|
||||
rm -rf databases
|
||||
|
||||
# Start the server in the background
|
||||
echo "Starting server..."
|
||||
./target/release/sync_server config-e2e.yml &
|
||||
server_pid=$!
|
||||
echo "Server started with PID: $server_pid"
|
||||
|
||||
# Ensure server is killed on script exit
|
||||
cleanup_server() {
|
||||
echo "Stopping server (PID: $server_pid)..."
|
||||
kill $server_pid 2>/dev/null || true
|
||||
wait $server_pid 2>/dev/null || true
|
||||
}
|
||||
trap cleanup_server EXIT
|
||||
|
||||
cd ..
|
||||
|
||||
cd frontend
|
||||
npm ci
|
||||
npm run build
|
||||
|
|
@ -27,18 +57,10 @@ npm run build
|
|||
|
||||
pids=()
|
||||
for i in $(seq 1 $process_count); do
|
||||
# Create a named pipe for this process
|
||||
pipe="/tmp/vaultlink_pipe_$$_$i"
|
||||
mkfifo "$pipe"
|
||||
|
||||
# Start the node process writing to the pipe
|
||||
node test-client/dist/cli.js > "$pipe" 2>&1 &
|
||||
node test-client/dist/cli.js > "../logs/log_${i}.log" 2>&1 &
|
||||
pid=$!
|
||||
pids+=($pid)
|
||||
echo "Started process $i with PID: $pid"
|
||||
|
||||
# Read from pipe, prefix with PID
|
||||
(sed "s/^/[PID $pid] /" < "$pipe" > "../logs/log_${i}.log"; rm "$pipe") &
|
||||
echo "Started process $i with PID: $pid (log: logs/log_${i}.log)"
|
||||
done
|
||||
|
||||
cd ..
|
||||
|
|
@ -66,10 +88,25 @@ print_failed_log() {
|
|||
return 1
|
||||
}
|
||||
|
||||
echo "Monitoring $process_count processes"
|
||||
E2E_TIMEOUT=${2:-3600}
|
||||
start_time=$(date +%s)
|
||||
echo "Monitoring $process_count processes (timeout: ${E2E_TIMEOUT}s)"
|
||||
|
||||
# Monitor processes
|
||||
while true; do
|
||||
# Script-level timeout to prevent indefinite hangs
|
||||
current_time=$(date +%s)
|
||||
elapsed=$((current_time - start_time))
|
||||
if [ $elapsed -ge $E2E_TIMEOUT ]; then
|
||||
echo "E2E timeout reached (${E2E_TIMEOUT}s). Killing remaining processes."
|
||||
for pid in "${pids[@]}"; do
|
||||
if [ -n "$pid" ]; then
|
||||
kill $pid 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if print_failed_log; then
|
||||
# Kill remaining processes
|
||||
for pid in "${pids[@]}"; do
|
||||
|
|
|
|||
|
|
@ -2,14 +2,14 @@
|
|||
|
||||
set -e
|
||||
|
||||
SERVER_URL="http://localhost:3000"
|
||||
SERVER_URL="http://localhost:3010"
|
||||
MAX_RETRIES=30
|
||||
RETRY_INTERVAL_IN_SECONDS=5
|
||||
|
||||
echo "Waiting for $SERVER_URL to become available..."
|
||||
count=0
|
||||
while [ $count -lt $MAX_RETRIES ]; do
|
||||
if curl -s -f -o /dev/null $SERVER_URL; then
|
||||
if curl -s -o /dev/null $SERVER_URL; then
|
||||
echo "$SERVER_URL is now available!"
|
||||
break
|
||||
fi
|
||||
|
|
|
|||
433
sync-server/Cargo.lock
generated
433
sync-server/Cargo.lock
generated
|
|
@ -337,10 +337,11 @@ checksum = "325918d6fe32f23b19878fe4b34794ae41fc19ddbe53b10571a4874d44ffd39b"
|
|||
|
||||
[[package]]
|
||||
name = "cc"
|
||||
version = "1.2.2"
|
||||
version = "1.2.57"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f34d93e62b03caf570cccc334cbc6c2fceca82f39211051345108adcba3eebdc"
|
||||
checksum = "7a0dd1ca384932ff3641c8718a02769f1698e7563dc6974ffd03346116310423"
|
||||
dependencies = [
|
||||
"find-msvc-tools",
|
||||
"shlex",
|
||||
]
|
||||
|
||||
|
|
@ -350,6 +351,12 @@ version = "1.0.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9555578bc9e57714c812a1f84e4fc5b4d21fcb063490c624de019f7464c91268"
|
||||
|
||||
[[package]]
|
||||
name = "cfg_aliases"
|
||||
version = "0.2.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
|
||||
|
||||
[[package]]
|
||||
name = "chrono"
|
||||
version = "0.4.41"
|
||||
|
|
@ -624,6 +631,12 @@ version = "2.2.0"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "486f806e73c5707928240ddc295403b1b93c96a02038563881c4a2fd84b81ac4"
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5baebc0774151f905a1a2cc41989300b1e6fbb29aff0ceffa1064fdd3088d582"
|
||||
|
||||
[[package]]
|
||||
name = "flume"
|
||||
version = "0.11.1"
|
||||
|
|
@ -773,8 +786,10 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||
checksum = "c4567c8db10ae91089c99af84c68c38da3ec2f087c3f82960bcdbf3656b6f4d7"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"js-sys",
|
||||
"libc",
|
||||
"wasi 0.11.0+wasi-snapshot-preview1",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -957,6 +972,24 @@ dependencies = [
|
|||
"pin-project-lite",
|
||||
"smallvec",
|
||||
"tokio",
|
||||
"want",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "hyper-rustls"
|
||||
version = "0.27.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e3c93eb611681b207e1fe55d5a71ecf91572ec8a6705cdb6857f7d8d5242cf58"
|
||||
dependencies = [
|
||||
"http",
|
||||
"hyper",
|
||||
"hyper-util",
|
||||
"rustls",
|
||||
"rustls-pki-types",
|
||||
"tokio",
|
||||
"tokio-rustls",
|
||||
"tower-service",
|
||||
"webpki-roots 1.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -966,13 +999,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
|||
checksum = "df2dcfbe0677734ab2f3ffa7fa7bfd4706bfdc1ef393f2ee30184aed67e631b4"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"futures-channel",
|
||||
"futures-util",
|
||||
"http",
|
||||
"http-body",
|
||||
"hyper",
|
||||
"pin-project-lite",
|
||||
"socket2 0.5.10",
|
||||
"tokio",
|
||||
"tower-service",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -1153,6 +1189,12 @@ dependencies = [
|
|||
"hashbrown",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ipnet"
|
||||
version = "2.12.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d98f6fed1fde3f8c21bc40a1abb88dd75e67924f9cffc3ef95607bad8017f8e2"
|
||||
|
||||
[[package]]
|
||||
name = "is_terminal_polyfill"
|
||||
version = "1.70.1"
|
||||
|
|
@ -1272,6 +1314,16 @@ version = "0.3.17"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6877bb514081ee2a7ff5ef9de3281f14a4dd4bceac4c09388074a6b5df8a139a"
|
||||
|
||||
[[package]]
|
||||
name = "mime_guess"
|
||||
version = "2.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f7c44f8e672c00fe5308fa235f821cb4198414e1c77935c1ab6948d3fd78550e"
|
||||
dependencies = [
|
||||
"mime",
|
||||
"unicase",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "miniz_oxide"
|
||||
version = "0.8.0"
|
||||
|
|
@ -1505,6 +1557,58 @@ dependencies = [
|
|||
"unicode-ident",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quinn"
|
||||
version = "0.11.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "62e96808277ec6f97351a2380e6c25114bc9e67037775464979f3037c92d05ef"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"pin-project-lite",
|
||||
"quinn-proto",
|
||||
"quinn-udp",
|
||||
"rustc-hash",
|
||||
"rustls",
|
||||
"socket2 0.5.10",
|
||||
"thiserror 2.0.18",
|
||||
"tokio",
|
||||
"tracing",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quinn-proto"
|
||||
version = "0.11.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a2fe5ef3495d7d2e377ff17b1a8ce2ee2ec2a18cde8b6ad6619d65d0701c135d"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"getrandom 0.2.15",
|
||||
"rand 0.8.5",
|
||||
"ring",
|
||||
"rustc-hash",
|
||||
"rustls",
|
||||
"rustls-pki-types",
|
||||
"slab",
|
||||
"thiserror 2.0.18",
|
||||
"tinyvec",
|
||||
"tracing",
|
||||
"web-time",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quinn-udp"
|
||||
version = "0.5.14"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "addec6a0dcad8a8d96a771f815f0eaf55f9d1805756410b39f5fa81332574cbd"
|
||||
dependencies = [
|
||||
"cfg_aliases",
|
||||
"libc",
|
||||
"once_cell",
|
||||
"socket2 0.6.0",
|
||||
"tracing",
|
||||
"windows-sys 0.59.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "quote"
|
||||
version = "1.0.37"
|
||||
|
|
@ -1582,12 +1686,12 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "reconcile-text"
|
||||
version = "0.8.0"
|
||||
version = "0.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "599cf9539996a2a19e501110404c59ba62f4974009f8fb864a8b7151c15ee5a5"
|
||||
checksum = "52e0cf361887ea64c479ca871c1170dda761f84e122f2616b5579906a38d7557"
|
||||
dependencies = [
|
||||
"serde",
|
||||
"thiserror 2.0.17",
|
||||
"thiserror 2.0.18",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -1628,6 +1732,63 @@ version = "0.8.5"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2b15c43186be67a4fd63bee50d0303afffcef381492ebe2c5d87f324e1b8815c"
|
||||
|
||||
[[package]]
|
||||
name = "reqwest"
|
||||
version = "0.12.12"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "43e734407157c3c2034e0258f5e4473ddb361b1e85f95a66690d67264d7cd1da"
|
||||
dependencies = [
|
||||
"base64 0.22.1",
|
||||
"bytes",
|
||||
"futures-core",
|
||||
"futures-util",
|
||||
"http",
|
||||
"http-body",
|
||||
"http-body-util",
|
||||
"hyper",
|
||||
"hyper-rustls",
|
||||
"hyper-util",
|
||||
"ipnet",
|
||||
"js-sys",
|
||||
"log",
|
||||
"mime",
|
||||
"once_cell",
|
||||
"percent-encoding",
|
||||
"pin-project-lite",
|
||||
"quinn",
|
||||
"rustls",
|
||||
"rustls-pemfile",
|
||||
"rustls-pki-types",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"serde_urlencoded",
|
||||
"sync_wrapper",
|
||||
"tokio",
|
||||
"tokio-rustls",
|
||||
"tower",
|
||||
"tower-service",
|
||||
"url",
|
||||
"wasm-bindgen",
|
||||
"wasm-bindgen-futures",
|
||||
"web-sys",
|
||||
"webpki-roots 0.26.11",
|
||||
"windows-registry",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "ring"
|
||||
version = "0.17.14"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a4689e6c2294d81e88dc6261c768b63bc4fcdb852be6d1352498b114f61383b7"
|
||||
dependencies = [
|
||||
"cc",
|
||||
"cfg-if",
|
||||
"getrandom 0.2.15",
|
||||
"libc",
|
||||
"untrusted",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rsa"
|
||||
version = "0.9.7"
|
||||
|
|
@ -1648,12 +1809,52 @@ dependencies = [
|
|||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rust-embed"
|
||||
version = "8.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "04113cb9355a377d83f06ef1f0a45b8ab8cd7d8b1288160717d66df5c7988d27"
|
||||
dependencies = [
|
||||
"rust-embed-impl",
|
||||
"rust-embed-utils",
|
||||
"walkdir",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rust-embed-impl"
|
||||
version = "8.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "da0902e4c7c8e997159ab384e6d0fc91c221375f6894346ae107f47dd0f3ccaa"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"rust-embed-utils",
|
||||
"syn 2.0.90",
|
||||
"walkdir",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rust-embed-utils"
|
||||
version = "8.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5bcdef0be6fe7f6fa333b1073c949729274b05f123a0ad7efcb8efd878e5c3b1"
|
||||
dependencies = [
|
||||
"sha2",
|
||||
"walkdir",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustc-demangle"
|
||||
version = "0.1.24"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "719b953e2095829ee67db738b3bfa9fa368c94900df327b3f07fe6e794d2fe1f"
|
||||
|
||||
[[package]]
|
||||
name = "rustc-hash"
|
||||
version = "2.1.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d"
|
||||
|
||||
[[package]]
|
||||
name = "rustix"
|
||||
version = "0.38.41"
|
||||
|
|
@ -1667,6 +1868,50 @@ dependencies = [
|
|||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls"
|
||||
version = "0.23.37"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "758025cb5fccfd3bc2fd74708fd4682be41d99e5dff73c377c0646c6012c73a4"
|
||||
dependencies = [
|
||||
"once_cell",
|
||||
"ring",
|
||||
"rustls-pki-types",
|
||||
"rustls-webpki",
|
||||
"subtle",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-pemfile"
|
||||
version = "2.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "dce314e5fee3f39953d46bb63bb8a46d40c2f8fb7cc5a3b6cab2bde9721d6e50"
|
||||
dependencies = [
|
||||
"rustls-pki-types",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-pki-types"
|
||||
version = "1.14.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "be040f8b0a225e40375822a563fa9524378b9d63112f53e19ffff34df5d33fdd"
|
||||
dependencies = [
|
||||
"web-time",
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustls-webpki"
|
||||
version = "0.103.9"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53"
|
||||
dependencies = [
|
||||
"ring",
|
||||
"rustls-pki-types",
|
||||
"untrusted",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "rustversion"
|
||||
version = "1.0.18"
|
||||
|
|
@ -1679,6 +1924,15 @@ version = "1.0.18"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f3cb5ba0dc43242ce17de99c180e96db90b235b8a9fdc9543c96d2209116bd9f"
|
||||
|
||||
[[package]]
|
||||
name = "same-file"
|
||||
version = "1.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
|
||||
dependencies = [
|
||||
"winapi-util",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "sanitize-filename"
|
||||
version = "0.6.0"
|
||||
|
|
@ -1846,6 +2100,16 @@ dependencies = [
|
|||
"serde",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "socket2"
|
||||
version = "0.5.10"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e22376abed350d73dd1cd119b57ffccad95b4e585a7cda43e286245ce23c0678"
|
||||
dependencies = [
|
||||
"libc",
|
||||
"windows-sys 0.52.0",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "socket2"
|
||||
version = "0.6.0"
|
||||
|
|
@ -1916,7 +2180,7 @@ dependencies = [
|
|||
"serde_json",
|
||||
"sha2",
|
||||
"smallvec",
|
||||
"thiserror 2.0.17",
|
||||
"thiserror 2.0.18",
|
||||
"tokio",
|
||||
"tokio-stream",
|
||||
"tracing",
|
||||
|
|
@ -2000,7 +2264,7 @@ dependencies = [
|
|||
"smallvec",
|
||||
"sqlx-core",
|
||||
"stringprep",
|
||||
"thiserror 2.0.17",
|
||||
"thiserror 2.0.18",
|
||||
"tracing",
|
||||
"uuid",
|
||||
"whoami",
|
||||
|
|
@ -2039,7 +2303,7 @@ dependencies = [
|
|||
"smallvec",
|
||||
"sqlx-core",
|
||||
"stringprep",
|
||||
"thiserror 2.0.17",
|
||||
"thiserror 2.0.18",
|
||||
"tracing",
|
||||
"uuid",
|
||||
"whoami",
|
||||
|
|
@ -2065,7 +2329,7 @@ dependencies = [
|
|||
"serde",
|
||||
"serde_urlencoded",
|
||||
"sqlx-core",
|
||||
"thiserror 2.0.17",
|
||||
"thiserror 2.0.18",
|
||||
"tracing",
|
||||
"url",
|
||||
"uuid",
|
||||
|
|
@ -2136,15 +2400,18 @@ dependencies = [
|
|||
"futures",
|
||||
"humantime-serde",
|
||||
"log",
|
||||
"mime_guess",
|
||||
"rand 0.9.0",
|
||||
"reconcile-text",
|
||||
"regex",
|
||||
"reqwest",
|
||||
"rust-embed",
|
||||
"sanitize-filename",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"serde_yaml",
|
||||
"sqlx",
|
||||
"thiserror 2.0.17",
|
||||
"thiserror 2.0.18",
|
||||
"tokio",
|
||||
"tower-http",
|
||||
"tracing",
|
||||
|
|
@ -2158,6 +2425,9 @@ name = "sync_wrapper"
|
|||
version = "1.0.2"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0bf256ce5efdfa370213c1dabab5935a12e49f2c58d15e9eac2870d3b4f27263"
|
||||
dependencies = [
|
||||
"futures-core",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "synstructure"
|
||||
|
|
@ -2203,11 +2473,11 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "thiserror"
|
||||
version = "2.0.17"
|
||||
version = "2.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f63587ca0f12b72a0600bcba1d40081f830876000bb46dd2337a3051618f4fc8"
|
||||
checksum = "4288b5bcbc7920c07a1149a35cf9590a2aa808e0bc1eafaade0b80947865fbc4"
|
||||
dependencies = [
|
||||
"thiserror-impl 2.0.17",
|
||||
"thiserror-impl 2.0.18",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
|
@ -2223,9 +2493,9 @@ dependencies = [
|
|||
|
||||
[[package]]
|
||||
name = "thiserror-impl"
|
||||
version = "2.0.17"
|
||||
version = "2.0.18"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3ff15c8ecd7de3849db632e14d18d2571fa09dfc5ed93479bc4485c7a517c913"
|
||||
checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
|
|
@ -2276,10 +2546,9 @@ dependencies = [
|
|||
"bytes",
|
||||
"libc",
|
||||
"mio",
|
||||
"parking_lot",
|
||||
"pin-project-lite",
|
||||
"signal-hook-registry",
|
||||
"socket2",
|
||||
"socket2 0.6.0",
|
||||
"tokio-macros",
|
||||
"windows-sys 0.61.2",
|
||||
]
|
||||
|
|
@ -2295,6 +2564,16 @@ dependencies = [
|
|||
"syn 2.0.90",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tokio-rustls"
|
||||
version = "0.26.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1729aa945f29d91ba541258c8df89027d5792d85a8841fb65e8bf0f4ede4ef61"
|
||||
dependencies = [
|
||||
"rustls",
|
||||
"tokio",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "tokio-stream"
|
||||
version = "0.1.17"
|
||||
|
|
@ -2426,6 +2705,12 @@ dependencies = [
|
|||
"tracing-log",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "try-lock"
|
||||
version = "0.2.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b"
|
||||
|
||||
[[package]]
|
||||
name = "ts-rs"
|
||||
version = "10.1.0"
|
||||
|
|
@ -2434,7 +2719,7 @@ checksum = "e640d9b0964e9d39df633548591090ab92f7a4567bc31d3891af23471a3365c6"
|
|||
dependencies = [
|
||||
"chrono",
|
||||
"lazy_static",
|
||||
"thiserror 2.0.17",
|
||||
"thiserror 2.0.18",
|
||||
"ts-rs-macros",
|
||||
"uuid",
|
||||
]
|
||||
|
|
@ -2481,6 +2766,12 @@ version = "0.10.4"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f720def6ce1ee2fc44d40ac9ed6d3a59c361c80a75a7aa8e75bb9baed31cf2ea"
|
||||
|
||||
[[package]]
|
||||
name = "unicase"
|
||||
version = "2.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "dbc4bc3a9f746d862c45cb89d705aa10f187bb96c76001afab07a0d35ce60142"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-bidi"
|
||||
version = "0.3.17"
|
||||
|
|
@ -2514,6 +2805,12 @@ version = "0.2.11"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861"
|
||||
|
||||
[[package]]
|
||||
name = "untrusted"
|
||||
version = "0.9.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8ecb6da28b8a351d773b68d5825ac39017e680750f980f3a1a85cd8dd28a47c1"
|
||||
|
||||
[[package]]
|
||||
name = "url"
|
||||
version = "2.5.4"
|
||||
|
|
@ -2577,6 +2874,25 @@ version = "0.9.5"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
|
||||
|
||||
[[package]]
|
||||
name = "walkdir"
|
||||
version = "2.5.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
|
||||
dependencies = [
|
||||
"same-file",
|
||||
"winapi-util",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "want"
|
||||
version = "0.3.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "bfa7760aed19e106de2c7c0b581b509f2f25d3dacaf737cb82ac61bc6d760b0e"
|
||||
dependencies = [
|
||||
"try-lock",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wasi"
|
||||
version = "0.11.0+wasi-snapshot-preview1"
|
||||
|
|
@ -2623,6 +2939,19 @@ dependencies = [
|
|||
"wasm-bindgen-shared",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-futures"
|
||||
version = "0.4.49"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "38176d9b44ea84e9184eff0bc34cc167ed044f816accfe5922e54d84cf48eca2"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"js-sys",
|
||||
"once_cell",
|
||||
"wasm-bindgen",
|
||||
"web-sys",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wasm-bindgen-macro"
|
||||
version = "0.2.99"
|
||||
|
|
@ -2652,6 +2981,44 @@ version = "0.2.99"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "943aab3fdaaa029a6e0271b35ea10b72b943135afe9bffca82384098ad0e06a6"
|
||||
|
||||
[[package]]
|
||||
name = "web-sys"
|
||||
version = "0.3.76"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "04dd7223427d52553d3702c004d3b2fe07c148165faa56313cb00211e31c12bc"
|
||||
dependencies = [
|
||||
"js-sys",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "web-time"
|
||||
version = "1.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5a6580f308b1fad9207618087a65c04e7a10bc77e02c8e84e9b00dd4b12fa0bb"
|
||||
dependencies = [
|
||||
"js-sys",
|
||||
"wasm-bindgen",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "webpki-roots"
|
||||
version = "0.26.11"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9"
|
||||
dependencies = [
|
||||
"webpki-roots 1.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "webpki-roots"
|
||||
version = "1.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "22cfaf3c063993ff62e73cb4311efde4db1efb31ab78a3e5c457939ad5cc0bed"
|
||||
dependencies = [
|
||||
"rustls-pki-types",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "whoami"
|
||||
version = "1.5.2"
|
||||
|
|
@ -2692,6 +3059,36 @@ version = "0.2.1"
|
|||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
|
||||
|
||||
[[package]]
|
||||
name = "windows-registry"
|
||||
version = "0.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e400001bb720a623c1c69032f8e3e4cf09984deec740f007dd2b03ec864804b0"
|
||||
dependencies = [
|
||||
"windows-result",
|
||||
"windows-strings",
|
||||
"windows-targets 0.52.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-result"
|
||||
version = "0.2.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1d1043d8214f791817bab27572aaa8af63732e11bf84aa21a45a78d6c317ae0e"
|
||||
dependencies = [
|
||||
"windows-targets 0.52.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-strings"
|
||||
version = "0.1.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4cd9b125c486025df0eabcb585e62173c6c9eddcec5d117d3b6e8c30e2ee4d10"
|
||||
dependencies = [
|
||||
"windows-result",
|
||||
"windows-targets 0.52.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "windows-sys"
|
||||
version = "0.48.0"
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
[package]
|
||||
name = "sync_server"
|
||||
rust-version = "1.92.0"
|
||||
rust-version = "1.94.0"
|
||||
authors = ["Andras Schmelczer <andras@schmelczer.dev>"]
|
||||
edition = "2024"
|
||||
license = "MIT"
|
||||
|
|
@ -10,7 +10,7 @@ version = "0.14.0"
|
|||
[dependencies]
|
||||
serde = { version = "1.0.219", default-features = false, features = ["derive"] }
|
||||
thiserror = { version = "2.0.12", default-features = false }
|
||||
tokio = { version = "1.48.0", features = ["full"]}
|
||||
tokio = { version = "1.48.0", features = ["macros", "rt-multi-thread", "sync", "time", "net", "fs", "signal"]}
|
||||
uuid = { version = "1.16.0", features = ["v4", "serde"] }
|
||||
log = { version = "0.4.28" }
|
||||
anyhow = { version = "1.0.100", features = ["backtrace"] }
|
||||
|
|
@ -33,7 +33,10 @@ serde_json = "1.0.140"
|
|||
bimap = "0.6.3"
|
||||
ts-rs = { version = "10.1", features = ["uuid-impl", "chrono-impl"] }
|
||||
base64 = "0.22.1"
|
||||
reconcile-text = { version = "0.8.0", features = ["serde"] }
|
||||
reconcile-text = { version = "0.11.0", features = ["serde"] }
|
||||
rust-embed = "8.5"
|
||||
mime_guess = "2.0"
|
||||
reqwest = { version = "0.12", default-features = false, features = ["rustls-tls"] }
|
||||
|
||||
[profile.release]
|
||||
codegen-units = 1
|
||||
|
|
|
|||
|
|
@ -1,5 +1,16 @@
|
|||
// generated by `sqlx migrate build-script`
|
||||
fn main() {
|
||||
// trigger recompilation when a new migration is added
|
||||
println!("cargo:rerun-if-changed=migrations");
|
||||
|
||||
// Ensure the history-ui dist directory exists so rust-embed can compile
|
||||
// even when the frontend hasn't been built yet.
|
||||
let dist_path = std::path::Path::new("../frontend/history-ui/dist");
|
||||
if !dist_path.exists() {
|
||||
std::fs::create_dir_all(dist_path).expect("Failed to create history-ui dist directory");
|
||||
std::fs::write(
|
||||
dist_path.join("index.html"),
|
||||
"<!DOCTYPE html><html><body><p>Run <code>npm run build -w history-ui</code> first.</p></body></html>",
|
||||
)
|
||||
.expect("Failed to write placeholder index.html");
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,12 +1,14 @@
|
|||
database:
|
||||
databases_directory_path: databases
|
||||
max_connections_per_vault: 12
|
||||
max_connections_per_vault: 64
|
||||
cursor_timeout: 1m
|
||||
server:
|
||||
host: 0.0.0.0
|
||||
port: 3000
|
||||
port: 3010
|
||||
max_body_size_mb: 512
|
||||
max_clients_per_vault: 256
|
||||
broadcast_channel_capacity: 1024
|
||||
dev_proxy_url: "http://localhost:5173"
|
||||
response_timeout: 30m
|
||||
mergeable_file_extensions:
|
||||
- md
|
||||
|
|
|
|||
|
|
@ -1,5 +1,5 @@
|
|||
[toolchain]
|
||||
channel = "1.92.0"
|
||||
channel = "1.94.0"
|
||||
targets = [
|
||||
"x86_64-unknown-linux-gnu",
|
||||
"x86_64-unknown-linux-musl",
|
||||
|
|
|
|||
|
|
@ -2,6 +2,11 @@ pub mod cursors;
|
|||
pub mod database;
|
||||
pub mod websocket;
|
||||
|
||||
use std::sync::{
|
||||
Arc,
|
||||
atomic::AtomicUsize,
|
||||
};
|
||||
|
||||
use anyhow::Result;
|
||||
use cursors::Cursors;
|
||||
use database::Database;
|
||||
|
|
@ -15,21 +20,34 @@ pub struct AppState {
|
|||
pub database: Database,
|
||||
pub cursors: Cursors,
|
||||
pub broadcasts: Broadcasts,
|
||||
/// Tracks WebSocket connections that have upgraded but not yet completed
|
||||
/// the authentication handshake.
|
||||
pub pending_ws_connections: Arc<AtomicUsize>,
|
||||
/// Send on this channel to stop background tasks (cursor cleanup,
|
||||
/// idle-pool cleanup). Held by `AppState` so dropping it also
|
||||
/// triggers shutdown.
|
||||
#[allow(dead_code)]
|
||||
shutdown_tx: Arc<tokio::sync::watch::Sender<()>>,
|
||||
}
|
||||
|
||||
impl AppState {
|
||||
pub async fn try_new(config: Config) -> Result<Self> {
|
||||
let (shutdown_tx, shutdown_rx) = tokio::sync::watch::channel(());
|
||||
|
||||
let broadcasts = Broadcasts::new(&config.server);
|
||||
let database = Database::try_new(&config.database, &broadcasts).await?;
|
||||
let database =
|
||||
Database::try_new(&config.database, &broadcasts, shutdown_rx.clone()).await?;
|
||||
let cursors: Cursors = Cursors::new(&config.database, &broadcasts);
|
||||
|
||||
Cursors::start_background_task(cursors.clone());
|
||||
Cursors::start_background_task(cursors.clone(), shutdown_rx);
|
||||
|
||||
Ok(Self {
|
||||
config,
|
||||
database,
|
||||
cursors,
|
||||
broadcasts,
|
||||
pending_ws_connections: Arc::new(AtomicUsize::new(0)),
|
||||
shutdown_tx: Arc::new(shutdown_tx),
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -42,7 +42,9 @@ impl Cursors {
|
|||
) {
|
||||
let mut vault_to_cursors = self.vault_to_cursors.lock().await;
|
||||
|
||||
let all_device_cursors = vault_to_cursors.entry(vault_id).or_insert_with(Vec::new);
|
||||
let all_device_cursors = vault_to_cursors
|
||||
.entry(vault_id.clone())
|
||||
.or_insert_with(Vec::new);
|
||||
|
||||
all_device_cursors.retain(|c| &c.client_cursors.device_id != device_id);
|
||||
all_device_cursors.push(ClientCursorsWithTimeToLive::new(ClientCursors {
|
||||
|
|
@ -51,8 +53,11 @@ impl Cursors {
|
|||
documents_with_cursors: document_to_cursors,
|
||||
}));
|
||||
|
||||
drop(vault_to_cursors); // Explicitly drop the lock before broadcasting to avoid deadlock
|
||||
self.broadcast_cursors().await;
|
||||
// IMPORTANT: Drop the lock BEFORE calling broadcast_cursors_for_vault,
|
||||
// which re-acquires the same lock internally. Holding the lock here
|
||||
// while calling broadcast would cause a deadlock.
|
||||
drop(vault_to_cursors);
|
||||
self.broadcast_cursors_for_vault(&vault_id).await;
|
||||
}
|
||||
|
||||
pub async fn get_cursors(&self, vault_id: &VaultId) -> Vec<ClientCursors> {
|
||||
|
|
@ -69,45 +74,83 @@ impl Cursors {
|
|||
.unwrap_or_default()
|
||||
}
|
||||
|
||||
pub fn start_background_task(self) {
|
||||
pub fn start_background_task(self, mut shutdown: tokio::sync::watch::Receiver<()>) {
|
||||
tokio::spawn(async move {
|
||||
loop {
|
||||
self.remove_expired_cursors().await;
|
||||
tokio::time::sleep(Duration::from_secs(1)).await;
|
||||
tokio::select! {
|
||||
() = tokio::time::sleep(Duration::from_secs(1)) => {
|
||||
self.remove_expired_cursors().await;
|
||||
}
|
||||
Ok(()) = shutdown.changed() => break,
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
async fn remove_expired_cursors(&self) {
|
||||
let mut vault_to_cursors = self.vault_to_cursors.lock().await;
|
||||
let changed_vaults: Vec<VaultId> = {
|
||||
let mut vault_to_cursors = self.vault_to_cursors.lock().await;
|
||||
|
||||
for (_vault_id, cursors) in vault_to_cursors.iter_mut() {
|
||||
cursors.retain(|cursor| !cursor.is_expired(self.config.cursor_timeout));
|
||||
let mut changed = Vec::new();
|
||||
for (vault_id, cursors) in vault_to_cursors.iter_mut() {
|
||||
let before = cursors.len();
|
||||
cursors.retain(|cursor| !cursor.is_expired(self.config.cursor_timeout));
|
||||
if cursors.len() != before {
|
||||
changed.push(vault_id.clone());
|
||||
}
|
||||
}
|
||||
|
||||
// Remove empty vault entries to prevent unbounded growth
|
||||
vault_to_cursors.retain(|_, cursors| !cursors.is_empty());
|
||||
|
||||
changed
|
||||
};
|
||||
|
||||
for vault_id in &changed_vaults {
|
||||
self.broadcast_cursors_for_vault(vault_id).await;
|
||||
}
|
||||
}
|
||||
|
||||
async fn broadcast_cursors(&self) {
|
||||
let vault_to_cursors = self.vault_to_cursors.lock().await;
|
||||
async fn broadcast_cursors_for_vault(&self, vault_id: &VaultId) {
|
||||
let client_cursors: Vec<ClientCursors> = {
|
||||
let vault_to_cursors = self.vault_to_cursors.lock().await;
|
||||
vault_to_cursors
|
||||
.get(vault_id)
|
||||
.map(|cursors| cursors.iter().map(|c| c.client_cursors.clone()).collect())
|
||||
.unwrap_or_default()
|
||||
};
|
||||
|
||||
for (vault_id, cursors) in vault_to_cursors.iter() {
|
||||
self.broadcasts
|
||||
.send_document_update(
|
||||
vault_id.clone(),
|
||||
WebSocketServerMessageWithOrigin::new(WebSocketServerMessage::CursorPositions(
|
||||
CursorPositionFromServer {
|
||||
clients: cursors.iter().map(|c| c.client_cursors.clone()).collect(),
|
||||
},
|
||||
)),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
self.broadcasts
|
||||
.send_document_update(
|
||||
vault_id.clone(),
|
||||
WebSocketServerMessageWithOrigin::new(WebSocketServerMessage::CursorPositions(
|
||||
CursorPositionFromServer {
|
||||
clients: client_cursors,
|
||||
},
|
||||
)),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
pub async fn remove_cursors_of_device(&self, vault_id: &str, device_id: &str) {
|
||||
let mut vault_to_cursors = self.vault_to_cursors.lock().await;
|
||||
pub async fn remove_cursors_of_device(&self, vault_id: &VaultId, device_id: &DeviceId) {
|
||||
let changed = {
|
||||
let mut vault_to_cursors = self.vault_to_cursors.lock().await;
|
||||
|
||||
if let Some(cursors) = vault_to_cursors.get_mut(vault_id) {
|
||||
cursors.retain(|c| c.client_cursors.device_id != device_id);
|
||||
if let Some(cursors) = vault_to_cursors.get_mut(vault_id) {
|
||||
let before = cursors.len();
|
||||
cursors.retain(|c| c.client_cursors.device_id != *device_id);
|
||||
let changed = cursors.len() != before;
|
||||
if cursors.is_empty() {
|
||||
vault_to_cursors.remove(vault_id);
|
||||
}
|
||||
changed
|
||||
} else {
|
||||
false
|
||||
}
|
||||
};
|
||||
|
||||
if changed {
|
||||
self.broadcast_cursors_for_vault(vault_id).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -6,14 +6,29 @@ use log::info;
|
|||
use models::{
|
||||
DocumentId, DocumentVersionWithoutContent, StoredDocumentVersion, VaultId, VaultUpdateId,
|
||||
};
|
||||
use sqlx::{ConnectOptions, sqlite::SqliteConnectOptions, types::chrono::Utc};
|
||||
use sqlx::{ConnectOptions, Connection, sqlite::SqliteConnectOptions, types::chrono::Utc};
|
||||
|
||||
pub mod models;
|
||||
use sqlx::{Pool, Sqlite, sqlite::SqlitePoolOptions};
|
||||
use tokio::sync::Mutex;
|
||||
use sqlx::{
|
||||
Pool, Sqlite, pool::PoolConnection, sqlite::SqliteConnection, sqlite::SqlitePoolOptions,
|
||||
};
|
||||
use tokio::sync::{Mutex, OnceCell};
|
||||
use tokio::time::Instant;
|
||||
use uuid::fmt::Hyphenated;
|
||||
|
||||
/// Row struct for vault history queries (used by `sqlx::query_as!`)
|
||||
#[derive(Debug)]
|
||||
struct VaultHistoryRow {
|
||||
vault_update_id: models::VaultUpdateId,
|
||||
document_id: models::DocumentId,
|
||||
relative_path: String,
|
||||
updated_date: chrono::DateTime<chrono::Utc>,
|
||||
is_deleted: bool,
|
||||
user_id: String,
|
||||
device_id: String,
|
||||
content_size: Option<u64>,
|
||||
}
|
||||
|
||||
use super::websocket::{
|
||||
broadcasts::Broadcasts,
|
||||
models::{WebSocketServerMessage, WebSocketServerMessageWithOrigin, WebSocketVaultUpdate},
|
||||
|
|
@ -21,32 +36,98 @@ use super::websocket::{
|
|||
use crate::config::database_config::DatabaseConfig;
|
||||
use crate::consts::IDLE_POOL_TIMEOUT;
|
||||
|
||||
#[derive(Clone)]
|
||||
struct PoolWithTimestamp {
|
||||
pool: Pool<Sqlite>,
|
||||
last_accessed: Instant,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for PoolWithTimestamp {
|
||||
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
|
||||
f.debug_struct("PoolWithTimestamp")
|
||||
.field("pool", &"Pool<Sqlite>")
|
||||
.field("last_accessed", &self.last_accessed)
|
||||
.finish()
|
||||
}
|
||||
#[derive(Debug)]
|
||||
struct VaultPool {
|
||||
cell: Arc<OnceCell<Pool<Sqlite>>>,
|
||||
last_accessed: Mutex<Instant>,
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct Database {
|
||||
config: DatabaseConfig,
|
||||
broadcasts: Broadcasts,
|
||||
connection_pools: Arc<Mutex<HashMap<VaultId, PoolWithTimestamp>>>,
|
||||
connection_pools: Arc<Mutex<HashMap<VaultId, Arc<VaultPool>>>>,
|
||||
}
|
||||
|
||||
pub type Transaction<'a> = sqlx::Transaction<'a, Sqlite>;
|
||||
/// A write transaction backed by a raw `BEGIN IMMEDIATE` instead of sqlx's
|
||||
/// savepoint-based `Transaction`. This avoids the savepoint mismatch caused
|
||||
/// by the old `END; BEGIN IMMEDIATE;` workaround.
|
||||
pub struct WriteTransaction {
|
||||
conn: Option<PoolConnection<Sqlite>>,
|
||||
}
|
||||
|
||||
impl WriteTransaction {
|
||||
async fn new(pool: &Pool<Sqlite>) -> Result<Self> {
|
||||
let mut conn = pool
|
||||
.acquire()
|
||||
.await
|
||||
.context("Cannot acquire connection for write transaction")?;
|
||||
sqlx::query("BEGIN IMMEDIATE")
|
||||
.execute(&mut *conn)
|
||||
.await
|
||||
.context("Cannot begin immediate transaction")?;
|
||||
Ok(Self { conn: Some(conn) })
|
||||
}
|
||||
|
||||
pub async fn commit(mut self) -> Result<()> {
|
||||
if let Some(mut conn) = self.conn.take() {
|
||||
sqlx::query("COMMIT")
|
||||
.execute(&mut *conn)
|
||||
.await
|
||||
.context("Failed to commit transaction")?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn rollback(mut self) -> Result<()> {
|
||||
if let Some(mut conn) = self.conn.take() {
|
||||
sqlx::query("ROLLBACK")
|
||||
.execute(&mut *conn)
|
||||
.await
|
||||
.context("Failed to rollback transaction")?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
impl Drop for WriteTransaction {
|
||||
fn drop(&mut self) {
|
||||
if self.conn.is_some() {
|
||||
// The connection is returned to the pool with an open transaction.
|
||||
// The pool's `before_acquire` hook issues a ROLLBACK before
|
||||
// handing it to the next consumer, so no async work is needed
|
||||
// here. If the pool is being shut down, SQLite itself rolls back
|
||||
// uncommitted transactions when the connection closes.
|
||||
log::warn!("WriteTransaction dropped without commit or rollback");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::Deref for WriteTransaction {
|
||||
type Target = SqliteConnection;
|
||||
fn deref(&self) -> &Self::Target {
|
||||
self.conn
|
||||
.as_ref()
|
||||
.expect("BUG: WriteTransaction dereferenced after being consumed")
|
||||
.deref()
|
||||
}
|
||||
}
|
||||
|
||||
impl std::ops::DerefMut for WriteTransaction {
|
||||
fn deref_mut(&mut self) -> &mut Self::Target {
|
||||
self.conn
|
||||
.as_mut()
|
||||
.expect("BUG: WriteTransaction dereferenced after being consumed")
|
||||
.deref_mut()
|
||||
}
|
||||
}
|
||||
|
||||
impl Database {
|
||||
pub async fn try_new(config: &DatabaseConfig, broadcasts: &Broadcasts) -> Result<Self> {
|
||||
pub async fn try_new(
|
||||
config: &DatabaseConfig,
|
||||
broadcasts: &Broadcasts,
|
||||
shutdown: tokio::sync::watch::Receiver<()>,
|
||||
) -> Result<Self> {
|
||||
tokio::fs::create_dir_all(&config.databases_directory_path)
|
||||
.await
|
||||
.with_context(|| {
|
||||
|
|
@ -71,13 +152,17 @@ impl Database {
|
|||
.trim_end_matches(".sqlite")
|
||||
.to_owned();
|
||||
|
||||
Self::validate_vault_id(&vault)?;
|
||||
|
||||
let pool = Self::create_vault_database(config, &vault).await?;
|
||||
let cell = Arc::new(OnceCell::new());
|
||||
cell.set(pool).expect("cell is new");
|
||||
connection_pools.insert(
|
||||
vault.clone(),
|
||||
PoolWithTimestamp {
|
||||
pool,
|
||||
last_accessed: Instant::now(),
|
||||
},
|
||||
Arc::new(VaultPool {
|
||||
cell,
|
||||
last_accessed: Mutex::new(Instant::now()),
|
||||
}),
|
||||
);
|
||||
}
|
||||
info!("Database migrations applied");
|
||||
|
|
@ -88,8 +173,7 @@ impl Database {
|
|||
broadcasts: broadcasts.clone(),
|
||||
};
|
||||
|
||||
// Start background task to cleanup idle connection pools
|
||||
database.start_idle_pool_cleanup();
|
||||
database.start_idle_pool_cleanup(shutdown);
|
||||
|
||||
Ok(database)
|
||||
}
|
||||
|
|
@ -102,91 +186,128 @@ impl Database {
|
|||
.databases_directory_path
|
||||
.join(format!("{vault}.sqlite"));
|
||||
|
||||
let connection_options = SqliteConnectOptions::new()
|
||||
// Database-level PRAGMAs (auto_vacuum, journal_mode) require a write
|
||||
// lock and persist across connections. Set them once with a dedicated
|
||||
// init connection so pool connections never need the write lock just to
|
||||
// open.
|
||||
let init_options = SqliteConnectOptions::new()
|
||||
.filename(file_name.clone())
|
||||
.create_if_missing(true)
|
||||
.auto_vacuum(sqlx::sqlite::SqliteAutoVacuum::Incremental)
|
||||
.busy_timeout(Duration::from_secs(30))
|
||||
.journal_mode(sqlx::sqlite::SqliteJournalMode::Wal);
|
||||
|
||||
// Run migrations on a dedicated connection, NOT through the pool.
|
||||
// The pool's `before_acquire` hook issues ROLLBACK on every checkout,
|
||||
// which can roll back the migration's bookkeeping transaction (the
|
||||
// _sqlx_migrations INSERT) while the DDL (ALTER TABLE) has already
|
||||
// auto-committed — leaving the migration in a dirty state.
|
||||
//
|
||||
// Uses `run_direct` instead of `run` because `run` takes
|
||||
// `impl Acquire<'_>`, whose lifetime bound prevents the enclosing
|
||||
// future from satisfying the `Send` requirement of axum handlers.
|
||||
let mut init_conn = sqlx::SqliteConnection::connect_with(&init_options).await?;
|
||||
sqlx::migrate!("src/app_state/database/migrations")
|
||||
.run_direct(&mut init_conn)
|
||||
.await
|
||||
.context("Cannot run pending migrations")?;
|
||||
drop(init_conn);
|
||||
|
||||
// Pool connections only set per-connection PRAGMAs that don't require a
|
||||
// write lock. journal_mode = WAL is a no-op on an already-WAL database.
|
||||
let pool_options = SqliteConnectOptions::new()
|
||||
.filename(file_name.clone())
|
||||
.journal_mode(sqlx::sqlite::SqliteJournalMode::Wal)
|
||||
.busy_timeout(Duration::from_secs(30))
|
||||
.log_slow_statements(log::LevelFilter::Warn, Duration::from_secs(30));
|
||||
|
||||
let pool = SqlitePoolOptions::new()
|
||||
.max_connections(config.max_connections_per_vault)
|
||||
.acquire_slow_threshold(Duration::from_secs(30))
|
||||
.test_before_acquire(true)
|
||||
.connect_with(connection_options)
|
||||
.before_acquire(|conn, _meta| {
|
||||
Box::pin(async move {
|
||||
// Ensure the connection has no leftover open transaction
|
||||
// (e.g. from a WriteTransaction that was dropped without
|
||||
// commit/rollback). ROLLBACK is a harmless no-op if no
|
||||
// transaction is active.
|
||||
if let Err(e) = sqlx::query("ROLLBACK").execute(&mut *conn).await {
|
||||
// "cannot rollback - no transaction is active" is the
|
||||
// common case (connection returned cleanly). Only
|
||||
// unexpected errors deserve attention.
|
||||
log::debug!("before_acquire ROLLBACK failed: {e}");
|
||||
}
|
||||
Ok(true)
|
||||
})
|
||||
})
|
||||
.connect_with(pool_options)
|
||||
.await
|
||||
.with_context(|| format!("Cannot open database at `{}`", file_name.display()))?;
|
||||
|
||||
Self::run_migrations(&pool).await?;
|
||||
|
||||
Ok(pool)
|
||||
}
|
||||
|
||||
async fn run_migrations(pool: &Pool<Sqlite>) -> Result<()> {
|
||||
sqlx::migrate!("src/app_state/database/migrations")
|
||||
.run(pool)
|
||||
.await
|
||||
.context("Cannot check for pending migrations")
|
||||
|
||||
fn validate_vault_id(vault: &VaultId) -> Result<()> {
|
||||
if vault.is_empty() {
|
||||
anyhow::bail!("Vault ID must not be empty");
|
||||
}
|
||||
if vault.contains('/')
|
||||
|| vault.contains('\\')
|
||||
|| vault.contains("..")
|
||||
|| vault.contains('\0')
|
||||
{
|
||||
anyhow::bail!(
|
||||
"Invalid vault ID: must not contain path separators, '..', or null bytes"
|
||||
);
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn get_connection_pool(&self, vault: &VaultId) -> Result<Pool<Sqlite>> {
|
||||
// First, check if the pool exists without holding the lock during creation
|
||||
{
|
||||
Self::validate_vault_id(vault)?;
|
||||
|
||||
// Get or create the VaultPool entry. The global lock is held only
|
||||
// long enough for a HashMap lookup/insert — never across
|
||||
// create_vault_database.
|
||||
let vault_pool = {
|
||||
let mut pools = self.connection_pools.lock().await;
|
||||
if let Some(pool_with_timestamp) = pools.get_mut(vault) {
|
||||
pool_with_timestamp.last_accessed = Instant::now();
|
||||
return Ok(pool_with_timestamp.pool.clone());
|
||||
}
|
||||
}
|
||||
pools
|
||||
.entry(vault.clone())
|
||||
.or_insert_with(|| {
|
||||
Arc::new(VaultPool {
|
||||
cell: Arc::new(OnceCell::new()),
|
||||
last_accessed: Mutex::new(Instant::now()),
|
||||
})
|
||||
})
|
||||
.clone()
|
||||
};
|
||||
|
||||
// Create the pool outside of the lock to avoid blocking other vaults
|
||||
// Note: This may result in multiple pools being created for the same vault
|
||||
// under high concurrency, but only one will be kept
|
||||
let new_pool = Self::create_vault_database(&self.config, vault).await?;
|
||||
|
||||
// Re-acquire lock and insert (or use existing if another task created it)
|
||||
let mut pools = self.connection_pools.lock().await;
|
||||
let pool_with_timestamp = pools
|
||||
.entry(vault.clone())
|
||||
.or_insert_with(|| PoolWithTimestamp {
|
||||
pool: new_pool.clone(),
|
||||
last_accessed: Instant::now(),
|
||||
});
|
||||
|
||||
pool_with_timestamp.last_accessed = Instant::now();
|
||||
Ok(pool_with_timestamp.pool.clone())
|
||||
}
|
||||
|
||||
/// Attempting to write from this transaction might result in a
|
||||
/// database locked error. Use this transaction for read-only operations.
|
||||
pub async fn create_readonly_transaction(
|
||||
&self,
|
||||
vault: &VaultId,
|
||||
) -> Result<Transaction<'static>> {
|
||||
self.get_connection_pool(vault)
|
||||
.await?
|
||||
.begin()
|
||||
.await
|
||||
.context("Cannot create transaction")
|
||||
}
|
||||
|
||||
pub async fn create_write_transaction(&self, vault: &VaultId) -> Result<Transaction<'static>> {
|
||||
let mut transaction = self.create_readonly_transaction(vault).await?;
|
||||
|
||||
// sqlx doesn't support immediate transactions for sqlite: https://github.com/launchbadge/sqlx/issues/481
|
||||
sqlx::query!("END; BEGIN IMMEDIATE;")
|
||||
.execute(&mut *transaction)
|
||||
// OnceCell::get_or_try_init guarantees exactly-once
|
||||
// initialization: concurrent callers for the same vault wait
|
||||
// here; callers for other vaults are not blocked.
|
||||
let config = self.config.clone();
|
||||
let vault_clone = vault.clone();
|
||||
let pool = vault_pool
|
||||
.cell
|
||||
.get_or_try_init(|| async {
|
||||
Self::create_vault_database(&config, &vault_clone).await
|
||||
})
|
||||
.await?;
|
||||
|
||||
Ok(transaction)
|
||||
*vault_pool.last_accessed.lock().await = Instant::now();
|
||||
Ok(pool.clone())
|
||||
}
|
||||
|
||||
pub async fn create_write_transaction(&self, vault: &VaultId) -> Result<WriteTransaction> {
|
||||
let pool = self.get_connection_pool(vault).await?;
|
||||
WriteTransaction::new(&pool).await
|
||||
}
|
||||
|
||||
/// Return the latest state of all documents in the vault
|
||||
pub async fn get_latest_documents(
|
||||
&self,
|
||||
vault: &VaultId,
|
||||
transaction: Option<&mut Transaction<'_>>,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<Vec<DocumentVersionWithoutContent>> {
|
||||
let query = sqlx::query!(
|
||||
r#"
|
||||
|
|
@ -204,8 +325,8 @@ impl Database {
|
|||
"#,
|
||||
);
|
||||
|
||||
if let Some(transaction) = transaction {
|
||||
query.fetch_all(&mut **transaction).await
|
||||
if let Some(conn) = connection {
|
||||
query.fetch_all(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_all(&self.get_connection_pool(vault).await?)
|
||||
|
|
@ -222,9 +343,7 @@ impl Database {
|
|||
is_deleted: row.is_deleted,
|
||||
user_id: row.user_id,
|
||||
device_id: row.device_id,
|
||||
content_size: row
|
||||
.content_size
|
||||
.expect("Content size can't be null but sqlx can't infer it"),
|
||||
content_size: row.content_size.unwrap_or(0),
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
|
|
@ -236,7 +355,7 @@ impl Database {
|
|||
&self,
|
||||
vault: &VaultId,
|
||||
vault_update_id: VaultUpdateId,
|
||||
transaction: Option<&mut Transaction<'_>>,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<Vec<DocumentVersionWithoutContent>> {
|
||||
let query = sqlx::query!(
|
||||
r#"
|
||||
|
|
@ -256,8 +375,8 @@ impl Database {
|
|||
vault_update_id
|
||||
);
|
||||
|
||||
if let Some(transaction) = transaction {
|
||||
query.fetch_all(&mut **transaction).await
|
||||
if let Some(conn) = connection {
|
||||
query.fetch_all(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_all(&self.get_connection_pool(vault).await?)
|
||||
|
|
@ -276,9 +395,7 @@ impl Database {
|
|||
is_deleted: row.is_deleted,
|
||||
user_id: row.user_id,
|
||||
device_id: row.device_id,
|
||||
content_size: row
|
||||
.content_size
|
||||
.expect("Content size can't be null but sqlx can't infer it"),
|
||||
content_size: row.content_size.unwrap_or(0),
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
|
|
@ -287,7 +404,7 @@ impl Database {
|
|||
pub async fn get_max_update_id_in_vault(
|
||||
&self,
|
||||
vault: &VaultId,
|
||||
transaction: Option<&mut Transaction<'_>>,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<i64> {
|
||||
let query = sqlx::query!(
|
||||
r#"
|
||||
|
|
@ -296,8 +413,8 @@ impl Database {
|
|||
"#,
|
||||
);
|
||||
|
||||
if let Some(transaction) = transaction {
|
||||
query.fetch_one(&mut **transaction).await
|
||||
if let Some(conn) = connection {
|
||||
query.fetch_one(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_one(&self.get_connection_pool(vault).await?)
|
||||
|
|
@ -311,7 +428,7 @@ impl Database {
|
|||
&self,
|
||||
vault: &VaultId,
|
||||
relative_path: &str,
|
||||
transaction: Option<&mut Transaction<'_>>,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<Option<StoredDocumentVersion>> {
|
||||
let query = sqlx::query_as!(
|
||||
StoredDocumentVersion,
|
||||
|
|
@ -337,8 +454,8 @@ impl Database {
|
|||
relative_path
|
||||
);
|
||||
|
||||
if let Some(transaction) = transaction {
|
||||
query.fetch_optional(&mut **transaction).await
|
||||
if let Some(conn) = connection {
|
||||
query.fetch_optional(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_optional(&self.get_connection_pool(vault).await?)
|
||||
|
|
@ -351,7 +468,7 @@ impl Database {
|
|||
&self,
|
||||
vault: &VaultId,
|
||||
document_id: &DocumentId,
|
||||
transaction: Option<&mut Transaction<'_>>,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<Option<StoredDocumentVersion>> {
|
||||
let document_id = document_id.as_hyphenated();
|
||||
let query = sqlx::query_as!(
|
||||
|
|
@ -374,8 +491,8 @@ impl Database {
|
|||
document_id
|
||||
);
|
||||
|
||||
if let Some(transaction) = transaction {
|
||||
query.fetch_optional(&mut **transaction).await
|
||||
if let Some(conn) = connection {
|
||||
query.fetch_optional(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_optional(&self.get_connection_pool(vault).await?)
|
||||
|
|
@ -388,7 +505,7 @@ impl Database {
|
|||
&self,
|
||||
vault: &VaultId,
|
||||
vault_update_id: VaultUpdateId,
|
||||
transaction: Option<&mut Transaction<'_>>,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<Option<StoredDocumentVersion>> {
|
||||
let query = sqlx::query_as!(
|
||||
StoredDocumentVersion,
|
||||
|
|
@ -409,8 +526,8 @@ impl Database {
|
|||
vault_update_id
|
||||
);
|
||||
|
||||
if let Some(transaction) = transaction {
|
||||
query.fetch_optional(&mut **transaction).await
|
||||
if let Some(conn) = connection {
|
||||
query.fetch_optional(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_optional(&self.get_connection_pool(vault).await?)
|
||||
|
|
@ -424,7 +541,7 @@ impl Database {
|
|||
&self,
|
||||
vault_id: &VaultId,
|
||||
version: &StoredDocumentVersion,
|
||||
transaction: Option<Transaction<'_>>,
|
||||
transaction: Option<WriteTransaction>,
|
||||
) -> Result<()> {
|
||||
let document_id = version.document_id.as_hyphenated();
|
||||
let query = sqlx::query!(
|
||||
|
|
@ -438,9 +555,10 @@ impl Database {
|
|||
is_deleted,
|
||||
user_id,
|
||||
device_id,
|
||||
idempotency_key
|
||||
idempotency_key,
|
||||
has_been_merged
|
||||
)
|
||||
values (?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
"#,
|
||||
version.vault_update_id,
|
||||
document_id,
|
||||
|
|
@ -450,7 +568,8 @@ impl Database {
|
|||
version.is_deleted,
|
||||
version.user_id,
|
||||
version.device_id,
|
||||
version.idempotency_key
|
||||
version.idempotency_key,
|
||||
version.has_been_merged
|
||||
);
|
||||
|
||||
if let Some(mut transaction) = transaction {
|
||||
|
|
@ -490,32 +609,36 @@ impl Database {
|
|||
&self,
|
||||
vault: &VaultId,
|
||||
idempotency_key: &str,
|
||||
transaction: Option<&mut Transaction<'_>>,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<Option<StoredDocumentVersion>> {
|
||||
// Start from the `documents` table (which has an index on
|
||||
// `idempotency_key`) to find the document_id, then join to
|
||||
// `latest_document_versions` for the latest state.
|
||||
let query = sqlx::query_as!(
|
||||
StoredDocumentVersion,
|
||||
r#"
|
||||
select
|
||||
d.vault_update_id,
|
||||
d.document_id as "document_id: Hyphenated",
|
||||
d.relative_path,
|
||||
d.updated_date as "updated_date: chrono::DateTime<Utc>",
|
||||
d.content,
|
||||
d.is_deleted,
|
||||
d.user_id,
|
||||
d.device_id,
|
||||
d.has_been_merged,
|
||||
d.idempotency_key
|
||||
from latest_document_versions d
|
||||
inner join documents d2 on d.document_id = d2.document_id
|
||||
where d2.idempotency_key = ?
|
||||
ldv.vault_update_id,
|
||||
ldv.document_id as "document_id: Hyphenated",
|
||||
ldv.relative_path,
|
||||
ldv.updated_date as "updated_date: chrono::DateTime<Utc>",
|
||||
ldv.content,
|
||||
ldv.is_deleted,
|
||||
ldv.user_id,
|
||||
ldv.device_id,
|
||||
ldv.has_been_merged,
|
||||
ldv.idempotency_key
|
||||
from documents d
|
||||
inner join latest_document_versions ldv on d.document_id = ldv.document_id
|
||||
where d.idempotency_key = ?
|
||||
order by ldv.vault_update_id desc
|
||||
limit 1
|
||||
"#,
|
||||
idempotency_key
|
||||
);
|
||||
|
||||
if let Some(transaction) = transaction {
|
||||
query.fetch_optional(&mut **transaction).await
|
||||
if let Some(conn) = connection {
|
||||
query.fetch_optional(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_optional(&self.get_connection_pool(vault).await?)
|
||||
|
|
@ -524,39 +647,192 @@ impl Database {
|
|||
.context("Cannot fetch document by idempotency key")
|
||||
}
|
||||
|
||||
/// Return all versions (without content) of a specific document, ordered by `vault_update_id`
|
||||
pub async fn get_document_versions(
|
||||
&self,
|
||||
vault: &VaultId,
|
||||
document_id: &DocumentId,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<Vec<DocumentVersionWithoutContent>> {
|
||||
let document_id = document_id.as_hyphenated();
|
||||
let query = sqlx::query!(
|
||||
r#"
|
||||
select
|
||||
vault_update_id,
|
||||
document_id as "document_id: Hyphenated",
|
||||
relative_path,
|
||||
updated_date as "updated_date: chrono::DateTime<Utc>",
|
||||
is_deleted,
|
||||
user_id,
|
||||
device_id,
|
||||
length(content) as "content_size: u64"
|
||||
from documents
|
||||
where document_id = ?
|
||||
order by vault_update_id
|
||||
"#,
|
||||
document_id,
|
||||
);
|
||||
|
||||
if let Some(conn) = connection {
|
||||
query.fetch_all(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_all(&self.get_connection_pool(vault).await?)
|
||||
.await
|
||||
}
|
||||
.with_context(|| format!("Cannot fetch document versions for document `{document_id}`"))
|
||||
.map(|rows| {
|
||||
rows.into_iter()
|
||||
.map(|row| DocumentVersionWithoutContent {
|
||||
vault_update_id: row.vault_update_id,
|
||||
document_id: row.document_id.into(),
|
||||
relative_path: row.relative_path,
|
||||
updated_date: row.updated_date,
|
||||
is_deleted: row.is_deleted,
|
||||
user_id: row.user_id,
|
||||
device_id: row.device_id,
|
||||
content_size: row.content_size.unwrap_or(0),
|
||||
})
|
||||
.collect()
|
||||
})
|
||||
}
|
||||
|
||||
/// Return all versions across all documents, paginated, ordered by `vault_update_id` DESC
|
||||
pub async fn get_vault_history(
|
||||
&self,
|
||||
vault: &VaultId,
|
||||
limit: i64,
|
||||
before_update_id: Option<VaultUpdateId>,
|
||||
connection: Option<&mut SqliteConnection>,
|
||||
) -> Result<Vec<DocumentVersionWithoutContent>> {
|
||||
let map_row = |row: VaultHistoryRow| DocumentVersionWithoutContent {
|
||||
vault_update_id: row.vault_update_id,
|
||||
document_id: row.document_id,
|
||||
relative_path: row.relative_path,
|
||||
updated_date: row.updated_date,
|
||||
is_deleted: row.is_deleted,
|
||||
user_id: row.user_id,
|
||||
device_id: row.device_id,
|
||||
content_size: row.content_size.unwrap_or(0),
|
||||
};
|
||||
|
||||
if let Some(before) = before_update_id {
|
||||
let query = sqlx::query_as!(
|
||||
VaultHistoryRow,
|
||||
r#"
|
||||
select
|
||||
vault_update_id,
|
||||
document_id as "document_id: Hyphenated",
|
||||
relative_path,
|
||||
updated_date as "updated_date: chrono::DateTime<Utc>",
|
||||
is_deleted,
|
||||
user_id,
|
||||
device_id,
|
||||
length(content) as "content_size: u64"
|
||||
from documents
|
||||
where vault_update_id < ?
|
||||
order by vault_update_id desc
|
||||
limit ?
|
||||
"#,
|
||||
before,
|
||||
limit,
|
||||
);
|
||||
|
||||
let rows = if let Some(conn) = connection {
|
||||
query.fetch_all(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_all(&self.get_connection_pool(vault).await?)
|
||||
.await
|
||||
}
|
||||
.context("Cannot fetch vault history")?;
|
||||
|
||||
Ok(rows.into_iter().map(map_row).collect())
|
||||
} else {
|
||||
let query = sqlx::query_as!(
|
||||
VaultHistoryRow,
|
||||
r#"
|
||||
select
|
||||
vault_update_id,
|
||||
document_id as "document_id: Hyphenated",
|
||||
relative_path,
|
||||
updated_date as "updated_date: chrono::DateTime<Utc>",
|
||||
is_deleted,
|
||||
user_id,
|
||||
device_id,
|
||||
length(content) as "content_size: u64"
|
||||
from documents
|
||||
order by vault_update_id desc
|
||||
limit ?
|
||||
"#,
|
||||
limit,
|
||||
);
|
||||
|
||||
let rows = if let Some(conn) = connection {
|
||||
query.fetch_all(&mut *conn).await
|
||||
} else {
|
||||
query
|
||||
.fetch_all(&self.get_connection_pool(vault).await?)
|
||||
.await
|
||||
}
|
||||
.context("Cannot fetch vault history")?;
|
||||
|
||||
Ok(rows.into_iter().map(map_row).collect())
|
||||
}
|
||||
}
|
||||
|
||||
/// Cleanup idle connection pools that haven't been accessed in more than 5 minutes
|
||||
async fn cleanup_idle_pools(&self) {
|
||||
let mut pools = self.connection_pools.lock().await;
|
||||
let now = Instant::now();
|
||||
// Collect idle vaults and remove them from the map while holding
|
||||
// the lock briefly. Close pools OUTSIDE the lock so that
|
||||
// pool.close().await doesn't block other get_connection_pool calls.
|
||||
let idle_pools: Vec<(VaultId, Arc<VaultPool>)> = {
|
||||
let mut pools = self.connection_pools.lock().await;
|
||||
let now = Instant::now();
|
||||
|
||||
// Collect vaults to remove
|
||||
let vaults_to_remove: Vec<VaultId> = pools
|
||||
.iter()
|
||||
.filter(|(_, pool_with_timestamp)| {
|
||||
now.duration_since(pool_with_timestamp.last_accessed) > IDLE_POOL_TIMEOUT
|
||||
})
|
||||
.map(|(vault_id, _)| vault_id.clone())
|
||||
.collect();
|
||||
let vaults_to_remove: Vec<VaultId> = pools
|
||||
.iter()
|
||||
.filter(|(_, vp)| {
|
||||
// If the lock is contested, the pool is actively used — not idle.
|
||||
let Ok(last) = vp.last_accessed.try_lock() else {
|
||||
return false;
|
||||
};
|
||||
now.duration_since(*last) > IDLE_POOL_TIMEOUT
|
||||
})
|
||||
.map(|(vault_id, _)| vault_id.clone())
|
||||
.collect();
|
||||
|
||||
// Close and remove idle pools
|
||||
for vault_id in &vaults_to_remove {
|
||||
if let Some(pool_with_timestamp) = pools.remove(vault_id) {
|
||||
vaults_to_remove
|
||||
.into_iter()
|
||||
.filter_map(|id| pools.remove(&id).map(|vp| (id, vp)))
|
||||
.collect()
|
||||
};
|
||||
|
||||
for (vault_id, vault_pool) in idle_pools {
|
||||
if let Some(pool) = vault_pool.cell.get() {
|
||||
info!("Closing idle database connection pool for vault `{vault_id}`");
|
||||
pool_with_timestamp.pool.close().await;
|
||||
pool.close().await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Start a background task that periodically cleans up idle connection pools
|
||||
fn start_idle_pool_cleanup(&self) {
|
||||
fn start_idle_pool_cleanup(&self, mut shutdown: tokio::sync::watch::Receiver<()>) {
|
||||
let database = self.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut interval = tokio::time::interval(Duration::from_secs(60)); // Check every minute
|
||||
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip);
|
||||
|
||||
loop {
|
||||
interval.tick().await;
|
||||
database.cleanup_idle_pools().await;
|
||||
tokio::select! {
|
||||
_ = interval.tick() => {
|
||||
database.cleanup_idle_pools().await;
|
||||
}
|
||||
_ = shutdown.changed() => {
|
||||
info!("Idle pool cleanup task shutting down");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
|
|
|||
|
|
@ -0,0 +1,2 @@
|
|||
CREATE UNIQUE INDEX IF NOT EXISTS idx_documents_idempotency_key
|
||||
ON documents (idempotency_key) WHERE idempotency_key IS NOT NULL AND is_deleted = 0;
|
||||
|
|
@ -0,0 +1,2 @@
|
|||
CREATE INDEX IF NOT EXISTS idx_documents_document_id
|
||||
ON documents (document_id, vault_update_id);
|
||||
|
|
@ -25,6 +25,8 @@ pub struct StoredDocumentVersion {
|
|||
pub idempotency_key: Option<String>,
|
||||
}
|
||||
|
||||
/// Equality is based solely on `vault_update_id` (the primary key).
|
||||
/// Two rows with the same PK are the same database record.
|
||||
impl PartialEq<Self> for StoredDocumentVersion {
|
||||
fn eq(&self, other: &Self) -> bool {
|
||||
self.vault_update_id == other.vault_update_id
|
||||
|
|
@ -34,7 +36,7 @@ impl PartialEq<Self> for StoredDocumentVersion {
|
|||
#[derive(TS, Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct DocumentVersionWithoutContent {
|
||||
#[ts(as = "i32")]
|
||||
#[ts(type = "number")]
|
||||
pub vault_update_id: VaultUpdateId,
|
||||
|
||||
pub document_id: DocumentId,
|
||||
|
|
@ -44,7 +46,7 @@ pub struct DocumentVersionWithoutContent {
|
|||
pub user_id: UserId,
|
||||
pub device_id: DeviceId,
|
||||
|
||||
#[ts(as = "i32")]
|
||||
#[ts(type = "number")]
|
||||
pub content_size: u64,
|
||||
}
|
||||
|
||||
|
|
@ -66,7 +68,7 @@ impl From<StoredDocumentVersion> for DocumentVersionWithoutContent {
|
|||
#[derive(TS, Debug, Clone, Serialize)]
|
||||
#[serde(rename_all = "camelCase")]
|
||||
pub struct DocumentVersion {
|
||||
#[ts(as = "i32")]
|
||||
#[ts(type = "number")]
|
||||
pub vault_update_id: VaultUpdateId,
|
||||
|
||||
pub document_id: DocumentId,
|
||||
|
|
|
|||
|
|
@ -1,24 +1,21 @@
|
|||
use std::{collections::HashMap, sync::Arc};
|
||||
|
||||
use anyhow::Context;
|
||||
use log::{debug, warn};
|
||||
use tokio::sync::{Mutex, broadcast};
|
||||
|
||||
use super::models::WebSocketServerMessageWithOrigin;
|
||||
use crate::{
|
||||
app_state::database::models::VaultId, config::server_config::ServerConfig, errors::server_error,
|
||||
};
|
||||
use crate::{app_state::database::models::VaultId, config::server_config::ServerConfig};
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
pub struct Broadcasts {
|
||||
max_clients_per_vault: usize,
|
||||
broadcast_channel_capacity: usize,
|
||||
tx: Arc<Mutex<HashMap<VaultId, broadcast::Sender<WebSocketServerMessageWithOrigin>>>>,
|
||||
}
|
||||
|
||||
impl Broadcasts {
|
||||
pub fn new(server_config: &ServerConfig) -> Self {
|
||||
Self {
|
||||
max_clients_per_vault: server_config.max_clients_per_vault,
|
||||
broadcast_channel_capacity: server_config.broadcast_channel_capacity,
|
||||
tx: Arc::new(Mutex::new(HashMap::new())),
|
||||
}
|
||||
}
|
||||
|
|
@ -26,10 +23,25 @@ impl Broadcasts {
|
|||
pub async fn get_receiver(
|
||||
&self,
|
||||
vault: VaultId,
|
||||
) -> broadcast::Receiver<WebSocketServerMessageWithOrigin> {
|
||||
let tx = self.get_or_create(vault).await;
|
||||
max_clients: usize,
|
||||
) -> Result<broadcast::Receiver<WebSocketServerMessageWithOrigin>, crate::errors::SyncServerError>
|
||||
{
|
||||
let mut tx_map = self.tx.lock().await;
|
||||
|
||||
tx.subscribe()
|
||||
// Prune senders for vaults with no active receivers
|
||||
tx_map.retain(|_, sender| sender.receiver_count() > 0);
|
||||
|
||||
let sender = tx_map
|
||||
.entry(vault)
|
||||
.or_insert_with(|| broadcast::channel(self.broadcast_channel_capacity).0);
|
||||
|
||||
if sender.receiver_count() >= max_clients {
|
||||
return Err(crate::errors::client_error(anyhow::anyhow!(
|
||||
"Vault has reached the maximum number of clients ({max_clients})"
|
||||
)));
|
||||
}
|
||||
|
||||
Ok(sender.subscribe())
|
||||
}
|
||||
|
||||
/// Notify all clients (who are subscribed to the vault) about an update.
|
||||
|
|
@ -39,31 +51,22 @@ impl Broadcasts {
|
|||
vault: VaultId,
|
||||
document: WebSocketServerMessageWithOrigin,
|
||||
) {
|
||||
let tx = self.get_or_create(vault.clone()).await;
|
||||
let mut tx_map = self.tx.lock().await;
|
||||
|
||||
if tx.receiver_count() == 0 {
|
||||
// Prune senders for vaults with no active receivers
|
||||
tx_map.retain(|_, sender| sender.receiver_count() > 0);
|
||||
|
||||
let sender = tx_map
|
||||
.entry(vault.clone())
|
||||
.or_insert_with(|| broadcast::channel(self.broadcast_channel_capacity).0);
|
||||
|
||||
if sender.receiver_count() == 0 {
|
||||
debug!("Skipping broadcast, no clients connected for vault `{vault}`");
|
||||
return;
|
||||
}
|
||||
|
||||
let result = tx
|
||||
.send(document)
|
||||
.context("Cannot broadcast server message to websocket listeners")
|
||||
.map_err(server_error);
|
||||
|
||||
if result.is_err() {
|
||||
warn!("Failed to send message: {result:?}");
|
||||
if let Err(e) = sender.send(document) {
|
||||
warn!("Failed to broadcast to vault `{vault}`: {e}");
|
||||
}
|
||||
}
|
||||
|
||||
async fn get_or_create(
|
||||
&self,
|
||||
vault: VaultId,
|
||||
) -> broadcast::Sender<WebSocketServerMessageWithOrigin> {
|
||||
let mut tx = self.tx.lock().await;
|
||||
|
||||
tx.entry(vault)
|
||||
.or_insert_with(|| broadcast::channel(self.max_clients_per_vault).0.clone())
|
||||
.clone()
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ pub struct WebSocketHandshake {
|
|||
pub token: String,
|
||||
pub device_id: DeviceId,
|
||||
|
||||
#[ts(as = "Option<i32>")]
|
||||
#[ts(type = "number | null")]
|
||||
pub last_seen_vault_update_id: Option<VaultUpdateId>,
|
||||
}
|
||||
|
||||
|
|
@ -28,7 +28,7 @@ pub struct DocumentWithCursors {
|
|||
// that it exists and can be client-side
|
||||
// interpolated. However, the actual
|
||||
// position is meaningless.
|
||||
#[ts(as = "Option<u32>")]
|
||||
#[ts(type = "number | null")]
|
||||
pub vault_update_id: Option<VaultUpdateId>,
|
||||
|
||||
pub document_id: DocumentId,
|
||||
|
|
@ -70,6 +70,7 @@ pub struct WebSocketVaultUpdate {
|
|||
pub enum WebSocketClientMessage {
|
||||
Handshake(WebSocketHandshake),
|
||||
CursorPositions(CursorPositionFromClient),
|
||||
Ping {},
|
||||
}
|
||||
|
||||
#[derive(TS, Serialize, Clone, Debug)]
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ use crate::{
|
|||
database::models::{DocumentVersionWithoutContent, VaultId, VaultUpdateId},
|
||||
},
|
||||
config::user_config::User,
|
||||
errors::{SyncServerError, server_error, unauthenticated_error},
|
||||
errors::{SyncServerError, client_error, server_error, unauthenticated_error},
|
||||
server::auth::auth,
|
||||
};
|
||||
|
||||
|
|
@ -26,16 +26,16 @@ pub fn get_authenticated_handshake(
|
|||
if let Some(Message::Text(message)) = message {
|
||||
let message: WebSocketClientMessage = serde_json::from_str(&message)
|
||||
.context("Failed to parse message")
|
||||
.map_err(server_error)?;
|
||||
.map_err(client_error)?;
|
||||
|
||||
match message {
|
||||
WebSocketClientMessage::Handshake(handshake) => {
|
||||
let user = auth(state, handshake.token.trim(), vault_id)?;
|
||||
Ok(AuthenticatedWebSocketHandshake { handshake, user })
|
||||
}
|
||||
WebSocketClientMessage::CursorPositions(_) => Err(unauthenticated_error(
|
||||
anyhow::anyhow!("Expected a handshake message"),
|
||||
)),
|
||||
WebSocketClientMessage::CursorPositions(_) | WebSocketClientMessage::Ping {} => Err(
|
||||
unauthenticated_error(anyhow::anyhow!("Expected a handshake message")),
|
||||
),
|
||||
}
|
||||
} else {
|
||||
Err(unauthenticated_error(anyhow::anyhow!(
|
||||
|
|
|
|||
|
|
@ -28,23 +28,20 @@ pub struct Config {
|
|||
|
||||
impl Config {
|
||||
pub async fn read_or_create(path: &Path) -> Result<Self> {
|
||||
let config = if path.exists() {
|
||||
info!(
|
||||
"Loading configuration from `{}`",
|
||||
path.canonicalize().unwrap().display()
|
||||
);
|
||||
Self::load_from_file(path).await?
|
||||
let display_path = path.canonicalize().unwrap_or_else(|_| path.to_path_buf());
|
||||
|
||||
if path.exists() {
|
||||
info!("Loading configuration from `{}`", display_path.display());
|
||||
Self::load_from_file(path).await
|
||||
} else {
|
||||
Self::default()
|
||||
};
|
||||
|
||||
config.write(path).await?;
|
||||
info!(
|
||||
"Updated configuration at `{}`",
|
||||
path.canonicalize().unwrap().display()
|
||||
);
|
||||
|
||||
Ok(config)
|
||||
let config = Self::default();
|
||||
config.write(path).await?;
|
||||
info!(
|
||||
"Created default configuration at `{}`",
|
||||
display_path.display()
|
||||
);
|
||||
Ok(config)
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn load_from_file(path: &Path) -> Result<Self> {
|
||||
|
|
|
|||
|
|
@ -1,10 +1,13 @@
|
|||
use anyhow::{Result, ensure};
|
||||
use log::debug;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::time::Duration;
|
||||
|
||||
use crate::consts::{
|
||||
DEFAULT_HOST, DEFAULT_MAX_BODY_SIZE_MB, DEFAULT_MAX_CLIENTS_PER_VAULT,
|
||||
DEFAULT_MERGEABLE_FILE_EXTENSIONS, DEFAULT_PORT, DEFAULT_RESPONSE_TIMEOUT_SECONDS,
|
||||
DEFAULT_ALLOWED_ORIGINS, DEFAULT_BROADCAST_CHANNEL_CAPACITY, DEFAULT_HOST,
|
||||
DEFAULT_MAX_BODY_SIZE_MB, DEFAULT_MAX_CLIENTS_PER_VAULT, DEFAULT_MAX_PENDING_WS_CONNECTIONS,
|
||||
DEFAULT_MERGEABLE_FILE_EXTENSIONS, DEFAULT_PORT, DEFAULT_RATE_LIMIT_PER_SECOND,
|
||||
DEFAULT_RESPONSE_TIMEOUT_SECONDS,
|
||||
};
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize, Clone, Default)]
|
||||
|
|
@ -21,11 +24,68 @@ pub struct ServerConfig {
|
|||
#[serde(default = "default_max_clients_per_vault")]
|
||||
pub max_clients_per_vault: usize,
|
||||
|
||||
#[serde(default = "default_broadcast_channel_capacity")]
|
||||
pub broadcast_channel_capacity: usize,
|
||||
|
||||
#[serde(default = "default_response_timeout", with = "humantime_serde")]
|
||||
pub response_timeout: Duration,
|
||||
|
||||
#[serde(default = "default_mergeable_file_extensions")]
|
||||
pub mergeable_file_extensions: Vec<String>,
|
||||
|
||||
/// Maximum requests per second (0 = disabled).
|
||||
#[serde(default = "default_rate_limit_per_second")]
|
||||
pub rate_limit_per_second: u64,
|
||||
|
||||
/// Allowed CORS origins. Default: `["*"]` (allow all).
|
||||
#[serde(default = "default_allowed_origins")]
|
||||
pub allowed_origins: Vec<String>,
|
||||
|
||||
/// Maximum concurrent unauthenticated WebSocket connections waiting for
|
||||
/// handshake. Limits resource consumption from clients that connect but
|
||||
/// never authenticate.
|
||||
#[serde(default = "default_max_pending_websocket_connections")]
|
||||
pub max_pending_websocket_connections: usize,
|
||||
|
||||
/// When set, proxies all UI requests (index, assets, Vite HMR) to this
|
||||
/// URL instead of serving embedded assets. Typically
|
||||
/// `http://localhost:5173` for the Vite dev server.
|
||||
#[serde(default)]
|
||||
pub dev_proxy_url: Option<String>,
|
||||
}
|
||||
|
||||
impl ServerConfig {
|
||||
pub fn validate(&self) -> Result<()> {
|
||||
ensure!(
|
||||
!self.response_timeout.is_zero(),
|
||||
"response_timeout must be greater than 0"
|
||||
);
|
||||
ensure!(
|
||||
self.max_body_size_mb > 0,
|
||||
"max_body_size_mb must be greater than 0"
|
||||
);
|
||||
ensure!(
|
||||
self.max_clients_per_vault > 0,
|
||||
"max_clients_per_vault must be greater than 0"
|
||||
);
|
||||
ensure!(
|
||||
self.broadcast_channel_capacity > 0,
|
||||
"broadcast_channel_capacity must be greater than 0"
|
||||
);
|
||||
ensure!(
|
||||
self.max_pending_websocket_connections > 0,
|
||||
"max_pending_websocket_connections must be greater than 0"
|
||||
);
|
||||
ensure!(
|
||||
self.max_clients_per_vault <= 10_000,
|
||||
"max_clients_per_vault must be at most 10000"
|
||||
);
|
||||
ensure!(
|
||||
self.broadcast_channel_capacity <= 1_000_000,
|
||||
"broadcast_channel_capacity must be at most 1000000"
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
fn default_host() -> String {
|
||||
|
|
@ -48,6 +108,11 @@ fn default_max_clients_per_vault() -> usize {
|
|||
DEFAULT_MAX_CLIENTS_PER_VAULT
|
||||
}
|
||||
|
||||
fn default_broadcast_channel_capacity() -> usize {
|
||||
debug!("Using default broadcast channel capacity: {DEFAULT_BROADCAST_CHANNEL_CAPACITY}");
|
||||
DEFAULT_BROADCAST_CHANNEL_CAPACITY
|
||||
}
|
||||
|
||||
fn default_response_timeout() -> Duration {
|
||||
debug!("Using default response timeout: {DEFAULT_RESPONSE_TIMEOUT_SECONDS:?}");
|
||||
DEFAULT_RESPONSE_TIMEOUT_SECONDS
|
||||
|
|
@ -60,3 +125,23 @@ fn default_mergeable_file_extensions() -> Vec<String> {
|
|||
.map(|s| (*s).to_owned())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn default_rate_limit_per_second() -> u64 {
|
||||
debug!("Using default rate limit per second: {DEFAULT_RATE_LIMIT_PER_SECOND}");
|
||||
DEFAULT_RATE_LIMIT_PER_SECOND
|
||||
}
|
||||
|
||||
fn default_allowed_origins() -> Vec<String> {
|
||||
debug!("Using default allowed origins: {DEFAULT_ALLOWED_ORIGINS:?}");
|
||||
DEFAULT_ALLOWED_ORIGINS
|
||||
.iter()
|
||||
.map(|s| (*s).to_owned())
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn default_max_pending_websocket_connections() -> usize {
|
||||
debug!(
|
||||
"Using default max pending WebSocket connections: {DEFAULT_MAX_PENDING_WS_CONNECTIONS}"
|
||||
);
|
||||
DEFAULT_MAX_PENDING_WS_CONNECTIONS
|
||||
}
|
||||
|
|
|
|||
|
|
@ -19,10 +19,19 @@ where
|
|||
let mut user_token_map = BiHashMap::new();
|
||||
for user in &users {
|
||||
if let Some(existing_name) = user_token_map.get_by_right(&user.token) {
|
||||
let redacted = if user.token.len() > 6 {
|
||||
format!(
|
||||
"{}...{}",
|
||||
&user.token[..3],
|
||||
&user.token[user.token.len() - 3..]
|
||||
)
|
||||
} else {
|
||||
"***".to_owned()
|
||||
};
|
||||
return Err(D::Error::custom(format!(
|
||||
"Duplicate user token found: `{}` for users `{}` and `{}`. User tokens must be \
|
||||
unique.",
|
||||
user.token, existing_name, user.name
|
||||
"Duplicate user token found: `{redacted}` for users `{}` and `{}`. User tokens \
|
||||
must be unique.",
|
||||
existing_name, user.name
|
||||
)));
|
||||
}
|
||||
|
||||
|
|
@ -41,10 +50,23 @@ where
|
|||
|
||||
impl UserConfig {
|
||||
pub fn get_user(&self, token: &str) -> Option<&User> {
|
||||
self.user_configs.iter().find(|u| u.token == token)
|
||||
self.user_configs
|
||||
.iter()
|
||||
.find(|u| constant_time_eq(u.token.as_bytes(), token.as_bytes()))
|
||||
}
|
||||
}
|
||||
|
||||
/// Constant-time byte comparison to prevent timing attacks on token lookups.
|
||||
fn constant_time_eq(a: &[u8], b: &[u8]) -> bool {
|
||||
if a.len() != b.len() {
|
||||
return false;
|
||||
}
|
||||
a.iter()
|
||||
.zip(b.iter())
|
||||
.fold(0u8, |acc, (x, y)| acc | (x ^ y))
|
||||
== 0
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize, Serialize, Clone)]
|
||||
pub struct User {
|
||||
pub name: String,
|
||||
|
|
|
|||
|
|
@ -13,12 +13,21 @@ pub const DEFAULT_PORT: u16 = 3000;
|
|||
pub const DEFAULT_MAX_BODY_SIZE_MB: usize = 4096;
|
||||
pub const DEFAULT_RESPONSE_TIMEOUT_SECONDS: Duration = Duration::from_mins(30);
|
||||
pub const DEFAULT_MAX_CLIENTS_PER_VAULT: usize = 256;
|
||||
pub const DEFAULT_BROADCAST_CHANNEL_CAPACITY: usize = 4096;
|
||||
pub const DEFAULT_MAX_PENDING_WS_CONNECTIONS: usize = 128;
|
||||
|
||||
pub const DEFAULT_LOG_DIRECTORY: &str = "logs";
|
||||
pub const DEFAULT_LOG_ROTATION_INTERVAL: Duration = Duration::from_hours(24);
|
||||
pub const IDLE_POOL_TIMEOUT: Duration = Duration::from_mins(5);
|
||||
pub const GRACEFUL_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(10);
|
||||
pub const DEFAULT_LOG_LEVEL: LogLevel = LogLevel::Info;
|
||||
|
||||
pub const DEFAULT_MERGEABLE_FILE_EXTENSIONS: &[&str] = &["md", "txt"];
|
||||
|
||||
/// 0 means rate limiting is disabled.
|
||||
pub const DEFAULT_RATE_LIMIT_PER_SECOND: u64 = 0;
|
||||
|
||||
/// Default: allow all origins.
|
||||
pub const DEFAULT_ALLOWED_ORIGINS: &[&str] = &["*"];
|
||||
|
||||
pub const SUPPORTED_API_VERSION: u32 = 3;
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ use axum::{
|
|||
http::StatusCode,
|
||||
response::{IntoResponse, Response},
|
||||
};
|
||||
use log::debug;
|
||||
use log::{debug, error, warn};
|
||||
use serde::Serialize;
|
||||
use thiserror::Error;
|
||||
use ts_rs::TS;
|
||||
|
|
@ -69,7 +69,19 @@ impl Display for SerializedError {
|
|||
|
||||
impl IntoResponse for SyncServerError {
|
||||
fn into_response(self) -> Response {
|
||||
let body = Json(self.serialize());
|
||||
let serialized = self.serialize();
|
||||
|
||||
match &self {
|
||||
Self::InitError(_) | Self::ServerError(_) => {
|
||||
error!("{serialized}");
|
||||
}
|
||||
Self::ClientError(_) | Self::NotFound(_) => {
|
||||
warn!("{serialized}");
|
||||
}
|
||||
Self::Unauthenticated(_) | Self::PermissionDeniedError(_) => {}
|
||||
}
|
||||
|
||||
let body = Json(serialized);
|
||||
|
||||
match self {
|
||||
Self::InitError(_) | Self::ServerError(_) => {
|
||||
|
|
|
|||
|
|
@ -41,7 +41,15 @@ async fn main() -> ExitCode {
|
|||
}
|
||||
};
|
||||
|
||||
let mut result = set_up_logging(&args, &config.logging);
|
||||
let mut result = config
|
||||
.server
|
||||
.validate()
|
||||
.context("Invalid server configuration")
|
||||
.map_err(init_error);
|
||||
|
||||
if result.is_ok() {
|
||||
result = set_up_logging(&args, &config.logging);
|
||||
}
|
||||
|
||||
if result.is_ok() {
|
||||
result = start_server(config).await;
|
||||
|
|
|
|||
|
|
@ -4,28 +4,31 @@ mod delete_document;
|
|||
mod device_id_header;
|
||||
mod fetch_document_version;
|
||||
mod fetch_document_version_content;
|
||||
mod fetch_document_versions;
|
||||
mod fetch_latest_document_version;
|
||||
mod fetch_latest_documents;
|
||||
mod fetch_vault_history;
|
||||
mod index;
|
||||
mod ping;
|
||||
mod rate_limit;
|
||||
mod requests;
|
||||
mod resolve_keys;
|
||||
mod responses;
|
||||
mod restore_document_version;
|
||||
mod update_document;
|
||||
mod websocket;
|
||||
|
||||
use anyhow::{Context as _, Result, anyhow};
|
||||
use anyhow::{Context as _, Result};
|
||||
use auth::auth_middleware;
|
||||
use axum::{
|
||||
Router,
|
||||
extract::{DefaultBodyLimit, Request},
|
||||
http::{self, HeaderValue, Method},
|
||||
middleware,
|
||||
response::IntoResponse,
|
||||
routing::{IntoMakeService, delete, get, post, put},
|
||||
};
|
||||
use device_id_header::DEVICE_ID_HEADER_NAME;
|
||||
use log::info;
|
||||
use log::{info, warn};
|
||||
use tokio::signal;
|
||||
use tower_http::{
|
||||
LatencyUnit,
|
||||
|
|
@ -42,7 +45,7 @@ use tracing::{Level, info_span};
|
|||
use crate::{
|
||||
app_state::AppState,
|
||||
config::{Config, server_config::ServerConfig},
|
||||
errors::{client_error, not_found_error},
|
||||
consts::GRACEFUL_SHUTDOWN_TIMEOUT,
|
||||
};
|
||||
|
||||
pub async fn create_server(config: Config) -> Result<()> {
|
||||
|
|
@ -52,26 +55,42 @@ pub async fn create_server(config: Config) -> Result<()> {
|
|||
|
||||
let server_config = app_state.config.server.clone();
|
||||
|
||||
let app = Router::new()
|
||||
let mut app = Router::new()
|
||||
.nest("/", get_authed_routes(app_state.clone()))
|
||||
.route("/", get(index::index))
|
||||
.route("/assets/*path", get(index::spa_assets))
|
||||
.route("/vaults/:vault_id/ping", get(ping::ping))
|
||||
.route("/vaults/:vault_id/ws", get(websocket::websocket_handler))
|
||||
.route("/vaults/:vault_id/ws", get(websocket::websocket_handler));
|
||||
|
||||
if app_state.config.server.dev_proxy_url.is_some() {
|
||||
info!(
|
||||
"Dev proxy enabled → {}",
|
||||
app_state.config.server.dev_proxy_url.as_deref().unwrap()
|
||||
);
|
||||
app = app.fallback(index::vite_proxy);
|
||||
}
|
||||
|
||||
let cors_layer = build_cors_layer(&server_config).context("Invalid CORS configuration")?;
|
||||
|
||||
if server_config.rate_limit_per_second > 0 {
|
||||
info!(
|
||||
"Rate limiting enabled: {} requests/second",
|
||||
server_config.rate_limit_per_second
|
||||
);
|
||||
let limiter = rate_limit::RateLimiter::new(server_config.rate_limit_per_second);
|
||||
app = app.layer(middleware::from_fn_with_state(
|
||||
limiter,
|
||||
rate_limit::rate_limit_middleware,
|
||||
));
|
||||
}
|
||||
|
||||
let app = app
|
||||
.layer(DefaultBodyLimit::disable())
|
||||
.layer(RequestBodyLimitLayer::new(
|
||||
app_state.config.server.max_body_size_mb * 1024 * 1024,
|
||||
))
|
||||
.layer(TimeoutLayer::new(server_config.response_timeout))
|
||||
.layer(
|
||||
CorsLayer::new()
|
||||
.allow_origin("*".parse::<HeaderValue>().expect("Failed to parse origin"))
|
||||
.allow_headers([
|
||||
http::header::CONTENT_TYPE,
|
||||
http::header::AUTHORIZATION,
|
||||
DEVICE_ID_HEADER_NAME.clone(),
|
||||
])
|
||||
.allow_methods([Method::GET, Method::POST, Method::PUT, Method::DELETE]),
|
||||
)
|
||||
.layer(cors_layer)
|
||||
.layer(
|
||||
TraceLayer::new_for_http()
|
||||
.make_span_with(|request: &Request<_>| {
|
||||
|
|
@ -92,13 +111,40 @@ pub async fn create_server(config: Config) -> Result<()> {
|
|||
.on_failure(DefaultOnFailure::new().level(Level::ERROR)),
|
||||
)
|
||||
.with_state(app_state)
|
||||
.fallback(handle_404)
|
||||
.fallback(handle_405)
|
||||
.into_make_service();
|
||||
|
||||
start_server(app, &server_config).await
|
||||
}
|
||||
|
||||
fn build_cors_layer(server_config: &ServerConfig) -> Result<CorsLayer> {
|
||||
let origins = &server_config.allowed_origins;
|
||||
|
||||
let cors = if origins.len() == 1 && origins[0] == "*" {
|
||||
info!("CORS: allowing all origins (wildcard)");
|
||||
let header: HeaderValue = "*"
|
||||
.parse()
|
||||
.context("Failed to parse wildcard CORS origin")?;
|
||||
CorsLayer::new().allow_origin(header)
|
||||
} else {
|
||||
let parsed: Vec<HeaderValue> = origins
|
||||
.iter()
|
||||
.map(|o| {
|
||||
o.parse::<HeaderValue>()
|
||||
.with_context(|| format!("Failed to parse CORS origin: `{o}`"))
|
||||
})
|
||||
.collect::<Result<Vec<_>>>()?;
|
||||
CorsLayer::new().allow_origin(parsed)
|
||||
};
|
||||
|
||||
Ok(cors
|
||||
.allow_headers([
|
||||
http::header::CONTENT_TYPE,
|
||||
http::header::AUTHORIZATION,
|
||||
DEVICE_ID_HEADER_NAME.clone(),
|
||||
])
|
||||
.allow_methods([Method::GET, Method::POST, Method::PUT, Method::DELETE]))
|
||||
}
|
||||
|
||||
fn get_authed_routes(app_state: AppState) -> Router<AppState> {
|
||||
Router::new()
|
||||
.route(
|
||||
|
|
@ -125,6 +171,10 @@ fn get_authed_routes(app_state: AppState) -> Router<AppState> {
|
|||
"/vaults/:vault_id/documents/:document_id/text",
|
||||
put(update_document::update_text),
|
||||
)
|
||||
.route(
|
||||
"/vaults/:vault_id/documents/:document_id/versions",
|
||||
get(fetch_document_versions::fetch_document_versions),
|
||||
)
|
||||
.route(
|
||||
"/vaults/:vault_id/documents/:document_id/versions/:vault_update_id",
|
||||
get(fetch_document_version::fetch_document_version),
|
||||
|
|
@ -137,6 +187,14 @@ fn get_authed_routes(app_state: AppState) -> Router<AppState> {
|
|||
"/vaults/:vault_id/documents/:document_id",
|
||||
delete(delete_document::delete_document),
|
||||
)
|
||||
.route(
|
||||
"/vaults/:vault_id/documents/:document_id/restore",
|
||||
post(restore_document_version::restore_document_version),
|
||||
)
|
||||
.route(
|
||||
"/vaults/:vault_id/history",
|
||||
get(fetch_vault_history::fetch_vault_history),
|
||||
)
|
||||
.layer(middleware::from_fn_with_state(app_state, auth_middleware))
|
||||
}
|
||||
|
||||
|
|
@ -153,26 +211,46 @@ async fn start_server(app: IntoMakeService<axum::Router>, config: &ServerConfig)
|
|||
.context("Failed to get local address")?
|
||||
);
|
||||
|
||||
axum::serve(listener, app)
|
||||
.with_graceful_shutdown(shutdown_signal())
|
||||
.tcp_nodelay(true)
|
||||
.await
|
||||
.context("Failed to start server")
|
||||
let (shutdown_tx, mut shutdown_rx) = tokio::sync::watch::channel(false);
|
||||
|
||||
let server = axum::serve(listener, app)
|
||||
.with_graceful_shutdown(async move {
|
||||
shutdown_signal().await;
|
||||
let _ = shutdown_tx.send(true);
|
||||
})
|
||||
.tcp_nodelay(true);
|
||||
|
||||
tokio::select! {
|
||||
result = server => result.context("Failed to start server"),
|
||||
() = async {
|
||||
let _ = shutdown_rx.changed().await;
|
||||
info!(
|
||||
"Shutdown signal received, waiting up to {}s for in-flight requests to complete...",
|
||||
GRACEFUL_SHUTDOWN_TIMEOUT.as_secs()
|
||||
);
|
||||
tokio::time::sleep(GRACEFUL_SHUTDOWN_TIMEOUT).await;
|
||||
warn!("Graceful shutdown timed out, forcing exit");
|
||||
} => Ok(()),
|
||||
}
|
||||
}
|
||||
|
||||
async fn shutdown_signal() {
|
||||
let ctrl_c = async {
|
||||
signal::ctrl_c()
|
||||
.await
|
||||
.expect("failed to install Ctrl+C handler");
|
||||
if let Err(e) = signal::ctrl_c().await {
|
||||
log::error!("Failed to install Ctrl+C handler: {e}");
|
||||
}
|
||||
};
|
||||
|
||||
#[cfg(unix)]
|
||||
let terminate = async {
|
||||
signal::unix::signal(signal::unix::SignalKind::terminate())
|
||||
.expect("failed to install signal handler")
|
||||
.recv()
|
||||
.await;
|
||||
match signal::unix::signal(signal::unix::SignalKind::terminate()) {
|
||||
Ok(mut signal) => {
|
||||
signal.recv().await;
|
||||
}
|
||||
Err(e) => {
|
||||
log::error!("Failed to install SIGTERM handler: {e}");
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
#[cfg(not(unix))]
|
||||
|
|
@ -183,11 +261,3 @@ async fn shutdown_signal() {
|
|||
() = terminate => {},
|
||||
}
|
||||
}
|
||||
|
||||
async fn handle_404() -> impl IntoResponse {
|
||||
not_found_error(anyhow!("Page not found"))
|
||||
}
|
||||
|
||||
async fn handle_405() -> impl IntoResponse {
|
||||
client_error(anyhow!("Method not allowed"))
|
||||
}
|
||||
|
|
|
|||
|
|
@ -9,7 +9,7 @@ use axum_extra::{
|
|||
TypedHeader,
|
||||
headers::{Authorization, authorization::Bearer},
|
||||
};
|
||||
use log::info;
|
||||
use log::{debug, info};
|
||||
|
||||
use crate::{
|
||||
app_state::{AppState, database::models::VaultId},
|
||||
|
|
@ -21,10 +21,12 @@ use crate::{
|
|||
pub async fn auth_middleware(
|
||||
State(state): State<AppState>,
|
||||
Path(path_params): Path<HashMap<String, String>>,
|
||||
TypedHeader(auth_header): TypedHeader<Authorization<Bearer>>,
|
||||
auth_header: Option<TypedHeader<Authorization<Bearer>>>,
|
||||
mut req: Request,
|
||||
next: Next,
|
||||
) -> Result<Response, SyncServerError> {
|
||||
let auth_header = auth_header
|
||||
.ok_or_else(|| unauthenticated_error(anyhow::anyhow!("Missing Authorization header")))?;
|
||||
let token = auth_header.token().trim();
|
||||
let vault_id = normalize_string(
|
||||
path_params
|
||||
|
|
@ -51,8 +53,8 @@ pub fn auth(state: &AppState, token: &str, vault_id: &VaultId) -> Result<User, S
|
|||
VaultAccess::AllowAccessToAll => true,
|
||||
VaultAccess::AllowList(AllowListedVaults { ref allowed }) => allowed.contains(vault_id),
|
||||
} {
|
||||
info!(
|
||||
"User `{}` is authenticated and is authorised to access to vault `{vault_id}`",
|
||||
debug!(
|
||||
"User `{}` is authenticated and is authorised to access vault `{vault_id}`",
|
||||
user.name
|
||||
);
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,3 @@
|
|||
use anyhow::Context;
|
||||
use axum::{
|
||||
Extension, Json,
|
||||
extract::{Path, State},
|
||||
|
|
@ -16,9 +15,13 @@ use crate::{
|
|||
},
|
||||
config::user_config::User,
|
||||
errors::{SyncServerError, server_error},
|
||||
server::{responses::DocumentUpdateResponse, update_document::merge_with_stored_version},
|
||||
server::{
|
||||
responses::DocumentUpdateResponse,
|
||||
update_document::{MergeInput, merge_with_stored_version},
|
||||
},
|
||||
utils::{
|
||||
find_first_available_path::find_first_available_path, normalize::normalize,
|
||||
dedup_paths::get_base_path, find_first_available_path::find_first_available_path,
|
||||
is_binary::is_binary, is_file_type_mergable::is_file_type_mergable, normalize::normalize,
|
||||
sanitize_path::sanitize_path,
|
||||
},
|
||||
};
|
||||
|
|
@ -32,7 +35,11 @@ pub struct CreateDocumentPathParams {
|
|||
/// Create a new document in case a document with the same doesn't exist
|
||||
/// already. If a document with the same path exists, a new version is created
|
||||
/// with their content merged.
|
||||
///
|
||||
/// Text content must be UTF-8 encoded. Clients are responsible for
|
||||
/// transcoding other encodings (e.g. UTF-16) to UTF-8 before sending.
|
||||
#[axum::debug_handler]
|
||||
#[allow(clippy::too_many_lines)]
|
||||
pub async fn create_document(
|
||||
Path(CreateDocumentPathParams { vault_id }): Path<CreateDocumentPathParams>,
|
||||
Extension(user): Extension<User>,
|
||||
|
|
@ -51,62 +58,133 @@ pub async fn create_document(
|
|||
if let Some(ref idempotency_key) = request.idempotency_key {
|
||||
let existing = state
|
||||
.database
|
||||
.get_document_by_idempotency_key(&vault_id, idempotency_key, Some(&mut transaction))
|
||||
.get_document_by_idempotency_key(&vault_id, idempotency_key, Some(&mut *transaction))
|
||||
.await
|
||||
.map_err(server_error)?;
|
||||
if let Some(existing) = existing {
|
||||
info!(
|
||||
"Found existing document with idempotency key `{idempotency_key}`, returning existing document"
|
||||
);
|
||||
transaction
|
||||
.rollback()
|
||||
.await
|
||||
.context("Failed to roll back transaction")
|
||||
.map_err(server_error)?;
|
||||
return Ok(Json(DocumentUpdateResponse::FastForwardUpdate(
|
||||
existing.into(),
|
||||
)));
|
||||
if existing.is_deleted {
|
||||
// The document was created (storing the key) and later deleted.
|
||||
// Don't return the deleted version — it would cause the client
|
||||
// to delete its local file. Instead, fall through to normal
|
||||
// create so the client's content is preserved as a new document.
|
||||
// The unique index excludes deleted rows (WHERE is_deleted = 0),
|
||||
// so keeping the key does NOT cause a constraint violation —
|
||||
// the new non-deleted version can safely reuse the same key.
|
||||
info!(
|
||||
"Idempotency key `{idempotency_key}` matches a deleted document, ignoring and creating fresh"
|
||||
);
|
||||
} else {
|
||||
// Return the LATEST version of the document, not the version
|
||||
// that originally stored the key. The document may have been
|
||||
// modified by other clients since the key was stored, and
|
||||
// returning a stale version would cause the client to cache
|
||||
// incorrect content, breaking subsequent diffs.
|
||||
let latest = state
|
||||
.database
|
||||
.get_latest_document(&vault_id, &existing.document_id, Some(&mut *transaction))
|
||||
.await
|
||||
.map_err(server_error)?
|
||||
.unwrap_or(existing);
|
||||
info!(
|
||||
"Found existing document with idempotency key `{idempotency_key}`, returning latest version"
|
||||
);
|
||||
transaction.rollback().await.map_err(server_error)?;
|
||||
return Ok(Json(DocumentUpdateResponse::FastForwardUpdate(
|
||||
latest.into(),
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let sanitized_relative_path = sanitize_path(&request.relative_path);
|
||||
|
||||
if sanitized_relative_path.is_empty() {
|
||||
transaction.rollback().await.map_err(server_error)?;
|
||||
return Err(crate::errors::client_error(anyhow::anyhow!(
|
||||
"Relative path is empty after sanitization"
|
||||
)));
|
||||
}
|
||||
|
||||
let new_content = request.content.contents.to_vec();
|
||||
|
||||
let latest_version = state
|
||||
.database
|
||||
.get_latest_non_deleted_document_by_path(
|
||||
&vault_id,
|
||||
&sanitized_relative_path,
|
||||
Some(&mut transaction),
|
||||
Some(&mut *transaction),
|
||||
)
|
||||
.await
|
||||
.map_err(server_error)?;
|
||||
|
||||
if let Some(latest_version) = latest_version {
|
||||
info!(
|
||||
"Document already exists at new location: `{sanitized_relative_path}` when trying to create it in vault `{vault_id}`, merging into existing document"
|
||||
);
|
||||
let is_mergeable_text = is_file_type_mergable(
|
||||
&sanitized_relative_path,
|
||||
&state.config.server.mergeable_file_extensions,
|
||||
) && !is_binary(&latest_version.content)
|
||||
&& !is_binary(&new_content);
|
||||
|
||||
return merge_with_stored_version(
|
||||
&sanitized_relative_path,
|
||||
&latest_version.content.clone(),
|
||||
latest_version,
|
||||
vault_id,
|
||||
user,
|
||||
device_id,
|
||||
state,
|
||||
&sanitized_relative_path,
|
||||
request.content.contents.to_vec(),
|
||||
transaction,
|
||||
request.idempotency_key,
|
||||
)
|
||||
.await;
|
||||
if is_mergeable_text || new_content == latest_version.content {
|
||||
return merge_with_stored_version(
|
||||
MergeInput {
|
||||
parent_content: &[],
|
||||
new_content,
|
||||
idempotency_key: request.idempotency_key,
|
||||
},
|
||||
latest_version,
|
||||
vault_id,
|
||||
user,
|
||||
device_id,
|
||||
state,
|
||||
transaction,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
|
||||
// For non-mergeable (binary) files with different content, don't
|
||||
// merge — create a separate document at a deconflicted path so
|
||||
// neither client's data is silently overwritten.
|
||||
}
|
||||
|
||||
// For creates at deconflicted paths (e.g., "file (2).bin"), the client's
|
||||
// ensureClearPath renamed a local file before uploading. Check if the
|
||||
// base path (e.g., "file.bin") has a document with identical content.
|
||||
// If so, merge with it instead of creating a duplicate document.
|
||||
let base_path = get_base_path(&sanitized_relative_path);
|
||||
if base_path != sanitized_relative_path {
|
||||
let base_doc = state
|
||||
.database
|
||||
.get_latest_non_deleted_document_by_path(&vault_id, &base_path, Some(&mut *transaction))
|
||||
.await
|
||||
.map_err(server_error)?;
|
||||
if let Some(base_doc) = base_doc
|
||||
&& new_content == base_doc.content
|
||||
{
|
||||
info!(
|
||||
"Create at deconflicted path `{sanitized_relative_path}` has identical content to document at base path `{base_path}`, merging"
|
||||
);
|
||||
return merge_with_stored_version(
|
||||
MergeInput {
|
||||
parent_content: &[],
|
||||
new_content,
|
||||
idempotency_key: request.idempotency_key,
|
||||
},
|
||||
base_doc,
|
||||
vault_id,
|
||||
user,
|
||||
device_id,
|
||||
state,
|
||||
transaction,
|
||||
)
|
||||
.await;
|
||||
}
|
||||
}
|
||||
|
||||
let document_id = uuid::Uuid::new_v4();
|
||||
|
||||
let last_update_id = state
|
||||
.database
|
||||
.get_max_update_id_in_vault(&vault_id, Some(&mut transaction))
|
||||
.get_max_update_id_in_vault(&vault_id, Some(&mut *transaction))
|
||||
.await
|
||||
.map_err(server_error)?;
|
||||
|
||||
|
|
@ -129,7 +207,7 @@ pub async fn create_document(
|
|||
vault_update_id: last_update_id + 1,
|
||||
document_id,
|
||||
relative_path: deduped_path,
|
||||
content: request.content.contents.to_vec(),
|
||||
content: new_content,
|
||||
updated_date: chrono::Utc::now(),
|
||||
is_deleted: false,
|
||||
user_id: user.name,
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
use anyhow::Context;
|
||||
use anyhow::{Context, anyhow};
|
||||
use axum::{
|
||||
Extension, Json,
|
||||
extract::{Path, State},
|
||||
|
|
@ -16,8 +16,8 @@ use crate::{
|
|||
},
|
||||
},
|
||||
config::user_config::User,
|
||||
errors::{SyncServerError, server_error},
|
||||
utils::{normalize::normalize, sanitize_path::sanitize_path},
|
||||
errors::{SyncServerError, not_found_error, server_error},
|
||||
utils::normalize::normalize,
|
||||
};
|
||||
|
||||
#[derive(Deserialize)]
|
||||
|
|
@ -37,7 +37,7 @@ pub async fn delete_document(
|
|||
Extension(user): Extension<User>,
|
||||
TypedHeader(device_id): TypedHeader<DeviceIdHeader>,
|
||||
State(state): State<AppState>,
|
||||
Json(request): Json<DeleteDocumentVersion>,
|
||||
Json(_request): Json<DeleteDocumentVersion>,
|
||||
) -> Result<Json<DocumentVersionWithoutContent>, SyncServerError> {
|
||||
debug!("Deleting document `{document_id}` in vault `{vault_id}`");
|
||||
|
||||
|
|
@ -59,6 +59,18 @@ pub async fn delete_document(
|
|||
.await
|
||||
.map_err(server_error)?;
|
||||
|
||||
if latest_version.is_none() {
|
||||
transaction
|
||||
.rollback()
|
||||
.await
|
||||
.context("Failed to roll back transaction")
|
||||
.map_err(server_error)?;
|
||||
|
||||
return Err(not_found_error(anyhow!(
|
||||
"Document `{document_id}` not found in vault `{vault_id}`"
|
||||
)));
|
||||
}
|
||||
|
||||
if let Some(latest_version) = &latest_version
|
||||
&& latest_version.is_deleted
|
||||
{
|
||||
|
|
@ -72,13 +84,14 @@ pub async fn delete_document(
|
|||
return Ok(Json(latest_version.clone().into()));
|
||||
}
|
||||
|
||||
let latest_content = latest_version.map_or_else(Vec::new, |version| version.content); // in case the document has never existed before deleting it
|
||||
// latest_version is guaranteed to be Some and not deleted at this point
|
||||
let latest_version = latest_version.expect("checked above: not None and not deleted");
|
||||
|
||||
let new_version = StoredDocumentVersion {
|
||||
vault_update_id: last_update_id + 1,
|
||||
document_id,
|
||||
relative_path: sanitize_path(&request.relative_path),
|
||||
content: latest_content, // copy the content from the latest version
|
||||
relative_path: latest_version.relative_path,
|
||||
content: latest_version.content,
|
||||
updated_date: chrono::Utc::now(),
|
||||
is_deleted: true,
|
||||
user_id: user.name,
|
||||
|
|
|
|||
|
|
@ -16,20 +16,31 @@ impl Header for DeviceIdHeader {
|
|||
{
|
||||
let value = values.next().ok_or_else(headers::Error::invalid)?;
|
||||
|
||||
Ok(DeviceIdHeader(
|
||||
value
|
||||
.to_str()
|
||||
.map_err(|_| headers::Error::invalid())?
|
||||
.to_owned(),
|
||||
))
|
||||
let s = value.to_str().map_err(|_| headers::Error::invalid())?;
|
||||
|
||||
if s.is_empty() || s.len() > 256 {
|
||||
return Err(headers::Error::invalid());
|
||||
}
|
||||
|
||||
// Only allow safe characters to prevent log injection and similar attacks.
|
||||
// Covers UUIDs, user-agent strings like "vault-link/1.0 (12345; linux)",
|
||||
// and human-readable device names.
|
||||
if !s
|
||||
.chars()
|
||||
.all(|c| c.is_ascii_alphanumeric() || "-_./ ();:@+,".contains(c))
|
||||
{
|
||||
return Err(headers::Error::invalid());
|
||||
}
|
||||
|
||||
Ok(DeviceIdHeader(s.to_owned()))
|
||||
}
|
||||
|
||||
fn encode<E>(&self, values: &mut E)
|
||||
where
|
||||
E: Extend<HeaderValue>,
|
||||
{
|
||||
let value = HeaderValue::from_static(Box::leak(self.0.clone().into_boxed_str()));
|
||||
|
||||
values.extend(std::iter::once(value));
|
||||
if let Ok(value) = HeaderValue::from_str(&self.0) {
|
||||
values.extend(std::iter::once(value));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ use crate::{
|
|||
AppState,
|
||||
database::models::{DocumentId, DocumentVersion, VaultId, VaultUpdateId},
|
||||
},
|
||||
errors::{SyncServerError, not_found_error, server_error},
|
||||
errors::{SyncServerError, client_error, not_found_error, server_error},
|
||||
utils::normalize::normalize,
|
||||
};
|
||||
|
||||
|
|
@ -52,7 +52,7 @@ pub async fn fetch_document_version(
|
|||
)?;
|
||||
|
||||
if result.document_id != document_id {
|
||||
return Err(not_found_error(anyhow!(
|
||||
return Err(client_error(anyhow!(
|
||||
"Document with document id `{document_id}` does not have a version with id \
|
||||
`{vault_update_id}`",
|
||||
)));
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ use crate::{
|
|||
AppState,
|
||||
database::models::{DocumentId, VaultId, VaultUpdateId},
|
||||
},
|
||||
errors::{SyncServerError, not_found_error, server_error},
|
||||
errors::{SyncServerError, client_error, not_found_error, server_error},
|
||||
utils::normalize::normalize,
|
||||
};
|
||||
|
||||
|
|
@ -52,7 +52,7 @@ pub async fn fetch_document_version_content(
|
|||
)?;
|
||||
|
||||
if result.document_id != document_id {
|
||||
return Err(not_found_error(anyhow!(
|
||||
return Err(client_error(anyhow!(
|
||||
"Document with document id `{document_id}` does not have a version with id \
|
||||
`{vault_update_id}`",
|
||||
)));
|
||||
|
|
|
|||
42
sync-server/src/server/fetch_document_versions.rs
Normal file
42
sync-server/src/server/fetch_document_versions.rs
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
use axum::{
|
||||
Json,
|
||||
extract::{Path, State},
|
||||
};
|
||||
use log::debug;
|
||||
use serde::Deserialize;
|
||||
|
||||
use crate::{
|
||||
app_state::{
|
||||
AppState,
|
||||
database::models::{DocumentId, DocumentVersionWithoutContent, VaultId},
|
||||
},
|
||||
errors::{SyncServerError, server_error},
|
||||
utils::normalize::normalize,
|
||||
};
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct FetchDocumentVersionsPathParams {
|
||||
#[serde(deserialize_with = "normalize")]
|
||||
vault_id: VaultId,
|
||||
|
||||
document_id: DocumentId,
|
||||
}
|
||||
|
||||
#[axum::debug_handler]
|
||||
pub async fn fetch_document_versions(
|
||||
Path(FetchDocumentVersionsPathParams {
|
||||
vault_id,
|
||||
document_id,
|
||||
}): Path<FetchDocumentVersionsPathParams>,
|
||||
State(state): State<AppState>,
|
||||
) -> Result<Json<Vec<DocumentVersionWithoutContent>>, SyncServerError> {
|
||||
debug!("Fetching all versions for document `{document_id}` in vault `{vault_id}`");
|
||||
|
||||
let versions = state
|
||||
.database
|
||||
.get_document_versions(&vault_id, &document_id, None)
|
||||
.await
|
||||
.map_err(server_error)?;
|
||||
|
||||
Ok(Json(versions))
|
||||
}
|
||||
70
sync-server/src/server/fetch_vault_history.rs
Normal file
70
sync-server/src/server/fetch_vault_history.rs
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
use axum::{
|
||||
Json,
|
||||
extract::{Path, Query, State},
|
||||
};
|
||||
use log::debug;
|
||||
use serde::Deserialize;
|
||||
|
||||
use super::responses::VaultHistoryResponse;
|
||||
use crate::{
|
||||
app_state::{
|
||||
AppState,
|
||||
database::models::{VaultId, VaultUpdateId},
|
||||
},
|
||||
errors::{SyncServerError, client_error, server_error},
|
||||
utils::normalize::normalize,
|
||||
};
|
||||
|
||||
const DEFAULT_LIMIT: i64 = 50;
|
||||
const MAX_LIMIT: i64 = 500;
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct FetchVaultHistoryPathParams {
|
||||
#[serde(deserialize_with = "normalize")]
|
||||
vault_id: VaultId,
|
||||
}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
pub struct QueryParams {
|
||||
limit: Option<i64>,
|
||||
before_update_id: Option<VaultUpdateId>,
|
||||
}
|
||||
|
||||
#[axum::debug_handler]
|
||||
pub async fn fetch_vault_history(
|
||||
Path(FetchVaultHistoryPathParams { vault_id }): Path<FetchVaultHistoryPathParams>,
|
||||
Query(QueryParams {
|
||||
limit,
|
||||
before_update_id,
|
||||
}): Query<QueryParams>,
|
||||
State(state): State<AppState>,
|
||||
) -> Result<Json<VaultHistoryResponse>, SyncServerError> {
|
||||
if let Some(id) = before_update_id
|
||||
&& id <= 0
|
||||
{
|
||||
return Err(client_error(anyhow::anyhow!(
|
||||
"before_update_id must be a positive integer"
|
||||
)));
|
||||
}
|
||||
|
||||
let limit = limit.unwrap_or(DEFAULT_LIMIT).clamp(1, MAX_LIMIT);
|
||||
|
||||
debug!(
|
||||
"Fetching vault history for vault `{vault_id}` (limit={limit}, before={before_update_id:?})"
|
||||
);
|
||||
|
||||
// Fetch one extra row to determine if there are more results
|
||||
let mut versions = state
|
||||
.database
|
||||
.get_vault_history(&vault_id, limit + 1, before_update_id, None)
|
||||
.await
|
||||
.map_err(server_error)?;
|
||||
|
||||
#[allow(clippy::cast_sign_loss)] // limit is clamped to [1, 500] above
|
||||
let has_more = versions.len() > limit as usize;
|
||||
if has_more {
|
||||
versions.pop();
|
||||
}
|
||||
|
||||
Ok(Json(VaultHistoryResponse { versions, has_more }))
|
||||
}
|
||||
|
|
@ -1,7 +1,146 @@
|
|||
use axum::response::{Html, IntoResponse};
|
||||
use axum::{
|
||||
body::Body,
|
||||
extract::{Path, State},
|
||||
http::{StatusCode, header},
|
||||
response::{Html, IntoResponse, Response},
|
||||
};
|
||||
use log::warn;
|
||||
use rust_embed::Embed;
|
||||
|
||||
pub async fn index() -> impl IntoResponse {
|
||||
const HTML_CONTENT: &str = include_str!("./assets/index.html");
|
||||
let html_content = HTML_CONTENT;
|
||||
Html(html_content)
|
||||
use crate::app_state::AppState;
|
||||
|
||||
#[derive(Embed)]
|
||||
#[folder = "../frontend/history-ui/dist/"]
|
||||
struct HistoryUiAssets;
|
||||
|
||||
pub async fn index(State(state): State<AppState>) -> impl IntoResponse {
|
||||
if let Some(proxy_url) = &state.config.server.dev_proxy_url {
|
||||
let response = proxy_request(proxy_url, "/").await;
|
||||
if response.status().is_success() {
|
||||
return response;
|
||||
}
|
||||
}
|
||||
|
||||
if let Some(content) = HistoryUiAssets::get("index.html") {
|
||||
Html(
|
||||
std::str::from_utf8(content.data.as_ref())
|
||||
.inspect_err(|e| warn!("Embedded index.html is not valid UTF-8: {e}"))
|
||||
.unwrap_or("<h1>VaultLink</h1>")
|
||||
.to_owned(),
|
||||
)
|
||||
.into_response()
|
||||
} else {
|
||||
warn!("No embedded index.html found — history UI may not have been built");
|
||||
Html("<h1>VaultLink server</h1>".to_owned()).into_response()
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn spa_assets(
|
||||
State(state): State<AppState>,
|
||||
Path(path): Path<String>,
|
||||
) -> impl IntoResponse {
|
||||
if let Some(proxy_url) = &state.config.server.dev_proxy_url {
|
||||
let response = proxy_request(proxy_url, &format!("/assets/{path}")).await;
|
||||
if response.status().is_success() {
|
||||
return response;
|
||||
}
|
||||
}
|
||||
|
||||
// The route is /assets/*path so path is relative to assets/.
|
||||
// The embedded files include the assets/ prefix from the dist directory.
|
||||
let full_path = format!("assets/{path}");
|
||||
if let Some(content) = HistoryUiAssets::get(&full_path) {
|
||||
let mime = mime_guess::from_path(&full_path).first_or_octet_stream();
|
||||
return Response::builder()
|
||||
.status(StatusCode::OK)
|
||||
.header(header::CONTENT_TYPE, mime.as_ref())
|
||||
.body(Body::from(content.data.to_vec()))
|
||||
.unwrap_or_else(|_| {
|
||||
Response::builder()
|
||||
.status(StatusCode::INTERNAL_SERVER_ERROR)
|
||||
.body(Body::empty())
|
||||
.unwrap_or_else(|_| Response::new(Body::empty()))
|
||||
});
|
||||
}
|
||||
|
||||
// Asset paths must match an embedded file — no SPA fallback.
|
||||
// Serving index.html here would return 200 with text/html for missing
|
||||
// .css/.js files, causing the browser to silently ignore the content.
|
||||
Response::builder()
|
||||
.status(StatusCode::NOT_FOUND)
|
||||
.body(Body::from("Not found"))
|
||||
.unwrap_or_else(|_| Response::new(Body::from("Not found")))
|
||||
}
|
||||
|
||||
/// Proxies unmatched paths to the Vite dev server for HMR support
|
||||
/// (`@vite/client`, `src/`, etc.).
|
||||
pub async fn vite_proxy(
|
||||
State(state): State<AppState>,
|
||||
request: axum::extract::Request,
|
||||
) -> impl IntoResponse {
|
||||
let proxy_url = state.config.server.dev_proxy_url.as_deref().unwrap_or("");
|
||||
let response = proxy_request(proxy_url, request.uri().path()).await;
|
||||
if !response.status().is_success() {
|
||||
return Response::builder()
|
||||
.status(StatusCode::NOT_FOUND)
|
||||
.body(Body::from("Not found"))
|
||||
.unwrap_or_else(|_| Response::new(Body::from("Not found")));
|
||||
}
|
||||
response
|
||||
}
|
||||
|
||||
/// SPA fallback for production: serves index.html for client-side routes
|
||||
/// (e.g. `/documents/123`). Only used when the dev proxy is disabled.
|
||||
pub async fn spa_fallback() -> impl IntoResponse {
|
||||
match HistoryUiAssets::get("index.html") {
|
||||
Some(content) => Response::builder()
|
||||
.status(StatusCode::OK)
|
||||
.header(header::CONTENT_TYPE, "text/html")
|
||||
.body(Body::from(content.data.to_vec()))
|
||||
.unwrap_or_else(|_| {
|
||||
Response::builder()
|
||||
.status(StatusCode::INTERNAL_SERVER_ERROR)
|
||||
.body(Body::empty())
|
||||
.unwrap_or_else(|_| Response::new(Body::empty()))
|
||||
}),
|
||||
None => Response::builder()
|
||||
.status(StatusCode::NOT_FOUND)
|
||||
.body(Body::from("Not found"))
|
||||
.unwrap_or_else(|_| Response::new(Body::from("Not found"))),
|
||||
}
|
||||
}
|
||||
|
||||
static DEV_PROXY_CLIENT: std::sync::LazyLock<reqwest::Client> = std::sync::LazyLock::new(|| {
|
||||
reqwest::Client::builder()
|
||||
.timeout(std::time::Duration::from_secs(5))
|
||||
.build()
|
||||
.unwrap_or_default()
|
||||
});
|
||||
|
||||
async fn proxy_request(proxy_url: &str, path: &str) -> Response {
|
||||
let url = format!("{proxy_url}{path}");
|
||||
match DEV_PROXY_CLIENT.get(&url).send().await {
|
||||
Ok(resp) => {
|
||||
let status =
|
||||
StatusCode::from_u16(resp.status().as_u16()).unwrap_or(StatusCode::BAD_GATEWAY);
|
||||
let mut builder = Response::builder().status(status);
|
||||
for (name, value) in resp.headers() {
|
||||
builder = builder.header(name.clone(), value.clone());
|
||||
}
|
||||
let bytes = resp.bytes().await.unwrap_or_default();
|
||||
builder.body(Body::from(bytes)).unwrap_or_else(|_| {
|
||||
Response::builder()
|
||||
.status(StatusCode::BAD_GATEWAY)
|
||||
.body(Body::empty())
|
||||
.unwrap_or_else(|_| Response::new(Body::empty()))
|
||||
})
|
||||
}
|
||||
Err(_) => {
|
||||
// Dev server not running — fall back to embedded assets
|
||||
Response::builder()
|
||||
.status(StatusCode::BAD_GATEWAY)
|
||||
.body(Body::empty())
|
||||
.unwrap_or_else(|_| Response::new(Body::empty()))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
72
sync-server/src/server/rate_limit.rs
Normal file
72
sync-server/src/server/rate_limit.rs
Normal file
|
|
@ -0,0 +1,72 @@
|
|||
use std::sync::{
|
||||
Arc,
|
||||
atomic::{AtomicU64, Ordering},
|
||||
};
|
||||
|
||||
use axum::{extract::Request, http::StatusCode, middleware::Next, response::Response};
|
||||
|
||||
/// Simple token-bucket rate limiter that refills every second.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct RateLimiter {
|
||||
inner: Arc<TokenBucket>,
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct TokenBucket {
|
||||
tokens: AtomicU64,
|
||||
max_tokens: u64,
|
||||
}
|
||||
|
||||
impl RateLimiter {
|
||||
/// Create a new rate limiter. Spawns a background task that refills tokens
|
||||
/// every second.
|
||||
///
|
||||
/// # Panics
|
||||
///
|
||||
/// Panics if `max_per_second` is 0.
|
||||
pub fn new(max_per_second: u64) -> Self {
|
||||
assert!(
|
||||
max_per_second > 0,
|
||||
"max_per_second must be > 0 (use 0 in config to disable rate limiting entirely)"
|
||||
);
|
||||
|
||||
let bucket = Arc::new(TokenBucket {
|
||||
tokens: AtomicU64::new(max_per_second),
|
||||
max_tokens: max_per_second,
|
||||
});
|
||||
|
||||
let bucket_clone = bucket.clone();
|
||||
tokio::spawn(async move {
|
||||
let mut interval = tokio::time::interval(std::time::Duration::from_secs(1));
|
||||
loop {
|
||||
interval.tick().await;
|
||||
bucket_clone
|
||||
.tokens
|
||||
.store(bucket_clone.max_tokens, Ordering::Release);
|
||||
}
|
||||
});
|
||||
|
||||
Self { inner: bucket }
|
||||
}
|
||||
|
||||
fn try_acquire(&self) -> bool {
|
||||
self.inner
|
||||
.tokens
|
||||
.fetch_update(Ordering::AcqRel, Ordering::Acquire, |current| {
|
||||
if current > 0 { Some(current - 1) } else { None }
|
||||
})
|
||||
.is_ok()
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn rate_limit_middleware(
|
||||
axum::extract::State(limiter): axum::extract::State<RateLimiter>,
|
||||
req: Request,
|
||||
next: Next,
|
||||
) -> Result<Response, StatusCode> {
|
||||
if limiter.try_acquire() {
|
||||
Ok(next.run(req).await)
|
||||
} else {
|
||||
Err(StatusCode::TOO_MANY_REQUESTS)
|
||||
}
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Add table
Add a link
Reference in a new issue