Fix syncing when network latency is present #4

Merged

schmelczer merged 58 commits from asch/fixes into master

2025-03-16 20:13:50 +00:00

schmelczer commented

2025-03-02 10:26:42 +00:00

(Migrated from github.com)

Big PR with big changes to fix concurrency issues

I wasn't sure going down this route would make sense but the changes proved to be a big success so I'm using this PR to clean up the diff.

Add E2E tests for verifying distributed correctness
Add CI task to run E2E tests
Run Docker build on all commits, not just releases
Make reconcile insertion order consistent
Deduplicate the overlap between subsequent competing inserts
Tokenize text by keeping the leading whitespace instead of the trailing to avoid joining words
Ensure that the insertion order is consistent across platforms
Use a separate DB per vault in the server to enable realistic parallel testing without contention from write transactions
Store documentIds as strings instead of binary blobs in DB
Stop taking createdDate in requests and use server time instead
Allow setting the documentId in the create endpoint
Return the deleted document in the delete endpoint
Publish SyncClient for both node and web targets
Improve syncing logic by handling create-after-deletes
Fix many concurrency bugs that the E2E test had uncovered

General clean-up

Delete fuzz project
Enable more clippy rules and fix them
Extract handling the sync on/off state so that it's not scattered
Add rust-toolchain.toml and remove rust installation instructions

# Big PR with big changes to fix concurrency issues I wasn't sure going down this route would make sense but the changes proved to be a big success so I'm using this PR to clean up the diff. - Add E2E tests for verifying distributed correctness - Add CI task to run E2E tests - Run Docker build on all commits, not just releases - Make reconcile insertion order consistent - Deduplicate the overlap between subsequent competing inserts - Tokenize text by keeping the leading whitespace instead of the trailing to avoid joining words - Ensure that the insertion order is consistent across platforms - Use a separate DB per vault in the server to enable realistic parallel testing without contention from write transactions - Store documentIds as strings instead of binary blobs in DB - Stop taking createdDate in requests and use server time instead - Allow setting the documentId in the create endpoint - Return the deleted document in the delete endpoint - Publish SyncClient for both node and web targets - Improve syncing logic by handling create-after-deletes - Fix many concurrency bugs that the E2E test had uncovered ## General clean-up - Delete fuzz project - Enable more clippy rules and fix them - Extract handling the sync on/off state so that it's not scattered - Add `rust-toolchain.toml` and remove rust installation instructions

schmelczer (Migrated from github.com) reviewed 2025-03-16 17:53:14 +00:00

.github/workflows/check.yml

schmelczer (Migrated from github.com) commented

2025-03-16 17:53:14 +00:00

It required node 22 but we only had 18