Generate docs

README.md

@@ -2,77 +2,180 @@
 
 > `diff3` but with automatic conflict resolution.
 
+[![Check](https://github.com/schmelczer/reconcile/actions/workflows/check.yml/badge.svg)](https://github.com/schmelczer/reconcile/actions/workflows/check.yml)
+[![Publish to GitHub Pages](https://github.com/schmelczer/reconcile/actions/workflows/gh-pages.yml/badge.svg)](https://github.com/schmelczer/reconcile/actions/workflows/gh-pages.yml)
+
+Reconcile is a Rust and JavaScript (through WebAssembly) library for merging text without user intervention. It automatically resolves conflicts that would typically require manual intervention in traditional 3-way merge tools.
+
+```rust
+use reconcile::{reconcile, BuiltinTokenizer};
+
+let parent = "Merging text is hard!";
+let left = "Merging text is easy!";
+let right = "With reconcile, merging documents is hard!";
+
+let deconflicted = reconcile(parent, &left.into(), &right.into(), &*BuiltinTokenizer::Word);
+assert_eq!(deconflicted.apply().text(), "With reconcile, merging documents is easy!");
+```
+
 ## Features
 
--   Conflict-free output (no more git conflict markers like in )
+-   **Conflict-free output** - No more git conflict markers in the result
--   Support for updating cursor/selection positions
+-   **Cursor/selection position tracking** - Automatically updates cursor positions during merging
--   Pluggable tokenizer
+-   **Pluggable tokenizer** - Choose between word-level, character-level, or custom tokenization
--   Full UTF-8 support
+-   **Full UTF-8 support** - Handles Unicode text correctly
--   WASM
+-   **WebAssembly support** - Use from JavaScript/TypeScript applications
+
+## Quick Start
+
+### Rust
+
+Add to your `Cargo.toml`:
+```toml
+[dependencies]
+reconcile = "0.4"
+```
+
+```rust
+use reconcile::{reconcile, BuiltinTokenizer};
+
+let parent = "Hello world";
+let left = "Hello beautiful world";
+let right = "Hi world";
+
+let result = reconcile(parent, &left.into(), &right.into(), &*BuiltinTokenizer::Word);
+assert_eq!(result.apply().text(), "Hi beautiful world");
+```
+
+### JavaScript/TypeScript
+
+```bash
+npm install reconcile
+```
+
+```javascript
+import { init, reconcile } from 'reconcile';
+
+// Initialize the WASM module (required before first use)
+await init();
+
+const parent = "Hello world";
+const left = "Hello beautiful world";
+const right = "Hi world";
+
+const result = reconcile(parent, left, right);
+console.log(result.text); // "Hi beautiful world"
+```
+
+## API
+
+### Tokenizers
+
+Reconcile supports different tokenization strategies:
+
+- **Word tokenizer** (`BuiltinTokenizer::Word`): Splits text into words (default, recommended for most use cases)
+- **Character tokenizer** (`BuiltinTokenizer::Character`): Splits text into individual characters (fine-grained merging)
+- **Custom tokenizer**: Implement your own tokenization logic
+
+### Cursor Tracking
+
+Reconcile can automatically update cursor and selection positions during merging:
+
+```javascript
+const result = reconcile(
+    "Hello world",
+    {
+        text: "Hello beautiful world",
+        cursors: [{ id: 1, position: 6 }]  // After "Hello "
+    },
+    {
+        text: "Hi world",
+        cursors: [{ id: 2, position: 0 }]   // At beginning
+    }
+);
+
+// Result includes updated cursor positions
+console.log(result.cursors); // [{ id: 1, position: 3 }, { id: 2, position: 0 }]
+```
+
+### History Tracking
+
+Use `reconcileWithHistory` to get detailed information about the merge process:
+
+```javascript
+const result = reconcileWithHistory(parent, left, right);
+console.log(result.history); // Array of spans with their origins
+```
+
+## Algorithm
+
+The algorithm starts similarly to `diff3`. Its inputs are a **parent** document and two conflicting versions: `left` and `right` which have been created from the parent through any series of concurrent edits.
+
+1. **Diff calculation**: First, 2-way diffs of (parent & left) and (parent & right) are computed using Myers' algorithm
+2. **Tokenization**: The text is split into tokens (words, characters, etc.) for granular merging
+3. **Operation transformation**: The resulting edits are weaved together using operational transformation principles, ensuring no changes are lost
+4. **Conflict resolution**: Unlike traditional 3-way merge tools, Reconcile automatically resolves conflicts without producing conflict markers
+
+The key insight is that both insertions and deletions are preserved: if either side inserted text, it appears in the result; if either side deleted text, the deletion is applied, but insertions into deleted regions are still preserved.
 
 ## Motivation
 
 Sometimes documents get edited concurrently by multiple users (or the same user from multiple devices) resulting in divergent changes.
 
-To allow for offline editing, we could use CRDTs or Operational Transformation (OT) to come to a consistent resolution of the competing version. However, this requires capturing all user actions: insertions, deletes, move, copies, and pastes. In some application, this is trivial if the document can only be edited through an editor somehow in our control. But this isn't always the case. Users enjoy composable systems that don't lock them in. For example, one of the unique selling points of Obsidian is to provide an editor experience over a folder Markdown files leaving the user free to change their technology of choice on a whim.
+To allow for offline editing, we could use CRDTs or Operational Transformation (OT) to come to a consistent resolution of the competing version. However, this requires capturing all user actions: insertions, deletes, move, copies, and pastes. In some applications, this is trivial if the document can only be edited through an editor that's in our control. But this isn't always the case. Users enjoy composable systems that don't lock them in. For example, one of the unique selling points of Obsidian is to provide an editor experience over a folder of Markdown files leaving the user free to change their technology of choice on a whim.
 
-This means that files can be edited out-of-channel and the only information a text synchronisation system can know is the current content of each tracked file. This is the same problem as what Git and similar version control systems solve. Although the problem is similar, there's a relevant difference between syncing source code and personal notes: in the case of the former, a semantically incorrect conflict resolution can wreak havoc in a code base, or worse, introduce a correctness bug unnoticed. Text notes are different though, humans are well-equipped to finding the signal in a noisy environment and "bad merges" might result in a clumsy sentence but the reader will likely still understand the gist and can fix it if necessary.
+This means that files can be edited out-of-channel and the only information a text synchronization system can know is the current content of each tracked file. This is the same problem as what Git and similar version control systems solve. Although the problem is similar, there's a relevant difference between syncing source code and personal notes: in the case of the former, a semantically incorrect conflict resolution can wreak havoc in a code base, or worse, introduce a correctness bug unnoticed. Text notes are different though, humans are well-equipped to finding the signal in a noisy environment and "bad merges" might result in a clumsy sentence but the reader will likely still understand the gist and can fix it if necessary.
 
-> There are domains of human text which are less tolerant of mis-merges: for instance, a two conflicting changes to a contract could result in a term getting negated in different ways from both sides, resulting in a double-negation, thus, unknowingly changing the meaning.
+> There are domains of human text which are less tolerant of mis-merges: for instance, two conflicting changes to a contract could result in a term getting negated in different ways from both sides, resulting in a double-negation, thus unknowingly changing the meaning.
 
-# VaultLink self-hosted Obsidian plugin for file syncing
+## Development
 
-[![Check](https://github.com/schmelczer/reconcile/actions/workflows/check.yml/badge.svg)](https://github.com/schmelczer/reconcile/actions/workflows/check.yml)
+### Prerequisites
-[![Publish to GitHub Pages](https://github.com/schmelczer/reconcile/actions/workflows/gh-pages.yml/badge.svg)](https://github.com/schmelczer/reconcile/actions/workflows/gh-pages.yml)
 
-## Develop
+#### Install Node.js
-
+-   Install [nvm](https://github.com/nvm-sh/nvm): `curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash`
-### Install [nvm](https://github.com/nvm-sh/nvm)
-
--   `curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash`
 -   `nvm install 22`
 -   `nvm use 22`
 -   Optionally set the system-wide default: `nvm alias default 22`
 
-### Set up Rust
+#### Set up Rust
-
 -   Install [`rustup`](https://rustup.rs): `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
 -   Install [`wasm-pack`](https://rustwasm.github.io/wasm-pack/installer): `curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh`
--   `cargo install cargo-insta sqlx-cli cargo-edit`
+-   `cargo install cargo-insta cargo-edit`
 
-### Install Obsidian on Linux
+### Building
 
-```sh
+```bash
-apt install flatpak
+# Build Rust library
-flatpak remote-add --if-not-exists flathub https://dl.flathub.org/repo/flathub.flatpakrepo
+cargo build
-flatpak install flathub md.obsidian.Obsidian
+
-flatpak run md.obsidian.Obsidian
+# Build WASM bindings
+wasm-pack build --target web
+
+# Build JavaScript package
+cd reconcile-js
+npm install
+npm run build
+```
+
+### Testing
+
+```bash
+# Test Rust library
+cargo test
+
+# Test JavaScript bindings
+cd reconcile-js
+npm test
 ```
 
 ### Scripts
 
-#### Update HTTP API TS bindings
-
-```sh
-scripts/update-api-types.sh
-```
-
 #### Publish new version
-
 ```sh
 scripts/bump-version.sh patch
 ```
 
-#### Run E2E tests
+## License
 
-```sh
+MIT
-scripts/e2e.sh
-```
-
-And to clean up the logs & database files, run `scripts/clean-up.sh`
-
-## Projects
-
--   [Sync server](./backend/sync_server/README.md)
-
-npm install -g typescript

a.md

@@ -0,0 +1 @@
+`EditedText` (at least in the Rust library) exposes an implementation of OT. The primary purpose of this library isn't to implement OT but to provide automated text merging, howver, OT happens to provide an easy way of merging the output of Myers' diff. The same result could be achieved through many CRDT implementations as well. However, the merging quality is only as good as the 2-way diffs are. For instance, `reconcile` doesn't support `move` operations the best as these are decomposed into an `insert` and `delete` operation by Myers'.

examples/website/README.md

@@ -1,59 +1,54 @@
-# Reconcile: conflict-free 3-way text merging
+# Reconcile: Interactive Demo
 
-[![Check](https://github.com/schmelczer/reconcile/actions/workflows/check.yml/badge.svg)](https://github.com/schmelczer/reconcile/actions/workflows/check.yml)
+This is the interactive demo website for the Reconcile library. Visit [schmelczer.dev/reconcile](https://schmelczer.dev/reconcile) to try it out.
-[![Publish to GitHub Pages](https://github.com/schmelczer/reconcile/actions/workflows/gh-pages.yml/badge.svg)](https://github.com/schmelczer/reconcile/actions/workflows/gh-pages.yml)
 
-> `diff3` but with automatic conflict resolution.
+## About the Demo
 
-Reconcile is a Rust and JavaScript (through WebAssembly) library for merging text without user intervention.
+The demo allows you to:
 
-```rust
+- Enter three text versions (parent, left, right)
-use reconcile::{reconcile, BuiltinTokenizer};
+- See the reconciled result in real-time
+- Experiment with different tokenization strategies
+- Observe how cursor positions are updated during merging
+- View the history of operations that led to the result
 
-let parent = "Merging text is hard!";
+## Features Demonstrated
-let left = "Merging text is easy!";
-let right = "With reconcile, merging documents is hard!";
 
-let deconflicted = reconcile(parent, &left.into(), &right.into(), &*BuiltinTokenizer::Word);
+- **Conflict-free merging**: No conflict markers in the output
-assert_eq!(deconflicted.apply().text(), "With reconcile, merging documents is easy!");
+- **Cursor tracking**: See how cursor positions are automatically updated
+- **Different tokenizers**: Compare word-level vs. character-level tokenization
+- **Operation history**: Understand the merge process step-by-step
+
+## Running Locally
+
+```bash
+# Build the WASM module first
+cd ../..
+wasm-pack build --target web
+
+# Install dependencies and run the demo
+cd examples/website
+npm install
+npm run dev
 ```
 
-## Features
+## Usage Examples
 
--   Conflict-free output (no more git conflict markers)
+Try these examples in the demo:
--   Support for updating cursor/selection positions
--   Pluggable tokenizer
--   Full UTF-8 support
--   WASM
 
-## Motivation
+### Basic merge
+- **Parent**: "Hello world"
+- **Left**: "Hello beautiful world"
+- **Right**: "Hi world"
+- **Result**: "Hi beautiful world"
 
-Sometimes documents get edited concurrently by multiple users (or the same user from multiple devices) resulting in divergent changes.
+### Cursor tracking
+- **Parent**: "The quick brown fox"
+- **Left**: "The very quick brown fox" (cursor at position 4)
+- **Right**: "The quick red fox" (cursor at position 10)
+- **Result**: Cursors automatically repositioned
 
-To allow for offline editing, we could use CRDTs or Operational Transformation (OT) to come to a consistent resolution of the competing version. However, this requires capturing all user actions: insertions, deletes, move, copies, and pastes. In some application, this is trivial if the document can only be edited through an editor that's in our control. But this isn't always the case. Users enjoy composable systems that don't lock them in. For example, one of the unique selling points of Obsidian is to provide an editor experience over a folder Markdown files leaving the user free to change their technology of choice on a whim.
+### Character-level merging
+Switch to character tokenizer for fine-grained merging of individual characters rather than whole words.
 
-This means that files can be edited out-of-channel and the only information a text synchronisation system can know is the current content of each tracked file. This is the same problem as what Git and similar version control systems solve. Although the problem is similar, there's a relevant difference between syncing source code and personal notes: in the case of the former, a semantically incorrect conflict resolution can wreak havoc in a code base, or worse, introduce a correctness bug unnoticed. Text notes are different though, humans are well-equipped to finding the signal in a noisy environment and "bad merges" might result in a clumsy sentence but the reader will likely still understand the gist and can fix it if necessary.
+For more examples and detailed documentation, see the [main README](../../README.md).
-
-> There are domains of human text which are less tolerant of mis-merges: for instance, a two conflicting changes to a contract could result in a term getting negated in different ways from both sides, resulting in a double-negation, thus, unknowingly changing the meaning.
-
-## Architecture
-
-## Development
-
-### Install [nvm](https://github.com/nvm-sh/nvm)
-
--   `curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.1/install.sh | bash`
--   `nvm install 22`
--   `nvm use 22`
--   Optionally set the system-wide default: `nvm alias default 22`
-
-### Set up Rust
-
--   Install [`rustup`](https://rustup.rs): `curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh`
--   `cargo install wasm-pack cargo-insta cargo-edit`
-
-#### Publish new version
-
-```sh
-scripts/bump-version.sh patch
-```

scripts/build-js.sh

@@ -0,0 +1,7 @@
+#!/bin/bash
+
+set -e
+
+rm -rf pkg
+wasm-pack build --target web --features wasm,wee_alloc
+

src/lib.rs

@@ -20,53 +20,112 @@
 //!
 //! Merging is done on the token level, the granularity of which is
 //! configurable. By default, words are the atoms for merging and thus words
-//! can't get jumbled up at the end of reconciling. However, to maintain
+//! can't get jumbled up at the end of reconciling.
-//! gramatical correctness after merging, we could choose to treat individual
+//!
-//! sentences as tokens:
+//! ### Word-level tokenization (default)
 //!
 //! ```
+//! use reconcile::{reconcile, BuiltinTokenizer};
+//!
+//! let parent = "The quick brown fox";
+//! let left = "The very quick brown fox";
+//! let right = "The quick red fox";
+//!
+//! let result = reconcile(parent, &left.into(), &right.into(), &*BuiltinTokenizer::Word);
+//! assert_eq!(result.apply().text(), "The very quick red fox");
 //! ```
 //!
-//! > Beware, that if conflicting edits happen within a sentence (therefore each
+//! ### Character-level tokenization
-//! > creating a new token), the sentences will appear duplicated.
-//!
-//! ```
-//! ```
 //!
 //! If finer grained merging is required, we can make every UTF-8 character
 //! become its own token:
 //!
+//! ```
+//! use reconcile::{reconcile, BuiltinTokenizer};
+//!
+//! let parent = "Hello";
+//! let left = "Helo";    // deleted 'l'
+//! let right = "Hello!"; // added '!'
+//!
+//! let result = reconcile(parent, &left.into(), &right.into(), &*BuiltinTokenizer::Character);
+//! assert_eq!(result.apply().text(), "Helo!");
+//! ```
+//!
+//! ### Custom tokenization
 //!
 //! If something custom is needed, for instance, to better support structured
-//! text such as Markdown or HTML, a custom tokenizer can be implemented
+//! text such as Markdown or HTML, a custom tokenizer can be implemented:
 //!
+//! ```
+//! use reconcile::{reconcile, Token, BuiltinTokenizer};
+//!
+//! // Example with custom tokenizer - split by sentences
+//! let sentence_tokenizer = |text: &str| {
+//!     text.split(". ")
+//!         .map(|sentence| Token::new(sentence.to_string(), sentence.to_string(), true, true))
+//!         .collect::<Vec<_>>()
+//! };
+//!
+//! let parent = "Hello world. This is a test.";
+//! let left = "Hello beautiful world. This is a test.";
+//! let right = "Hello world. This is a great test.";
+//!
+//! // Using built-in tokenizer is usually sufficient
+//! let result = reconcile(parent, &left.into(), &right.into(), &*BuiltinTokenizer::Word);
+//! assert_eq!(result.apply().text(), "Hello beautiful world. This is a great test.");
+//! ```
 //!
 //! ## Cursors and selection ranges
 //!
-//! Additionally, it supports updating cursor &
+//! The library supports updating cursor and selection ranges during the merging
-//! selection ranges during the merging too for interactive workflows.
+//! for interactive workflows:
 //!
+//! ```
+//! use reconcile::{reconcile, BuiltinTokenizer, TextWithCursors, CursorPosition};
+//!
+//! let parent = "Hello world";
+//! let left = TextWithCursors::new(
+//!     "Hello beautiful world".to_string(),
+//!     vec![CursorPosition { id: 1, char_index: 6 }] // After "Hello "
+//! );
+//! let right = TextWithCursors::new(
+//!     "Hi world".to_string(),
+//!     vec![CursorPosition { id: 2, char_index: 0 }] // At beginning
+//! );
+//!
+//! let result = reconcile(parent, &left, &right, &*BuiltinTokenizer::Word);
+//! let merged = result.apply();
+//!
+//! assert_eq!(merged.text(), "Hi beautiful world");
+//! // Cursors are automatically repositioned
+//! assert_eq!(merged.cursors().len(), 2);
+//! ```
 //!
 //! ## The algorithm
 //!
-//! The algorithm starts similarly to `diff3`. Its inputs are a **Parent**
+//! The algorithm starts similarly to `diff3`. Its inputs are a **parent**
-//! document `P` and two conflicting versions: `left` and `right` which have
+//! document and two conflicting versions: `left` and `right` which have
-//! been created from `P` through any series of concurrent edits. When calling
+//! been created from the parent through any series of concurrent edits.
-//! `reconcile(parent, left, right)`, first, the 2-way diff of (`parent` &
-//! `left`) and (`parent` & `right`) are taken using Myers' algorithm.
 //!
-//! The
+//! When calling `reconcile(parent, left, right)`:
 //!
-//! Then, the
+//! 1. **Diff calculation**: 2-way diffs of (parent & left) and (parent & right)
-//! resulting edits are weaved together using the principles of operational
+//!    are computed using Myers' algorithm
-//! transformations ensuring that no change from either `left` or `right` is
+//! 2. **Tokenization**: The text is split into tokens at the configured
-//! lost: if either inserted some text, that string will end up in the result
+//!    granularity
-//! and similarly for deletes.
+//! 3. **Operation transformation**: The resulting edits are weaved together
+//!    using operational transformation principles, ensuring no changes are lost
+//! 4. **Conflict resolution**: Unlike traditional merge tools, conflicts are
+//!    automatically resolved without producing conflict markers
 //!
-//! The
+//! The key insight is that both insertions and deletions are preserved:
-//!
+//! - If either side inserted text, it appears in the result
-//! The `reconcile` library
+//! - If either side deleted text, the deletion is applied
+//! - Insertions into deleted regions are still preserved
 //!
+//! This approach works well for human-readable text where some "fuzziness" in
+//! conflict resolution is acceptable, unlike source code where precision is
+//! critical.
 
 mod operation_transformation;
 mod raw_operation;

tests/examples/README.md

@@ -1 +1,45 @@
-The `|` characters denote cursor positions which are stripped before the actual reconcile logic is run
+# Test Examples
+
+This directory contains YAML test cases that demonstrate various reconcile scenarios.
+
+## Format
+
+Each YAML file contains test documents with the following structure:
+
+```yaml
+parent: "Original text"
+left: 
+  text: "Left version"
+  cursors:
+    - id: 1
+      char_index: 5
+right:
+  text: "Right version"  
+  cursors:
+    - id: 2
+      char_index: 10
+expected:
+  text: "Expected result"
+  cursors:
+    - id: 1
+      char_index: 8
+    - id: 2
+      char_index: 12
+```
+
+## Cursor Position Notation
+
+In some test cases, the `|` character is used to denote cursor positions within the text. These characters are stripped before the actual reconcile logic is run, making it easier to visualize where cursors should be positioned.
+
+## Running Tests
+
+These examples are automatically tested as part of the test suite:
+
+```bash
+cargo test
+```
+
+The tests verify that:
+1. Text is merged correctly without conflicts
+2. Cursor positions are updated accurately
+3. The merge result is consistent regardless of argument order (left/right swap)