Update content & design (#75)
All checks were successful
Deploy to Pages / build (push) Successful in 2m58s
All checks were successful
Deploy to Pages / build (push) Successful in 2m58s
Reviewed-on: https://home.schmelczer.dev/git/git/andras/schmelczer-dev/pulls/75
This commit is contained in:
parent
0be50b6c24
commit
b554e92e9f
83 changed files with 2995 additions and 723 deletions
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: A 3-Way Text Merger That Never Shows Conflict Markers
|
||||
description: How reconcile-text borrows the idea of operational transformation and applies it to consolidated diffs to auto-resolve conflicting edits.
|
||||
description: reconcile-text merges Markdown notes from three editors I don't control, with no operation history. Here's why git, CRDTs, and diff-match-patch each failed me.
|
||||
date: 2026-05-21
|
||||
projectPeriod: '2025'
|
||||
thumbnail:
|
||||
|
|
@ -8,10 +8,10 @@ thumbnail:
|
|||
alt: The reconcile-text logo and tagline "Conflict-free 3-way text merging".
|
||||
tags: ['systems', 'tools', 'web']
|
||||
featuredOrder: 2
|
||||
role: Author
|
||||
role: Library author
|
||||
stack: ['Rust', 'WebAssembly', 'Python', 'pyo3', 'wasm-bindgen']
|
||||
scale: One Rust core, three published packages (crates.io, npm, PyPI), driving an Obsidian sync plugin
|
||||
outcome: A small, well-tested library that fills a gap between git, CRDTs, and patch-based merging
|
||||
outcome: A small Rust library that auto-resolves prose conflicts, with WASM and Python bindings
|
||||
audience: recruiter-relevant
|
||||
links:
|
||||
- label: Demo
|
||||
|
|
@ -28,55 +28,40 @@ media:
|
|||
- type: image
|
||||
src: ./_assets/reconcile.png
|
||||
alt: The reconcile-text logo, a stylised merge arrow, with the tagline "Conflict-free 3-way text merging".
|
||||
caption: reconcile-text resolves conflicting edits to prose by weaving them together instead of asking a human to choose.
|
||||
caption: reconcile-text weaves conflicting edits together instead of asking a human to choose.
|
||||
---
|
||||
|
||||
`reconcile-text` started from a concrete need. I wanted to synchronise Markdown notes across devices where the editor was not under my control, and where the only thing I could observe was the final text on each side. Vim on one machine, VS Code on another, Obsidian on a third. No keystroke stream, no operation log, just the documents and a shared common ancestor from the last successful sync.
|
||||
## Why I wrote it
|
||||
|
||||
That setting is awkward for almost every existing tool. Git is the closest fit, but `git merge-file` answers conflicts with markers, which is exactly what a sync tool cannot ship to a user's note. CRDTs and operational transformation assume you control the editing infrastructure all the way down to the keystroke. `diff-match-patch` produces patches without a common ancestor, and on adjacent edits it silently corrupts the output. None of these matched the shape of the problem I had.
|
||||
I keep Markdown notes in three editors I don't control the internals of: Vim on my laptop, VS Code on my work machine, Obsidian on my phone. When two of them edit the same note between syncs, I have three files: the last-synced parent and two divergent children. That's the input. I want one merged file out, and I want to hand it back to the editors without conflict markers, because `<<<<<<< HEAD` is not something a notes app should ever show me.
|
||||
|
||||
So I wrote a library that does one specific thing: given a parent and two edited versions, return a single merged text that contains both sets of changes, without conflict markers and without dropping edits on the floor.
|
||||
Every existing tool got close and missed:
|
||||
|
||||
## The Problem
|
||||
- `git merge-file` does exactly the right thing structurally, then writes markers into the output. That's correct for source code and wrong for prose.
|
||||
- CRDTs and OT both assume you own the editing pipeline down to the keystroke. I don't. I'm looking at three files.
|
||||
- `diff-match-patch` doesn't take a common ancestor. On adjacent edits it quietly produces wrong output. I have a runnable example in the repo.
|
||||
|
||||
The hard part is not detecting a conflict. The hard part is resolving it well enough that a human is happy to read the result without thinking about merge mechanics.
|
||||
So the library does exactly one thing: pure function from three strings to one. No async, no networking, no concurrency, no plugins. Anything outside that boundary is somebody else's library.
|
||||
|
||||
Source code has hard correctness requirements, so refusing to choose and emitting markers is the right default. Human prose is more forgiving. A merged paragraph that is slightly clumsy is almost always preferable to one that interrupts the reader with `<<<<<<< HEAD`. That observation is the entire reason this library exists in the form it does.
|
||||
## The decisions worth naming
|
||||
|
||||
The challenge was to commit to that asymmetry honestly. The library should always produce a result. It should never silently lose an edit. It should preserve cursors so a collaborative editor can rely on it. And it should do all of this from end states alone, with no operation history available.
|
||||
**Myers diff per side, then weave the diffs.** Each child is diffed against the parent, the two edit scripts are optimised so adjacent changes group cleanly, then a single weaving pass interleaves them into one ordered op sequence that produces the merged text. The weave borrows the shape of operational transformation, but the inputs are batched complete diffs, not live keystrokes, so it only runs once per merge.
|
||||
|
||||
## Constraints
|
||||
**Tokeniser is the user knob.** This is the choice I'd defend hardest. Most of what people want when they say "merge differently" isn't a new algorithm; it's a different unit. Word-level tokenisation turns most "conflicts" in prose into two adjacent edits that coexist. Line-level makes it behave like `git merge-file`. Markdown-level merges on headings and list items. Same engine, four different products depending on what you call a token.
|
||||
|
||||
The library had to live in three places: a Rust crate, a JavaScript package built through WebAssembly, and a Python package built through `pyo3`. The cross-language story was a constraint, not a stretch goal. The Obsidian plugin I was writing alongside it consumed the npm build, but I also wanted a clean Rust crate for sync engines and a Python package for scripting.
|
||||
**Cursors are first-class merge inputs.** Each cursor has a stable ID and rides through the merge so a collaborative editor can ask "where did this cursor go?" without reconstructing it from the output text. This is the bit that made it useful to anything that wasn't just [the Obsidian sync plugin I wrote alongside it](/articles/vault-link-obsidian-sync/).
|
||||
|
||||
That ruled out anything that depended on language-specific runtime tricks. Generics, closures, and trait objects could live freely inside the Rust core, but the public surface had to be flat enough to cross both `wasm-bindgen` and `pyo3` without per-binding glue.
|
||||
**The Rust core is generic; the FFI surface is not.** Inside Rust, the tokeniser is a `dyn Fn(&str) -> Vec<Token<T>>`. That dies the moment you try to pass it through wasm-bindgen or pyo3. The fix was a closed enum of built-in tokenisers for non-Rust callers, with the generic version reserved for Rust users. Not elegant, but the alternative was per-binding glue forever.
|
||||
|
||||
It also had to be predictable. There is no async story, no networking, no concurrency. A merge is a pure function from three strings to one string with some metadata. Everything that is not the merge itself was deliberately kept out.
|
||||
**WASM size mattered enough to tune for it.** The release profile is aggressive about size, and the JS package ships a small leak detector that warns if you forget to free wasm-bindgen objects. I lost an afternoon to that the first time and didn't want anyone else to.
|
||||
|
||||
## Design
|
||||
## What's held up, what I'd change
|
||||
|
||||
The pipeline is short. The library tokenises the parent and the two edited versions, runs Myers' diff to compare each edited version against the parent, optimises the resulting edit sequences so that adjacent changes group together cleanly, and then weaves the two diffs into a single ordered sequence of operations that produces the merged text.
|
||||
- **Kept:** the never-emits-markers, never-drops-edits guarantee. It's the only reason a sync engine can call this library without an escape hatch.
|
||||
- **Kept:** the comparison example against `diff-match-patch`. It's a runnable program in the repo showing exact inputs where the alternative is wrong. Way more convincing than a benchmark table.
|
||||
- **Cut:** the snapshot tests do well on regressions and badly on unknown edge cases. Three-way merging is exactly what proptest was made for, and I should have written generators on day one.
|
||||
- **Next:** I want to be more explicit about the boundary. reconcile-text is a merge primitive, not a live collab engine. If you have a keystroke stream and a real-time channel, use Yjs or Automerge. This library is for when you don't.
|
||||
|
||||
The weaving step borrows the concept of operational transformation, but applies it to a different problem. Classic OT transforms individual keystrokes against each other in real time. Here, OT is applied to the consolidated diff output of two complete edits. The structure is similar, but the inputs are batched and the algorithm only needs to run once per merge point. It became the simplest way I could find to describe how two sets of changes should be interleaved.
|
||||
## If you take one idea from this
|
||||
|
||||
The tokeniser turned out to be more important than I initially expected. It is what decides whether a conflict exists in the first place. Word-level tokenisation, the default for prose, often turns a "conflict" into two adjacent independent edits that can coexist. Line-level tokenisation makes the library behave more like `git merge-file`. Markdown-level tokenisation merges on headings and list items rather than characters. Exposing this as a user-facing knob meant the library could be shaped to the document, not the other way around.
|
||||
|
||||
Cursors and selections were added as first-class merge inputs rather than something users reconstruct after the fact. Each cursor carries a stable ID and rides through the merge, ending up at a sensible position even when both sides edited the surrounding text. This is what made the library useful to anything resembling a collaborative editor.
|
||||
|
||||
The cross-language surface needed extra care. The tokeniser inside Rust is a `dyn Fn(&str) -> Vec<Token<T>>`, which is convenient in Rust and impossible to pass through `wasm-bindgen` or `pyo3`. The fix was to expose a closed enum of built-in tokenisers to non-Rust callers and reserve the generic version for Rust users. WebAssembly users also paid a real binary-size cost, so the release profile is tuned aggressively, and the JS package ships a small leak detector to remind callers that wasm-bindgen objects must be freed explicitly.
|
||||
|
||||
## What Worked
|
||||
|
||||
The strongest part of the project is that the result never has conflict markers and never silently drops an edit. That sounds modest, but it is exactly the property that makes the library usable inside a sync engine without an escape hatch.
|
||||
|
||||
Choosing the tokeniser as the main user-facing knob also held up well. Most of the "tuning" people want when merging prose is not a different algorithm, it is a different idea of what counts as a unit. Letting users choose between character, word, line, and Markdown granularity covered the realistic cases without inventing new merge strategies.
|
||||
|
||||
The comparison example against `diff-match-patch` was probably the most useful piece of writing in the repository. It is a runnable program, not a benchmark table, showing concrete cases where a popular alternative quietly produces wrong output. Having that as a falsifiable claim in the source tree made the value proposition much clearer than any prose description would have.
|
||||
|
||||
## What I Would Change
|
||||
|
||||
If I revisited this now, I would invest more in formal property tests around the merge. Three-way merging is exactly the kind of problem where generated inputs find behaviours that hand-written tests do not, and the snapshot tests I have are good at catching regressions but not at finding unknown edge cases.
|
||||
|
||||
I would also be more explicit about the boundary the library does not cross. It is a merge point primitive, not a live collaboration engine. CRDTs and OT remain the right tools when you actually have a keystroke stream and a real-time channel. `reconcile-text` is for the part of the problem space where you do not.
|
||||
|
||||
The part I would keep is the asymmetry the project rests on. Human text deserves a merger that prefers a slightly imperfect sentence over a conflict marker, and that decision is what shaped every other choice in the design.
|
||||
Prose deserves a merger that prefers a slightly clumsy sentence over a marker. Code doesn't. That one asymmetry is the whole reason the library exists in the shape it does; everything else fell out of taking it seriously.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue