andras/reconcile
1
0
Fork 0
Code Issues Pull requests 6 Projects Releases Packages Wiki Activity Actions

Improve docs and compare with alternatives

Browse source
This commit is contained in:
Andras Schmelczer 2026-03-10 20:29:35 +00:00
parent 5962feb90a
commit 3d382ad741
14 changed files with 106 additions and 69 deletions
Download patch file Download diff file Expand all files Collapse all files

10
src/tokenizer/token.rs
View file

@ -3,13 +3,11 @@ use std::fmt::Debug;
#[cfg(feature = "serde")]
use serde::{Deserialize, Serialize};
/// A token is a string that has been normalized in some way.
/// A token with a normalized form (used for diffing) and an original form
/// (used when applying operations). Joinability flags control whether
/// adjacent insertions interleave or group.
///
/// A token consists of the normalized form is used for comparison, and the
/// original form used for subsequently applying `Operation`-s to a text
/// document.
///
/// It's UTF-8 compatible.
/// UTF-8 compatible.
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
#[derive(Debug, Clone)]
pub struct Token<T>