Add docs
This commit is contained in:
parent
56c1f4d58b
commit
50a95b114d
19 changed files with 4663 additions and 1 deletions
344
docs/architecture/index.md
Normal file
344
docs/architecture/index.md
Normal file
|
|
@ -0,0 +1,344 @@
|
|||
# Architecture Overview
|
||||
|
||||
VaultLink is built as a distributed system with a central sync server and multiple clients. This document explains the high-level architecture and design decisions.
|
||||
|
||||
## System Components
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ Clients │
|
||||
├─────────────────────┬───────────────────┬───────────────────┤
|
||||
│ Obsidian Plugin │ Obsidian Plugin │ CLI Client │
|
||||
│ (User A - Device1) │ (User A - Device2│ (Server/Backup) │
|
||||
└──────────┬──────────┴─────────┬─────────┴──────────┬────────┘
|
||||
│ │ │
|
||||
│ WebSocket │ WebSocket │ WebSocket
|
||||
│ │ │
|
||||
└────────────────────┼────────────────────┘
|
||||
│
|
||||
┌───────────▼───────────┐
|
||||
│ Sync Server │
|
||||
│ (Rust + Axum) │
|
||||
│ │
|
||||
│ ┌─────────────────┐ │
|
||||
│ │ WebSocket Hub │ │
|
||||
│ └────────┬────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────▼────────┐ │
|
||||
│ │ Sync Engine │ │
|
||||
│ │ (OT Algorithm) │ │
|
||||
│ └────────┬────────┘ │
|
||||
│ │ │
|
||||
│ ┌────────▼────────┐ │
|
||||
│ │ SQLite Database │ │
|
||||
│ │ (Per Vault) │ │
|
||||
│ └─────────────────┘ │
|
||||
└───────────────────────┘
|
||||
```
|
||||
|
||||
## Core Components
|
||||
|
||||
### Sync Server
|
||||
|
||||
The central authority for synchronization, written in Rust using Axum framework.
|
||||
|
||||
**Responsibilities**:
|
||||
- Accept WebSocket connections from clients
|
||||
- Authenticate users via token-based auth
|
||||
- Store document versions in SQLite
|
||||
- Coordinate real-time updates between clients
|
||||
- Apply operational transformation for conflict resolution
|
||||
- Manage vault access control
|
||||
|
||||
**Technology**:
|
||||
- **Language**: Rust 1.89+
|
||||
- **Framework**: Axum (async web framework)
|
||||
- **Database**: SQLite with SQLx
|
||||
- **Protocol**: WebSockets for real-time communication
|
||||
- **Sync Algorithm**: reconcile-text (operational transformation)
|
||||
|
||||
### Sync Client Library
|
||||
|
||||
TypeScript library providing core synchronization logic, used by both the Obsidian plugin and CLI client.
|
||||
|
||||
**Responsibilities**:
|
||||
- Manage WebSocket connection to server
|
||||
- Watch local filesystem for changes
|
||||
- Upload and download files
|
||||
- Apply remote changes locally
|
||||
- Handle conflict resolution
|
||||
- Maintain sync metadata
|
||||
|
||||
**Technology**:
|
||||
- **Language**: TypeScript
|
||||
- **Build**: Webpack
|
||||
- **Protocol**: WebSocket client
|
||||
- **File System**: Node.js `fs` API / Obsidian API
|
||||
|
||||
### Obsidian Plugin
|
||||
|
||||
Integration layer between sync client and Obsidian.
|
||||
|
||||
**Responsibilities**:
|
||||
- Provide UI for configuration
|
||||
- Bridge sync client with Obsidian's file system API
|
||||
- Handle Obsidian lifecycle events
|
||||
- Display sync status to users
|
||||
|
||||
**Technology**:
|
||||
- **Platform**: Obsidian Plugin API
|
||||
- **Core**: sync-client library
|
||||
- **UI**: Obsidian settings UI
|
||||
|
||||
### CLI Client
|
||||
|
||||
Standalone executable for syncing vaults without Obsidian.
|
||||
|
||||
**Responsibilities**:
|
||||
- Command-line interface
|
||||
- File system access via Node.js
|
||||
- Daemon mode for continuous sync
|
||||
- Health check endpoint for monitoring
|
||||
|
||||
**Technology**:
|
||||
- **Language**: TypeScript
|
||||
- **Runtime**: Node.js
|
||||
- **CLI**: Commander.js
|
||||
- **Core**: sync-client library
|
||||
|
||||
## Data Flow
|
||||
|
||||
### Initial Connection
|
||||
|
||||
1. Client connects via WebSocket to server
|
||||
2. Server authenticates using provided token
|
||||
3. Server verifies user has access to requested vault
|
||||
4. Connection established, sync begins
|
||||
|
||||
### File Upload Flow
|
||||
|
||||
```
|
||||
Client Server
|
||||
│ │
|
||||
│ 1. File changed locally │
|
||||
│ │
|
||||
│ 2. Read file content │
|
||||
│ │
|
||||
│ 3. WebSocket: Upload file │
|
||||
├──────────────────────────────►│
|
||||
│ │ 4. Store in SQLite
|
||||
│ │
|
||||
│ │ 5. Broadcast to other clients
|
||||
│ ├───────────────────────►
|
||||
│ 6. Ack upload │
|
||||
│◄──────────────────────────────┤
|
||||
```
|
||||
|
||||
### File Download Flow
|
||||
|
||||
```
|
||||
Client A Server Client B
|
||||
│ │ │
|
||||
│ │ 1. File uploaded │
|
||||
│ │◄────────────────────────┤
|
||||
│ │ │
|
||||
│ │ 2. Store in DB │
|
||||
│ │ │
|
||||
│ 3. Push notification │ │
|
||||
│◄────────────────────────┤ │
|
||||
│ │ │
|
||||
│ 4. Download file │ │
|
||||
├────────────────────────►│ │
|
||||
│ │ │
|
||||
│ 5. Write locally │ │
|
||||
│ │ │
|
||||
```
|
||||
|
||||
### Conflict Resolution
|
||||
|
||||
When two clients edit the same file simultaneously:
|
||||
|
||||
```
|
||||
Client A Server Client B
|
||||
│ │ │
|
||||
│ 1. Edit file │ │ 1. Edit same file
|
||||
│ │ │
|
||||
│ 2. Upload changes │ │ 2. Upload changes
|
||||
├────────────────────────►│◄────────────────────────┤
|
||||
│ │ │
|
||||
│ │ 3. Apply OT algorithm │
|
||||
│ │ - Merge both edits │
|
||||
│ │ - Preserve all changes│
|
||||
│ │ │
|
||||
│ 4. Receive merged ver. │ 5. Receive merged ver. │
|
||||
│◄────────────────────────┤────────────────────────►│
|
||||
│ │ │
|
||||
│ 6. Apply locally │ │ 6. Apply locally
|
||||
```
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
### Server Storage
|
||||
|
||||
Each vault has its own SQLite database:
|
||||
|
||||
```
|
||||
databases/
|
||||
├── vault-1.db
|
||||
├── vault-2.db
|
||||
└── shared-team.db
|
||||
```
|
||||
|
||||
**Database Schema** (simplified):
|
||||
- **documents**: File metadata (path, size, modified time)
|
||||
- **versions**: Document content with version history
|
||||
- **cursors**: Client sync state
|
||||
|
||||
### Client Storage
|
||||
|
||||
Clients maintain sync metadata:
|
||||
|
||||
```
|
||||
.vaultlink/
|
||||
├── metadata.json # Sync state
|
||||
└── cache/ # Optional local cache
|
||||
```
|
||||
|
||||
The `.vaultlink` directory tracks which files have been synced and their versions to enable efficient synchronization.
|
||||
|
||||
## Communication Protocol
|
||||
|
||||
### WebSocket Messages
|
||||
|
||||
Client-server communication uses JSON messages over WebSocket.
|
||||
|
||||
**Message Types**:
|
||||
- `upload_file`: Client → Server (file upload)
|
||||
- `download_file`: Client → Server (request file)
|
||||
- `file_updated`: Server → Client (file changed notification)
|
||||
- `file_deleted`: Server → Client (file deleted notification)
|
||||
- `sync_complete`: Server → Client (initial sync finished)
|
||||
|
||||
### Authentication
|
||||
|
||||
Token-based authentication on connection:
|
||||
|
||||
```typescript
|
||||
// Client sends token on connect
|
||||
{
|
||||
type: "auth",
|
||||
token: "user-auth-token",
|
||||
vault: "vault-name"
|
||||
}
|
||||
|
||||
// Server responds
|
||||
{
|
||||
type: "auth_success"
|
||||
}
|
||||
// or
|
||||
{
|
||||
type: "auth_error",
|
||||
message: "Invalid token"
|
||||
}
|
||||
```
|
||||
|
||||
## Scalability Considerations
|
||||
|
||||
### Current Architecture
|
||||
|
||||
- **SQLite per vault**: Simple, performant, limited to single server
|
||||
- **WebSocket connections**: Stateful, requires sticky sessions for load balancing
|
||||
- **Operational transformation**: Centralized on server
|
||||
|
||||
### Scaling Approaches
|
||||
|
||||
**Vertical Scaling**:
|
||||
- Increase server resources (CPU, RAM, storage)
|
||||
- Optimize database queries and indexing
|
||||
- Tune connection limits
|
||||
|
||||
**Horizontal Scaling** (future):
|
||||
- Separate vault servers (vault sharding)
|
||||
- Load balancer with sticky sessions
|
||||
- Shared storage layer for SQLite databases
|
||||
- Consider alternative databases (PostgreSQL) for multi-server setups
|
||||
|
||||
### Performance Characteristics
|
||||
|
||||
- **Small vaults** (< 1000 files): Excellent performance
|
||||
- **Medium vaults** (1000-10000 files): Good performance with tuning
|
||||
- **Large vaults** (> 10000 files): May require optimization
|
||||
- **Concurrent users**: Tested with dozens of simultaneous clients per vault
|
||||
|
||||
## Security Model
|
||||
|
||||
### Authentication
|
||||
|
||||
- Token-based authentication
|
||||
- Tokens configured in server `config.yml`
|
||||
- No password hashing (tokens are secrets)
|
||||
|
||||
### Authorization
|
||||
|
||||
- Per-user vault access control
|
||||
- Allow-list or deny-list patterns
|
||||
- Global access or vault-specific access
|
||||
|
||||
### Network Security
|
||||
|
||||
- WebSocket over TLS (WSS) for encrypted transport
|
||||
- No built-in SSL (use reverse proxy)
|
||||
- CORS configured for web clients
|
||||
|
||||
### Data Security
|
||||
|
||||
- No encryption at rest (use encrypted filesystems if needed)
|
||||
- No end-to-end encryption (server sees all content)
|
||||
- Self-hosted model: you control the data
|
||||
|
||||
## Technology Choices
|
||||
|
||||
### Why Rust for Server?
|
||||
|
||||
- **Performance**: Low latency for real-time sync
|
||||
- **Memory safety**: No crashes from memory bugs
|
||||
- **Concurrency**: Excellent async support with Tokio
|
||||
- **Type safety**: Catch bugs at compile time
|
||||
- **SQLx**: Compile-time SQL verification
|
||||
|
||||
### Why SQLite?
|
||||
|
||||
- **Simplicity**: No separate database server required
|
||||
- **Performance**: Fast for read-heavy workloads
|
||||
- **Reliability**: Battle-tested, ACID compliant
|
||||
- **Portability**: Single file per vault
|
||||
- **Backups**: Simple file copy
|
||||
|
||||
### Why WebSocket?
|
||||
|
||||
- **Real-time**: Bidirectional push for instant updates
|
||||
- **Efficiency**: Persistent connection, no polling overhead
|
||||
- **Simplicity**: Built-in browser/Node.js support
|
||||
- **Standards**: Well-supported protocol
|
||||
|
||||
### Why Operational Transformation?
|
||||
|
||||
- **Automatic conflict resolution**: No manual merging required
|
||||
- **Preserves intent**: All edits are kept
|
||||
- **Real-time collaboration**: Users see changes as they happen
|
||||
- **Proven algorithm**: Used by Google Docs, etc.
|
||||
|
||||
## Design Principles
|
||||
|
||||
1. **Self-hosted first**: Users control their data and infrastructure
|
||||
2. **Simplicity**: Easy to deploy and operate
|
||||
3. **Real-time**: Changes appear immediately
|
||||
4. **Reliability**: Handle network failures gracefully
|
||||
5. **Performance**: Fast sync for typical vault sizes
|
||||
6. **Privacy**: No third-party services or telemetry
|
||||
|
||||
## Next Steps
|
||||
|
||||
- [Learn about the sync algorithm →](/architecture/sync-algorithm)
|
||||
- [Understand data flow in detail →](/architecture/data-flow)
|
||||
- [Deploy the server →](/guide/server-setup)
|
||||
Loading…
Add table
Add a link
Reference in a new issue