NeuroSkill™: Proactive Real-Time Agentic System Capable of Modeling Human State of Mind

Abstract

Key Contributions

1. 1

   GPU-accelerated local EEG pipeline

   A complete signal chain — BLE acquisition, bandpass/notch filtering, Welch PSD, band power extraction, cross-channel ratios, nonlinear complexity, artifact detection, and composite score computation — implemented entirely as wgpu compute shaders. Runs at ≈4 Hz on Metal (macOS), Vulkan (Linux), and DX12 (Windows) without any cloud round-trip.

2. 2

   ZUNA neural encoder + HNSW index

   A 186 K-parameter 1-D convolutional encoder maps each 5-second epoch to a 32-D L2-normalised embedding. An HNSW index (hnswlib) builds incrementally across sessions, enabling cosine-similarity search at <1 ms per query over 10,000+ epochs. Text queries are supported via a jointly-trained MLP that aligns tag-name BERT embeddings with the 32-D EEG space.

3. 3

   Brain Nebula™ (GPU-accelerated UMAP)

   A wgpu/CubeCL backend for fast-umap runs UMAP dimensionality reduction on dual-session embedding batches in under 15 seconds on Apple M1. The result is visualised as an interactive 3D Threlte (Three.js) point cloud in the dashboard, with cluster separation analysis (inter/intra centroid distance, outlier detection) exposed via the CLI and WebSocket API.

4. 4

   AASM sleep staging from frontal/temporal EEG

   Epoch-by-epoch classification into Wake, N1, N2, N3, REM using band-power ratios, sleep spindle detection (matched filter, 11–16 Hz), K-complex template matching, and DFA exponent trend analysis — all from only four electrodes. Outputs efficiency, onset/REM latency, transition count, awakening count, and per-stage bout statistics.

5. 5

   Comprehensive session analytics

   Side-by-side A/B session comparison with per-metric absolute and percentage deltas, improved/declined classification, and auto-queued UMAP. Recording history tracks total hours, session streak, and today-vs-7-day averages for every metric. Labels are stored with timestamps in SQLite and visualised as markers on all charts.

6. 6

   Local WebSocket API (9 commands, 2 events)

   A tokio-tungstenite server binds to 127.0.0.1:8375 exclusively, broadcasting eeg-bands (~4 Hz, 60+ metrics) and muse-status (~1 Hz, device health). Nine request-response commands cover status, sessions, label, search, compare, sleep, umap, umap_poll, and export. All data stays on-device.

7. 7

   TypeScript CLI with mDNS discovery

   cli.ts wraps the WebSocket API with mDNS auto-discovery (bonjour-service), lsof fallback port scanning, session-aware auto-range defaults, colourised terminal output, and --json mode for piping to jq or any external processor. No global install required — runs via {CLI_PREFIX}.

System Architecture

NeuroSkill™ is a Tauri 2 application: a Rust backend handles all signal processing, storage, and networking; a SvelteKit frontend renders the dashboard using Threlte (Three.js) for Brain Nebula™. The two communicate over Tauri's IPC channel. External clients (CLI, Python scripts, web apps) communicate with the Rust backend via the local WebSocket server — the frontend is completely bypass-able.

The GPU pipeline runs inside the Rust process — no separate GPU daemon. wgpu selects the best available backend at startup (Metal preferred on macOS, Vulkan on Linux, DX12 on Windows) and falls back to software rendering on headless/CI hosts.

Signal Pipeline

Raw EEG arrives at 256 Hz from four electrodes (AF7, AF8 frontal; TP9, TP10 temporal). Each 5-second epoch (1,280 samples per channel) is dispatched to the GPU for the following sequential stages. PPG and IMU are processed on a separate lower-frequency path.

ZUNA Embeddings

Encoder architecture

ZUNA (Zero-latency Universal Neuro-Annotation) is a 1-D depthwise-separable convolutional encoder designed to run in under 5 ms on CPU (CoreML on macOS, ONNX Runtime on Linux/Windows) so embedding generation never becomes a pipeline bottleneck. Input shape: [4 channels × 1280 samples].

Total encoder parameters: 186 K. Full autoencoder (used for training only): 1.2 M.

Training

Training dataset: 42,000 labelled 5-second epochs from 38 healthy adult subjects across 8 cognitive-state conditions recorded with Muse 2 headsets. Contrastive objective: NT-Xent loss (temperature τ = 0.07, batch size 512) — epochs with the same label are attracted, different-label epochs repelled in the 32-D sphere. Optimiser: AdamW (lr = 1×10⁻³, cosine decay, 100 epochs). Label conditions: N-back (2-back, 3-back), Stroop colour-word, eyes-closed rest, guided mindfulness, passive video, post-wake drowsiness, motor imagery.

Text projection (natural-language search)

A two-layer MLP is trained jointly to project BERT-base embeddings of label-name strings into the 32-D EEG space (mean squared error loss). This allows the search command and API to accept arbitrary natural-language queries — e.g. "calm, focused, low anxiety" — without requiring the user to provide example epochs.

HNSW index

Embeddings are inserted into an hnswlib HNSW index (M = 16, ef_construction = 200, cosine metric) immediately after generation. The index is serialised to ~/.skill/hnsw_index.bin and loaded at startup. Query time is O(log N) — measured at <1 ms for N = 10,000 on M1. The search command returns neighbours with timestamps, labels, and the aggregate analysis (distance stats, top days, temporal distribution).

Sleep Staging

Each 30-second epoch is classified against AASM (2007/2017) guidelines using a rule-based decision tree operating on the band-power ratios and temporal features computed by the main pipeline. Because Muse headsets provide only frontal (AF7, AF8) and temporal (TP9, TP10) electrodes, occipital alpha and central EEG features are approximated.

The sleep command returns a per-epoch hypnogram array alongside a summary (total epochs, stage breakdown in count and minutes) and an analysis object with efficiency, onset latency, REM latency, transition count, awakening count, and per-stage bout statistics (count, mean duration, max duration).

Limitation: without occipital or central electrodes, classification is an approximation. Do not use for clinical diagnosis. See the Limitations section.

Dashboard Windows

The SvelteKit frontend communicates with the Rust backend over Tauri IPC. All panels update from the eeg-bands event stream at ~4 Hz. Each window can be undocked, resized, or pinned to any monitor.

WebSocket API

The server binds to ws://127.0.0.1:8375 exclusively. Any number of clients can connect simultaneously; all receive the same broadcast stream. Commands are JSON objects sent from client to server; responses arrive on the same connection. Full schemas and example payloads are in the API reference ↗.

Broadcast events (server → all clients)

Request-response commands (client → server)

All commands are sent as {"command": "...", ...params} and receive {"command": "...", "ok": true, ...data} or {"command": "...", "ok": false, "error": "..."}. See the full API reference and recipes for example code.

Command-Line Interface

cli.ts is a TypeScript program (Node ≥ 18, dependencies: ws, bonjour-service, tsx) that wraps every WebSocket API command in a ergonomic terminal interface.

Auto-discovery

Connection resolution order: (1) explicit --port <n>, (2) mDNS browse for _skill._tcp (5 s timeout), (3) lsof scan for processes named "skill" with TCP LISTEN sockets, probing each port with a 1.5 s WebSocket handshake. Once found, connects with up to 3 retries (1 s delay, 5 s handshake timeout). Global 10-minute timeout prevents hangs.

Session-aware auto-range defaults

When time parameters are omitted, the CLI fetches sessions and picks smart defaults. A rerun: line is always printed for reproducibility.

See the full CLI reference ↗ for all flags, output schemas, and terminal examples. See Recipes → CLI for copy-paste usage patterns.

Use Cases

Data Storage & Persistence

All user data is stored under ~/.skill/ (macOS/Linux) or %APPDATA%\NeuroSkill™\ (Windows). No data is written outside this directory without an explicit export command.

~/.skill/
  embeddings/
    20260224.sqlite # per-day epoch database 20260223.sqlite
    …
  labels.sqlite # all annotations, all time hnsw_index.bin # hnswlib serialised index umap_cache/
    3_result.json # cached UMAP job outputs …
  exports/ # CSV/JSON exports from the export command preferences.json # notch frequency, overlay prefs, etc.

Epoch database schema (per-day SQLite)

Each YYYYMMDD.sqlite contains one epochs table:

CREATE TABLE epochs (
  id INTEGER PRIMARY KEY,
  timestamp REAL NOT NULL, -- Unix seconds (float) device_name TEXT,
  embedding BLOB NOT NULL, -- 32× float32, LE rel_delta REAL, rel_theta REAL, rel_alpha REAL,
  rel_beta REAL, rel_gamma REAL,
  focus REAL, relaxation REAL, engagement REAL,
  hr REAL, rmssd REAL, spo2 REAL -- (all 60+ metric columns present; abbreviated here) ); CREATE INDEX idx_timestamp ON epochs(timestamp);

Growth rate

At 4 Hz × ~200 bytes per epoch row + 32-byte BLOB, each hour of recording produces approximately 2.9 MB of SQLite data. A 45-minute session ≈ 2.2 MB. The HNSW index adds ≈0.5 MB per 1,000 epochs (~70 min of recording).

Privacy Model

Validation

Full benchmark methodology, per-metric reference tables, confusion matrices (sleep staging), and per-subject retrieval break-downs are in the paper supplementary materials. Read the paper →

Limitations

Cite this Work

If you use NeuroSkill™ in your research, please cite:

@software{neuroskill2026,
  title     = {{NeuroSkill™}: Real-Time {EXG} Analysis with Neural Embeddings
               for Wearable Brain--Computer Interfaces},
  author    = {Nataliya Kosmyna, Eugene Hauptmann},
  year      = {2026},
  url       = {https://arxiv.org/abs/2603.03212},
  version   = {0.0.55},
  license   = {GPL-3.0},
  note      = {Open-source desktop agentic State of Mind based on Brain Computer Interfaces},
}

DOI and final author list will be updated upon formal publication. Check the repository for the latest recommended citation format.