Architecture & Under the Hood

SlyPilot's responsiveness and stability come from a deliberately plain, low-level Win32 design. This page describes the engineering principles, how the pieces talk to each other, and the rules that keep it safe. It is a description of how the program is built, not a performance benchmark: it makes no speed claims.

🏗️ System Architecture Overview

                      ┌──────────────────────────────────────────────┐
                      │                 SlyPilot.exe                 │
                      │  AutoHotkey v2 (64-bit) runtime + the script │
                      │  (Free Core; UI adapters & desktop engine)   │
                      └──────┬───────────────┬──────────────┬────────┘
                             │               │              │
      ┌──────────────────────┴──────┐        │        ┌─────┴────────────────────────┐
      ▼                             ▼        ▼        ▼                              ▼
 ┌─────────────┐             ┌─────────────┐ ┌─────────────────┐              ┌─────────────┐
 │ Win32 GUI   │             │ COM Interop │ │ Input & Window  │              │ Core API    │
 │ Subsystem   │             │ Layer       │ │ Hooks           │              │ (WM_COPYDATA│
 └──────┬──────┘             └──────┬──────┘ └────────┬────────┘              └──────┬──────┘
        │                           │                 │                              │
        │ • Custom GUIs             │ • Ciantic       │ • WM_POINTERDOWN/UP          │ Local IPC
        │ • WS_EX_NOACTIVATE        │   VirtualDesktop│ • Tablet flick suppression   │ to Pro\
        │ • WS_CLIPCHILDREN         │   Accessor.dll  │ • Keyboard hooks, hotkeys    │
        │ • Double-buffered GDI+    │ • IVirtualDesk- │ • SetCapture mouse tracking  ▼
        │ • State memoization       │   topManager    │ • Shell hook window tracker  ┌─────────────┐
        │                           │   Internal      │ • Surface Dial wheel tools   │ Pro/*.exe   │
        └───────────────────────────┴─────────────────┴──────────────────────────────┤ (Pro, C#)   │
                                                                                     │ • SlyPro.exe│
                                                                                     │   (service &│
                                                                                     │   licence)  │
                                                                                     │ • SlyMeet   │
                                                                                     │ • SlyRemote │
                                                                                     │ • SlyTts    │
                                                                                     └─────────────┘

SlyPilot uses an open-core architecture. The core desktop companion (SlyPilot.ahk) is compiled into a 64-bit executable together with the AutoHotkey v2 runtime. It is free, MIT-licensed, and its source is included. The parts of the optional SlyPilot Pro are separate, proprietary programs written in C# (.NET Framework 4.8) that live in the Pro\ directory next to the program: SlyPro.exe (the Pro service and the licence window), SlyMeet.exe (dual-channel meeting recorder), SlyRemote.exe (phone remote server), SlyTts.exe (natural-voice engine), SlyShot.exe (screenshot editor, screen recorder, scrolling screenshots and guides) and SlyHome.exe (the Home Link preview); AI dictation is unlocked by the same licence. The core never contains Pro code and starts a Pro program only after the licence allows that part. Each Pro program verifies the user's signed 30-day lease (SlyPilot.lease, Ed25519, obtained by activating a SLY-... key online and renewed about twice a day while running) or offline licence file (SlyPilot.lic, ECDSA P-256) on the PC by itself. The program has no usage telemetry. Speech models (Whisper, Parakeet) and natural-voice packs are downloaded on request after showing file size and checksum manifests. A separate, tiny client, SlyCtl.exe, sits next to the program for scripts, macro pads, and Stream Decks.

💎 Critical Engineering Invariants

1. Zero Focus Loss Architecture

In Windows, clicking a standard toolbar makes the OS activate that window, deselecting text in your editor and dismissing menus.

SlyPilot enforces zero focus loss across its tool windows (the command palette is the one deliberate exception, because you type into it):

• Extended window styling: tool windows use WS_EX_NOACTIVATE and WS_EX_TOOLWINDOW, or are click-through.
• Message trapping: the window procedure answers WM_MOUSEACTIVATE with MA_NOACTIVATE.
• Target tracking (GetActiveTargetHwnd): window actions (snap, fling, pin, minimise) never rely on "the active window", which could be one of SlyPilot's own windows. SlyPilot tracks the last real application window you worked in and acts on that.
• Own-window recognition (IsAppOwnWindow): hooks, window events and active-target captures check whether a handle belongs to one of SlyPilot's windows before acting.

2. Double-Buffering & Zero-Flicker Rendering

Layered, dark-themed windows can erase their child controls whenever the parent repaints:

• Paint clipping (WS_CLIPCHILDREN): every custom window with controls declares it, so the parent's background paint skips the child areas.
• State memoization: frequent redraw functions such as the Touch Bar update remember the visual state they last drew (current desktop, indicator style, accent, counts, orientation) and return at once if nothing changed.

3. COM Interop for Virtual Desktops (VirtualDesktopAccessor.dll)

Windows' virtual desktops are driven through undocumented internal COM interfaces (IVirtualDesktopManagerInternal):

• Instead of spawning scripts or child processes, SlyPilot loads a small native library into its own process and registers for desktop-change notifications.
• Asking Windows which desktop every window is on is a cross-process call to Explorer, so window lists are cached for a short time, invisible and cloaked windows are filtered out before any desktop query, and live updates come from shell-hook events (window created / destroyed) rather than polling.
• Because the interfaces are undocumented, a big Windows update can break the helper; SlyPilot checks at start and says so plainly.

4. Direct Windows Ink Pointer Handling

Stylus input normally passes through the Windows Ink gesture pipeline, which buffers contact to detect press-and-hold:

• Gesture suppression: SlyPilot sets the tablet-service property (MicrosoftTabletPenServiceProperty, 0x00010019) on its own windows to switch off press-and-hold, flicks and pen-tap feedback.
• Direct pointer messages: hold-to-talk buttons handle WM_POINTERDOWN / WM_POINTERUP themselves, so dictation starts when the pen touches the glass instead of waiting for a buffered mouse click, and a held key is released when the same pointer lifts.
• Stick-proof keys: held keys and buttons are released on exit, reload, hide, pointer-up, and when a phone disconnects.

5. One Writer for the Settings File

All settings live in one human-readable text file (SlyPilot.ini, UTF-16). Two programs writing it at once can damage it, so:

• Only the running program writes it. Short-lived command-line processes (/say, Stream Deck switches, macro pads) hand their changes to the running instance in a window message instead of writing the file themselves.
• A safety net: a few seconds after start SlyPilot checks the file; a damaged one (a repeated section header, stray fragments) is repaired by keeping the first copy of every section, and the damaged copy is kept. A healthy file is copied once a day to %LOCALAPPDATA%\SlyPilot\Backups (outside any synced folder), and the newest seven stay.
• Secrets are never in the settings file in plain text. An AI service key is stored encrypted for your Windows user (DPAPI) and is excluded from exports and diagnostics.

🗣️ The Voice Engine

Every built-in voice command is one entry in a command registry. The registry generates the speech grammar, drives the dispatcher, builds the command guide and carries each command's safety rules, so those cannot drift apart. A recognised phrase is only queued by the speech event handler; wake word, confidence and chain-splitting happen in the queue processor, and the dispatcher checks your own commands first, then the registry. Commands that can destroy something are marked in the registry and ask you to say "confirm". A self-test at start checks that every grammar phrase reaches a handler. The same dispatcher serves the command palette, the phone remote, macro pads (/say) and AI agents, so one safety rule covers every path.

🧠 Local Helpers: Isolation & IPC

SlyPilot does not run large engines inside its own process:

• Separate helper processes: SlyPilot Pro subsystems (SlyPro.exe for the service and licence, SlyMeet.exe for meetings, SlyRemote.exe for phone control, SlyTts.exe for natural voices) and downloaded engine servers (whisper.cpp, CrispASR, llama-server) run in separate processes, placed in a Windows job object so they terminate automatically when SlyPilot exits.
• Local IPC: The core communicates with the Pro C# helpers through structured WM_COPYDATA JSON messages (Windows window messages on your PC, no network port) for UI-critical events.
• Loopback only for network servers: The downloaded speech engines and local language-model server listen strictly on 127.0.0.1 on random ephemeral ports, accepting requests exclusively from SlyPilot (the language-model server also requires a cryptographically random session key passed via protected file).
• Supervision & Process Lifecycle: SlyPilot continuously supervises its helpers; if an engine becomes unresponsive, it is restarted cleanly in the background without affecting active desktop windows or losing work.
• Memory efficiency & idle behavior: The natural-voice engine unloads voice weights after 30 minutes of inactivity; the local LLM server halts after 10 quiet minutes. The core application uses about 60 MB of memory when idle (the Pro service adds about 30 MB only while a licence is installed).
• Cryptographic verification: Models, voices, and dependencies are fetched only after explicit user consent and are verified against hardcoded SHA-256 digests. Pro licences are validated offline via ECDSA P-256 signatures.

✍️ AI Actions: The Request Path

AI actions ("SlyAction") take text from where you are working, send it to a model you chose, and show the answer for you to accept. The path is designed so that a mistake fails safe:

 trigger                    context                  gate                    request                  answer
 ───────                    ───────                  ────                    ───────                  ──────
 hotkey / palette `a ...`   selection (copy),        provider set up?        OpenAI-compatible        checked (empty, far too
 voice "fix that"           or @screen (offline      outside this PC?        chat request to the      long, a refusal, or a
 Touch Bar button           OCR), @clip, @meeting    needs https + your      address you chose;       model "thinking out
 phone AI tab               (never from protected    tick naming the host    streamed to a card       loud" are not accepted
                            windows or private       BEFORE any request      through a short helper   as the answer)
                            clipboards)                                                                       │
                                                                                                              ▼
                                                                     a card: Replace selection / Insert after / Copy / Discard
                                                                     (the answer is only ever pasted as text, never run)

• The model address decides what is allowed. An address on your PC (127.0.0.1, localhost) is local. Anything else must be https and needs a tick, in AI Actions Setup, that names the host; the card names the host on every answer. The check happens before a request is built.
• The selected text is data, not instructions. It is passed to the model between tags, and the answer is treated as text only: it cannot trigger a command, a macro or a keystroke.
• The built-in local model is a small llama.cpp server that SlyPilot starts when you ask for something and stops after a quiet period. It can also serve as a fallback: when your chosen server is unreachable, slow or rate-limited, a request can be answered once on this PC instead, and the card says so. Nothing leaves the PC in a fallback.
• Secrets and logs. The API key is stored encrypted for your Windows user and handed to the streaming helper through its environment, never on a command line. The selected text, the instruction and the answer are never written to the log, and the temporary request file is deleted when the answer ends.
• Model hygiene. Free models that write out their reasoning instead of the answer are detected, skipped and remembered for a few hours; OpenRouter requests carry a fallback list so a busy free model is replaced rather than failing.
• Per-program and per-use settings (a usual action per program, a different model for dictation polish or meeting summaries) change only the action and the model, never where a request goes.

🎙️ Meeting Notes: The Recorder (SlyPilot Pro)

Meeting notes (a SlyPilot Pro part) are built as a separate helper with a short chain of custody:

• You start it, every time. The tray item, the hotkey and the indicator menu are the only ways to begin. No agent verb, MCP tool, command-line switch, macro or voice command can start a recording. The first use shows a consent window; each start shows a notice with a sentence you can read out to the other people.
• SlyMeet.exe captures two streams, your microphone ("You") and the sound the PC plays ("Them"), and cuts them into short pieces in a private temporary folder.
• SlyPilot sends each piece to the local speech engine and deletes the piece as soon as it has been read. No audio is kept, and nothing is sent anywhere.
• The transcript is a plain text file in %LOCALAPPDATA%\SlyPilot\Transcripts (not next to the program, which may be in a synced folder). It never enters the log, a diagnostics report, an exported settings file or the agent interface.
• An echo guard drops a microphone line that repeats a "Them" line from the same moment, which happens when speakers are on.
• A red indicator stays on screen, on every virtual desktop, while recording, and cannot be hidden.
• Summaries go through the same AI model path as AI actions, with the same rules; a model outside your PC is asked about each transcript separately and the host is named.

🔍 The Magnifier, Presenter and Drawing Helpers

The glass, strip, loupe, watched areas, the second-screen view, the presenter tools, the drawing board and the marks an assistant draws live in one small helper program, SlyMag.exe, started on demand and ended with SlyPilot. Its windows are layered, click-through (the board takes input only while it is up) and never take the focus; the glass is excluded from screen captures so that it does not magnify itself. Each helper process (glass, drawing board, presenter, marks) has its own pipe to SlyPilot with buffered writes and a time limit on every request, so a stuck helper can never freeze the program (learned from a real 45-second hang, now covered by a test).

🧰 SlyCtl.exe: The Fast Command-Line Client

Starting SlyPilot.exe just to hand one command to the already-running program means loading and parsing the whole program each time. SlyCtl.exe is a tiny separate client, built next to the program, that finds the running SlyPilot window and delivers /say <voice command> and the speech-button switch directly through a window message; any other switch is passed on to SlyPilot.exe with its exit code. It exists for Stream Deck buttons, macro pads and scripts. See the command-line guide.

📱 Phone Remote: The Network Side (SlyPilot Pro)

The Phone Remote (a SlyPilot Pro part, used with Chrome on Android) is the one part of SlyPilot that listens on a network, so it is built as a small, separate, locked-down helper:

• A separate helper process (SlyRemote.exe, in the Pro folder next to SlyPilot.exe). SlyPilot starts it only when you switch the remote on. It serves the phone page and talks to the phone over one WebSocket per phone. Everything that touches your desktop (typing, clicking, windows, dictation, screen pictures) happens inside SlyPilot, which accepts messages only from its own helper (checked by process).
• Local only. It answers only clients on private network addresses, refuses connections from the PC itself, caps connections per address and in total, and limits message sizes and rates. It never contacts the internet.
• TLS with pairing proof in both directions. The helper makes a self-signed certificate on your PC, kept protected for your Windows user (DPAPI). Pairing uses a one-time secret in the code you scan; the phone and the PC each prove they know it (HMAC), and the phone then receives its own device token, which is stored protected on the PC. A stranger on the same Wi-Fi cannot complete the handshake.
• Per-phone permissions, checked at both ends. Each message type needs a permission (buttons, media, trackpad, keyboard, voice, windows, AI, screen, clipboard, files, power, agent approvals, unrestricted). The helper refuses what a phone may not do, SlyPilot checks again, and a permission change reaches a connected phone at once.
• No stuck keys or buttons. Held mouse buttons and held Touch Bar keys are released when a phone goes silent, disconnects, or the remote stops.
• Zero focus loss, as everywhere. Touch Bar presses from the phone use the same focus-safe path as touching the real bar; typing from the phone goes to the program you work in, and never into SlyPilot's own windows.
• Screen pictures are made by the same capture code as the agent API: protected windows are painted black unless the phone is unrestricted. A live view sends at most five pictures a second, one in flight at a time, nothing while the screen is unchanged, and a zoomed view asks only for the part you are looking at.
• Tested. The remote has its own automated tests (handshake and attacks, permissions, typing, the screen view, the Touch Bar, files, and a check that the phone page runs without script errors in a real browser).

🤖 Agent API & MCP

The agent API and the MCP server (SlyPilot.exe mcp, 15 typed tools) are thin front ends to the running program. They are off until you choose a level in the tray menu; higher levels are time-limited, risky requests need a click on an approval window, and one hotkey pauses every agent and phone. Agents name windows by handle or title, never "whatever is in front", and protected windows (password managers, private browsing, plus any you add) are invisible to them and painted black in screenshots. The classic command-line switches and /say are not covered by the agent gate.

✅ How It Is Tested

• An automated suite of over 300 test cases covers static checks (syntax, a scanner for accidentally-local variables, duplicate case labels, the settings file's encoding, the installer), the voice engine's self-test, the command-line switch table, dictation against golden files, and, in a full run, the live program: the Touch Bar states, microphone mute, workspaces, windows, the agent and MCP surfaces, the phone remote and more.
• Tests run in a throw-away copy of the program. A run copies the program to a temporary folder with a seed settings file, runs everything against the copy and closes it afterwards, so a test can neither fail because of how you have configured SlyPilot nor leave test values in your real settings. The real settings file is hashed before and after, and a change is reported.
• Tests never touch your downloaded engines and models. Anything that deletes, uninstalls or moves files works on a name or folder of its own and proves that the real data folder is unchanged. Installer tests compile their own copy of the setup with another application identity.
• Privacy is tested too: tests scan for code that would log spoken or dictated text, meeting transcripts or selected text, and for AI requests that could leave the PC without the opt-in.