AI Assistants That Point, Draw and Walk You Through Your Own Screen

"Where is the Save button?" is a question an AI assistant can now answer by circling it on your screen. An assistant such as Claude Code, Claude Desktop, Cursor, VS Code (Copilot), Gemini CLI or Codex CLI can put boxes, rings, arrows and notes on your real windows, dim everything except one place, walk you through a task step by step, open a whiteboard and draw an explanation, run the presenter tools for a talk, and read a longer text aloud.

It does this through the MCP server and the command-line agent API that are built into SlyPilot.exe. The marks are drawn on windows that never take a click, a touch or the keyboard focus, so you keep working while the assistant shows you things. And you stay in charge: nothing works until you switch agent control on.

What is free and what is Pro.
• Free: the whole agent side. The MCP server, the command-line agent API, marks and notes on the screen, circling words by OCR, the spot, the tour, the drawing board for agents, the presenter verbs, read-aloud with a Windows voice, saving and listing workspaces.
• SlyPilot Pro: things an assistant can use but that are Pro by themselves: natural voices for reading aloud, recording meetings (an assistant can read the notes of meetings you recorded, but SlyPilot Pro does the recording), and approving an assistant's request from your phone. SlyPilot Pro is the optional one-time purchase.

   You: "Show me where the Save button is."
                      │
                      ▼
   ┌──────────────────────────────┐
   │ AI assistant                 │   Claude Code · Claude Desktop · Cursor
   │ (it looks, then decides)     │   VS Code · Gemini CLI · Codex CLI ...
   └──────────────┬───────────────┘
                  │  MCP (SlyPilot.exe mcp)   or   SlyPilot.exe agent ...
                  ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ SlyPilot gate                                                │
   │  level (off / read / see / control ...) · your Allow clicks  │
   │  protected windows black · scope · pause key · agent.log     │
   └──────────────┬───────────────────────────────────────────────┘
                  ▼
   ┌──────────────────────────────────────────────────────────────┐
   │ Your screen                                                  │
   │  ┌──────── Notepad ────────┐                                 │
   │  │ File  Edit  View        │   ( Save )  <- ring + note      │
   │  │ ...                     │      "Claude: click Save"       │
   │  └─────────────────────────┘   click-through, never takes    │
   │                                 your mouse or your focus     │
   └──────────────────────────────────────────────────────────────┘

✨ At a Glance

🎯 Pointing: Marks That Never Take a Click

An assistant can put these on your screen:

How it behaves:

• Click-through. The marks sit on windows that never take a click, a touch or the focus. Your mouse and your pen work on the program underneath.
• They fade by themselves. After 10 seconds unless the assistant asks to keep them (a kept mark ends after 30 minutes anyway).
• Every batch carries the assistant's name in a small chip, so you always know whose marks they are.
• They are hidden while a SlyPilot window is in front. An approval window or a dialog is never covered, so a mark can never dress up a fake Allow button.
• At most 60 marks at once. Up to 20 marks go in one request, so a whole picture is one action.
• They are on the screen, so a screenshot you take contains them.

Circle what I name

The assistant does not have to guess pixels. around looks for words on the screen with Windows OCR, or for a control by its name through UI Automation, and puts a mark round it.

You: "Circle the word Save in Notepad."
The assistant finds the window, reads it, and draws a ring with the label "Save". If the word appears twice it asks which one, or circles both.

Windows OCR must have an engine for the language on screen. Windows that are protected (below) are refused.

🚶 A Tour and a Spot

Walk me through exporting this as a PDF becomes a tour: a list of steps, each a few seconds long, each with its own marks. One request starts it, SlyPilot shows the steps by itself, and each step replaces the one before.

  step 1 (5 s)            step 2 (5 s)            step 3 (6 s)
  ┌─────────────┐         ┌─────────────┐         ┌─────────────┐
  │ [File] <-1  │   ->    │ File menu:  │   ->    │ Save as:    │
  │             │         │ [Export] <-2│         │ [PDF]  <-3  │
  └─────────────┘         └─────────────┘         └─────────────┘

• Up to 12 steps, 5 minutes in all, up to 30 seconds per step.
• Say "clear agent drawings" at any moment and the tour ends.
• For a single place the spot is stronger than a ring: the screen goes dark except a round or square hole where you should look.

🖍️ The Drawing Board as a Whiteboard

"Draw me a diagram of how this works." The assistant can open SlyPilot's drawing board on your screen and draw on it.

You click Allow first. The first time an assistant asks to open the board, an approval window appears. Nothing opens before your click on Allow once.

Then:

• The toolbar says who is drawing: "Drawn with Claude Code" (or the name the assistant gave).
• The assistant draws real strokes. You can undo, move and erase them like your own.
• It can add pages (up to 30), change paper (plain, grid, lines) and background (screen, white, black), and look at the page as a picture to check its work.
• It can save the pages as a PNG per page and one PDF in your screenshot folder.
• It is your board. It takes your mouse and pen until you press Done (or Esc); you can take over at any time.
• Your own strokes are protected. If you drew since the assistant's last request, its undo and clear are refused (user_drew) unless you told it to go ahead.
• Pausing agent control closes the board. The next time it asks again.

🎤 For a Talk or a Lesson: the Presenter Tools

Assistants can run SlyPilot's presenter tools:

• Presenting mode on or off, with an optional countdown in minutes. It uses your Presenting Mode settings (keep the PC awake, highlight the pointer, quiet notifications, and so on).
• Countdown on the screen.
• Show my shortcuts: the audience sees Ctrl+S, Alt+Tab and the like (never what you type).
• Spotlight and pointer highlight.
• Spot a place for the audience.
• Read here: reads aloud what is under the pointer.

Pausing agent control ends a presentation the assistant started.

🔊 Reading Aloud, Meeting Notes and Workspaces

Narration

An assistant can read a longer text aloud, up to 3000 characters, in the voice you chose for reading aloud. The screen shows the first 160 characters. Together with a tour this is "show and tell": the voice explains while the marks move. A Windows voice is free; natural voices are Pro.

Your recorded meeting notes

If you recorded meetings with SlyPilot Pro's meeting notes, an assistant can list the notes files and read one. Each file needs your click on Allow: the window names the file and its size, because these are your colleagues' words. It reads in pieces (20000 characters by default). It cannot start a recording: that stays with you, with its consent question and red indicator. It can stop one that runs.

Workspaces

The assistant can list your saved workspaces, save the current layout as one, and open one. Opening a workspace moves windows and starts programs, so it needs the system level and your click.

🛡️ What You Stay in Control Of

1. It is off until you say so. Tray menu, Agent Control. Nothing works at level off, except a status question.
2. You pick the level. Each includes the ones above:

1. Drawing has its own switch. Tray, Agent Control, Allow agents to draw on the screen. Turn it off and every draw request is refused.
2. Approvals. Risky requests (the board's first opening, reading a meeting file, launching a program) show a small window with Allow once and Deny. It never takes keyboard focus, so a stray key cannot approve.
3. The assistant's name is on every mark. The name is not verified: it is what the program calls itself.
4. Clear agent drawings. Tray, Agent Control, or say "clear agent drawings". It ends a tour too.
5. Pause everything. Ctrl+Alt+Shift+F12, the tray, or say "pause agent control". Marks go, the board closes, a presentation the assistant started ends.
6. Protected windows. Password managers, private browsing windows and anything you add are invisible to the assistant and painted black in what it sees, including a board page made from the screen.
7. Scope (experimental). Share only some windows with the assistant.
8. A log, and a window to read it. logs\agent.log: what was asked, never what was typed. Tray → Agent Control → Agent Activity... ("show agent activity") lists it in plain words, in sessions per assistant, with a Pause button.

What SlyPilot sends: nothing. There is no telemetry in the program. But an assistant that takes a screenshot passes it to its own AI service, as it would any picture you give it. That is between you and the assistant's provider. The protected-window rule and your level are how you limit what it can look at.

💬 What to Ask

In Claude Code and other clients that show MCP prompts as slash commands, the ready-made recipes are there too (see the guide).

🖥️ The Same Thing From a Script

Everything above is also a command, for assistants that live in a shell and for your own scripts. Each call prints one line of JSON and goes through the same gate.

# circle the word "Save" in a window whose title contains "notes"
.\SlyPilot.exe agent draw around --text "Save" --title notes --kind ring --label "click here" --as "My script"

# dim everything except a round place at x 900, y 500, radius 60
.\SlyPilot.exe agent draw spot 900 500 60 --ttl 15

# take the marks away
.\SlyPilot.exe agent draw clear

The classic command-line switches (SlyPilot.exe snap left) are a different door and are not behind the agent gate. The agent commands are.

🧭 A Short Conversation

1. You ask: "Walk me through turning on dark mode in this program."
2. The assistant asks SlyPilot for its own status. If agent control is off it shows you a message saying which level it needs, and waits. It does not look for another way in.
3. At level See or higher it looks: the list of windows, a small screenshot of the one that matters, the words on it.
4. At level Control it starts a tour of four notes with numbered circles. You follow along and keep clicking in your own program.
5. You say "clear agent drawings" when you are done, or let them fade.

If a step needs your approval (the board, a meeting file), a small window asks and the assistant waits for your click instead of retrying.

⚠️ Honest Limits

• It is as good as the assistant. SlyPilot gives it the tools; a model can still pick the wrong word. Look at the circle. A mark that is wrong is easy to clear.
• Marks sit on screen pixels. They do not follow a window that you move. A short tour on a window you are not touching works best.
• OCR and UI Automation have limits. OCR reads text only (no icons), in the languages Windows has an OCR engine for. Chrome and Electron programs build their control tree lazily. Programs run as administrator cannot be read.
• Only 60 marks at once, 20 per request, 12 tour steps.
• The board takes your mouse until you press Done. A good assistant asks first.
• Real-hardware tests. The agent drawing, the board and the MCP server were tested with the program's automatic tests, and Claude Code connects to the server and sees its tools. Other clients, pen and touch on the board with an assistant, and monitors with different scaling were only partly tried.
• Prepared, not yet tried: the Codex CLI, Windsurf and Zed snippets and the Claude Desktop .mcpb extension are prepared but have not yet been tried in those programs.

Next Steps

• Set it up step by step: the AI assistant guide →
• MCP server and the command-line agent API →
• The drawing board → · Presenter tools →
• Teachers and trainers →
• Meeting notes (SlyPilot Pro) → · Privacy → · SlyPilot Pro →