AI Assistant Guide: Let an Assistant See, Point, Draw and Present on Your Screen

A step-by-step guide to AI Assistants That Point and Draw and to the MCP server behind it.

• Free: all of it. The MCP server, the command-line agent API, marks, the tour, the spot, the board, the presenter verbs, read-aloud with a Windows voice, workspaces.
• SlyPilot Pro: only the parts that are Pro by themselves: recording meetings (an assistant can read the notes you recorded), natural voices, approving requests from your phone. See SlyPilot Pro.

Agent control is off until you choose a level in step 1. SlyPilot sends nothing anywhere; an assistant passes what it sees to its own AI service, as it would with any picture you give it.

1. Turn Agent Control On

1. Right-click the SlyPilot tray icon and open Agent Control.
2. Pick a level. For a first try choose Read only (look, never touch); to let the assistant draw, choose Control windows and desktops.

Input and System are asked for with a time (15 minutes, 1 hour, until SlyPilot restarts, or until you turn it off) and are never kept across a restart.

1. Check that Allow agents to draw on the screen is ticked if you want marks and the board.
2. Keep the stop in mind: Ctrl+Alt+Shift+F12, or say "pause agent control", stops every assistant at once.

2. Connect an Assistant

Run this in a terminal. It prints the setup text for each client, with the real path of your SlyPilot.exe:

.\SlyPilot.exe mcp-config

Or use the tray menu: Agent Control, Copy MCP setup for Claude Code.

Claude Desktop and most clients:

{
  "mcpServers": {
    "slypilot": {
      "command": "C:\\Path\\To\\SlyPilot.exe",
      "args": ["mcp"]
    }
  }
}

VS Code:

{
  "servers": {
    "slypilot": {
      "type": "stdio",
      "command": "C:\\Path\\To\\SlyPilot.exe",
      "args": ["mcp"]
    }
  }
}

Zed:

{
  "context_servers": {
    "slypilot": {
      "command": { "path": "C:\\Path\\To\\SlyPilot.exe", "args": ["mcp"] }
    }
  }
}

What has been tried. Claude Code connects to the server over stdio, sees the tools and the resources, and closes cleanly. The Codex CLI, Windsurf and Zed snippets and the Claude Desktop .mcpb extension (one click: powershell -File tools\make_mcpb.ps1 in the project builds it) are prepared, but have not yet been tried in those programs. If one does not work, send the output of mcp-config and the client's log to support.

The MCP server starts SlyPilot if it is not running. The first call may take a few seconds then.

3. First Checks

Ask the assistant: "Call slypilot_state with status true and tell me what you may do."

status: true is answered at every level, even Off. It says:

• your level,
• whether you are paused,
• whether drawing is allowed,
• the scope.

Then ask: "List my windows." At Read level that works. If you see agent_off, go back to step 1.

4. The 20 Tools

Kept few on purpose, because every tool description costs the assistant context on every turn.

Resources (things a client can read): slypilot://voice-commands, slypilot://agent-api (the rules), slypilot://recipes, slypilot://cli-commands, and three live ones, slypilot://status, slypilot://state and slypilot://windows.

5. Prompts: Ready-Made Recipes

Many clients show MCP prompts as slash commands. Each one is a short recipe that names the tools to use, in order, and the rules that matter.

In Claude Code, type / and look for slypilot. Where a client has no prompts, just ask in plain words.

6. Pointing, Touring and the Spot

Needs level Control (circling words by OCR also needs See, because it reads the screen).

Try:

1. "Circle the word Save in Notepad." The assistant lists windows, finds Notepad, and draws a ring with a label.
2. "Walk me through printing this page." A tour of notes and numbered circles, a few seconds each.
3. "Dim everything except the Share button." The spot.

What to expect:

• Marks fade after 10 seconds. Ask the assistant to keep them, or say "clear agent drawings" to remove them.
• Every batch carries the assistant's name.
• You can click through the marks.
• If a SlyPilot window is in front (an approval window), the marks are hidden until it goes.
• A tour: up to 12 steps, 5 minutes. A mark does not move when you move the window.

7. The Drawing Board

1. Ask: "Draw a diagram of how my three services connect." A good assistant asks first, because the board takes your mouse and pen.
2. An approval window appears: Allow once or Deny. Click Allow once.
3. The board opens. The toolbar says Drawn with .
4. The assistant draws boxes, arrows, text and numbers as real strokes. You can undo, move and erase them. You can draw too.
5. If you drew, its undo and clear are refused until it has looked at the page again (or you tell it to go ahead). Your work is protected.
6. Press Done or Esc when you are finished. With more than one page the pages are saved as a PNG each and one PDF in your screenshot folder.

Pausing agent control closes the board at once; the next opening asks you again.

More about the board itself: drawing board and its guide.

8. Presenting

Ask: "Set my screen up for a 30 minute talk." The assistant uses slypilot_present:

A presentation the assistant started ends if you pause agent control. See presenting guide.

9. Meeting Notes and Workspaces

Meeting notes. Recording meetings is SlyPilot Pro. If you have notes, ask: "Summarize the last meeting." The assistant lists the files, asks to read one, and a window names the file and its size. You click Allow for each file. It reads in pieces and cannot start a recording.

Workspaces. "What workspaces do I have?" lists the layouts you saved. "Save this layout as Writing" saves one. "Open Writing" moves windows and starts programs, so it needs the "Everything" (System) level and your click.

Reading aloud. "Read me this summary" uses slypilot_notify with aloud: up to 3000 characters in the voice you chose for reading aloud.

10. Approvals, Cancelling and Pausing

• Approval window. A risky request shows Allow once and Deny. It never takes the keyboard focus. It covers exactly that request, once. The assistant's name in it is not verified.
• From your phone. With a paired phone (SlyPilot Pro) the same request shows as a card on your phone.
• Cancelling. If the assistant cancels a call that is waiting for you, the approval window or the question goes away.
• A question. slypilot_ask shows a card with its choices. Your answer is only information; it never approves anything.
• Pause. Ctrl+Alt+Shift+F12, the tray item Pause agent control, or say "pause agent control". Marks go, the board closes, a started presentation ends. SlyPilot also pauses itself if requests run away.
• Protected windows. Password managers and private browsing windows (and what you add under Protected windows...) are black in everything an assistant sees.
• Scope (experimental). Tray, Agent Control, Scope: the assistant sees only windows you share.
• The log. Tray, Agent Control, Open agent log: what was asked, never what was typed.

11. The Command Line, for Scripts

Everything the tools do is also a command: one line of JSON back, an exit code, the same gate.

.\SlyPilot.exe agent status
.\SlyPilot.exe agent windows --exe notepad
.\SlyPilot.exe agent draw around --text "Save" --exe notepad --kind ring --label "click here" --as "My script"
.\SlyPilot.exe agent draw note 800 400 Click here first --keep
.\SlyPilot.exe agent draw spot 900 500 60 --ttl 15
.\SlyPilot.exe agent draw tour --file steps.json
.\SlyPilot.exe agent draw clear
.\SlyPilot.exe agent board open --bg white --paper grid
.\SlyPilot.exe agent present on --minutes 20
.\SlyPilot.exe agent meeting list
.\SlyPilot.exe agent notify "Reading the summary" --aloud

--as "<name>" is the name shown on the marks and in the approval window. Exit code 3 means an approval is needed: agent approval <id> tells its state, then the identical request is sent again with --approval <id>. Details: command line and Stream Deck.

12. Troubleshooting

More help: troubleshooting and the Doctor · MCP and agent automation.