Microsoft’s Skill Recorder: Record a Task Once, Let Your AI Agent Repeat It

Do you ever find yourself doing the same sequence of clicks, forms, and file uploads every week? You know the steps by heart, but automating them feels like more work than just doing them manually again. Microsoft recently open‑sourced a tool called Skill Recorder that tackles this from a different angle.

Instead of giving you another screen recorder or a fragile macro tool, Skill Recorder captures your screen and actions while you work, then uses GitHub Copilot to turn that recording into a structured procedure. That procedure becomes a skill that your AI agent can run on demand – or an automation that runs on a schedule.

The key insight is simple: you demonstrate the task once, and the AI learns to do it – not by replaying your mouse clicks, but by understanding your intent and using its own tools to achieve the same outcome.

How This Differs from Screen Recorders and RPA Tools

It’s easy to mistake Skill Recorder for yet another screen‑capture app like Loom or OBS. Those tools produce videos that humans watch to learn. Skill Recorder produces a structured, machine‑readable document that AI agents can execute.

And compared to traditional RPA (Robotic Process Automation) tools like UiPath – which record exact screen coordinates and keystrokes – Skill Recorder takes a completely different approach. RPA scripts are precise but brittle: move a button a few pixels, and the script breaks. Skill Recorder never stores coordinates. It stores what you accomplished, and then lets the AI decide how to accomplish it.

For example, if you record yourself creating a GitHub issue through a browser, the generated skill will likely use the gh CLI or the GitHub API instead of simulating a browser session. API calls are more stable, faster, and immune to UI changes. Even better, the AI generalises from your single demonstration: you showed it how to create one issue, but it learns how to create any issue with different content – no re‑recording needed.

The Four‑Step Workflow from Recording to Execution

Skill Recorder’s workflow is straightforward and linear. Let’s walk through each stage.

Step 1: Record

Launch the app and hit the big red record button, or use the global shortcut ⌘⇧R (macOS) / Ctrl+Shift+R (Windows). A small, always‑on‑top control bar appears; it stays visible even when you switch between applications.

During recording, you can:

• •
  
  Pause and resume at any time.
• •
  
  Mute/unmute your microphone.
• •
  
  Switch between audio input sources.
• •
  
  Stop and save the session, or discard it with a confirmation prompt (to prevent accidental loss).

Behind the scenes, Skill Recorder logs much more than just screen pixels:

• •
  
  Active application and window switches.
• •
  
  Browser URLs (currently on macOS) – the pages you visit.
• •
  
  Clipboard content previews (short snippets) to help connect steps.
• •
  
  Optional voice narration if you have the microphone enabled.

All this data stays local on your machine until you explicitly click Analyze. Nothing leaves your computer during the recording phase.

Step 2: Control and Review While Recording

The floating control bar gives you full control without interrupting your flow. You can check the recording timer, microphone status, and adjust settings – all without jumping back to the main window. If you make a mistake, you can discard the session; the app asks for confirmation before deleting, so you don’t lose good takes by accident.

Step 3: Analyze – Let Copilot Turn Your Recording into Steps

When you finish recording, click the Analyze button. Skill Recorder packages the recorded timeline (window titles, URLs, clipboard previews, screen snapshots, and transcribed voice) and sends it to GitHub Copilot for processing.

This is the only step that requires an internet connection and a Copilot‑enabled GitHub account. Copilot processes the data and produces two things:

1. Intent – a one‑sentence summary of what you accomplished.
2. Ordered Steps – a numbered list of high‑level actions, written in descriptive language (e.g., “Click ‘New Issue’”) rather than coordinate‑based instructions.

The output is intentionally abstract: it describes what you did, not how your mouse moved. This abstraction makes the steps understandable, editable, and reusable.

Once the analysis appears, you can freely edit the intent and the steps. The AI usually gets about 80% of the structure right – you fine‑tune the remaining 20% to ensure accuracy and completeness.

Step 4: Create – Generate a Skill or an Automation

When you’re satisfied with the analysis, you can generate one of two outputs:

• •
  
  Skill – a SKILL.md file containing the intent and step list. Any compatible AI agent can read this file and execute the procedure on demand.
• •
  
  Automation – the same procedure, but with an added trigger: a schedule (e.g., every Monday at 9 AM) or an event‑based trigger.

The underlying content is identical – only the invocation method changes. Skills are “run when I say”, while Automations are “run when the clock strikes”.

Step 5: Execute – Let the AI Do the Work

Now comes the payoff. Next time you need that task done, simply tell your AI agent “run the ‘create issue’ skill”. The agent reads the SKILL.md file, selects the appropriate native tools (CLI, API, or browser automation), and performs every step without your intervention. You only need to verify the final result.

Installation: One‑Line Commands for macOS, Ubuntu, and Windows 11

Skill Recorder is distributed as a source release. Instead of a pre‑compiled installer, you get an installation script that fetches the exact source revision for a given release, then builds the app on your machine. This approach guarantees transparency and reproducibility.

macOS and Ubuntu

Open a terminal and run (replace <40-character-release-commit> with the actual commit hash from the latest release):

commit="<40-character-release-commit>"
curl -fsSL "https://raw.githubusercontent.com/microsoft/skill-recorder/$commit/install.sh" | SKILL_RECORDER_COMMIT="$commit" bash

On macOS, this adds a Skill Recorder (Source) app to ~/Applications. You can launch it from Spotlight, Launchpad, or the Dock.

If you want the app to stay running after closing the terminal, add the detached flag:

commit="<40-character-release-commit>"
curl -fsSL "https://raw.githubusercontent.com/microsoft/skill-recorder/$commit/install.sh" | SKILL_RECORDER_COMMIT="$commit" SKILL_RECORDER_DETACHED=1 bash

Ubuntu users follow the same command; it adds a corresponding application entry.

Windows 11 (PowerShell)

In PowerShell, run:

$commit="<40-character-release-commit>"
$env:SKILL_RECORDER_COMMIT=$commit
irm "https://raw.githubusercontent.com/microsoft/skill-recorder/$commit/install.ps1" | iex

This creates Skill Recorder (Source) shortcuts on your Desktop and in the Start Menu. Both x64 and ARM64 architectures are supported.

First Launch: Don’t Forget the Permission Step

On macOS, the first launch triggers a system prompt asking for Screen Recording permission. This is required because the underlying Chromium engine captures the screen. Without it, recording won’t work – grant the permission and you’re good to go.

The first time you click Analyze, you’ll be guided through signing in to GitHub and authorising Copilot if you haven’t already. After that, the app remembers your credentials.

Privacy: What Stays Local and What Goes to the Cloud

Privacy is a legitimate concern for any tool that records your screen. Skill Recorder draws a clear line between local storage and cloud processing.

During Recording and Storage (all local)

• •
  
  Window tracking: active app and window titles.
• •
  
  Browser URLs: pages you visit (macOS only at the moment).
• •
  
  Screen video: captured by Chromium, with low‑rate snapshots kept only when the screen changes or a heartbeat timer fires.
• •
  
  Clipboard previews: short text snippets that help connect steps.
• •
  
  Voice narration (optional): recorded via microphone.

None of this leaves your computer until you explicitly choose to analyse.

During Analysis (sent to the cloud)

When you click Analyse, the following data is transmitted to GitHub Copilot:

• •
  
  The event timeline (window titles, document titles, URLs, clipboard previews).
• •
  
  Extracted screen images (key frames).
• •
  
  Transcribed voice text (the transcription itself is generated locally – more on that below).

Voice transcription is on‑device. Skill Recorder uses the Whisper model; it downloads a ~252 MB model file on first use, and all subsequent transcription runs entirely on your machine. The raw audio is never uploaded.

The Hard Rule: Never Record Secrets

The app shows a clear warning before every recording: don’t record, type, paste, or narrate passwords, access tokens, API keys, credentials, or any confidential information. This isn’t boilerplate – since the screen content is captured, anything visible or audible during the session becomes part of the recording. If your task involves sensitive data, use a test environment or dummy data instead, then edit the analysis results afterwards.

Why Source‑Based Distribution?

Choosing a source release over pre‑built binaries has three main advantages:

1. Integrity – you run exactly the same code that’s in the open‑source repository; no opaque binary blobs.
2. Auditability – you can inspect the source at any time to verify behaviour.
3. Customisability – you can fork the repo, modify the code, and build your own version.

Each release pins a specific commit hash. The installation script fetches that commit and builds from it, ensuring that the version you install today is identical to what anyone else installs tomorrow.

The first‑time installation downloads a fixed Node.js runtime (currently Node.js 24) and runs npm ci to install dependencies, all within a local scope – it doesn’t interfere with any existing Node.js setup on your system.

For Developers: Running from Source and Running Evals

If you want to hack on the code or run the test suite, clone the repository and check out the desired revision. Then:

npm ci
npm run compliance:licenses
npm run dev

npm run dev starts the Vite development server and launches the Electron app with hot‑reload enabled. The global recording shortcut (⌘⇧R / Ctrl+Shift+R) works in dev mode too.

Evaluation Suite

Skill Recorder includes a fixture‑based evaluation harness for the Copilot describer and builder components:

npm run eval            # scores the describer against synthetic recordings
npm run eval:builder    # scores the skill/automation generalisation quality

The eval suite uses fixed test data and doesn’t require external dependencies – useful for validating changes before a release.

When to Use It – and When to Avoid It

Skill Recorder shines in certain scenarios, but it’s not a universal hammer.

Good fits

• •
  
  Repetitive manual workflows – weekly reports, data exports, form submissions – record once, let the AI handle future runs.
• •
  
  Process documentation – instead of writing a long manual, record a session, generate the steps, and you have a draft that’s already structured.
• •
  
  Personal productivity – turn your own frequent routines into skills; trigger them with a single command.
• •
  
  Automation discovery – if you’re planning to write scripts, Skill Recorder can quickly surface the logical steps you take, giving you a head start.

Poor fits

• •
  
  One‑off or highly variable tasks – if the process changes each time, recording a single example won’t capture the variability.
• •
  
  Sensitive data handling – if passwords or personal information appear on screen, it’s risky to record that session.
• •
  
  Tasks requiring complex decision‑making – the tool is designed for deterministic, step‑by‑step procedures, not for processes that require frequent human judgement.

Practical Tips for Your First Recording

If you’re ready to give it a spin, here are a few tips to make your first session smooth:

1. Start small. Pick a task that takes 3–4 steps – e.g., open a web page and take a screenshot. Walk through the full flow (record → analyse → generate → execute) so you understand the output format before tackling a complex workflow.

2. Clean your screen. Close unrelated windows and turn off notifications. The fewer distractions on screen, the cleaner the analysis will be.

3. Edit the analysis result. The AI gives you a solid skeleton, but you know the nuances. Add conditional branches, timeout handling, or edge‑case notes that the recording couldn’t capture.

4. Test the skill in a sandbox. Run the generated skill in a test environment first. Once you’re confident it works correctly, move to production.

Frequently Asked Questions

What’s the difference between Skill Recorder and a regular screen recorder?
A regular screen recorder produces a video for human viewing; Skill Recorder produces a structured procedure that AI agents can execute.

Does the analysis always require an internet connection?
Yes – clicking Analyse sends the event timeline and screen snapshots to GitHub Copilot. This is necessary for the AI processing.

Where does voice transcription happen?
On your machine. Skill Recorder downloads the Whisper model (~252 MB) once and performs all transcription locally.

Can I use generated skills with AI agents other than GitHub Copilot?
The SKILL.md file is plain Markdown, so any agent that understands that format could potentially use it. However, the tool is designed primarily for the Copilot ecosystem.

Are all features identical on Windows and macOS?
Core functionality is the same. Browser URL tracking is currently available only on macOS, with Windows support being worked on.

Where is the recording data stored?
It stays in a local directory until you analyse or delete it. Check the project documentation for the exact location.

Do I need a Copilot subscription?
Yes. The analysis step requires GitHub Copilot access. If you don’t have it, you won’t be able to use the analysis feature.

One‑Page Summary

Final Thoughts

Skill Recorder doesn’t try to be a universal automation platform. It does one thing well: bridge the gap between “how a human does a task” and “how an AI can repeat it”. The approach is refreshingly practical – you demonstrate once, the AI learns the pattern, and you stop wasting time on repetitive clicks.

If you have a recurring task and a Copilot subscription, invest half an hour to try it out. Record a simple workflow, generate the skill, and watch the AI execute it. The first time it works, you’ll see the value immediately – and you’ll probably start thinking about which other tasks you can hand off next.