Code2Skill Explained: Turning GitHub Source Code into a Million-Record Agent Skill Bank

Skill augmentation is the cheapest way to give an AI agent domain knowledge it doesn’t have in its weights. Skills can be updated, versioned, and deployed independently, with no retraining involved. The hard part has always been sourcing them. Code2Skill’s answer is source code: it mines 19,769 actively maintained GitHub repositories and distills them into 1,006,822 evidence-backed, auditable skill records — built offline, before the target agent runs a single task. This article walks through the construction pipeline, the verification gate that keeps bad records out, and the measured results across nine model settings and eight benchmarks, separating what you can adopt directly from what deserves a question mark.

Where Should Agent Skills Come From?

Source code occupies a position trajectories and documents can’t: it exists before any agent interaction, and it stays inspectable after abstraction. That’s the whole reason this pipeline starts from repositories rather than execution logs or manuals.

Trajectory-based methods distill skills from execution history — successful episodes, failures, recurring workarounds. This supports self-evolving agents, but the skills end up welded to whatever produced them. Change the model, the task distribution, the toolset, or the harness, and distilled skills can go stale overnight. Their quality ceiling is also the generating agent’s own competence: it only knows what it has done.

Document-based methods avoid trajectory dependence by extracting from static text. The problem is that documents describe intent and conventions, with no concrete implementation sitting behind the claims. Whether the procedure actually works, and what it does when it fails, remains unverifiable.

Source code has both properties. It predates downstream interaction, records real control flow, state changes, constraints, and failure paths, and code as a medium natively supports execution, evaluation, verification, maintenance, and versioning. GitHub has accumulated an enormous supply of procedures that were implemented, debugged, and refined over years.

Automatic distillation is still not trivial. Real code mixes transferable procedures with framework glue, project-local identifiers, duplicated implementations, implicit dependencies, and edge cases the author never resolved. A qualified skill has to answer five questions clearly: when does the procedure apply, which steps are essential, which invariants must hold, which failures change the execution path, and which generalizations fall outside its scope.

Reading this closely, the gap between “writing a code summary” and “extracting a skill” lives exactly in those five questions. A summary describes what the code does. A skill pins down applicability and failure paths, or the downstream agent following it won’t even know where it went wrong.

In short: trajectories bind skills to past experience, documents lack execution evidence, and source code offers both availability before interaction and inspectability after abstraction.

How Does the Code2Skill Pipeline Work?

Four stages: select behavior-rich source units, lift them into typed skill records, verify each candidate through blind reconstruction against the original implementation, then store accepted records with their evidence and build retrieval views.

Stage 1 — Select source units. The pipeline scans GitHub repositories available by April 14, 2026, keeping only projects with more than 500 stars, for a source pool of 19,769 repositories. Within each repository it parses functions, methods, command-line entry points, and file-level components, then an LLM tagger scores every unit against six signals:

• Does it implement a meaningful, reusable operation beyond project-local glue?
• Are there ordered steps, branching, state or resource updates?
• Are boundary conditions, invariants, and failure handling visible?
• Can the behavior be described without project-specific names?
• Are inputs, outputs, side effects, and required context recoverable?
• Is it more than a getter, thin wrapper, or boilerplate adapter?

Units that clear the configured gate and the per-repository cap move forward. This stage only nominates candidates — no acceptance decisions happen here.

Stage 2 — Extract typed records. The extractor maps each selected unit to one of three granularities. Atomic skills capture a single well-defined operation inside one function or method. Composite skills capture ordered workflows coordinating multiple operations. Recurring-pattern skills capture higher-level implementation practices beyond a single localized operation. Three tiers exist for a practical reason: a single granularity would either fragment multi-step procedures or overgeneralize local behavior. Candidates where worth_extracting is false or skill_value_score falls below 0.45 get filtered out. The core schema looks like this:

{
  "level": "atomic | composite | pattern",
  "name": "<short skill name>",
  "summary": "<behavioral summary, under 80 words>",
  "inputs":  ["<input expectations>"],
  "outputs": ["<outputs or side effects>"],
  "workflow": ["<ordered steps>"],
  "invariants": ["<invariants>"],
  "error_cases": ["<failure and edge cases>"],
  "implementation_notes": ["<constraints>"],
  "evidence": ["<behaviors directly supported by the code>"],
  "anti_goals": ["<behaviors to avoid>"],
  "confidence": 0.0
}

Each array holds at most 6 items of 25 words or fewer, and the JSON must not quote source code. Records are still candidates at this point.

Stage 3 — Blind reconstruction and comparison. This is the most interesting part of the design. The reconstructor receives only the skill record plus interface metadata — language, symbol, signature. The source body, repository, and file path are hidden, and it must regenerate the code from the record alone. A source-aware judge then puts the reconstruction side by side with the original implementation: sufficiently consistent candidates are accepted directly, inconsistent ones go to an adjudicator that distinguishes a flawed skill description from a weak reconstruction run. If the skill itself holds up, it’s accepted through adjudication; otherwise it’s rejected.

The design swaps “does this description sound right” for “can an independent model reproduce equivalent behavior from this description alone.” Vague descriptions leak immediately — steps go missing, failure handling drifts. The cost is one extra generation-plus-judge pass per candidate; the payoff is that every accepted record carries a reviewable acceptance trace.

Stage 4 — Store and index. Accepted records retain their repository, file, symbol, source span, decision rationale, and construction status. Feature tagging builds a task-facing retrieval view for each record with one-to-one correspondence to the evidence archive. Purpose indexing runs a separate deterministic pass — clustering by task family, intent action, and intent target, then selecting one existing record per cluster as the representative. No LLM synthesizes new cluster-level records anywhere in that pass.

In short: the quality gate sits at verification, and every accepted record has passed blind reconstruction — the line that separates this from an ordinary code-summary corpus.

What Does a Real Skill Card Look Like?

One concrete example beats ten principles. The card below, recoverDoQAfterCachedQUICFailure, is a composite workflow taken from AdGuard’s open-source DNS proxy repository. It handles a nasty real-world failure: a DNS-over-QUIC resolver reuses QUIC connections, but cached connections silently die after server restarts, idle closes, stateless resets, NAT breakage, or 0-RTT token rejection.

Its control rule has three clauses: retry only failures from a cached connection; close the failed connection before reacquiring; clear the token store only for Err0RTTRejected.

The workflow: obtain a cached or freshly opened QUIC connection, pack the DNS query, open a stream with a deadline, write the length-prefixed message, close the write side, and read the response. If a cached exchange fails, close that connection, reset token state only on 0-RTT rejection, reacquire, and retry once.

Invariants and failures are equally pinned down: never retry a freshly opened connection that failed; never leave a failed cached connection in the cache; restore DNS message IDs; surface bootstrap, dial, stream, read, parse, and final-retry errors.

The anti-goals list four: no unbounded retry loop, no global resolver reset, no silent error swallowing, no token-store reset for unrelated transport errors.

The transferable value here is not “retry on error.” It’s a bounded recovery policy that pins down what may be retried, what must happen before retrying, and the one condition under which token state gets touched. Code written against this card behaves very differently under real failures — server restarts, NAT switches — than code that wraps everything in a blanket retry. The source-evidence field even links to specific symbols at commit f35ca3e: the Exchange retry branch, openStream, closeConnWithError, and isQUICRetryError. Audit is one click away.

Those anti-goals would slot straight into a code review checklist, honestly. Silent error swallowing and unbounded retries are two of the most common ways production code rots.

How Much Does the Skill Bank Improve Agent Performance?

Across 72 protocol-matched comparisons, 57 improved. The macro-average rose from 42.90 to 47.90 — an 11.7% relative gain. The improvement is broad but not uniform, and the tables below show both facts.

Table 1: Eight-benchmark averages across nine model settings (unweighted mean)

Every model-setting average improves, with relative gains from 6.0% to 20.7%. The default experimental retrieval pool is a random 10% sample of all available records, for computational efficiency.

Table 2: Improvement count per benchmark

The math and software engineering benchmarks are nearly a clean sweep. The boundary shows up on BigCodeBench under reasoning mode and AgentBench-OS. The explanation is straightforward: on short, self-contained problems that capable models already solve directly, an injected procedure has nothing to grab onto. Repository navigation and multi-step interaction, by contrast, hand the stored workflows, invariants, and error cases direct leverage. Structured checks also transfer beyond coding — the AIME, HMMT, and GPQA gains say so.

Head-to-head against trajectory-derived skill banks

Four skill banks, one shared agent loop, one downstream interface, evaluated with DS4-Flash reasoning, five runs reported as mean ± standard deviation (the baselines were built with Qwen3.5-397B-A17B on task splits that don’t overlap evaluation):

Code2Skill ranks first on all seven shared benchmarks, averaging 49.5. Even an oracle that picks the strongest trajectory baseline per benchmark only reaches 40.1 — 9.5 points behind. Per-benchmark, it beats the best trajectory method by 6.6 to 13.3 points. The same interface produced these gaps, so the advantage belongs to the skill bank itself, not the agent loop.

In short: the gains are wide but conditional. Short self-contained tasks see little benefit; multi-step tasks and review scenarios are the home turf.

Where Should Skills Enter the Agent Workflow?

The same record can play four different roles depending on where it enters: constraining a plan before execution, sitting in the generation prompt, serving as a checklist after a draft exists, or feeding policy, verifier, or reviewer during coding RL. Placement determines what the knowledge actually does computationally.

Planning is the cleanest case. DS4-Flash improves on all four shared benchmarks (AIME +10.0, TerminalBench +14.5), and Qwen3.5 goes positive everywhere too. Retrieved records constrain task decomposition before execution starts.

Generation-time prompting is subtler. DS4-Flash gains everywhere with a big AIME jump (+22.0), but Qwen3.5 drops 5.6 points on TerminalBench and sits flat on HMMT. One plausible read: generation-time use forces the model to digest retrieved guidance before it has formed a task-specific plan, and useful procedures drown in the noise.

Post-generation review has the broadest positive footprint — the 57/72 improvements in the macro table all come from this interface. Once a draft exists, every invariant, error case, and anti-goal translates into a concrete check: “does this candidate do X or not?”

The RL results are the early signal worth watching. Same Qwen3-32B SWE-World checkpoint, same coding-RL protocol, resolve rate at step 150:

All four skill interfaces beat the control, with post-generation review leading by 14 points. That reads to me as a structural signal: during review, the candidate is in hand and retrieved procedures map onto it one by one; policy prompting has to operationalize them before any solution exists; the reward side has exactly one channel — the training signal — to influence anything. The caveat matters, though: this is a single checkpoint with no repeated seeds and no aligned learning curves. It does not establish faster learning or stronger final convergence.

In short: retrieval quality matters, and so does the decision point where the skill gets seen. Planning and post-generation review are the two consistently winning positions.

707 Characters Doing the Work of 6,352: Cutting Context Cost

Compress nearly nine tenths of the retrieved text and the utility mostly survives. That’s the headline from the retrieval ablations, and it changes how you should configure any skill-augmented agent.

Three variables, tested on BigCodeBench Instruct-Hard with DS4-Flash and Qwen3.5:

Retrieval depth. Increasing k from 1 to 10 grows average rendered context from 2.1K to 17.8K characters with little added utility. Qwen3.5 gains 1.40 points (33.10 to 34.50); DS4-Flash stays at or below its no-skill baseline.

Rendering. The clearest trade-off in the study. At k=3, summary rendering cuts average skill context from 6,352 to 707 characters — an 88.9% reduction. Qwen3.5 holds its full-record score, and DS4-Flash actually improves from 28.40 to 31.80.

Purpose indexing. Deduplicating near-identical candidates via cluster representatives shrinks full-record context from roughly 6.4K to 5.0K characters, but pass-rate effects are mixed: DS4-Flash’s full-record condition rises back to its no-skill baseline while the other three model-renderer pairs decline. Summary context sits near 0.7K with or without indexing, so there’s nothing left for the index to squeeze.

Why does deleting 90% of the text cost so little? Because the procedural core a task actually needs is worth far more than the implementation detail wrapped around it. The mixed indexing results likely come from global representatives displacing locally relevant candidates.

If you’re configuring your own system, start with k=3 plus summary rendering, and treat purpose indexing as a corpus-level deduplication tool rather than a default-on feature.

In short: depth isn’t where the value is; compact procedural descriptions are. Full evidence stays in the archive for audit and doesn’t need to ride along on every call.

What’s Inside a Million Skill Records?

Semantic annotation over all 1,006,822 accepted records paints a clear picture: this is a procedural knowledge bank, not an API cheat sheet. Multi-step procedures account for 57.3% of records and constraint reasoning for another 33.5%, against just 8.6% surface API calls — meaning 90.8% of records preserve an execution sequence or the conditions governing correct execution.

Problem types skew toward data transformation (35.8%) and state updates (33.7%). The leading reuse mechanisms are parameterization (39.9%), validation (38.6%), state-machine control (34.5%), and API usage (28.4%). The labels cohere structurally too: state updates pair with state-machine reuse, decision validation with validation patterns, external interaction with API usage, error recovery with robustness. Retrieval can therefore select records by “task objective plus reusable procedure” instead of falling back on loose keyword overlap.

One set of numbers deserves particular attention: 49.4% of records depend on library or framework behavior, 47.5% on language semantics, 47.2% on data formats or schemas, and 79.6% are correctness-critical. Read together, these explain why every record carries source coordinates. Much of the bank concerns library behavior and schema assumptions where one wrong generalization changes behavior — a skill without an evidence chain is a liability in exactly those situations.

On the use side, code generation leads at 69.1%, SWE agent workflows at 58.3%, and tool use at 41.8%. The records most often teach what to check (79.1%) and how to structure a solution (66.7%); merely providing copyable code is the minority case.

Granularity composition of the purpose cards: atomic-only 65.35% (490,457 cards), composite-only 31.12% (233,549), mixed 3.53% (26,530). The latter two together hold 34.65% — purpose indexing retains plenty of workflow-level content after compression.

Human judgment backs the automated gate. Annotators evaluated samples from four construction outcomes. Directly accepted records scored 96% description accuracy, 84% correct reconstruction, 76% retention value. Adjudicated accepts scored 88% accuracy and 84% retention — even though none of their initial reconstructions were judged correct, the descriptions themselves remain reusable. Rejected records came in at 32% accuracy, 28% retention, zero correct reconstructions. Bank-level summary: 92% accurate descriptions, 80% worth retaining, 84% supporting correct reconstruction.

The repository pool itself holds up under inspection: median 3,133 stars and 82 merged pull requests, 78.3% of repos with at least 1,000 stars, 46.9% with at least 100 merged PRs, and 66.0% pushed within the previous year. Language coverage is broad — TypeScript 16.0%, Python 8.2%, JavaScript 6.8%, with the major ecosystems represented.

In short: the bank’s substance is “how to do things correctly” knowledge, its density rests on the verification gate, and its evidence chains make it auditable rather than just large.

Can AI-Generated Code Feed the Skill Bank?

Yes — and the numbers are almost indistinguishable from human-written code. A skill bank extracted from tested AI-generated implementations scored 93.50% pass rate on a 400-task LiveCodeBench subset, against 93.00% for the human-code bank.

The experimental design is worth appreciating. The researchers sampled 25 tested source implementations from the accepted archive, had Codex with GPT-5.1 at extra-high reasoning effort generate interface-matched AI implementations, and required both sides to pass the same public and hidden tests. Origin labels were withheld from the extractor. Each implementation yielded one “how to implement” record and one “how to verify” record, producing a 50-skill bank per source. Evaluation used DS4-Flash at high reasoning effort, temperature 0, top-k=4 retrieval, with slots matched by functional pair, skill type, and rank.

Aggregate scores differ by just 0.5 percentage points, but the two banks disagree on 16 tasks — the human bank uniquely solves seven, the AI bank nine. The AI bank supplies genuinely different guidance under the source-controlled protocol, not a rerun of the human bank.

The practical implication is sustainability. As AI-generated code takes a growing share of new software, tested AI implementations can flow through the same extraction pipeline, letting the bank keep expanding as new code appears. The authors keep the claim appropriately modest: similar aggregates don’t establish equivalence between the two sources. But as initial evidence goes, it’s a strong start.

In short: AI-generated code can enter the bank, provided it passes the same tests first — and the two sources produce distinct, complementary signals.

Closing Thoughts

Code2Skill moves the “where do skills come from” question out of an agent’s execution history and into the world’s code repositories. Two properties of source code — existing before interaction, remaining inspectable after abstraction — combined with a blind-reconstruction verification gate yield a million records that each carry their own evidence chain. For anyone building agent systems, the two transferable judgments are probably these: a skill’s utility depends heavily on where it enters the workflow, and procedural cores survive aggressive compression far better than implementation detail does.

Practical Takeaways / Action Checklist

• Build the skill bank offline, before your target agent accumulates any trajectories
• Prefer planning-time and post-generation-review placements; pilot generation-time prompting before committing, since it’s model-dependent
• Start retrieval at k=3 with summary rendering; pushing depth to 10 mostly burns context
• During review, walk the record’s invariants, error cases, and anti-goals as an explicit checklist
• Keep provenance intact — repository, file, symbol, source span, acceptance trace — for audit and refresh
• In coding RL, try the post-generation-review interface first; it led the no-skill control by 14 points in the pilot
• Admit AI-generated implementations into the pipeline as long as they pass the same tests

One-Page Overview

• What it is: a fully automated pipeline that turns source code into developer skills — atomic operations, composite workflows, and recurring patterns
• Scale: 19,769 GitHub repositories, 1,006,822 accepted records, 10% default experimental retrieval pool
• Verification: source-body-blind reconstruction plus source-aware comparison; consistent records enter directly, others face adjudication or rejection
• Impact: macro-average 42.90 to 47.90, an 11.7% relative gain, 57/72 comparisons improved
• Versus trajectory methods: first on all seven shared benchmarks, averaging 49.5 against 40.1 for a per-benchmark oracle
• Best placements: planning improved 8/8 shared pairs, post-generation review showed the broadest positive pattern, coding RL led control by 14 points
• Context efficiency: k=3 summary rendering cuts context 88.9% with equal or better performance
• Human-verified quality: 92% accurate descriptions, 80% worth retaining, 84% supporting correct reconstruction

FAQ

Does Code2Skill require successful agent trajectories first? No. Skills are constructed offline from source code, before any downstream agent interaction takes place.

What are the three skill record types? Atomic skills capture one operation inside a single function or method; composite skills capture multi-step orchestration; recurring patterns capture higher-level implementation practices beyond a single localized operation.

How is record quality enforced? A model regenerates code from the record alone, without seeing the source body, and a source-aware judge compares the result against the original implementation. Consistent records are accepted directly; inconsistent ones go to adjudication or rejection.

How many records should you retrieve at once? The experiments sampled 10% of the bank as the default retrieval pool, and k=3 with summary rendering was the best value-for-context configuration tested.

Where should skills be injected into the agent workflow? Planning improved all eight shared pairs and post-generation review showed the most stable positive pattern. Generation-time prompting is mixed and depends on the model.

Can AI-generated code enter the skill bank? Yes. A bank extracted from tested AI implementations scored 93.50% on LiveCodeBench, slightly above the 93.00% of the human-code bank.

How much context overhead does it add? At k=3 with summary rendering, average context drops from 6,352 to 707 characters — an 88.9% reduction — with performance held or improved.

How does it compare against trajectory-derived skill banks? Under one shared interface it ranks first on all seven benchmarks, averaging 49.5 versus 40.1 for an oracle picking the best trajectory baseline per benchmark.