feedgrab: Pass It a URL, Get Structured Content Back

I’ve lost count of how many tools call themselves “universal content extractors.” Most support a handful of platforms and fold the moment they hit a bot detector. feedgrab is different — it actually delivers on the promise: give it any URL, get structured Markdown back. The six-tier fallback chain it uses for X/Twitter is something I haven’t seen in other open-source projects at this level of depth.

This post breaks feedgrab down: what it grabs, how to install it, how to use it, and where the rough edges are.

What can feedgrab actually extract?

It supports 17+ platforms covering articles, videos, podcasts, tweets, forum threads, knowledge base docs, and paywalled content. Everything comes out as Markdown with YAML front matter, organized into platform-specific subdirectories.

By category:

• Articles: WeChat Official Accounts (Sogou search + MP backend API batch by account + album batch), Zhihu columns, Medium, HackerNews, Reddit, Weibo, paywalled news sites (NYT, WSJ, FT, Economist, Bloomberg, SCMP, and 300+ others via a 7-tier paywall bypass)
• Video/Audio: YouTube (InnerTube API, zero dependencies, zero quota), Bilibili (3-tier subtitle fallback), Xiaoyuzhou (__NEXT_DATA__ SSR extraction + Groq Whisper transcription), Ximalaya (Web Revision API + Whisper transcription)
• Social platforms: X/Twitter (6-tier fallback), Xiaohongshu/RED (API + Pinia Store injection + browser, 3-tier strategy), Douyin/TikTok China (CDP Chrome reuse + SSR RENDER_DATA parsing)
• Knowledge bases/Docs: Feishu/Lark (Open API + CDP + Playwright + recursive wiki batch), KDocs/WPS (ProseMirror DOM + virtual scrolling), FlowUs (Notion-style block-tree rendering), Youdao Notes
• Developer platforms: GitHub (REST API, Chinese README priority + subdirectory language link search + relative image link completion), LinuxDo/IDCFlare (Discourse forums, JSON API → CDP → browser → Jina 4-tier fallback), Telegram (Telethon)
• Paid content: Zsxq/Knowledge Planet (5 topic morphologies covered: talk, question+answer, article, solution; short link 302 resolution; 3 comment modes)
• Any webpage: JSON-LD front detection → Jina Reader fallback

If you read it, feedgrab can probably grab it.

Three usage layers — pick what you need

feedgrab ships in three tiers. Use one, use all three, doesn’t matter.

Most people start with the CLI and never need more. If you live in Claude Code and regularly analyze content, the skill layer makes video transcription seamless — yt-dlp pulls subtitles, falls back to Groq Whisper if that fails, and the whole pipeline triggers automatically. The MCP server is for when you want to expose “read a webpage” as a tool to an AI Agent. It exposes four tools: read_url, read_batch, list_inbox, and detect_platform.

The Claude Code skill layer includes five skills: /feedgrab for core extraction, /feedgrab-batch for batch operations, /feedgrab-setup for installation guidance, /analyze for multi-dimensional structured analysis reports, and video for auto-triggered transcription. Once installed, just send a URL in Claude Code and the matching skill fires automatically.

Why X/Twitter’s six-tier fallback deserves its own section

Extracting X/Twitter content is where feedgrab is at its most thorough. Most tools either hit oEmbed for plain text or spin up a browser — neither gives you good data fast. feedgrab chains six degradation tiers:

With valid cookies, Tier 0 GraphQL runs and you get everything. When cookies expire, it doesn’t just crash — it drops to FxTwitter (a third-party public API, no auth needed, data completeness close to GraphQL, with a circuit breaker that skips after 3 consecutive failures), then to Syndication (missing retweets/bookmarks/views but the body and media are there). By the time you hit oEmbed, you’re down to plain text, but at least you get something instead of an error.

I initially thought the FxTwitter tier was redundant. It’s not. When all your cookies expire simultaneously, FxTwitter means you still pull 80% of the data without stopping to re-login. During batch bookmark extraction, if one account hits a 429, feedgrab drops to FxTwitter and keeps going rather than aborting the entire run.

The Tier 0 GraphQL layer inherits capabilities from dotey’s baoyu-danger-x-to-markdown skill: dynamic queryId resolution (extracted from X’s frontend JS bundle), full thread reconstruction (author self-reply chains), multi-phase pagination (up + down + continuation pages), complete media extraction, and full engagement metrics. On top of that, feedgrab adds bookmark batching, user tweet batching, list tweet batching, browser search supplementation (breaking through the ~800-tweet UserTweets limit with monthly chunked searches), and a global dedup index.

Four ways to configure Twitter cookies

The GraphQL tier needs two cookie values: auth_token and ct0. Priority from highest to lowest:

Environment variables (highest priority, overrides everything else) — copy manually from browser DevTools, add to .env:

X_AUTH_TOKEN=your_value
X_CT0=your_value

Playwright session — run feedgrab login twitter, a browser opens, you log in, cookies save to sessions/twitter.json. The easiest path.

Cookie file — manually create sessions/x.json with the same format.

Chrome CDP auto-extraction — if Chrome is already running and logged in with Remote Debugging enabled (chrome://inspect/#remote-debugging), run CHROME_CDP_LOGIN=true feedgrab login twitter and it pulls cookies instantly without opening another browser window.

Multi-account rotation to handle rate limits

Batch tweet extraction regularly triggers 429s on GraphQL. Drop multiple cookie files into sessions/ and feedgrab rotates automatically:

sessions/
├── twitter.json    ← Primary account (auto-generated by feedgrab login twitter)
├── x_2.json        ← Second account (manual creation, same format)
├── x_3.json        ← Third

Grab auth_token and ct0 from browser DevTools (F12 → Application → Cookies → https://x.com) and fill them in. Cookies aren’t bound to IP or device — they work across machines. When a 429 hits, feedgrab switches to the next unrestricted account and auto-recovers after a 15-minute cooldown.

TwitterAPI.io paid API: the server-friendly alternative

When tweet counts exceed ~800, the default strategy launches a browser and does monthly chunked searches. That doesn’t work on headless servers. Set TWITTERAPI_IO_KEY and it uses the paid API instead — $0.15 per 1,000 tweets, no count limit, with checkpoint resumption (discovery phase writes cache in real-time, so restarting doesn’t re-consume quota).

TWITTERAPI_IO_KEY=your_api_key
X_API_PROVIDER=api          # Full paid API, no cookies or browser needed
X_API_SAVE_DIRECTLY=false   # false = GraphQL supplements media (recommended); true = faster but no images

You can also filter by minimum likes, retweets, or views (OR relationship between the three) via X_API_MIN_LIKES, X_API_MIN_RETWEETS, and X_API_MIN_VIEWS in .env.

Platform-specific extraction strategies

Different platforms have wildly different anti-bot measures and data structures. feedgrab doesn’t use a one-size-fits-all approach — each platform gets tailored handling.

Bilibili’s 3-tier subtitle chain: Hits /x/player/v2 first, falls back to /x/player/wbi/v2 (requires WBI signature), then drops to Whisper transcription if both fail. Shares the Whisper pipeline with YouTube.

Xiaohongshu’s 4-tier strategy: Single-note extraction tries xhshow API first (no login, full metadata + comments), degrades to Pinia Store injection (executes native browser requests without third-party signature libraries, XHS_PINIA_ENABLED=true by default), then Jina, then Playwright. Keyword search (xhs-so) and author homepage batching follow the same chain.

Feishu’s virtual directory tree: Wiki batch extraction handles documents nested in multi-level directories with lazy-loaded sheets. feedgrab implements virtual directory tree traversal, lazy-loaded Sheet preheating, /sheet/block merging, and duplicate cell misalignment correction. The extraction chain: Open API → CDP direct → Playwright PageMain → Jina.

Zsxq/Knowledge Planet’s 5 topic morphologies: talk (regular posts), question+answer, article (long-form), solution — each with a different data structure, all covered. Short links (t.zsxq.com/<code>) auto-resolve via 302. Comments support three modes: none, all, author (author’s replies only).

FlowUs block-tree rendering: Public share links need zero cookies (pure HTTP /api/docs/{uuid}). Paid/private docs need next_auth + next_auth.sig dual cookies. The rendering layer parses 8 block types + 5 enhancers + link fragments in Notion-style block-tree format. When localizing images, a headless browser renders to capture signed cdn2.flowus.cn URLs, then pulls them directly.

7-tier paywall bypass: JSON-LD detection → Googlebot/Bingbot UA → AMP pages → archive.today → Google Cache. Doesn’t work on every site, but 300+ sites is a wide enough net.

Discourse forums (LinuxDo/IDCFlare): Prioritizes the Discourse Topic JSON API, drops to CDP Chrome reuse for Cloudflare-strict sites, then Playwright in-page fetch, then Jina. Reply mode is configurable: author (OP post + OP self-replies), all (full thread), none (OP post only).

Installation: zero to working

# Basic install
pip install git+https://github.com/iBigQiang/feedgrab.git

# Recommended: stealth browser + TLS fingerprinting (patchright + browserforge + curl_cffi)
pip install "feedgrab[stealth] @ git+https://github.com/iBigQiang/feedgrab.git"
patchright install chromium

The [stealth] extras give the strongest anti-detection capability. If you need Twitter search enhancement (x-client-transaction-id signing for x-so), add [twitter]. For Xiaohongshu API enhancement (xhshow signing for xhs-so), add [xhs]. Or just use [all] and install everything at once.

Then run the setup wizard:

feedgrab setup

Five steps: environment check → config file generation → UA detection → platform login → feature enablement. Each step is skippable, and re-running automatically skips completed items.

Video/audio transcription needs yt-dlp and ffmpeg separately:

# macOS
brew install yt-dlp ffmpeg

# Linux
pip install yt-dlp
apt install ffmpeg

Whisper transcription requires a Groq API key (free):

export GROQ_API_KEY=your_key_here

Image source: Unsplash

Day-to-day CLI usage

Single URL extraction is straightforward:

feedgrab https://mp.weixin.qq.com/s/abc123

In PowerShell, URLs containing & throw errors. Use feedgrab clip to read from the clipboard instead. You can also pass multiple URLs at once: feedgrab https://url1.com https://url2.com.

Here are the batch scenarios that come up most often:

Xiaohongshu search (no login required, uses xhshow API):

feedgrab xhs-so "AI Agent" --sort popular --type video --limit 50
feedgrab xhs-so "claude code,openclaw,lobster farming" --merge   # Merge keywords into one table
feedgrab xhs-so "claude code,openclaw"                             # Separate tables per keyword

Twitter search (summary tables sorted by engagement):

feedgrab x-so openclaw                                        # Last 24h + Chinese + Latest tab
feedgrab x-so "AI Agent" --days 7 --min-faves 50 --sort top   # Custom parameters
feedgrab x-so "VPN,proxy,v2ray,shadowrocket" --merge          # Merge multiple keywords

YouTube search and download:

feedgrab ytb-so "AI Agent" --channel @AndrewNg --order viewCount