Site icon Efficient Coder

Tired of Paywalls? Host This Googlebot Impersonator: Ladder HTTP Proxy

高效码农

1. What Exactly Is Ladder—and Why Should You Care?

Ladder is an open-source, Go-based HTTP proxy that clones the core trick used by sites such as 1ft.io and 12ft.io: it dresses up as Googlebot, asks the target page for the “search-engine” version, strips the paywall markup, and hands you a clean article. You host it yourself, so nobody logs your reading list and no third-party limits your usage.

Who this guide is for

  • College (and above) reading level
  • Zero tolerance for “marketing fluff”
  • Comfortable copying commands into a terminal or Docker prompt

What you will get

  • A private proxy running on your laptop, home server, or cloud instance
  • Step-by-step copy-paste blocks for Windows, macOS, Linux, Docker, Docker Compose, Helm
  • A plain-English tour of every setting and YAML knob—so you can fix it when (not if) a site changes its layout

2. How the Magic Happens (The 30-Second Version)

  1. Browser → Ladder (asks for article)
  2. Ladder → Website (pretends to be Googlebot)
  3. Website → Ladder (sends full HTML)
  4. Ladder scrubs ads, paywall scripts, and CORS headers → Browser

That’s it. No browser extension, no VPN, no account cookies.

3. Pick Your Install Style in One Table

Route Time When to Use
Pre-built binary 2 min Quick look on your own PC
Docker one-liner 3 min Always-on box at home
Docker Compose 5 min Keeps your config in Git
Helm chart 10 min Kubernetes cluster

All four routes are covered below—jump to the one you like.

4. Binary Install (Windows / macOS / Linux)

  1. Download the latest release

    • Windows: ladder_windows_amd64.exe
    • macOS Intel: ladder_darwin_amd64
    • macOS Apple Silicon: ladder_darwin_arm64
    • Linux x86-64: ladder_linux_amd64

  2. Make it executable (macOS/Linux)

    chmod +x ladder

  3. Start it

    • Windows: double-click or .\ladder.exe
    • macOS/Linux: ./ladder

  4. Open http://localhost:8080
    Paste any article URL, press Enter.

Security heads-up
Default port 8080 is wide open and unauthenticated. If another person can reach your IP, add USERPASS (shown in §6) before you finish your coffee.

5. Docker One-Liner

docker run -d \
  -p 8080:8080 \
  -e RULESET=https://t.ly/14PSf \
  --name ladder \
  ghcr.io/everywall/ladder:latest

Check logs:

docker logs -f ladder

Update later:

docker pull ghcr.io/everywall/ladder:latest
docker stop ladder && docker rm ladder
# re-run the same docker run string

6. Docker Compose (Recommended for Homelabs)

Create docker-compose.yaml:

version: "3.9"
services:
  ladder:
    image: ghcr.io/everywall/ladder:latest
    ports:
      - "8080:8080"
    environment:
      USERPASS: "admin:YOUR_STRONG_PASS"
      RULESET: "https://raw.githubusercontent.com/everywall/ladder-rules/main/ruleset.yaml"
      LOG_URLS: "true"
    restart: unless-stopped

Bring it up:

docker compose up -d

Down or upgrade:

docker compose down
docker compose pull && docker compose up -d

7. Kubernetes / Helm

Add the repo:

helm repo add ladder https://everywall.github.io/ladder
helm install myladder ladder/ladder \
  --set userpass=admin:YOUR_STRONG_PASS \
  --set service.type=ClusterIP

See helm-chart/README.md for Ingress TLS, persistent rules, and resource limits.

8. Configuration Dictionary (All Env Vars)

Variable Default Purpose Example
PORT 8080 TCP port inside container 8888
PREFORK false Spawn multiple workers true
USER_AGENT Googlebot 2.1 Outgoing browser signature Mozilla/5.0 (Windows NT 10.0; Win64; x64) …
X_FORWARDED_FOR 66.249.66.1 Fake crawler IP empty = don’t send
USERPASS (blank) Basic auth colon-separated admin:StrongPass
LOG_URLS true Print every fetched URL false
DISABLE_FORM false Hide the front-page URL box true
FORM_PATH (blank) Path to custom HTML form /opt/mysite/form.html
RULESET official URL YAML file or folder or URL ./rules.yaml
EXPOSE_RULESET true Let other Ladder nodes pull your rules false
ALLOWED_DOMAINS (blank) Comma list of whitelisted domains nytimes.com,washingtonpost.com
ALLOWED_DOMAINS_RULESET false Auto-allow domains found in rules true

Note
ALLOWED_DOMAINS and ALLOWED_DOMAINS_RULESET are combined. If both are empty, the proxy accepts any target.

9. Quick Usage Examples

A. Browser bookmarklet
Create a bookmark with this location:

javascript:window.location.href="http://YOUR_IP:8080/"+encodeURIComponent(location.href)

Click it when you hit a paywall.

B. Raw HTML API

curl http://YOUR_IP:8080/raw/https://www.example.com/article

C. JSON-free API

curl http://YOUR_IP:8080/api/https://www.example.com/article

10. Rule Sets: Teaching Ladder New Tricks

A rule is plain YAML telling Ladder:

  • Which domains it applies to
  • How to rewrite request/response headers
  • Regex replacements in HTML
  • Code snippets to inject

Minimal working example:

- domain: example.com
  headers:
    user-agent: Mozilla/5.0 (Googlebot 2.1)
  regexRules:
    - match: <script[^>]*src="(/[^"]*)"
      replace: <script src="https://www.example.com$1"
  injections:
    - position: head
      append: |
        <script>
          window.localStorage.clear();
        </script>

Drop the file anywhere and point RULESET to it, or host it on a static URL so multiple Ladder nodes can share it.

11. Anatomy of a Rule (Line-by-Line)

Section What You Can Do
domain: Main domain (catches sub-domains)
domains: Extra exact-match domains
paths: Limit rule to certain URL paths
headers: Add/overwrite/delete request or response headers (delete by setting value to none)
regexRules: List of match/replace pairs, applied in order
injections: append, prepend, or replace HTML at CSS selector or standard positions (head, body)
urlMods: Rewrite query strings, domain, or path before fetching
googleCache: Fetch from webcache.googleusercontent.com instead of live site

12. Real-World Pattern: Removing a Soft Paywall

The following snippet is only a technical illustration; effectiveness changes whenever the publisher redesigns.

- domain: nytimes.com
  headers:
    referer: https://www.google.com/
  regexRules:
    - match: <div[^>]*class="[^"]*css-paywall[^"]*"[^>]*>.*?</div>
      replace: ""
  injections:
    - position: head
      prepend: |
        <style>.css-paywall { display:none !important; }</style>

Save as nyt.yaml, restart Ladder with:

docker run -d -p 8080:8080 -e RULESET=./nyt.yaml -v $(pwd)/nyt.yaml:/nyt.yaml ghcr.io/everywall/ladder:latest

13. Troubleshooting Checklist

Symptom Likely Cause Fix
Images not loading Lazy-load data-src not converted Add regex rule to copy data-src to src
502 returned Target blocks Googlebot IP Change USER_AGENT to desktop string, empty X_FORWARDED_FOR
YAML parse error on start Mixed spaces/tabs Validate with online YAML linter
permission denied in Docker bind mount Host file not readable chmod 644 rules.yaml

14. Updating & Rolling Back

Binary

  • Keep old binary named ladder.bak
  • Swap file, restart process

Docker

  • docker pull newer tag
  • docker stop / rm / run
  • Rollback: tag the previous image explicitly

Helm

helm upgrade myladder ladder/ladder --version x.y.z
helm rollback myladder REVISION

15. Development Setup (If You Want to Hack the Code)

Requirements

  • Go 1.20+
  • pnpm (only if you tweak the default form styling)

Clone and run:

git clone https://github.com/everywall/ladder.git
cd ladder
echo "dev" > handlers/VERSION
RULESET="./ruleset.yaml" go run cmd/main.go

Live-reload option (Air):

go install github.com/cosmtrek/air@latest
air

Change CSS classes inside form.html? Rebuild:

pnpm install
pnpm build

16. Security & Legal Reminders (Don’t Skip)

  1. Copyright still exists. Ladder is a tool; whether bypassing a particular paywall is legal depends on your jurisdiction and the site’s TOS.
  2. Basic auth is mandatory on any public IP. An open proxy becomes an abuse magnet—bandwidth, DMCA, worse.
  3. Don’t log what you don’t need. Flip LOG_URLS=false if you want less liability surface.
  4. Keep the image updated. New builds occasionally patch the Go standard library or the proxy parser.

17. Quick Reference: All URLs You Need

Purpose Path
Human UI http://IP:PORT/
API (full processing) http://IP:PORT/api/https://...
Raw HTML only http://IP:PORT/raw/https://...
View running rules http://IP:PORT/ruleset

18. Frequently Asked Questions

Q1. Will it work on every site?
No. Sites that never serve the full text to search engines (hard paywalls) return blank or partial pages. A future Ladder release may retry via Google Cache.

Q2. Why are comments or images still broken?
Some media loads through JavaScript bundles that require a browser environment. Ladder fetches static HTML only; it does not execute JS.

Q3. Does Ladder store anything on disk?
Not by default. Logs go to stdout; you can redirect or disable them.

Q4. Can I chain a TOR exit node?
The feature is on the roadmap but not implemented yet.

Q5. Is there a rate limit?
Ladder itself does not throttle. Respect individual publishers’ politeness policies to avoid IP bans.

19. TL;DR for the Impatient

  1. docker run -p 8080:8080 -e USERPASS=you:pass ghcr.io/everywall/ladder:latest
  2. Browse to http://localhost:8080
  3. Paste article URL → Enter
  4. Read peacefully
  5. When bored, open docker-compose.yaml, add domains, volumes, env vars—tinker away.

20. Parting Thought

Tools come and go; paywalls adapt. Hosting your own proxy keeps the experiment alive without handing your reading history to a stranger. Use it wisely, patch it often, and remember that sustainable journalism deserves support. When you find yourself relying on a particular publication, consider subscribing.

Exit mobile version