Open Scouts: Build Your Custom AI Web Monitors to Track What Matters 24/7

Executive Summary

Open Scouts is an AI-powered monitoring platform that enables users to create automated tasks (scouts) for continuous web information tracking. Built on a tech stack including Next.js, Supabase, and Firecrawl, it supports email notifications and semantic search capabilities. This article provides a comprehensive guide to its features, technical architecture, deployment steps, design system, and analytics integration, ensuring full transparency and operational clarity.

What Is Open Scouts?

Imagine having a team of tireless monitors that work around the clock to keep an eye on the web for information you care about—newly opened local restaurants, the latest AI industry updates, product price fluctuations, or any other specific content. The moment they find what you’re looking for, they deliver timely alerts right to you. That’s exactly what Open Scouts offers.

Open Scouts is an AI-driven monitoring platform that allows users to create “scouts”—automated tasks that run on predefined schedules to continuously search and track web-based information. Whether you want to stay updated on nearby coffee shops, monitor AI-related news, or keep tabs on any type of online updates, these scouts operate 24/7 to capture relevant content and notify you as soon as new findings emerge.

open-scouts_4

The Technical Stack Behind Open Scouts: Core Infrastructure

To support a platform that delivers continuous monitoring, intelligent analysis, and timely notifications, Open Scouts relies on a robust and integrated set of technologies, each playing a critical role in its functionality:

  • Next.js 16: Leveraging the App Router and Turbopack, it powers the frontend application and server-side logic, ensuring high efficiency and smooth user experience.
  • React 19: Used for building interactive user interfaces, enabling seamless interactions and responsive design across the platform.
  • TypeScript: Provides static type checking to reduce code errors and enhance development reliability and maintainability.
  • Tailwind CSS v4: Facilitates rapid styling with utility-first classes, ensuring consistent and visually cohesive UI design.
  • Supabase: Serves as the all-in-one backend solution, combining database management, user authentication, and edge function deployment.
  • pgvector: Handles vector embeddings to enable semantic search, allowing the platform to understand the contextual meaning of content rather than just matching keywords.
  • Firecrawl SDK (@mendable/firecrawl-js): Responsible for web scraping and content retrieval, equipping scouts with the ability to gather information from across the internet.
  • OpenAI API: Powers the AI agent and embedding generation, enabling scouts to analyze data and produce concise, context-aware summaries.
  • Resend: Manages email notifications, ensuring users receive timely alerts when their scouts uncover relevant results.

How to Get Started with Open Scouts: Step-by-Step Deployment Guide

If you want to deploy and use Open Scouts for your own monitoring needs, follow this detailed, actionable guide to set up the platform successfully:

Prerequisites: What You Need to Prepare

Before starting the deployment process, ensure you have the following tools and credentials ready:

  • Node.js 18 or higher
  • A package manager (bun is recommended by default; npm or pnpm are also compatible)
  • A Supabase account (register at supabase.com)
  • An OpenAI API key (obtainable from platform.openai.com)
  • A Firecrawl API key (available via firecrawl.dev)
  • A Resend API key (from resend.com, required for email notifications)
  • A Google Cloud Console account (optional, for enabling Google OAuth login)

Step 1: Clone the Repository and Install Dependencies

First, clone the Open Scouts codebase to your local machine and install all required dependencies. Open your terminal and run the following commands:

git clone https://github.com/leonardogrig/open-scout
cd open-scout
bun install  # For npm users, replace with npm install; for pnpm, use pnpm install

Step 2: Create a Supabase Project

Supabase is the backbone of Open Scouts’ backend, so creating a project is the foundational step for database and authentication management:

  1. Navigate to the Supabase Dashboard (supabase.com/dashboard)
  2. Create a new project by following the on-screen prompts
  3. Wait for the project to finish provisioning (this process may take a few minutes)

Step 3: Enable Required Extensions

Several database extensions are critical for Open Scouts’ core features to function properly. Enable them through the Supabase Dashboard:

  1. Go to the “Database → Extensions” section in your Supabase project
  2. Search for and enable the following extensions:

    • pg_cron: Enables scheduled tasks to run scouts at predefined intervals
    • pg_net: Supports sending HTTP requests directly from the database
    • vector: Powers AI-driven semantic search for execution summaries
    • supabase_vault: Provides secure storage for credentials (usually enabled by default)

Step 4: Configure Environment Variables

Environment variables store sensitive credentials and configuration details required for the platform to connect to external services. Set them up as follows:

  1. Create a .env file in the project root directory by copying the example template:

    cp .env.example .env
  2. Fill in the actual values for each variable in the .env file. The .env.example file includes detailed instructions and direct links to obtain each API key, making the configuration process straightforward.

Step 5: Run the Database Setup Script

This step configures the database tables, access policies, and core infrastructure required for Open Scouts to operate. Follow these sub-steps:

  1. Link your local environment to your Supabase project (required for secret synchronization):

    bunx supabase login        # One-time login to the Supabase CLI
bunx supabase link --project-ref <your-project-ref>  # Find the project ref in your Supabase Dashboard URL
  2. Execute the database setup script:

    bun run setup:db  # For npm/pnpm users, replace with npm run setup:db or pnpm run setup:db

The setup script performs a range of critical tasks, including:

  • Creating all necessary database tables (such as scouts, scout_executions, and scout_execution_steps)
  • Adding user authentication support (including user ID columns and Row Level Security policies)
  • Enabling real-time subscription functionality for live result updates
  • Configuring vector embeddings for AI-generated execution summaries
  • Setting up a scalable dispatcher architecture (combining pg_cron, pg_net, and supabase_vault)
  • Automatically storing the Supabase URL and service role key in the vault for secure access
  • Creating cron jobs for scout dispatching and stuck execution cleanup
  • Syncing edge function secrets (OPENAI_API_KEY, FIRECRAWL_API_KEY, RESEND_API_KEY) from the .env file to Supabase

Note: The script will verify that all required extensions are enabled. If any are missing, follow the on-screen prompts to enable them in the Supabase Dashboard, then re-run the script.

Step 6: Set Up User Authentication

Open Scouts uses Supabase Auth for user identity management, supporting both email/password and Google OAuth login methods.

Enable Email/Password Authentication (Enabled by Default)

  1. Navigate to “Authentication → Providers → Email” in the Supabase Dashboard
  2. Ensure the “Enable Email Provider” toggle is switched on
  3. Customize email templates as needed in the “Authentication → Email Templates” section

Enable Google OAuth (Optional but Recommended)

To allow users to log in with their Google accounts, complete the following configuration:

  1. Create Google OAuth Credentials:

    • Go to the Google Cloud Console (console.cloud.google.com)
    • Create a new project or select an existing one
    • Navigate to “APIs & Services → Credentials”
    • Click “Create Credentials → OAuth client ID”
    • Select “Web application” as the application type
    • Add authorized JavaScript origins:

      • Development environment: http://localhost:3000
      • Production environment: https://your-domain.com (replace with your actual domain)
    • Add authorized redirect URIs:

      • https://<your-project-ref>.supabase.co/auth/v1/callback
    • Copy the generated “Client ID” and “Client Secret”

  2. Configure in Supabase:

    • Go to “Authentication → Providers → Google” in the Supabase Dashboard
    • Toggle on “Enable Google Provider”
    • Paste the copied Client ID and Client Secret
    • Save the configuration

Step 7: Deploy Edge Functions

Edge functions handle scout execution and email notification delivery, and they need to be deployed to Supabase Cloud:

bunx supabase functions deploy scout-cron
bunx supabase functions deploy send-test-email

Note: Secrets (OPENAI_API_KEY, FIRECRAWL_API_KEY, RESEND_API_KEY) are automatically synced when you run the setup:db script. If you need to update them manually, use the following command (replace the placeholder with your actual key):

bunx supabase secrets set OPENAI_API_KEY=sk-proj-...

Step 8: Configure Resend for Email Notifications

When scouts find relevant results, they send email alerts to users’ registered accounts, which requires Resend configuration:

  1. Create a Resend account at resend.com
  2. Retrieve your API key from the Resend Dashboard
  3. Add the key to your .env file and re-run setup:db to sync it, or set it manually via:

    bunx supabase secrets set RESEND_API_KEY=re_...  # Replace with your actual Resend API key
  4. Verify a custom domain at resend.com/domains to enable sending emails to any recipient address

Important – Free Tier Limitations:

  • Without a verified domain, Resend only allows sending emails to your registered Resend account email
  • The free tier includes 3,000 emails per month (with a 100-email daily limit)

Testing Email Configuration:

  1. Navigate to the “Settings” page in the Open Scouts application
  2. Click the “Send Test Email” button to verify the setup
  3. Check your inbox to confirm receipt of the test email

Step 9: Configure Firecrawl for Web Scraping

Open Scouts relies on Firecrawl for web scraping and content retrieval, with two configuration options available:

Option A: Standard API Key (Recommended for Contributors)

This is the simplest setup, where all users share a single API key:

  1. Register an account at firecrawl.dev
  2. Obtain your API key from the Firecrawl Dashboard (www.firecrawl.dev/app/api-keys)
  3. Add the key to your .env file:

    FIRECRAWL_API_KEY=fc-your-key-here  # Replace with your actual key
  4. Set the edge function secret for Firecrawl:

    npx supabase secrets set FIRECRAWL_API_KEY=fc-your-key-here

Option B: Partner Integration (For Production Deployments)

For deployments serving multiple users with per-user API key management, use the partner integration:

  1. Contact Firecrawl to obtain a partner key
  2. Add the partner key to your .env file:

    FIRECRAWL_API_KEY=your-partner-key  # Replace with your partner key
  3. Set the edge function secret:

    npx supabase secrets set FIRECRAWL_API_KEY=your-partner-key

How Partner Integration Works:

  • A unique Firecrawl API key is automatically created for each user upon signup
  • Each user’s usage is tracked individually
  • Keys are securely stored in the user_preferences database table
  • If a user’s key becomes invalid, the system automatically falls back to the shared partner key
  • Users can check their connection status in the “Settings → Firecrawl Integration” section of the app

Benefits of Partner Integration:

  • Granular usage tracking per user
  • Ability to revoke individual user keys
  • Automated key provisioning for new users
  • Self-healing functionality (automatic fallback for invalid keys)

Note: The partner integration is fully backward-compatible. Without a partner key, the system functions identically to Option A, using a shared API key for all users.

Step 10: Launch the Development Server

Once all configurations are complete, start the development server to access the Open Scouts application locally:

bun run dev  # For npm/pnpm users, use npm run dev or pnpm run dev

Then, open http://localhost:3000 in your browser to interact with the platform.

How Open Scouts Works: Core Operational Logic

Now that you know how to deploy Open Scouts, let’s break down its end-to-end operational flow, from user authentication to scout execution and notification delivery.

User Authentication Flow

  1. Public Landing Page: Users can browse the platform’s homepage without logging in
  2. Scout Creation Trigger: When a user enters a search query and presses Enter, they are prompted to log in or sign up
  3. Authentication Process: Users can authenticate via email/password or Google OAuth
  4. Seamless Flow Continuation: After successful authentication, the system automatically resumes the scout creation process
  5. User Data Isolation: Each user can only view and manage their own scouts, ensuring data privacy and security

Scout System: From Creation to Notification

  1. Create a Scout: Define the information you want to monitor (e.g., “Track newly opened Indian restaurants near me” or “Monitor the latest AI industry news”)
  2. AI Agent Configuration: The system automatically generates search queries and monitoring strategies tailored to your input
  3. Set Execution Frequency: Choose how often the scout runs (options include hourly, every 3 days, weekly, and more)
  4. Configure Notifications: Add your email address in the Settings page to receive alerts when the scout finds relevant results
  5. Continuous Monitoring: The dispatcher checks for due scouts every minute and triggers their execution individually
  6. AI-Generated Summaries: Each successful scout execution produces a concise, one-sentence summary with semantic embeddings for easy content scanning
  7. Notification Delivery: If email notifications are enabled, you receive an alert as soon as the scout uncovers new results
  8. Result Viewing: Access all scout findings and their AI summaries in real time on the scout’s dedicated page

Manual Scout Execution

For immediate information retrieval, users can click the “Run Now” button on any scout’s page to trigger execution instantly, bypassing the scheduled cron job.

Email Notification Details

Email alerts are a core feature of Open Scouts, with the following characteristics:

  • Conditional Delivery: Emails are only sent when the scout successfully finds new, relevant results
  • Rich Content Format: Alerts use aesthetically pleasing HTML templates that include full scout results and direct links to the platform
  • Test Functionality: The “Send Test Email” button in Settings allows users to verify their notification configuration before relying on live alerts

Email Service Provider: Notifications are powered by Resend, which offers 3,000 free emails per month in its basic tier.

Note: On Resend’s free tier without a verified custom domain, emails can only be sent to the user’s registered Resend account email. To send alerts to any recipient, verify a custom domain at resend.com/domains.

Underlying Architecture

Open Scouts’ architecture is designed for scalability and reliability, with each component serving a distinct purpose:

  • Frontend: A Next.js application that leverages Supabase Realtime for live result updates, ensuring users see new findings without refreshing the page
  • Database: A PostgreSQL database (hosted on Supabase) that uses pg_cron for task scheduling and pgvector for semantic search capabilities
  • Authentication: Supabase Auth manages user identities, supporting both email/password and Google OAuth login methods
  • AI Agent: OpenAI GPT-4 with function-calling capabilities powers the scout’s decision-making, using web scraping and search tools to gather information
  • AI Summaries: The system generates concise one-sentence summaries for each successful execution, with vector embeddings for enhanced content organization
  • Edge Functions: Deno-based serverless functions orchestrate scout execution, ensuring isolated and scalable processing
  • Web Scraping: Firecrawl API handles content retrieval, with support for per-user API keys via the partner integration model

Scalable Dispatcher Architecture

Open Scouts uses a dispatcher pattern designed to scale to thousands of concurrent scouts, with the following workflow:

Every minute:
pg_cron → dispatch_due_scouts() → identifies due scouts → pg_net HTTP POST
                                                          ↓
                                    ┌──────────────────────┼──────────────────────┐
                                    ↓                      ↓                      ↓
                              Edge Function          Edge Function          Edge Function
                              (scout A)              (scout B)              (scout C)
                              [isolated environment] [isolated environment] [isolated environment]
  • Dispatcher (SQL-based): Runs every minute via pg_cron, queries the database for scouts that are due for execution, and sends individual HTTP requests to trigger each scout
  • Isolated Execution: Each scout runs in its own edge function instance, with dedicated resources (256MB of memory and a 400-second timeout) to prevent resource contention
  • Automatic Cleanup: A separate cron job runs every 5 minutes to clean up stuck or failed executions
  • Vault Integration: Supabase credentials are securely stored in supabase_vault and retrieved by the dispatcher for authenticated operations

Security Measures: Protecting User Data and Platform Integrity

Security is a top priority for Open Scouts, with multiple layers of protection to safeguard user data and platform operations:

  • Row Level Security (RLS): All database tables are protected by RLS policies, ensuring users can only access data associated with their own accounts
  • User Isolation: Scouts, execution results, and notifications are all tied to authenticated user IDs, preventing unauthorized access to sensitive information
  • Secure Authentication Flow: OAuth tokens and user sessions are managed by Supabase Auth, which follows industry-standard security protocols
  • Service Role Access: Server-side operations (such as cron jobs and edge functions) use the Supabase service role for privileged access, with strict permission controls
  • API Key Storage: When using the Firecrawl partner integration, user-specific API keys are stored in the server-side user_preferences table and never exposed to the client-side application, eliminating key leakage risks

Firecrawl Design System: Building a Consistent and User-Friendly UI

The Open Scouts interface is built on the Firecrawl design system, which ensures visual consistency across the application and streamlines frontend development for contributors.

Design System Overview

The Firecrawl design system is organized around a modular component architecture located in the components-new/ directory. It integrates multiple UI libraries to deliver a unified visual language for the application.

Core Technologies

  • Tailwind CSS: A utility-first CSS framework with custom configurations tailored to the Firecrawl design language
  • shadcn/ui: High-quality, accessible React components built on top of Radix UI
  • Custom Components: Application-specific shared components designed for Open Scouts’ unique use cases

Directory Structure

components-new/
├── ui/                    # Core UI components
│   ├── shadcn/           # shadcn/ui component library
│   ├── magic/            # Magic UI animated components
│   ├── tremor/           # Tremor data visualization components
│   └── motion/           # Motion and animation utilities
├── shared/               # Shared application components
│   ├── icons/            # Icon components and brand assets
│   ├── buttons/          # Custom button components
│   ├── cards/            # Card UI components
│   ├── effects/          # Visual effects and animations
│   ├── layout/           # Layout utilities
│   └── ui/               # Shared UI helper components
├── app/                  # Application-specific components
│   ├── brand/            # Brand-related components
│   ├── pricing/          # Pricing page components
│   └── (home)/           # Homepage components
└── providers/            # Context providers and theme management

Color System

The design system uses a comprehensive color palette defined in colors.json and styles/colors.json, supporting consistent theming across both light and dark modes.

Color Categories

Heat Colors

The primary brand color with varying opacity levels:

  • heat-4 to heat-100: An orange-red brand color (#fa5d19) with opacity ranging from 4% to 100%
Accent Colors

Semantic colors used to indicate different UI states and contexts:

  • accent-black: Dark neutral (#262626)
  • accent-white: Pure white (#ffffff)
  • accent-amethyst: Purple accent (#9061ff)
  • accent-bluetron: Blue accent (#2a6dfb)
  • accent-crimson: Red accent (#eb3424)
  • accent-forest: Green accent (#42c366)
  • accent-honey: Yellow accent (#ecb730)
Alpha Variants

Transparent overlays used for creating visual depth and layering:

  • black-alpha-1 to black-alpha-88: Black with opacity from 1% to 88%
  • white-alpha-56 and white-alpha-72: White with 56% and 72% opacity, respectively
UI Colors

Dedicated colors for specific interface elements:

  • border-faint, border-muted, border-loud: Gradated border colors for different visual emphasis levels
  • illustrations-faint, illustrations-muted, illustrations-default: Color variants for in-app illustrations
  • background-lighter, background-base: Base background colors for different page sections

Color Implementation

Colors are integrated into Tailwind CSS via CSS custom properties, enabling dynamic theming and consistent usage across components:

const colors = Object.keys(colorsJson).reduce(
  (acc, key) => {
    acc[key] = `var(--${key})`;
    return acc;
  },
  {} as Record<string, string>
);

This setup allows for straightforward application of brand and UI colors in components:

<div className="bg-heat-100 text-accent-white">
  Primary brand styling
</div>

<div className="border border-border-muted bg-background-base">
  Subtle interface element
</div>

Tailwind Configuration

The Tailwind configuration file (tailwind.config.ts) extends the default framework settings with custom typography, spacing rules, and utility classes built specifically for the Firecrawl design system.

Typography Scale

The system includes a semantic typography scale with predefined sizes for different content types:

Titles
  • title-h1: 60px font size, 64px line height, -0.3px letter spacing
  • title-h2: 52px font size, 56px line height, -0.52px letter spacing
  • title-h3: 40px font size, 44px line height, -0.4px letter spacing
  • title-h4: 32px font size, 36px line height, -0.32px letter spacing
  • title-h5: 24px font size, 32px line height, -0.24px letter spacing
Body Text
  • body-x-large: 20px font size, 28px line height, -0.1px letter spacing
  • body-large: 16px font size, 24px line height
  • body-medium: 14px font size, 20px line height, 0.14px letter spacing
  • body-small: 13px font size, 20px line height
  • body-input: 15px font size, 24px line height (for form input fields)
Labels
  • label-x-large: 20px font size, 28px line height, 450 font weight
  • label-large: 16px font size, 24px line height, 450 font weight
  • label-medium: 14px font size, 20px line height, 450 font weight
  • label-small: 13px font size, 20px line height, 450 font weight
  • label-x-small: 12px font size, 20px line height, 450 font weight
Monospace Text
  • mono-medium: 14px font size, 22px line height
  • mono-small: 13px font size, 20px line height, 500 font weight
  • mono-x-small: 12px font size, 16px line height

Font Families

  • Sans: SuisseIntl (primary font) with system font fallbacks
  • Mono: Geist Mono with system font fallbacks
  • ASCII: Roboto Mono with system font fallbacks

Custom Utility Classes

The configuration includes a set of custom utility classes to streamline common UI patterns:

Border Utilities
  • .inside-border: Absolute-positioned border overlays
  • .inside-border-x: Horizontal-only border overlays
  • .inside-border-y: Vertical-only border overlays
  • .mask-border: CSS mask-based border effects
Positioning Utilities
  • .center-x: Horizontal centering for elements
  • .center-y: Vertical centering for elements
  • .center: Full horizontal and vertical centering
  • .flex-center: Flexbox-based centering for container children
Layout Utilities
  • .overlay: Full-screen overlay positioning
  • .text-gradient: Gradient text styling effects
Custom Sizing Utilities
  • cw-{size}: Centered width positioning
  • ch-{size}: Centered height positioning
  • cs-{size}: Centered square sizing
  • cmw-{maxWidth},{padding}: Centered max-width with built-in padding
  • mw-{maxWidth},{padding}: Max-width container with padding

Critical: Custom Sizing System

A key distinction of the Firecrawl design system is that numeric values in Tailwind classes correspond to literal pixels, not rem units as in standard Tailwind.

What This Means

In tailwind.config.ts, a custom sizes object (lines 16–37) maps numeric values to pixel measurements:

const sizes = Array.from({ length: 1000 }, (_, i) => i).reduce(
  (acc, curr) => {
    acc[curr] = `${curr}px`;  // e.g., 3 = "3px", 8 = "8px", 100 = "100px"
    return acc;
  },
  { /* fractional percentages */ }
);

This sizes object is applied to multiple CSS properties (lines 337–344):

  • spacing: Affects padding (p-*), margin (m-*), and gap (gap-*)
  • width: Controls element width (w-*)
  • height: Controls element height (h-*)
  • size: Shorthand for width + height (size-*)
  • inset: Affects positional properties (top-*, left-*, etc.)
Comparison with Standard Tailwind
Class Standard Tailwind Firecrawl System
w-3 0.75rem (12px) 3px
h-8 2rem (32px) 8px
size-4 1rem (16px) 4px
p-12 3rem (48px) 12px
gap-24 6rem (96px) 24px
Correct Usage Guidelines

✅ For Spacing (padding, margin, gap):

<div className="p-24 gap-16 mb-8">  {/* 24px padding, 16px gap, 8px bottom margin */}

✅ For Border Radius (pixel-based values):

<div className="rounded-8">  {/* 8px border radius */}
<div className="rounded-6">  {/* 6px border radius */}
<div className="rounded-4">  {/* 4px border radius */}

✅ For Border Width (explicit pixel values):

<div className="border-1">  {/* 1px border */}

❌ Avoid for Component Heights/Widths (risk of creating undersized elements):

{/* WRONG - Results in a 9px tall button! */}
<Button className="h-9" />
{/* WRONG - Creates a 4px × 4px icon! */}
<Icon className="size-4" />
Working with Components

Problem: Many UI components (buttons, icons, inputs) are designed to use rem-based h-* and size-* classes, which do not translate to the Firecrawl system’s pixel-based values.

Solution: Use explicit pixel values for component dimensions that require larger sizes:

{/* Instead of size-4 (4px), use explicit dimensions */}
<Icon className="w-16 h-16" />  {/* 16px × 16px icon */}
{/* Alternatively, use the style prop for non-spacing dimensions */}
<Icon style={{ width: '1rem', height: '1rem' }} />  {/* 16px × 16px */}
Common Component Fixes

Input Components:

// ❌ WRONG - Creates a 9px tall input field
<Input className="h-9 px-3 py-1 rounded-md border text-sm" />
// ✅ CORRECT - Creates a properly sized 40px tall input
<Input className="h-40 px-12 py-8 rounded-6 border-1 text-body-input" />

Button Components:

// ❌ WRONG - Creates a 9px tall button
<Button className="h-9 px-4 gap-2">Click me</Button>
// ✅ CORRECT - Creates a standard 36px tall button
<Button className="h-36 px-16 gap-8">Click me</Button>

Textarea Components:

// ❌ WRONG - Applies insufficient 10px vertical padding
<Textarea className="py-10 px-12" />
// ✅ CORRECT - Uses appropriate 16px padding
<Textarea className="py-16 px-16" />

Icon Components:

// ❌ WRONG - Results in a tiny 4px × 4px icon
<Icon className="size-4" />
// ✅ CORRECT - Renders a standard 16px × 16px icon
<Icon className="w-16 h-16" />
Migration Guide for Legacy Components

When porting components from standard Tailwind projects to the Firecrawl system:

  1. Spacing values remain consistent – p-24 still equals 24px of padding
  2. Heights require conversion – h-9 (36px in standard Tailwind) becomes h-144 (144px) or uses inline styles
  3. Sizes require conversion – size-4 (16px in standard Tailwind) becomes size-64 (64px) or uses explicit dimensions
  4. Border radius uses pixel values – Replace rounded-lg with rounded-8
  5. Border width is explicit – Use border-1 instead of the generic border class
Typography Exception

Notably, typography uses semantic sizing that is unaffected by the custom pixel-based system:

<h1 className="text-title-h3">  {/* Uses the predefined 40px size from the typography config */}
<p className="text-body-medium">  {/* Uses the predefined 14px size from the typography config */}

Border Radius System

Border radius uses pixel-based numeric values instead of standard Tailwind’s named classes (e.g., sm, md, lg):

// Available values: rounded-{0-32} in 1px increments
rounded-0   // 0px (no rounding)
rounded-4   // 4px (for small buttons and badges)
rounded-6   // 6px (for inputs and small cards)
rounded-8   // 8px (for large cards and modals)
rounded-16  // 16px (for highly rounded elements)
rounded-32  // 32px (maximum rounding)
rounded-full // 999px (perfect circles or pills)

Common Usage Patterns:

  • Small UI elements (buttons, badges): rounded-4
  • Medium components (inputs, small cards): rounded-6
  • Large components (cards, modals): rounded-8
  • Circular/pill-shaped elements: rounded-full

❌ Do not use standard Tailwind border radius classes:

{/* WRONG - These classes are not defined in the Firecrawl config */}
<div className="rounded-sm rounded-md rounded-lg rounded-xl" />
{/* CORRECT - Use pixel-based values */}
<div className="rounded-4 rounded-6 rounded-8 rounded-16" />

Opacity System

The design system includes a custom opacity scale ranging from 0 to 99 (percentage-based):

// Available values: opacity-{0-99}
opacity-0   // 0% (completely invisible)
opacity-10  // 10% (very faint)
opacity-50  // 50% (semi-transparent)
opacity-80  // 80% (mostly opaque)
opacity-100 // 100% (fully opaque) - use sparingly; prefer opacity-99 for better visual consistency

Practical Examples:

<div className="opacity-50 hover:opacity-100">Fade in on hover</div>
<div className="bg-black opacity-20">Subtle background overlay</div>

Transition System

Custom transition timing functions and durations ensure consistent animation behavior across the application:

Default Timing Function

All transition classes use this cubic-bezier curve by default for smooth, natural motion:

transition-timing-function: cubic-bezier(0.25, 0.1, 0.25, 1)

Transition Durations

Duration classes use a multiplier system where each numeric value equals 50ms (e.g., duration-4 = 200ms):

duration-0   // 0ms (instant transition)
duration-4   // 200ms (default quick transition)
duration-10  // 500ms (moderate transition)
duration-20  // 1000ms (slow transition)
duration-40  // 2000ms (very slow transition)

Usage Examples:

<div className="transition-all duration-4">Quick state transition (200ms)</div>
<div className="transition-opacity duration-10">Moderate fade effect (500ms)</div>

Custom Animations

The system includes a set of predefined keyframe animations for common UI interactions:

// Accordion animations
animate-accordion-down  // Smooth accordion expansion
animate-accordion-up    // Smooth accordion collapse

// Fade animations
animate-fade-in        // Fade from 0% to 100% opacity on mount
animate-fade-up        // Fade in while sliding up 10px

// Special effects
animate-screenshot-scroll  // 15-second infinite screenshot scroll animation
animate-selection-pulse-green  // Green pulsing effect for selected items
animate-button-press   // Button press feedback (scale down then up)

Implementation Examples:

<div className="animate-fade-in">Fades in automatically when rendered</div>
<button className="animate-button-press">Press for tactile feedback</button>

Recommended Spacing Values

To maintain visual consistency across the application, the design system specifies standard spacing values for different use cases:

Micro Spacing (Tight Layouts)

  • gap-4, p-4, m-4 = 4px
  • gap-8, p-8, m-8 = 8px

Standard Spacing (Most Common Use Cases)

  • gap-12, p-12, m-12 = 12px
  • gap-16, p-16, m-16 = 16px
  • gap-24, p-24, m-24 = 24px

Large Spacing (Sections and Containers)

  • gap-32, p-32, m-32 = 32px
  • gap-48, p-48, m-48 = 48px
  • gap-64, p-64, m-64 = 64px

Extra Large Spacing (Page Layouts)

  • gap-80, p-80, m-80 = 80px
  • gap-96, p-96, m-96 = 96px
  • gap-128, p-128, m-128 = 128px

Fractional Percentage Spacing (also supported for responsive layouts):

w-1/2   // 50% width
w-1/3   // 33.3% width
w-2/3   // 66.6% width
w-1/4   // 25% width
w-1/6   // 16.6% width
w-5/6   // 83.3% width

Responsive Breakpoints

The design system defines custom responsive breakpoints to support multiple device sizes:

screens: {
  xs: { min: "390px" },
  "xs-max": { max: "389px" },
  sm: { min: "576px" },
  "sm-max": { max: "575px" },
  md: { min: "768px" },
  "md-max": { max: "767px" },
  lg: { min: "996px" },
  "lg-max": { max: "995px" },
  xl: { min: "1200px" },
  "xl-max": { max: "1199px" }
}

Component Architecture

The Firecrawl design system’s component architecture is divided into three core layers: UI components, shared components, and application-specific components.

UI Components (components-new/ui/)

The UI layer combines three specialized component libraries to deliver diverse functionality:

shadcn/ui Components

High-quality, accessible React components that form the foundation of the UI:

  • Form controls: Button, Input, Select, Checkbox, Switch
  • Layout components: Card, Sheet, Dialog, Tabs, Accordion
  • Navigation components: NavigationMenu, DropdownMenu, ContextMenu
  • Feedback components: Toast, Alert, Progress, Badge
  • Data components: Table, DataTable, Calendar
Magic UI Components

Animated and interactive components that enhance user engagement:

  • animated-shiny-text: Shimmering text effects for emphasis
  • animated-list: Animated transitions for list items
  • dot-pattern: Decorative background dot patterns
  • dock: macOS-style dock component for navigation
  • ripple: Ripple effect for interactive elements
  • gradual-spacing: Animated text spacing for visual interest
Tremor Components

Data visualization and dashboard-focused components for analytics:

  • Charts: LineChart, BarChart, AreaChart
  • Controls: Button, Badge, Dropdown
  • Layout: Card, Calendar, DatePicker
  • Progress indicators: ProgressBar

Shared Components (components-new/shared/)

Shared components are reusable across the application and include utility and brand-specific elements:

Icons (shared/icons/)

Organized exports for brand and utility icons, ensuring consistent usage:

// Brand Icons
export { default as SymbolWhite } from './symbol-white';
export { default as SymbolColored } from './symbol-colored';
export { default as WordmarkWhite } from './wordmark-white';
export { default as WordmarkColored } from './wordmark-colored';

// Utility Icons
export { default as AnimatedLogo } from './animated-logo';
export { default as Check } from './check';
export { default as GitHub } from './github';
Buttons (shared/buttons/)

Custom button components styled to match the Firecrawl brand:

export { SlateButton } from './slate-button';
export { HeatButton } from './heat-button';
export { FireActionLink } from './fire-action-link';
Layout Components

Specialized layout utilities for common UI patterns:

  • curvy-rect: Curved rectangle shapes for decorative elements
  • animated-height: Smooth height transitions for collapsible content
  • animated-width: Smooth width transitions for responsive elements
  • unified-blur-overlay: Consistent backdrop blur effects for modals and overlays
Effects and Animations

Visual effects that add polish to the user experience:

  • flame/: Fire-themed animation effects for brand alignment
  • animated-beam: Connecting beam animations for showing data flow
  • data-sources-beam: Visualization for data source connections

Application Components (components-new/app/)

Application-specific components are built for unique pages and features within Open Scouts:

Brand Components (app/brand/)

Components dedicated to brand presentation and asset management:

  • brand-hero.tsx: Hero section for the brand assets page
  • brand-assets-copy.tsx: Functionality for copying brand asset links
  • brand-assets-download.tsx: Asset download functionality
  • brand-group.tsx: Grouping component for organizing brand assets
  • firecrawl-logo.tsx: Multiple variants of the Firecrawl logo
  • firecrawl-wordmark.tsx: Text-based brand wordmark components

Example implementation of the brand hero component:

export default function BrandHero() {
  return (
    <section className='max-w-[1112px] mx-auto -mt-1'>
      <SectionHead
        action={(
          <a className="contents" href="/brand/brand-assets.zip" download>
            <Button className='mx-auto' size='large' variant='primary'>
              Download brand assets
            </Button>
          </a>
        )}
        description="Welcome to the Firecrawl brand hub..."
        title={<><span className="text-heat-100">Firecrawl </span> Brand Assets</>}
        titleClassName='text-title-h3'
      >
        <DeveloperFlame />
      </SectionHead>
    </section>
  );
}

Brand Assets

The Firecrawl brand assets are stored in the public/brand/ directory and include logos, marketing materials, and distribution packages.

Asset Organization

Logo Variants
  • firecrawl-logo.svg/png: Primary brand logo
  • firecrawl-light-logo.svg/png: Light theme logo variant for dark backgrounds
  • firecrawl-wordmark.svg/png: Text-only brand wordmark
  • firecrawl-light-wordmark.svg/png: Light theme wordmark variant
  • firecrawl-icon.png: App icon for mobile and desktop applications
  • firecrawl-app-icon.png: Alternative app icon variant
Special Assets
  • firecrawl-logo-transparent.png: Logo with a transparent background for flexible usage
  • firecrawl-logo-with-fire.png: Logo incorporating the brand’s fire element
  • logo_fire.png: Standalone fire element for decorative usage
Marketing Materials
  • scrape-data-from-any-site--firecrawl.jpg
  • turn-websites-into-llm-ready-data--firecrawl.jpg
  • we-handle-all-the-hard-stuff--firecrawl.jpg
  • trusted-by-devs-at-top-companies--firecrawl.jpg
Asset Distribution Package
  • brand-assets.zip: Complete bundle of all brand assets for easy download and distribution

Brand Usage Guidelines

To maintain brand consistency, the design system provides clear guidelines for asset usage:

Logo Usage Rules
  • Use firecrawl-logo.svg for primary brand representation in most contexts
  • Use firecrawl-light-logo.svg when displaying the logo on dark backgrounds
  • Use firecrawl-wordmark.svg in space-constrained layouts where the full logo does not fit
  • Maintain proper spacing and sizing ratios around all logo variants to preserve brand integrity
Color Usage Rules
  • The primary brand color is heat-100 (#fa5d19), to be used for key brand elements and calls to action
  • Accent colors should be used sparingly, only for highlighting specific UI elements or CTAs
  • Ensure all color combinations meet accessibility contrast ratios to support users with visual impairments

PostHog Integration: Tracking User Behavior and Platform Health

Open Scouts integrates PostHog, a product analytics tool, to understand how users interact with the platform and measure overall product performance.

What We Track

PostHog captures two core categories of data: user journey events and execution performance metrics.

User Journey Tracking

The platform tracks the complete user lifecycle from signup to active engagement:

  1. Authentication: Events for sign-up, login, and Google OAuth initiation
  2. Scout Creation: Events triggered when users create new monitoring scouts
  3. Configuration Progress: Events tracking completion of scout setup steps (goal definition, query configuration, location setting, frequency selection)
  4. Activation: Events when users enable scouts for automated monitoring
  5. Ongoing Engagement: Events for manual scout runs, result views, and scout management actions

Execution Metrics (Server-Side)

Server-side tracking captures the technical performance of scout operations:

  • Execution success and failure rates
  • Execution duration and number of steps completed
  • Duplicate detection (when results match previous scout runs)
  • Email notification delivery status (success/failure)

Key Funnels for Performance Analysis

Funnels help identify bottlenecks in user behavior and platform functionality:

Activation Funnel

Measures how effectively users progress from signup to having an active scout:

Sign Up → Create Scout → Complete Configuration → Activate Scout

Engagement Funnel

Tracks ongoing user interaction with their scouts after activation:

View Results → Manual Trigger → Check Email Notification

Product Health Metrics

The following metrics provide a holistic view of platform performance:

Metric Description
Execution Success Rate Percentage of scout runs that complete without technical errors
Duplicate Detection Rate Percentage of runs that identify already-reported information (to avoid redundant alerts)
Configuration Completion Percentage of created scouts that have all required fields configured
Activation Rate Percentage of fully configured scouts that users enable for automated monitoring
Email Delivery Rate Percentage of notification emails that are successfully sent to users

PostHog Setup

Configuring PostHog requires setting environment variables and implementing a multi-layered tracking architecture.

Environment Variables

Add these variables to your .env file to enable PostHog integration:

NEXT_PUBLIC_POSTHOG_KEY=your_project_api_key
NEXT_PUBLIC_POSTHOG_HOST=https://us.posthog.com  # Optional; defaults to PostHog cloud

The PostHog API key must also be accessible to Supabase edge functions to enable server-side event tracking.

Tracking Architecture

PostHog tracking is implemented across three layers of the application for comprehensive data capture:

┌─────────────────────────────────────────────────────────────┐
│                        Browser                               │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  posthog-js (instrumentation-client.ts)             │    │
│  │  - User identification                              │    │
│  │  - Page views & UI interactions                     │    │
│  │  - Client-side events                               │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                    Next.js Server                            │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  posthog-node (lib/posthog-server.ts)               │    │
│  │  - Server-side event capture                        │    │
│  │  - API route tracking                               │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘
                              │
                              ▼
┌─────────────────────────────────────────────────────────────┐
│                 Supabase Edge Functions                      │
│  ┌─────────────────────────────────────────────────────┐    │
│  │  HTTP Capture API (supabase/functions/.../posthog.ts)│    │
│  │  - Execution lifecycle events                       │    │
│  │  - Email notification tracking                      │    │
│  │  - Performance metrics                              │    │
│  └─────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────┘

Events Reference

A comprehensive list of tracked events and their trigger conditions:

Authentication Events

Event Trigger Condition
user_signed_up When a new user creates an account
user_logged_in When an existing user logs into their account
google_auth_initiated When a user clicks the Google OAuth login button

Scout Management Events

Event Trigger Condition
scout_created When a user creates a new scout
scout_configuration_completed When all required fields for a scout are filled
scout_activated When a user enables a scout for automated monitoring
scout_deactivated When a user disables an active scout
scout_deleted When a user permanently deletes a scout

Execution Events (Server-Side)

Event Trigger Condition
scout_execution_started When a scout begins its scheduled or manual run
scout_execution_completed When a scout finishes running without errors
scout_execution_failed When a scout run encounters a technical error
scout_duplicate_detected When a scout’s results match a previous run’s findings
scout_email_notification_sent When the system attempts to send a notification email

User Engagement Events

Event Trigger Condition
scout_execution_triggered When a user manually runs a scout
scout_results_viewed When a user accesses a scout’s execution results
execution_history_cleared When a user deletes a scout’s execution history

Settings Events

Event Trigger Condition
test_email_sent When a user sends a test notification email
firecrawl_key_regenerated When a user regenerates their Firecrawl API key
location_updated When a user changes their default location setting

Adding New Events

Implementing new tracked events varies by application layer:

Client-Side Events

To track events in the browser, use the posthog-js library:

import posthog from "posthog-js";
posthog.capture("event_name", {
  relevant_property: "value",
});

Server-Side Events (API Routes)

For events in Next.js API routes, use the server-side PostHog client:

import { getPostHogClient } from "@/lib/posthog-server";
const posthog = getPostHogClient();
posthog.capture({
  distinctId: user.id,
  event: "event_name",
  properties: { ... },
});

Edge Function Events

To track events in Supabase edge functions, use the dedicated capture utility:

import { captureEvent } from "./posthog.ts";
captureEvent({
  event: "event_name",
  distinctId: userId,
  properties: { ... },
});

Privacy Considerations

Open Scouts prioritizes user privacy in its analytics implementation:

  • User identification uses the Supabase user.id (not email addresses) as the primary identifier
  • Email addresses are stored only as user properties for customer support purposes, not for analytics
  • No personally identifiable information (PII) is included in event properties
  • PostHog’s proxy path (/ingest) is used to improve data collection reliability while maintaining privacy

Frequently Asked Questions (FAQ)

To address common user questions and concerns, we’ve compiled this FAQ section:

Basic Open Scouts Questions

  1. What use cases is Open Scouts best suited for?
    Open Scouts excels at any scenario requiring continuous web monitoring, including tracking industry news, monitoring local business openings, following product price changes, and staying updated on niche topic developments.

  2. Do I need programming experience to use Open Scouts?
    Basic usage does not require advanced programming skills, as the platform provides a user-friendly interface for creating scouts. However, setting up the platform locally requires familiarity with terminal commands and configuration file management.

Technical and Setup Questions

  1. Are all the listed technologies required for Open Scouts to work?
    Yes, each technology serves a critical role in the platform’s functionality: Supabase manages the database and authentication, Firecrawl handles web scraping, OpenAI powers AI analysis, and Resend delivers email notifications. Omitting any component will break core features.

  2. Is the Resend free tier sufficient for personal use?
    The free tier (3,000 emails per month, 100 per day) is typically enough for individual users with a small number of scouts. For enterprise or high-volume use cases, upgrading to a paid tier is recommended.

  3. How can I verify if my scouts are running correctly?
    You can check the execution history on each scout’s page, which displays status (success/failure) and error messages for failed runs. Email notifications will also confirm when scouts find new results.

Design System Questions

  1. Why does the Firecrawl design system use pixel-based sizing instead of standard Tailwind rem units?
    The custom pixel-based system enables precise control over UI element dimensions, ensuring consistent visual alignment across the application, especially for brand-critical components that require exact sizing.

  2. Can I customize the design system’s color palette?
    The color palette can be modified by updating the colors.json file and adjusting the Tailwind configuration, but we recommend preserving the primary heat-100 brand color to maintain visual consistency with the Firecrawl ecosystem.

PostHog Analytics Questions

  1. Is the user data collected by PostHog secure?
    Yes, all tracked data is secured with industry-standard encryption, and no PII is included in event properties. User identification relies on non-identifiable Supabase user.id values instead of personal information.

  2. Can I disable PostHog analytics if I don’t want to track user behavior?
    While there is no built-in toggle, you can disable analytics by omitting the PostHog environment variables from your .env file during setup, which will prevent all event tracking.

Conclusion

Open Scouts is a powerful, AI-driven monitoring platform that empowers users to create custom web monitors that work around the clock to track the information that matters most to them. With a robust technical stack, user-friendly interface, and comprehensive design system, it delivers both functionality and visual consistency.

The platform’s scalable architecture ensures it can handle thousands of concurrent scouts, while its PostHog integration provides valuable insights into user behavior and system performance. Whether you’re a personal user tracking niche interests or a team monitoring industry developments, Open Scouts offers a reliable, customizable solution for web information gathering.

By following the deployment guide outlined in this article, you can set up your own instance of Open Scouts and start leveraging the power of AI to stay ahead of the curve with timely, relevant web information.