Docs

Ingestion

Run the ingest CLI to load crawler output into Supabase and populate the Stocks view.

Open stocks

Quick start

Run the CLI from the repo root to ingest crawler output into Supabase.

  1. Prepare a crawler output JSON file (single envelope or NDJSON).
  2. Run the ingest command.
  3. Open the Stocks page to confirm tickers appear.
pnpm ingest --file /path/to/crawler-output.json
pnpm ingest --file /path/to/crawler-output.json --dry-run

The CLI prints an ingestion summary with counts for sources, items, and mentions.

Requirements

The ingest CLI writes with the Supabase service role key.

  • Set SUPABASE_URL and SUPABASE_SERVICE_ROLE_KEY in a .env or .env.local file.
  • scripts/lib/load-env.js loads env files from the repo root, apps/web, and apps/worker.
  • Use --dry-run to validate payloads without writing data.
SUPABASE_URL="https://your-project.supabase.co"
SUPABASE_SERVICE_ROLE_KEY="your-service-role-key"

Input format

Accepts JSON (single envelope or array) or NDJSON (one envelope per line).

  • source.name is required to identify the origin.
  • Each item needs external_id and at least one stock ticker.
  • published_at is optional but recommended for accurate recency.
{
  "source": {
    "name": "Reddit r/stocks",
    "type": "reddit",
    "base_url": "https://www.reddit.com/r/stocks"
  },
  "items": [
    {
      "external_id": "t3_example",
      "title": "NVDA momentum keeps building",
      "url": "https://www.reddit.com/r/stocks/comments/example",
      "published_at": "2024-06-18T12:34:00Z",
      "stocks": [
        { "ticker": "NVDA", "count": 3, "confidence": 0.9 }
      ]
    }
  ]
}