My Hermes Agent does a lot of web work: it extracts the content of pages as clean markdown, and it searches the web. Both go through Firecrawl, a scraping API that exists as a hosted cloud service and as an open source, self-hostable stack. When I described my Hermes setup I mentioned the Firecrawl integration, but not where it points: to my own instance of Firecrawl, running in Docker next to the agent on my home server.

Why self-host it

• No usage meter. The cloud plan is usage-based, and an agent is bursty: several pages per question, many questions per day.
• Privacy. Every page the agent reads passes through the scraper. Local means that traffic stays on my own server.
• It fits the stack. Hermes already runs in a Docker container next to my other self-hosted services, so a local Firecrawl is one more folder in the same stack, not a new subscription.

The stack

Self-hosted Firecrawl is a Docker Compose stack, not a single container: the API on port 3002, a Playwright microservice that renders JavaScript-heavy pages, and supporting services for queues and storage (Redis, RabbitMQ, Postgres, FoundationDB). Together it sits around 4 GB of RAM on my server.

One environment variable matters most: with USE_DB_AUTHENTICATION=false, the API requires no key, so calls from my local network need no credentials. And I pinned the clone to a release tag instead of main: the compose contract changes between releases, and upstream verifies a specific release in its own self-hosting guide.

Setting it up

Clone the repo pinned to the release you want:

git clone --depth 1 --branch v2.11.162 https://github.com/firecrawl/firecrawl
cd firecrawl

Create a minimal .env (the template in their self-hosting guide lists many optional variables, but these three boot the stack):

PORT=3002
HOST=0.0.0.0
USE_DB_AUTHENTICATION=false

Build and start:

docker compose build
docker compose up -d

The default compose file builds the API, Playwright and Postgres images from source; prebuilt images exist on ghcr.io if you prefer to swap the build: lines for image: lines. The first start takes a while. Afterwards, test with a real scrape:

curl http://localhost:3002/v0/health/readiness
curl -X POST http://localhost:3002/v1/scrape \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "formats": ["markdown"]}'

The readiness endpoint only says the process is up; the scrape proves the workers actually return content.

Search: SearXNG next to the API

The self-hosted API exposes a POST /v2/search endpoint, but the cloud search backend (fire-engine) is not part of the self-hosted package, so search falls back to DuckDuckGo. That fallback rate-limits: during bursts of agent searches, my server IP got blocked (“Blocked by anti-bot measures”).

The fix is a SearXNG container next to the API: it fans each query out to many search engines at once. Two additions. A compose.override.yaml with the service and the endpoint variable on the API:

services:
  api:
    environment:
      SEARXNG_ENDPOINT: http://searxng:8080
  searxng:
    image: searxng/searxng:latest
    ports:
      - "127.0.0.1:8081:8080"
    volumes:
      - ./searxng:/etc/searxng

And a searxng/settings.yml enabling the JSON format the API consumes, with the limiter off since the caller is my own agent:

use_default_settings: true
server:
  secret_key: "<a random string>"
  limiter: false
search:
  formats:
    - html
    - json

With that, /v2/search tries SearXNG first and keeps DuckDuckGo as an automatic fallback when it returns nothing:

curl -X POST http://localhost:3002/v2/search \
  -H 'Content-Type: application/json' \
  -d '{"query": "home assistant docker", "limit": 3}'

One Docker gotcha worth knowing: bind mounts resolve on the host filesystem. If the SearXNG container boots with the bundled template instead of your settings.yml, the file did not reach the host mount; write it on the host and recreate the container.

Pointing Hermes at it

Hermes picks Firecrawl from its environment file (~/.env):

FIRECRAWL_API_URL=http://172.20.0.1:3002
# FIRECRAWL_API_KEY=...  # commented out: when the URL is set, it wins

When FIRECRAWL_API_URL is set, Hermes talks to that URL and needs no API key; set only the key, and it goes to the cloud service. The web backend in Hermes’ config is set to firecrawl.

The address 172.20.0.1 is the Docker bridge gateway: Hermes runs in its own container, and localhost:3002 inside that container does not reach a port published on the host. If your agent runs directly on the host, http://localhost:3002 is enough.

What you don’t get

Self-hosted Firecrawl does not include fire-engine, Firecrawl’s anti-bot layer with proxies and block handling. Pages behind aggressive bot walls fail locally where the cloud service would pass. Screenshots and page actions are also cloud-only, and a few cloud-only response formats are rejected by the self-hosted API (the 400 error lists what your instance accepts).

The core paths all work: scraping to markdown, JavaScript rendering, crawling and search. And when a scrape really needs the cloud’s anti-bot layer, the fallback is the commented key in ~/.env: comment the URL line, uncomment the key, and Hermes is back on the cloud service.