Async Website Crawl and Extraction — POST /v1/crawl

Use POST /v1/crawl to recursively crawl an entire website and extract content from every page. The crawl runs asynchronously — the endpoint returns a job_id immediately and processes pages in the background via CROWD (Firecrawl). Poll GET /v1/crawl/{job_id} to check progress and retrieve results as they accumulate. If you need to stop a crawl early, send DELETE /v1/crawl/{job_id}.

Before crawling, use POST /v1/map to discover the URL structure of the site. Map is much faster and lets you filter include_paths or exclude_paths to a relevant subset before committing to a full crawl.

Start a Crawl

POST https://api.piramyd.cloud/v1/crawl

Authentication: Authorization: Bearer sk-<key>

Request Parameters

string

required

Root URL to begin crawling from. The crawler follows links discovered on each page, staying within the same domain.

integer

default:"10"

Maximum number of pages to crawl. Accepted range: 1–100.

integer

Maximum link depth from the root URL. Accepted range: 1–10. Omit to crawl without a depth limit (up to limit pages).

string[]

default:"[\"markdown\"]"

Output formats to extract per page. Any combination of "markdown", "html", "links", "screenshot".

boolean

default:"true"

Strip navigation, headers, footers, and boilerplate from each page, returning only main content.

string[]

URL path patterns to skip during the crawl. Supports glob syntax (e.g. ["/blog/*", "/tag/*"]). Pages matching any pattern are not fetched.

string[]

Only crawl URLs matching at least one of these path patterns. Pages that do not match are skipped. Useful for restricting crawls to a specific section (e.g. ["/docs/*"]).

Response

{
  "success": true,
  "job_id": "crawl_abc123",
  "url": "https://docs.python.org/3/"
}

Field	Type	Description
`success`	boolean	`true` if the crawl job was created successfully.
`job_id`	string	Unique job identifier. Use this to poll or cancel the crawl.
`url`	string	The root URL the crawl was started from.

Poll for Results

GET https://api.piramyd.cloud/v1/crawl/{job_id}

Poll this endpoint until status is completed, failed, or cancelled. Results accumulate progressively — data may already contain finished pages while status is still scraping.

Response

string

Current crawl status. One of:

scraping — crawl is in progress
completed — all pages processed successfully
failed — crawl encountered a fatal error
cancelled — crawl was stopped by a DELETE request

integer

Total number of pages queued for crawling.

integer

Number of pages that have finished processing so far.

integer

Number of CROWD credits consumed by this crawl job.

array

Array of page result objects. Each object contains the extracted content for one crawled page.

Show page result fields

string

Cleaned markdown content of the page (when "markdown" is in formats).

object

Page metadata.

Show metadata fields

string

The URL of this specific page.

integer

HTTP status code returned by the page.

Cancel a Crawl

DELETE https://api.piramyd.cloud/v1/crawl/{job_id}

Cancels a running crawl job. Any pages already processed are still available via GET /v1/crawl/{job_id} with status: "cancelled". Returns 200 on success.

Code Example

import httpx
import time

PIRAMYD_KEY = "sk-YOUR_KEY"
BASE = "https://api.piramyd.cloud/v1"
HEADERS = {"Authorization": f"Bearer {PIRAMYD_KEY}"}

# 1. Start the crawl
start = httpx.post(
    f"{BASE}/crawl",
    headers=HEADERS,
    json={
        "url": "https://docs.python.org/3/",
        "limit": 20,
        "max_depth": 3,
        "formats": ["markdown"],
        "only_main_content": True,
        "exclude_paths": ["/genindex/*", "/py-modindex/*"],
        "include_paths": ["/library/*"],
    },
)
start.raise_for_status()
job_id = start.json()["job_id"]
print(f"Crawl started: {job_id}")

# 2. Poll until completed
while True:
    poll = httpx.get(f"{BASE}/crawl/{job_id}", headers=HEADERS)
    poll.raise_for_status()
    status_data = poll.json()

    status = status_data["status"]
    completed = status_data.get("completed", 0)
    total = status_data.get("total", "?")
    print(f"Status: {status} ({completed}/{total} pages)")

    if status in ("completed", "failed", "cancelled"):
        break

    time.sleep(3)

# 3. Print results
if status == "completed":
    pages = status_data.get("data", [])
    print(f"\nCrawl complete — {len(pages)} pages retrieved")
    for page in pages:
        url = page["metadata"]["sourceURL"]
        preview = page.get("markdown", "")[:200].replace("\n", " ")
        print(f"\n  {url}\n  {preview}...")
else:
    print(f"Crawl ended with status: {status}")

{
  "url": "https://docs.python.org/3/",
  "limit": 20,
  "max_depth": 3,
  "formats": ["markdown"],
  "only_main_content": true,
  "exclude_paths": ["/genindex/*", "/py-modindex/*"],
  "include_paths": ["/library/*"]
}

{
  "status": "completed",
  "total": 20,
  "completed": 20,
  "creditsUsed": 20,
  "data": [
    {
      "markdown": "# Python 3.13 Documentation\n\n...",
      "metadata": {
        "sourceURL": "https://docs.python.org/3/",
        "statusCode": 200
      }
    }
  ]
}

Errors

Error code	HTTP status	Description
`crawl_timeout`	504	The crawl job timed out. Retry with backoff, or reduce `limit` / `max_depth`.
`crawl_upstream_error`	502	CROWD encountered an upstream error during crawling. Retry with backoff.
`crawl_unavailable`	502	The CROWD service is temporarily unavailable. Retry with backoff.
`crawl_internal_error`	500	Unexpected internal error. Retry with backoff.
`crawl_job_not_found`	404	No crawl job found with the given `job_id`. Check the ID or confirm the job wasn’t already purged.

Crawl jobs are ephemeral. Results are not stored indefinitely — retrieve them promptly after the status reaches completed. Polling a job_id for a purged job returns crawl_job_not_found (404).

Inference

Models

Web Intelligence

Account

Async Website Crawl and Extraction — POST /v1/crawl

Start a Crawl

Request Parameters

Response

Poll for Results

Response

Cancel a Crawl

Code Example

Errors

​Start a Crawl

​Request Parameters

​Response

​Poll for Results

​Response

​Cancel a Crawl

​Code Example

​Errors

Start a Crawl

Request Parameters

Response

Poll for Results

Response

Cancel a Crawl

Code Example

Errors