web.robots-evaluate

Run a task for free

This starter input is synthetic. Replace it with your own public or sanitized input. It sends one request when you press Run.

Inspect the API contract

Read this tool’s schema and execution URL as JSON · Read a fixed input/output example

Use HTTP or MCP

POST https://agent-utilities.agent-utilities.workers.dev/v1/tools/web.robots-evaluate
Content-Type: application/json

{
  "url": "https://example.com/private",
  "robots": "User-agent: *\nDisallow: /private",
  "userAgent": "ExampleBot"
}

No key or payment. Hosted MCP full tool: web_robots_evaluate, arguments:

{
  "input": {
    "url": "https://example.com/private",
    "robots": "User-agent: *\nDisallow: /private",
    "userAgent": "ExampleBot"
  }
}

Input schema

{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "maxLength": 2048
    },
    "robots": {
      "type": "string",
      "maxLength": 100000
    },
    "userAgent": {
      "type": "string",
      "minLength": 1,
      "maxLength": 200
    }
  },
  "required": [
    "url",
    "robots",
    "userAgent"
  ],
  "additionalProperties": false
}

Output schema

{
  "type": "object",
  "properties": {
    "allowed": {
      "type": [
        "boolean",
        "null"
      ]
    },
    "crawlDelay": {
      "type": [
        "number",
        "null"
      ]
    },
    "sitemaps": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "scope": {
      "type": "string"
    }
  },
  "required": [
    "allowed",
    "crawlDelay",
    "sitemaps",
    "scope"
  ],
  "additionalProperties": false
}

Input ≤128 KiB; output ≤64 KiB. Schema limits may be smaller. Network sources retain their access and upstream limits. Free results are not stored for recovery.