Agent Utilities
← Web policies & fetch

web.robots-evaluate · v1.0.0

Robots Evaluate
for your agent.

Evaluate supplied robots.txt for a URL and user agent; no network fetch.

Use this tool View JSON contract
$0.0003 / call300 micro-dollar credits
131,072 bytes max input65,536 bytes max output
Works on supplied inputNo external network fetch

Example input

{
  "url": "https://example.com/private",
  "robots": "User-agent: *\nDisallow: /private",
  "userAgent": "ExampleBot"
}

Example output

{
  "allowed": false,
  "crawlDelay": null,
  "sitemaps": [],
  "scope": "Evaluates supplied policy only; does not establish authorization to access a site."
}

Generated from the actual handler using this example input; this is a preview, not a paid API call. Paid responses wrap the result with a receipt and recovery expiry.

Call it from your agent

Add credits in the workspace and keep your API key in private environment storage. Use the same request ID and exact body to recover a lost response within ten minutes.

// Node.js 22+. Read the key from your private environment.
const requestId = Date.now() + '_' + crypto.randomUUID();
const response = await fetch('https://agent-utilities.agent-utilities.workers.dev/v1/tools/web.robots-evaluate', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': 'Bearer ' + process.env.AGENT_UTILITIES_API_KEY,
    'Idempotency-Key': requestId
  },
  body: JSON.stringify({
  "url": "https://example.com/private",
  "robots": "User-agent: *\nDisallow: /private",
  "userAgent": "ExampleBot"
})
});
const data = await response.json();
if (!response.ok) throw new Error(data.error?.message ?? 'Request failed');
console.log(data);
// On response loss, retry this SAME requestId, tool and body.
// Do not rerun the line that generates requestId for a retry.
Input JSON Schema
{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "maxLength": 2048
    },
    "robots": {
      "type": "string",
      "maxLength": 100000
    },
    "userAgent": {
      "type": "string",
      "minLength": 1,
      "maxLength": 200
    }
  },
  "required": [
    "url",
    "robots",
    "userAgent"
  ],
  "additionalProperties": false
}
Output JSON Schema
{
  "type": "object",
  "properties": {
    "allowed": {
      "type": [
        "boolean",
        "null"
      ]
    },
    "crawlDelay": {
      "type": [
        "number",
        "null"
      ]
    },
    "sitemaps": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "scope": {
      "type": "string"
    }
  },
  "required": [
    "allowed",
    "crawlDelay",
    "sitemaps",
    "scope"
  ],
  "additionalProperties": false
}

Before you integrate

Prices are experimental. The service currently allows 1,000 new calls per day across all accounts. Failed operations return reserved credits. Successful results are encrypted for ten-minute retry recovery. Review data handling and service limits.

Related tools

web $0.0020 / call

Readable Fetch

Fetch readable text from an approved public HTTPS host.

web.readable-fetch

web $0.0010 / call

Redirect Trace

Trace up to four redirects across approved public HTTPS hosts.

web.redirect-trace