---
name: robots-txt-tester
description: "Robots.txt Tester (Robots.txt Tester) via devops.majbase.com's free public API, no key required. Keywords: test robots.txt rules, check if page is crawlable."
---

# Robots.txt Tester

Robots.txt Tester. Calls devops.majbase.com's free public API — no API key or authentication required.

## Endpoint

```
GET/POST https://devops.majbase.com/robots-txt-tester/api
```

## Request / Response

```
API: GET or POST /robots-txt-tester/api
  robots_txt=<string>   (required — raw robots.txt content; fetch it yourself
                          first if you need the live version of a domain's)
  user_agent=<string>    (optional, default '*')
  path=<string>           (optional, default '/')
Returns {"groups", "sitemaps", "allowed", "matched_rule", "matched_user_agent"},
or {"error": "..."}.
```

## Example

```bash
curl -sS -X POST https://devops.majbase.com/robots-txt-tester/api -H "Content-Type: application/json" \
  -d '{"robots_txt":"User-agent: *\nDisallow: /admin/","user_agent":"Googlebot","path":"/admin/page"}'
```

## Notes

- No API key or authentication required.
- Full interactive tool: https://devops.majbase.com/robots-txt-tester/