Let crawlers ask, and tell them there is nothing to index

GET

/robots.txt

const url = 'https://api.sloose.com/robots.txt';
const options = {method: 'GET'};

try {
  const response = await fetch(url, options);
  const data = await response.json();
  console.log(data);
} catch (error) {
  console.error(error);
}

curl --request GET \
  --url https://api.sloose.com/robots.txt

Unauthenticated. User-agent: * and Allow: /: crawlers may ask, and every response of the API carries X-Robots-Tag: noindex, nofollow (a Disallow would stop a crawler reading that header, and an address it may not fetch can still be listed).

Responses

200

The robots file.

text/plain

string

Report a problem with this page

sloose.com