Crawler policy

robots.txt and crawler policy

pasal.biz operates a crawler (PasalBot) to index Nepali product pages for shopper search. This page documents exactly which signals we honour and what each one does to your listing.

Updated 2026-05-22

The two signals we read

We honour the two standard robots-control mechanisms the rest of the web already uses, so you don't have to learn anything pasal-specific.

robots.txt at yourshop.com.np/robots.txt — site-wide rules that apply to all crawlers (or to ours specifically as User-agent: PasalBot).
<meta name="robots"> in the <head> of an individual page — per-page directives that override the site-wide rule for just that URL.

What each signal does on pasal.biz

Directive	Where	Effect on pasal.biz
`Disallow: /` for `PasalBot`	robots.txt	We stop crawling on the next pass. Existing listings remain until the next re-sync attempts to refresh them and is blocked, then they age out.
`Disallow: /some-path/`	robots.txt	We never fetch URLs under that path. Useful for hiding internal sections like `/admin/` or `/cart/` while still allowing product pages.
`Crawl-delay: N`	robots.txt	We wait at least N seconds between requests to your server. Capped at 30s on our side so a hostile `robots.txt` can't stall the run forever.
`noindex`	meta robots	That specific product page is skipped — not added to our index. We still fetched it to read the meta tag, but no listing is created.
`none`	meta robots	Equivalent to `noindex, nofollow`. Same effect as `noindex` for us.
`noarchive` / `nocache`	meta robots	We still create the listing (title, photo, price, link) but we do not store the product description body. Shoppers see the listing card and are forwarded to your site; no description snippet is shown on pasal.biz.
`nosnippet`	meta robots	Same effect as `noarchive` on pasal.biz — description body is suppressed.

The googlebot meta name is also read, so if you're already controlling Google with <meta name="googlebot">, the same directives apply to PasalBot. The catch-all <meta name="robots"> takes precedence when both are present.

Recipes

Block PasalBot entirely

User-agent: PasalBot
Disallow: /

Stay indexed but suppress cached snippets

On each product page, add this to the <head>:

<meta name="robots" content="noarchive">

Listing card still appears on pasal.biz; description body is not stored.

Hide one specific product from pasal.biz

<meta name="robots" content="noindex">

Slow us down without blocking

User-agent: PasalBot
Crawl-delay: 10

What we do not read (yet)

For transparency, here's what is on the standards list but not in our v1 reader:

X-Robots-Tag HTTP response header. We're moving to honour this in a future release; until then, put the directive in a <meta name="robots"> tag so we pick it up.
Wildcard (*) and end-anchor ($) patterns in robots.txt. If you ship a wildcard rule we will over-fetch — never under-fetch — until the parser is upgraded.
max-snippet, max-image-preview, and similar quantitative directives. Treated as no-ops; we don't pretend to enforce them.

For anything not covered by the directives above, email [email protected].

Removal request

If you need a listing removed immediately and editing your site's robots.txt or meta tags isn't an option, see the takedown request page. We act on takedown emails directly without waiting for the next crawl pass.