Voorbeeld robot.txt
Voorbeeld van ROBOT.TXT Met vertraging in het laten doorzoeken van een website!
Select & Copy
Wildcards etc
| Voorbeelden (ROBOT.TXT) |
/ |
Matches the root and any lower level URL. |
/* |
Equivalent to /. The trailing wildcard is ignored. |
/$ |
Matches only the root. Any lower level URL is allowed for crawling. |
/fish |
Matches any path that starts with /fish. Note that the matching is case-sensitive.
Matches:
/fish/fish.html/fish/salmon.html/fishheads/fishheads/yummy.html/fish.php?id=anything
Doesn’t match:
/Fish.asp/catfish/?id=fish/desert/fish
|
/fish* |
Equivalent to /fish. The trailing wildcard is ignored.
Matches:
/fish/fish.html/fish/salmon.html/fishheads/fishheads/yummy.html/fish.php?id=anything
Doesn’t match:
/Fish.asp/catfish/?id=fish/desert/fish
|
/fish/ |
Matches anything in the /fish/ folder.
Matches:
/fish//fish/?id=anything/fish/salmon.htm
Doesn’t match:
/fish/fish.html/animals/fish//Fish/Salmon.asp
|
/*.php |
Matches any path that contains .php.
Matches:
/index.php/filename.php/folder/filename.php/folder/filename.php?parameters/folder/any.php.file.html/filename.php/
Doesn’t match:
/ (even if it maps to /index.php)/windows.PHP
|
/*.php$ |
Matches any path that ends with .php.
Matches:
/filename.php/folder/filename.php
Doesn’t match:
/filename.php?parameters/filename.php//filename.php5/windows.PHP
|
/fish*.php |
Matches any path that contains /fish and .php, in that order.
Matches:
/fish.php/fishheads/catfish.php?parameters
Doesn’t match: /Fish.PHP
|
Tags: Robot.txtServerWebdesign
