Файл robots.txt

Источник Robots.txt: на сайте http://astanafans.com/robots.txt на момент обработки проекта.

User-agent: *
Disallow: /?comments_popup=
Disallow: /?p=
Disallow: /?attachment_id=

Disallow: /*.html/*
Disallow: /*.html/feed



### Topics
Disallow: /topics/*/page/*

### Archives
Disallow: /2007/*
Disallow: /2008/*
Disallow: /2009/*
Disallow: /2010/*
Disallow: /2011/*
Disallow: /2012/*
Disallow: /2013/*
Disallow: /2014/*
Disallow: /2015/*
Disallow: /2016/*
Disallow: /2017/*
Disallow: /2018/*
Disallow: /2019/*
Disallow: /2020/*


### WP
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /xmlrpc.php
Disallow: /archives/
Disallow: /tags/
Disallow: /tags/*/feed
Disallow: /wap.php*
Disallow: /wap-comments.php*



### Search
Disallow: /?s=
Disallow: /*/?s=

### Feed
Disallow: /*/feed


### Forum
Disallow: /forum/register
Disallow: /forum/viewforum.php?id=
Disallow: /forum/viewtopic.php?id=
Disallow: /forum/viewtopic.php?pid=
Disallow: /forum/profile.php?id=
Disallow: /forum/post*.html
Disallow: /forum/search-topics-user*.html
Disallow: /forum/search-posts-user*.html
Disallow: /forum/feed-atom*.xml
Disallow: /forum/feed-rss-*.xml
Disallow: /forum/extern.php?action=feed
Disallow: /forum/userlist.php
Disallow: /forum/admin/
Disallow: /forum/cache/
Disallow: /forum/search.php
Disallow: /forum/user


### Musar
Disallow: /?shoutboxOp=