🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 41 (from laksa170)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa041.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://video.shakhtar.com/ru/video/f5e44be0-e799-4ffa-bb55-e4758942fe2a\')), getAhrefsUnparsedNoserviceFromURL(\'https://video.shakhtar.com/ru/video/f5e44be0-e799-4ffa-bb55-e4758942fe2a\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/video.shakhtar.com\/ru\/video\/f5e44be0-e799-4ffa-bb55-e4758942fe2a","crawl_time":1759493517,"first_indexed_time":1564322320,"http_code":200,"src_unparsed":"com,shakhtar!video,\/ru\/video\/f5e44be0-e799-4ffa-bb55-e4758942fe2a s443","src_root_hash":"15812233617324491841","history_drop_reason":null,"meta_title":"Шахтер – Динамо – 3:1 (02.09.2012): как это было","meta_descriptions":[""],"attrs_boilerpipe_text":"© FC Shakhtar Donetsk, 1998-2025. All rights reserved.","attrs_markdown":"menu\n\n[![FC Shakhtar](https:\/\/video.shakhtar.com\/ru\/video\/assets\/images\/logo.svg)](https:\/\/shakhtar.com\/)\n\nperm\\_identity\n\n[![FC Shakhtar](https:\/\/video.shakhtar.com\/ru\/video\/assets\/images\/logo.svg)](https:\/\/shakhtar.com\/)\n\n- [1\\.49m](https:\/\/www.facebook.com\/fcshakhtar)\n- [113k](https:\/\/twitter.com\/fcshakhtar)\n- [239k](https:\/\/www.youtube.com\/user\/shakhtartv)\n- [631k](https:\/\/www.instagram.com\/fcshakhtar\/)\n\n# Шахтер – Динамо – 3:1 (02.09.2012): как это было\ncode\n\nvisibility\n###\n##\n\n###\n####\n\n[![](https:\/\/shakhtar.com\/-\/media\/fcsd\/sponsors\/favbet\/favbet_gapped_eng.png)](https:\/\/www.favbet.ua\/)[![](https:\/\/shakhtar.com\/-\/media\/fcsd\/sponsors\/puma\/puma_gapped.png)](https:\/\/ua.puma.com\/)[![](https:\/\/shakhtar.com\/-\/media\/fcsd\/sponsors\/scm\/scm_gapped.png)](https:\/\/www.scm.com.ua\/)\n\n[![](https:\/\/shakhtar.com\/-\/media\/fcsd\/sponsors\/parallel\/parallel_gapped.png)](https:\/\/parallel.ua\/)[![](https:\/\/shakhtar.com\/-\/media\/fcsd\/sponsors\/traumeel\/traumel_gapped.png)](https:\/\/heel-apteka.com.ua\/preparati\/bezrecepturni\/traumel-s-gel%20)[![](https:\/\/shakhtar.com\/-\/media\/fcsd\/sponsors\/skyup\/skyup_gapped.png)](https:\/\/skyup.aero\/)[![](https:\/\/shakhtar.com\/-\/media\/fcsd\/sponsors\/voda-ua\/voda_gapped.png)](https:\/\/vodaua.ua\/)[![](https:\/\/shakhtar.com\/-\/media\/fcsd\/sponsors\/pumb\/pumb_gapped.png)](https:\/\/www.pumb.ua\/)\n\n© FC Shakhtar Donetsk, 1998-2025. All rights reserved.\n\n[Terms of Use](https:\/\/shakhtar.com\/\/terms-of-use)\n\nВстроить","attrs_readable_markdown":null,"meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

🚫

NOT INDEXABLE

✅

CRAWLED

6 months ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	FAIL	`download_stamp > now() - 6 MONTH`	6.3 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://video.shakhtar.com/ru/video/f5e44be0-e799-4ffa-bb55-e4758942fe2a
Last Crawled	2025-10-03 12:11:57 (6 months ago)
First Indexed	2019-07-28 13:58:40 (6 years ago)
HTTP Status Code	200
Meta Title	Шахтер – Динамо – 3:1 (02.09.2012): как это было
Meta Description	null
Meta Canonical	null
Boilerpipe Text	© FC Shakhtar Donetsk, 1998-2025. All rights reserved.
Markdown	menu [![FC Shakhtar](https://video.shakhtar.com/ru/video/assets/images/logo.svg)](https://shakhtar.com/) perm\_identity [![FC Shakhtar](https://video.shakhtar.com/ru/video/assets/images/logo.svg)](https://shakhtar.com/) - [1\.49m](https://www.facebook.com/fcshakhtar) - [113k](https://twitter.com/fcshakhtar) - [239k](https://www.youtube.com/user/shakhtartv) - [631k](https://www.instagram.com/fcshakhtar/) # Шахтер – Динамо – 3:1 (02.09.2012): как это было code visibility ### ## ### #### [![](https://shakhtar.com/-/media/fcsd/sponsors/favbet/favbet_gapped_eng.png)](https://www.favbet.ua/)[![](https://shakhtar.com/-/media/fcsd/sponsors/puma/puma_gapped.png)](https://ua.puma.com/)[![](https://shakhtar.com/-/media/fcsd/sponsors/scm/scm_gapped.png)](https://www.scm.com.ua/) [![](https://shakhtar.com/-/media/fcsd/sponsors/parallel/parallel_gapped.png)](https://parallel.ua/)[![](https://shakhtar.com/-/media/fcsd/sponsors/traumeel/traumel_gapped.png)](https://heel-apteka.com.ua/preparati/bezrecepturni/traumel-s-gel%20)[![](https://shakhtar.com/-/media/fcsd/sponsors/skyup/skyup_gapped.png)](https://skyup.aero/)[![](https://shakhtar.com/-/media/fcsd/sponsors/voda-ua/voda_gapped.png)](https://vodaua.ua/)[![](https://shakhtar.com/-/media/fcsd/sponsors/pumb/pumb_gapped.png)](https://www.pumb.ua/) © FC Shakhtar Donetsk, 1998-2025. All rights reserved. [Terms of Use](https://shakhtar.com//terms-of-use) Встроить
Readable Markdown	null
Shard	41 (laksa)
Root Hash	15812233617324491841
Unparsed URL	com,shakhtar!video,/ru/video/f5e44be0-e799-4ffa-bb55-e4758942fe2a s443