Crawler Inspector

1. Shard Calculation

Query:

Response:

Calculated Shard: 125 (from laksa016)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa125.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://bbs.xiaopeng.com/thread/2997509\')), getAhrefsUnparsedNoserviceFromURL(\'https://bbs.xiaopeng.com/thread/2997509\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/bbs.xiaopeng.com\/thread\/2997509","crawl_time":1756310224,"first_indexed_time":1745974549,"http_code":200,"src_unparsed":"com,xiaopeng!bbs,\/thread\/2997509 s443","src_root_hash":"14483017276979023925","history_drop_reason":null,"meta_title":"今年商城双十一都没有","meta_descriptions":["小鹏汽车，一起聊聊智能生活"],"attrs_boilerpipe_text":"今年商城双十一都没有","attrs_markdown":"[![](https:\/\/s.xiaopeng.com\/bbs\/public\/img\/app_icon.3db442c6.png)小鹏社区Beta版](https:\/\/bbs.xiaopeng.com\/ \"小鹏汽车-互联网基因的智能电动汽车\")\n\n- [推荐](https:\/\/bbs.xiaopeng.com\/ \"推荐\")\n- [鹏友圈](https:\/\/bbs.xiaopeng.com\/thread.html \"鹏友圈\")\n\n登录\n\n![头像](https:\/\/xps14.xiaopeng.com\/719b1707c7418df8b1450d7a9f3cbed0.png?x-oss-process=image\/resize,m_mfit,w_80,h_80,limit_1\/crop,w_80,h_80\/format,webp\/quality,q_90)\n\nN\\&J\n\n![](https:\/\/s.xiaopeng.com\/bbs\/assets\/tag\/g6.png)\n\n2024-10-29 21:17\n\n今年商城双十一都没有搞什么活动呀\n\n最新评论\n\n![](https:\/\/s.xiaopeng.com\/bbs\/public\/img\/blank.8b66d659.png)暂无评论\n\n[![小鹏汽车](https:\/\/s.xiaopeng.com\/bbs\/assets\/images\/h5\/test-drive.png)](https:\/\/www.xiaopeng.com\/appointment.html?ch=00327&ps=app)\n\n![小鹏汽车APP](https:\/\/s.xiaopeng.com\/bbs\/assets\/images\/h5\/download-app.png)\n\n扫描二维码下载小鹏App","attrs_readable_markdown":null,"meta_canonical":null}

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	FAIL	`download_stamp > now() - 6 MONTH`	7.6 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Property	Value
URL	https://bbs.xiaopeng.com/thread/2997509
Last Crawled	2025-08-27 15:57:04 (7 months ago)
First Indexed	2025-04-30 00:55:49 (11 months ago)
HTTP Status Code	200
Meta Title	今年商城双十一都没有
Meta Description	小鹏汽车，一起聊聊智能生活
Meta Canonical	null
Boilerpipe Text	今年商城双十一都没有
Markdown	[![](https://s.xiaopeng.com/bbs/public/img/app_icon.3db442c6.png)小鹏社区Beta版](https://bbs.xiaopeng.com/ "小鹏汽车-互联网基因的智能电动汽车") - [推荐](https://bbs.xiaopeng.com/ "推荐") - [鹏友圈](https://bbs.xiaopeng.com/thread.html "鹏友圈") 登录 ![头像](https://xps14.xiaopeng.com/719b1707c7418df8b1450d7a9f3cbed0.png?x-oss-process=image/resize,m_mfit,w_80,h_80,limit_1/crop,w_80,h_80/format,webp/quality,q_90) N\&J ![](https://s.xiaopeng.com/bbs/assets/tag/g6.png) 2024-10-29 21:17 今年商城双十一都没有搞什么活动呀最新评论 ![](https://s.xiaopeng.com/bbs/public/img/blank.8b66d659.png)暂无评论 [![小鹏汽车](https://s.xiaopeng.com/bbs/assets/images/h5/test-drive.png)](https://www.xiaopeng.com/appointment.html?ch=00327&ps=app) ![小鹏汽车APP](https://s.xiaopeng.com/bbs/assets/images/h5/download-app.png) 扫描二维码下载小鹏App
Readable Markdown	null
Shard	125 (laksa)
Root Hash	14483017276979023925
Unparsed URL	com,xiaopeng!bbs,/thread/2997509 s443

🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

1. Shard Calculation

Query:

Response:

2. Crawled Status Check

Query:

Response:

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

Page Info Filters

Page Details