Crawler Inspector

1. Shard Calculation

Query:

Response:

Calculated Shard: 125 (from laksa029)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa125.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://bbs.xiaopeng.com/thread/3008713\')), getAhrefsUnparsedNoserviceFromURL(\'https://bbs.xiaopeng.com/thread/3008713\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/bbs.xiaopeng.com\/thread\/3008713","crawl_time":1757551623,"first_indexed_time":1746292095,"http_code":200,"src_unparsed":"com,xiaopeng!bbs,\/thread\/3008713 s443","src_root_hash":"14483017276979023925","history_drop_reason":null,"meta_title":"双十一来啦！","meta_descriptions":["小鹏汽车，一起聊聊智能生活"],"attrs_boilerpipe_text":"双十一来啦！","attrs_markdown":"[![](https:\/\/s.xiaopeng.com\/bbs\/public\/img\/app_icon.3db442c6.png)小鹏社区Beta版](https:\/\/bbs.xiaopeng.com\/ \"小鹏汽车-互联网基因的智能电动汽车\")\n\n- [推荐](https:\/\/bbs.xiaopeng.com\/ \"推荐\")\n- [鹏友圈](https:\/\/bbs.xiaopeng.com\/thread.html \"鹏友圈\")\n\n登录\n\n![头像](https:\/\/xps14.xiaopeng.com\/a5d3a01f9fe12b9cd2b1b0d6fcb4136e.png?x-oss-process=image\/resize,m_mfit,w_80,h_80,limit_1\/crop,w_80,h_80\/format,webp\/quality,q_90)\n\nPu.j\n\n![](https:\/\/s.xiaopeng.com\/bbs\/assets\/tag\/p5.png)\n\n2024-11-02 13:25\n\n双十一来啦！![](https:\/\/s.xiaopeng.com\/bbs\/uploaded\/20241102\/media_202411021325_8358_file_w_786@h_1402.png?x-oss-process=image\/resize,m_mfit,w_1150,limit_1\/format,webp\/quality,q_90)\n\n最新评论\n\n![](https:\/\/s.xiaopeng.com\/bbs\/public\/img\/blank.8b66d659.png)暂无评论\n\n[![小鹏汽车](https:\/\/s.xiaopeng.com\/bbs\/assets\/images\/h5\/test-drive.png)](https:\/\/www.xiaopeng.com\/appointment.html?ch=00327&ps=app)\n\n![小鹏汽车APP](https:\/\/s.xiaopeng.com\/bbs\/assets\/images\/h5\/download-app.png)\n\n扫描二维码下载小鹏App","attrs_readable_markdown":null,"meta_canonical":null}

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	FAIL	`download_stamp > now() - 6 MONTH`	7.3 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Property	Value
URL	https://bbs.xiaopeng.com/thread/3008713
Last Crawled	2025-09-11 00:47:03 (7 months ago)
First Indexed	2025-05-03 17:08:15 (11 months ago)
HTTP Status Code	200
Meta Title	双十一来啦！
Meta Description	小鹏汽车，一起聊聊智能生活
Meta Canonical	null
Boilerpipe Text	双十一来啦！
Markdown	[![](https://s.xiaopeng.com/bbs/public/img/app_icon.3db442c6.png)小鹏社区Beta版](https://bbs.xiaopeng.com/ "小鹏汽车-互联网基因的智能电动汽车") - [推荐](https://bbs.xiaopeng.com/ "推荐") - [鹏友圈](https://bbs.xiaopeng.com/thread.html "鹏友圈") 登录 ![头像](https://xps14.xiaopeng.com/a5d3a01f9fe12b9cd2b1b0d6fcb4136e.png?x-oss-process=image/resize,m_mfit,w_80,h_80,limit_1/crop,w_80,h_80/format,webp/quality,q_90) Pu.j ![](https://s.xiaopeng.com/bbs/assets/tag/p5.png) 2024-11-02 13:25 双十一来啦！![](https://s.xiaopeng.com/bbs/uploaded/20241102/media_202411021325_8358_file_w_786@h_1402.png?x-oss-process=image/resize,m_mfit,w_1150,limit_1/format,webp/quality,q_90) 最新评论 ![](https://s.xiaopeng.com/bbs/public/img/blank.8b66d659.png)暂无评论 [![小鹏汽车](https://s.xiaopeng.com/bbs/assets/images/h5/test-drive.png)](https://www.xiaopeng.com/appointment.html?ch=00327&ps=app) ![小鹏汽车APP](https://s.xiaopeng.com/bbs/assets/images/h5/download-app.png) 扫描二维码下载小鹏App
Readable Markdown	null
Shard	125 (laksa)
Root Hash	14483017276979023925
Unparsed URL	com,xiaopeng!bbs,/thread/3008713 s443

🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

1. Shard Calculation

Query:

Response:

2. Crawled Status Check

Query:

Response:

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

Page Info Filters

Page Details