Crawler Inspector

1. Shard Calculation

Query:

Response:

Calculated Shard: 192 (from laksa036)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa192.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://finance.yahoo.com/chart/btc-USD/\')), getAhrefsUnparsedNoserviceFromURL(\'https://finance.yahoo.com/chart/btc-USD/\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/finance.yahoo.com\/chart\/btc-USD\/","crawl_time":1772741547,"first_indexed_time":1599337674,"http_code":200,"src_unparsed":"com,yahoo!finance,\/chart\/btc-USD\/ s443","src_root_hash":"3401813433841292792","history_drop_reason":null,"meta_title":"BTC-USD Interactive Stock Chart | Bitcoin USD Stock - Yahoo Finance","meta_descriptions":["At Yahoo Finance, you get free stock quotes, up-to-date news, portfolio management resources, international market data, social interaction and mortgage rates that help you manage your financial life."],"attrs_boilerpipe_text":"70,991.46\n-2,611.81\n(-3.55%)\nAs of 8:06:00 PM UTC. Market Open.","attrs_markdown":"Oops, something went wrong\n\n[![Yahoo Finance Logo](https:\/\/s.yimg.com\/rz\/p\/yahoo_finance_en-US_h_w_finance_2.png)](https:\/\/finance.yahoo.com\/)\n\n# Bitcoin USD ([BTC-USD](https:\/\/finance.yahoo.com\/quote\/BTC-USD\/) )\nCCC - CoinMarketCap • USD\n\n70,991.46  \\-2,611.81  (-3.55%)\n\nAs of 8:06:00 PM UTC. Market Open.\n\nFollow\n\nComparisons\n\nIndicators\n\nTechnicals\n\nCorporate events\n\nDraw\n\nSettings\n\nSelect\n\nChart Range Bar\n\n1D\n\n5D\n\n1M\n\n3M\n\n6M\n\nYTD\n\n1Y\n\n5Y\n\nAll\n\nInterval:\n\n1 min\n\nIn partnership with ChartIQ\n\n© 2026 Yahoo. All rights reserved.In partnership with ChartIQ\n\n[Data Disclaimer](https:\/\/help.yahoo.com\/kb\/finance-for-web\/SLN2310.html?locale=en_US) [Help](https:\/\/help.yahoo.com\/kb\/finance-for-web) [Feedback](https:\/\/yahoo.uservoice.com\/forums\/952723-finance-b3) [Sitemap](https:\/\/finance.yahoo.com\/sitemap\/) [What's New](https:\/\/finance.yahoo.com\/about\/whatsnew\/) [About Our Ads](https:\/\/policies.oath.com\/us\/en\/oath\/privacy\/adinfo\/index.html) [Terms](https:\/\/guce.yahoo.com\/terms?locale=en-US) [Privacy Policy](https:\/\/guce.yahoo.com\/privacy-policy?locale=en-US)","attrs_readable_markdown":"70,991.46  \\-2,611.81  (-3.55%)\n\nAs of 8:06:00 PM UTC. Market Open.","meta_canonical":null}

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	1.2 months ago (distributed domain, exempt)
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Property	Value
URL	https://finance.yahoo.com/chart/btc-USD/
Last Crawled	2026-03-05 20:12:27 (1 month ago)
First Indexed	2020-09-05 20:27:54 (5 years ago)
HTTP Status Code	200
Meta Title	BTC-USD Interactive Stock Chart \| Bitcoin USD Stock - Yahoo Finance
Meta Description	At Yahoo Finance, you get free stock quotes, up-to-date news, portfolio management resources, international market data, social interaction and mortgage rates that help you manage your financial life.
Meta Canonical	null
Boilerpipe Text	70,991.46 -2,611.81 (-3.55%) As of 8:06:00 PM UTC. Market Open.
Markdown	Oops, something went wrong [![Yahoo Finance Logo](https://s.yimg.com/rz/p/yahoo_finance_en-US_h_w_finance_2.png)](https://finance.yahoo.com/) # Bitcoin USD ([BTC-USD](https://finance.yahoo.com/quote/BTC-USD/) ) CCC - CoinMarketCap • USD 70,991.46 \-2,611.81 (-3.55%) As of 8:06:00 PM UTC. Market Open. Follow Comparisons Indicators Technicals Corporate events Draw Settings Select Chart Range Bar 1D 5D 1M 3M 6M YTD 1Y 5Y All Interval: 1 min In partnership with ChartIQ © 2026 Yahoo. All rights reserved.In partnership with ChartIQ [Data Disclaimer](https://help.yahoo.com/kb/finance-for-web/SLN2310.html?locale=en_US) [Help](https://help.yahoo.com/kb/finance-for-web) [Feedback](https://yahoo.uservoice.com/forums/952723-finance-b3) [Sitemap](https://finance.yahoo.com/sitemap/) [What's New](https://finance.yahoo.com/about/whatsnew/) [About Our Ads](https://policies.oath.com/us/en/oath/privacy/adinfo/index.html) [Terms](https://guce.yahoo.com/terms?locale=en-US) [Privacy Policy](https://guce.yahoo.com/privacy-policy?locale=en-US)
Readable Markdown	70,991.46 \-2,611.81 (-3.55%) As of 8:06:00 PM UTC. Market Open.
Shard	192 (laksa)
Root Hash	3401813433841292792
Unparsed URL	com,yahoo!finance,/chart/btc-USD/ s443

🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

1. Shard Calculation

Query:

Response:

2. Crawled Status Check

Query:

Response:

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

Page Info Filters

Page Details