Crawler Inspector

1. Shard Calculation

Query:

Response:

Calculated Shard: 92 (from laksa034)

2. Crawled Status Check

Query:

curl -X POST \
  'http://laksa092.int.ahrefs:8124/' \
  -H 'Content-Type: text/plain' \
  -H 'X-ClickHouse-Database: crawler3' \
  -H 'Authorization: Basic YXBpOg==' \
  -d 'SELECT getAhrefsURLFromUnparsed(src_unparsed) AS found_url, ifNull(toUnixTimestamp(download_stamp), 0) AS crawl_time, ifNull(toUnixTimestamp(props_url_first_seen), 0) AS first_indexed_time, download_http_code AS http_code, src_unparsed AS src_unparsed, src_root_hash AS src_root_hash, history_drop_reason AS history_drop_reason, meta_title AS meta_title, meta_descriptions AS meta_descriptions, attrs_boilerpipe_text AS attrs_boilerpipe_text, attrs_markdown AS attrs_markdown, attrs_readable_markdown AS attrs_readable_markdown, meta_canonical AS meta_canonical FROM crawler3.page_info_local FINAL PREWHERE (src_root_hash, src_unparsed) IN ((getAhrefsRootHashFromUnparsed(getAhrefsUnparsedNoserviceFromURL(\'https://courses.cs.umbc.edu/341/fall99/frey/Lectures/Hashing/hashFunc.html\')), getAhrefsUnparsedNoserviceFromURL(\'https://courses.cs.umbc.edu/341/fall99/frey/Lectures/Hashing/hashFunc.html\'))) FORMAT JSONEachRow'

Response:

{"found_url":"https:\/\/courses.cs.umbc.edu\/341\/fall99\/frey\/Lectures\/Hashing\/hashFunc.html","crawl_time":1753290519,"first_indexed_time":0,"http_code":200,"src_unparsed":"edu,umbc!cs,courses,\/341\/fall99\/frey\/Lectures\/Hashing\/hashFunc.html s443","src_root_hash":"6541913094267234292","history_drop_reason":null,"meta_title":"Hashing Functions","meta_descriptions":[],"attrs_boilerpipe_text":"h ( k ) = |_ M * ( k * A - |_ k * A _| ) _|     where A is some real positive constant. \n    note that ( k * A - |_ k * A _| ) is between 0.0 and\n1.0 Good choice for A -- the inverse of the golden ratio    golden ratio  Ø =  \n( 1 + sqrt (5) ) \/ 2 = 1.618033989....     so Ø -1  = 1 \/ 1.618033989... \n=  0.618033989 Because of the relationship between the golden ratio and \nFibonacci numbers, this is also called \"Fibonacci hashing\"     h ( k ) = |_ M * ( k * Ø -1 \n- |_ k * Ø -1  _| ) _|     Some values for k and h ( k ) k h ( k ) 0 0 1 0.618 * M 2 0.236 * M 3 0.854 * M 4 0.472 * M 5 0.090 * M 6 0.708 * M 7 0.326 * M 8 0.562 * M 9 0.562 * M","attrs_markdown":null,"attrs_readable_markdown":null,"meta_canonical":null}

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	FAIL	`download_stamp > now() - 6 MONTH`	8.9 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Property	Value
URL	https://courses.cs.umbc.edu/341/fall99/frey/Lectures/Hashing/hashFunc.html
Last Crawled	2025-07-23 17:08:39 (8 months ago)
First Indexed	not set
HTTP Status Code	200
Meta Title	Hashing Functions
Meta Description	null
Meta Canonical	null
Boilerpipe Text	h ( k ) = \|_ M * ( k * A - \|_ k * A _\| ) _\| where A is some real positive constant. note that ( k * A - \|_ k * A _\| ) is between 0.0 and 1.0 Good choice for A -- the inverse of the golden ratio golden ratio Ø = ( 1 + sqrt (5) ) / 2 = 1.618033989.... so Ø -1 = 1 / 1.618033989... = 0.618033989 Because of the relationship between the golden ratio and Fibonacci numbers, this is also called "Fibonacci hashing" h ( k ) = \|_ M * ( k * Ø -1 - \|_ k * Ø -1 _\| ) _\| Some values for k and h ( k ) k h ( k ) 0 0 1 0.618 * M 2 0.236 * M 3 0.854 * M 4 0.472 * M 5 0.090 * M 6 0.708 * M 7 0.326 * M 8 0.562 * M 9 0.562 * M
Markdown	null
Readable Markdown	null
Shard	92 (laksa)
Root Hash	6541913094267234292
Unparsed URL	edu,umbc!cs,courses,/341/fall99/frey/Lectures/Hashing/hashFunc.html s443

🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

1. Shard Calculation

Query:

Response:

2. Crawled Status Check

Query:

Response:

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

Page Info Filters

Page Details