🕷️ Crawler Inspector

URL Lookup

Direct Parameter Lookup

Raw Queries and Responses

1. Shard Calculation

Query:

Response:

Calculated Shard: 15 (from laksa173)

2. Crawled Status Check

Query:

Response:

{"found_url":"https:\/\/consumer.sd.gov\/","crawl_time":1775753267,"first_indexed_time":1522369941,"http_code":200,"src_unparsed":"gov,sd!consumer,\/ s443","src_root_hash":"12968601681144549415","history_drop_reason":null,"meta_title":"South Dakota Consumer Protection","meta_descriptions":[],"attrs_boilerpipe_text":"Welcome to South Dakota's Consumer Reports Website!\nMENU\nCONSUMER ALERTS\nLAWS\nFAST FACTS\nRESOURCES\nCOMPLAINT\nFORM\nCOMPANY SEARCH\nCONTACT US\nCITIZEN\n&#x2713","attrs_markdown":"# Welcome to South Dakota's Consumer Reports Website\\!\n[WHO WE ARE](https:\/\/consumer.sd.gov\/whoweare.aspx) [NEWSCASTS](https:\/\/consumer.sd.gov\/newscasts.aspx)\n\nMENU\n\n- [CONSUMER ALERTS](https:\/\/consumer.sd.gov\/consumeralerts.aspx)\n- [LAWS](https:\/\/consumer.sd.gov\/laws.aspx)\n- [FAST FACTS](https:\/\/consumer.sd.gov\/fastfacts.aspx)\n- [RESOURCES](https:\/\/consumer.sd.gov\/resources.aspx)\n\n[![Consumer Protection](https:\/\/consumer.sd.gov\/img\/cplogo.png)](https:\/\/consumer.sd.gov\/)\n\n- [COMPLAINT FORM](https:\/\/consumer.sd.gov\/complaintform.aspx)\n- [COMPANY SEARCH](https:\/\/consumer.sd.gov\/companysearch\/)\n- [CONTACT US](https:\/\/consumer.sd.gov\/contact.aspx)\n- [CITIZEN **\\&\\#x2713**](https:\/\/consumer.sd.gov\/citizencheck\/)\n\n[![](https:\/\/consumer.sd.gov\/img\/slider\/banner-landlordTenant.jpg)](https:\/\/consumer.sd.gov\/docs\/LLTen_Statutes43-32.pdf)\n\n[![](https:\/\/consumer.sd.gov\/img\/slider\/banner-companySearch.png)](https:\/\/consumer.sd.gov\/companysearch\/)\n\n[![](https:\/\/consumer.sd.gov\/img\/slider\/HangUpGiftCardScams.png)](https:\/\/consumer.sd.gov\/)\n\n## [CREDIT  CARDS](https:\/\/consumer.sd.gov\/fastfacts\/credit.aspx)\n## [VEHICLES](https:\/\/consumer.sd.gov\/fastfacts\/automotive.aspx)\n## [DOOR-TO  -DOOR](https:\/\/consumer.sd.gov\/fastfacts\/doortodoor.aspx)\n## [IDENTITY  THEFT](https:\/\/consumer.sd.gov\/fastfacts\/identitytheft.aspx)\n## [MEDICAL](https:\/\/consumer.sd.gov\/fastfacts\/medical.aspx)\n## [BUSINESS-TO  -BUSINESS](https:\/\/consumer.sd.gov\/fastfacts\/businesstobusiness.aspx)\n## [CHARITY](https:\/\/consumer.sd.gov\/fastfacts\/charity.aspx)\n## [DEBT  COLLECTION](https:\/\/consumer.sd.gov\/fastfacts\/debt.aspx)\n## [LANDLORD  TENANT](https:\/\/consumer.sd.gov\/fastfacts\/landlordtenant.aspx)\n## [MAGAZINES](https:\/\/consumer.sd.gov\/fastfacts\/magazines.aspx)\n## [SWEEPSTAKES](https:\/\/consumer.sd.gov\/fastfacts\/sweepstakes.aspx)\n## [TRAVEL  VACATION](https:\/\/consumer.sd.gov\/fastfacts\/travel.aspx)\n## [CONSTRUCTION](https:\/\/consumer.sd.gov\/fastfacts\/construction.aspx)\n\n[HOME](https:\/\/consumer.sd.gov\/) \\| [ATG SITE](http:\/\/atg.sd.gov\/) \\| [SD.GOV](https:\/\/www.sd.gov\/)\n\nFont Size:\n\n- [S](https:\/\/consumer.sd.gov\/#nogo)\n- [M](https:\/\/consumer.sd.gov\/#nogo)\n- [L](https:\/\/consumer.sd.gov\/#nogo)\n- [XL](https:\/\/consumer.sd.gov\/#nogo)","attrs_readable_markdown":"## Welcome to South Dakota's Consumer Reports Website\\!\nMENU\n\n- [CONSUMER ALERTS](https:\/\/consumer.sd.gov\/consumeralerts.aspx)\n- [LAWS](https:\/\/consumer.sd.gov\/laws.aspx)\n- [FAST FACTS](https:\/\/consumer.sd.gov\/fastfacts.aspx)\n- [RESOURCES](https:\/\/consumer.sd.gov\/resources.aspx)\n\n[![Consumer Protection](https:\/\/consumer.sd.gov\/img\/cplogo.png)](https:\/\/consumer.sd.gov\/)\n\n- [COMPLAINT FORM](https:\/\/consumer.sd.gov\/complaintform.aspx)\n- [COMPANY SEARCH](https:\/\/consumer.sd.gov\/companysearch\/)\n- [CONTACT US](https:\/\/consumer.sd.gov\/contact.aspx)\n- [CITIZEN **\\&\\#x2713**](https:\/\/consumer.sd.gov\/citizencheck\/)\n\n[![](https:\/\/consumer.sd.gov\/img\/slider\/banner-landlordTenant.jpg)](https:\/\/consumer.sd.gov\/docs\/LLTen_Statutes43-32.pdf)\n\n[![](https:\/\/consumer.sd.gov\/img\/slider\/banner-companySearch.png)](https:\/\/consumer.sd.gov\/companysearch\/)\n\n[![](https:\/\/consumer.sd.gov\/img\/slider\/HangUpGiftCardScams.png)](https:\/\/consumer.sd.gov\/)","meta_canonical":null}

3. Robots.txt Check

Query:

Response:

4. Spam/Ban Check

Query:

Response:

5. Seen Status Check

ℹ️ Skipped - page is already crawled

📄

INDEXABLE

✅

CRAWLED

5 days ago

🤖

ROBOTS ALLOWED

Page Info Filters

Filter	Status	Condition	Details
HTTP status	PASS	`download_http_code = 200`	HTTP 200
Age cutoff	PASS	`download_stamp > now() - 6 MONTH`	0.2 months ago
History drop	PASS	`isNull(history_drop_reason)`	No drop reason
Spam/ban	PASS	`fh_dont_index != 1 AND ml_spam_score = 0`	ml_spam_score=0
Canonical	PASS	`meta_canonical IS NULL OR = '' OR = src_unparsed`	Not set

Page Details

Property	Value
URL	https://consumer.sd.gov/
Last Crawled	2026-04-09 16:47:47 (5 days ago)
First Indexed	2018-03-30 00:32:21 (8 years ago)
HTTP Status Code	200
Meta Title	South Dakota Consumer Protection
Meta Description	null
Meta Canonical	null
Boilerpipe Text	Welcome to South Dakota's Consumer Reports Website! MENU CONSUMER ALERTS LAWS FAST FACTS RESOURCES COMPLAINT FORM COMPANY SEARCH CONTACT US CITIZEN &#x2713
Markdown	# Welcome to South Dakota's Consumer Reports Website\! [WHO WE ARE](https://consumer.sd.gov/whoweare.aspx) [NEWSCASTS](https://consumer.sd.gov/newscasts.aspx) MENU - [CONSUMER ALERTS](https://consumer.sd.gov/consumeralerts.aspx) - [LAWS](https://consumer.sd.gov/laws.aspx) - [FAST FACTS](https://consumer.sd.gov/fastfacts.aspx) - [RESOURCES](https://consumer.sd.gov/resources.aspx) [![Consumer Protection](https://consumer.sd.gov/img/cplogo.png)](https://consumer.sd.gov/) - [COMPLAINT FORM](https://consumer.sd.gov/complaintform.aspx) - [COMPANY SEARCH](https://consumer.sd.gov/companysearch/) - [CONTACT US](https://consumer.sd.gov/contact.aspx) - [CITIZEN \&\#x2713](https://consumer.sd.gov/citizencheck/) [![](https://consumer.sd.gov/img/slider/banner-landlordTenant.jpg)](https://consumer.sd.gov/docs/LLTen_Statutes43-32.pdf) [![](https://consumer.sd.gov/img/slider/banner-companySearch.png)](https://consumer.sd.gov/companysearch/) [![](https://consumer.sd.gov/img/slider/HangUpGiftCardScams.png)](https://consumer.sd.gov/) ## [CREDIT CARDS](https://consumer.sd.gov/fastfacts/credit.aspx) ## [VEHICLES](https://consumer.sd.gov/fastfacts/automotive.aspx) ## [DOOR-TO -DOOR](https://consumer.sd.gov/fastfacts/doortodoor.aspx) ## [IDENTITY THEFT](https://consumer.sd.gov/fastfacts/identitytheft.aspx) ## [MEDICAL](https://consumer.sd.gov/fastfacts/medical.aspx) ## [BUSINESS-TO -BUSINESS](https://consumer.sd.gov/fastfacts/businesstobusiness.aspx) ## [CHARITY](https://consumer.sd.gov/fastfacts/charity.aspx) ## [DEBT COLLECTION](https://consumer.sd.gov/fastfacts/debt.aspx) ## [LANDLORD TENANT](https://consumer.sd.gov/fastfacts/landlordtenant.aspx) ## [MAGAZINES](https://consumer.sd.gov/fastfacts/magazines.aspx) ## [SWEEPSTAKES](https://consumer.sd.gov/fastfacts/sweepstakes.aspx) ## [TRAVEL VACATION](https://consumer.sd.gov/fastfacts/travel.aspx) ## [CONSTRUCTION](https://consumer.sd.gov/fastfacts/construction.aspx) [HOME](https://consumer.sd.gov/) \\| [ATG SITE](http://atg.sd.gov/) \\| [SD.GOV](https://www.sd.gov/) Font Size: - [S](https://consumer.sd.gov/#nogo) - [M](https://consumer.sd.gov/#nogo) - [L](https://consumer.sd.gov/#nogo) - [XL](https://consumer.sd.gov/#nogo)
Readable Markdown	## Welcome to South Dakota's Consumer Reports Website\! MENU - [CONSUMER ALERTS](https://consumer.sd.gov/consumeralerts.aspx) - [LAWS](https://consumer.sd.gov/laws.aspx) - [FAST FACTS](https://consumer.sd.gov/fastfacts.aspx) - [RESOURCES](https://consumer.sd.gov/resources.aspx) [![Consumer Protection](https://consumer.sd.gov/img/cplogo.png)](https://consumer.sd.gov/) - [COMPLAINT FORM](https://consumer.sd.gov/complaintform.aspx) - [COMPANY SEARCH](https://consumer.sd.gov/companysearch/) - [CONTACT US](https://consumer.sd.gov/contact.aspx) - [CITIZEN \&\#x2713](https://consumer.sd.gov/citizencheck/) [![](https://consumer.sd.gov/img/slider/banner-landlordTenant.jpg)](https://consumer.sd.gov/docs/LLTen_Statutes43-32.pdf) [![](https://consumer.sd.gov/img/slider/banner-companySearch.png)](https://consumer.sd.gov/companysearch/) [![](https://consumer.sd.gov/img/slider/HangUpGiftCardScams.png)](https://consumer.sd.gov/)
Shard	15 (laksa)
Root Hash	12968601681144549415
Unparsed URL	gov,sd!consumer,/ s443