Crawler.sh
Free Local AEO & SEO Spider and a Markdown content extractor
464 followers
Free Local AEO & SEO Spider and a Markdown content extractor
464 followers
A fast, local-first web crawler and AEO & SEO analysis tool. Crawl entire sites in seconds from the terminal or a native desktop app, run automated SEO checks, extract content as clean Markdown, and export to JSON, CSV, or Sitemap XML.
Crawler.sh
Hey Product Hunt! I built crawler.sh because I kept running into the same problem: every time I needed to audit a website's SEO or extract its content, I had to choose between bloated enterprise tools, slow cloud services, or stitching together a bunch of scripts.
crawler.sh is a single tool that does all three: crawling, SEO analysis, and content extraction, from the terminal or a native desktop app. It's built in Rust so it's fast, it runs locally so your data stays private, and it outputs standard formats so you can take the results anywhere.
@mehmetkose Congrats on your launch! You got me at bloated enterprise tools. After an SEO/AEO analysis, does this also offer suggestions to fix the detected issues?
Crawler.sh
@jerrybyday Thank you! and YES. You can see them on the UI or download them. I'm adding new features every day in our core modules to discover issues smartly
@mehmetkose Hey Mehmet
Thanks for that, it's just on time, the topic is hot
But what’s the exact moment or event in Crawler that makes me think: ‘This is worth paying for’?
like aha-moment
Loving the ease of this. Is there a team plan?
Crawler.sh
@jacklyn_i Thank you Jacklyn William I'm thrilled! I'm adding this on the roadmap! Will let you know when is implemented. Also the cloud functionality is on the way
@mehmetkose Awesome! I’d follow your account to stay connected on your journey as you continue to grow. Congrats on your launch again.
Teract
Man, markdown extraction is such a lifesaver when dealing with giant websites. Way easier than my old 'copy-paste marathon' method. 😂
Crawler.sh
@djordjevic_nikola Oh that's exactly what i would like to hear. Give me some more :)) Also check it out that you can copy your SEO issues as a prompt so your llm can go fix.
Congrats on the launch and the product!
I'm curious if you spotted a pattern regarding how people are using the product already - is it more focused on their own sites or competitor analysis?
Love the positioning, local, fast, no bloat is refreshing. How deep does the AEO analysis go? Is it just schema and structured data checks, or does it also evaluate content for AI-answer readiness?
I really like this local-first approach. Congrats!
Does this handle JavaScript-heavy sites (React, Next.js)? Or is it pure HTML crawling? I believe that's usually where terminal crawlers fall short vs the bloated enterprise tools, but I also might be mistaken.
What analysis is done for AEO? And what do you think plays crucial role to optimize sites for AEO?