markdown-crawler

A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG

Homepage ↗ Repo ↗ AI & Machine Learning html-to-markdown html-to-markdown-converter html2md llm llmops markdown markdown-crawler markdown-parser markdown-scraper md-crawler rag web-scraper

⚙ Agent Friendliness

N/A

Not evaluated

Can an agent use this?

🔒 Security

N/A

Not evaluated

Is it safe for agents?

⚡ Reliability

N/A

Not evaluated

Does it work consistently?

API endpoint ↗ Agent guide ↗ Report inaccuracy

Scores are editorial opinions as of unknown date.