ScraperAI

⚡ Scraping has never been easier ⚡

ScraperAI is an open-source, AI-powered tool designed to simplify web scraping for users of all skill levels. By leveraging Large Language Models, such as ChatGPT, ScraperAI extracts data from web pages and generates reusable and shareable scraping configs.

Forget about manually extracting selectors from HTML pages using Developer Consoles. ScraperAI handles this process for you.

Features

• Serializable & reusable Scraper Configs
• Automatic data detection
• Automatic XPATHs detection
• Automatic pagination & page type detection
• HTML minification
• ChatGPT support
• Custom LLMs support
• Selenium support
• Custom crawlers support

How does ScraperAI work?

ScraperAI employs AI models to analyze web pages and generate scraping tasks. These tasks are then utilized to collect data from websites in structured formats such as JSON, CSV, and others.

Who should use ScraperAI?

ScraperAI is designed for scientists, analysts, entrepreneurs, and SEO specialists seeking to effortlessly scrape data from the web in a no-code manner. It is particularly useful for those unfamiliar with various scraping techniques or looking to save time and effort when collecting the data.

How to get started with ScraperAI?

To get started with ScraperAI, you first need to install it. You can do this by running the following command:

pip install scraperai

Once you have installed ScraperAI, you can start using a CLI application by calling

scraperai --url https://www.ycombinator.com/companies

command in your console.

Demo

We put some examples of scraper usage here.

Support

If you have any questions or need help, please open a new issue.