
Apify is a powerful tool for web scraping, allowing you to extract data from websites with ease. It's a cloud-based platform that uses a visual interface to create and manage scrapers.
Apify supports over 1000 data sources, including popular websites like Amazon, eBay, and Wikipedia. This makes it an ideal choice for web scraping tasks that require data from multiple sources.
With Apify, you can create custom scrapers using a visual interface, making it easy to extract data even if you have no coding experience. You can also use pre-built scrapers or purchase them from the Apify Store.
On a similar theme: How to Make a Server Hosting Website
What Is Apify Web Scraper?
Apify Web Scraper is a versatile tool that can extract data from any website. It's a game-changer for anyone who needs to scrape product prices, real estate data, or social media profiles.
This tool is incredibly configurable with an intuitive user interface, making it easy to use even for those who aren't tech-savvy. You can handle almost any website and scrape dynamic content with ease.
One of the coolest features of Apify Web Scraper is its ability to scrape a list of URLs or crawl an entire website by following links. This is especially useful for extracting data from complex websites.
Apify Web Scraper runs entirely on the Apify platform, so you don't need to manage servers or proxies. This means you can focus on what matters most – extracting the data you need.
You can even set your scraper to run on a schedule and get data delivered automatically. This is perfect for tasks that require regular updates, like monitoring product prices or tracking social media activity.
Here are some of the key benefits of using Apify Web Scraper:
- Configurable with an intuitive user interface
- Can handle almost any website and can scrape dynamic content
- Scrape a list of URLs or crawl an entire website by following links
- Runs entirely on the Apify platform; no need to manage servers or proxies
- Set your scraper to run on a schedule and get data delivered automatically
- Can be used as a template to create your own scraper
Getting Started
To start using Apify Web Scraper, create a free account using your email and open Web Scraper. You can then add one or more URLs you want to scrape.
Apify Web Scraper offers a user-friendly interface, making it easy to get started. With just a few clicks, you can configure the page function that determines the data that needs to be scraped.
If this caught your attention, see: Calculator Html Css and Javascript
To scrape data from web pages, you can use Instant Data Scraper, which can do the heavy lifting for you. It's available in the Apify Store and can be used for free.
Here's a step-by-step guide to scraping data with Instant Data Scraper:
- Find Instant Web Scraper in Apify Store and click the Try for free button.
- Enter the URL of the webpage you want to scrape and select the 'Find tables' action to identify tabular data.
- Run the Actor and review the output dataset.
- Note the 'table number' and 'column numbers' of the data points to be scraped.
- Define your data extraction preferences in the "Column mappings" section.
- Enter the noted 'table number' from the previous run.
- Find the next page button selector from the target web page and enter it.
- Choose the 'Scrape data' action and run the Actor again.
- Optionally, pass cookies and set proxies if needed for advanced scraping scenarios.
- Preview and download your dataset in JSON, XML, CSV, Excel, or HTML, or export it via API.
How To Use
To get started with scraping data, you'll need to create a free Apify account and open Web Scraper. You can then add one or more URLs you want to scrape, and set paths that you'd like to include or exclude from crawling by configuring glob patterns or pseudo-URLs.
To scrape data from web pages, you can use Instant Data Scraper, which can be found in the Apify Store. This data scraping program allows you to scrape data from web pages in just ten steps.
You can start by finding Instant Web Scraper in the Apify Store and clicking the Try for free button. Then, enter the URL of the webpage you want to scrape and select the 'Find tables' action to identify tabular data.
Expand your knowledge: Chrome Web Store Software

To extract data from a webpage, you'll need to configure the page function that determines the data that needs to be scraped. This can be done by defining your data extraction preferences in the "Column mappings" section.
To scrape data from multiple pages, you can use the next page button selector from the target web page and enter it. This will allow you to scrape data from multiple pages and extract the data you need.
Here are the basic steps to follow:
- Create a free Apify account and open Web Scraper
- Add one or more URLs you want to scrape
- Set paths that you’d like to include or exclude from crawling
- Configure the page function that determines the data that needs to be scraped
- Click the “Start” button and wait for the data to be extracted
- Download your data in JSON, XML, CSV, Excel, or HTML
To make the most of your data scraping experience, be sure to follow along with the video tutorial or read the article on scraping with Web Scraper.
Start URLs
To start scraping, you'll need to add the initial URLs of the pages you want to visit to the Start URLs field. These URLs can be entered manually one by one, uploaded from a CSV file, or linked from a Google Sheets document.
Each URL must start with either a http:// or https:// protocol prefix. This is a requirement for the scraper to recognize the URL as valid.
You can add new URLs to scrape on the fly using the Link selector and Glob Patterns/Pseudo-URLs options. Alternatively, you can call await context.enqueueRequest() inside the Page function to add new URLs dynamically.
You can associate custom user data with each URL, which is a JSON object that can be referenced from your JavaScript code in the Page function. This is useful for determining which start URL is currently loaded, allowing you to perform page-specific actions.
For example, when crawling an online store, you might want to perform different actions on a page listing products versus a product detail page.
Configuration Settings
The Input tab is where you create your scraping configuration. It's the place where you tell the Actor what to do, and the Actor's creator prepares the Input form to guide you through the process.
You can enter configuration settings either manually in the user interface in Apify Console or programmatically in a JSON object using the Apify API. For a complete list of input fields and their type, check the input tab.
Don't worry about the Build, Timeout, and Memory options for now, just keep them at default settings.
Additional reading: Web Camera Settings
Input Configurations
Input Configurations are crucial for setting up your Web Scraper properly. You can enter these configurations manually in the user interface in Apify Console.
The Web Scraper Actor accepts a number of configuration settings on input, which can be entered in the user interface or programmatically in a JSON object using the Apify API. For a complete list of input fields and their type, check the input tab.
You can find detailed instructions on advanced configuration settings for Web Scraper in the relevant documentation. Advanced configuration settings are available for those who need more control over their scraping process.
The Input tab is where you create your scraping configuration. It's prepared by the Actor's creator so that you can tell the Actor what to do.
Glob Patterns
Glob Patterns are a powerful tool in configuration settings, allowing you to specify which types of URLs to add to the request queue. They're essentially strings with wildcard characters.
A glob pattern can match multiple URLs at once, which is super convenient. For example, the pattern http://www.example.com/pages/**/* will match URLs like http://www.example.com/pages/deeper-level/page, http://www.example.com/pages/my-awesome-page, and http://www.example.com/pages/something.
You can use glob patterns to match URLs with varying depths, like the ** wildcard, which matches any directory level. This can be really helpful when working with complex URL structures.
Here are some examples of glob patterns and what they match:
- http://www.example.com/pages/**/*: Matches URLs like http://www.example.com/pages/deeper-level/page, http://www.example.com/pages/my-awesome-page, and http://www.example.com/pages/something
- http://www.example.com/*: Matches URLs like http://www.example.com/ and http://www.example.com/about
By using glob patterns, you can make your configuration settings more flexible and efficient.
Advanced Features
Apify Web Scraper offers a range of advanced features that can help you scrape data more efficiently.
One of the most useful advanced features is the ability to configure settings in detail, which is outlined in the Advanced Configuration Settings section.
You can also use the advanced configuration settings to fine-tune your scraping process for optimal results.
Apify Web Scraper provides detailed instructions on how to use these advanced features, making it easy to get started.
With Apify Web Scraper, you can take your web scraping to the next level by using these advanced features to scrape data more effectively.
Discover more: Use Android Phone as Web Cam
Output and Export
You can access and export the data scraped by Apify Web Scraper in various formats such as JSON, XML, CSV, or as an Excel spreadsheet. The results can be downloaded using the Apify API or through the Apify Console.
The data is stored in a dataset, which you can find in the Output tab. Here are two example datasets:
To export the data, you can use the Apify API or the Apify Console. This will give you the flexibility to choose your preferred format.
You can export the data in various formats, including CSV, JSON, and Excel. This makes it easy to use the data in other tools or applications.
Related reading: Html to Pdf Api
To access the dataset, go to the Output tab, where you'll find the results wrapped into a dataset. This is where you can view and export the data.
Apify Web Scraper can be connected with various cloud services and web apps, including Google Drive, Zapier, and Slack. This means you can integrate your scraped databases with these services.
Here are the steps to export the data:
1. Select the target website.
2. Identify column mappings.
3. Target next page selectors.
4. Add proxies, cookies, or customize wait time.
5. Run the scraper.
6. Export the data in your preferred format.
You can export the data in various formats, including CSV, JSON, and Excel.
See what others are reading: Amazon Web Services
Proxy and Crawling
You can configure proxies for Apify Web Scraper using Apify Proxy or custom HTTP or SOCKS5 proxy servers. The scraper requires a proxy to run.
Apify Proxy offers automatic mode, where the proxy uses all available proxy groups and selects a proxy that hasn't been used in the longest time for the specific hostname. This reduces the chance of detection by the website.
For another approach, see: Proxy Server for Web Scraping
To use custom proxies, specify them in the `scheme://user:password@host:port` format, separated by a space or new line. The URL scheme can be either `HTTP` or `SOCKS5`, and the port must always be present.
Here's a table summarizing the available proxy configuration options:
By configuring proxies, you can enhance privacy, distribute requests, and prevent blocking while scraping data.
Using Proxies
You can use proxies to prevent your web scraper from being detected by target websites. Apify Proxy is one option available, which can be set up automatically or with specific groups of target proxy servers.
Apify Proxy can use all available proxy groups to reduce the chance of detection by the website. You can view the list of available proxy groups on the Proxy page in Apify Console.
To use custom proxies, you need to specify them in the `scheme://user:password@host:port` format, with multiple proxies separated by a space or new line. The URL scheme can be either `HTTP` or `SOCKS5`, and the port must always be present.
See what others are reading: Smiley-http-proxy-servlet Proxy Websocket
Here's an example of how to specify custom proxies: `http://bob:[email protected]:8000 http://bob:[email protected]:8000`
A web scraping proxy is an intermediary server that acts as a gateway between your web scraper and the target website. It hides your IP address, allowing you to make requests anonymously and avoid IP bans or access restrictions.
Proxies help to enhance privacy, distribute requests, and prevent blocking while scraping data.
Discover more: Webp Example
Crawling
Crawling is an essential part of web scraping, and it's great that you're interested in learning more about it.
You can control the crawling behavior of Web Scraper by specifying start URLs, defining link selectors, and entering pseudo-URLs to guide the scraper in following specific page links.
To find the right pseudo-URL, you can open some of the Actor details in the store and look for the pattern in the URLs. For example, if the URLs are structured as "https://example.com/OWNER/NAME", you can use a pseudo-URL to describe this structure.
A pseudo-URL is a pattern that describes a URL structure, and it's used to add links that follow the given structure to the crawling queue. You can add a label to the pseudo-URL, just like you would with a start URL.
To crawl websites with pseudo-URLs, you need to find the pattern in the URLs first. This can be done by opening some of the Actor details in the store and looking for the structure in the URLs. Once you've found the pattern, you can use a pseudo-URL to describe it.
Here are some key features of Web Scraper's crawling behavior:
- Start URLs: Specify the URLs you want to start crawling from.
- Link selectors: Define which links to follow on a page.
- Pseudo-URLs: Enter a pattern to describe a URL structure.
- Recursive crawling: Follow links recursively to crawl a website.
- Targeted extraction: Extract specific data from a website.
By controlling the crawling behavior of Web Scraper, you can customize the crawling process to suit your needs. This includes specifying start URLs, defining link selectors, and entering pseudo-URLs to guide the scraper in following specific page links.
Additional reading: Distributed Web Crawling
Waiting for Dynamic Content
Waiting for dynamic content can be a challenge when web scraping. Dynamic content is content that changes on the page after it's loaded, often due to JavaScript executing or network requests being made.
For more insights, see: Dynamic Web Page
The Apify Store is a JavaScript application, which means that the button might not exist in the page when the scraper runs the pageFunction. This is because the scraper only waits for the page to load its HTML, but not for the additional JavaScript to run.
You can't rely on the scraper to wait for all the JavaScript in the page to finish executing, as it would never stop waiting. It's up to you, the programmer, to wait for the elements you need.
The context.page.waitFor() function is available to help with this. It accepts a number of milliseconds to wait, a selector to await in the page, or a function to execute, and will stop waiting once the time elapses, the selector appears, or the provided function returns true.
The default timeout for the page.waitFor() function is 30 seconds, but you can override it by providing an options object with a timeout property.
Function and Customization
The pageFunction is a powerful tool in Apify web scraper that allows you to execute custom JavaScript code for each page visited. It's like having a superpower that lets you dive into the inner workings of the page.
To write a pageFunction, you need to inspect the page's structure using your browser's developer tools, specifically DevTools. This will give you a clear understanding of how the page is organized and what you need to target.
Custom data can be passed to the scraper using the Custom data input field in Advanced configuration. This allows you to add arbitrary data to the scraper and access it under the customData context key as an object within the pageFunction.
MCP Server
MCP Server is a powerful tool that allows you to connect to Apify Actors in various ways. This includes using clients like ClaudeDesktop and LibreChat, or even building your own custom connection.
To set up an MCP server for Web Scraper, start by going to the MCP tab. From there, follow these steps:
- Start a Server-Sent Events (SSE) session to receive a sessionId
- Send API messages using that sessionId to trigger the scraper
- The message starts the Web Scraper with the provided input
- The response should be: Accepted
You can also use the MCP server to connect to other Actors in conjunction with an MCP server, as mentioned earlier.
Post Navigation Hooks

Post Navigation Hooks allow you to execute custom functions after the main pageFunction has run.
These functions are stored in an array, which means you can add multiple hooks to be executed in a specific order.
The only parameter available in these functions is the CrawlingContext object, which provides access to the current crawling context.
To learn more about the objects passed into these functions, check out the docs for Post-navigation hooks and the PuppeteerHook type.
A fresh viewpoint: Web Navigation
The Function
The page function is a JavaScript function that gets executed for each page the scraper visits. It's a crucial part of the scraping process, and it can be customized to suit your needs.
You can use the page function to extract data from a page, such as a URL, unique identifier, title, description, last modification date, and number of runs. To do this, you need to inspect the page's structure to get an idea of its inner workings.
The page.$eval function allows you to run a function in the browser, with the selected element as the first argument. This is useful for extracting text content from a page.
If you're visiting multiple types of pages, you can define separate functions for each page type. For example, you can define a function for the list page and a function for the detail page.
The page function can return nothing, null, Object, or Object[]. If it returns an Object, it will be saved as a single result. If it returns an Array of Objects, each item in the array will be saved as a separate result.
Here are the possible return values of the page function:
- Nothing
- Null
- Object
- Object[]
Note that if you return an Array of Objects, each item in the array will be saved as a separate result. This can be useful if you need to extract multiple pieces of data from a page.
Should I Build a Tool?

Building a tool from scratch can be a daunting task, especially if you're short on time or don't have the necessary skills. Consider using existing libraries or tools unless you have specific needs that warrant custom development.
It's worth noting that building a web scraping tool from scratch is indeed time-consuming, requiring advanced skills, as mentioned in the article.
Using existing libraries or tools can save you a significant amount of time and effort, allowing you to focus on other important aspects of your project.
For example, building a web scraping tool from scratch is a task that is often better left to professionals or those with extensive experience in the field.
Discover more: Building Web Pages
Limitations of Using?
Using Apify Web Scraper can be a great way to extract data from websites, but it's not without its limitations. One of the main limitations is that it's designed to be user-friendly and generic, which can affect its performance and flexibility compared to more specialized solutions.

This generic approach means that Apify Web Scraper may not be as efficient as more tailored solutions, which can be a drawback for users who need to scrape large amounts of data.
Apify Web Scraper uses a resource-intensive Chromium browser to support client-side JavaScript code, which can impact its performance and battery life.
Instant as API
You can use the Apify API to access Instant Web Scraper programmatically.
The Apify API is organized around RESTful HTTP endpoints that enable you to manage, schedule, and run Apify Actors, access datasets, monitor actor performance, fetch results, create and update versions, and more.
To access the API using Node.js, use the apify-client NPM package, or use the apify-client PyPI package for Python.
There are also API endpoints available for extracting website data without a client.
For detailed information and code examples, see the API tab or refer to the Apify API documentation.
Theory and Information
Apify web scraper is a powerful tool for web scraping, but before we dive into the nitty-gritty, let's cover the basics.
To get started, you'll need to understand the information presented in the Apify Store, which is a great resource for scraping practice. The Store is built on popular technologies and displays a variety of items in different categories.
The Apify Store is a typical scraping target, making it an excellent choice for learning web scraping techniques. You can access the Store's detail pages to gather information about Actors, which are configurations for web scrapers.
Here's a quick rundown of the information you can find on an Actor's detail page:
- URL - The direct link to the Actor's detail page.
- Unique identifier - Such as apify/web-scraper.
- Title - The title visible on the Actor's detail page.
- Description - The Actor's description.
- Last modification date - When the Actor was last modified.
- Number of runs - How many times the Actor was run.
This information is essential for understanding the Actor's capabilities and limitations, which will help you create effective web scraping tasks.
Theory
Theory is what makes web scraping work. It's all about understanding how the process flows from start to finish.
Web scraping extracts data from websites by sending requests, analyzing the HTML responses, and saving the data. This is done through a series of steps that are repeated until all the desired data is collected.

The scraper lifecycle is quite straightforward. It visits the first Start URL, waits for the page to load, and then executes the pageFunction. It finds all the elements matching the Link selector, extracts their href attributes (URLs), and uses pseudo URLs to filter the extracted URLs.
A scraper repeats steps 1 and 2 when not using the request queue. This is useful when you already know all the URLs you want to visit, such as when you have a pre-existing list of a thousand URLs.
Some information can be scraped directly from listing pages, but for the rest, we need to visit the detail pages of all the Actors. The information we're interested in includes the URL, unique identifier, title, description, last modification date, and number of runs.
Here are the details we're looking to extract:
- URL - The URL that goes directly to the Actor's detail page.
- Unique identifier - Such as apify/web-scraper.
- Title - The title visible in the Actor's detail page.
- Description - The Actor's description.
- Last modification date - When the Actor was last modified.
- Number of runs - How many times the Actor was run.
Information
The Information tab is where the magic happens. It shows you all the information about the underlying Actor, which is the Apify scraper of your choice.
You can see the available versions and their READMEs, which is always a good idea to read first before creating a task for it. Reading an Actor's README will give you a better understanding of what it can do and how to use it effectively.
Testing and Debugging
To test and debug your Apify web scraper, start by running a test run to see if everything works as expected. Save and run the task, then check the dataset by clicking on the Results card and previewing the URLs and unique identifiers scraped.
You should see a nice table of all the attributes correctly scraped, which means you've nailed it! Just hit that Save & Run button and visit the Dataset preview of clean items.
If you're stuck, you can insert a breakpoint in your scraper by setting the Run mode to DEVELOPMENT and adding a debugger statement within your Page function. This will stop the execution of code at the breakpoint until you manually resume it in the DevTools window.
Insert Breakpoint
Inserting a breakpoint in your scraper can be super helpful during testing and debugging. This feature allows you to pause the execution of code at a specific location, giving you a chance to inspect the page and identify any issues.
To insert a breakpoint, you'll need to set the Run mode to DEVELOPMENT. This is a crucial step, as breakpoints won't work if you're in PRODUCTION mode.
In DEVELOPMENT mode, you can add a breakpoint by clicking on the Insert breakpoint property and selecting the location where you want the execution to stop.
A unique perspective: Outline of Web Design and Web Development
Browser Log
Browser Log can be a valuable tool for debugging, but it can also be overwhelming.
Setting the Browser Log to true includes console messages from the browser in the Actor's log, which can result in a flood of error messages, warnings, and other messages of little value.
This can be especially problematic with high concurrency, where many tasks are happening at the same time.
Test Run
To run a test, save your configuration and click the Save & Run button. This will start the task, and you should see the scraper visit the Start URL and then several Actor details matching the Pseudo URL in the log.
You might like: What Browesr Does Not save as Webp
Set Max pages per run to 10 and keep the pageFunction as it is. The scraper will first visit the Start URL, then several Actor details matching the Pseudo URL.
As you run the task, keep an eye on the log to see the scraper's progress. You can also check the Results card to see if everything is working as expected.
Once the task finishes, click on the Results card and then Preview to see the URLs and unique identifiers scraped. This will give you an idea of what data is being extracted.
Save & Run the task and visit the Dataset preview of clean items to see the attributes correctly scraped.
See what others are reading: See Html Code in Chrome
Best Practices
To use Apify Web Scraper effectively, it's essential to follow best practices.
Apify Web Scraper can handle up to 1000 concurrent tasks, making it suitable for large-scale web scraping projects.
To avoid overwhelming the scraper, it's recommended to start with smaller tasks and gradually increase the number of concurrent tasks.
Make sure to check the website's terms of service before scraping, as some websites may have restrictions or requirements for web scraping.
Practice

Practice is essential to become proficient in web scraping. It can take a few days to a week to learn the basics, depending on your prior programming experience.
To get started, you'll want to practice scraping data that's already available in a webpage's URL. This will help you understand how to extract the URL and a unique identifier from the request.
Learning web scraping basics can take a few days to a week, depending on your prior programming experience. Several months of practice may be needed to master advanced techniques.
Online tutorials can help you get started with web scraping practice. They'll provide you with the guidance and resources you need to begin extracting data from webpages.
Worth a look: Web Programming Language Crossword
Wrapping It Up
Wrapping it up is all about efficiency and speed. You can achieve this by adding all the data you need in a single object, as shown in the previous example.
Having all the necessary data in one place makes it easier to work with. This can be especially helpful when dealing with large datasets.
To speed things up, consider running multiple functions concurrently. This is called concurrency or parallelism. It's a good idea to run functions that don't rely on a specific order concurrently.
Concurrent execution can significantly reduce the time it takes to complete tasks. This is especially true when dealing with back and forth communication between different systems, like Node.js and the browser.
Frequently Asked Questions
Is web scraping legal in Apify?
Web scraping is allowed in Apify as long as you're scraping publicly available data on the internet. However, be aware of international regulations protecting personal data, intellectual property, and confidential information
Is Apify scraper free?
Apify's free plan is limited, and you'll need to pay for additional features like the TikTok actor to scrape certain websites. For most scraping tasks, you'll need to upgrade to a paid plan.
Featured Images: pexels.com

