Puppeteer Web Scraping Made Easy with Examples

Puppeteer web scraping is a powerful tool for extracting data from websites.

With Puppeteer, you can launch a browser instance in a Node.js environment, allowing you to interact with web pages programmatically.

Puppeteer provides a high-level API that makes it easy to automate tasks such as clicking buttons, filling out forms, and navigating between pages.

Automating tasks with Puppeteer can save you a significant amount of time and effort compared to manual web scraping methods.

To get started with Puppeteer web scraping, create a new folder for your project and initialize a Node.js repository with a package.json file. This will help you manage your project's dependencies and keep track of your NPM packages.

You'll need to install the Puppeteer library, which can be done by running npm install puppeteer in your terminal. This will ensure that the library is available for use in your project.

Here are the basic steps to initialize your project:

A headless browser is an actual browser without a graphical user interface, often referred to as a hidden browser.

It has complete functionality offered by a browser, but is faster and takes up less memory because there is no user interface.

Headless browsers can be controlled programmatically, which is a big advantage.

Chrome and Firefox are two of the most commonly used browsers that support headless mode.

Chromium, an open-source web browser made by Google, is also a headless browser option.

Note that Chromium and Chrome are two different browsers, with Chromium being an open-source project and Chrome being built on top of it.

Recommended read: Axiom Browser Automation & Web Scraping

Programmatic browser control is a powerful tool that allows you to automate browser interactions programmatically. This is achieved through the use of headless browsers, which are actual browsers without a graphical user interface.

A headless browser is simply an actual browser but without a graphical user interface. Think of it as a hidden browser. Headless browsers have complete functionality offered by a browser while being faster and taking up a lot less memory because there is no user interface.

Expand your knowledge: Capture Html of a Link without Opening It

Some of the most commonly used browsers that support headless mode include Chrome, Firefox, and Chromium. Chromium is an open-source web browser made by Google, and it's worth noting that Chromium and Chrome are two different browsers.

To control headless browsers programmatically, you can use solutions like Selenium or Puppeteer. Puppeteer is a lightweight and fast headless browser that's specifically designed for web scraping, making it a better choice than Selenium for tasks that require a high level of speed and efficiency.

Here are some of the key benefits of using Puppeteer for programmatic browser control:

By using Puppeteer for programmatic browser control, you can automate browser interactions with ease and speed, making it an ideal choice for web scraping tasks.

Glob Patterns are a powerful tool for specifying which types of URLs to add to the request queue.

A Glob Pattern is simply a string with wildcard characters, making it easy to match multiple URLs with a single pattern.

For example, a glob pattern like http://www.example.com/pages/**/* will match all URLs within the /pages/ directory and its subdirectories.

You can use wildcard characters like ** to match directories and * to match any characters.

Here's a breakdown of how the glob pattern http://www.example.com/pages/**/* would match the following URLs:

With Glob Patterns, you can easily match a wide range of URLs and add them to the request queue with minimal effort.

To set up the environment for puppeteer web scraping, start by creating a new folder on your computer, such as `first-puppeteer-scraper-example`, to hold your project.

Create a `package.json` file in this folder to store information about your repository and NPM packages, including the Puppeteer library.

Install the Puppeteer library by running a command in your terminal, which will allow you to use its functionality in your project.

By default, the browser is launched in headless mode, but you can modify this to include an object as a parameter if you need a user interface.

Intriguing read: Javascript Web Scraping Library

Create a new file in your node project directory, such as `example1.js`, to hold the code for your puppeteer scraper.

Set up a `.puppeteerrc.cjs` file to configure the browser executable path, which is necessary for running your scraper in Lambda.

Run `npm install` to install the necessary packages, including Puppeteer, and set up a GitHub repository for your project to connect to AWS CodeBuild.

Suggestion: How to Get Html File from Website

Web scraping is a process of automatically extracting data from websites.

Puppeteer is a Node.js library developed by the Chrome team that provides a high-level API to control a headless Chrome or Chromium browser.

With Puppeteer, you can scrape websites by navigating to a URL, waiting for the page to load, and then extracting the data you need. For example, you can use Puppeteer to scrape a webpage's HTML content, wait for a specific element to appear, and then extract the text within that element.

To get started with web scraping using Puppeteer, you'll need to install the library and set up a basic project structure.

A different take: C# Web Scraping Library

Pyppeteer is an unofficial port of Puppeteer for Python, bundling Chromium and working smoothly with it. It can also work with Chrome, similar to Puppeteer.

Pyppeteer uses the asyncio library for Python, which means its syntax is similar to Puppeteer, but with some differences due to the language.

You can use Pyppeteer to load a page and take a screenshot of it, just like with Puppeteer. For example, here's a Pyppeteer script that does just that: it loads a page and takes a screenshot of it.

Expand your knowledge: Java Web Scraper

To fetch current page quotes, you'll need to use the `document.querySelectorAll` function to select all elements with the class name "quote". This function returns a NodeList, which needs to be converted to an array using `Array.from()`.

You can then use a loop to extract the quote text and author from each element. To do this, you'll need to use the `querySelector` function to select the text and author elements within each quote element.

Here's a step-by-step guide:

1. Replace `document.querySelector` with `document.querySelectorAll` to select all quote elements.

2. Convert the NodeList to an array using `Array.from()`.

3. Use a loop to extract the quote text and author from each element.

4. Use `querySelector` to select the text and author elements within each quote element.

Here's an example of how the updated code might look:

```javascript

const quoteList = await page.evaluate(() => {

const quotes = document.querySelectorAll('.quote');

return Array.from(quotes).map((quote) => {

const text = quote.querySelector('.text').textContent;

const author = quote.querySelector('.author').textContent;

return { text, author };

});

});

```

This code uses the `page.evaluate` function to execute the JavaScript code on the page, and returns the extracted quote data as an array of objects.

Note that this assumes that the quote elements have the class names "quote", "text", and "author". You may need to adjust the code to match the actual class names used on the page.

Related reading: Html Web Page in a Web Page

The Clickable Elements Selector is a powerful tool in web scraping that allows you to target specific elements on a page that don't have href attributes.

To use it, you'll need to pass a CSS Selector to the Clickable elements selector, which should match the elements that lead to the URL you want to queue up.

This CSS selector is like a roadmap for the scraper, telling it exactly which elements to click on. Any triggered requests, navigations, or open tabs will be intercepted and filtered using Globs and/or Pseudo URLs.

The filtered URLs will then be added to the request queue, where they can be crawled and extracted. Leave this field empty to prevent the scraper from clicking on any elements.

Using the Clickable Elements Selector can impact performance, so it's essential to use it judiciously and only when necessary.

A unique perspective: Beautifulsoup Css Selector

Post-Navigation Hooks are a powerful feature in web scraping that allows you to execute custom code after the main pageFunction is run. This is where the magic happens, and you can add extra functionality to your scraper.

You can use Post-Navigation Hooks to perform various tasks, such as clicking on links or buttons that are not directly accessible through the pageFunction. For example, in Example 2, we used the Post-Navigation Hooks to click on the "Next page" button after processing all instructions.

To use Post-Navigation Hooks, you need to define an array of functions that will be executed after the main pageFunction is run. Each function will receive a PuppeteerCrawlingContext object as a parameter. This object contains various properties, including Actor (or Apify) and customData.

Here's an example of how you can use Post-Navigation Hooks to click on a link:

```html

// Define the Post-Navigation Hooks array

postNavigationHooks: [

async (context) => {

// Click on the link

await context.page.click(".pager > .next > a");

}

]

```

By using Post-Navigation Hooks, you can extend the functionality of your scraper and perform tasks that are not directly accessible through the pageFunction. This feature is particularly useful when dealing with complex web applications that require additional interactions to extract the desired data.

For more insights, see: Web Navigation

Pseudo URLs are a powerful tool in web scraping that allow you to specify which types of URLs to add to the request queue. They're essentially URLs with special directives enclosed in [] brackets.

The most commonly used directive is [regexp], which defines a JavaScript-style regular expression to match against the URL. For example, a pseudo-URL like http://www.example.com/pages/[(\w|-)*] will match all URLs that contain a word or hyphen in the path.

This can be really useful for scraping specific types of pages, such as blog posts or product pages. By using a pseudo-URL, you can avoid scraping every single URL on the site, which can be a huge time-saver.

If you need to include special characters like "[" or "]" in your pseudo-URL, you'll need to encode them as [\x5B] or [\x5D], respectively. For example, the pseudo-URL http://www.example.com/[pages] will match the URL http://www.example.com/pages.

You can also associate user data with each pseudo-URL, which can be referenced from your Page function using context.request.label. This allows you to determine which kind of page is currently loaded in the browser.

Extracting data from a web page is a crucial step in web scraping with Puppeteer. You can extract any data from the page by using the evaluate() function, which allows you to execute JavaScript functions like document.querySelector(). This makes it easy to extract any Element from the DOM.

To extract text from an element, you can use the text property, like this: element.text. This will return the text content of the element. You can also use the querySelector() function to select multiple elements at once, and then extract their text content using a loop.

One way to extract multiple elements is to use the querySelectorAll() function, which returns a NodeList of all elements matching the selector. You can then convert this NodeList to an array using the Array.from() function, and use the map() function to process each element in the array. This can be done using the page.evaluate() function to execute the code on the page.

A different take: Web Scraping Is Used to Extract What Type of Data

Puppeteer loads the complete page in DOM, allowing us to extract any data from the page.

To extract an element, use the evaluate() function, which enables us to execute JavaScript functions like document.querySelector(). This allows us to extract any element from the DOM.

Using this function, you'll immediately see that your desired tag is extracted. The evaluate() function returns one element from the page.

For the extracted element, you might only need the text. In that case, you can use the text property to extract the text. The text property can be accessed using the following code: element.text.

The result of the evaluate() function can be stored in a variable to complete the functionality. Don't forget to close the browser after you're done.

Here's a quick rundown of how to extract an element:

Scraping multiple elements is a crucial step in extracting data from a webpage. You can use the querySelectorAll function to get all elements matching the selector.

To extract multiple elements, you'll need to use three steps: querySelectorAll, create an array, and call the map function. The map function can be used to process each element in the array and return it.

You can store the result of the querySelectorAll function in a variable, such as heading_elements, which is of type NodeList. Then, you can call the map function on this variable to process each element.

The map function can be used to return each element, but you can also use the Array.from function to convert the NodeList to an array directly. This can be useful if you prefer a more concise code.

By following these steps, you can extract multiple elements from a webpage using Puppeteer. This is a powerful tool for web scraping, and understanding how to use it is essential for extracting data from websites.

Proxies are intermediaries between your Puppeteer browser and the target website, allowing you to rotate IP addresses and mask your original one, which helps you avoid rate limits, CAPTCHAs, and geo-restrictions while scraping.

There are two main types of proxies: residential and datacenter proxies. Residential proxies use real devices with IP addresses assigned by internet service providers (ISPs) and are highly effective at avoiding detection and bypassing sophisticated anti-bot systems.

Datacenter proxies, on the other hand, are hosted on servers in data centers and offer high speed and availability, making them more cost-effective than residential proxies and suitable for less protected websites or when volume and speed matter more than stealth.

To configure proxies in Puppeteer, you can use the Proxy configuration (proxyConfiguration) option, which enables you to set proxies that will be used by the scraper to prevent its detection by target websites.

You can choose from three options: Apify Proxy (automatic), Apify Proxy (selected groups), and custom proxies. If you choose custom proxies, you must specify them in the scheme://user:password@host:port format, and multiple proxies should be separated by a space or a new line.

Here are the available options for the proxy configuration setting:

You can also use free proxies available for testing purposes or small-scale projects, but keep in mind that they might not be as effective as paid proxies.

Puppeteer's built-in support for asynchronous programming allows for efficient web scraping by leveraging the browser's event loop. This enables Puppeteer to handle multiple requests and responses simultaneously, making it ideal for large-scale web scraping tasks.

Handling cookies and session management is crucial for maintaining a consistent user experience. As mentioned earlier, Puppeteer provides a simple way to set and manage cookies through the `page` object's `setCookie` method.

Puppeteer's `page.goto` method allows for easy navigation to URLs, and its `page.waitForNavigation` method ensures that the page has fully loaded before proceeding with the next step. This is especially useful when dealing with dynamic content that requires a full page load.

Automating web scraping can be a complex task, but it's a crucial step in extracting data from websites. Generally, there are two methods of accessing and parsing web pages.

Directly sending a get request to a web page using packages like Axios is a fast method, but it has its limitations, especially when it comes to dynamic sites that are rendered using JavaScript. Unfortunately, loading a browser would take a lot of resources because it has to load a lot of other things like the toolbar and buttons.

Fortunately, there are better solutions – headless browsers.

See what others are reading: Web Scraping Using Google Colab

Using a Chromium web browser can be a significant resource hog, especially for websites that don't rely on client-side JavaScript for dynamic content rendering.

The Actor might be overkill for such sites, making it a less-than-ideal choice for scraping.

Cheerio Scraper is a better option for these types of websites, as it downloads and processes raw HTML pages without the extra overhead of a web browser.

Puppeteer Scraper may be too complex for non-seasoned developers, who might prefer a simpler setup process.

Web Scraper is a good alternative, as it also uses Puppeteer under the hood but has a more streamlined setup process.

You can control which pages the scraper accesses by calling await context.enqueueRequest() from the Page function, making the Glob Patterns setting unnecessary.

Similarly, you can control which pages the scraper accesses by calling await context.enqueueRequest() from the Page function, making the Pseudo-URLs setting unnecessary.

Saving a snapshot of the current page's HTML and a screenshot can be done using the saveSnapshot helper function, which overwrites the previous snapshot and throttles pageFunction invocations to prevent abuse.

You can find the latest screenshot under the SNAPSHOT-SCREENSHOT key and the HTML under the SNAPSHOT-BODY key.

The scraping results returned by Page function are stored in the default dataset associated with the Actor run, from which you can export them to formats such as JSON, XML, CSV, or Excel.

You can save a snapshot of the current page's HTML and a screenshot of the current page into the default key-value store using the saveSnapshot function.

Each snapshot overwrites the previous one, so you don't need to worry about storing multiple versions of the snapshot. The pageFunction's invocations will also be throttled if saveSnapshot is invoked more than once in 2 seconds to prevent abuse.

The latest screenshot can be found under the SNAPSHOT-SCREENSHOT key and the HTML under the SNAPSHOT-BODY key.

The results of your web scraping project are stored in the default dataset associated with the Actor run.

You can export them in various formats, including JSON, XML, CSV, or Excel. The results are stored in the default dataset associated with the Actor run, which you can find in Apify Console.

For each object returned by the Page function, Puppeteer Scraper pushes one record into the dataset. This includes metadata such as the URL of the web page where the results come from.

The full object stored in the dataset includes the #error and #debug metadata fields, which you can skip by adding the clean=true query parameter to the API URL or selecting the Clean items option when downloading the dataset in Apify Console.

You can also customize the format of the results by setting the format query parameter to xml, xlsx, csv, html, etc.

To set up a Puppeteer scraper, start by creating a new folder for your project. Then, initialize your Node.js repository with a package.json file, which will help you manage your project's dependencies, including the Puppeteer library.

You can install the Puppeteer library by running the npm install command in your terminal. This command will download a recent version of Chromium, which is bundled with Puppeteer.

To use Puppeteer Scraper, you'll need to specify the Start URLs, which are the web pages you want the scraper to load. You can also configure the scraper to follow page links and extract data from each page using the Link selector, Glob Patterns, and Pseudo-URLs.

To handle each request and extract data, you'll need to provide a Page function, which is JavaScript code executed in the Node.js environment. The scraper uses the full-featured Chromium browser, so you can use the page object within the Page function's context to execute client-side logic.

You can create an index.js file in your project repository root to serve as the application entry point. This file will contain your scraper's logic, such as opening a new page and navigating to the desired website.

To start a Puppeteer session, you'll need to launch a browser using puppeteer.launch, which will instantiate a browser variable. Then, you can open a new page using browser.newPage, which will instantiate a page variable.

Here's a basic outline of the steps you'll need to follow:

Note that you'll need to close the browser to stop the running application, unless you comment out the instruction to close the browser.

Scraping a hotel listing page is a great way to extract valuable data, and I'm excited to walk you through the process.

Airbnb's page structure changes constantly, so it's essential to find the right selectors each time.

To scrape a hotel listing page, start by identifying the selector for each hotel section, as shown in the example where a NodeList of length 20 is returned and stored in the variable root.

The hotel name can be extracted by concatenating querySelectors, which effectively extracts the first hotel name with the code:

The URL of the photo of the hotel can be extracted with a similar code, and the map() function will be used to extract text or attributes from the array of elements.

Suggestion: Web Scraping Hotel Prices