Complete Guide to Ruby Web Scraping

Author

Reads 307

Green and Black Spider on Spider Web in Close Up Photography
Credit: pexels.com, Green and Black Spider on Spider Web in Close Up Photography

Ruby web scraping is a powerful tool for extracting data from websites, and it's surprisingly easy to get started. You can use Ruby's built-in libraries, such as Nokogiri and Mechanize, to navigate and parse web pages.

One of the most important things to keep in mind when web scraping is the website's terms of service. As we discussed in the "Understanding Website Terms" section, some websites prohibit web scraping altogether, while others require permission or have specific rules.

To avoid getting blocked, it's essential to respect a website's robots.txt file, which specifies which pages can be crawled and which cannot. We learned about this in the "Working with Robots.txt" section, where we saw how to use Nokogiri to parse the file.

Ruby's Nokogiri library is particularly well-suited for web scraping, thanks to its fast and efficient parsing capabilities. With Nokogiri, you can extract specific data from web pages, even if it's buried deep within the HTML structure.

Check this out: What Is Robot Txt

Setting Up Your Project

Credit: youtube.com, Ruby Web Scraping Tutorial

Next, initialize a `scraper.rb` file inside the project folder with the following content: `This should print this message:`

Time to import your project in your IDE and start defining some advanced Ruby scraping logic.

To do this, start Visual Studio Code, click the “Extensions” icon in the left bar, and type “Ruby” in the search input on the top.

Click on the “Install” button on the first element to add Ruby highlighting capabilities to VS Code.

Wait for the plugin to be added to the IDE. Then, open the `ruby-web-scraper` folder with “File,” “Open Folder…”.

You might like: Ruby Web Programming

Making HTTP Requests

Making HTTP requests is a crucial step in web scraping with Ruby. You've got several options, including Net::HTTP, HTTP.rb, HTTParty, Typhoeus, RestClient, and Faraday.

Faraday is a popular choice due to its flexibility, pluggability, and ease of use. It's like a reliable morning coffee - you can configure it to your liking and it always gets the job done.

Credit: youtube.com, Ruby Scraper - Nokogiri - HTTParty - Crawl Websites and Scrape Public DATA

To send HTTP requests, you can use Faraday's GET method, which takes only one line of code to replicate a browser's GET request. The response object contains useful properties, such as the status code and the actual returned HTML.

Here are some popular HTTP clients for Ruby:

To avoid getting blocked by anti-bot technologies, you can specify a valid User-Agent in HTTParty. This will make the requests appear to come from a legitimate browser, like Google Chrome 112.

Parsing HTML

Parsing HTML is a crucial step in Ruby web scraping. Nokogiri is a popular HTML and XML parser in Ruby that makes it easy to dig into HTML and pull out the juicy bits.

Nokogiri is super fast, works with both XPath and CSS selectors, and has been around forever. You can use the HTML4 module of Nokogiri to parse data in your HTML string, but note that before version 1.12, the HTML4 module didn't exist, so use the HTML module instead.

Intriguing read: Ruby Golang

Credit: youtube.com, Scraping the Web with Ruby - Part 1 (2020)

To parse the HTML document, you can pass its content to the Nokogiri HTML() function, which will give you a parsed document object that you can query using CSS Selectors or XPath. The two most important methods to select HTML elements are xpath() and css(), but CSS queries are usually the easiest way to express what you are looking for.

Here are some common CSS selectors used in HTML parsing:

  • `#id` selects an element with the given id
  • `.class` selects an element with the given class
  • `tag` selects all elements with the given tag name
  • `tag[attr]` selects all elements with the given tag name and attribute

To locate HTML elements via CSS selectors, you can use the `find_elements` method, which takes either a css or an xpath parameter to locate the HTML element and return an array of all matching elements. If you only need one element, you can use the `find_element` variation of this function, which returns only the first matching element.

Here's a step-by-step guide to parsing HTML with Nokogiri:

1. Inspect the page to understand its structure

2. Use the `Nokogiri::HTML()` function to parse the HTML document

3. Use CSS Selectors to locate the HTML elements of interest

4. Use the `xpath()` or `css()` method to select the HTML elements

5. Use the selected elements to extract the data you need

By following these steps and using Nokogiri, you can easily parse HTML and extract the data you need for your Ruby web scraping project.

Consider reading: Beautifulsoup Css Selector

Selenium and Capybara

Credit: youtube.com, System Tests With Capybara and Selenium | Ruby On Rails 7 Tutorial

Selenium and Capybara are powerful tools that can help you scrape dynamic web pages. They let you drive a real browser, load pages, wait for JavaScript to execute, click buttons, and fill out forms.

To use Selenium or Capybara, you need a real browser installed, either Chrome or Firefox, along with a matching driver that lets Selenium control it.

These tools are particularly useful for scraping sites that load content dynamically with JavaScript, lazy-load elements, or hide data behind clicks and scrolling. They can simulate what a real user would do, making them a great solution for scraping JavaScript-driven pages with interaction.

One popular gem for scraping dynamic pages is Kimurai, but it hasn't been updated yet and doesn't work with Ruby version 3. Fortunately, Selenium works perfectly with Ruby and is a time-tested solution for running a headless browser.

You can run Selenium in headless mode, which means it runs the browser without showing any UI – completely silent, but still fully functional. This is great for automation, background jobs, or running things on a server.

Installing and Configuring

Credit: youtube.com, Ruby Mechanize Web Scraping

Installing Ruby is a straightforward process. You can download and run the Ruby Installer on Windows, or use a package manager like Chocolatey.

To install Ruby on macOS, use a package manager like Homebrew, which will download and install the Ruby development environment. For Linux, use the package manager for your distro, such as running a specific command for Ubuntu.

Once you have Ruby installed, you'll need to install the required gems, including HTTParty, Nokogiri, and CSV. You can do this by running specific commands in the terminal, or by creating a Gemfile and running a command to install all the gems listed.

Installing

Installing Ruby is a straightforward process that depends on your operating system. For Windows, you can download and run the Ruby Installer.

If you're using a package manager like Chocolatey, you can install Ruby by running a command in the command prompt.

macOS comes with Ruby version 2 by default, but it's recommended to install a separate instance of Ruby version 3.

Yellow Spider on Web in Close Up Photography
Credit: pexels.com, Yellow Spider on Web in Close Up Photography

To install Ruby on macOS, use a package manager like Homebrew and enter a specific command in the terminal.

For Linux, use the package manager for your distro, such as running a command for Ubuntu.

You'll also need to install the required gems, including HTTParty, Nokogiri, and CSV, using the command line or a Gemfile.

Here's a list of the required gems:

  • HTTParty
  • Nokogiri
  • CSV

Alternatively, you can create a Gemfile and add the following lines:

```ruby

gem 'httparty'

gem 'nokogiri'

gem 'csv'

```

Then, open the terminal, navigate to the directory, and run the command `bundle install` to install all the gems listed in the Gemfile.

Sending Emails for Success

Sending emails is a great way to notify users of successful web scraper results.

For example, if you're using a Raspberry Pi to scrape a web page, you'll want to set up an email notification system.

This allows you to receive an email as soon as the Raspberry Pi 5 is in stock.

You can use this same principle to notify users of other successful web scraper results.

To set up email notifications, you'll need to configure your web scraper to send emails to the user's inbox.

Working with Data

Credit: youtube.com, Scraping data with Rails, Copybara and Selenium driver example in 1 min

You can export the scraped data to CSV and JSON using Ruby's built-in capabilities. The CSV gem can be used to export data to a CSV file, and you can write the data to the file immediately to keep chunks in the CSV file if a script fails mid-execution.

To export data to CSV, you can use the CSV gem and write the data to a file using the `<<` operator, which will automatically convert each instance to an array of strings as required by the CSV class. This will create a CSV file with the header record and the scraped data.

You can also export data to JSON using Ruby's built-in JSON capabilities, which will generate a JSON file containing the scraped data. This is a great way to provide data to other teams in a format they can easily work with.

Here's an interesting read: Web Scraping Using Google Colab

Scrape Data

You can use Nokogiri to retrieve data from a target HTML webpage.

Credit: youtube.com, What is Web Scraping and What is it Used For? | Definition and Examples EXPLAINED

Nokogiri's at_css() function returns the first element that matches a CSS query, which is a shortcut for using the CSS selector method.

To store the collected data, define a UseCase class with a Struct in Ruby. This allows you to bundle one or more attributes in the same data class. The struct has three attributes corresponding to the info to retrieve from each use case card.

You can use attribute() to select an attribute from the current HTML element, and then get its value with the value field. The text field directly returns all text contained in the current HTML node as a plain string.

With Nokogiri, web scraping is pretty simple. You can scrape data from a webpage and use it to instantiate a new UseCase object and add it to a list.

To collect search results, find list items under the #productGrid and fetch links defined by the a.productClickItemV2 tags.

You can use Ruby's Struct to define a data class with multiple attributes, making it easy to store and retrieve data from a webpage.

Export Scraped Data

Credit: youtube.com, Web Scraping Tutorial | Data Scraping from Websites to Excel | Web Scraper Chorme Extension

Exporting scraped data is a crucial step in the web scraping process. You can export your data to a CSV file using the CSV gem, which is a built-in Ruby library.

To export to CSV, you'll need to require the CSV gem and open a CSV file in write or append mode, writing the headers as you go. This ensures that your data is properly formatted and can be easily read by others.

The CSV gem will automatically convert each instance of your data to an array of strings, making it easy to export your data to a CSV file.

You can also export your data to JSON format, which is another common format for exchanging data between teams and applications.

To export to JSON, you can use the built-in JSON conversion capabilities of Ruby, which are similar to the CSV gem.

By exporting your scraped data to CSV and JSON, you can easily share it with others and use it in a variety of applications and tools.

In addition to exporting to CSV and JSON, you can also store your web scraper results in ActiveRecord models, which is a great way to keep track of the web pages you want to monitor and scrape data from.

Handling Complex Pages

Credit: youtube.com, Data Scraping and More with Ruby Nokogiri Sinatra and Heroku

Nokogiri can't handle complex pages with JavaScript-driven content, but Capybara with Selenium can.

You can use Selenium to simulate user interactions, such as clicking links and buttons, to load dynamic content.

For example, you can use Selenium to open the Rotten Tomatoes homepage, find the first movie, click its link, wait for the movie page to load, and then click the trailer button, as demonstrated in Example 2.

Alternatively, you can use Kimurai, but be aware that it hasn't been updated and may not work with Ruby version 3.

If you need to scrape dynamic pages, you can use a headless browser like Selenium, which works perfectly with Ruby, as mentioned in Example 4.

Here are some popular gems for scraping dynamic pages:

  • Kimurai (note: not recommended due to lack of updates)
  • Selenium

Keep in mind that Selenium is a powerful tool that can handle complex pages, but it may require more code and setup than simpler scraping methods.

Handling Pagination

Handling pagination can be a challenge when scraping web pages, but it's not insurmountable.

Credit: youtube.com, Pagination Made Simple

A common issue arises when the next button on the last page is not found, causing an error.

To mitigate this, a while-true loop can be used to scrape the data.

The loop will be terminated using a begin-rescue-end block.

This approach allows the scraper to continue running even if it encounters an error.

To create a CSV file, the same approach discussed in the earlier section can be used.

Dynamic Pages

Handling dynamic pages can be a real challenge. Nokogiri can't handle JavaScript-driven pages, but Capybara with Selenium can.

You'll need to use a gem like Kimurai, but it hasn't been updated yet and doesn't work with Ruby version 3. A time-tested solution is to run a headless browser using Selenium.

To load a dynamic website, you'll need to create an instance of Selenium WebDriver for Chrome. This will create a headed browser, but you can make it headless by setting the Chrome::Options to false.

Banded garden spider on web wrapping prey in cocoon
Credit: pexels.com, Banded garden spider on web wrapping prey in cocoon

Here's a simple table to illustrate the process:

To handle pagination, you can use a while-true loop to scrape the data. This will continue until the next button isn't found, preventing an error. You can also use a begin-rescue-end block to handle any potential errors.

Saving and Deploying

You can save scraped data to a CSV file using the `csv` module in Ruby, as shown in the "Saving Data" section. This is useful for storing large amounts of data for later use.

The `csv` module is a built-in Ruby library that makes it easy to work with CSV files. It's a great way to store and retrieve data from your scraper.

To deploy your Ruby web scraper, you can use a tool like Capistrano, which allows you to automate the deployment process. This is especially useful if you're working on a team or need to deploy your scraper to multiple servers.

Young boy smiling while saving money in a crowned piggy bank, demonstrating financial responsibility.
Credit: pexels.com, Young boy smiling while saving money in a crowned piggy bank, demonstrating financial responsibility.

Make sure to test your scraper on different websites and servers to ensure it works as expected. This will help you catch any issues before they become major problems.

Ruby's `open-uri` library can be used to make HTTP requests to websites, which is essential for web scraping. This library is also useful for testing your scraper's deployment.

By following these steps, you can successfully save and deploy your Ruby web scraper. Remember to always test your scraper thoroughly before deploying it to production.

Wrapping Up in Rails

In Ruby, you can create a data scraping script in just 50 lines of code.

If you're new to web scraping, don't worry if it seems overwhelming at first - with Ruby, you can get started quickly and easily.

Ruby is a great choice for web scraping because it's a versatile and efficient language that can handle complex tasks with ease.

The key to a successful web scraping script is to keep it simple and focused, just like the 50-line script mentioned earlier.

By following these tips and using the right tools, you can create a web scraping script that gets the job done without breaking the bank.

Jeannie Larson

Senior Assigning Editor

Jeannie Larson is a seasoned Assigning Editor with a keen eye for compelling content. With a passion for storytelling, she has curated articles on a wide range of topics, from technology to lifestyle. Jeannie's expertise lies in assigning and editing articles that resonate with diverse audiences.

Love What You Read? Stay Updated!

Join our community for insights, tips, and more.