rvest: Easily Harvest (Scrape) Web Pages

Description

Wrappers around the 'xml2' and 'httr' packages to make it easy to download, then manipulate, HTML and XML.

Author(s)

Maintainer: Hadley Wickham hadley@posit.co

Other contributors:

• Posit Software, PBC [copyright holder, funder]

See Also

Useful links:

• https://rvest.tidyverse.org/

• https://github.com/tidyverse/rvest

• Report bugs at https://github.com/tidyverse/rvest/issues

Interact with a live web page

Description

You construct an LiveHTML object with read_html_live() and then interact, like you're a human, using the methods described below. When debugging a scraping script it is particularly useful to use ⁠$view()⁠, which will open a live preview of the site, and you can actually see each of the operations performed on the real site.

rvest provides relatively simple methods for scrolling, typing, and clicking. For richer interaction, you probably want to use a package that exposes a more powerful user interface, like selendir.

Public fields

Methods

Public methods

• LiveHTML$new()

• LiveHTML$print()

• LiveHTML$view()

• LiveHTML$html_elements()

• LiveHTML$click()

• LiveHTML$get_scroll_position()

• LiveHTML$scroll_into_view()

• LiveHTML$scroll_to()

• LiveHTML$scroll_by()

• LiveHTML$type()

• LiveHTML$press()

• LiveHTML$clone()

Method new()

initialize the object

Usage

LiveHTML$new(url)

Arguments

Method print()

Called when print()ed

Usage

LiveHTML$print(...)

Arguments

Method view()

Display a live view of the site

Usage

LiveHTML$view()

Method html_elements()

Extract HTML elements from the current page.

Usage

LiveHTML$html_elements(css, xpath)

Arguments

Method click()

Simulate a click on an HTML element.

Usage

LiveHTML$click(css, n_clicks = 1)

Arguments

Method get_scroll_position()

Get the current scroll position.

Usage

LiveHTML$get_scroll_position()

Method scroll_into_view()

Scroll selected element into view.

Usage

LiveHTML$scroll_into_view(css)

Arguments

Method scroll_to()

Scroll to specified location

Usage

LiveHTML$scroll_to(top = 0, left = 0)

Arguments

Method scroll_by()

Scroll by the specified amount

Usage

LiveHTML$scroll_by(top = 0, left = 0)

Arguments

Method type()

Type text in the selected element

Usage

LiveHTML$type(css, text)

Arguments

Method press()

Simulate pressing a single key (including special keys).

Usage

LiveHTML$press(css, key_code, modifiers = character())

Arguments

Method clone()

The objects of this class are cloneable with this method.

Usage

LiveHTML$clone(deep = FALSE)

Arguments

Examples

## Not run: 
# To retrieve data for this paginated site, we need to repeatedly push
# the "Load More" button
sess <- read_html_live("https://www.bodybuilding.com/exercises/finder")
sess$view()

sess %>% html_elements(".ExResult-row") %>% length()
sess$click(".ExLoadMore-btn")
sess %>% html_elements(".ExResult-row") %>% length()
sess$click(".ExLoadMore-btn")
sess %>% html_elements(".ExResult-row") %>% length()

## End(Not run)

Make link to google form given id

Description

Make link to google form given id

Usage

google_form(x)

Arguments

Get element attributes

Description

html_attr() gets a single attribute; html_attrs() gets all attributes.

Usage

html_attr(x, name, default = NA_character_)

html_attrs(x)

Arguments

Value

A character vector (for html_attr()) or list (html_attrs()) the same length as x.

Examples

html <- minimal_html('<ul>
  <li><a href="https://a.com" class="important">a</a></li>
  <li class="active"><a href="https://c.com">b</a></li>
  <li><a href="https://c.com">b</a></li>
  </ul>')

html %>% html_elements("a") %>% html_attrs()

html %>% html_elements("a") %>% html_attr("href")
html %>% html_elements("li") %>% html_attr("class")
html %>% html_elements("li") %>% html_attr("class", default = "inactive")

Get element children

Description

Get element children

Usage

html_children(x)

Arguments

Examples

html <- minimal_html("<ul><li>1<li>2<li>3</ul>")
ul <- html_elements(html, "ul")
html_children(ul)

html <- minimal_html("<p>Hello <b>Hadley</b><i>!</i>")
p <- html_elements(html, "p")
html_children(p)

Select elements from an HTML document

Description

html_element() and html_elements() find HTML element using CSS selectors or XPath expressions. CSS selectors are particularly useful in conjunction with https://selectorgadget.com/, which makes it very easy to discover the selector you need.

Usage

html_element(x, css, xpath)

html_elements(x, css, xpath)

Arguments

Value

html_element() returns a nodeset the same length as the input. html_elements() flattens the output so there's no direct way to map the output to the input.

CSS selector support

CSS selectors are translated to XPath selectors by the selectr package, which is a port of the python cssselect library, https://pythonhosted.org/cssselect/.

It implements the majority of CSS3 selectors, as described in https://www.w3.org/TR/2011/REC-css3-selectors-20110929/. The exceptions are listed below:

• Pseudo selectors that require interactivity are ignored: ⁠:hover⁠, ⁠:active⁠, ⁠:focus⁠, ⁠:target⁠, ⁠:visited⁠.

• The following pseudo classes don't work with the wild card element, *: ⁠*:first-of-type⁠, ⁠*:last-of-type⁠, ⁠*:nth-of-type⁠, ⁠*:nth-last-of-type⁠, ⁠*:only-of-type⁠

• It supports ⁠:contains(text)⁠

• You can use !=, ⁠[foo!=bar]⁠ is the same as ⁠:not([foo=bar])⁠

• ⁠:not()⁠ accepts a sequence of simple selectors, not just a single simple selector.

Examples

html <- minimal_html("
  <h1>This is a heading</h1>
  <p id='first'>This is a paragraph</p>
  <p class='important'>This is an important paragraph</p>
")

html %>% html_element("h1")
html %>% html_elements("p")
html %>% html_elements(".important")
html %>% html_elements("#first")

# html_element() vs html_elements() --------------------------------------
html <- minimal_html("
  <ul>
    <li><b>C-3PO</b> is a <i>droid</i> that weighs <span class='weight'>167 kg</span></li>
    <li><b>R2-D2</b> is a <i>droid</i> that weighs <span class='weight'>96 kg</span></li>
    <li><b>Yoda</b> weighs <span class='weight'>66 kg</span></li>
    <li><b>R4-P17</b> is a <i>droid</i></li>
  </ul>
")
li <- html %>% html_elements("li")

# When applied to a node set, html_elements() returns all matching elements
# beneath any of the inputs, flattening results into a new node set.
li %>% html_elements("i")

# When applied to a node set, html_element() always returns a vector the
# same length as the input, using a "missing" element where needed.
li %>% html_element("i")
# and html_text() and html_attr() will return NA
li %>% html_element("i") %>% html_text2()
li %>% html_element("span") %>% html_attr("class")

Guess faulty character encoding

Description

html_encoding_guess() helps you handle web pages that declare an incorrect encoding. Use html_encoding_guess() to generate a list of possible encodings, then try each out by using encoding argument of read_html(). html_encoding_guess() replaces the deprecated guess_encoding().

Usage

html_encoding_guess(x)

Arguments

Examples

# A file with bad encoding included in the package
path <- system.file("html-ex", "bad-encoding.html", package = "rvest")
x <- read_html(path)
x %>% html_elements("p") %>% html_text()

html_encoding_guess(x)
# Two valid encodings, only one of which is correct
read_html(path, encoding = "ISO-8859-1") %>% html_elements("p") %>% html_text()
read_html(path, encoding = "ISO-8859-2") %>% html_elements("p") %>% html_text()

Parse forms and set values

Description

Use html_form() to extract a form, set values with html_form_set(), and submit it with html_form_submit().

Usage

html_form(x, base_url = NULL)

html_form_set(form, ...)

html_form_submit(form, submit = NULL)

Arguments

Value

• html_form() returns as S3 object with class rvest_form when applied to a single element. It returns a list of rvest_form objects when applied to multiple elements or a document.

• html_form_set() returns an rvest_form object.

• html_form_submit() submits the form, returning an httr response which can be parsed with read_html().

See Also

HTML 4.01 form specification: https://www.w3.org/TR/html401/interact/forms.html

Examples

html <- read_html("http://www.google.com")
search <- html_form(html)[[1]]

search <- search %>% html_form_set(q = "My little pony", hl = "fr")

# Or if you have a list of values, use !!!
vals <- list(q = "web scraping", hl = "en")
search <- search %>% html_form_set(!!!vals)

# To submit and get result:
## Not run: 
resp <- html_form_submit(search)
read_html(resp)

## End(Not run)

Get element name

Description

Get element name

Usage

html_name(x)

Arguments

Value

A character vector the same length as x

Examples

url <- "https://rvest.tidyverse.org/articles/starwars.html"
html <- read_html(url)

html %>%
  html_element("div") %>%
  html_children() %>%
  html_name()

Parse an html table into a data frame

Description

The algorithm mimics what a browser does, but repeats the values of merged cells in every cell that cover.

Usage

html_table(
  x,
  header = NA,
  trim = TRUE,
  fill = deprecated(),
  dec = ".",
  na.strings = "NA",
  convert = TRUE
)

Arguments

Value

When applied to a single element, html_table() returns a single tibble. When applied to multiple elements or a document, html_table() returns a list of tibbles.

Examples

sample1 <- minimal_html("<table>
  <tr><th>Col A</th><th>Col B</th></tr>
  <tr><td>1</td><td>x</td></tr>
  <tr><td>4</td><td>y</td></tr>
  <tr><td>10</td><td>z</td></tr>
</table>")
sample1 %>%
  html_element("table") %>%
  html_table()

# Values in merged cells will be duplicated
sample2 <- minimal_html("<table>
  <tr><th>A</th><th>B</th><th>C</th></tr>
  <tr><td>1</td><td>2</td><td>3</td></tr>
  <tr><td colspan='2'>4</td><td>5</td></tr>
  <tr><td>6</td><td colspan='2'>7</td></tr>
</table>")
sample2 %>%
  html_element("table") %>%
  html_table()

# If a row is missing cells, they'll be filled with NAs
sample3 <- minimal_html("<table>
  <tr><th>A</th><th>B</th><th>C</th></tr>
  <tr><td colspan='2'>1</td><td>2</td></tr>
  <tr><td colspan='2'>3</td></tr>
  <tr><td>4</td></tr>
</table>")
sample3 %>%
  html_element("table") %>%
  html_table()

Get element text

Description

There are two ways to retrieve text from a element: html_text() and html_text2(). html_text() is a thin wrapper around xml2::xml_text() which returns just the raw underlying text. html_text2() simulates how text looks in a browser, using an approach inspired by JavaScript's innerText(). Roughly speaking, it converts ⁠<br />⁠ to "\n", adds blank lines around ⁠<p>⁠ tags, and lightly formats tabular data.

html_text2() is usually what you want, but it is much slower than html_text() so for simple applications where performance is important you may want to use html_text() instead.

Usage

html_text(x, trim = FALSE)

html_text2(x, preserve_nbsp = FALSE)

Arguments

Value

A character vector the same length as x

Examples

# To understand the difference between html_text() and html_text2()
# take the following html:

html <- minimal_html(
  "<p>This is a paragraph.
    This another sentence.<br>This should start on a new line"
)

# html_text() returns the raw underlying text, which includes whitespace
# that would be ignored by a browser, and ignores the <br>
html %>% html_element("p") %>% html_text() %>% writeLines()

# html_text2() simulates what a browser would display. Non-significant
# whitespace is collapsed, and <br> is turned into a line break
html %>% html_element("p") %>% html_text2() %>% writeLines()

# By default, html_text2() also converts non-breaking spaces to regular
# spaces:
html <- minimal_html("<p>x&nbsp;y</p>")
x1 <- html %>% html_element("p") %>% html_text()
x2 <- html %>% html_element("p") %>% html_text2()

# When printed, non-breaking spaces look exactly like regular spaces
x1
x2
# But aren't actually the same:
x1 == x2
# Which you can confirm by looking at their underlying binary
# representaion:
charToRaw(x1)
charToRaw(x2)

Create an HTML document from inline HTML

Description

Create an HTML document from inline HTML

Usage

minimal_html(html, title = "")

Arguments

Examples

minimal_html("<p>test</p>")

Static web scraping (with xml2)

Description

read_html() works by performing a HTTP request then parsing the HTML received using the xml2 package. This is "static" scraping because it operates only on the raw HTML file. While this works for most sites, in some cases you will need to use read_html_live() if the parts of the page you want to scrape are dynamically generated with javascript.

Generally, we recommend using read_html() if it works, as it will be faster and more robust, as it has fewer external dependencies (i.e. it doesn't rely on the Chrome web browser installed on your computer.)

Usage

read_html(x, encoding = "", ..., options = c("RECOVER", "NOERROR", "NOBLANKS"))

Arguments

Examples

# Start by reading a HTML page with read_html():
starwars <- read_html("https://rvest.tidyverse.org/articles/starwars.html")

# Then find elements that match a css selector or XPath expression
# using html_elements(). In this example, each <section> corresponds
# to a different film
films <- starwars %>% html_elements("section")
films

# Then use html_element() to extract one element per film. Here
# we the title is given by the text inside <h2>
title <- films %>%
  html_element("h2") %>%
  html_text2()
title

# Or use html_attr() to get data out of attributes. html_attr() always
# returns a string so we convert it to an integer using a readr function
episode <- films %>%
  html_element("h2") %>%
  html_attr("data-id") %>%
  readr::parse_integer()
episode

Live web scraping (with chromote)

Description

read_html() operates on the HTML source code downloaded from the server. This works for most websites but can fail if the site uses javascript to generate the HTML. read_html_live() provides an alternative interface that runs a live web browser (Chrome) in the background. This allows you to access elements of the HTML page that are generated dynamically by javascript and to interact with the live page by clicking on buttons or typing in forms.

Behind the scenes, this function uses the chromote package, which requires that you have a copy of Google Chrome installed on your machine.

Usage

read_html_live(url)

Arguments

Value

read_html_live() returns an R6 LiveHTML object. You can interact with this object using the usual rvest functions, or call its methods, like ⁠$click()⁠, ⁠$scroll_to()⁠, and ⁠$type()⁠ to interact with the live page like a human would.

Examples

## Not run: 
# When we retrieve the raw HTML for this site, it doesn't contain the
# data we're interested in:
static <- read_html("https://www.forbes.com/top-colleges/")
static %>% html_elements(".TopColleges2023_tableRow__BYOSU")

# Instead, we need to run the site in a real web browser, causing it to
# download a JSON file and then dynamically generate the html:

sess <- read_html_live("https://www.forbes.com/top-colleges/")
sess$view()
rows <- sess %>% html_elements(".TopColleges2023_tableRow__BYOSU")
rows %>% html_element(".TopColleges2023_organizationName__J1lEV") %>% html_text()
rows %>% html_element(".grant-aid") %>% html_text()

## End(Not run)

Objects exported from other packages

Description

These objects are imported from other packages. Follow the links below to see their documentation.

magrittr

%>%

xml2

url_absolute

Functions renamed in rvest 1.0.0

Description

rvest 1.0.0 renamed a number of functions to ensure that every function has a common prefix, matching tidyverse conventions that emerged since rvest was first created.

• set_values() -> html_form_set()

• submit_form() -> session_submit()

• xml_tag() -> html_name()

• xml_node() & html_node() -> html_element()

• xml_nodes() & html_nodes() -> html_elements()

(html_node() and html_nodes() are only superseded because they're so widely used.)

Additionally all session related functions gained a common prefix:

• html_session() -> session()

• forward() -> session_forward()

• back() -> session_back()

• jump_to() -> session_jump_to()

• follow_link() -> session_follow_link()

Usage

set_values(form, ...)

submit_form(session, form, submit = NULL, ...)

xml_tag(x)

xml_node(...)

xml_nodes(...)

html_nodes(...)

html_node(...)

back(x)

forward(x)

jump_to(x, url, ...)

follow_link(x, ...)

html_session(url, ...)

Repair faulty encoding

Description

This function has been deprecated because it doesn't work. Instead re-read the HTML file with correct encoding argument.

Usage

repair_encoding(x, from = NULL)

Arguments

Simulate a session in web browser

Description

This set of functions allows you to simulate a user interacting with a website, using forms and navigating from page to page.

• Create a session with session(url)

• Navigate to a specified url with session_jump_to(), or follow a link on the page with session_follow_link().

• Submit an html_form with session_submit().

• View the history with session_history() and navigate back and forward with session_back() and session_forward().

• Extract page contents with html_element() and html_elements(), or get the complete HTML document with read_html().

• Inspect the HTTP response with httr::cookies(), httr::headers(), and httr::status_code().

Usage

session(url, ...)

is.session(x)

session_jump_to(x, url, ...)

session_follow_link(x, i, css, xpath, ...)

session_back(x)

session_forward(x)

session_history(x)

session_submit(x, form, submit = NULL, ...)

Arguments

Examples

s <- session("http://hadley.nz")
s %>%
  session_jump_to("hadley-wickham.jpg") %>%
  session_jump_to("/") %>%
  session_history()

s %>%
  session_jump_to("hadley-wickham.jpg") %>%
  session_back() %>%
  session_history()


s %>%
  session_follow_link(css = "p a") %>%
  html_elements("p")