Force execution of an Impala query

Description

compute()

Executes the query and stores the result in a new Impala table

collect()

Executes the query and returns the result to R as a data frame tbl

collapse()

Generates the query for later execution

Usage

## S3 method for class 'tbl_impala'
compute(
  x,
  name,
  temporary = TRUE,
  unique_indexes = NULL,
  indexes = NULL,
  analyze = FALSE,
  external = FALSE,
  overwrite = FALSE,
  force = FALSE,
  field_terminator = NULL,
  line_terminator = NULL,
  file_format = NULL,
  ...
)

## S3 method for class 'tbl_impala'
collect(x, ..., n = Inf, warn_incomplete = TRUE)

## S3 method for class 'tbl_impala'
collapse(x, vars = NULL, ...)

Arguments

Note

Impala does not support temporary tables. When using compute() to store results in an Impala table, you must set temporary = FALSE.

Copy a (very small) local data frame to Impala

Description

copy_to inserts the contents of a local data frame into a new Impala table. copy_to is intended to be used only with very small data frames. It uses the SQL INSERT ... VALUES() technique, which is not suitable for loading large amounts of data. By default, this function will throw an error if you attempt to copy a data frame with more than 1000 row/column positions. You can increase this limit at your own risk by setting the option implyr.copy_to_size_limit to a higher number.

This package does not provide tools for loading larger amounts of local data into Impala tables. This is because Impala can query data stored in several different filesystems and storage systems (HDFS, Apache Kudu, Apache HBase, Amazon S3, Microsoft ADLS, and Dell EMC Isilon) and Impala does not include built-in capability for loading local data into these systems.

Usage

## S3 method for class 'src_impala'
copy_to(
  dest,
  df,
  name = deparse(substitute(df)),
  overwrite = FALSE,
  types = NULL,
  temporary = TRUE,
  unique_indexes = NULL,
  indexes = NULL,
  analyze = FALSE,
  external = FALSE,
  force = FALSE,
  field_terminator = NULL,
  line_terminator = NULL,
  file_format = NULL,
  ...
)

Arguments

Value

An object with class tbl_impala, tbl_sql, tbl_lazy, tbl

Note

Impala does not support temporary tables. When using copy_to() to insert local data into an Impala table, you must set temporary = FALSE.

Examples

library(nycflights13)
dim(airlines) # airlines data frame is very small
# [1] 16  2

## Not run: 
copy_to(impala, airlines, temporary = FALSE)
## End(Not run)

Close the connection to Impala

Description

Closes (disconnects) the connection to Impala.

Usage

## S4 method for signature 'src_impala'
dbDisconnect(conn, ...)

Arguments

Value

Returns TRUE, invisibly

Examples

## Not run: 
dbDisconnect(impala)
## End(Not run)

Execute an Impala statement that returns no result

Description

Executes an Impala statement that returns no result.

Usage

## S4 method for signature 'src_impala,character'
dbExecute(conn, statement, ...)

Arguments

Value

Depending on the package used to connect to Impala, either a scalar numeric that specifies the number of rows affected by the statement, or NULL

Note

This method is for statements that return no result, such as data definition or data manipulation statements. Use dbGetQuery() for SELECT queries.

Examples

## Not run: 
dbExecute(impala, "INVALIDATE METADATA")
## End(Not run)

Send SQL query to Impala and retrieve results

Description

Returns the result of an Impala SQL query as a data frame.

Usage

## S4 method for signature 'src_impala,character'
dbGetQuery(conn, statement, ...)

Arguments

Value

A data.frame with as many rows as records were fetched and as many columns as fields in the result set, even if the result is a single value or has one or zero rows

Note

This method is for SELECT queries only. Use dbExecute() for data definition or data manipulation statements.

Examples

## Not run: 
flights_by_carrier_df <- dbGetQuery(
  impala,
  "SELECT carrier, COUNT(*) FROM flights GROUP BY carrier"
)
## End(Not run)

Describe the Impala data source

Description

Describe the Impala data source

Usage

## S3 method for class 'impala_connection'
db_desc(x)

Arguments

Value

A string containing information about the connection to Impala

Unnest a complex column in an Impala table

Description

impala_unnest() unnests a column of type ARRAY, MAP, or STRUCT in a tbl_impala. These column types are referred to as complex or nested types.

Usage

impala_unnest(data, col, ...)

Arguments

Details

impala_unnest() currently can unnest only one column, can only be applied once to a tbl_impala, and must be applied to a tbl_impala representing an Impala table or view before applying any other operations.

Value

an object with class tbl_impala with the complex column unnested into two or more separate columns

See Also

Impala Complex Types

Objects exported from other packages

Description

These objects are imported from other packages. Follow the links below to see their documentation.

dbplyr

in_schema

List all available databases

Description

Returns a character vector containing the names of all the available databases, in alphabetical order, including the _impala_builtins database.

Usage

src_databases(src, ...)

src_schemas(src, ...)

Arguments

Details

src_schemas() is an alias for src_databases()

Connect to Impala and create a remote dplyr data source

Description

src_impala creates a SQL backend to dplyr for Apache Impala, the massively parallel processing query engine for Apache Hadoop.

src_impala can work with any DBI-compatible interface that provides connectivity to Impala. Currently, two packages that can provide this connectivity are odbc and RJDBC.

Usage

src_impala(drv, ..., auto_disconnect = TRUE)

Arguments

Value

An object with class src_impala, src_sql, src

See Also

Impala ODBC driver, Impala JDBC driver

Examples

# Using ODBC connectivity:

## Not run: 
library(odbc)
drv <- odbc::odbc()
impala <- src_impala(
  drv = drv,
  driver = "Cloudera ODBC Driver for Impala",
  host = "host",
  port = 21050,
  database = "default",
  uid = "username",
  pwd = "password"
)
## End(Not run)

# Using JDBC connectivity:

## Not run: 
library(RJDBC)
Sys.setenv(JAVA_HOME = "/path/to/java/home/")
impala_classpath <- list.files(
  path = "/path/to/jdbc/driver",
  pattern = "\\.jar$",
  full.names = TRUE
)
.jinit(classpath = impala_classpath)
drv <- JDBC(
  driverClass = "com.cloudera.impala.jdbc41.Driver",
  classPath = impala_classpath,
  identifier.quote = "`"
)
impala <- src_impala(
  drv,
  "jdbc:impala://host:21050",
  "username",
  "password"
)
## End(Not run)

Create a lazy tbl from an Impala table

Description

Create a lazy tbl from an Impala table

Usage

## S3 method for class 'src_impala'
tbl(src, from, ...)

Arguments

Value

An object with class tbl_impala, tbl_sql, tbl_lazy, tbl

See Also

in_schema

Examples

## Not run: 
flights_tbl <- tbl(impala, "flights")

flights_tbl <- tbl(impala, in_schema("nycflights13", "flights"))
## End(Not run)