funModeling: Exploratory data analysis, data preparation and model performance

Description

funModeling is intimately related to the Data Science Live Book -Open Source- (2017) in the sense that most of its functionality is used to explain different topics addressed by the book.

Details

To start using funModeling you can start by the vignette: 'browseVignettes(package = "funModeling")'

Or you can read the Data Science Live Book, fully accessible at: https://livebook.datascienceheroes.com

Author(s)

Maintainer: Pablo Casas pcasas.biz@gmail.com

See Also

Useful links:

• https://livebook.datascienceheroes.com

• Report bugs at https://github.com/pablo14/funModeling/issues

Reduce cardinality in categorical variable by automatic grouping

Description

Reduce the cardinality of an input variable based on a target -binary by now- variable based on attribitues of accuracy and representativity, for both input and target variable. It uses a cluster model to create the new groups.

Usage

auto_grouping(data, input, target, n_groups, model = "kmeans", seed = 999)

Arguments

Value

A list containing 3 elements: recateg_results which contains the description of the target variable with the new groups; df_equivalence is a data frame containing the input category and the new category; fit_cluster which is the cluster model used to do the re-grouping

Examples

# Reducing quantity of countries based on has_flu variable
auto_grouping(data=data_country, input='country', target="has_flu", n_groups=8)

Profiling analysis of categorical vs. target variable

Description

Retrieves a complete summary of the grouped input variable against the target variable. Type of target variable must be binary for now. A positive case will be the less representative one. It returns the total positive cases (sum_target)); pecentage of total positive cases (perc_target) that fell in that category (this column sums 1); likelihood or mean of positive cases (mean_target) measured by the total positive cases over total cases in that category; quantity of rows of that category (q_rows) and in percentage (perc_rows) -this column sums 1.

Usage

categ_analysis(data, input, target)

Arguments

Value

if input has 1 variable, it retrurns a data frame indicating all the metrics, otherwise prints in console all variable results.

Examples

categ_analysis(data_country, "country", "has_flu")

Compare two data frames by keys

Description

Obtain differences between two data frames

Usage

compare_df(dfcomp_x, dfcomp_y, keys_x, keys_y = NA, compare_values = FALSE)

Arguments

Value

Differences and coincident values

Examples

data(heart_disease)
a=heart_disease
b=heart_disease
a=subset(a, age >45)
b=subset(b, age <50)
b$gender='male'
b$chest_pain=ifelse(b$chest_pain ==3, 4, b$chest_pain)
res=compare_df(a, b, c('age', 'gender'))
# Print the keys that didn't match
res
# Accessing the keys not present in the first data frame
res[[1]]$rows_not_in_X
# Accessing the keys not present in the second data frame
res[[1]]$rows_not_in_Y
# Accessing the keys which coincide completely
res[[1]]$coincident
# Accessing the rows whose values did not coincide
res[[1]]$different_values

Concatenate 'N' variables

Description

Concatenate 'N' variables using the char pipe: <|>. This function is used when there is the need of measuring the mutual information and/or the information gain between 'N' input variables an against a target variable. This function makes sense when it is used based on categorical data.

Usage

concatenate_n_vars(data, vars)

Arguments

Value

vector containing the concatenated values for the given variables

Examples

new_variable=concatenate_n_vars(mtcars, c("cyl", "disp"))
# Checking new variable
head(new_variable)

Convert every column in a data frame to character

Description

It converts all the variables present in 'data' to character. Criteria conversion is based on two functions, discretize_get_bins plus discretize_df, which will discretize all the numerical variables based on equal frequency criteria, with the number of bins equal to 'n_bins'. This only applies for numerical variables which unique valuesare more than 'n_bins' parameter. After this step, it may happen that variables remain non-character, so these variables will be converting directly into character.

Usage

convert_df_to_categoric(data, n_bins)

Arguments

Value

data frame containing all variables as character

Examples

# before
df_status(heart_disease)

# after
new_df=convert_df_to_categoric(data=heart_disease, n_bins=5)
df_status(new_df)

Coordinate plot

Description

Calculate the means (or other function defined in 'group_func' parameter) per group to analyze how each segment behave. It scales each variable mean inti the 0 to 1 range to easily profile the groups according to its mean. It also calculate the mean regardless the grouping. This function is also useful when you want to profile cluster results in terms of its means.

Usage

coord_plot(data, group_var, group_func = mean, print_table = FALSE)

Arguments

Value

coordinate plot, if print_table=T it also prints a table with the average per column plus the average of the whole column

Examples

# calculating the differences based on function 'mean'
coord_plot(data=mtcars, group_var="cyl")
# printing the table used to generate the coord_plot
coord_plot(data=mtcars, group_var="cyl", print_table=TRUE)
# printing the table used to generate the coord_plot
coord_plot(data=mtcars, group_var="cyl", group_func=median, print_table=TRUE)

Get correlation against target variable

Description

Obtain correlation table for all variables against target variable. Only numeric variables are analyzed (factor/character are skippted automatically).

Usage

correlation_table(data, target)

Arguments

Value

Correlation index for all data input variable

Examples

correlation_table(data=heart_disease, target="has_heart_disease")

Cross-plotting input variable vs. target variable

Description

The cross_plot shows how the input variable is correlated with the target variable, getting the likelihood rates for each input's bin/bucket .

Usage

cross_plot(data, input, target, path_out, auto_binning, plot_type = "both")

Arguments

Value

cross plot

Examples

## Example 1:
cross_plot(data=heart_disease, input="chest_pain", target="has_heart_disease")

## Example 2: Disabling auto_binning:
cross_plot(data=heart_disease, input="oldpeak",
		target="has_heart_disease", auto_binning=FALSE)

## Example 3: Saving the plot into a folder:
#cross_plot(data=heart_disease, input="oldpeak",
#		target="has_heart_disease", path_out = "my_folder")

## Example 4: Running with multiple input variables at the same time:
cross_plot(data=heart_disease, input=c("age", "oldpeak", "max_heart_rate"),
		target="has_heart_disease")

People with flu data

Description

Each row represents a person from different countries indicating if he or she has or not flu. Colmuns person: unique id country: country of the person, 70 different countries has_flu: character variable with values "yes" or "no" indicating if the person has flu

Usage

data_country

Format

A data frame with 910 rows and 3 variables

Play golf

Description

This well known small data frame containst 14 cases indicating wheter or not play golf based on wheather conditions. Target variable: 'play_golf.'

Usage

data_golf

Format

A data frame with 14 rows and 3 variables

Data integrity

Description

A handy function to return different vectors of variable names aimed to quickly filter NA, categorical (factor / character), numerical and other types (boolean, date, posix). It also returns a vector of variables which have high cardinality. It returns an 'integrity' object, which has: 'status_now' (comes from status function), and 'results' list, following elements can be found:

vars_cat: Vector containing the categorical variables names (factor or character)

vars_num: Vector containing the numerical variables names

vars_char: Vector containing the character variables names

vars_factor: Vector containing the factor variables names

vars_other: Vector containing the other variables names (date time, posix and boolean)

vars_num_with_NA: Summary table for numerical variables with NA

vars_cat_with_NA: Summary table for categorical variables with NA

vars_cat_high_card: Summary table for high cardinality variables (where thershold = MAX_UNIQUE parameter)

vars_one_value: Vector containing the variables names with 1 unique different value

Explore the NA and high cardinality variables by doing summary(integrity_object), or a full summary by doing print(integrity_object)

Usage

data_integrity(data, MAX_UNIQUE = 35)

Arguments

Value

An 'integrity' object.

Examples

# Example 1:
data_integrity(heart_disease)
# Example 2:
# changing the default minimum threshold to flag a variable as high cardiniality
data_integrity(data=data_country, MAX_UNIQUE=50)

Check data integrity model

Description

Given a data frame, we need to create models (xgboost, random forest, regression, etc). Each one of them has its constraints regarding data types. Many errors appear when we are creating models just because of data format. This function returns, given a certain model, which are the constraints that the data is not satisfying. This way we can anticipate and correct errors before we call for model creation. This function is quite related to data_integrity.

Usage

data_integrity_model(data, model_name, MAX_UNIQUE = 35)

Arguments

Value

an 'integritymodel' object

Profiling categorical variable

Description

Calculate the means (or other function) per group to analyze how each segment behave. It scales each variable mean inti the 0 to 1 range to easily profile the groups according to its mean. It also calculate the mean regardless the grouping. This function is also useful when you want to profile cluster results in terms of its means. It automatically adds a row representing the sumarization of the column regardless the group_var categories, this is useful to compare each segement with the whole population. It will exclude all factor/character variables.

Usage

desc_groups(data, group_var, group_func = mean, add_all_data_row = T)

Arguments

Value

grouped data frame

Examples

# default grouping function: mean
desc_groups(data=mtcars, group_var="cyl")

# using the median as the grouping function
desc_groups(data=mtcars, group_var="cyl", group_func=median)

# using the max as the grouping function
desc_groups(data=mtcars, group_var="gear", group_func=max)

Profiling categorical variable (rank)

Description

Similar to 'desc_groups' function, this one computes the rank of each value in order to quickly know what is the value in each segment that has the highest value (rank=1). 1 represent the highest number. It will exclude all factor/character variables.

Usage

desc_groups_rank(data, group_var, group_func = mean)

Arguments

Value

grouped data frame, showing the rank instead of the absolute values/

Examples

# default grouping function: mean
desc_groups_rank(data=mtcars, group_var="gear")

# using the median as the grouping function
desc_groups(data=mtcars, group_var="cyl", group_func=median)

# using the max as the grouping function
desc_groups_rank(data=mtcars, group_var="gear", group_func=max)

Get a summary for the given data frame (o vector).

Description

For each variable it returns: Quantity and percentage of zeros (q_zeros and p_zeros respectevly). Same metrics for NA values (q_NA/p_na), and infinite values (q_inf/p_inf). Last two columns indicates data type and quantity of unique values. This function print and return the results.

Usage

df_status(data, print_results)

Arguments

Value

Metrics data frame

Examples

df_status(heart_disease)

Discretize a data frame

Description

Converts all numerical variables into factor or character, depending on 'stringsAsFactors' parameter, based on equal frequency criteria. The thresholds for each segment in each variable are generated based on the output of discretize_get_bins function, which returns a data frame containing the threshold for each variable. This result is must be the 'data_bins' parameter input. Important to note that the returned data frame contains the non-transformed variables plus the transformed ones. More info about converting numerical into categorical variables can be found at: https://livebook.datascienceheroes.com/data-preparation.html#data_types

Usage

discretize_df(data, data_bins, stringsAsFactors = TRUE)

Arguments

Value

Data frame with the transformed variables

Examples

# Getting the bins thresholds for each. If input is missing, 
# will run for all numerical variables.
d_bins=discretize_get_bins(data=heart_disease,
input=c("resting_blood_pressure", "oldpeak"), n_bins=5)

# Now it can be applied on the same data frame,
# or in a new one (for example in a predictive model that 
# change data over time)
heart_disease_discretized=
discretize_df(data=heart_disease, 
data_bins=d_bins, 
stringsAsFactors=TRUE)

Get the data frame thresholds for discretization

Description

It takes a data frame and returns another data frame indicating the threshold for each bin (or segment) in order to discretize the variable.

Usage

discretize_get_bins(data, n_bins = 5, input = NULL)

Arguments

Value

Data frame containing the thresholds or cuts to bin every variable

Variable discretization by gain ratio maximization

Description

Discretize numeric variable by maximizing the gain ratio between each bucket and the target variable.

Usage

discretize_rgr(input, target, min_perc_bins = 0.1, max_n_bins = 5)

Arguments

Value

discretized variable (factor)

Examples

library(funModeling)
data=heart_disease
input=data$oldpeak
target=as.character(data$has_heart_disease)

input2=discretize_rgr(input, target)

# checking:
summary(input2)

Computes the entropy between two variables

Description

It calculates the entropy between two categorical variables using log2. This log2 is mentioned in most of the Claude Shannon bibliography. Input/target can be numeric or character.

Usage

entropy_2(input, target)

Arguments

Value

Entropy measured in bits

Examples

# Measuring entropy between input and target variable
entropy_2(input=data_golf$outlook, target=data_golf$play_golf)

Equal frequency binning

Description

Equal frequency tries to put the same quantity of cases per bin when possible. It's a wrapper of function cut2 from Hmisc package.

Usage

equal_freq(var, n_bins)

Arguments

Value

The binned variable.

Examples

## Example 1
summary(heart_disease$age)
age_2=equal_freq(var=heart_disease$age, n_bins = 10)
summary(age_2)

## Example 2
age_3=equal_freq(var=heart_disease$age, n_bins = 5)
summary(age_3)

Export plot to jpeg file

Description

Export 'object_plot' to jpeg file under the name 'file_name' in the directory 'path_out'

Usage

export_plot(object_plot, path_out, file_name)

Arguments

Value

none

Fibonacci series

Description

It retrieves a vector containing the first N numbers specified in 'length' parameter of the Fibonacci series.

Usage

fibonacci(length, remove_first = FALSE)

Arguments

Value

vector

Examples

# Get the first 4 elements of Fibonacci series
fibonacci(4)

Frequency table for categorical variables

Description

Retrieves the frequency and percentage for input

Usage

freq(data, input = NA, plot = TRUE, na.rm = FALSE, path_out)

Arguments

Value

vector with the values scaled into the 0 to 1 range

Examples

freq(data=heart_disease$thal)
freq(data=heart_disease, input = c('thal','chest_pain'))

Generates lift and cumulative gain performance table and plot

Description

It retrieves the cumulative positive rate -gain curve- and the lift chart & plot when score is divided in 5, 10 or 20 segments. Both metrics give a quality measure about how well the model predicts. Higher values at the beginning of the population implies a better model. More info at: https://livebook.datascienceheroes.com/model-performance.html#scoring_data

Usage

gain_lift(data, score, target, q_segments = 10)

Arguments

Value

lift/gain table, column: gain implies how much positive cases are catched if the cut point to define the positive class is set to the column "Score Point"

Examples

fit_glm=glm(has_heart_disease ~ age + oldpeak, data=heart_disease, family = binomial)
heart_disease$score=predict(fit_glm, newdata=heart_disease, type='response')
gain_lift(data=heart_disease, score='score', target='has_heart_disease')

Gain ratio

Description

Computes the information gain between an 'input' and 'target' variable (using log2). Similar to information gain but less sensitive to high cardinality variables.

Usage

gain_ratio(input, target)

Arguments

Value

gain ratio

Examples

gain_ratio(input=data_golf$outlook, target=data_golf$play_golf)

Sampling training and test data

Description

Split input data into training and test set, retrieving always same sample by setting the seed.

Usage

get_sample(data, percentage_tr_rows = 0.8, seed = 987)

Arguments

Value

TRUE/FALSE vector same length as 'data' param. TRUE represents that row position is for training data

Examples

# Training and test data. Percentage of training cases default value=80%.
index_sample=get_sample(data=heart_disease, percentage_tr_rows=0.8)
# Generating the samples
data_tr=heart_disease[index_sample,]
data_ts=heart_disease[-index_sample,]

Hampel Outlier Threshold

Description

Retrieves the bottom and top boundaries to flag outliers or extreme values, according to the Hampel method. This technique takes into account the median and MAD value, which is a is a robust measure of the variability of a univariate sample of quantitative data (Wikipedia). Similar to standard deviation but less sensitve to outliers. This function is used in 'prep_outliers' function. All 'NA's values are automatically excluded. More information at: https://livebook.datascienceheroes.com/data-preparation.html#how_to_deal_with_outliers_in_r.

Usage

hampel_outlier(input, k_mad_value = 3)

Arguments

Value

A two-item vector, the first value represents the bottom threshold, while the second one is the top threshold

Examples

hampel_outlier(heart_disease$age)

Heart Disease Data

Description

There are variables related to patient clinic trial. The variable to predict is 'has_heart_disease'.

Usage

heart_disease

Format

A data frame with 303 rows and 16 variables:

https://archive.ics.uci.edu/ml/datasets/Heart+Disease

Computes several information theory metrics between two vectors

Description

It retrieves the same as var_rank_info but receiving two vectors. Metrics are: entropy (en), mutual information (mi), information gain (ig) and gain ratio (gr).

Usage

infor_magic(input, target)

Arguments

Value

Matrix of 1 row and 4 columns, where each column represent the mentioned metrics

Examples

infor_magic(data_golf$outlook, data_golf$play_golf)

Information gain

Description

Computes the information gain between an 'input' and 'target' variable (using log2). In general terms, the higher the more predictable the input is.

Usage

information_gain(input, target)

Arguments

Value

information gain

Examples

information_gain(input=data_golf$outlook, target=data_golf$play_golf)

Metadata models data integrity

Description

Metadata models data integrity

Usage

metadata_models

Format

Tibble

Plotting numerical data

Description

Retrieves one plot containing all the histograms for numerical variables. NA values will not be displayed.

Usage

plot_num(data, bins = 10, path_out = NA)

Arguments

Value

plot containing all numerical variables

Examples

plot_num(mtcars)
# changing the bins parameter and exporting the plot
# plot_num(data=mtcars, bins=5, path_out="my_folder")

Correlation plots

Description

Visual correlation analysis. Plot different graphs in order to expose the inner information of any numeric variable against the target variable

Usage

plotar(data, input, target, plot_type, path_out)

Arguments

Value

Single or multiple plots specified by 'plot_type' parameter

Examples

## It runs for all numeric variables automatically
plotar(data=heart_disease, target="has_heart_disease", plot_type="histdens")

plotar(heart_disease, input = 'age', target = 'chest_pain', plot_type = "boxplot")

Outliers Data Preparation

Description

Deal with outliers by setting an 'NA value' or by 'stopping' them at a certain. There are three supported methods to flag the values as outliers: "bottom_top", "tukey" and "hampel". The parameters: 'top_percent' and/or 'bottom_percent' are used only when method="bottom_top".

For a full reference please check the official documentation at: https://livebook.datascienceheroes.com/data-preparation.html#treatment_outliers> Setting NA is recommended when doing statistical analysis, parameter: type='set_na'. Stopping is recommended when creating a predictive model without biasing the result due to outliers, parameter: type='stop'.

The function can take a data frame, and returns the same data plus the transformations specified in the input parameter. Or it can take a single vector (in the same 'data' parameter), and it returns a vector.

Usage

prep_outliers(
  data,
  input = NA,
  type = NA,
  method = NA,
  bottom_percent = NA,
  top_percent = NA,
  k_mad_value = NA
)

Arguments

Value

A data frame with the desired outlier transformation

Examples

# Creating data frame with outliers
set.seed(10)
df=data.frame(var1=rchisq(1000,df = 1), var2=rnorm(1000))
df=rbind(df, 1135, 2432) # forcing outliers
df$id=as.character(seq(1:1002))

# for var1: mean is ~ 4.56, and max 2432
summary(df)

########################################################
### PREPARING OUTLIERS FOR DESCRIPTIVE STATISTICS
########################################################

#### EXAMPLE 1: Removing top 1%% for a single variable
# checking the value for the top 1% of highest values (percentile 0.99), which is ~ 7.05
quantile(df$var1, 0.99)

# Setting type='set_na' sets NA to the highest value specified by top_percent.
# In this case 'data' parameter is single vector, thus it returns a single vector as well.
var1_treated=prep_outliers(data = df$var1, type='set_na', top_percent  = 0.01,method = "bottom_top")

# now the mean (~ 1) is more accurate, and note that: 1st, median and 3rd
#  quartiles remaining very similar to the original variable.
summary(var1_treated)

#### EXAMPLE 2: Removing top and bottom 1% for the specified input variables.
vars_to_process=c('var1', 'var2')
df_treated3=prep_outliers(data = df, input = vars_to_process, type='set_na',
 bottom_percent = 0.01, top_percent  = 0.01, method = "bottom_top")
summary(df_treated3)

########################################################
### PREPARING OUTLIERS FOR PREDICTIVE MODELING
########################################################

data_prep_h=funModeling::prep_outliers(data = heart_disease,
input = c('age','resting_blood_pressure'),
 method = "hampel",  type='stop')

# Using Hampel method to flag outliers:
summary(heart_disease$age);summary(data_prep_h$age)
# it changed from 29 to 29.31, and the max remains the same at 77
hampel_outlier(heart_disease$age) # checking the thresholds

data_prep_a=funModeling::prep_outliers(data = heart_disease,
input = c('age','resting_blood_pressure'),
 method = "tukey",  type='stop')

max(heart_disease$age);max(data_prep_a$age)
# remains the same (77) because the max thers for age is 100
tukey_outlier(heart_disease$age)

Profiling numerical data

Description

Get a metric table with many indicators for all numerical variables, automatically skipping the non-numerical variables. Current metrics are: mean, std_dev: standard deviation, all the p_XX: percentile at XX number, skewness, kurtosis, iqr: inter quartile range, variation_coef: the ratio of sd/mean, range_98 is the limit for which the 98

Usage

profiling_num(data)

Arguments

Value

metrics table

Examples

profiling_num(mtcars)

Transform a variable into the [0-1] range

Description

Range a variable into [0-1], assigning 0 to the min and 1 to the max of the input variable. All NA values will be removed.

Usage

range01(var)

Arguments

Value

vector with the values scaled into the 0 to 1 range

Examples

range01(mtcars$cyl)

Get a summary for the given data frame (o vector).

Description

For each variable it returns: Quantity and percentage of zeros (q_zeros and p_zeros respectevly). Same metrics for NA values (q_NA/p_na), and infinite values (q_inf/p_inf). Last two columns indicates data type and quantity of unique values. 'status' function is the evolution of 'df_status'. Main change is to have the decimal points as it is, except in percentage. For example now p_na=0.04 means 4 This time it's easier to embbed in a data process flow and to take actions based on this number.

Usage

status(data)

Arguments

Value

Tibble with metrics

Examples

status(heart_disease)

Tukey Outlier Threshold

Description

Retrieves the bottom and top boundaries to flag outliers or extreme values, according to the Tukey's test. More info at https://en.wikipedia.org/wiki/Outlier#Tukey.27s_test This function is used in 'prep_outliers' function. All 'NA's values are automatically excluded. More information at: https://livebook.datascienceheroes.com/data-preparation.html#how_to_deal_with_outliers_in_r.

Usage

tukey_outlier(input)

Arguments

Value

A two-item vector, the first value represents the bottom threshold, while the second one is the top threshold

Examples

tukey_outlier(heart_disease$age)

Compare two vectors

Description

Obtaing coincident and not coincident elements between two vectors.

Usage

v_compare(vector_x, vector_y)

Arguments

Value

Correlation index for all data input variable

Examples

v1=c("height","weight","age")
v2=c("height","weight","location","q_visits")
res=v_compare(vector_x=v1, vector_y=v2)
# Print the keys that didn't match
res
# Accessing the keys not present in

Importance variable ranking based on information theory

Description

Retrieves a data frame containing several metrics related to information theory. Metrics are: entropy (en), mutual information (mi), information gain (ig) and gain ratio (gr).

Usage

var_rank_info(data, target)

Arguments

Value

data frame ordered by gain ratio metric

Examples

var_rank_info(data_golf, "play_golf")