Batch Contextual Epsilon-Greedy Policy

Description

Batch Contextual Epsilon-Greedy Policy

Batch Contextual Epsilon-Greedy Policy

Details

Implements an epsilon-greedy exploration strategy for contextual bandits with batched updates.

Super class

cramR::NA

Public fields

Methods

Public methods

• BatchContextualEpsilonGreedyPolicy$new()

• BatchContextualEpsilonGreedyPolicy$set_parameters()

• BatchContextualEpsilonGreedyPolicy$get_action()

• BatchContextualEpsilonGreedyPolicy$set_reward()

• BatchContextualEpsilonGreedyPolicy$clone()

Method new()

Constructor for the Batch Epsilon-Greedy policy.

Usage

BatchContextualEpsilonGreedyPolicy$new(epsilon = 0.1, batch_size = 1)

Arguments

Method set_parameters()

Initializes the parameter structures for each arm.

Usage

BatchContextualEpsilonGreedyPolicy$set_parameters(context_params)

Arguments

Method get_action()

Chooses an arm based on epsilon-greedy logic and the current estimates.

Usage

BatchContextualEpsilonGreedyPolicy$get_action(t, context)

Arguments

Returns

A list with the selected action.

Method set_reward()

Updates model statistics based on observed reward. Updates occur once per batch.

Usage

BatchContextualEpsilonGreedyPolicy$set_reward(t, context, action, reward)

Arguments

Returns

Updated parameter estimates.

Method clone()

The objects of this class are cloneable with this method.

Usage

BatchContextualEpsilonGreedyPolicy$clone(deep = FALSE)

Arguments

Batch Contextual Thompson Sampling Policy

Description

Batch Contextual Thompson Sampling Policy

Batch Contextual Thompson Sampling Policy

Details

Implements Thompson Sampling for linear contextual bandits with batch updates.

Methods

- 'initialize(v = 0.2, batch_size = 1)': Constructor, sets variance and batch size. - 'set_parameters(context_params)': Initializes arm-level matrices. - 'get_action(t, context)': Samples from the posterior and selects action. - 'set_reward(t, context, action, reward)': Updates posterior statistics using observed feedback.

Super class

cramR::NA

Public fields

Methods

Public methods

• BatchContextualLinTSPolicy$new()

• BatchContextualLinTSPolicy$set_parameters()

• BatchContextualLinTSPolicy$get_action()

• BatchContextualLinTSPolicy$set_reward()

• BatchContextualLinTSPolicy$clone()

Method new()

Constructor for the batch-based Thompson Sampling policy.

Usage

BatchContextualLinTSPolicy$new(v = 0.2, batch_size = 1)

Arguments

Method set_parameters()

Initializes per-arm sufficient statistics.

Usage

BatchContextualLinTSPolicy$set_parameters(context_params)

Arguments

Method get_action()

Samples from the posterior distribution of expected rewards and selects an action.

Usage

BatchContextualLinTSPolicy$get_action(t, context)

Arguments

Returns

A list with the chosen arm ('choice').

Method set_reward()

Updates Gram matrix and response vector for the chosen arm. Parameters are refreshed every 'batch_size' rounds.

Usage

BatchContextualLinTSPolicy$set_reward(t, context, action, reward)

Arguments

Returns

Updated internal parameters.

Method clone()

The objects of this class are cloneable with this method.

Usage

BatchContextualLinTSPolicy$clone(deep = FALSE)

Arguments

Batch Disjoint LinUCB Policy with Epsilon-Greedy

Description

Batch Disjoint LinUCB Policy with Epsilon-Greedy

Batch Disjoint LinUCB Policy with Epsilon-Greedy

Details

Implements the disjoint LinUCB algorithm with upper confidence bounds and epsilon-greedy exploration, using batched updates.

Methods

- 'initialize(alpha = 1.0, epsilon = 0.1, batch_size = 1)': Constructor. - 'set_parameters(context_params)': Initializes sufficient statistics for each arm. - 'get_action(t, context)': Selects an arm using UCB scores and epsilon-greedy rule. - 'set_reward(t, context, action, reward)': Updates statistics and refreshes model at batch intervals.

Super class

cramR::NA

Public fields

Methods

Public methods

• BatchLinUCBDisjointPolicyEpsilon$new()

• BatchLinUCBDisjointPolicyEpsilon$set_parameters()

• BatchLinUCBDisjointPolicyEpsilon$get_action()

• BatchLinUCBDisjointPolicyEpsilon$set_reward()

• BatchLinUCBDisjointPolicyEpsilon$clone()

Method new()

Constructor for batched LinUCB with epsilon-greedy exploration.

Usage

BatchLinUCBDisjointPolicyEpsilon$new(alpha = 1, epsilon = 0.1, batch_size = 1)

Arguments

Method set_parameters()

Initialize arm-specific parameter containers.

Usage

BatchLinUCBDisjointPolicyEpsilon$set_parameters(context_params)

Arguments

Method get_action()

Chooses an arm based on UCB and epsilon-greedy sampling.

Usage

BatchLinUCBDisjointPolicyEpsilon$get_action(t, context)

Arguments

Returns

A list with the selected action.

Method set_reward()

Updates arm-specific sufficient statistics based on observed reward. Parameter updates occur only at the end of a batch.

Usage

BatchLinUCBDisjointPolicyEpsilon$set_reward(t, context, action, reward)

Arguments

Returns

Updated internal model parameters.

Method clone()

The objects of this class are cloneable with this method.

Usage

BatchLinUCBDisjointPolicyEpsilon$clone(deep = FALSE)

Arguments

Contextual Linear Bandit Environment

Description

Contextual Linear Bandit Environment

Contextual Linear Bandit Environment

Details

An R6 class for simulating a contextual linear bandit environment with normally distributed rewards.

Methods

- 'initialize(k, d, list_betas, sigma = 0.1, binary_rewards = FALSE)': Constructor. - 'post_initialization()': Loads correct coefficients based on 'sim_id'. - 'get_context(t)': Returns context and sets internal reward vector. - 'get_reward(t, context_common, action)': Returns observed reward for an action.

Super class

cramR::NA -> ContextualLinearBandit

Public fields

Methods

Public methods

• ContextualLinearBandit$new()

• ContextualLinearBandit$post_initialization()

• ContextualLinearBandit$get_context()

• ContextualLinearBandit$get_reward()

• ContextualLinearBandit$clone()

Method new()

Usage

ContextualLinearBandit$new(
  k,
  d,
  list_betas,
  sigma = 0.1,
  binary_rewards = FALSE
)

Arguments

Method post_initialization()

Set the simulation-specific coefficients for the current simulation.

Usage

ContextualLinearBandit$post_initialization()

Returns

No return value; modifies the internal state of the object.

Method get_context()

Usage

ContextualLinearBandit$get_context(t)

Arguments

Returns

A list containing context vector 'X' and arm count 'k'

Method get_reward()

Usage

ContextualLinearBandit$get_reward(t, context_common, action)

Arguments

Returns

A list with reward and optimal arm/reward info

Method clone()

The objects of this class are cloneable with this method.

Usage

ContextualLinearBandit$clone(deep = FALSE)

Arguments

LinUCB Disjoint Policy with Epsilon-Greedy Exploration

Description

LinUCB Disjoint Policy with Epsilon-Greedy Exploration

LinUCB Disjoint Policy with Epsilon-Greedy Exploration

Details

Implements the disjoint LinUCB algorithm with upper confidence bounds and epsilon-greedy exploration.

Methods

- 'initialize(alpha = 1.0, epsilon = 0.1)': Create a new LinUCBDisjointPolicyEpsilon object. - 'set_parameters(context_params)': Initialize arm-level parameters. - 'get_action(t, context)': Selects an arm using epsilon-greedy UCB. - 'set_reward(t, context, action, reward)': Updates internal statistics based on observed reward.

Super class

cramR::NA

Public fields

Methods

Public methods

• LinUCBDisjointPolicyEpsilon$new()

• LinUCBDisjointPolicyEpsilon$set_parameters()

• LinUCBDisjointPolicyEpsilon$get_action()

• LinUCBDisjointPolicyEpsilon$set_reward()

• LinUCBDisjointPolicyEpsilon$clone()

Method new()

Initializes the policy with UCB parameter alpha and exploration rate epsilon.

Usage

LinUCBDisjointPolicyEpsilon$new(alpha = 1, epsilon = 0.1)

Arguments

Method set_parameters()

Set arm-specific parameter structures.

Usage

LinUCBDisjointPolicyEpsilon$set_parameters(context_params)

Arguments

Method get_action()

Selects an arm using epsilon-greedy Upper Confidence Bound (UCB).

Usage

LinUCBDisjointPolicyEpsilon$get_action(t, context)

Arguments

Returns

A list containing the selected action.

Method set_reward()

Updates internal statistics using the observed reward for the selected arm.

Usage

LinUCBDisjointPolicyEpsilon$set_reward(t, context, action, reward)

Arguments

Returns

Updated internal parameters.

Method clone()

The objects of this class are cloneable with this method.

Usage

LinUCBDisjointPolicyEpsilon$clone(deep = FALSE)

Arguments

Cram Bandit: On-policy Statistical Evaluation in Contextual Bandits

Description

Performs the Cram method for On-policy Statistical Evaluation in Contextual Bandits

Usage

cram_bandit(pi, arm, reward, batch = 1, alpha = 0.05)

Arguments

Value

A list containing:

Examples

# Example with batch size of 1

# Set random seed for reproducibility
set.seed(42)

# Define parameters
T <- 100  # Number of timesteps
K <- 4    # Number of arms

# Simulate a 3D array pi of shape (T, T, K)
# - First dimension: Individuals (context Xj)
# - Second dimension: Time steps (pi_t)
# - Third dimension: Arms (depth)
pi <- array(runif(T * T * K, 0.1, 1), dim = c(T, T, K))

# Normalize probabilities so that each row sums to 1 across arms
for (t in 1:T) {
  for (j in 1:T) {
    pi[j, t, ] <- pi[j, t, ] / sum(pi[j, t, ])
    }
 }

# Simulate arm selections (randomly choosing an arm)
arm <- sample(1:K, T, replace = TRUE)

# Simulate rewards (assume normally distributed rewards)
reward <- rnorm(T, mean = 1, sd = 0.5)

result <- cram_bandit(pi, arm, reward, batch=1, alpha=0.05)
result$raw_results
result$interactive_table

Cram Bandit Policy Value Estimate

Description

This function implements the contextual armed bandit on-policy evaluation by providing the policy value estimate.

Usage

cram_bandit_est(pi, reward, arm, batch = 1)

Arguments

Value

The estimated policy value.

Cram Bandit Simulation

Description

This function runs on-policy simulation for contextual bandit algorithms using the Cram method. It evaluates the statistical properties of policy value estimates.

Usage

cram_bandit_sim(
  horizon,
  simulations,
  bandit,
  policy,
  alpha = 0.05,
  do_parallel = FALSE,
  seed = 42
)

Arguments

Value

A list containing:

Examples

# Number of time steps
horizon       <- 500L

# Number of simulations
simulations   <- 100L

# Number of arms
k = 4

# Number of context features
d= 3

# Reward beta parameters of linear model (the outcome generation models,
# one for each arm, are linear with arm-specific parameters betas)
list_betas <- cramR::get_betas(simulations, d, k)

# Define the contextual linear bandit, where sigma is the scale
# of the noise in the outcome linear model
bandit        <- cramR::ContextualLinearBandit$new(k = k,
                                                    d = d,
                                                    list_betas = list_betas,
                                                    sigma = 0.3)

# Define the policy object (choose between Contextual Epsilon Greedy,
# UCB Disjoint and Thompson Sampling)
policy <- cramR::BatchContextualEpsilonGreedyPolicy$new(epsilon=0.1,
                                                         batch_size=5)
# policy <- cramR::BatchLinUCBDisjointPolicyEpsilon$new(alpha=1.0,epsilon=0.1,batch_size=1)
# policy <- cramR::BatchContextualLinTSPolicy$new(v = 0.1, batch_size=1)


sim <- cram_bandit_sim(horizon, simulations,
                       bandit, policy,
                       alpha=0.05, do_parallel = FALSE)
sim$summary_table

Cram Bandit Variance of the Policy Value Estimate

Description

This function implements the crammed variance estimate of the policy value estimate for the contextual armed bandit on-policy evaluation setting.

Usage

cram_bandit_var(pi, reward, arm, batch = 1)

Arguments

Value

The crammed variance estimate of the policy value estimate.

Cram Policy Estimator for Policy Value Difference (Delta)

Description

This function returns the cram policy estimator for the policy value difference (delta).

Usage

cram_estimator(X, Y, D, pi, batch_indices, propensity = NULL)

Arguments

Value

The estimated policy value difference (Delta).

Cram ML Expected Loss Estimate

Description

This function computes the Cram ML expected loss estimator based on the given loss matrix and batch indices.

Usage

cram_expected_loss(loss, batch_indices)

Arguments

Value

The Cram ML expected loss estimate

Cram Policy Learning

Description

This function performs the learning part of the Cram Policy method.

Usage

cram_learning(
  X,
  D,
  Y,
  batch,
  model_type = "causal_forest",
  learner_type = "ridge",
  baseline_policy = NULL,
  parallelize_batch = FALSE,
  model_params = NULL,
  custom_fit = NULL,
  custom_predict = NULL,
  n_cores = detectCores() - 1,
  propensity = NULL
)

Arguments

Value

A list containing:

See Also

causal_forest, cv.glmnet, keras_model_sequential

Cram ML: Simultaneous Machine Learning and Evaluation

Description

Performs the Cram method for simultaneous machine learning and evaluation.

Usage

cram_ml(
  data,
  batch,
  formula = NULL,
  caret_params = NULL,
  parallelize_batch = FALSE,
  loss_name = NULL,
  custom_fit = NULL,
  custom_predict = NULL,
  custom_loss = NULL,
  alpha = 0.05,
  classify = FALSE
)

Arguments

Value

A list containing:

• raw_results: Data frame with performance metrics

• interactive_table: The same performance metrics in a user-friendly interface

• final_ml_model: Trained model object

See Also

train for model training parameters

Examples

# Load necessary libraries
library(caret)

# Set seed for reproducibility
set.seed(42)

# Generate example dataset
X_data <- data.frame(x1 = rnorm(100), x2 = rnorm(100), x3 = rnorm(100))
Y_data <- rnorm(100)  # Continuous target variable for regression
data_df <- data.frame(X_data, Y = Y_data)  # Ensure target variable is included

# Define caret parameters for simple linear regression (no cross-validation)
caret_params_lm <- list(
  method = "lm",
  trControl = trainControl(method = "none")
)

nb_batch <- 5

# Run ML learning function
result <- cram_ml(
  data = data_df,
  formula = Y ~ .,  # Linear regression model
  batch = nb_batch,
  loss_name = 'se',
  caret_params = caret_params_lm
)

result$raw_results
result$interactive_table
result$final_ml_model

Cram Policy: Efficient Simultaneous Policy Learning and Evaluation

Description

This function performs the cram method (simultaneous policy learning and evaluation) for binary policies on data including covariates (X), binary treatment indicator (D) and outcomes (Y).

Usage

cram_policy(
  X,
  D,
  Y,
  batch,
  model_type = "causal_forest",
  learner_type = "ridge",
  baseline_policy = NULL,
  parallelize_batch = FALSE,
  model_params = NULL,
  custom_fit = NULL,
  custom_predict = NULL,
  alpha = 0.05,
  propensity = NULL
)

Arguments

Value

A list containing:

• raw_results: A data frame summarizing key metrics with truncated decimals:

  • Delta Estimate: The estimated treatment effect (delta).

  • Delta Standard Error: The standard error of the delta estimate.

  • Delta CI Lower: The lower bound of the confidence interval for delta.

  • Delta CI Upper: The upper bound of the confidence interval for delta.

  • Policy Value Estimate: The estimated policy value.

  • Policy Value Standard Error: The standard error of the policy value estimate.

  • Policy Value CI Lower: The lower bound of the confidence interval for policy value.

  • Policy Value CI Upper: The upper bound of the confidence interval for policy value.

  • Proportion Treated: The proportion of individuals treated under the final policy.

• interactive_table: An interactive table summarizing key metrics for detailed exploration.

• final_policy_model: The final fitted policy model based on model_type and learner_type or custom_fit.

See Also

causal_forest, cv.glmnet, keras_model_sequential

Examples

# Example data
X_data <- matrix(rnorm(100 * 5), nrow = 100, ncol = 5)
D_data <- as.integer(sample(c(0, 1), 100, replace = TRUE))
Y_data <- rnorm(100)
nb_batch <- 5

# Perform CRAM policy
result <- cram_policy(X = X_data,
                          D = D_data,
                          Y = Y_data,
                          batch = nb_batch)

# Access results
result$raw_results
result$interactive_table
result$final_policy_model

Cram Policy: Estimator for Policy Value (Psi)

Description

This function returns the cram estimator for the policy value (psi).

Usage

cram_policy_value_estimator(X, Y, D, pi, batch_indices, propensity = NULL)

Arguments

Value

The estimated policy value.

Cram Policy Simulation

Description

This function performs the cram method (simultaneous learning and evaluation) on simulation data, for which the data generation process (DGP) is known. The data generation process for X can be given directly as a function or induced by a provided dataset via row-wise bootstrapping. Results are averaged across Monte Carlo replicates for the given DGP.

Usage

cram_simulation(
  X = NULL,
  dgp_X = NULL,
  dgp_D,
  dgp_Y,
  batch,
  nb_simulations,
  nb_simulations_truth = NULL,
  sample_size,
  model_type = "causal_forest",
  learner_type = "ridge",
  alpha = 0.05,
  baseline_policy = NULL,
  parallelize_batch = FALSE,
  model_params = NULL,
  custom_fit = NULL,
  custom_predict = NULL,
  propensity = NULL
)

Arguments

Value

A list containing:

avg_proportion_treated

The average proportion of treated individuals across simulations.

avg_delta_estimate

The average delta estimate across simulations.

avg_delta_standard_error

The average standard error of delta estimates.

delta_empirical_bias

The empirical bias of delta estimates.

delta_empirical_coverage

The empirical coverage of delta confidence intervals.

avg_policy_value_estimate

The average policy value estimate across simulations.

avg_policy_value_standard_error

The average standard error of policy value estimates.

policy_value_empirical_bias

The empirical bias of policy value estimates.

policy_value_empirical_coverage

The empirical coverage of policy value confidence intervals.

See Also

causal_forest, cv.glmnet, keras_model_sequential

Examples

set.seed(123)

# dgp_X <- function(n) {
#   data.table::data.table(
#     binary     = rbinom(n, 1, 0.5),
#     discrete   = sample(1:5, n, replace = TRUE),
#     continuous = rnorm(n)
#   )
# }

n <- 100

X_data <- data.table::data.table(
  binary     = rbinom(n, 1, 0.5),
  discrete   = sample(1:5, n, replace = TRUE),
  continuous = rnorm(n)
)


dgp_D <- function(X) rbinom(nrow(X), 1, 0.5)

dgp_Y <- function(D, X) {
  theta <- ifelse(
    X[, binary] == 1 & X[, discrete] <= 2,  # Group 1: High benefit
    1,
    ifelse(X[, binary] == 0 & X[, discrete] >= 4,  # Group 3: Negative benefit
           -1,
           0.1)  # Group 2: Neutral effect
  )
  Y <- D * (theta + rnorm(length(D), mean = 0, sd = 1)) +
    (1 - D) * rnorm(length(D))  # Outcome for untreated
  return(Y)
}

# Parameters
nb_simulations <- 100
nb_simulations_truth <- 200
batch <- 5

# Perform CRAM simulation
result <- cram_simulation(
  X = X_data,
  dgp_D = dgp_D,
  dgp_Y = dgp_Y,
  batch = batch,
  nb_simulations = nb_simulations,
  nb_simulations_truth = nb_simulations_truth,
  sample_size = 500
)

result$raw_results
result$interactive_table

Cram ML: Variance Estimate of the crammed expected loss estimate

Description

This function computes the variance estimator based on the given loss matrix and batch indices.

Usage

cram_var_expected_loss(loss, batch_indices)

Arguments

Value

The variance estimate of the crammed expected loss estimate

Cram Policy: Variance Estimate of the crammed Policy Value Difference (Delta)

Description

This function estimates the asymptotic variance of the cram estimator for the policy value difference (delta).

Usage

cram_variance_estimator(X, Y, D, pi, batch_indices, propensity = NULL)

Arguments

Value

The estimated variance of the policy value difference (Delta)

Cram Policy: Variance Estimate of the crammed Policy Value estimate (Psi)

Description

This function estimates the asymptotic variance of the cram estimator for the policy value (psi).

Usage

cram_variance_estimator_policy_value(
  X,
  Y,
  D,
  pi,
  batch_indices,
  propensity = NULL
)

Arguments

Value

The variance estimate of the crammed Policy Value estimate (Psi)

Cram Policy: Fit Model

Description

This function trains a given unfitted model with the provided data and parameters, according to model type and learner type.

Usage

fit_model(model, X, Y, D, model_type, learner_type, model_params, propensity)

Arguments

Value

The fitted model object.

Cram ML: Fit Model ML

Description

This function trains a given unfitted model with the provided data and parameters, according to model type and learner type.

Usage

fit_model_ml(data, formula, caret_params, classify)

Arguments

Value

The fitted model object.

Generate Reward Parameters for Simulated Linear Bandits

Description

Creates a list of matrices representing the arm-specific reward-generating parameters (betas) used in contextual linear bandit simulations. Each matrix corresponds to one simulation and contains normalized random coefficients.

Usage

get_betas(simulations, d, k)

Arguments

Value

A list of length simulations + 1 (first element being discarded in the underlying simulation package), each containing a d x k matrix of normalized reward parameters.

Cram ML: Generalized ML Learning

Description

This function performs batch-wise learning for machine learning models.

Usage

ml_learning(
  data,
  formula = NULL,
  batch,
  parallelize_batch = FALSE,
  loss_name = NULL,
  caret_params = NULL,
  custom_fit = NULL,
  custom_predict = NULL,
  custom_loss = NULL,
  n_cores = detectCores() - 1,
  classify = FALSE
)

Arguments

Value

A list containing:

Cram Policy: Predict with the Specified Model

Description

This function performs inference using a trained model, providing flexibility for different types of models such as Causal Forest, Ridge Regression, and Feedforward Neural Networks (FNNs).

Usage

model_predict(model, X, D, model_type, learner_type, model_params)

Arguments

Value

A vector of binary policy assignments, depending on the model_type and learner_type.

Cram ML: Predict with the Specified Model

Description

This function performs inference using a trained model

Usage

model_predict_ml(
  model,
  data,
  formula,
  caret_params,
  cram_policy_handle = FALSE
)

Arguments

Value

Predictions of the model on the data

Cram Policy: Set Model

Description

This function maps the model type and learner type to the corresponding model function.

Usage

set_model(model_type, learner_type, model_params)

Arguments

Value

The instantiated model object or the corresponding model function.

Validate or Set the Baseline Policy

Description

This function validates a provided baseline policy or sets a default baseline policy of zeros for all individuals.

Usage

test_baseline_policy(baseline_policy, n)

Arguments

Value

A validated or default baseline policy as a list of numeric values.

Validate or Generate Batch Assignments

Description

This function validates a provided batch assignment or generates random batch assignments for individuals.

Usage

test_batch(batch, n)

Arguments

Value

A list containing:

batches

A list where each element contains the indices of individuals assigned to a specific batch.

nb_batch

The total number of batches.

Cram Policy: Validate User-Provided Parameters for a Model

Description

This function validates user-provided parameters against the formal arguments of a specified model function. It ensures that all user-specified parameters are recognized by the model and raises an error for invalid parameters.

Usage

validate_params(model_function, model_type, learner_type, user_params)

Arguments

Value

A named list of validated parameters that are safe to pass to the model function.

Cram Policy: Validate Parameters for Feedforward Neural Networks (FNNs)

Description

This function validates user-provided parameters for a Feedforward Neural Network (FNN) model. It ensures the correct structure for input_layer, layers, output_layer, compile_args and fit_params.

Usage

validate_params_fnn(model_type, learner_type, model_params, X)

Arguments

Value

A named list of validated parameters merged with defaults for any missing values.