Title:	Analysing Non-Verbal Communication in Dyadic Interactions from Video Data
Version:	0.1.1
Description:	Analyzes non-verbal communication by processing data extracted from video recordings of dyadic interactions. It supports integration with open source tools, currently limited to 'OpenPose' (Cao et al. (2019) <doi:10.1109/TPAMI.2019.2929257>), converting its outputs into CSV format for further analysis. The package includes functions for data pre-processing, visualization, and computation of motion indices such as velocity, acceleration, and jerkiness (Cook et al. (2013) <doi:10.1093/brain/awt208>), facilitating the analysis of non-verbal cues in paired interactions and contributing to research on human communication dynamics.
License:	MIT + file LICENSE
SystemRequirements:	FFmpeg (https://ffmpeg.org/)
Imports:	dplyr, ggplot2, ggthemes, graphics, grDevices, kza, parallel, patchwork, reshape2, rjson, rlang, signal, stats, stringr, tidyr, tidyselect, tools, utils, zoo
Suggests:	spelling, testthat (≥ 3.0.0)
Config/testthat/edition:	3
Encoding:	UTF-8
Language:	en-US
RoxygenNote:	7.3.2
NeedsCompilation:	no
Packaged:	2025-01-16 11:32:32 UTC; tefthimi
Author:	Themis Efthimiou [aut, cre]
Maintainer:	Themis Efthimiou <themiftw20@gmail.com>
Repository:	CRAN
Date/Publication:	2025-01-17 08:50:02 UTC

Animate OpenPose data for a dyad across a range of frames (Video)

Description

This function generates a video of the OpenPose data for both persons in a dyad across a specified range of frames.

Usage

op_animate_dyad(
  data,
  output_file,
  lines = FALSE,
  keylabels = FALSE,
  label_type = "names",
  fps = 24,
  min_frame = NULL,
  max_frame = NULL,
  hide_labels = FALSE,
  left_color = "blue",
  right_color = "red",
  background_color = "white",
  background_colour = NULL
)

Arguments

Value

No return value. This function generates a video file as a side effect, saved at the specified output path.

Examples

## Not run: 
# Example OpenPose data
data <- data.frame(
  frame = rep(1:10, each = 2),
  person = rep(c("left", "right"), times = 10),
  x0 = runif(20, 0, 1920), y0 = runif(20, 0, 1080),
  x1 = runif(20, 0, 1920), y1 = runif(20, 0, 1080)
)

# Output file path
output_file <- tempfile("output_video", fileext = ".mp4")

# Generate video
op_animate_dyad(
  data = data,
  output_file = output_file,
  fps = 24,
  left_color = "blue",
  right_color = "red"
)

## End(Not run)

Rename Columns Based on Region

Description

This function renames columns of a dataframe based on the specified region.

Usage

op_apply_keypoint_labels(df)

Arguments

Value

Dataframe with renamed columns.

Examples

# Example dataframe
df <- data.frame(
  region = rep(c("body", "hand_left", "hand_right", "face"), each = 3),
  x0 = rnorm(12), y0 = rnorm(12), c0 = rnorm(12),
  x1 = rnorm(12), y1 = rnorm(12), c1 = rnorm(12)
)

# Apply keypoint labels
df_renamed <- op_apply_keypoint_labels(df)
print(df_renamed)

Process all Dyad Directories to Create CSV Files Using MultimodalR

Description

This function processes all dyad directories in the specified input base path, applying the op_create_csv function from the package, and saves the output in the corresponding directories in the output base path.

Usage

op_batch_create_csv(
  input_base_path,
  output_base_path,
  include_filename = TRUE,
  include_labels = FALSE,
  frame_width = 1920,
  export_type = "dyad",
  model = "all",
  overwrite = FALSE
)

Arguments

Value

None. The function is called for its side effects.

Compute Acceleration

Description

This function calculates the acceleration for each column that begins with 'x' and 'y' and removes all columns that start with 'c'. It takes either the fps or the video duration as input to compute the acceleration.

Usage

op_compute_acceleration(
  data,
  fps = NULL,
  video_duration = NULL,
  overwrite = FALSE,
  merge_xy = FALSE
)

Arguments

Value

A data frame with acceleration columns added and 'c' columns removed.

Examples

# Load example data from the package
data_path <- system.file("extdata/csv_data/A-B_body_dyad_velocity.csv", package = "duet")
data <- read.csv(data_path)

# Compute acceleration
result <- op_compute_acceleration(
  data = data,
  fps = 30,
  overwrite = FALSE,
  merge_xy = TRUE
)

print(result)

Compute Jerk

Description

This function calculates the jerk for each column that begins with 'x' and 'y' and removes all columns that start with 'c'. It takes either the fps or the video duration as input to compute the jerk.

Usage

op_compute_jerk(
  data,
  fps = NULL,
  video_duration = NULL,
  overwrite = FALSE,
  merge_xy = FALSE
)

Arguments

Value

A data frame with jerk columns added and 'c' columns removed.

Examples

# Load example data from the package
data_path <- system.file("extdata/csv_data/A-B_body_dyad_accel.csv", package = "duet")
data <- read.csv(data_path)

# Compute jerk
result <- op_compute_jerk(
  data = data,
  fps = 30,
  overwrite = FALSE,
  merge_xy = TRUE
)

print(result)

Compute Velocity

Description

This function calculates the velocity for each column that begins with 'x' and 'y' and removes all columns that start with 'c'. It takes either the fps or the video duration as input to compute the velocity.

Usage

op_compute_velocity(
  data,
  fps = NULL,
  video_duration = NULL,
  overwrite = FALSE,
  merge_xy = FALSE
)

Arguments

Value

A data frame with velocity columns added and 'c' columns removed.

Examples

# Load example data from the package
data_path <- system.file("extdata/csv_data/A-B_body_dyad.csv", package = "duet")
data <- read.csv(data_path)

# Compute velocity
result <- op_compute_velocity(
  data = data,
  fps = 30,
  overwrite = FALSE,
  merge_xy = TRUE
)

print(result)

Create CSV from JSON files

Description

This function reads JSON files from the specified directory, processes the pose keypoints, and saves the results into CSV files.

Usage

op_create_csv(
  input_path,
  output_path = input_path,
  model = "all",
  include_filename = FALSE,
  include_labels = FALSE,
  frame_width = 1920,
  export_type = "dyad"
)

Arguments

Value

No return value. This function is called for its side effects, which include writing CSV files to the specified output directory.

Examples

# Path to example JSON files included with the package
input_path <- system.file("extdata/json_files", package = "duet")

# Temporary output directory
output_path <- tempfile("output_csv")
dir.create(output_path)

# Run the function using the provided data
op_create_csv(
  input_path = input_path,
  output_path = output_path,
  model = "body",
  include_filename = TRUE,
  include_labels = TRUE,
  frame_width = 1920,
  export_type = "dyad"
)

Interpolate missing or low-confidence values in a dataset

Description

This function performs interpolation for x and y coordinate columns in a dataset based on confidence thresholds. It groups the data by person and region and uses spline interpolation to estimate missing or low-confidence values.

Usage

op_interpolate(
  data,
  confidence_threshold,
  missing = TRUE,
  treat_na_conf_as_low = FALSE
)

Arguments

Value

A modified data frame with interpolated x and y values for low-confidence or missing rows.

Examples

# Load example data from the package
data_path <- system.file("extdata/csv_data/A-B_body_dyad.csv", package = "duet")
data <- read.csv(data_path)

# Interpolate missing or low-confidence values
result <- op_interpolate(
  data = data,
  confidence_threshold = 0.5,
  missing = TRUE,
  treat_na_conf_as_low = TRUE
)

print(result)

Merge CSV files for each dyad

Description

This function merges all CSV files in each dyad directory within the specified input base path.

Usage

op_merge_dyad(input_base_path, output_base_path)

Arguments

Value

None. The function is called for its side effects.

Examples

# Load example data paths from the package
input_base_path <- system.file("extdata/csv_data/dyad_1", package = "duet")
output_base_path <- tempfile("merged_dyads")

# Ensure input files exist
input_files <- list.files(input_base_path, pattern = "\\.csv$", full.names = TRUE)
if (length(input_files) > 0) {
  # Merge CSV files for each dyad
  op_merge_dyad(input_base_path, output_base_path)

  # Check merged files
  merged_files <- list.files(output_base_path, pattern = "\\.csv$", full.names = TRUE)
  print(merged_files)

  # Read and display merged data
  if (length(merged_files) > 0) {
    merged_data <- read.csv(merged_files[1])
    print(merged_data)
  } else {
    message("No merged files were created.")
  }
} else {
  message("No input files found to process.")
}

Plot OpenPose Data for a Specified Frame

Description

This function visualizes keypoints and their connections from OpenPose data for a specified frame. The function allows customization of the plot, including the option to display labels, lines between keypoints, and different colours for left and right persons.

Usage

op_plot_openpose(
  data,
  frame_num,
  person = c("both", "left", "right"),
  lines = TRUE,
  keylabels = FALSE,
  label_type = c("names", "numbers"),
  hide_labels = FALSE,
  left_color = "blue",
  right_color = "red",
  background_color = "white",
  background_colour = NULL,
  line_width = 2,
  point_size = 1.5,
  text_color = "black"
)

Arguments

Value

No return value, called for side effects (plotting to screen).

Examples

# Path to example CSV file included with the package
file_path <- system.file("extdata/csv_data/A-B_body_dyad.csv", package = "duet")

# Load the data
data <- read.csv(file_path)

# Plot the data for the specified frame
op_plot_openpose(
  data = data,
  frame_num = 1,
  person = "both",
  lines = TRUE,
  keylabels = TRUE,
  label_type = "names",
  left_color = "blue",
  right_color = "red",
  background_colour = "grey90"
)

Plot Data Quality (Confidence Ratings or Completeness)

Description

This function plots either the mean confidence ratings, the percentage of completeness (i.e., data present), or both for the given dataframe. It can handle data for one or multiple persons and regions, creating separate panels for each.

Usage

op_plot_quality(df, plot_type = "confidence", threshold_line = 50)

Arguments

Value

A ggplot object or a combined plot if "both" is selected.

Examples

# Example usage:
# Path to example CSV file included with the package
file_path <- system.file("extdata/csv_data/A-B_body_dyad.csv", package = "duet")

# Load the data
data <- read.csv(file_path)

# plot <- op_plot_data_quality(df, plot_type = "both", threshold_line = 75)
# print(plot)

Plot Keypoints with Facet Wrap

Description

This function plots the specified keypoints over time with facet wrapping. It allows the user to overlay axes or separate them, and to filter the data by person. If the number of facets exceeds the max_facets limit, a warning is issued, and the function will not return a plot.

Usage

op_plot_timeseries(
  data,
  keypoints = NULL,
  free_y = TRUE,
  overlay_axes = FALSE,
  person = "both",
  max_facets = 10
)

Arguments

Value

A ggplot object or NULL if the maximum number of facets is exceeded.

Examples

# Load example data from the package
data_path <- system.file("extdata/csv_data/dyad_1/A_body.csv", package = "duet")
data <- read.csv(data_path)

# Plot with overlay_axes = TRUE
op_plot_timeseries(data = data, keypoints = c("0", "1"), overlay_axes = TRUE, person = "both")

# Plot with overlay_axes = FALSE
op_plot_timeseries(data = data, keypoints = c("0", "1"), overlay_axes = FALSE, person = "left")

Remove Keypoints Based on Various Criteria

Description

This function removes keypoints and their corresponding columns based on several criteria: user-specified keypoints, low total confidence values over time, exceeding a threshold of missing/zero values, or if all data for a keypoint is missing (i.e., all zeros).

Usage

op_remove_keypoints(
  df,
  remove_specific_keypoints = NULL,
  remove_undetected_keypoints = FALSE,
  remove_keypoints_total_confidence = NULL,
  remove_keypoints_missing_data = NULL,
  apply_removal_equally = TRUE
)

Arguments

Value

A data frame with specified keypoints and corresponding columns removed.

Examples

# Load example data from the package
data_path <- system.file("extdata/csv_data/dyad_1/A_body.csv", package = "duet")
df <- read.csv(data_path)

# Remove keypoints based on various criteria
result <- op_remove_keypoints(
  df = df,
  remove_specific_keypoints = c("1", "2"), # Remove specific keypoints (e.g., keypoints 1 and 2)
  remove_undetected_keypoints = TRUE,      # Remove keypoints with all zero confidence
  remove_keypoints_total_confidence = 0.5, # Remove keypoints with mean confidence below 0.5
  remove_keypoints_missing_data = 0.2,     # Remove keypoints with >20% missing data
  apply_removal_equally = TRUE             # Apply removal equally across the dataset
)

# Display the result
print(result)

Smooth Time Series Data with Various Methods

Description

This function applies different smoothing techniques to time series data for the selected columns (keypoints), including moving average, Kalman-Ziegler Adaptive (KZA), Savitzky-Golay filter, and Butterworth filter. It can optionally plot the smoothed data alongside the original data, with faceting based on the person and keypoint columns.

Arguments

Value

A data frame with the smoothed time series data for the specified keypoints. If plot = TRUE, a plot is displayed comparing the original and smoothed data.

Examples

# Load example data from the package
data_path <- system.file("extdata/csv_data/dyad_1/A_body.csv", package = "duet")
data <- read.csv(data_path)

# Smooth the time series data using the Savitzky-Golay filter
smoothed_data <- op_smooth_timeseries(
  data = data,
  method = "savitzky",
  sg_window = 5,
  sg_order = 3,
  plot = TRUE,
  keypoints = c("x0", "y0") # Specify keypoints to smooth
)

# Print the smoothed data
print(smoothed_data)