Package {IBclust}

IBclust: Information Bottleneck Clustering for Mixed-Type Data

Description

The IBclust package provides clustering methods based on the Information Bottleneck (IB) principle, supporting continuous, categorical (nominal and ordinal), and mixed-type data. Five algorithmic variants are implemented: AIBmix for hierarchical (agglomerative) clustering, DIBmix for hard partitional clustering, IBmix for soft (fuzzy) clustering, GIBmix for the generalised case interpolating between the two, and sIBmix for hard partitional clustering with hierarchical assignment steps (hybrid method).

Choosing a method

All methods share the same kernel similarity machinery; they differ in the form of the partition produced and the objective optimised:

• AIBmix: hierarchical, hard clustering via greedy bottom-up merging that minimises Jensen-Shannon divergence between cluster distributions. Returns a full hierarchy from n singletons to one cluster.

• DIBmix: hard partitional clustering for a fixed number of clusters; uses dynamic regularisation to guarantee non-empty clusters.

• IBmix: soft (fuzzy) partitional clustering; each observation has a probability vector over clusters.

• GIBmix: generalised case with a tunable fuzziness parameter \alpha \in [0, 1] that interpolates between DIBmix (\alpha = 0) and IBmix (\alpha = 1).

• sIBmix: hard partitional clustering for a fixed number of clusters; contrary to DIBmix, it does not make use of regularisation but rather starts by a random partition into the required number of clusters and uses the bottom-up approach rule used in AIBmix for re-assigning observations into clusters.

For mathematical details of each algorithm, see the references on the individual function help pages, or Costa et al. (2026) for a unified treatment.

Kernel functions

The functions in IBclust use kernel density estimation with generalised product kernels to construct the matrix P_{Y|X} of kernel weights. Bandwidth parameters control the smoothness of the estimate and can be either user-supplied or automatically selected.

For continuous variables:

• Gaussian (RBF) kernel (Silverman 1998):

  K_c\left(\frac{x - x'}{s}\right) = \frac{1}{\sqrt{2\pi}} \exp\left\{-\frac{\left(x - x'\right)^2}{2s^2}\right\}, \quad s > 0.

• Epanechnikov kernel (Epanechnikov 1969):

  K_c(x - x'; s) = \begin{cases} \frac{3}{4\sqrt{5}}\left(1 - \frac{(x-x')^2}{5s^2} \right), & \text{if } \frac{(x - x')^2}{s^2} < 5 \\ 0, & \text{otherwise} \end{cases}, \quad s > 0.

For nominal (unordered categorical) variables:

• Aitchison & Aitken kernel (Aitchison and Aitken 1976):

  K_u(x = x'; \lambda) = \begin{cases} 1 - \lambda, & \text{if } x = x' \\ \frac{\lambda}{\ell - 1}, & \text{otherwise} \end{cases}, \quad 0 \leq \lambda \leq \frac{\ell - 1}{\ell}.

• Li & Racine kernel (Ouyang et al. 2006):

  K_u(x = x'; \lambda) = \begin{cases} 1, & \text{if } x = x' \\ \lambda, & \text{otherwise} \end{cases}, \quad 0 \leq \lambda \leq 1.

For ordinal (ordered categorical) variables:

• Li & Racine kernel (Li and Racine 2003):

  K_o(x = x'; \nu) = \begin{cases} 1, & \text{if } x = x' \\ \nu^{|x - x'|}, & \text{otherwise} \end{cases}, \quad 0 \leq \nu \leq 1.

• Wang & van Ryzin kernel (Wang and Van Ryzin 1981):

  K_o(x = x'; \nu) = \begin{cases} 1 - \nu, & \text{if } x = x' \\ \frac{1-\nu}{2}\nu^{|x - x'|}, & \text{otherwise} \end{cases}, \quad 0 \leq \nu \leq 1.

\ell is the number of levels of the categorical variable. For ordinal variables, the lambda argument is used to define \nu. Bandwidths are automatically selected using the approach in Costa et al. (2026) when set to -1.

Nyström approximation

Constructing the kernel similarity matrix P_{Y|X} has cost O(n^2), which becomes prohibitive for large datasets. The partitional methods (DIBmix, IBmix, GIBmix) support a Nyström approximation (Williams and Seeger 2000) of the form P_{Y|X} \approx C W C^\top, where C is an n \times m matrix of kernel weights between all observations and m landmark points, and W is the m \times m kernel matrix over the landmarks alone. This reduces the cost of constructing P_{Y|X} from O(n^2) to O(nm), with m \ll n. The heuristic m \approx \sqrt{n} is used as the default. The approximation is enabled by setting nystrom = TRUE in the fit functions; the number of landmark points is controlled via n_landmarks, and specific landmark observations can be requested via landmark_indices. The Nyström approximation is not available for AIBmix, since its greedy merging procedure has cost O(n^3) regardless of how P_{Y|X} is constructed. This is also the case for sIBmix, where the cost is fixed at O(n^2).

Author(s)

Maintainer: Angelos Markos amarkos@gmail.com

Authors:

• Efthymios Costa

• Ioanna Papatsouma

References

Aitchison J, Aitken CG (1976). “Multivariate binary discrimination by the kernel method.” Biometrika, 63(3), 413–420.

Li Q, Racine J (2003). “Nonparametric estimation of distributions with categorical and continuous data.” Journal of Multivariate Analysis, 86(2), 266–292.

Silverman BW (1998). Density Estimation for Statistics and Data Analysis (1st Ed.). Routledge.

Ouyang D, Li Q, Racine J (2006). “Cross-validation and the estimation of probability distributions with categorical data.” Journal of Nonparametric Statistics, 18(1), 69–100.

Wang M, Van Ryzin J (1981). “A class of smooth estimators for discrete distributions.” Biometrika, 68(1), 301–309.

Epanechnikov VA (1969). “Non-parametric estimation of a multivariate probability density.” Theory of Probability & Its Applications, 14(1), 153–158.

Costa E, Papatsouma I, Markos A (2026). “A deterministic information bottleneck method for clustering mixed-type data.” Pattern Recognition, 179, 113580. ISSN 0031-3203, doi:10.1016/j.patcog.2026.113580.

Williams C, Seeger M (2000). “Using the Nyström method to speed up kernel machines.” Advances in Neural Information Processing Systems, 13.

Agglomerative Information Bottleneck Clustering for Mixed-Type Data

Description

The AIBmix function implements the Agglomerative Information Bottleneck (AIB) algorithm for hierarchical clustering of datasets containing mixed-type variables, including categorical (nominal and ordinal) and continuous variables. This method merges clusters so that information retention is maximised at each step to create meaningful clusters, leveraging bandwidth parameters to handle different categorical data types (nominal and ordinal) effectively (Slonim and Tishby 1999).

Usage

AIBmix(
  X,
  s = -1,
  lambda = -1,
  scale = TRUE,
  contkernel = "gaussian",
  nomkernel = "aitchisonaitken",
  ordkernel = "liracine",
  cat_first = FALSE,
  keep_data = TRUE
)

Arguments

Details

The AIBmix function produces a hierarchical agglomerative clustering of the data while retaining maximal information about the original variable distributions. The Agglomerative Information Bottleneck algorithm uses an information-theoretic criterion to merge clusters so that information retention is maximised at each step, hence creating meaningful clusters with maximal information about the original distribution. Bandwidth parameters for categorical (nominal, ordinal) and continuous variables are adaptively determined if not provided. This process identifies stable and interpretable cluster assignments by maximizing mutual information while controlling complexity. The method is well-suited for datasets with mixed-type variables and integrates information from all variable types effectively.

See IBclust-package for details on the available kernel functions and their bandwidth parameters.

Value

An object of class "aibclust" representing the final clustering result. The returned object is a list with the following components:

An object of class aibclust. See aibclust-methods for the available S3 methods (print, summary, plot, fitted, coef, info_metrics, as.hclust, as.dendrogram).

Author(s)

Efthymios Costa, Ioanna Papatsouma, Angelos Markos

References

Slonim N, Tishby N (1999). “Agglomerative Information Bottleneck.” Advances in Neural Information Processing Systems, 12.

Examples

# Example dataset with categorical, ordinal, and continuous variables
set.seed(123)
data_mix <- data.frame(
  cat_var = factor(sample(letters[1:3], 100, replace = TRUE)),      # Nominal categorical variable
  ord_var = factor(sample(c("low", "medium", "high"), 100, replace = TRUE),
                   levels = c("low", "medium", "high"),
                   ordered = TRUE),                                # Ordinal variable
  cont_var1 = rnorm(100),                                          # Continuous variable 1
  cont_var2 = runif(100)                                           # Continuous variable 2
)

# Perform Mixed-Type Hierarchical Clustering with Agglomerative IB
result_mix <- AIBmix(X = data_mix, lambda = -1, s = -1, scale = TRUE)

# Print clustering results
plot(result_mix, type = "dendrogram", xlab = "", sub = "", cex = 0.5)  # Plot dendrogram
plot(result_mix, type = "info", col = "black", pch = 16)  # Plot info retention curve

# Simulated categorical data example
set.seed(123)
data_cat <- data.frame(
  Var1 = as.factor(sample(letters[1:3], 200, replace = TRUE)),  # Nominal variable
  Var2 = as.factor(sample(letters[4:6], 200, replace = TRUE)),  # Nominal variable
  Var3 = factor(sample(c("low", "medium", "high"), 200, replace = TRUE),
                levels = c("low", "medium", "high"), ordered = TRUE)  # Ordinal variable
)

# Run AIBmix with automatic lambda selection
result_cat <- AIBmix(X = data_cat, lambda = -1)
coef(result_cat)                       # Check bandwidths chosen
fitted(result_cat, ncl = 3)            # Partition at the 3-cluster cut
info_metrics(result_cat, ncl = 3)      # Information-theoretic quantities at 3 clusters
plot(result_cat, ncl = 3, type = "similarity") # Plot of similarity matrix at 3 clusters
plot(result_cat, type = "dendrogram", xlab = "", sub = "", cex = 0.5)  # Plot dendrogram

# Results summary
summary(result_cat)

# Simulated continuous data example
set.seed(123)
# Continuous data with 200 observations, 5 features
data_cont <- as.data.frame(matrix(rnorm(1000), ncol = 5))

# Run AIBmix with automatic bandwidth selection
result_cont <- AIBmix(X = data_cont, s = -1, scale = TRUE)

# Print concise summary of output
print(result_cont)

# Convert to hclust for standard tree tools
hc <- as.hclust(result_cont)
cutree(hc, k = 3)

Deterministic Information Bottleneck Clustering for Mixed-Type Data

Description

The DIBmix function implements the Deterministic Information Bottleneck (DIB) algorithm for clustering datasets containing continuous, categorical (nominal and ordinal), and mixed-type variables. This method optimises an information-theoretic objective to preserve relevant information in the cluster assignments while achieving effective data compression (Costa et al. 2026).

Usage

DIBmix(
  X,
  ncl,
  randinit = NULL,
  s = -1,
  lambda = -1,
  scale = TRUE,
  maxiter = 100,
  nstart = 100,
  contkernel = "gaussian",
  nomkernel = "aitchisonaitken",
  ordkernel = "liracine",
  cat_first = FALSE,
  verbose = FALSE,
  nystrom = FALSE,
  n_landmarks = NULL,
  landmark_indices = NULL,
  keep_data = TRUE
)

Arguments

Details

The DIBmix function clusters data while retaining maximal information about the original variable distributions. The Deterministic Information Bottleneck algorithm optimises an information-theoretic objective that balances information preservation and compression. Bandwidth parameters for categorical (nominal, ordinal) and continuous variables are adaptively determined if not provided. This iterative process identifies stable and interpretable cluster assignments by maximising mutual information while controlling complexity. The method is well-suited for datasets with mixed-type variables and integrates information from all variable types effectively. For data sets with over a thousand observations (n > 1000), a Nyström approximation of the kernel Gram matrix can be enabled via nystrom = TRUE; see IBclust-package for details.

See IBclust-package for details on the available kernel functions and their bandwidth parameters.

Value

An object of class gibclust. See gibclust-methods for the available S3 methods (print, summary, plot, fitted, coef, info_metrics, predict).

Author(s)

Efthymios Costa, Ioanna Papatsouma, Angelos Markos

References

Costa E, Papatsouma I, Markos A (2026). “A deterministic information bottleneck method for clustering mixed-type data.” Pattern Recognition, 179, 113580. ISSN 0031-3203, doi:10.1016/j.patcog.2026.113580.

Examples

# Example 1: Basic Mixed-Type Clustering
set.seed(123)

# Create a more realistic dataset with mixed variable types
data_mix <- data.frame(
  # Categorical variables
  education = factor(sample(c("High School", "Bachelor", "Master", "PhD"), 150,
                           replace = TRUE, prob = c(0.4, 0.3, 0.2, 0.1))),
  employment = factor(sample(c("Full-time", "Part-time", "Unemployed"), 150,
                            replace = TRUE, prob = c(0.6, 0.25, 0.15))),

  # Ordinal variable
  satisfaction = factor(sample(c("Low", "Medium", "High"), 150, replace = TRUE),
                       levels = c("Low", "Medium", "High"), ordered = TRUE),

  # Continuous variables
  income = rlnorm(150, meanlog = 10, sdlog = 0.5),  # Log-normal income
  age = rnorm(150, mean = 35, sd = 10),             # Normally distributed age
  experience = rpois(150, lambda = 8)               # Years of experience
)

# Perform Mixed-Type Clustering
result_mix <- DIBmix(X = data_mix, ncl = 3, nstart = 5)

# View results
print(result_mix)
summary(result_mix)
table(fitted(result_mix))

# Example 2: Comparing cat_first parameter
# When categorical variables are more informative
result_cat_first <- DIBmix(X = data_mix, ncl = 3,
                           cat_first = TRUE,  # Prioritise categorical variables
                           nstart = 5)

# When continuous variables are more informative (default)
result_cont_first <- DIBmix(X = data_mix, ncl = 3,
                            cat_first = FALSE,
                            nstart = 5)

# Compare clustering performance
if (requireNamespace("mclust", quietly = TRUE)){  # For adjustedRandIndex
  print(paste("Agreement between approaches:",
              round(mclust::adjustedRandIndex(fitted(result_cat_first),
                    fitted(result_cont_first)), 3)))
  }

plot(result_cat_first, type = "sizes") # Bar plot of cluster sizes
plot(result_cat_first, type = "info")  # Information-theoretic quantities plot
plot(result_cat_first, type = "beta")  # Plot of evolution of beta
plot(result_cat_first, type = "importance") # Variable importance plot
plot(result_cat_first, type = "similarity") # Similarity matrix plot

# Simulated categorical data example
data_cat <- data.frame(
  Var1 = as.factor(sample(letters[1:3], 200, replace = TRUE)),  # Nominal variable
  Var2 = as.factor(sample(letters[4:6], 200, replace = TRUE)),  # Nominal variable
  Var3 = factor(sample(c("low", "medium", "high"), 200, replace = TRUE),
                levels = c("low", "medium", "high"), ordered = TRUE)  # Ordinal variable
)

# Perform hard clustering on categorical data with Deterministic IB
result_cat <- DIBmix(X = data_cat, ncl = 3, lambda = -1, nstart = 5)

# Print clustering results
fitted(result_cat)        # Cluster assignments
info_metrics(result_cat)  # Information-theoretic quantities
coef(result_cat)          # Hyperparameter values

# Simulated continuous data example
set.seed(123)
# Continuous data with 200 observations, 5 features
data_cont <- as.data.frame(matrix(rnorm(1000), ncol = 5))

# Perform hard clustering on continuous data with Deterministic IB
result_cont <- DIBmix(X = data_cont, ncl = 3, s = -1, nstart = 5)

# Print clustering results
fitted(result_cont)       # Cluster assignments

# Summary of output
print(result_cont)
summary(result_cont)

Generalised Information Bottleneck Clustering for Mixed-Type Data

Description

The GIBmix function implements the Generalised Information Bottleneck (GIB) algorithm for clustering datasets containing continuous, categorical (nominal and ordinal), and mixed-type variables. This method optimises an information-theoretic objective to preserve relevant information in the cluster assignments while achieving effective data compression (Strouse and Schwab 2017).

Usage

GIBmix(
  X,
  ncl,
  beta,
  alpha,
  randinit = NULL,
  s = -1,
  lambda = -1,
  scale = TRUE,
  maxiter = 100,
  nstart = 100,
  conv_tol = 1e-05,
  contkernel = "gaussian",
  nomkernel = "aitchisonaitken",
  ordkernel = "liracine",
  cat_first = FALSE,
  verbose = FALSE,
  nystrom = FALSE,
  n_landmarks = NULL,
  landmark_indices = NULL,
  keep_data = TRUE
)

Arguments

Details

The GIBmix function produces a fuzzy clustering of the data while retaining maximal information about the original variable distributions. The Generalised Information Bottleneck algorithm optimises an information-theoretic objective that balances information preservation and compression. Bandwidth parameters for categorical (nominal, ordinal) and continuous variables are adaptively determined if not provided. This iterative process identifies stable and interpretable cluster assignments by maximising mutual information while controlling complexity. The method is well-suited for datasets with mixed-type variables and integrates information from all variable types effectively. Set \alpha = 1 and \alpha = 0 to recover the Information Bottleneck and its Deterministic variant, respectively. If \alpha = 0, the algorithm ignores the value of the regularisation parameter \beta. For data sets with over a thousand observations (n > 1000), a Nyström approximation of the kernel Gram matrix can be enabled via nystrom = TRUE; see IBclust-package for details.

See IBclust-package for details on the available kernel functions and their bandwidth parameters.

Value

An object of class "gibclust" representing the final clustering result. The returned object is a list with the following components:

An object of class gibclust. See gibclust-methods for the available S3 methods (print, summary, plot, fitted, coef, info_metrics, predict).

Author(s)

Efthymios Costa, Ioanna Papatsouma, Angelos Markos

References

Strouse DJ, Schwab DJ (2017). “The Deterministic Information Bottleneck.” Neural Computation, 29(6), 1611–1630.

Examples

# Example dataset with categorical, ordinal, and continuous variables
set.seed(123)
data_mix <- data.frame(
  cat_var = factor(sample(letters[1:3], 100, replace = TRUE)),      # Nominal categorical variable
  ord_var = factor(sample(c("low", "medium", "high"), 100, replace = TRUE),
                   levels = c("low", "medium", "high"),
                   ordered = TRUE),                                # Ordinal variable
  cont_var1 = rnorm(100),                                          # Continuous variable 1
  cont_var2 = runif(100)                                           # Continuous variable 2
)

# Perform Mixed-Type Fuzzy Clustering with Generalised IB
result_mix <- GIBmix(X = data_mix, ncl = 3, beta = 2, alpha = 0.5, nstart = 5)

# Print clustering results
fitted(result_mix, method = "soft")  # Cluster membership matrix
info_metrics(result_mix)             # Information-theoretic quantities
coef(result_mix)                     # Hyperperameters used

# Summary of output
summary(result_mix)

# Simulated categorical data example
set.seed(123)
data_cat <- data.frame(
  Var1 = as.factor(sample(letters[1:3], 200, replace = TRUE)),  # Nominal variable
  Var2 = as.factor(sample(letters[4:6], 200, replace = TRUE)),  # Nominal variable
  Var3 = factor(sample(c("low", "medium", "high"), 200, replace = TRUE),
                levels = c("low", "medium", "high"), ordered = TRUE)  # Ordinal variable
)

# Perform Fuzzy Clustering on categorical data with Generalised IB
result_cat <- GIBmix(X = data_cat, ncl = 2, beta = 25, alpha = 0.75, lambda = -1, nstart = 5)

# Print clustering results
fitted(result_cat, method = "soft")       # Cluster membership matrix
fitted(result_cat, method = "classes")    # Hardened cluster memberships

# Simulated continuous data example
set.seed(123)
# Continuous data with 200 observations, 5 features
data_cont <- as.data.frame(matrix(rnorm(1000), ncol = 5))

# Perform Fuzzy Clustering on continuous data with Generalised IB
result_cont <- GIBmix(X = data_cont, ncl = 2, beta = 50, alpha = 0.75, s = -1, nstart = 5)

# Print clustering results
print(result_cont) 

plot(result_cont, type = "sizes") # Bar plot of cluster sizes (hardened assignments)
plot(result_cont, type = "info")  # Information-theoretic quantities plot
# Variable importance plot (hardened assignments)
plot(result_cont, type = "importance")
plot(result_cont, type = "membership") # Cluster membership plot
plot(result_cont, type = "similarity") # Similarity matrix plot

Information Bottleneck Clustering for Mixed-Type Data

Description

The IBmix function implements the Information Bottleneck (IB) algorithm for clustering datasets containing continuous, categorical (nominal and ordinal), and mixed-type variables. This method optimises an information-theoretic objective to preserve relevant information in the cluster assignments while achieving effective data compression (Strouse and Schwab 2019).

Usage

IBmix(
  X,
  ncl,
  beta,
  randinit = NULL,
  s = -1,
  lambda = -1,
  scale = TRUE,
  maxiter = 100,
  nstart = 100,
  conv_tol = 1e-05,
  contkernel = "gaussian",
  nomkernel = "aitchisonaitken",
  ordkernel = "liracine",
  cat_first = FALSE,
  verbose = FALSE,
  nystrom = FALSE,
  n_landmarks = NULL,
  landmark_indices = NULL,
  keep_data = TRUE
)

Arguments

Details

The IBmix function produces a fuzzy clustering of the data while retaining maximal information about the original variable distributions. The Information Bottleneck algorithm optimises an information-theoretic objective that balances information preservation and compression. Bandwidth parameters for categorical (nominal, ordinal) and continuous variables are adaptively determined if not provided. This iterative process identifies stable and interpretable cluster assignments by maximising mutual information while controlling complexity. The method is well-suited for datasets with mixed-type variables and integrates information from all variable types effectively. For data sets with over a thousand observations (n > 1000), a Nyström approximation of the kernel Gram matrix can be enabled via nystrom = TRUE; see IBclust-package for details.

See IBclust-package for details on the available kernel functions and their bandwidth parameters.

Value

An object of class gibclust representing the final clustering result. The returned object is a list with the following components:

An object of class gibclust. See gibclust-methods for the available S3 methods (print, summary, plot, fitted, coef, info_metrics, predict).

Author(s)

Efthymios Costa, Ioanna Papatsouma, Angelos Markos

References

Strouse DJ, Schwab DJ (2019). “The information bottleneck and geometric clustering.” Neural Computation, 31(3), 596–612.

Examples

# Example dataset with categorical, ordinal, and continuous variables
set.seed(123)
data_mix <- data.frame(
  cat_var = factor(sample(letters[1:3], 100, replace = TRUE)),      # Nominal categorical variable
  ord_var = factor(sample(c("low", "medium", "high"), 100, replace = TRUE),
                   levels = c("low", "medium", "high"),
                   ordered = TRUE),                                # Ordinal variable
  cont_var1 = rnorm(100),                                          # Continuous variable 1
  cont_var2 = runif(100)                                           # Continuous variable 2
)

# Perform Mixed-Type Fuzzy Clustering
result_mix <- IBmix(X = data_mix, ncl = 3, beta = 2, nstart = 1)

# Print clustering results
summary(result_mix)                    # Output summary
fitted(result_mix, method = "soft")    # Fuzzy clustering output
fitted(result_mix, method = "classes") # Hardened classes
coef(result_mix)                       # Hyperparameter values used
info_metrics(result_mix)               # Information-theoretic quantities


# Simulated categorical data example
set.seed(123)
data_cat <- data.frame(
  Var1 = as.factor(sample(letters[1:3], 100, replace = TRUE)),  # Nominal variable
  Var2 = as.factor(sample(letters[4:6], 100, replace = TRUE)),  # Nominal variable
  Var3 = factor(sample(c("low", "medium", "high"), 100, replace = TRUE),
                levels = c("low", "medium", "high"), ordered = TRUE)  # Ordinal variable
)

# Perform fuzzy clustering on categorical data with standard IB
result_cat <- IBmix(X = data_cat, ncl = 3, beta = 15, lambda = -1, nstart = 2, maxiter = 200)


plot(result_cat, type = "sizes") # Bar plot of cluster sizes (hardened assignments)
plot(result_cat, type = "info")  # Information-theoretic quantities plot
# Variable importance plot (hardened assignments)
plot(result_cat, type = "importance")
# Similarity plot
plot(result_cat, type = "similarity")
# Cluster membership plot
plot(result_cat, type = "membership")

# Simulated continuous data example
set.seed(123)
# Continuous data with 100 observations, 5 features
data_cont <- as.data.frame(matrix(rnorm(500), ncol = 5))

# Perform fuzzy clustering on continuous data with standard IB
result_cont <- IBmix(X = data_cont, ncl = 3, beta = 50, s = -1, nstart = 2)

# Print clustering results
print(result_cont) 

Methods for aibclust objects

Description

S3 methods available for aibclust objects, including extractors for the cluster assignments and model parameters, an information-metrics accessor, conversion methods to hclust and dendrogram, and diagnostic plotting.

Usage

## S3 method for class 'aibclust'
print(x, ...)

## S3 method for class 'aibclust'
summary(object, k = NULL, m = NULL, ...)

## S3 method for class 'summary.aibclust'
print(x, ...)

## S3 method for class 'aibclust'
plot(
  x,
  type = c("dendrogram", "info", "importance", "similarity"),
  X = NULL,
  ncl = NULL,
  color_by_type = TRUE,
  col = NULL,
  colorbar = TRUE,
  main = NULL,
  labels = TRUE,
  ...
)

## S3 method for class 'aibclust'
fitted(object, ncl, ...)

## S3 method for class 'aibclust'
coef(object, ...)

Arguments

Details

The following methods are available:

• print and summary: concise and detailed descriptions of the cluster hierarchy.

• fitted: extract the cluster partition at a requested number of clusters via the ncl argument.

• coef: extract the model's bandwidth hyperparameters (s, lambda).

• info_metrics: extract information-theoretic quantities. Optional ncl argument returns scalar values at the chosen cluster count.

• as.hclust and as.dendrogram: convert to standard hclust and dendrogram objects, enabling cutree() and dendrogram-based tools.

• plot: produce diagnostic plots (type = "dendrogram", "info", "importance", or "similarity").

See Also

AIBmix

Convert an aibclust object to hclust or dendrogram

Description

Enables use of standard hierarchical-clustering methods (e.g., cutree, as.dendrogram) on AIBmix output.

Usage

## S3 method for class 'aibclust'
as.hclust(x, ...)

## S3 method for class 'aibclust'
as.dendrogram(object, ...)

Arguments

Value

For as.hclust.aibclust, an object of class "hclust". For as.dendrogram.aibclust, an object of class "dendrogram".

Detect a knee/elbow in a monotone curve

Description

Identifies the point on a curve that lies at maximum perpendicular distance from the straight line connecting two endpoints. For information retention curves this is a common heuristic for choosing the number of clusters.

Usage

find_elbow(x, y, x_start, x_end)

Arguments

Details

The function expects x sorted in strictly ascending order, and y either non-decreasing or non-increasing throughout (no direction reversals). The endpoints x_start and x_end must be values present in x; the segment of the curve between them is searched for the elbow/knee.

Value

A list with components:

Examples

# Synthetic monotone-decreasing curve with a clear elbow
x <- 1:20
y <- 1 / (1 + exp(0.5 * (x - 5)))
elbow <- find_elbow(x, y, x_start = 1, x_end = 20)
elbow$x
plot(x, y, type = "b")
points(elbow$x, elbow$y, col = "red", pch = 19, cex = 1.5)

# Applied to an AIBmix information retention curve

fit <- AIBmix(iris[, -5])
find_elbow(seq_along(fit$info_ret), fit$info_ret,
           x_start = 1, x_end = 10)

Methods for gibclust objects

Description

S3 methods available for gibclust objects, including extractors for the cluster assignments and model parameters, an information-metrics accessor, a prediction method for new data, and diagnostic plotting.

Usage

## S3 method for class 'gibclust'
print(x, ...)

## S3 method for class 'gibclust'
summary(object, ...)

## S3 method for class 'summary.gibclust'
print(x, ...)

## S3 method for class 'gibclust'
plot(
  x,
  type = c("sizes", "info", "beta", "importance", "similarity", "membership"),
  X = NULL,
  color_by_type = TRUE,
  col = NULL,
  order_by_cluster = TRUE,
  groups = NULL,
  colorbar = TRUE,
  main = NULL,
  ...
)

## S3 method for class 'gibclust'
fitted(object, method = c("classes", "soft"), ...)

## S3 method for class 'gibclust'
coef(object, ...)

Arguments

Details

The following methods are available:

• print and summary: concise and detailed descriptions of the cluster solution.

• fitted: extract cluster assignments. The argument method = "classes" (default) returns hard cluster labels; method = "soft" returns the membership matrix (one-hot for DIBmix; fuzzy for IBmix and GIBmix).

• coef: extract the model's hyperparameters as a list, including bandwidths (s, lambda) and IB parameters (beta, alpha).

• info_metrics: extract information-theoretic quantities H(T), H(T \mid X), I(T; X), and I(T; Y).

• predict: assign new observations to clusters. For DIBmix fits, returns hard labels via the minimisation of Kullback–Leibler divergence; for IBmix and GIBmix fits, returns a soft membership matrix via the soft IB assignment rule.

• plot: produce diagnostic plots (type = "sizes", "info", "beta", "importance", "similarity", or "membership").

See Also

DIBmix, IBmix, GIBmix

Extract information-theoretic metrics from an IBclust fit

Description

Returns the entropy, conditional entropy, and mutual information quantities computed by the chosen Information Bottleneck variant. Methods are provided for gibclust, aibclust, and sibclust objects, returning a parallel set of quantities with class-specific extras.

Usage

info_metrics(object, ...)

## S3 method for class 'gibclust'
info_metrics(object, ...)

## S3 method for class 'sibclust'
info_metrics(object, ...)

## S3 method for class 'aibclust'
info_metrics(object, ncl = NULL, ...)

Arguments

Details

The shared quantities across both classes are:

• H_T: the entropy H(T) of the cluster assignment.

• H_T_X: the conditional entropy H(T \mid X) of the cluster assignment given the observation weights. This is 0 for hard partitions, since T is a deterministic function of X.

• I_T_X: the mutual information I(T; X) between the cluster assignment and the observation weights. Equals H_T for hard partitions.

• I_T_Y: the mutual information I(T; Y) between the cluster assignment and the joint distribution Y.

For gibclust and sibclust objects, each quantity is a single numeric value corresponding to the fitted partition. For aibclust objects, each quantity is a numeric vector indexed by the number of clusters m, from m = 1 (a single cluster) to m = n (all singletons). Supplying ncl extracts the scalar values at the chosen cut.

The aibclust output additionally includes:

• I_X_Y: the scalar baseline I(X; Y), the mutual information between observation weights and the joint distribution.

• info_ret: the proportion of baseline information retained, I(T_m; Y) / I(X; Y). Either a vector over m or a scalar at the requested cut.

Value

A named list of information-theoretic quantities. For gibclust and sibclust, contains H_T, H_T_X, I_T_X, I_T_Y. For aibclust, contains those four quantities plus I_X_Y and info_ret.

See Also

DIBmix, IBmix, GIBmix, AIBmix, sIBmix.

Examples

# gibclust: single values per quantity
fit_dib <- DIBmix(iris[, -5], ncl = 3, nstart = 5)
info_metrics(fit_dib)

# sibclust: single values per quantity
fit_sib <- sIBmix(iris[, -5], ncl = 3, nstart = 5)
info_metrics(fit_sib)

# aibclust: full vectors over cluster counts
fit_aib <- AIBmix(iris[, -5])
metrics_all <- info_metrics(fit_aib)
str(metrics_all)

# aibclust: scalar values at a specific cut
info_metrics(fit_aib, ncl = 3)

Predict cluster assignments for new observations

Description

Assigns new observations to clusters using a fitted gibclust model. For hard fits (DIBmix, alpha = 0), returns integer cluster labels based on the minimisation of the Kullback–Leibler divergence between the new observation's conditional distribution and each cluster conditional density. For soft fits (IBmix, GIBmix), returns the full membership matrix based on the (G)IB update, with the fitted \beta.

Usage

## S3 method for class 'gibclust'
predict(object, newdata = NULL, X = NULL, ...)

Arguments

Details

Prediction consists of three steps. First, the conditional distribution p(y \mid x_{\text{new}}) of each new observation over the training data is computed using the same kernel and bandwidths as the fit. Second, the cluster profiles q(y \mid t) are reconstructed from the training data and the stored cluster assignments. Third, the Kullback–Leibler divergence D_{KL}(p(y \mid x_{\text{new}}) \,||\, q(y \mid t)) is computed for each (new observation, cluster) pair. New observations are then assigned via argmin (DIBmix) or via the soft assignment rule q(t \mid x_{\text{new}}) \propto q(t) \exp(-\beta \, D_{KL}) (IBmix and GIBmix).

Continuous variables in newdata are scaled using the means and standard deviations of the training data X when object$scale is TRUE. Categorical variables are relabelled to match the encoding used at fit time. New observations containing categorical levels not seen in the training data will produce an error.

Value

For DIBmix fits, an integer vector of length nrow(newdata) giving the predicted cluster label for each new observation. For IBmix and GIBmix fits, a numeric matrix of dimension object$ncl by nrow(newdata) containing soft membership probabilities, with columns summing to 1.

See Also

DIBmix, IBmix, GIBmix, fitted.

Examples

# Hold-out prediction on the iris data
set.seed(1)
test_idx  <- sample(seq_len(nrow(iris)), 30)
train_idx <- setdiff(seq_len(nrow(iris)), test_idx)

fit <- DIBmix(iris[train_idx, -5], ncl = 3, nstart = 5, keep_data = TRUE)
preds <- predict(fit, newdata = iris[test_idx, -5])
table(preds, iris$Species[test_idx])

# Grid prediction for visualising the decision boundary
fit2 <- DIBmix(faithful, ncl = 3, nstart = 5, keep_data = TRUE)
grid <- expand.grid(
  eruptions = seq(min(faithful$eruptions), max(faithful$eruptions), length = 50),
  waiting = seq(min(faithful$waiting), max(faithful$waiting), length = 50)
)
grid_pred <- predict(fit2, newdata = grid)
plot(grid, col = grid_pred + 1, pch = 15, cex = 0.6)
points(faithful, pch = fit2$Cluster, col = "black")

Predict cluster assignments for new observations

Description

Assigns new observations to clusters using a fitted sibclust model. Each new observation is assigned to the cluster minimising the sequential merge cost, i.e. the (prior-weighted) loss of information incurred by absorbing the observation into the cluster, (p_x + p_t)\,\mathrm{JS}_\pi(p(Y \mid x_{\text{new}}), q(Y \mid t)), i.e. the same criterion optimised during fitting.

Usage

## S3 method for class 'sibclust'
predict(object, newdata = NULL, X = NULL, ...)

Arguments

Details

Continuous variables in newdata are scaled using the means and standard deviations of the training data X when object$scale is TRUE. Categorical variables are relabelled to match the encoding used at fit time; new observations with categorical levels unseen in the training data produce an error.

Value

An integer vector of length nrow(newdata) giving the predicted cluster label for each new observation.

See Also

sIBmix, fitted.

Sequential Information Bottleneck Clustering for Mixed-Type Data

Description

The sIBmix function implements the sequential Information Bottleneck (sIB) algorithm for clustering datasets containing continuous, categorical (nominal and ordinal), and mixed-type variables. Unlike the agglomerative method (AIBmix), sIB maintains exactly ncl clusters throughout and is guaranteed to converge to a local maximum of the mutual information I(Y; T) (Slonim et al. 2002).

Usage

sIBmix(
  X,
  ncl,
  randinit = NULL,
  s = -1,
  lambda = -1,
  scale = TRUE,
  maxiter = 100,
  nstart = 100,
  eps = 0,
  contkernel = "gaussian",
  nomkernel = "aitchisonaitken",
  ordkernel = "liracine",
  cat_first = FALSE,
  verbose = FALSE,
  keep_data = TRUE
)

Arguments

Details

The sIBmix function clusters data by maximising the information that the cluster assignments retain about the original variable distributions, while holding the number of clusters fixed at ncl. In contrast to the hierarchical agglomerative approach of AIBmix, the algorithm maintains a partition into exactly ncl clusters throughout. At each step a single observation is removed from its current cluster and re-assigned to the cluster for which the resulting loss of mutual information is smallest. Each such move can only increase the retained information, or leave it unchanged, so the procedure is guaranteed to converge to a local optimum. The criterion used to score each re-assignment is the same information-loss measure that underlies AIBmix, so the two methods optimise the same objective through different search strategies. Bandwidth parameters for categorical (nominal, ordinal) and continuous variables are adaptively determined if not provided. The algorithm is run from several random initial partitions (controlled by nstart) and the most informative solution is returned. The method is well-suited for datasets with mixed-type variables and integrates information from all variable types effectively.

See IBclust-package for details on the available kernel functions and their bandwidth parameters.

Value

An object of class sibclust. See sibclust-methods for the available S3 methods (print, summary, plot, fitted, coef, info_metrics, predict).

Author(s)

Efthymios Costa, Ioanna Papatsouma, Angelos Markos

References

Slonim N, Friedman N, Tishby N (2002). “Unsupervised document classification using sequential information maximization.” In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 129–136.

Examples

set.seed(123)
data_mix <- data.frame(
  cat_var = factor(sample(letters[1:3], 100, replace = TRUE)),
  ord_var = factor(sample(c("low", "medium", "high"), 100, replace = TRUE),
                   levels = c("low", "medium", "high"), ordered = TRUE),
  cont_var1 = rnorm(100),
  cont_var2 = runif(100)
)

# Hard partitional clustering with sequential IB
result <- sIBmix(X = data_mix, ncl = 3, nstart = 5)

# Print output and provide summary
print(result)
summary(result)
fitted(result)               # Hard cluster labels
coef(result)                 # Bandwidths
info_metrics(result)         # information-theoretic quantities

plot(result, type = "sizes")      # Plot of cluster sizes
plot(result, type = "info")       # Plot of information-theoretic quantities
plot(result, type = "importance") # Variable importance plot
plot(result, type = "similarity") # Similarity matrix plot

predict(result, newdata = data_mix[1:5, ])   # Predict integer labels for new data

Methods for sibclust objects

Description

S3 methods available for sibclust objects, including extractors for the cluster assignments and model parameters, an information-metrics accessor, a prediction method for new data, and diagnostic plotting.

Usage

## S3 method for class 'sibclust'
print(x, ...)

## S3 method for class 'sibclust'
summary(object, ...)

## S3 method for class 'summary.sibclust'
print(x, ...)

## S3 method for class 'sibclust'
plot(
  x,
  type = c("sizes", "info", "importance", "similarity"),
  X = NULL,
  color_by_type = TRUE,
  col = NULL,
  order_by_cluster = TRUE,
  colorbar = TRUE,
  main = NULL,
  ...
)

## S3 method for class 'sibclust'
fitted(object, method = c("classes", "soft"), ...)

## S3 method for class 'sibclust'
coef(object, ...)

Arguments

Details

The following methods are available:

• print and summary: concise and detailed descriptions of the cluster solution.

• fitted: extract cluster assignments. The argument method = "classes" (default) returns hard cluster labels; method = "soft" returns the equivalent one-hot membership matrix.

• coef: extract the model's bandwidth hyperparameters (s, lambda).

• info_metrics: extract information-theoretic quantities H(T), H(T \mid X), I(T; X), and I(T; Y).

• predict: assign new observations to clusters via the sequential merge-cost (Jensen–Shannon) rule.

• plot: produce diagnostic plots (type = "sizes", "info", "importance", or "similarity").

See Also

sIBmix