Calculate mutual information between groups of joint variates

This function calculates various entropic information measures between two grops of joint variates: the mutual information, the conditional entropies, and the entropies.

Usage

mutualinfo(
  Y1names,
  Y2names = NULL,
  X = NULL,
  learnt,
  tails = NULL,
  n = NULL,
  unit = "Sh",
  parallel = TRUE,
  verbose = FALSE
)

Arguments

Y1names: Character vector: first group of joint variates
Y2names: Character vector or NULL: second group of joint variates
X: Matrix or data.frame or NULL: values of some variates conditional on which we want the probabilities.
learnt: Either a character with the name of a directory or full path for an 'learnt.rds' object, or such an object itself.
tails: Named vector or list, or NULL (default). The names must match some or all of the variates in arguments X. For variates in this list, the probability conditional is understood in a semi-open interval sense: $X \le x$ or $X \ge x$, an so on. See analogous argument in Pr().
n: Integer or NULL (default): number of samples from which to approximately calculate the mutual information. Default as many as Monte Carlo samples in learnt.
unit: Either one of 'Sh' for shannon (default), 'Hart' for hartley, 'nat' for natural unit, or a positive real indicating the base of the logarithms to be used.
parallel: Logical or positive integer or cluster object. TRUE (default): use roughly half of available cores; FALSE: use serial computation; integer: use this many cores. It can also be a cluster object previously created with parallel::makeCluster(); in this case the parallel computation will use this object.
verbose: Logical, default FALSE: give messages about parallel processing?

Value

A list consisting of the following elements:

MI, a vector of value and accuracy: the mutual information between (joint) variates Y1names and (joint) variates Y2names.
CondEn12, CondEn21, vectors of value and accuracy: the conditional entropy of the first variate given the second, and vice versa.
En1, En2, vectors of value and accuracy: the (differential) entropies of the first and second variates.
MI.rGauss, a vector of value and accuracy: the absolute value of the Pearson correlation coefficient $r$ of a multivariate Gaussian distribution having mutual information MI; the two are related by $\mathrm{MI} = -\ln(1 - r^2)/2$. It may provide a vague intuition for the MI value for people more familiar with Pearson's correlation, but should be taken with a grain of salt.
unit, Y1names, Y1names: same as the input arguments, included for the user's convenience.

Details

If $Y_1$ and $Y_2$ are two variates, each of which can be a joint variate such as $Y_1 = (Y_{1,1}, Y_{1,2}, \dotsc)$, and $X$ a third, also possibly join, variate, then the mutual information $\mathit{MI}$ between $Y_1$ and $Y_2$, conditional on $X = x$, is given by $$\mathit{MI}(Y_1, Y_2 \vert X = x) \mathrel{:=} \sum_{y_1, y_2} \mathrm{Pr}(Y_1 = y_1, Y_2 = y_2 \vert X = x, \text{data}) \log_2\frac{ \mathrm{Pr}(Y_1 = y_1, Y_2 = y_2 \vert X = x, \text{data}) }{ \mathrm{Pr}(Y_1 = y_1 \vert X = x, \text{data}) \cdot \mathrm{Pr}(Y_2 = y_2 \vert X = x, \text{data}) } \, \mathrm{Sh} $$ an expression which can also be written in several other equivalent ways. It is an information-theoretic measure of association that is model-free, that is, does not depend on assumptions such as linearity, gaussianity, and similar. See vignette('mutualinfo') for discussion and example uses, and also the "References" section. If $Y_1, Y_2$ are jointly gaussian variates, then there is a mathematical correspondence between their mutual information and their Pearson correlation coefficient; see output MI.rGauss in the "Value" section.

The conditional entropy of $Y_1$ with respect to $Y_2$, conditional on $X = x$, is given by $$\mathit{CondEn12}(Y_1, Y_2 \vert X = x) \mathrel{:=} -\sum_{y_1, y_2} \mathrm{Pr}(Y_1 = y_1 \vert Y_2 = y_2, X = x, \text{data}) \log_2 \mathrm{Pr}(Y_1 = y_1 \vert Y_2 = y_2, X = x, \text{data}) \cdot \mathrm{Pr}(Y_2 = y_2 \vert X = x, \text{data}) \, \mathrm{Sh} $$

The (differential) entropy of $Y_1$, conditional on $X = x$, is given by $$\mathit{En1}(Y_1 \vert X = x) \mathrel{:=} -\sum_{y_1} \mathrm{Pr}(Y_1 = y_1 \vert X = x, \text{data}) \log_2 \mathrm{Pr}(Y_1 = y_1 \vert X = x, \text{data}) \, \mathrm{Sh} $$

see "References" section for discussions about entropy and conditional entropy.

The function mutualinfo() calculates the quantities above for the joint variates specified in the arguments Y1names and Y2names, conditional on the values of the variates specified in the data frame X. If X is omitted or NULL, then the posterior probabilities $\mathrm{Pr}(Y_1 | \text{data})$ etc. are used. Each variate in the argument X can be specified either as a point-value $X = x$ or as a left-open interval $X \le x$ or as a right-open interval $X \ge x$, through the argument tails.

The computation of these quantities is done via Monte Carlo integration, using the samples produced by the learn() function. The present function also output the numerical error associated with this computation.

Examples