Policies.klUCBPlusPlus module

The improved kl-UCB++ policy, for one-parameter exponential distributions. Reference: [Menard & Garivier, ALT 2017](https://hal.inria.fr/hal-01475078)

Policies.klUCBPlusPlus.logplus(x)

..math:: log^+(x) := max(0, log(x)).

Policies.klUCBPlusPlus.g(t, T, K)

The exploration function g(t) (for t current time, T horizon, K nb arms), as defined in page 3 of the reference paper.

\[\begin{split}g(t, T, K) &:= \log^+(y (1 + \log^+(y)^2)),\\ y &:= \frac{T}{K t}.\end{split}\]

Policies.klUCBPlusPlus.g_vect(t, T, K)

The exploration function g(t) (for t current time, T horizon, K nb arms), as defined in page 3 of the reference paper, for numpy vectorized inputs.

\[\begin{split}g(t, T, K) &:= \log^+(y (1 + \log^+(y)^2)),\\ y &:= \frac{T}{K t}.\end{split}\]

class Policies.klUCBPlusPlus.klUCBPlusPlus(nbArms, horizon=None, tolerance=0.0001, klucb=CPUDispatcher(<function klucbBern>), c=1.0, lower=0.0, amplitude=1.0)

Bases: Policies.klUCB.klUCB

The improved kl-UCB++ policy, for one-parameter exponential distributions. Reference: [Menard & Garivier, ALT 2017](https://hal.inria.fr/hal-01475078)

__init__(nbArms, horizon=None, tolerance=0.0001, klucb=CPUDispatcher(<function klucbBern>), c=1.0, lower=0.0, amplitude=1.0)

New generic index policy.

• nbArms: the number of arms,

• lower, amplitude: lower value and known amplitude of the rewards.

horizon = None

Parameter \(T\) = known horizon of the experiment.

__str__()

-> str

computeIndex(arm)

Compute the current index, at time t and after \(N_k(t)\) pulls of arm k:

\[\begin{split}\hat{\mu}_k(t) &= \frac{X_k(t)}{N_k(t)}, \\ U_k(t) &= \sup\limits_{q \in [a, b]} \left\{ q : \mathrm{kl}(\hat{\mu}_k(t), q) \leq \frac{c g(N_k(t), T, K)}{N_k(t)} \right\},\\ I_k(t) &= U_k(t).\end{split}\]

If rewards are in \([a, b]\) (default to \([0, 1]\)) and \(\mathrm{kl}(x, y)\) is the Kullback-Leibler divergence between two distributions of means x and y (see Arms.kullback), and c is the parameter (default to 1), and where \(g(t, T, K)\) is this function:

\[\begin{split}g(t, T, K) &:= \log^+(y (1 + \log^+(y)^2)),\\ y &:= \frac{T}{K t}.\end{split}\]

computeAllIndex()

Compute the current indexes for all arms, in a vectorized manner.

__module__ = 'Policies.klUCBPlusPlus'