Table of Contents¶
1 An example of a small Multi-Player simulation, with Centralized Algorithms
1.1 Creating the problem
1.1.1 Parameters for the simulation
1.1.2 Three MAB problems with Bernoulli arms
1.1.3 Some RL algorithms
1.2 Creating the EvaluatorMultiPlayers objects
1.3 Solving the problem
1.4 Plotting the results
1.4.1 First problem
1.4.2 Second problem
1.4.3 Third problem
1.4.4 Comparing their performances
An example of a small Multi-Player simulation, with Centralized Algorithms¶
First, be sure to be in the main folder, or to have SMPyBandits installed, and import EvaluatorMultiPlayers from Environment package:
[1]:
!pip install SMPyBandits watermark
%load_ext watermark
%watermark -v -m -p SMPyBandits -a "Lilian Besson"
Requirement already satisfied: SMPyBandits in ./venv3/lib/python3.6/site-packages (0.9.4)
Requirement already satisfied: watermark in ./venv3/lib/python3.6/site-packages (1.7.0)
Requirement already satisfied: numpy in ./venv3/lib/python3.6/site-packages (from SMPyBandits) (1.15.4)
Requirement already satisfied: scikit-learn in ./venv3/lib/python3.6/site-packages (from SMPyBandits) (0.20.0)
Requirement already satisfied: scikit-optimize in ./venv3/lib/python3.6/site-packages (from SMPyBandits) (0.5.2)
Requirement already satisfied: matplotlib>=2 in ./venv3/lib/python3.6/site-packages (from SMPyBandits) (3.0.2)
Requirement already satisfied: joblib in ./venv3/lib/python3.6/site-packages (from SMPyBandits) (0.13.0)
Requirement already satisfied: scipy>0.9 in ./venv3/lib/python3.6/site-packages (from SMPyBandits) (1.1.0)
Requirement already satisfied: seaborn in ./venv3/lib/python3.6/site-packages (from SMPyBandits) (0.9.0)
Requirement already satisfied: ipython in ./venv3/lib/python3.6/site-packages (from watermark) (7.1.1)
Requirement already satisfied: python-dateutil>=2.1 in ./venv3/lib/python3.6/site-packages (from matplotlib>=2->SMPyBandits) (2.7.5)
Requirement already satisfied: kiwisolver>=1.0.1 in ./venv3/lib/python3.6/site-packages (from matplotlib>=2->SMPyBandits) (1.0.1)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in ./venv3/lib/python3.6/site-packages (from matplotlib>=2->SMPyBandits) (2.3.0)
Requirement already satisfied: cycler>=0.10 in ./venv3/lib/python3.6/site-packages (from matplotlib>=2->SMPyBandits) (0.10.0)
Requirement already satisfied: pandas>=0.15.2 in ./venv3/lib/python3.6/site-packages (from seaborn->SMPyBandits) (0.23.4)
Requirement already satisfied: prompt-toolkit<2.1.0,>=2.0.0 in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (2.0.7)
Requirement already satisfied: decorator in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (4.3.0)
Requirement already satisfied: pickleshare in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (0.7.5)
Requirement already satisfied: jedi>=0.10 in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (0.13.1)
Requirement already satisfied: setuptools>=18.5 in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (40.6.2)
Requirement already satisfied: traitlets>=4.2 in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (4.3.2)
Requirement already satisfied: pexpect; sys_platform != "win32" in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (4.6.0)
Requirement already satisfied: pygments in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (2.2.0)
Requirement already satisfied: backcall in ./venv3/lib/python3.6/site-packages (from ipython->watermark) (0.1.0)
Requirement already satisfied: six>=1.5 in ./venv3/lib/python3.6/site-packages (from python-dateutil>=2.1->matplotlib>=2->SMPyBandits) (1.11.0)
Requirement already satisfied: pytz>=2011k in ./venv3/lib/python3.6/site-packages (from pandas>=0.15.2->seaborn->SMPyBandits) (2018.7)
Requirement already satisfied: wcwidth in ./venv3/lib/python3.6/site-packages (from prompt-toolkit<2.1.0,>=2.0.0->ipython->watermark) (0.1.7)
Requirement already satisfied: parso>=0.3.0 in ./venv3/lib/python3.6/site-packages (from jedi>=0.10->ipython->watermark) (0.3.1)
Requirement already satisfied: ipython-genutils in ./venv3/lib/python3.6/site-packages (from traitlets>=4.2->ipython->watermark) (0.2.0)
Requirement already satisfied: ptyprocess>=0.5 in ./venv3/lib/python3.6/site-packages (from pexpect; sys_platform != "win32"->ipython->watermark) (0.6.0)
Info: Using the Jupyter notebook version of the tqdm() decorator, tqdm_notebook() ...
Lilian Besson
CPython 3.6.6
IPython 7.1.1
SMPyBandits 0.9.4
compiler : GCC 8.0.1 20180414 (experimental) [trunk revision 259383
system : Linux
release : 4.15.0-38-generic
machine : x86_64
processor : x86_64
CPU cores : 4
interpreter: 64bit
[2]:
# Local imports
from SMPyBandits.Environment import EvaluatorMultiPlayers, tqdm
We also need arms, for instance Bernoulli-distributed arm:
[3]:
# Import arms
from SMPyBandits.Arms import Bernoulli
And finally we need some single-player and multi-player Reinforcement Learning algorithms:
[4]:
# Import algorithms
from SMPyBandits.Policies import *
from SMPyBandits.PoliciesMultiPlayers import *
[5]:
# Just improving the ?? in Jupyter. Thanks to https://nbviewer.jupyter.org/gist/minrk/7715212
from __future__ import print_function
from IPython.core import page
def myprint(s):
try:
print(s['text/plain'])
except (KeyError, TypeError):
print(s)
page.page = myprint
For instance, this imported the UCB algorithm:
[6]:
UCBalpha?
Init signature: UCBalpha(nbArms, alpha=4, lower=0.0, amplitude=1.0)
Docstring:
The UCB1 (UCB-alpha) index policy, modified to take a random permutation order for the initial exploration of each arm (reduce collisions in the multi-players setting).
Reference: [Auer et al. 02].
Init docstring:
New generic index policy.
- nbArms: the number of arms,
- lower, amplitude: lower value and known amplitude of the rewards.
File: /tmp/SMPyBandits/notebooks/venv3/lib/python3.6/site-packages/SMPyBandits/Policies/UCBalpha.py
Type: type
As well as the CentralizedMultiplePlay multi-player policy:
[7]:
CentralizedMultiplePlay?
Init signature: CentralizedMultiplePlay(nbPlayers, nbArms, playerAlgo, uniformAllocation=False, *args, **kwargs)
Docstring:
CentralizedMultiplePlay: a multi-player policy where ONE policy is used by a centralized agent; asking the policy to select nbPlayers arms at each step.
Init docstring:
- nbPlayers: number of players to create (in self._players).
- playerAlgo: class to use for every players.
- nbArms: number of arms, given as first argument to playerAlgo.
- uniformAllocation: Should the affectations of users always be uniform, or fixed when UCB indexes have converged? First choice is more fair, but linear nb of switches, second choice is not fair, but cst nb of switches.
- `*args`, `**kwargs`: arguments, named arguments, given to playerAlgo.
Examples:
>>> import sys; sys.path.insert(0, '..'); from Policies import *
>>> s = CentralizedMultiplePlay(2, 3, UCB)
>>> [ child.choice() for child in s.children ]
[2, 0]
- To get a list of usable players, use ``s.children``.
- Warning: ``s._players`` is for internal use ONLY!
File: /tmp/SMPyBandits/notebooks/venv3/lib/python3.6/site-packages/SMPyBandits/PoliciesMultiPlayers/CentralizedMultiplePlay.py
Type: type
We also need a collision model. The usual ones are defined in the CollisionModels package, and the only one we need is the classical one, where two or more colliding users don’t receive any rewards.
[8]:
# Collision Models
from SMPyBandits.Environment.CollisionModels import onlyUniqUserGetsReward
onlyUniqUserGetsReward?
Signature: onlyUniqUserGetsReward(t, arms, players, choices, rewards, pulls, collisions)
Docstring:
Simple collision model where only the players alone on one arm samples it and receives the reward.
- This is the default collision model, cf. [[Multi-Player Bandits Revisited, Lilian Besson and Emilie Kaufmann, 2017]](https://hal.inria.fr/hal-01629733).
- The numpy array 'choices' is increased according to the number of users who collided (it is NOT binary).
File: /tmp/SMPyBandits/notebooks/venv3/lib/python3.6/site-packages/SMPyBandits/Environment/CollisionModels.py
Type: function
Creating the problem¶
Parameters for the simulation¶
\(T = 10000\) is the time horizon,
\(N = 100\) is the number of repetitions (should be larger to have consistent results),
\(M = 2\) is the number of players,
N_JOBS = 4is the number of cores used to parallelize the code.
[9]:
HORIZON = 10000
REPETITIONS = 100
NB_PLAYERS = 2
N_JOBS = 4
collisionModel = onlyUniqUserGetsReward
Three MAB problems with Bernoulli arms¶
We consider in this example \(3\) problems, with Bernoulli arms, of different means.
The first problem is very easy, with two good arms and three arms, with a fixed gap \(\Delta = \max_{\mu_i \neq \mu_j}(\mu_{i} - \mu_{j}) = 0.1\).
The second problem is as easier, with a larger gap.
Third problem is harder, with a smaller gap, and a very large difference between the two optimal arms and the suboptimal arms.
Note: right now, the multi-environments evaluator does not work well for MP policies, if there is a number different of arms in the scenarios. So I use the same number of arms in all the problems.
[10]:
ENVIRONMENTS = [ # 1) Bernoulli arms
{ # Scenario 1 from [Komiyama, Honda, Nakagawa, 2016, arXiv 1506.00779]
"arm_type": Bernoulli,
"params": [0.3, 0.4, 0.5, 0.6, 0.7]
},
{ # Classical scenario
"arm_type": Bernoulli,
"params": [0.1, 0.3, 0.5, 0.7, 0.9]
},
{ # Harder scenario
"arm_type": Bernoulli,
"params": [0.005, 0.01, 0.015, 0.84, 0.85]
}
]
Some RL algorithms¶
We will compare Thompson Sampling against \(\mathrm{UCB}_1\), using two different centralized policy:
CentralizedMultiplePlayis the naive use of a Bandit algorithm for Multi-Player decision making: at every step, the internal decision making process is used to determine not \(1\) arm but \(M\) to sample. For UCB-like algorithm, the decision making is based on a \(\arg\max\) on UCB-like indexes, usually of the form \(I_j(t) = X_j(t) + B_j(t)\), where \(X_j(t) = \hat{\mu_j}(t) = \sum_{\tau \leq t} r_j(\tau) / N_j(t)\) is the empirical mean of arm \(j\), and \(B_j(t)\) is a bias term, of the form \(B_j(t) = \sqrt{\frac{\alpha \log(t)}{2 N_j(t)}}\).CentralizedIMPis very similar, but instead of following the internal decision making for all the decisions, the system uses just the empirical means \(X_j(t)\) to determine \(M-1\) arms to sample, and the bias-corrected term (i.e., the internal decision making, can be sampling from a Bayesian posterior for instance) is used just for one decision. It is an heuristic, proposed in [Komiyama, Honda, Nakagawa, 2016].
[12]:
nbArms = len(ENVIRONMENTS[0]['params'])
assert all(len(env['params']) == nbArms for env in ENVIRONMENTS), "Error: not yet support if different environments have different nb of arms"
nbArms
SUCCESSIVE_PLAYERS = [
CentralizedMultiplePlay(NB_PLAYERS, nbArms, UCBalpha, alpha=1).children,
CentralizedIMP(NB_PLAYERS, nbArms, UCBalpha, alpha=1).children,
CentralizedMultiplePlay(NB_PLAYERS, nbArms, Thompson).children,
CentralizedIMP(NB_PLAYERS, nbArms, Thompson).children
]
SUCCESSIVE_PLAYERS
[12]:
5
- One new child, of index 0, and class #1<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
- One new child, of index 1, and class #2<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
- One new child, of index 0, and class #1<CentralizedIMP(UCB($\alpha=1$))> ...
- One new child, of index 1, and class #2<CentralizedIMP(UCB($\alpha=1$))> ...
- One new child, of index 0, and class #1<CentralizedMultiplePlay(Thompson)> ...
- One new child, of index 1, and class #2<CentralizedMultiplePlay(Thompson)> ...
- One new child, of index 0, and class #1<CentralizedIMP(Thompson)> ...
- One new child, of index 1, and class #2<CentralizedIMP(Thompson)> ...
[12]:
[[CentralizedMultiplePlay(UCB($\alpha=1$)),
CentralizedMultiplePlay(UCB($\alpha=1$))],
[CentralizedIMP(UCB($\alpha=1$)), CentralizedIMP(UCB($\alpha=1$))],
[CentralizedMultiplePlay(Thompson), CentralizedMultiplePlay(Thompson)],
[CentralizedIMP(Thompson), CentralizedIMP(Thompson)]]
The mother class in this case does all the job here, as we use centralized learning.
[13]:
OnePlayer = SUCCESSIVE_PLAYERS[0][0]
OnePlayer.nbArms
OneMother = OnePlayer.mother
OneMother
OneMother.nbArms
[13]:
5
[13]:
<SMPyBandits.PoliciesMultiPlayers.CentralizedMultiplePlay.CentralizedMultiplePlay at 0x7fc67216f908>
[13]:
5
Complete configuration for the problem:
[14]:
configuration = {
# --- Duration of the experiment
"horizon": HORIZON,
# --- Number of repetition of the experiment (to have an average)
"repetitions": REPETITIONS,
# --- Parameters for the use of joblib.Parallel
"n_jobs": N_JOBS, # = nb of CPU cores
"verbosity": 6, # Max joblib verbosity
# --- Collision model
"collisionModel": onlyUniqUserGetsReward,
# --- Arms
"environment": ENVIRONMENTS,
# --- Algorithms
"successive_players": SUCCESSIVE_PLAYERS,
}
Creating the EvaluatorMultiPlayers objects¶
We will need to create several objects, as the simulation first runs one policy against each environment, and then aggregate them to compare them.
[15]:
%%time
N_players = len(configuration["successive_players"])
# List to keep all the EvaluatorMultiPlayers objects
evs = [None] * N_players
evaluators = [[None] * N_players] * len(configuration["environment"])
for playersId, players in tqdm(enumerate(configuration["successive_players"]), desc="Creating"):
print("\n\nConsidering the list of players :\n", players)
conf = configuration.copy()
conf['players'] = players
evs[playersId] = EvaluatorMultiPlayers(conf)
Considering the list of players :
[CentralizedMultiplePlay(UCB($\alpha=1$)), CentralizedMultiplePlay(UCB($\alpha=1$))]
Number of players in the multi-players game: 2
Time horizon: 10000
Number of repetitions: 100
Sampling rate for plotting, delta_t_plot: 1
Number of jobs for parallelization: 4
Using collision model onlyUniqUserGetsReward (function <function onlyUniqUserGetsReward at 0x7fc672469bf8>).
More details:
Simple collision model where only the players alone on one arm samples it and receives the reward.
- This is the default collision model, cf. [[Multi-Player Bandits Revisited, Lilian Besson and Emilie Kaufmann, 2017]](https://hal.inria.fr/hal-01629733).
- The numpy array 'choices' is increased according to the number of users who collided (it is NOT binary).
Using accurate regrets and last regrets ? True
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.3, 0.4, 0.5, 0.6, 0.7]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.3, 0.4, 0.5, 0.6, 0.7]
- with 'arms' = [B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)]
- with 'means' = [0.3 0.4 0.5 0.6 0.7]
- with 'nbArms' = 5
- with 'maxArm' = 0.7
- with 'minArm' = 0.3
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- with 'arms' represented as: $[B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)^*]$
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.1, 0.3, 0.5, 0.7, 0.9]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.1, 0.3, 0.5, 0.7, 0.9]
- with 'arms' = [B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)]
- with 'means' = [0.1 0.3 0.5 0.7 0.9]
- with 'nbArms' = 5
- with 'maxArm' = 0.9
- with 'minArm' = 0.1
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- with 'arms' represented as: $[B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)^*]$
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.005, 0.01, 0.015, 0.84, 0.85]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.005, 0.01, 0.015, 0.84, 0.85]
- with 'arms' = [B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)]
- with 'means' = [0.005 0.01 0.015 0.84 0.85 ]
- with 'nbArms' = 5
- with 'maxArm' = 0.85
- with 'minArm' = 0.005
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- with 'arms' represented as: $[B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)^*]$
Number of environments to try: 3
Considering the list of players :
[CentralizedIMP(UCB($\alpha=1$)), CentralizedIMP(UCB($\alpha=1$))]
Number of players in the multi-players game: 2
Time horizon: 10000
Number of repetitions: 100
Sampling rate for plotting, delta_t_plot: 1
Number of jobs for parallelization: 4
Using collision model onlyUniqUserGetsReward (function <function onlyUniqUserGetsReward at 0x7fc672469bf8>).
More details:
Simple collision model where only the players alone on one arm samples it and receives the reward.
- This is the default collision model, cf. [[Multi-Player Bandits Revisited, Lilian Besson and Emilie Kaufmann, 2017]](https://hal.inria.fr/hal-01629733).
- The numpy array 'choices' is increased according to the number of users who collided (it is NOT binary).
Using accurate regrets and last regrets ? True
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.3, 0.4, 0.5, 0.6, 0.7]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.3, 0.4, 0.5, 0.6, 0.7]
- with 'arms' = [B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)]
- with 'means' = [0.3 0.4 0.5 0.6 0.7]
- with 'nbArms' = 5
- with 'maxArm' = 0.7
- with 'minArm' = 0.3
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- with 'arms' represented as: $[B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)^*]$
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.1, 0.3, 0.5, 0.7, 0.9]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.1, 0.3, 0.5, 0.7, 0.9]
- with 'arms' = [B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)]
- with 'means' = [0.1 0.3 0.5 0.7 0.9]
- with 'nbArms' = 5
- with 'maxArm' = 0.9
- with 'minArm' = 0.1
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- with 'arms' represented as: $[B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)^*]$
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.005, 0.01, 0.015, 0.84, 0.85]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.005, 0.01, 0.015, 0.84, 0.85]
- with 'arms' = [B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)]
- with 'means' = [0.005 0.01 0.015 0.84 0.85 ]
- with 'nbArms' = 5
- with 'maxArm' = 0.85
- with 'minArm' = 0.005
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- with 'arms' represented as: $[B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)^*]$
Number of environments to try: 3
Considering the list of players :
[CentralizedMultiplePlay(Thompson), CentralizedMultiplePlay(Thompson)]
Number of players in the multi-players game: 2
Time horizon: 10000
Number of repetitions: 100
Sampling rate for plotting, delta_t_plot: 1
Number of jobs for parallelization: 4
Using collision model onlyUniqUserGetsReward (function <function onlyUniqUserGetsReward at 0x7fc672469bf8>).
More details:
Simple collision model where only the players alone on one arm samples it and receives the reward.
- This is the default collision model, cf. [[Multi-Player Bandits Revisited, Lilian Besson and Emilie Kaufmann, 2017]](https://hal.inria.fr/hal-01629733).
- The numpy array 'choices' is increased according to the number of users who collided (it is NOT binary).
Using accurate regrets and last regrets ? True
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.3, 0.4, 0.5, 0.6, 0.7]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.3, 0.4, 0.5, 0.6, 0.7]
- with 'arms' = [B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)]
- with 'means' = [0.3 0.4 0.5 0.6 0.7]
- with 'nbArms' = 5
- with 'maxArm' = 0.7
- with 'minArm' = 0.3
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- with 'arms' represented as: $[B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)^*]$
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.1, 0.3, 0.5, 0.7, 0.9]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.1, 0.3, 0.5, 0.7, 0.9]
- with 'arms' = [B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)]
- with 'means' = [0.1 0.3 0.5 0.7 0.9]
- with 'nbArms' = 5
- with 'maxArm' = 0.9
- with 'minArm' = 0.1
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- with 'arms' represented as: $[B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)^*]$
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.005, 0.01, 0.015, 0.84, 0.85]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.005, 0.01, 0.015, 0.84, 0.85]
- with 'arms' = [B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)]
- with 'means' = [0.005 0.01 0.015 0.84 0.85 ]
- with 'nbArms' = 5
- with 'maxArm' = 0.85
- with 'minArm' = 0.005
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- with 'arms' represented as: $[B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)^*]$
Number of environments to try: 3
Considering the list of players :
[CentralizedIMP(Thompson), CentralizedIMP(Thompson)]
Number of players in the multi-players game: 2
Time horizon: 10000
Number of repetitions: 100
Sampling rate for plotting, delta_t_plot: 1
Number of jobs for parallelization: 4
Using collision model onlyUniqUserGetsReward (function <function onlyUniqUserGetsReward at 0x7fc672469bf8>).
More details:
Simple collision model where only the players alone on one arm samples it and receives the reward.
- This is the default collision model, cf. [[Multi-Player Bandits Revisited, Lilian Besson and Emilie Kaufmann, 2017]](https://hal.inria.fr/hal-01629733).
- The numpy array 'choices' is increased according to the number of users who collided (it is NOT binary).
Using accurate regrets and last regrets ? True
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.3, 0.4, 0.5, 0.6, 0.7]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.3, 0.4, 0.5, 0.6, 0.7]
- with 'arms' = [B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)]
- with 'means' = [0.3 0.4 0.5 0.6 0.7]
- with 'nbArms' = 5
- with 'maxArm' = 0.7
- with 'minArm' = 0.3
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- with 'arms' represented as: $[B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)^*]$
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.1, 0.3, 0.5, 0.7, 0.9]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.1, 0.3, 0.5, 0.7, 0.9]
- with 'arms' = [B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)]
- with 'means' = [0.1 0.3 0.5 0.7 0.9]
- with 'nbArms' = 5
- with 'maxArm' = 0.9
- with 'minArm' = 0.1
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- with 'arms' represented as: $[B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)^*]$
Creating a new MAB problem ...
Reading arms of this MAB problem from a dictionnary 'configuration' = {'arm_type': <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>, 'params': [0.005, 0.01, 0.015, 0.84, 0.85]} ...
- with 'arm_type' = <class 'SMPyBandits.Arms.Bernoulli.Bernoulli'>
- with 'params' = [0.005, 0.01, 0.015, 0.84, 0.85]
- with 'arms' = [B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)]
- with 'means' = [0.005 0.01 0.015 0.84 0.85 ]
- with 'nbArms' = 5
- with 'maxArm' = 0.85
- with 'minArm' = 0.005
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- with 'arms' represented as: $[B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)^*]$
Number of environments to try: 3
CPU times: user 170 ms, sys: 20.5 ms, total: 191 ms
Wall time: 194 ms
Solving the problem¶
Now we can simulate the \(2\) environments, for the successive policies. That part can take some time.
[16]:
%%time
for playersId, evaluation in tqdm(enumerate(evs), desc="Policies"):
for envId, env in tqdm(enumerate(evaluation.envs), desc="Problems"):
# Evaluate just that env
evaluation.startOneEnv(envId, env)
# Storing it after simulation is done
evaluators[envId][playersId] = evaluation
Evaluating environment: MAB(nbArms: 5, arms: [B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)], minArm: 0.3, maxArm: 0.7)
- Adding player # 1 = #1<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
Using this already created player 'player' = #1<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
- Adding player # 2 = #2<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
Using this already created player 'player' = #2<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 7.2s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 27.5s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 1.4min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)], minArm: 0.1, maxArm: 0.9)
- Adding player # 1 = #1<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
Using this already created player 'player' = #1<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
- Adding player # 2 = #2<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
Using this already created player 'player' = #2<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 10.9s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 47.0s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 1.7min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)], minArm: 0.005, maxArm: 0.85)
- Adding player # 1 = #1<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
Using this already created player 'player' = #1<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
- Adding player # 2 = #2<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
Using this already created player 'player' = #2<CentralizedMultiplePlay(UCB($\alpha=1$))> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 7.7s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 55.0s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 1.8min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)], minArm: 0.3, maxArm: 0.7)
- Adding player # 1 = #1<CentralizedIMP(UCB($\alpha=1$))> ...
Using this already created player 'player' = #1<CentralizedIMP(UCB($\alpha=1$))> ...
- Adding player # 2 = #2<CentralizedIMP(UCB($\alpha=1$))> ...
Using this already created player 'player' = #2<CentralizedIMP(UCB($\alpha=1$))> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 19.6s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 1.5min
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 3.6min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)], minArm: 0.1, maxArm: 0.9)
- Adding player # 1 = #1<CentralizedIMP(UCB($\alpha=1$))> ...
Using this already created player 'player' = #1<CentralizedIMP(UCB($\alpha=1$))> ...
- Adding player # 2 = #2<CentralizedIMP(UCB($\alpha=1$))> ...
Using this already created player 'player' = #2<CentralizedIMP(UCB($\alpha=1$))> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 23.3s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 1.6min
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 3.7min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)], minArm: 0.005, maxArm: 0.85)
- Adding player # 1 = #1<CentralizedIMP(UCB($\alpha=1$))> ...
Using this already created player 'player' = #1<CentralizedIMP(UCB($\alpha=1$))> ...
- Adding player # 2 = #2<CentralizedIMP(UCB($\alpha=1$))> ...
Using this already created player 'player' = #2<CentralizedIMP(UCB($\alpha=1$))> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 16.0s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 1.6min
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 3.8min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)], minArm: 0.3, maxArm: 0.7)
- Adding player # 1 = #1<CentralizedMultiplePlay(Thompson)> ...
Using this already created player 'player' = #1<CentralizedMultiplePlay(Thompson)> ...
- Adding player # 2 = #2<CentralizedMultiplePlay(Thompson)> ...
Using this already created player 'player' = #2<CentralizedMultiplePlay(Thompson)> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 7.2s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 47.9s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 1.7min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)], minArm: 0.1, maxArm: 0.9)
- Adding player # 1 = #1<CentralizedMultiplePlay(Thompson)> ...
Using this already created player 'player' = #1<CentralizedMultiplePlay(Thompson)> ...
- Adding player # 2 = #2<CentralizedMultiplePlay(Thompson)> ...
Using this already created player 'player' = #2<CentralizedMultiplePlay(Thompson)> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 7.9s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 45.6s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 1.7min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)], minArm: 0.005, maxArm: 0.85)
- Adding player # 1 = #1<CentralizedMultiplePlay(Thompson)> ...
Using this already created player 'player' = #1<CentralizedMultiplePlay(Thompson)> ...
- Adding player # 2 = #2<CentralizedMultiplePlay(Thompson)> ...
Using this already created player 'player' = #2<CentralizedMultiplePlay(Thompson)> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 8.1s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 46.2s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 1.7min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.3), B(0.4), B(0.5), B(0.6), B(0.7)], minArm: 0.3, maxArm: 0.7)
- Adding player # 1 = #1<CentralizedIMP(Thompson)> ...
Using this already created player 'player' = #1<CentralizedIMP(Thompson)> ...
- Adding player # 2 = #2<CentralizedIMP(Thompson)> ...
Using this already created player 'player' = #2<CentralizedIMP(Thompson)> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 7.7s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 46.7s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 1.8min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.1), B(0.3), B(0.5), B(0.7), B(0.9)], minArm: 0.1, maxArm: 0.9)
- Adding player # 1 = #1<CentralizedIMP(Thompson)> ...
Using this already created player 'player' = #1<CentralizedIMP(Thompson)> ...
- Adding player # 2 = #2<CentralizedIMP(Thompson)> ...
Using this already created player 'player' = #2<CentralizedIMP(Thompson)> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 6.4s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 32.1s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 1.0min finished
Evaluating environment: MAB(nbArms: 5, arms: [B(0.005), B(0.01), B(0.015), B(0.84), B(0.85)], minArm: 0.005, maxArm: 0.85)
- Adding player # 1 = #1<CentralizedIMP(Thompson)> ...
Using this already created player 'player' = #1<CentralizedIMP(Thompson)> ...
- Adding player # 2 = #2<CentralizedIMP(Thompson)> ...
Using this already created player 'player' = #2<CentralizedIMP(Thompson)> ...
[Parallel(n_jobs=4)]: Using backend LokyBackend with 4 concurrent workers.
[Parallel(n_jobs=4)]: Done 5 tasks | elapsed: 3.8s
[Parallel(n_jobs=4)]: Done 42 tasks | elapsed: 26.9s
[Parallel(n_jobs=4)]: Done 100 out of 100 | elapsed: 57.2s finished
CPU times: user 2min 45s, sys: 913 ms, total: 2min 46s
Wall time: 27min 44s
Plotting the results¶
And finally, visualize them, with the plotting method of a EvaluatorMultiPlayers object:
[24]:
def plotAll(evaluation, envId):
evaluation.printFinalRanking(envId)
# Rewards
evaluation.plotRewards(envId)
# Fairness
#evaluation.plotFairness(envId, fairness="STD")
# Centralized regret
evaluation.plotRegretCentralized(envId, subTerms=True)
#evaluation.plotRegretCentralized(envId, semilogx=True, subTerms=True)
# Number of switches
#evaluation.plotNbSwitchs(envId, cumulated=False)
evaluation.plotNbSwitchs(envId, cumulated=True)
# Frequency of selection of the best arms
evaluation.plotBestArmPulls(envId)
# Number of collisions - not for Centralized* policies
#evaluation.plotNbCollisions(envId, cumulated=False)
#evaluation.plotNbCollisions(envId, cumulated=True)
# Frequency of collision in each arm
#evaluation.plotFrequencyCollisions(envId, piechart=True)
[25]:
import matplotlib as mpl
mpl.rcParams['figure.figsize'] = (12.4, 7)
First problem¶
\(\mu = [0.3, 0.4, 0.5, 0.6, 0.7]\) was an easy Bernoulli problem.
[26]:
for playersId in tqdm(range(len(evs)), desc="Policies"):
evaluation = evaluators[0][playersId]
plotAll(evaluation, 0)
Final ranking for this environment # 0 : CentralizedMultiplePlay(UCB($\alpha=1$)) ...
- Player # 2 / 2, CentralizedMultiplePlay(UCB($\alpha=1$)) was ranked 1 / 2 for this simulation (last rewards = 6516.6).
- Player # 1 / 2, CentralizedMultiplePlay(UCB($\alpha=1$)) was ranked 2 / 2 for this simulation (last rewards = 6346.1).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 9 ...
- For 2 players, our lower bound gave = 18 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 12.1 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- [Anandtharam et al] centralized lower-bound = 9,
- [Anandkumar et al] decentralized lower-bound = 12.1
- Our better (larger) decentralized lower-bound = 18,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 0 : CentralizedIMP(UCB($\alpha=1$)) ...
- Player # 2 / 2, CentralizedIMP(UCB($\alpha=1$)) was ranked 1 / 2 for this simulation (last rewards = 6464.3).
- Player # 1 / 2, CentralizedIMP(UCB($\alpha=1$)) was ranked 2 / 2 for this simulation (last rewards = 6400.6).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 9 ...
- For 2 players, our lower bound gave = 18 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 12.1 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- [Anandtharam et al] centralized lower-bound = 9,
- [Anandkumar et al] decentralized lower-bound = 12.1
- Our better (larger) decentralized lower-bound = 18,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 0 : CentralizedMultiplePlay(Thompson) ...
- Player # 1 / 2, CentralizedMultiplePlay(Thompson) was ranked 1 / 2 for this simulation (last rewards = 6495.3).
- Player # 2 / 2, CentralizedMultiplePlay(Thompson) was ranked 2 / 2 for this simulation (last rewards = 6394).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 9 ...
- For 2 players, our lower bound gave = 18 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 12.1 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- [Anandtharam et al] centralized lower-bound = 9,
- [Anandkumar et al] decentralized lower-bound = 12.1
- Our better (larger) decentralized lower-bound = 18,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 0 : CentralizedIMP(Thompson) ...
- Player # 2 / 2, CentralizedIMP(Thompson) was ranked 1 / 2 for this simulation (last rewards = 6550.7).
- Player # 1 / 2, CentralizedIMP(Thompson) was ranked 2 / 2 for this simulation (last rewards = 6346.3).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 9 ...
- For 2 players, our lower bound gave = 18 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 12.1 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- [Anandtharam et al] centralized lower-bound = 9,
- [Anandkumar et al] decentralized lower-bound = 12.1
- Our better (larger) decentralized lower-bound = 18,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Second problem¶
\(\mu = [0.1, 0.3, 0.5, 0.7, 0.9]\) was an easier Bernoulli problem, with larger gap \(\Delta = 0.2\).
[27]:
for playersId in tqdm(range(len(evs)), desc="Policies"):
evaluation = evaluators[1][playersId]
plotAll(evaluation, 1)
Final ranking for this environment # 1 : CentralizedMultiplePlay(UCB($\alpha=1$)) ...
- Player # 1 / 2, CentralizedMultiplePlay(UCB($\alpha=1$)) was ranked 1 / 2 for this simulation (last rewards = 8059.3).
- Player # 2 / 2, CentralizedMultiplePlay(UCB($\alpha=1$)) was ranked 2 / 2 for this simulation (last rewards = 7817.7).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 4.23 ...
- For 2 players, our lower bound gave = 8.46 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 5.35 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- [Anandtharam et al] centralized lower-bound = 4.23,
- [Anandkumar et al] decentralized lower-bound = 5.35
- Our better (larger) decentralized lower-bound = 8.46,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 1 : CentralizedIMP(UCB($\alpha=1$)) ...
- Player # 1 / 2, CentralizedIMP(UCB($\alpha=1$)) was ranked 1 / 2 for this simulation (last rewards = 8184.8).
- Player # 2 / 2, CentralizedIMP(UCB($\alpha=1$)) was ranked 2 / 2 for this simulation (last rewards = 7700).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 4.23 ...
- For 2 players, our lower bound gave = 8.46 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 5.35 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- [Anandtharam et al] centralized lower-bound = 4.23,
- [Anandkumar et al] decentralized lower-bound = 5.35
- Our better (larger) decentralized lower-bound = 8.46,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 1 : CentralizedMultiplePlay(Thompson) ...
- Player # 1 / 2, CentralizedMultiplePlay(Thompson) was ranked 1 / 2 for this simulation (last rewards = 8084.4).
- Player # 2 / 2, CentralizedMultiplePlay(Thompson) was ranked 2 / 2 for this simulation (last rewards = 7815.1).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 4.23 ...
- For 2 players, our lower bound gave = 8.46 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 5.35 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- [Anandtharam et al] centralized lower-bound = 4.23,
- [Anandkumar et al] decentralized lower-bound = 5.35
- Our better (larger) decentralized lower-bound = 8.46,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 1 : CentralizedIMP(Thompson) ...
- Player # 1 / 2, CentralizedIMP(Thompson) was ranked 1 / 2 for this simulation (last rewards = 8008.4).
- Player # 2 / 2, CentralizedIMP(Thompson) was ranked 2 / 2 for this simulation (last rewards = 7885.5).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 4.23 ...
- For 2 players, our lower bound gave = 8.46 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 5.35 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- [Anandtharam et al] centralized lower-bound = 4.23,
- [Anandkumar et al] decentralized lower-bound = 5.35
- Our better (larger) decentralized lower-bound = 8.46,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Third problem¶
\(\mu = [0.005, 0.01, 0.015, 0.84, 0.85]\) is an harder Bernoulli problem, as there is a huge gap between suboptimal and optimal arms.
[28]:
for playersId in tqdm(range(len(evs)), desc="Policies"):
evaluation = evaluators[2][playersId]
plotAll(evaluation, 2)
Final ranking for this environment # 2 : CentralizedMultiplePlay(UCB($\alpha=1$)) ...
- Player # 2 / 2, CentralizedMultiplePlay(UCB($\alpha=1$)) was ranked 1 / 2 for this simulation (last rewards = 8399.9).
- Player # 1 / 2, CentralizedMultiplePlay(UCB($\alpha=1$)) was ranked 2 / 2 for this simulation (last rewards = 8392.3).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 1.41 ...
- For 2 players, our lower bound gave = 2.83 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 2.78 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- [Anandtharam et al] centralized lower-bound = 1.41,
- [Anandkumar et al] decentralized lower-bound = 2.78
- Our better (larger) decentralized lower-bound = 2.83,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 2 : CentralizedIMP(UCB($\alpha=1$)) ...
- Player # 2 / 2, CentralizedIMP(UCB($\alpha=1$)) was ranked 1 / 2 for this simulation (last rewards = 8403.6).
- Player # 1 / 2, CentralizedIMP(UCB($\alpha=1$)) was ranked 2 / 2 for this simulation (last rewards = 8387.9).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 1.41 ...
- For 2 players, our lower bound gave = 2.83 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 2.78 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- [Anandtharam et al] centralized lower-bound = 1.41,
- [Anandkumar et al] decentralized lower-bound = 2.78
- Our better (larger) decentralized lower-bound = 2.83,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 2 : CentralizedMultiplePlay(Thompson) ...
- Player # 2 / 2, CentralizedMultiplePlay(Thompson) was ranked 1 / 2 for this simulation (last rewards = 8414.9).
- Player # 1 / 2, CentralizedMultiplePlay(Thompson) was ranked 2 / 2 for this simulation (last rewards = 8391).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 1.41 ...
- For 2 players, our lower bound gave = 2.83 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 2.78 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- [Anandtharam et al] centralized lower-bound = 1.41,
- [Anandkumar et al] decentralized lower-bound = 2.78
- Our better (larger) decentralized lower-bound = 2.83,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Final ranking for this environment # 2 : CentralizedIMP(Thompson) ...
- Player # 2 / 2, CentralizedIMP(Thompson) was ranked 1 / 2 for this simulation (last rewards = 8404.6).
- Player # 1 / 2, CentralizedIMP(Thompson) was ranked 2 / 2 for this simulation (last rewards = 8400.8).
Warning: forcing to use putatright = False because there is 2 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 1.41 ...
- For 2 players, our lower bound gave = 2.83 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 2.78 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- [Anandtharam et al] centralized lower-bound = 1.41,
- [Anandkumar et al] decentralized lower-bound = 2.78
- Our better (larger) decentralized lower-bound = 2.83,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Warning: forcing to use putatright = False because there is 2 items in the legend.
Comparing their performances¶
[29]:
def plotCombined(e0, eothers, envId):
# Centralized regret
e0.plotRegretCentralized(envId, evaluators=eothers)
# Fairness
e0.plotFairness(envId, fairness="STD", evaluators=eothers)
# Number of switches
e0.plotNbSwitchsCentralized(envId, cumulated=True, evaluators=eothers)
# Number of collisions - not for Centralized* policies
#e0.plotNbCollisions(envId, cumulated=True, evaluators=eothers)
[30]:
N = len(configuration["environment"])
for envId, env in enumerate(configuration["environment"]):
e0, eothers = evaluators[envId][0], evaluators[envId][1:]
plotCombined(e0, eothers, envId)
- For 2 players, Anandtharam et al. centralized lower-bound gave = 9 ...
- For 2 players, our lower bound gave = 18 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 12.1 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 9.46 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 60.00% ...
- [Anandtharam et al] centralized lower-bound = 9,
- [Anandkumar et al] decentralized lower-bound = 12.1
- Our better (larger) decentralized lower-bound = 18,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 4 items in the legend.
Warning: forcing to use putatright = False because there is 4 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 4.23 ...
- For 2 players, our lower bound gave = 8.46 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 5.35 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 3.12 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 40.00% ...
- [Anandtharam et al] centralized lower-bound = 4.23,
- [Anandkumar et al] decentralized lower-bound = 5.35
- Our better (larger) decentralized lower-bound = 8.46,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 4 items in the legend.
Warning: forcing to use putatright = False because there is 4 items in the legend.
- For 2 players, Anandtharam et al. centralized lower-bound gave = 1.41 ...
- For 2 players, our lower bound gave = 2.83 ...
- For 2 players, the initial lower bound in Theorem 6 from [Anandkumar et al., 2010] gave = 2.78 ...
This MAB problem has:
- a [Lai & Robbins] complexity constant C(mu) = 27.3 for 1-player problem ...
- a Optimal Arm Identification factor H_OI(mu) = 29.40% ...
- [Anandtharam et al] centralized lower-bound = 1.41,
- [Anandkumar et al] decentralized lower-bound = 2.78
- Our better (larger) decentralized lower-bound = 2.83,
Warning: forcing to use putatright = False because there is 7 items in the legend.
Warning: forcing to use putatright = False because there is 4 items in the legend.
Warning: forcing to use putatright = False because there is 4 items in the legend.
That’s it for this demo!