Experiment 20220109-MTOA

Experiment design

20220109-MTOA

Date: 2021-12-09 (Andreas Kalaitzakis)

Hypothesis

An agent will improve its accuracy on one task by carrying out another task.

Experimental setting

20 runs; 80000 games

Agents learn multi-task ontologies. Agents will coordinate on a set of decision tasks, making a decision about an object.

Variables independent variables: ['numberOfTasks']

dependent variables: ['t1_accuracy']

Experiment

Date: 2022-01-09 (Andreas Kalaitzakis)

Computer: Dell Precision-5540 (CC: 12 * Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz with 16GB RAM OS: Linux 5.4.0-92-generic)

Duration : 15 minutes

Lazy lavender hash: ceb1c5d1ca8109373d293b687fc55953fce5241d

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

. params.sh

CURRDIR=$(pwd)
OUTPUT=${CURRDIR}/${DIRPREF}
# cd ${LLPATH}
cd lazylav
# this sample runs ExperimentalPlan. It can be replaced with Monitor if parameters are not varied.
bash scripts/runexp.sh -p ${CURRDIR} -d ${DIRPREF} java -Dlog.level=INFO -cp ${JPATH} fr.inria.exmo.lazylavender.engine.ExperimentalPlan -Dexperiment=fr.inria.exmo.lazylavender.decisiontaking.multitask.MultitaskExperiment ${OPT} -DresultDir=${OUTPUT}

Analysis

/home/andreas/.local/lib/python3.6/site-packages/pandas/core/frame.py:4308: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,
/home/andreas/.local/lib/python3.6/site-packages/pandas/core/frame.py:4308: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  errors=errors,

Raw data

Full results can be found at: Zenodo DOI



Table 1

Table 1 consists of the final achieved ontology accuracy values, i.e., the accuracy after the last iteration, with respect to task 1. Each column corresponds to a different number of tasks, while each row corresponds to a different run, for the same number of tasks.

Table 1 : t1 accuracy for different number of tackled tasks.

Out[7]:
1 2 4
0 0.562500 0.875000 0.937500
1 0.687500 0.750000 0.750000
2 0.687500 0.875000 0.937500
3 0.500000 0.812500 0.812500
4 0.562500 0.812500 0.812500
5 0.562500 0.812500 0.687500
6 0.500000 0.812500 0.875000
7 0.562500 0.875000 1.000000
8 0.625000 0.812500 0.937500
9 0.687500 0.875000 0.812500
10 0.625000 0.812500 0.687500
11 0.500000 0.812500 0.812500
12 0.625000 0.812500 0.812500
13 0.625000 0.812500 0.625000
14 0.687500 0.750000 0.937500
15 0.625000 0.687500 0.875000
16 0.562500 0.875000 0.750000
17 0.562500 0.750000 0.937500
18 0.687500 0.687500 0.812500
19 0.562500 0.687500 0.875000

Figures

Figure 1 depicts the evolution of the accuracy for task_1 (yaxis), depending on the number of tasks. The x-axis is scaled to the number of tackled tasks, and thus does not correspond to the total interactions for each run. Each point x,y corresponds to the accuracy for task 1 when x interactions regarding task 1 have taken place. The figure demonstrates how knowledge with respect to one task is affected by tackling another task. Results support the hypothesis, showing that the presence of additional tasks affects the accuracy for task_1. In particular, tackling one additional task is shown to improve, on this particular setting, the accuracy for task 1 by 33% while tackling 3 additional tasks improves the accuracy fortask 1 by 39 %.

Analysis of variance (ANOVA)

Based on Table 1, we perform one-way ANOVA, testing if the independent variable "numberOfTasks" has a statistically significant effect on the final average accuracy values.

One-way Anova on the independent variable numberOfTasks
F : 52.79446640316205
p : 1.0625281464640243e-13

Performing one-way ANOVA on the values of Table 1 returns a p value of ~1.06e-13. This value is lower than 0.05, thus we consider that the number of tackled tasks statistically significantly affects the obtained final task 1 accuracy values.

Conclusions

In this work, we hypothesize that by agents will improve their knowledge on one task, by carrying out a different one (additional). Results support this hypothesis, suggesting the formation of general purpose knowledge.