Experiment 20231220-MTOA¶

Experiment design¶

20231220-MTOA

Date: 2023-12-20 (Andreas Kalaitzakis)

Hypotheses¶

Favoring the reproduction of agents with the lowest success rate (agents that agree less with their peers) will allow agent populations to equally explore all tasks. Hence, agent populations will become equally accurate on all tasks and thus improve their efficiency.

Experimental setting¶

36 agents; 3 tasks; 6 features (2 independent features per task); 4 decision classes; 10 runs; 80000 games; 20000 games per generation

Each agent initially trains on all tasks. The agents then carry all tasks. When the agents disagree the following take place:

(a) The agent with the lower score will adapt its knowledge accordingly. If memory capacity limit is attained, the agent will try to forget knowledge.

(b) The agent with the highest score will decide for both agents. If this agent's decision is correct, the agent will receive the points corresponding to both agents.

Every 20000 games, 9 new agents are born and 9 agents are removed. Each new agent will be trained with examples provided by the two parents.

The first parent is selected either randomly, or based on their collected points/success rate (low success rate, high success rate, low points, high points). The second parent is selected based on its success rate with respect to the first parent (parent 2 is the the agent that agrees more often with parent 1).

Agents undertake 3 tasks having a limited memory, enough for learning 1/3 tasks accurately (4 and 12 classes respectively)

Variables independent variables: ['birthCriteria','birthSortingOrder','knowledgeLimit','parentSelectionPolicy']

dependent variables: ['avg_min_accuracy', 'avg_accuracy', 'avg_max_accuracy', 'success_rate', 'correct_decision_rate', 'delegation_rate', 'accuracy_relative_standard_deviation', 'total_points', 'points_distribution', 'games_relative_standard_deviation']

Experiment¶

Date: 2023-12-20 (Andreas Kalaitzakis)

Computer: Dell Precision-5540 (CC: 12 * Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz with 16GB RAM OS: Linux 5.4.0-92-generic)

Duration : 1080 minutes

Lazy lavender hash: ceb1c5d1ca8109373d293b687fc55953fce5241d

Parameter file: params.sh

Executed command (script.sh):

Analysis¶

Raw data¶

Full results can be found at:

Table 1: Final success rate values¶

Table 1 consists of the final achieved average success rate values, i.e., the average success rate after the last iteration.

Table 2: Final average minimum accuracy values¶

Table 2 consists of the final average minimum accuracy values with respect to the worst task, i.e., the accuracy after the last iteration for the task for which the agent scores the lowest accuracies.

Table 3: Final average accuracy values¶

Table 3 consists of the final achieved average ontology accuracy with respect to all tasks, i.e., the accuracy after the last iteration averaged on all tasks and agents.

Table 4: Final average best task accuracy values¶

Table 4 consists of the final average best task accuracy values with respect to the best task, i.e., the accuracy after the last iteration for the task for which the agent score the highest accuracies.

Table 5: Final average scope accuracy values¶

Table 5 consists of the final average scope accuracy values.

Table 6: Final correct decision rate values¶

Table 6 consists of the final achieved correct decision rate values, i.e., the average correct decision rate after the last iteration.

Table 7: Final accuracy for Task 1 values¶

Table 8: Final accuracy for Task 0 values¶

Table 9: Relative standard deviation of task undertakings¶

Figures¶

Figure 1: Success rate¶

Figure 1 displays the evolution of the average success rate (y-axis) as the number of iterations increases (x-axis), depending on the (a) birth criteria, (b) birth sorting order, (c) knowledge limit and (d) parent selection policy. Two things are shown. First, that favoring the reproduction of agents that are less successful or have accumulated less points, leads to a significantly lower success rate. This is to be expected according to the pressure exerted during the selection. Second, in both subfigures it is shown that the success rate stabilizes, but does not converge to 1. This indicates that either the agents' final ontologies do not allow them to agree on all decisions. Whether agents memory is equivalent to 4 or 12 classes, there is a statistically significant difference between the random selection of parents and the selection of the parents that have the lowest success rate (ASC_SRATE).

Figure 2: Average minimum accuracy (average accuracy of agents' accuracy on their worst task)¶

Figure 2 displays the evolution of the average minimum accuracy (y-axis) as the number of iterations increases (x-axis), depending on the (a) birth criteria, (b) birth sorting order, (c) knowledge limit and (d) parent selection policy. It is shown that the (a) and (b) and (d) have no statistically significant impact on the observed average minimum accuracy. Whether agents memory is equivalent to 4 or 12 classes, there is no statistically significant difference between the random selection of parents and the selection of the parents that have the lowest success rate (ASC_SRATE).

Figure 3: Average accuracy¶

Figure 3 displays the evolution of the average accuracy over all tasks (y-axis) as the number of iterations increases (x-axis), depending on the (a) birth criteria, (b) birth sorting order, (c) knowledge limit and (d) parent selection policy. It is shown that favoring the reproduction of agents that are less successful or have accumulated less points, leads to a significantly lower average accuracy for a given knowledge limit. Whether agents memory is of 4 or 12 classes, there is a statistically significant difference between the random selection of parents and the selection of the parents that have the lowest success rate (ASC_SRATE).

Figure 4: Average maximum accuracy (average accuracy of agents' accuracy on their best task)¶

Figure 4 displays the evolution of the average maximum accuracy over all agents (y-axis) as the number of iterations increases (x-axis), depending on the (a) birth criteria, (b) birth sorting order, (c) knowledge limit and (d) parent selection policy. It is shown that favoring the reproduction of agents that are less successful or have accumulated less points, leads to a significantly lower average maximum accuracy for a given knowledge limit. When agents memory holds 4 classes, there is no statistically significant difference between the random selection of parents and the selection of the parents that have the lowest success rate (ASC_SRATE). When the agents memory holds 12 classes, there is a statistically significant difference between the random selection of parents and the selection of parents that have the lowest success rate (ASC_RATE), i.e., agree less with their peers.

Figure 5: Correct decision rate¶

Figure 5 displays the evolution of the average correct decision rate (y-axis) as the number of iterations increases (x-axis), depending on the (a) birth criteria, (b) birth sorting order, (c) knowledge limit and (d) parent selection policy. It is shown that favoring the reproduction of agents that are less successful or have accumulated less points, leads to a significantly lower correct decision rate for a given knowledge limit. Whether agents memory is equivalent to 4 or 12 classes, there is a statistically significant difference between the random selection of parents and the selection of the parents that have the lowest success rate (ASC_SRATE).

Figure 6: Total points¶

Figure 6 displays the evolution of the average accumulated points by all agents (y-axis) as the number of iterations increases (x-axis), depending on the (a) birth criteria, (b) birth sorting order, (c) knowledge limit and (d) parent selection policy. It is shown that the reproduction of agents that agree less is favored, agent societies collectively accumulate less points. Whether agents memory is equivalent to 4 or 12 classes, there is a statistically significant difference between the random selection of parents and the selection of the parents that have the lowest success rate (ASC_SRATE), i.e., the parents that agree less with their peers.

Figure 7: Points distribution¶

Figure 7 displays the evolution of the distribution of the accumulated points to agents (y-axis) as the number of iterations increases (x-axis), depending on the (a) birth criteria, (b) birth sorting order, (c) knowledge limit and (d) parent selection policy. Several things are shown.First, within each generation, points equitability decreases with iterations. Second, points are more equitably distributed generation after generation. Third, after each new generation, the difference in final distribution values for different (a), (b) and (d) increases. Fourth, while the reproduction of less successful agents (in terms of points and communication) favors the relative standard deviation for the different tasks accuracy, leads into less equitable point distributions. When agents memory holds 4 classes, there is no statistically significant difference between the random selection of parents and the selection of the parents that have the lowest success rate (ASC_SRATE). When the agents memory holds 12 classes, there is a statistically significant difference between the random selection of parents and the selection of parents that have the lowest success rate (ASC_RATE), i.e., agree less with their peers.

Accuracy relative standard deviation¶

Figure 8 displays the evolution of the relative standard deviation for the average accuracy of each task over all agents (y-axis) as the number of iterations increases (x-axis), depending on the (a) birth criteria, (b) birth sorting order, (c) knowledge limit and (d) parent selection policy. The following are shown. First, when the reproduction of successful agents is favored, the relative standard deviation shows a slow increase. Contrariwise, when the reproduction of the less successful agents is favored, the standard deviation reaches a local maximum. It then decreases until it stabilizes. When agents memory holds 4 classes, there is no statistically significant difference between the random selection of parents and the selection of the parents that have the lowest success rate (ASC_SRATE). When the agents memory holds 12 classes, there is a statistically significant difference betwwen the random selection of parents and the selection of parents that have the lowest success rate (ASC_RATE), i.e., agree less with their peers.

Summary¶

When agents tackle all tasks, favoring the reproduction of agents whose knowledge is conflicting, leads to agent societies that are more equitably accurate on all tasks. Counter-intuitively, this is not shown to be the result of agent societies becoming more accurate on previously under-represented tasks. On the contrary, results show that balance in task representation results from agent societies becoming less accurate on previously over-represented tasks. As a consequence, agents become less efficient over time. Simply put, agent societies can either favor task representation in detriment of their efficiency, or favor their efficiency in detriment of task representation.