Experiment design¶

Agents are playing the knowledge-based agreement game using their symbolic knowledge base. Agents adapt their knowledge base by using a single adaptation operator.

Date: 2026-01-22

Designer: Richard Trézeux

Hypothesis: Agents using a single adaptation operator do not reach a consensus on decisions to take

10 agents; 2 environments; 5 runs per environment.

Experiment¶

Date: 2026-01-22 (Richard Trézeux)

Link to code

Simulator code hash: 5da10fed4b6c78179ddf3ecd7b46b0cc4ac7c25a

Parameter file: params.sh

Executed command (script.sh):

#!/bin/bash

# retrieve code

. params.sh

git clone https://gitlab.inria.fr/moex/karlOperators.git code
cd code
git checkout $LLHASH
cd ..

pip install -r code/requirements.txt

# run
        
METHODS=("singleop1" "singleop2" "singleop3" "singleop4" "singleop5" "singleop6")
SEEDS=(24 2727)
INIT=(random consistent)

MAX_JOBS=5

for method in "${METHODS[@]}"; do
  for seed in "${SEEDS[@]}"; do
    for init in "${INIT[@]}"; do
    
        while [ "$(jobs -r | wc -l)" -ge "$MAX_JOBS" ]; do
        sleep 1
        done
        echo "Running $method seed $seed init $init"

        python code/main.py \
            --config params.sh \
            --method $method \
            --seed $seed \
            --init $init &
    done
  done
done

wait

Experimental Plan¶

The independent variables have been varied as follows:
LEARNING_METHOD = [singleop1, singleop2, singleop3, singleop4, singleop5, singleop6]
SEED = [24, 2727] (randomly selected)
KB_INIT_METHOD = [random, consistent]

Constants of the experiment:
NBINTERACTIONS = 100000
NBPROPERTIES = 6
NBDECISIONS = 4
ADAPTING_AGENT_SELECTION = accuracy

Full results are available on Zenodo

Key parameters description¶

Out[28]:

Parameter	Description
NBAGENTS	Size of the agent population.
NBGENERATIONS	Number of generations or iterations.
NBINTERACTIONS	Maximum number of interactions for a population (just a security if agents did not converge to a consensus).
NBPROPERTIES	Number of properties in the environment (influences complexity).
NBDECISIONS	Number of decisions to discriminate objects.
KB_INIT_METHOD	KB initialization methods. Two different methods are implemented (see paper).
ADAPTING_AGENT_SELECTION	The type of score used to select which agent will be adapting its knowledge (accuracy or successrate).
LEARNING_METHOD	Type of updates and policy for agents.
SEED	Random seed of the experiment. Controls the generation of objects in the environment

NBPROPERTIES, NBCLASSES, NBAGENTS, and NBGENERATIONS have a strong impact on computational time.
Large values may lead to long simulations.

The directory folder structure we used for analysis is :

results   
│
└───method1
│   └───seed1
│       │   logs1_gen1.jsonl
│       │   logs1_gen2.jsonl
│       │   ...
|       |   metrics.json
|   └───seed2
|       |   logs1_gen1.jsonl
│       │   logs1_gen2.jsonl
│       │   ...
|       |   metrics.json
|   └───seed3...
│   
└───method2
│   └───seed1
│       │   logs1_gen1.jsonl
│       │   logs1_gen2.jsonl
│       │   ...
|       |   metrics.json
|   └───seed2...

Hypothese: Agents do not reach a consensus under single adaptation operator strategies¶

Consistent initialization

Random initialization

We observe that agents never reach 100% success rates under any of the single adaptation operator strategies.

Environment generation and seeds used for experiments from the paper¶

The following trees represent the different $d^*$ function (environment objects generation) used.

SEEDS USED :  [24, 2727]

No description has been provided for this image

Hyperparameters table¶

Out[27]:

Name	Symbol	Value	Description
Environment-related
Number of properties	$\|\mathcal{P}\|$	(cfg)	Number of object types is $2^{\|\mathcal{P}\|}$
Number of decisions	$\|\mathcal{D}\|$	(cfg)
Number of agents	$N_{\text{agents}}$	(cfg)
Number of generations	$N_{\text{gen}}$	(cfg)	Total training iterations
Interaction window size	--	500	Window for population success rate computation
Environment Tree generation	$(N_{\text{props}}, N_{\text{classes}}, p_{\text{stop}})$	$p_{\text{stop}} = 0.6$	$p_{\text{stop}}$ controls depth of the tree
KB initialization (opt. 1)	$N(K_{init})$	$Bin(n = \frac{\|\text{obj}\|}{2}, p = 0.5)$	Choose clauses among correct ones only
KB initialization (opt. 2)	$N(K_{init})$	$Bin(n = \|\text{obj}\|, p = 0.5)$	Choose clauses among correct and incorrect ones

Experiment 20260122-KARL¶