Deliberation

Prelude.Deliberation puts learning in the one place where it is safe for an agent: choosing among the options it is already permitted to take. Something else — typically the agent’s logical theory — decides which plans or actions are licensed and hands over a 0/1 licence mask m. This module orders the licensed options by learned scores. A learner that learns badly therefore picks a worse licensed option; it can never pick an unlicensed one.

The guarantee is arithmetic, not a promise. Before any policy sees the scores, every unlicensed entry is replaced by a floor of \(-10^{300}\), which behaves as \(-\infty\): no score can buy its way past it. Each policy below is the corresponding one from Policies applied to those masked scores, so every one of them satisfies \(\operatorname{supp}(\pi) \subseteq \operatorname{supp}(m)\). The mask is deliberately not applied afterwards as a filter — normalising over all options and then zeroing the unlicensed ones would leave rows that no longer sum to one. The BDI agents guide shows where the licence mask comes from: an agent’s .ha theory computes which plans it may adopt, and this module orders them.

Conventions: q holds the learned scores and m the licence mask, both n×A matrices (use 1×A for a single decision). At least one option per row must be licensed.

Notation. \(Q \in \mathbb{R}^{n \times A}\) is q and \(M \in \{0, 1\}^{n \times A}\) is m, with at least one \(M_{ia} = 1\) in every row \(i\). \(\odot\) is the elementwise product, \([\,\cdot\,]\) the Iverson bracket, and \(\operatorname{safe}(d) = d + [\,d = 0\,]\) the guarded divisor from Linear algebra.

open Prelude.Deliberation

Licensing

licensed

licensed(q, m) · q scores matrix n×A · m licence mask matrix n×A → masked scores matrix n×A

Let \(Q \in \mathbb{R}^{n \times A}\) (the argument q) be the scores and \(M \in \{0, 1\}^{n \times A}\) (the argument m) the licence mask. Then

\[\operatorname{licensed}(Q, M) = M \odot Q + (1 - M) \cdot \big({-10^{300}}\big), \qquad \operatorname{licensed}(Q, M)_{ia} = \begin{cases} Q_{ia} & M_{ia} = 1, \\ -10^{300} & M_{ia} = 0. \end{cases}\]

The scores with every unlicensed entry replaced by the floor.

Intuition Push every forbidden option so far down that no learned score can ever lift it back. Masking the scores rather than the resulting probabilities means every policy computed afterwards stays correctly normalised and simply never sees the forbidden options as contenders.

Use it when some rule outside the learner — a logical theory, a safety constraint, a business rule, an action unavailable in this state — decides what is allowed, and you want a learned ranking only among the allowed options. The prefer… policies call it for you. Every row needs at least one licensed option; otherwise all entries sit at the floor and the licence no longer separates anything.

safety-constraintsdecision-makingmasking

def licensed(q, m)  ≔ m · q + (1 − m) · barred

Building blocks

barred — The constant \(\operatorname{barred} = 0 - 10^{300} = -10^{300}\). The floor given to unlicensed options.

Policies over the licensed set

preferBest

preferBest(q, m) · q scores matrix n×A · m licence mask matrix n×A → policy matrix n×A

Let \(Q \in \mathbb{R}^{n \times A}\) (the argument q) and \(M \in \{0, 1\}^{n \times A}\) (the argument m), and let \(L = \operatorname{licensed}(Q, M)\). Then

\[\operatorname{preferBest}(Q, M)_{ia} = \operatorname{greedyOf}(L)_{ia} = \frac{[\,L_{ia} = \max_b L_{ib}\,]}{\sum_{a'} [\,L_{ia'} = \max_b L_{ib}\,]} .\]

The greedy policy over licensed options (for scores above \(-10^{300}\)); ties share probability.

Intuition Take the best-scoring option among those allowed, and split evenly when several tie. It is pure exploitation: whatever the learner currently believes is best, within the licence.

Use it when you are acting for real rather than learning — deploying a trained agent, choosing a plan once scores are trusted, or evaluating the learned ranking. Because it never tries anything else, scores for untried options never improve; while still learning, use preferExploring, preferSoftly or preferConfident.

decision-makingsafety-constraintsplanning

def preferBest(q, m)          ≔ greedyOf(licensed(q, m))

preferEvenly

preferEvenly(q, m) · q scores (ignored) matrix n×A · m licence mask matrix n×A → policy matrix n×A

Let \(Q \in \mathbb{R}^{n \times A}\) (the argument q, not used) and \(M \in \{0, 1\}^{n \times A}\) (the argument m). Then

\[\operatorname{preferEvenly}(Q, M)_{ia} = \operatorname{shared}(M)_{ia} = \frac{M_{ia}}{\operatorname{safe}\big(\sum_{b} M_{ib}\big)} .\]

The uniform policy over licensed options.

Intuition Know nothing, prefer nothing: every allowed option is equally likely. The scores are ignored, so this is the policy of an agent that has not learned yet.

Use it when you need a baseline to compare a learned policy against, a cold start before any rewards have been seen, or data collection that should cover every licensed option evenly. It is also the exploration part of preferExploring. Keep the scores argument for a uniform signature; its values do not matter.

explorationmodel-evaluationsafety-constraints

def preferEvenly(q, m)        ≔ shared(m)

preferExploring

preferExploring(q, m, eps) · q scores matrix n×A · m licence mask matrix n×A · eps exploration rate ε scalar → policy matrix n×A

Let \(Q \in \mathbb{R}^{n \times A}\) (the argument q), \(M \in \{0, 1\}^{n \times A}\) (the argument m) and \(\varepsilon \in [0, 1]\) (the argument eps). Then

\[\operatorname{preferExploring}(Q, M, \varepsilon) = (1 - \varepsilon)\, \operatorname{preferBest}(Q, M) + \varepsilon\, \operatorname{preferEvenly}(Q, M) .\]

ε-greedy restricted to licensed options: \((1-\varepsilon)\) on the best licensed option and \(\varepsilon\) spread evenly over all licensed options.

Intuition Mostly take the best allowed option, but with probability \(\varepsilon\) pick any allowed option at random, so every licensed option keeps being tried and its score can still improve. The random share includes the best option, so it ends up with \(1 - \varepsilon + \varepsilon/k\) for \(k\) licensed options.

Use it when the agent must keep learning while acting and you want the simplest exploration with one intuitive knob. Typical \(\varepsilon\) is 0.05–0.2, often decayed over time (decayed in Policies). It explores blindly — a clearly bad option is tried as often as a promising one; preferSoftly or preferConfident explore more selectively.

explorationdecision-makingsafety-constraints

def preferExploring(q, m, eps) ≔ (1 − eps) · preferBest(q, m) + eps · preferEvenly(q, m)

preferSoftly

preferSoftly(q, m, tau) · q scores matrix n×A · m licence mask matrix n×A · tau temperature τ scalar → policy matrix n×A

Let \(Q \in \mathbb{R}^{n \times A}\) (the argument q), \(M \in \{0, 1\}^{n \times A}\) (the argument m) and \(\tau > 0\) (the argument tau), and let \(L = \operatorname{licensed}(Q, M)\). Then

\[\operatorname{preferSoftly}(Q, M, \tau)_{ia} = \operatorname{boltzmann}(L, \tau)_{ia} = \frac{e^{L_{ia}/\tau}}{\sum_{b} e^{L_{ib}/\tau}} ,\]

computed stably by subtracting each row’s maximum. The softmax policy over licensed options; unlicensed options receive probability 0, because at any ordinary temperature their weight \(e^{(-10^{300} - \max_b L_{ib})/\tau}\) underflows to exactly zero.

Intuition Allowed options are chosen in proportion to \(e^{\text{score}/\tau}\): better options are more likely, near-ties are treated almost equally, and clearly bad options are rarely tried. The temperature sets how strongly score differences matter — near zero it behaves like preferBest, very large like preferEvenly.

Use it when you want exploration that respects how much better one option looks than another, or a smooth, stochastic choice among licensed plans. Pick \(\tau\) on the scale of the score differences you care about, since the result depends on scores divided by \(\tau\); rescaling rewards means retuning it.

explorationdecision-makingsafety-constraints

def preferSoftly(q, m, tau)   ≔ boltzmann(licensed(q, m), tau)

preferConfident

preferConfident(q, m, c, counts) · q scores matrix n×A · m licence mask matrix n×A · c exploration weight scalar · counts times each option was chosen matrix n×A → policy matrix n×A

Let \(Q \in \mathbb{R}^{n \times A}\) (the argument q), \(M \in \{0, 1\}^{n \times A}\) (the argument m), \(c \ge 0\) and \(N \in \mathbb{N}^{n \times A}\) (the argument counts) with row totals \(N_i = \sum_b N_{ib}\). Let \(U_{ia} = Q_{ia} + c\,\sqrt{\ln \max(N_i, 1) / \operatorname{safe}(N_{ia})}\). Then

\[\operatorname{preferConfident}(Q, M, c, N) = \operatorname{greedyOf}\big(\operatorname{licensed}(U, M)\big) .\]

UCB over licensed options: the greedy licensed option after adding the confidence bonus of confident.

Intuition Optimism in the face of uncertainty: add to each score a bonus that is large for options chosen rarely compared with the total, then act greedily among the allowed options. Poorly known options get tried until their score is trustworthy, and the bonus shrinks as they are chosen.

Use it when choices are repeated with the same options (bandit-style plan selection) and you want exploration that stops on its own without a schedule. \(c\) trades exploring against exploiting; with rewards in \([0, 1]\), \(\sqrt{2}\) is the classical value. Note the guards: an option never chosen gets the bonus of one chosen once, not an infinite one, and with a row total of at most one there is no bonus at all.

explorationdecision-makingsafety-constraints

def preferConfident(q, m, c, counts) ≔ preferBest(confident(q, c, counts), m)

What a desire is worth

An agent’s own history of cycles — belief, choice, outcome, reward — is already a dataset of transitions, so the value of pursuing a desire can be measured as a discounted return over what the agent has actually done rather than a hand-tuned number.

worth

worth(rewards, gamma) · rewards rewards in order vector T · gamma discount scalar → discounted return scalar

Let \(r = (r_0, \dots, r_{T-1}) \in \mathbb{R}^T\) (the argument rewards) and \(\gamma \in [0, 1]\) (the argument gamma). Then

\[\operatorname{worth}(r, \gamma) = \operatorname{returnOf}(r, \gamma) = \sum_{t=0}^{T-1} \gamma^{t}\, r_t .\]

The discounted return of the reward sequence from its first step (returnOf from Returns).

Intuition Score a desire by what pursuing it actually paid: the rewards of the cycles that followed, with later rewards discounted by \(\gamma\) per cycle. The agent’s own journal becomes the evidence, rather than a value somebody set by hand.

Use it when you rate a desire or plan from one uninterrupted run of cycles, compare how different pursuits paid off, or produce a learning target for plan scores. The sequence is taken as one episode, starting at its first reward; when the rewards span several episodes, use episodeReturns from Returns, and for a value at every step use returns.

policy-evaluationcredit-assignmentplanning

def worth(rewards, gamma)        ≔ returnOf(rewards, gamma)

worthWithin

worthWithin(rewards, gamma, ended) · rewards rewards vector T · gamma discount scalar · ended end-of-episode flags vector T → see below

Let \(r \in \mathbb{R}^T\) (the argument rewards), \(\gamma \in [0, 1]\) (the argument gamma) and \(d \in \{0, 1\}^T\) (the argument ended). Then, as written,

\[\operatorname{worthWithin}(r, \gamma, d) = \operatorname{decayWithin}(r, \gamma, d) = \operatorname{decayFrom}(r, \gamma) \odot \operatorname{sameRun}(d),\]

whereas decayWithin(g, rw, done) in Returns expects the discount first and the rewards second. Intended as the episode-aware counterpart of worth, it therefore passes the rewards in the discount position and the discount in the reward position, and would return a T×T decay matrix rather than a return. Called with a scalar discount it fails at run time. Use episodeReturns(rewards, gamma, ended) from Returns instead.

Intuition Meant to be worth for a journal that holds several episodes: returns that stop at each episode’s end instead of leaking into the next one. As written, it hands its arguments to decayWithin in the wrong order, so it does not compute that.

Use it when — do not use it for now. To value desires across a multi-episode journal, call episodeReturns(rewards, gamma, ended) from Returns, which computes exactly the per-episode discounted returns this definition was meant to give. For a single episode, worth is correct.

policy-evaluationepisode-segmentation

def worthWithin(rewards, gamma, ended) ≔ decayWithin(rewards, gamma, ended)

Checking the invariant

leaked

leaked(pol, m) · pol policy matrix n×A · m licence mask matrix n×A → leaked probability per row vector n

Let \(\pi \in [0, 1]^{n \times A}\) (the argument pol) be a policy and \(M \in \{0, 1\}^{n \times A}\) (the argument m) the licence mask. Then, for \(i = 1, \dots, n\),

\[\operatorname{leaked}(\pi, M)_i = \sum_{a=1}^{A} (1 - M_{ia})\, \pi_{ia} .\]

The total probability a policy puts on unlicensed options. For every policy in this module it is zero; assert that in tests. Checking only that the licensed options sum to one would miss a policy that also puts weight elsewhere.

Intuition The safety invariant turned into a number: the probability mass a policy puts outside the licence. Zero means the policy can never choose a forbidden option; anything above zero is a violation, however small.

Use it when you test or monitor a policy — especially one built outside this module, loaded from elsewhere, or combined by hand from others. Assert that every row is zero rather than only checking that licensed options sum to one. Use a tolerance near floating-point rounding only if the policy was computed by a route that can leave tiny residues.

safety-constraintsinvariant-checkingmodel-evaluation

def leaked(pol, m) ≔ einsum("ij->i", (1 − m) · pol)

Examples

The examples below live in one file, Licensed.hb, so the class is Licensed. Matrices are passed as Tensor.

The best option is not licensed

Option 0 has the highest score but is not licensed. The greedy policy chooses option 2, the best licensed one; ε-greedy with \(\varepsilon = 0.3\) gives \(0.7 + 0.15\) to option 2 and \(0.15\) to option 1; the softmax gives option 0 nothing.

open Prelude.Deliberation

fn bestLicensed(q, m) ↦ preferBest(q, m)
fn exploreLicensed(q, m, eps) ↦ preferExploring(q, m, eps)
fn softLicensed(q, m, tau) ↦ preferSoftly(q, m, tau)

using MLambda.Hilbert.Runtime;
using Examples;

var scores  = (Tensor)new double[,] { { 5, 1, 3 } };
var licence = (Tensor)new double[,] { { 0, 1, 1 } };

var best    = Licensed.BestLicensed(scores, licence);
var explore = Licensed.ExploreLicensed(scores, licence, 0.3);
var soft    = Licensed.SoftLicensed(scores, licence, 1.0);

Output (verified, rounded to six digits):

preferBest = shape [1, 3] values [0, 0, 1]
preferExploring = shape [1, 3] values [0, 0.15, 0.85]
preferSoftly = shape [1, 3] values [0, 0.119203, 0.880797]

Detecting a policy that leaks

A policy built elsewhere puts 0.2 on the unlicensed option; leaked reports it.

open Prelude.Deliberation

fn leakage(pol, m) ↦ leaked(pol, m)

using MLambda.Hilbert.Runtime;
using Examples;

var licence = (Tensor)new double[,] { { 0, 1, 1 } };
var leak    = Licensed.Leakage((Tensor)new double[,] { { 0.2, 0.3, 0.5 } }, licence);

Output (verified):

leaked = [0.2]

The worth of a desire from its rewards

Three cycles each rewarded 1, discounted at 0.5: \(1 + 0.5 + 0.25\).

open Prelude.Deliberation

fn desireWorth(rewards, gamma) ↦ worth(rewards, gamma)

using MLambda.Hilbert.Runtime;
using Examples;

var value = Licensed.DesireWorth((Tensor)new[] { 1d, 1d, 1d }, 0.5);

Output (verified):

worth = 1.75