How I'm using RL to shorten market research surveys

Brand-tracking surveys are long. You pick a list of brands and ask every respondent the same block of questions about each one. Have you heard of it? Would you consider it? Is it good value? Which of these statements describe it? Multiply that by fifteen brands and the respondent is answering hundreds of items. Somewhere around the middle they stop reading and start clicking.

Most of those questions tell you very little. If someone has barely heard of a brand, you learn almost nothing by asking them whether it’s worth the money. Human survey designers already know this, so surveys come with routing rules: “if not aware, skip the brand block.” The rules are hand-written, all or nothing, and they never change.

I wanted to see whether a model could learn a softer version of those rules. For each question, for each respondent, it should decide: ask it or skip it?

Why a bandit and not a classifier #

The first idea most people have is supervised learning. Predict how a respondent will answer, and skip the question if you can already guess the answer. That fails for two reasons.

1. The thing you want isn’t a label. You don’t want to predict the answer. You want to decide whether asking is worth it, and that depends on a cost you choose: how much respondent fatigue you’ll accept to get one more answer. That’s a decision problem, not a prediction problem.
2. Your own choices change what you know. Once you skip a question, you never see its answer. Later decisions have less information to work with. A classifier trained on complete historical surveys never faces that.

A contextual bandit fits this shape well. At each step the agent sees a context (what it knows so far), picks an action (ASK or SKIP), and gets a reward. It never sees what the other action would have earned. The goal is to maximise total reward while still trying the less-tested action often enough to learn from it.

It’s the simplest form of reinforcement learning. There’s no long-horizon credit assignment like in full RL. Each decision earns its own reward right away. That’s a good fit here, and it’s why the model is small enough to explain in one blog post.

LinUCB in one page #

LinUCB (Li et al., 2010)^[1] assumes each action’s expected reward is linear in the context vector x:

E[reward | x, arm] = x · θ_arm

Each arm keeps its own ridge regression. The “disjoint” in disjoint LinUCB means the arms don’t share parameters. Each arm stores just two things:

• A: a d × d matrix, the identity plus the sum of x xᵀ over every context this arm has been updated with
• b: a d-vector, the sum of reward · x

The estimate is θ = A⁻¹ b. To choose an arm, LinUCB doesn’t just take the highest estimate. It adds a bonus for uncertainty:

score(arm) = x · θ_arm  +  α · sqrt( xᵀ A_arm⁻¹ x )
             └ exploit ┘    └──── explore ────┘

The square-root term is large when x points in a direction the arm has rarely seen, and small when it has seen many similar contexts. So the agent is optimistic about what it hasn’t tried yet. That’s the upper confidence bound. α sets how optimistic it is.

The update needs one practical trick. You need A⁻¹ for every score, and inverting a matrix at every step is wasteful. Each update adds a rank-one term x xᵀ, so the Sherman–Morrison formula updates the inverse directly in O(d²):

class DisjointLinUCB:
    def __init__(self, n_features, exploration_alpha=0.2):
        self.alpha = exploration_alpha
        self.arms = {
            arm: {"a_inv": np.eye(n_features), "b": np.zeros(n_features)}
            for arm in ("ASK", "SKIP")
        }

    def scores(self, x):
        out = {}
        for arm, s in self.arms.items():
            theta = s["a_inv"] @ s["b"]
            variance = max(0.0, float(x @ s["a_inv"] @ x))
            out[arm] = float(x @ theta + self.alpha * np.sqrt(variance))
        return out

    def update(self, arm, x, reward):
        s = self.arms[arm]
        ax = s["a_inv"] @ x
        s["a_inv"] = s["a_inv"] - np.outer(ax, ax) / (1.0 + x @ ax)
        s["b"] = s["b"] + reward * x

That’s the whole algorithm. Everything else in this project is about what goes into x, what the reward is, and how to evaluate it honestly.

The reward is the product decision #

The reward function is where you write down what you actually care about. For each question I computed a value score between 0 and 1: how much the answer tells you. A top-box answer, like “first choice” or “use it most often”, is worth close to 1. A “never heard of it” is worth 0. The rewards are:

ASK   →  value − ask_cost
SKIP  →  −missed_weight · value

Asking always costs a little (ask_cost, for fatigue). Skipping costs you in proportion to what you threw away (missed_weight). Put those together and there’s a break-even value:

ask  iff  value > ask_cost / (1 + missed_weight)

Every behaviour of the policy follows from these two numbers. If you raise ask_cost, the survey gets shorter and loses more. If you raise missed_weight, it gets more cautious. Those are business decisions, so they belong in a small, frozen config object and not scattered through the training loop.

Context without leakage #

This is the part I got wrong first, and I think it’s the most useful lesson in the project.

The training data is historical surveys in which everyone answered everything. That’s great for training, because you know the reward for both arms. It’s also a trap. The answer to the current question is sitting right there in the row, and it’s very easy to let it seep into the features. One-hot encode the “answer option” column, or include a score derived from the answer, and the model looks brilliant offline. In a live survey it’s useless, because it decided using information it could only have had after asking.

The fix is to replay each respondent one step at a time. The context for a question includes only:

• what the question is (metric, brand)
• where we are in the survey (share of steps done, share of questions asked so far)
• a prior for how valuable this question tends to be, learned from the training split only
• answers collected at earlier steps where the policy chose ASK

That last point matters. If the policy skipped the familiarity question for a brand, then its downstream decisions for that brand have no familiarity signal. They have to cope, exactly as they would in a live survey.

flowchart LR
  Q[Next question] --> C[Build context<br/>metadata + earlier ASKed answers]
  C --> G{Routing gate<br/>allows both?}
  G -- only SKIP --> S[SKIP]
  G -- both --> B[LinUCB picks<br/>ASK or SKIP]
  B -- ASK --> A[Collect answer] --> U[Update respondent state]
  B -- SKIP --> U
  S --> U
  U --> Q

The current answer is used only after the decision, to compute the reward.

Training with full feedback #

Because historical surveys have every answer, training can do something a live bandit can’t. It updates both arms with their true rewards at every step:

for arm, reward in reward_contract.rewards(value).items():
    model.update(arm, x, reward)

This is full-information learning, not true bandit feedback, and it’s worth saying so plainly. It converges faster and removes the need for off-policy correction. The exploration term still matters, though. Deployed live, the model would only see the reward for the arm it picked, and the UCB bonus is what stops it from settling too early. Training this way is a warm start. It doesn’t replace online learning.

The respondent state still evolves from the action the policy actually chose. So the model is trained on the same partially observed contexts it will face at inference time.

Keep the rules as rules #

Not every decision should belong to the model. Some are structural:

• Hard gates. If a respondent has never heard of a brand, don’t ask anything else about it. That’s a survey logic rule, and a model should never be able to break it.
• The funnel. The top-of-funnel metrics (awareness and consideration) are what the headline reports are built on. I didn’t want a learned policy quietly reshaping them. So they use a fixed, outcome-independent sample: a stable hash of respondent and brand, kept for a set fraction of pairs. I tuned that fraction on the validation split only, and chose the lowest rate that still kept the funnel close to the full survey.

The bandit decides everything else. The model only ever chooses from the actions the gate allows:

action, score = model.select_arm(x, allowed_actions)

This split turned out to matter a lot for evaluation, as you’ll see below.

Why accuracy is the wrong score #

My first instinct for evaluation was a confusion matrix: ASK/SKIP against “was it a valuable answer”. That’s the wrong frame. The model isn’t classifying anything. It’s making trade-offs under a cost you chose. Precision on “valuable” rewards a policy that asks everything, which is exactly what you’re trying to avoid.

What I track instead:

Regret per decision. Under the reward contract, an oracle that knew every answer would ask exactly when value clears the break-even. Regret is how much reward the policy gave up compared with that oracle. It’s never negative, and it counts both failure modes at once: a bad ask (you asked something worthless) and a bad skip (you skipped something valuable). This is the number to push down.

Bad-ask and bad-skip rates. These are guardrails. Regret is a single number, and a single number can hide one failure mode getting worse while the other gets better.

Score only the model’s own decisions. Gated questions and funnel-sampled questions aren’t the model’s choice. Mix them in and the model gets blamed, or credited, for survey design. When I separated them, the picture changed completely. Almost all of the information lost to skips came from the structure: if the funnel sample drops a brand’s awareness question, the gate then cascades and skips that brand’s whole block. The model’s own discretionary skips lost very little. That decides where to put effort next, and it’s the opposite of where I’d have looked from a single aggregate number.

Does the headline still hold? An adaptive survey is only useful if the numbers people report don’t move. For each brand I compared the adaptive rates with the full-survey rates on the same held-out respondents. The first check is a fixed percentage-point tolerance. The second is a pair of z-tests: one treating the two as independent samples, and one treating the full survey as the benchmark.

What I’d take from this #

• LinUCB is a very good first RL model. It’s about thirty lines, it’s fast, it’s deterministic given its state, and you can look at θ to see what it learned. You don’t need a deep policy network to decide whether to ask a question.
• The reward function is the spec. Spend your design time there, keep it in one frozen place, and use the same one for training and evaluation.
• Leakage in sequential problems is sneaky. If your offline data is complete but your live system isn’t, replay it step by step and only reveal what the policy has earned.
• Separate structure from policy, both in the system and in the evaluation. Otherwise you’ll tune the model to fix a problem the survey design created.
• Pick metrics that match the problem. For a decision policy that means regret, guardrail rates, and whether downstream numbers survive. Not accuracy.

The obvious next step is an engagement feature. The model’s worst misses were respondents who were unusually engaged with a brand, in ways the context couldn’t see. After that, the step I want most is running it live, where it only sees the reward for the arm it picks, and the exploration term has to earn its keep.