LLM Generation Parameters: Decoding Method Decides

Thirteen knobs sit in a watsonx.ai generation config, and one of them silently decides whether the three most famous ones do anything at all — and it isn't temperature.

A model produces one token at a time: score the vocabulary, turn scores into probabilities, pick one, repeat. Every generation parameter changes how the pick is made, when the loop stops, or the plumbing around the call.

This piece is in two parts, and the line between them matters. Part 1 is what IBM documents — wording quoted from the API reference and SDK docs, with the ranges. Part 2 is my take — start values, traps, a symptom-to-fix table and four configs. Part 2 is opinion from use, not documentation; treat it as such.

Part 1 — Reference: what watsonx.ai documents

The gate: decoding_method

Two consequences that follow from the documentation, not from opinion:

Group 1 — Randomness (sampling mode only)

Mechanism, computed — one step, three candidate tokens with logits 4.0 / 2.0 / 1.0, softmax after dividing by temperature (cut-off uses the "add until the sum reaches p" convention from the docs):

Read across: temperature reshapes the distribution but removes nothing; top_p widens as the model gets less sure; top_k keeps a fixed count regardless. At temperature 0.5, top_p: 0.9 already leaves one candidate — the two parameters interact rather than stack.

Group 2 — Length

Group 3 — Repetition

Group 4 — Service

SDK note: GenTextParamsMetaNames.DECODING_METHOD is literally the string "decoding_method"; a plain dict with the same keys is accepted everywhere params is.

from ibm_watsonx_ai.foundation_models import ModelInference
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
 
params = {
    GenParams.DECODING_METHOD: "greedy",
    GenParams.MAX_NEW_TOKENS: 300,
    GenParams.STOP_SEQUENCES: ["\n\n"],
}
model = ModelInference(model_id="ibm/granite-3-8b-instruct", params=params,
                       credentials=credentials, project_id=project_id)
print(model.generate_text("Extract the invoice total as a number only:\n" + document))

Part 2 — My take: how I set them

Everything below is judgement from use. Where it leans on a documented fact, the fact is in Part 1.

Decision rule

Set decoding_method explicitly on every call. It costs one line and removes a whole class of "why is temperature doing nothing" questions.

One caveat on "deterministic": the docs describe greedy as deterministic, and it is by definition. Hosted inference has still shown run-to-run differences under greedy in the wild (there's a public langchain-ibm issue about it). For tests, assert on structure or with tolerance, not on byte equality.

Start values and traps

stop_sequences recipes

repetition_penalty ladder

Symptom → parameter

Four configs

EXTRACT  = {"decoding_method": "greedy", "max_new_tokens": 300,
            "stop_sequences": ["\n\n"], "include_stop_sequence": False,
            "repetition_penalty": 1.0}
ASSIST   = {"decoding_method": "sample", "temperature": 0.7, "top_p": 0.9,
            "max_new_tokens": 800, "repetition_penalty": 1.05}
IDEATE   = {"decoding_method": "sample", "temperature": 1.0, "top_p": 0.95,
            "max_new_tokens": 500, "repetition_penalty": 1.1}
EVALUATE = {**ASSIST, "random_seed": 42}

Key takeaway: decoding_method isn't one of the thirteen parameters — it's the gate three of them have to pass through.