Unit 02 of 03
Reasoning Techniques
Measuring, on two tasks at once, what asking the model to take more steps gains and what it costs: chain of thought paying off on multi-step calculation but multiplying tokens without a gain on single-step classification; self-consistency accuracy and call cost as the number of samples goes from one to nine, and consensus breaking on a wrong answer; the number of calls growing as the branching factor rises and the point where the gain stops; the difference between a two-stage prompt that asks for the principle first and the solution second; the share of responses with a valid action when thought and action are requested in the same prompt; the token cost of intermediate steps that are invisible before the response and the task where the difference stays unmeasurable; the overlap between the errors of different prompts; the gain in validation not carrying over to the test as the search budget grows.
Published Updated
Lessons
- 01 Chain of Thought Measuring, on two separate tasks, what happens when a prompt asks the model for the intermediate steps instead of the result: on the multi-step fictional tariff calculation accuracy rises from 0.3000 to 0.8833 and tokens per correct response fall from 243.2 to 150.1, while on single-step support ticket classification the difference changes sign across the three runs as -0.0500, -0.0417 and +0.0750 and stays inside the run band. Total tokens grow about 1.8 times on both tasks; in classification the cost per correct response rises from 59.1 to 114.1 tokens. 5 min read
- 02 Self-Consistency The gain and the cost of running the same prompt several times and voting on the responses: as the number of samples goes from one to nine, accuracy rises from 0.7667 to 0.8833 and stops after seven, the number of calls goes from 60 to 540 and the tokens per correct response from 172.9 to 1343.6. At nine samples a consensus forms on 58 records, but on five of them the consensus is on a wrong answer; the consumption of those five records lies between 41 and 45 m³, that is, where the error is systematic rather than scattered. 4 min read
- 03 Tree of Thoughts Measuring branching from the middle of a solution: on the same 48 fictional support tickets, the arrangement that asks for all thirty-two fault codes in a single prompt gives 0.4792 accuracy with 48 calls; the tree of thoughts, which splits the solution into steps and asks for several extensions at every step, falls to 0.0833 at branch 1, rises to 0.6250 at branch 3 and stops at branch 4. As branching goes from 3 to 4, accuracy slips by 0.0208 while the number of calls goes from 795 to 1086 and tokens from 43182 to 59835; since this difference is smaller than the 0.0417 difference between two runs of the same arrangement, it counts as unmeasured. 12 min read
- 04 Step-Back Prompting Measuring what it is worth to ask for the principle before solving: on the same 48 fictional support tickets, the single-stage prompt gives 0.5417 code accuracy with 48 calls, while the two-stage prompt, which first asks for the general rule and then asks for the solution, gives 0.7500 and does it with 96 calls and 1.66 times the tokens. The whole gain depends on the first stage: under the right principle code accuracy is 0.8000, and when a wrong principle is imposed it drops to between 0.0000 and 0.1458, that is, below the single-stage prompt. The same arrangement does not move accuracy at all in a task that asks only for the degree, but it raises the tokens to 2.69 times. 6 min read
- 05 Reason and Act Measuring what it takes to ask for the thought field and the action field together in the same prompt: on 120 fictional support tickets, when an action field is requested only 0.1000 of the responses produce a valid action field; when the action set is written into the prompt, 0.5250; when the thought is cut to a single sentence, 0.8000. The cost rises from 5300 to 8060 in input tokens and falls from 645.8 tokens to 102.1 tokens per valid action field. The two-run comparison leaves the difference of writing the action set unmeasurable. 9 min read
- 06 Reasoning Models The yardstick for the class of models that produce invisible intermediate steps before writing a response: on the same 120 fictional support tickets, in the single-field task the accuracy difference between the two classes stays at −0.0333 and +0.1167; because the two-run difference is 0.1167 it counts as unmeasured, and tokens rise 1.75 times. In the two-field task the difference rises above the band with +0.1000 and +0.1667; the single-pass class contradicts its own label in the code field in 0.2333 of the responses. The cost per correct response rises from 52.0 to 95.1 and from 83.9 to 110.1 tokens. 9 min read
- 07 Prompt Ensemble The same 120 fictional support tickets are examined with three separate instructions, and the assumption the ensemble rests on is tested: do different prompts make different errors. The three prompts alone give 0.7750, 0.6583 and 0.7833, on nine records all three are wrong at once and the ceiling comes out at 0.9250. The majority vote stays at 0.8000: 0.1250 below the ceiling, only 0.0167 above the best single prompt, and this gain is below the 0.0333 to 0.1333 difference the same prompts show between two runs. The cost is written in number of calls: 120 calls become 360 calls, and 61.1 tokens per correct label become 180.7 tokens. 13 min read
- 08 Automatic Prompt Optimization Instead of being written by hand, the prompt is searched with a criterion: ten candidates are produced mechanically from four parts, and the fictional support tickets are split into a 40-record validation set and an 80-record test set. As the search budget goes from one candidate to ten, the best score on validation rises from 0.7750 to 0.9000 and the test score of the same candidate falls from 0.6750 to 0.6500: validation gain +0.1250, test gain -0.0250, and the chosen candidate's optimism margin 0.2500. The search goes from 40 calls to 400 calls, the candidate that is truly best on the test (0.8000) is never chosen, and in the second run the search chooses a different candidate. 13 min read