Skip to content
academia.sh
tr
Log in

Course Intermediate

Prompt Engineering

Published Updated

By the end of this course

Start course

01

Prompt Design

Splitting a prompt into its components and measuring the cost of each design decision next to its number: the token share of input, context and instruction and what happens when a component is removed, the total that repeating the system instruction in every request inflates and the accuracy that shortening it gives up, comparing the difference that role prompting adds with the difference between two runs, in how many different forms an ambiguous prompt is answered over ten runs, the share of parseable responses under a format constraint and the cost of a length limit, the point where the gain saturates as the example count goes from zero to eight and the effect of example order, the case where a negative constraint increases violations and its comparison with a positive statement, how the validation and test numbers diverge as the iteration count grows.

  1. 01 What Is a Prompt Splitting a prompt into input, context and instruction components and measuring the token share of each: across 120 fictional support tickets the instruction holds 0.3945, the context 0.3114 and the input 0.2941. When a single component is removed, the number and the cost are read side by side. The full prompt gives a label hit rate of 0.7833 and 5974 tokens; with the context removed it gives 0.7083 and 4177, with the instruction removed 0.7167 and 4751, with the input removed 0.1417 and 4197. The two-run difference is 0.0583 for the full prompt and rises to between 0.2000 and 0.2583 for prompts missing the context or the instruction.
  2. 02 System Instruction Measuring how a persistent behavior definition is sent again in every request and how its cost is multiplied by the number of requests: a 62-token instruction holds 0.8378 of a single prompt and 7440 tokens across 120 requests. Four lengths are set side by side. The prompt without an instruction gives a label hit rate of 0.4583 and 2985 tokens, the short instruction 0.7167 and 4751, the medium instruction 0.7500 and 5856, the long instruction 0.7417 and 9343. Cutting the long instruction to the medium one saves 3487 tokens and gives up nothing in hit rate; cutting the medium one to the short one saves 1105 tokens but raises the unlabeled response share from 0.0250 to 0.2167.
  3. 03 Specifying Role and Behavior Measuring the difference that adding a role and behavior prefix to a prompt makes and comparing it with the run difference of the same prompt. On the same 120 fictional support tickets the hit rate of the prefixed prompts moves -0.0667, -0.0583, -0.2208 and +0.0125 against the 0.7500 base, while the band of the base across four seeds is 0.1500. None of the four prefixes leaves the band, so none of the differences counts as measured. The only thing measured is the cost: prefixes of 9 to 27 tokens add 1080 to 3240 tokens over 120 requests and raise the cost per correct label from 65.1 to between 83.6 and 123.8.
  4. 04 Clear and Measurable Requests Measuring that an ambiguous request produces not a single wrong response but a distribution: the same ambiguous prompt gives a mean of 4.81 distinct responses per record in ten runs, the most frequent response is repeated in only 0.3944 of the runs, and no label can be read from 0.4333 of the responses. Four request forms are set side by side. Clarifying the verb adds two tokens and lowers the distribution to 4.53; closing the option set adds 15 tokens and lowers the distribution to 3.89 and the unlabeled share to 0.2889; adding an ambiguity rule adds 10 more tokens and raises the unlabeled share back to 0.3722. The band of the hit rate read in a single run is between 0.2778 and 0.3333 and cannot tell the four requests apart.
  5. 05 Format and Length Control The response to a clear request is still free text, and a program cannot read it. On the same 120 fictional support tickets a three-field record is requested: the share that the strict field parser reads in the free response is 0.0000, it becomes 0.9750 when a format declaration is added, and the label hit rate rises from 0.3083 to 0.6833. The cost is a rise from 6140 to 8780 in input tokens and from 1392 to 2026 in output tokens. When the maximum output length is lowered from 24 to 10, the output tokens fall from 2026 to 1170 but the parseable share drops from 0.9750 to 0.0000; at the intermediate values the curve stays tied exactly to the number of cut responses.
  6. 06 Using Examples Zero-shot, one-shot and few-shot prompting are measured side by side on the same task. As the number of examples rises from zero to eight, the hit rate on 112 fictional support tickets goes from 0.8571 to 0.9732, but the gain saturates at the third example: from zero to three examples +0.1071 hit rate comes with 4928 extra input tokens, from three to eight examples +0.0089 hit rate comes with 8736 extra tokens, and the mean between four and eight examples, 0.9250, stays below the 0.9643 of three examples. When the same four examples are written with only their order changed, the hit rate wanders between 0.7679 and 0.9196 and the input tokens stay the same in every order.
  7. 07 Stating Constraints and Prohibitions The same rule is written in two separate forms and measured on 120 fictional support tickets. With no statement the violation rate is 0.2583; when a negative constraint is added it rises to 0.5333, and the number of responses carrying one of the forbidden words goes from 5 to 35. The prompt that states the same rule positively brings the violation down to 0.0167 and raises the label hit rate from 0.6667 to 0.8333. When both are written together the violation is 0.1583: adding the prohibition to the positive statement makes the number about ten times worse. The cost is written side by side; the positive statement costs 115.9 input tokens per prevented violation.
  8. 08 Prompt Iteration The loop of improving by measuring is itself put under measurement. Ten candidate changes are searched greedily over eight iterations on a validation set of 30 records: the validation hit rate rises from 0.4000 to 1.0000 and hits the ceiling at the fifth iteration, and the test hit rate reaches its highest value of 0.9333 at the same point and falls to 0.8111 at the eighth iteration. The mean of the two-run difference on validation is 0.1037 and the mean step gain is 0.0625: only two of the eight iterations are larger than their own measurement noise. When the search is repeated in another run, only one of the eight steps chooses the same change at the same place. The cost is 2640 calls, and three of the eight chosen changes are irrelevant to the task.

02

Reasoning Techniques

Measuring, on two tasks at once, what asking the model to take more steps gains and what it costs: chain of thought paying off on multi-step calculation but multiplying tokens without a gain on single-step classification; self-consistency accuracy and call cost as the number of samples goes from one to nine, and consensus breaking on a wrong answer; the number of calls growing as the branching factor rises and the point where the gain stops; the difference between a two-stage prompt that asks for the principle first and the solution second; the share of responses with a valid action when thought and action are requested in the same prompt; the token cost of intermediate steps that are invisible before the response and the task where the difference stays unmeasurable; the overlap between the errors of different prompts; the gain in validation not carrying over to the test as the search budget grows.

  1. 01 Chain of Thought Measuring, on two separate tasks, what happens when a prompt asks the model for the intermediate steps instead of the result: on the multi-step fictional tariff calculation accuracy rises from 0.3000 to 0.8833 and tokens per correct response fall from 243.2 to 150.1, while on single-step support ticket classification the difference changes sign across the three runs as -0.0500, -0.0417 and +0.0750 and stays inside the run band. Total tokens grow about 1.8 times on both tasks; in classification the cost per correct response rises from 59.1 to 114.1 tokens.
  2. 02 Self-Consistency The gain and the cost of running the same prompt several times and voting on the responses: as the number of samples goes from one to nine, accuracy rises from 0.7667 to 0.8833 and stops after seven, the number of calls goes from 60 to 540 and the tokens per correct response from 172.9 to 1343.6. At nine samples a consensus forms on 58 records, but on five of them the consensus is on a wrong answer; the consumption of those five records lies between 41 and 45 m³, that is, where the error is systematic rather than scattered.
  3. 03 Tree of Thoughts Measuring branching from the middle of a solution: on the same 48 fictional support tickets, the arrangement that asks for all thirty-two fault codes in a single prompt gives 0.4792 accuracy with 48 calls; the tree of thoughts, which splits the solution into steps and asks for several extensions at every step, falls to 0.0833 at branch 1, rises to 0.6250 at branch 3 and stops at branch 4. As branching goes from 3 to 4, accuracy slips by 0.0208 while the number of calls goes from 795 to 1086 and tokens from 43182 to 59835; since this difference is smaller than the 0.0417 difference between two runs of the same arrangement, it counts as unmeasured.
  4. 04 Step-Back Prompting Measuring what it is worth to ask for the principle before solving: on the same 48 fictional support tickets, the single-stage prompt gives 0.5417 code accuracy with 48 calls, while the two-stage prompt, which first asks for the general rule and then asks for the solution, gives 0.7500 and does it with 96 calls and 1.66 times the tokens. The whole gain depends on the first stage: under the right principle code accuracy is 0.8000, and when a wrong principle is imposed it drops to between 0.0000 and 0.1458, that is, below the single-stage prompt. The same arrangement does not move accuracy at all in a task that asks only for the degree, but it raises the tokens to 2.69 times.
  5. 05 Reason and Act Measuring what it takes to ask for the thought field and the action field together in the same prompt: on 120 fictional support tickets, when an action field is requested only 0.1000 of the responses produce a valid action field; when the action set is written into the prompt, 0.5250; when the thought is cut to a single sentence, 0.8000. The cost rises from 5300 to 8060 in input tokens and falls from 645.8 tokens to 102.1 tokens per valid action field. The two-run comparison leaves the difference of writing the action set unmeasurable.
  6. 06 Reasoning Models The yardstick for the class of models that produce invisible intermediate steps before writing a response: on the same 120 fictional support tickets, in the single-field task the accuracy difference between the two classes stays at −0.0333 and +0.1167; because the two-run difference is 0.1167 it counts as unmeasured, and tokens rise 1.75 times. In the two-field task the difference rises above the band with +0.1000 and +0.1667; the single-pass class contradicts its own label in the code field in 0.2333 of the responses. The cost per correct response rises from 52.0 to 95.1 and from 83.9 to 110.1 tokens.
  7. 07 Prompt Ensemble The same 120 fictional support tickets are examined with three separate instructions, and the assumption the ensemble rests on is tested: do different prompts make different errors. The three prompts alone give 0.7750, 0.6583 and 0.7833, on nine records all three are wrong at once and the ceiling comes out at 0.9250. The majority vote stays at 0.8000: 0.1250 below the ceiling, only 0.0167 above the best single prompt, and this gain is below the 0.0333 to 0.1333 difference the same prompts show between two runs. The cost is written in number of calls: 120 calls become 360 calls, and 61.1 tokens per correct label become 180.7 tokens.
  8. 08 Automatic Prompt Optimization Instead of being written by hand, the prompt is searched with a criterion: ten candidates are produced mechanically from four parts, and the fictional support tickets are split into a 40-record validation set and an 80-record test set. As the search budget goes from one candidate to ten, the best score on validation rises from 0.7750 to 0.9000 and the test score of the same candidate falls from 0.6750 to 0.6500: validation gain +0.1250, test gain -0.0250, and the chosen candidate's optimism margin 0.2500. The search goes from 40 calls to 400 calls, the candidate that is truly best on the test (0.8000) is never chosen, and in the second run the search chooses a different candidate.

03

Output Control

Measuring how the output of an uncertain generator is constrained in form and in content: the combined behavior of temperature, top-p (nucleus) and top-k, and which of them narrows the difference between two runs; the repetition rate and the number of prematurely cut responses when a repetition penalty and a stop sequence are added; the schema match rate and token cost of free text, a format description and an imposed schema; the schema and business rule violations that validation catches, the responses that retrying recovers and the extra calls it costs; a wrong response that is sometimes fixed by a better instruction and sometimes only by missing information; how much balancing through the prompt narrows the subgroup gap and that it does not solve data bias.

  1. 01 Sampling Parameters Measuring temperature, top-p (nucleus) and top-k together on the same prompt: label accuracy 0.6417 and two-run difference 0.5667 on the base setting, 0.8750 and 0.0417 at temperature 0.2, 0.8417 and 0.1333 at top-p 0.50, 0.7833 and 0.3583 at top-k 3, 0.8667 and 0.0083 when all three narrow together. The fixed cut leaves three candidates on average while the top-p cut leaves 1.20 candidates. Input tokens stay constant at 5300 and the call count stops at 120, so the cost of the change is thirteen output tokens.
  2. 02 Penalties and Stop Sequences Measuring the repetition penalty and the stop sequence on one response set: on the base scaffolding the repetition rate is 0.3431 and 107 of 120 responses are truncated at the length limit; a negative constraint written into the prompt lowers the rate to 0.2167 while raising the input tokens from 2734 to 3574, and a 0.45 repetition penalty does the same job with zero extra tokens, down to 0.1972. The stop sequence lowers the number of truncated responses to 55, and to 24 together with the penalty, and brings the output tokens from 4073 to 2653, but it cuts twenty-five responses prematurely, before the label sentence is written, and lowers usability from 0.8833 to 0.7417. The two-run difference on the stop sequence row is 0.0543, and the 0.0746 gain over the base is marginal.
  3. 03 Structured Output Measuring free text, a format description written into the prompt and an imposed schema side by side on the same 120 fictional tickets: the rate of schema-conforming responses is 0.0000 in free text, 0.8000 with the format description, 0.8833 when an example is added and 1.0000 with the imposed schema. The two-run difference of the format description row is 0.0833 and, since the gain from adding an example is also 0.0833, that gain counts as unmeasured, and it adds 2040 tokens. Topic hit rate stays fixed at 0.8833 at all four levels. Input tokens rise from 2254 to 5614 with the format description and to 7654 with the example, while the imposed schema stays at 2974; that is, the cheapest prompt gives the highest conformance.
  4. 04 Output Validation Counting schema validation and business rule checking separately and building a retry loop on top: 24 of 120 fictional responses do not conform to the schema, 24 of the 96 records that pass the schema get caught on a business rule, and with no validation the faulty record rate is 0.4000. The two checks together bring the acceptance rate down to 0.6000, one retry recovers 26 records and raises it to 0.8167 at a cost of 48 extra calls. The second retry recovers eight more records but its gain of 0.0666 stays below the two-run difference of 0.0667. The retry that writes the error into the prompt recovers 23 records and adds 514 tokens.
  5. 05 Prompt and Context Engineering Difference Comparing two separate fixes on the same set of errors: on 120 fictional tickets the baseline topic hit rate is 0.8833, the fault code hit rate is 0.4083, and both correct is 0.4083. Writing a distinguishing rule into the prompt raises the topic hit rate to 0.9667 and does not touch the code hit rate at all; putting the fictional code table into the prompt raises the code hit rate to 0.9167 and does not touch the topic hit rate at all. Of the fourteen topic errors ten are fixed by the instruction alone, of the seventy-one code errors sixty-six are fixed by information alone, and one error in each class is fixed by neither. The costs diverge: the instruction adds 1680 tokens, the information adds 14520.
  6. 06 Bias-Reducing Prompts Measuring how much of a subgroup gap prompt balancing closes and what it leaves: on 200 fictional tickets the central hit rate is 0.9016, the rural hit rate 0.5513 and the gap 0.3504. A bias-reducing notice in the prompt moves the gap to 0.3586 and adds 3800 tokens; a balanced example brings it down to 0.2888 and adds 14400 tokens. The hit rate of the sampled local spelling rises from 0.4848 to 0.8788, while the unsampled one stays fixed at 0.3023. The prompt closes only a 0.1758 share of the gap. The course wrap-up gathers the previous prompt, changed prompt and cost rows of twenty-two lessons.

Final exam

Prompt Engineering — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close