Skip to content
academia.sh
tr
Log in

Course Beginner

Introduction to Machine Learning

Published Updated

By the end of this course

Start course

01

Fundamental Concepts

Counting what a procedure that learns its decision from data does and how much it gains, together with the baseline: the difference between a hand-written rule set and a learner that derives its rules from data on the same problem, the number of labels asked for by supervised, unsupervised, semi-supervised and reinforcement setups, what a pretext task whose label is derived from the data adds to the actual task, the four separate baselines of classification, regression, clustering and ranking problems, the number of returns in the workflow from problem definition to monitoring, and the same problem turning into a different output in three roles.

  1. 01 What Is Machine Learning The same illicit-use suspicion problem is solved with three procedures and three numbers are printed side by side: the non-learning baseline scores 0.7533 on the held-out set, the hand-written rule set written by the field expert scores 0.7507, and the three-question decision tree that derives its rules from data scores 0.8594. The hand-written rule falls 0.0027 below the baseline, the tree rises 0.1061 above it. The difference does not come from the thresholds: when the same thresholds are moved into the hand-written rule, the number drops to 0.7480. Of the 377 subscribers in the held-out set, 93 are suspicious; the hand-written rule flags 151 subscribers and 76 turn out empty, the tree flags 66 subscribers and 13 turn out empty. The dataset is reduced from a readings table of 3,199 rows to a subscriber table of 1,260 rows, seed 20260218.
  2. 02 Types of Learning The same illicit-use suspicion problem is set up in four setups and the number of labels each setup asks for is counted: supervised 0.8594 on the held-out set with 883 labels, unsupervised 0.8117 with 6 labels, semi-supervised 0.8090 with 60 labels, reinforcement 0.8090 with 120 visits. The baseline is 0.7533. As the label budget falls from 883 to 30, the held-out figure drops from 0.8594 to 0.7745, while the training figure rises from 0.8709 to 0.9400. The two-cluster split sits exactly on the baseline, the six-cluster split gains 0.0584. The visit policy finds 73 illicit cases in 120 visits; if the same budget were spent at random, 25 would be found.
  3. 03 Self-Supervised Learning The label is derived from the data: the last reading of each subscriber is hidden and predicted from the earlier readings. On the pretext task the baseline gives a mean absolute error of 6.3776 m³ on the held-out set and the least squares model 5.9661 m³ — a gain of 0.4115 m³, obtained with zero labels. When the three numbers produced by the pretext task are carried to the main task, the 0.7851 measured with the record features alone rises to 0.8037. When the same representation is built from the baseline of the pretext task, it gives 0.8064, so the contribution of the learned part is minus 0.0027: the difference comes not from what the pretext task learned but from the raw reading that was carried along. The nine hand-derived features still give 0.8594.
  4. 04 Application Areas Four questions are asked of the same fictional subscriber table and four separate baselines are chosen: in classification the most frequent class gives 0.7533 on the held-out set and the tree 0.8594; in regression the overall mean gives a mean absolute error of 5.2255 m³ and least squares 4.1275 m³; in clustering a random four-way grouping gives a within-cluster distance of 2.7683 and the four learned centers 2.1551; in ranking a random order finds 20 illicit cases in the first 100 visits, an order starting from the highest consumption finds 30, and an ensemble of nine trees finds 71. Of the four measures, a large number is good in two and a small number is good in two; the numbers can be compared not with one another but only with their own baselines.
  5. 05 Machine Learning Workflow The seven-step flow produces three returns, and the number each return moved is printed. The gate at the fifth step tests the gap over the baseline with a threshold of 0.0500: the model built with three columns gives 0.0318 and stays at the gate, with nine columns it gives 0.1061 and passes. At the sixth step the model flags 240 of 1,260 subscribers, the daily capacity is 40 visits and the list spreads over six days; the constraint turns the problem from classification into ranking, and the ranking finds 30 illicit cases in the first 40 visits, the random order 14. At the seventh step the measurement policy changes, the label does not change, yet the same model falls to 23 and returns to 30 when rebuilt on the drifted data. The seven-step flow is completed with a total of 22 step visits.
  6. 06 Roles The same illicit-use problem turns into three separate outputs and three separate baselines in three job definitions. The baseline of the data scientist is the field's current procedure and finds 13 illicit cases in the first 40 visits, the model order delivered finds 30, the gap is 17. The baseline of the machine learning engineer is the model running today and finds 23 on the drifted data, the rebuilt procedure finds 30, the gap is 7. The baseline of the AI engineer is the system without a model and finds 21 in the worst case, the system with a signal finds 30, the gap is 9. The same delivered count produces 17, 7 and 9 in three roles; what determines the number is not the model but the chosen baseline. The system that falls back gives 21 on the drifted data, the one that never falls back 23.

02

Generalization

Measuring what a model will do on data it has not seen and how this measure gets corrupted: the reasoning behind separating the training, validation and test sets and how many times the test set may be looked at, the point where the training and held-out numbers diverge as capacity rises, how much the same procedure swings across different training samples, the two numbers moving in opposite directions as the penalty coefficient grows and reading the best point from the validation set, and how much four kinds of leakage inflate the held-out number.

  1. 01 Training, Validation and Test Sets Measuring that the same model gives two different numbers on data it has seen and data it has not seen: 1,260 subscribers are split 756/252/252, a depth 6 decision tree gives 0.8466 accuracy on training, 0.7897 on validation and 0.7778 on test, and the baseline is also three different numbers on the three sets: 0.7778, 0.8135, 0.7579. As selection is made among 48 candidates produced by the same procedure, the gap between the number on the selection set and the number on the unseen set grows from 0.0211 to 0.0411. On a time-ordered split the test accuracy drops to 0.6865, but the difference above the baseline grows from 0.0198 to 0.0397.
  2. 02 Overfitting and Underfitting Measuring how the training and held-out numbers diverge as capacity is raised step by step: as the depth of the decision tree goes from 1 to 14, the training accuracy rises from 0.7778 to 0.9352, the held-out accuracy peaks at 0.8214 and falls to 0.7302, and the difference between the two numbers widens from -0.0357 to +0.2050. Only four of the eleven capacities can match the baseline of 0.8135 on the held-out set. When the training set is raised from 100 to 756 rows, the difference falls from 0.2219 to 0.1680; when the minimum leaf size is raised from 2 to 80, it falls from 0.1680 to -0.0198 and the model turns into exactly the baseline.
  3. 03 Bias–Variance Tradeoff Splitting the error on the held-out set into two components: on training sets built by sampling 30 times with replacement, the total squared error comes to 41.242 for the baseline, 29.768 for the depth 3 tree and 33.810 for the depth 12 tree, in m³ squared; the variance component rises from 0.040 to 4.409 while the bias and noise component falls from 41.202 to 28.448 and stops there. The bias that capacity can remove is 12.754 m³ squared, and the remaining 28.4 m³ squared cannot be removed by any capacity. For the same subscriber, the mean range of the predictions given by 30 models is 3.587 m³ at depth 1 and 8.172 m³ at depth 12.
  4. 04 Regularization Intuition Measuring how penalizing complexity moves two numbers in opposite directions: in a linear model with 29 columns on 120 rows, when the penalty coefficient is raised from 0.001 to 1000 the weight magnitude falls from 10.040 to 0.601 and the training error worsens from 4.603 to 5.453 m³, while the validation error falls from 5.924 to 5.553 and returns to 5.932. Decomposed by sampling with replacement, the variance falls from 12.988 to 0.181, while the bias hits a floor at 30.976 and jumps to 35.295. The best penalty read from validation is 100 and gives 5.559 m³ on the test set; the baseline is 6.556 and the unpenalized model 5.841 m³.
  5. 05 Data Leakage Measuring how sources other than learning inflate the held-out number: against the honest line, which gives an accuracy of 0.7778 on the test set with a baseline of 0.7579, scaling before the split inflates this number by 0.0119, a feature that sees the future by 0.0754, a duplicated record by 0.0542 and a feature derived from the target by 0.1429; all four have the same sign. A model whose features have been replaced with six random columns reads an accuracy of 0.8757 on the held-out set while the duplicates are present and beats the baseline of 0.7857 by 9 points; on the rows whose twin it has not seen, it stays 9.2 points below the baseline. At the course wrap-up, the baseline, training and held-out numbers of eleven lessons are gathered in a single table.

Final exam

Introduction to Machine Learning — Final Exam

In preparation

This course's question bank is not complete yet. The exam opens here automatically once enough questions are added.

Start typing to search.

↑↓ ↵ Esc navigate · open · close