Unit 02 of 02
Computer Vision
Measuring each task against its own baseline on fictional images generated from the coverage grid: the contribution of normalization and augmentation and the transform that breaks the label in augmentation, end-to-end training against the mean image baseline and how many examples it takes for transfer to add something, the number a fixed box placed in the middle of the image gets on the overlap metric and the fact that both the method and the fixed box drop as the threshold rises from 0.3 to 0.7, the baseline that assigns every pixel to the dominant class scoring high because of pixel imbalance and semantic segmentation being unable to separate two adjacent objects, the inconsistency of independent per-frame decisions against a decision that uses time across consecutive frames, the success of a joint representation that matches a note with its image against random matching, and the share a single modality gives on its own.
Published Updated
Lessons
- 01 Image Data and Preprocessing Measuring how much the way a pixel is turned into a number carries: when the same two procedures are read on three representations, dataset-wide scaling moves no number (0.5500 and 0.6937 stay as they are), while per-image normalization raises the threshold rule from 0.6937 to 0.9250 and lowers the mean image baseline from 0.5500 to 0.4875. Of five augmentation transforms, the quarter turn flips the label in 160 of 160 examples, and the shift breaks it in 1. With forty labels, nearest neighbor reads 0.5938 and 0.6250 without augmentation, two flips read 0.6875 and 0.6250, and a quarter turn added with its label kept reads 0.5188 and 0.5000: below the 0.5312 of the most frequent class. 13 min read
- 02 Image Classification Measuring end-to-end training and transfer learning against the domain baseline: while the most frequent class reads 0.5312 and the mean image 0.5500, the network with 1,169 parameters reads 0.6000 in both runs on 144 pixels, and the network with 305 parameters reads 0.6687 and 0.7063 on 36 pooled numbers; both memorize the training set completely. With twenty labels, training from scratch stays below the domain baseline at 0.5188 and 0.5250, while freezing a pretrained model produced from a fictional source task gives 0.6312 and 0.5500, and fine-tuning gives 0.5938 and 0.5625. At sixty labels the advantage closes, and at one hundred sixty training from scratch pulls ahead with 0.6125 and 0.6875; the gap of 0.0812 between the two runs of freezing is larger than the difference transfer adds. 13 min read
- 03 Object Detection Reading intersection over union and its threshold together with a procedure that looks at no image: a fixed 4x4 box placed in the middle of the grid gets a mean overlap of 0.3011 and counts as a detection in 0.5125 of the test images at a threshold of 0.3, but falls to 0.1938 at a threshold of 0.5 and to 0.0125 at 0.7. A fixed-size window with 81 candidates that searches only position reads 0.9750, 0.6875, and 0.2375 with a mean of 0.5687; a window with 2,025 candidates that also searches size reads 0.9812, 0.9625, and 0.8500 with a mean of 0.9076. The two procedures are separated by 0.0062 at a threshold of 0.3, and the gap between them rises to 0.6125 at a threshold of 0.7. When the confidence threshold is pulled to the score of the first quarter of the validation set, precision rises from 0.9812 to 1.0000 and recall drops to 0.7063. 7 min read
- 04 Segmentation Reading the decision at the pixel level against its own baseline: because 2,193 of the 17,280 pixels in the test set are object, the baseline that assigns every pixel to the dominant class gets 0.8731 in pixel accuracy, while the object intersection over union of the same output is 0.0000. A linear model deciding from a single pixel value gives 0.9540 and 0.6412, and the same model with a 3x3 neighborhood window gives 0.9851 and 0.8843; the difference comes not from the model but from the representation. 45 of the one hundred twenty images carry adjacent blocks, and even with a perfect mask, counting connected components gets the instance count wrong in these 45 images; rectangle decomposition fixes 24 of them and stops at 21, while on the predicted mask the wrong count rises from 46 to 79. 4 min read
- 05 Video and Temporal Models Counting the instability over time of the per-frame decision: on the 432 frames of thirty-six test sequences, the baseline that writes the dominant class into every frame gives 0.7014 accuracy and 0 instability, while the true label sequences carry 71 transitions. The independent frame decision produces 99 transitions with 0.8634 accuracy, which is 28 more than the truth. When a transition penalty is applied to the same scores, accuracy rises to 0.8657 at a penalty of 0.05, instability settles on the truth's 71 at a penalty of 0.15, and at a penalty of 4.00 the decision falls to the baseline itself, 0.7014 and 1 transition: on this metric the domain baseline is not a separate procedure but the extreme value of the penalty. 4 min read
- 06 Multimodal Models Measuring a joint representation, which brings the note and the coverage frame of the same event together in a single space, against its own baseline: in rooms of twenty-five candidates, random matching gives 0.0400 while the joint representation learned with a contrastive objective gives 0.1660, and the ceiling of the shared information carried by the fiction is 0.2659. On the event kind decision, the most frequent class baseline is 0.4420, note only 0.8940, frame only 0.6140, joint representation 0.8660: the representation that wins at matching stays below what the note alone gives in classification. The course wrap-up collects the domain baseline of fourteen lessons, the method's number, and the difference in a single table. 10 min read