Buffers

In the previous lesson, chunks coming from a stream showed up with type Buffer, and the encoding name was given by hand when converting to text. What happens if a chunk splits in the middle of a multi-byte character was left unasked.

This lesson answers that question. The subject is a direct continuation of the character encoding and byte order lessons from the How Computers Work course: the concepts established there turn into working code here.

A Buffer as a Byte Array

A buffer is an object representing a fixed-length raw byte array. It derives from the language’s Uint8Array type, so its indexed access, length, and iteration behavior are the same as a typed array. What is added on top is encoding conversions and methods for reading and writing numeric fields.

The difference between a string and a buffer shows up in the unit of measure:

// buffers.mjs
const text = 'sıcaklık';

const buf = Buffer.from(text, 'utf8');
console.log('character count :', text.length);
console.log('byte count      :', buf.length, '=', Buffer.byteLength(text, 'utf8'));
console.log('hex             :', buf.toString('hex'));
console.log('is Uint8Array   :', buf instanceof Uint8Array);
console.log('first three bytes:', [...buf.subarray(0, 3)]);

node buffers.mjs

character count : 8
byte count      : 10 = 10
hex             : 73c4b163616b6cc4b16b
is Uint8Array   : true
first three bytes: [ 115, 196, 177 ]

An eight-character string held ten bytes; the letter ı is encoded in UTF-8 with two bytes (c4 b1) and appears twice in the word. This distinction matters when writing a Content-Length header on an HTTP response: the number that has to go in the header is the byte count, not the character count. Buffer.byteLength gives this number without producing a buffer.

Allocation, View, and Copy

There are three ways to produce a buffer, and the difference between them is in memory behavior.

// allocation.mjs
const zeroed = Buffer.alloc(8);
console.log('alloc(8)          :', zeroed.toString('hex'));
console.log('allocUnsafe(8) len:', Buffer.allocUnsafe(8).length);

// subarray is not a copy, it is a view opened onto the same memory
const full = Buffer.from('edge-01', 'utf8');
const view = full.subarray(0, 5);
view[0] = 'E'.charCodeAt(0);
console.log('view changed ->', full.toString('utf8'));

// A copy is requested explicitly when one is needed
const copy = Buffer.from(full.subarray(0, 5));
copy[0] = 'x'.charCodeAt(0);
console.log('copy changed ->', full.toString('utf8'), '|', copy.toString('utf8'));

// Joining pieces into a single buffer
const joined = Buffer.concat([Buffer.from('edge'), Buffer.from('-'), Buffer.from('01')]);
console.log('concat        ->', joined.toString('utf8'), `(${joined.length} bytes)`);

node allocation.mjs

alloc(8)          : 0000000000000000
allocUnsafe(8) len: 8
view changed -> Edge-01
copy changed -> Edge-01 | xdge-
concat        -> edge-01 (7 bytes)

Buffer.alloc gives zeroed memory of the requested size. Buffer.allocUnsafe skips the zeroing step and is faster; in exchange, the buffer contains whatever was previously in that memory. Using this buffer without writing over its content start to finish can leak data found elsewhere in the process. Rule: use zeroed allocation unless you are going to fill the buffer immediately and completely.

The third line of the output shows view behavior: subarray does not allocate new memory, it returns a window opened onto the same memory. Writing to the window changes the original buffer. This is valuable in stream code because it allows slicing without copying; unnoticed, it produces hard-to-explain bugs. When an independent copy is needed, it is requested explicitly with Buffer.from.

Buffer.concat joins a list of pieces into a single buffer. This is the standard way to gather chunks coming from a stream and convert them to a single piece of text at the end; converting pieces to strings and joining them leads to the problem below.

Splitting at a Character Boundary

A chunk can end in the middle of a multi-byte character. If that chunk is converted to text on its own, a replacement character is put in place of the incomplete character, and information is lost irreversibly.

// boundary.mjs
import { StringDecoder } from 'node:string_decoder';

const buf = Buffer.from('sıcaklık', 'utf8');   // 10 bytes
const chunkA = buf.subarray(0, 2);              // cuts through the middle of 'ı'
const chunkB = buf.subarray(2);

console.log('direct decode :', chunkA.toString('utf8') + chunkB.toString('utf8'));

const decoder = new StringDecoder('utf8');
console.log('with decoder  :', decoder.write(chunkA) + decoder.write(chunkB) + decoder.end());

node boundary.mjs

direct decode : s��caklık
with decoder  : sıcaklık

The two replacement characters on the first line came from the two bytes of the split ı being decoded separately. The decoder in the node:string_decoder module keeps an incomplete byte sequence in itself and joins it with the next write; the end call reports any leftover incomplete sequence.

This exposes a hidden assumption in the previous lesson’s LineSplitter class. There, every chunk was converted directly with toString('utf8'); because the measurement file only contains ASCII characters, no problem was visible. The same code breaks on a dataset with non-ASCII characters in node names. The correct fix is keeping a decoder in the transform and writing decoder.write(chunk) instead of chunk.toString('utf8').

Binary Record Format

A buffer offers methods that read and write numeric fields at a given width and byte order. This is how you parse a binary protocol coming over the network, and how you define your own format to save space.

A measurement record is fit into six bytes below: one byte for the node number, one byte for the metric number, four bytes for a single-precision floating-point value.

// binary.mjs
// Binary format fitting a measurement into 6 bytes:
// [0] node no (uint8)  [1] metric no (uint8)  [2..5] value (float32, big-endian)
function encode(nodeNo, metricNo, value) {
  const buf = Buffer.alloc(6);          // zeroed memory
  buf.writeUInt8(nodeNo, 0);
  buf.writeUInt8(metricNo, 1);
  buf.writeFloatBE(value, 2);
  return buf;
}

function decode(buf) {
  return {
    nodeNo: buf.readUInt8(0),
    metricNo: buf.readUInt8(1),
    value: Number(buf.readFloatBE(2).toFixed(2)),
  };
}

const record = encode(1, 0, 21.4);
console.log('binary form   :', record.toString('hex'));
console.log('decoded record:', decode(record));

const littleEndian = Buffer.alloc(4);
littleEndian.writeFloatLE(21.4, 0);
console.log('little-endian :', littleEndian.toString('hex'));
console.log('big-endian    :', record.subarray(2).toString('hex'));

const textSize = Buffer.byteLength(JSON.stringify({ node: 'edge-01', metric: 'temperature', value: 21.4 }));
console.log(`text form ${textSize} bytes, binary form ${record.length} bytes`);

node binary.mjs

binary form   : 010041ab3333
decoded record: { nodeNo: 1, metricNo: 0, value: 21.4 }
little-endian : 3333ab41
big-endian    : 41ab3333
text form 54 bytes, binary form 6 bytes

The BE and LE suffixes in the method names select the big-endian and little-endian layouts introduced in the How Computers Work course. The third and fourth lines of the output show the same number’s two layouts side by side: the byte sequence is reversed. The conventional choice in network protocols is big-endian layout; when defining a format, which endianness is used has to be written down, or the two ends read each other’s data wrong.

The last line shows why binary formats exist: the same measurement takes 54 bytes as text, 6 bytes as binary. In exchange, the binary form is unreadable by a human, does not describe itself, and breaks old decoders when a field is added. This trade-off was discussed in the How the Internet Works course while justifying why HTTP is text-based; the choice is between size and inspectability.

Squeezing the value into four bytes has one more cost. Single-precision floating-point representation carries about seven significant decimal digits; the value 21.4 cannot be represented exactly in this form. This is why the example rounds it with toFixed(2). If measurement precision needs to exceed this limit, double-precision read and write methods are used.

Buffer methods do bounds checking: a call that tries to write outside the array’s range throws RangeError. This blocks, at the runtime level, the class of unbounded-memory-access errors covered in the How Computers Work course.

Summary

• A buffer is a raw byte array built on Uint8Array; character count and byte count diverge in multi-byte encodings, and it is the byte count that goes into protocol headers.
• Buffer.alloc gives zeroed memory; allocation that skips zeroing leaks old memory content when not filled completely.
• subarray returns a view, not a copy; an independent copy is requested explicitly with Buffer.from.
• When a multi-byte character splits at a chunk boundary, direct decoding corrupts the data; node:string_decoder prevents this by holding onto partial bytes.
• Numeric fields are read and written by choosing a width and an endianness; a binary format saves space at the cost of inspectability and extensibility.

Next Step

Up to here, streams were said to emit events, but the event mechanism itself was never opened up. The measurement collector’s next step needs a structure that notifies interested parties when data arrives: a mechanism that produces a warning when a threshold is crossed, reports a broken line, announces that the source has closed. The next lesson covers this mechanism’s base class — the event emitter.