Remote Procedure Call and gRPC

The previous lesson tested REST’s constraints. What those constraints have in common is that they lean the contract on the rules of the network: addresses, methods, status codes. The contract’s detail — a field’s type, whether a field is required — stays outside the wire, in the documentation.

There is an approach that does the opposite. Remote procedure call presents an operation over the network as though it were a local function call: the client calls a function named GetBook, the call goes out over the network, runs on the other side, and returns a result. For this presentation to work, the two sides must agree on the call’s signature, and this agreement lives not in documentation but in a machine-readable schema.

The Limit of the Procedure Call Analogy

The analogy is useful but not exact, and knowing where it breaks down determines the interface’s design. A local call is too fast to measure; a remote call takes at least one network round trip in the best case. A local call either runs or throws an exception; a remote call can produce a third outcome — partial failure: it is not known whether the request went out, or whether the work was done. In a local call, arguments are objects in memory; in a remote call, they have to be converted into a byte sequence, and the rules of that conversion are the contract itself.

These three differences show up in the design of interfaces that use the remote-procedure-call style: calls are kept coarse-grained (a small number of full calls instead of a large number of small ones), retry behavior is defined explicitly, and the data format is tightly specified.

The Contract Is Written First

The schema file is the interface’s single source of truth. The following file defines a message type and a service; every field has a number, a type, a name, and a requiredness.

message Book {
  1 string  isbn      required
  2 string  title     required
  3 int     pages     optional
  4 bool    onLoan    required
}
service Catalog {
  GetBook(GetBookRequest) -> Book
  SearchBooks(SearchRequest) -> stream Book
}

Two things are generated from this file: a validator that checks the presence and type of fields, and an encoder that converts an object into a byte sequence. The encoder’s format never puts field names on the wire at all; every field is referred to by its own number, and numbers are written at variable length.

// generate.mjs — generates a validator and a binary encoder from the schema file (the contract comes first).
import { readFileSync } from "node:fs";

export const readSchema = (file) => {
  const messages = {}, services = {};
  let open = null;
  for (const raw of readFileSync(file, "utf8").split("\n")) {
    const line = raw.trim();
    if (!line) continue;
    let e;
    if ((e = /^message (\w+) \{$/.exec(line))) { open = messages[e[1]] = []; continue; }
    if ((e = /^service (\w+) \{$/.exec(line))) { open = services[e[1]] = []; continue; }
    if (line === "}") { open = null; continue; }
    if ((e = /^(\d+) (\w+)\s+(\w+)\s+(\S+)$/.exec(line)))
      open.push({ number: +e[1], type: e[2], name: e[3], required: e[4] === "required" });
    else if ((e = /^(\w+)\((\w+)\) -> (stream )?(\w+)$/.exec(line)))
      open.push({ method: e[1], input: e[2], stream: Boolean(e[3]), output: e[4] });
  }
  return { messages, services };
};

const TYPES = { string: "string", int: "number", bool: "boolean" };

// Generated validator: checks fields for presence and type against the schema.
export const makeValidator = (fields) => (obj) => {
  const defects = [];
  for (const f of fields) {
    const v = obj[f.name];
    if (v === undefined) { if (f.required) defects.push(`${f.name}: required field missing`); continue; }
    if (typeof v !== TYPES[f.type]) defects.push(`${f.name}: expected ${f.type}, got ${typeof v}`);
  }
  for (const name of Object.keys(obj))
    if (!fields.some((f) => f.name === name)) defects.push(`${name}: not in schema`);
  return defects;
};

// Variable-length integer: small numbers take fewer bytes.
const writeVarint = (n) => {
  const b = [];
  do { let p = n & 0x7f; n >>>= 7; b.push(n ? p | 0x80 : p); } while (n);
  return Buffer.from(b);
};
const readVarint = (b, i) => {
  let n = 0, k = 0;
  while (b[i] & 0x80) { n |= (b[i++] & 0x7f) << k; k += 7; }
  return [n | (b[i++] << k), i];
};

// Binary framing: each field is written with a (number<<3 | wireType) key; the field name never reaches the wire.
export const makeEncoder = (fields) => ({
  encode(obj) {
    const parts = [];
    for (const f of fields) {
      const v = obj[f.name];
      if (v === undefined) continue;
      const wireType = f.type === "string" ? 2 : 0;
      parts.push(writeVarint((f.number << 3) | wireType));
      if (wireType === 0) parts.push(writeVarint(f.type === "bool" ? (v ? 1 : 0) : v));
      else { const buf = Buffer.from(v, "utf8"); parts.push(writeVarint(buf.length), buf); }
    }
    return Buffer.concat(parts);
  },
  decode(bytes) {
    const obj = {};
    let i = 0;
    while (i < bytes.length) {
      let key; [key, i] = readVarint(bytes, i);
      const field = fields.find((f) => f.number === key >> 3);
      if ((key & 7) === 2) {
        let length; [length, i] = readVarint(bytes, i);
        obj[field?.name ?? `unknown_${key >> 3}`] = bytes.toString("utf8", i, i + length);
        i += length;
      } else {
        let value; [value, i] = readVarint(bytes, i);
        obj[field?.name ?? `unknown_${key >> 3}`] = field?.type === "bool" ? Boolean(value) : value;
      }
    }
    return obj;
  },
});

Measurement

The measurement answers four questions: what surface is generated from the schema, which records the validator rejects, how many bytes the binary format takes, and what happens to the bytes on the wire when a field name changes?

// measure.mjs — tests the schema-first contract: validation, binary/text byte comparison, field renaming
import { writeFileSync } from "node:fs";
import { readSchema, makeValidator, makeEncoder } from "./generate.mjs";

const { messages, services } = readSchema("library.schema");
const fields = messages.Book;
const validate = makeValidator(fields);
const { encode, decode } = makeEncoder(fields);

console.log("-- 1) service surface generated from the schema --");
for (const y of services.Catalog)
  console.log(`  ${y.method}(${y.input}) -> ${y.stream ? "stream " : ""}${y.output}`);

console.log("\n-- 2) generated validator --");
const SAMPLES = [
  { isbn: "978-0262033848", title: "Introduction to Algorithms", pages: 1312, onLoan: false },
  { isbn: "978-0201896831", title: "The Art of Computer Programming", onLoan: true },   // no optional field
  { isbn: "978-0131103627", onLoan: true },                                             // required field missing
  { isbn: "978-0596007126", title: "Head First", pages: "272", onLoan: true },          // wrong type
  { isbn: "978-0000000000", title: "Test", onLoan: true, shelf: "R-12" },               // field not in schema
];
for (const o of SAMPLES) {
  const defects = validate(o);
  console.log(`  ${JSON.stringify(o).slice(0, 52).padEnd(54)} ${defects.length ? "FAIL  " + defects[0] : "PASS"}`);
}

console.log("\n-- 3) byte count of binary framing vs. text format --");
const valid = SAMPLES.filter((o) => validate(o).length === 0);
for (const o of valid) {
  const binary = encode(o), text = Buffer.from(JSON.stringify(o), "utf8");
  console.log(`  ${o.isbn}  binary=${String(binary.length).padStart(3)} B  ` +
    `text=${String(text.length).padStart(3)} B  gain=%${Math.round(100 * (1 - binary.length / text.length))}`);
}
const totalBinary = valid.reduce((t, o) => t + encode(o).length, 0);
const totalText = Buffer.byteLength(JSON.stringify(valid));
console.log(`  total           binary=${totalBinary} B  text=${totalText} B  ` +
  `gain=%${Math.round(100 * (1 - totalBinary / totalText))}`);

console.log("\n-- 4) no field name on the wire: the schema changes the field's name --");
const encoded = encode(valid[0]);
console.log(`  decoded with old schema : ${JSON.stringify(decode(encoded))}`);
writeFileSync("new.schema", `message Book {
  1 string  isbn      required
  2 string  heading   required
  3 int     pages     optional
  4 bool    onLoan    required
}
`);
const renamed = makeEncoder(readSchema("new.schema").messages.Book);
console.log(`  decoded with new schema : ${JSON.stringify(renamed.decode(encoded))}`);
console.log(`  same bytes, ${encoded.length} B; the only thing that changed is the name attached to the field number.`);

-- 1) service surface generated from the schema --
  GetBook(GetBookRequest) -> Book
  SearchBooks(SearchRequest) -> stream Book

-- 2) generated validator --
  {"isbn":"978-0262033848","title":"Introduction to Al   PASS
  {"isbn":"978-0201896831","title":"The Art of Compute   PASS
  {"isbn":"978-0131103627","onLoan":true}                FAIL  title: required field missing
  {"isbn":"978-0596007126","title":"Head First","pages   FAIL  pages: expected int, got string
  {"isbn":"978-0000000000","title":"Test","onLoan":tru   FAIL  shelf: not in schema

-- 3) byte count of binary framing vs. text format --
  978-0262033848  binary= 49 B  text= 90 B  gain=%46
  978-0201896831  binary= 51 B  text= 81 B  gain=%37
  total           binary=100 B  text=174 B  gain=%43

-- 4) no field name on the wire: the schema changes the field's name --
  decoded with old schema : {"isbn":"978-0262033848","title":"Introduction to Algorithms","pages":1312,"onLoan":false}
  decoded with new schema : {"isbn":"978-0262033848","heading":"Introduction to Algorithms","pages":1312,"onLoan":false}
  same bytes, 49 B; the only thing that changed is the name attached to the field number.

What the Measurement Says

Validation is generated from the contract, not hand-written. Three of the five records were rejected, and the three rejections are of three separate kinds: a missing required field, a wrong type, a field not in the schema. None of these checks were written in the application code; they came out of the lines in the schema file. When the schema changes, validation changes with it, and the two sides’ rules cannot drift apart.

The binary format reduces the same record to roughly half the bytes. The source of the gain is not compression but the unnecessary never being written at all: field names, quotation marks, colons, and commas never reach the wire. Numbers, too, are written not as text but as a variable-length integer — 1312 takes two bytes instead of four characters.

The field number is the identity, not the field name. The fourth part repeats a change considered breaking in the first lesson: the title field is renamed to heading. In the text format, this change had broken the consumer. In the binary format, the same 49-byte sequence decodes without any trouble under the new schema; only the key in the output changes. The identity on the wire is the number 2.

The mirror image of this is also true, and more dangerous: changing the number breaks silently. If two fields’ numbers are swapped, the validator says nothing, because the bytes are valid; only the values land in the wrong fields. This is where schema-first design’s invariant rule comes from: once a field number has been assigned, it is never given to another field, and a retired number is never reused.

What gRPC Brings

The schema, validator, and framing in this lesson are a small example of this design. gRPC offers the same structure as a ready-made whole: messages are defined with a Protocol Buffers schema, both the client and server sides are generated from that schema, and HTTP/2 is used as the transport layer.

Choosing HTTP/2 is not a detail; it is what makes four call types possible. The stream marker in the schema file shows this: a call can be single-request–single-response, but it can also be server-side streaming (one request, many responses), client-side streaming (many requests, one response), or bidirectional streaming. In the library service, jobs like “send the shelf count as a stream” or “give catalog matches as they come in” fit these types.

Its cost is just as clear. The content on the wire cannot be read by eye; inspecting a request requires a tool that knows the schema. A browser cannot speak this protocol directly; a translation layer has to sit in between. And using the interface requires access to the other side’s schema — where REST is content with sending a request to an address and reading the response, here not a single byte can be understood without the contract.

These costs also explain where gRPC is used the most: among a system’s own components, in places where both sides are managed by the same team and call volume is high.

Summary

• Remote procedure call presents a network operation as though it were a local call; the analogy breaks down at latency, partial failure, and byte conversion, and these points shape the design.
• In schema-first design, the contract is a machine-readable file; the validator and encoder are generated from it, so the two sides’ rules cannot drift apart.
• In the measurement, binary framing reduced the same records to roughly half the bytes of the text format; the gain comes from field names and delimiters never reaching the wire at all.
• The identity on the wire is the field number: when the field name changes, the same bytes decode without trouble, but when the number changes, the error shows up silently as a wrong value.
• gRPC combines this design with a Protocol Buffers schema and HTTP/2 transport; it makes four call types possible, and in return readability and direct browser access are lost.

Next Step

Schema-first design fixes the response shape in the contract, and this returns to the problem measured in the second lesson: when the client’s requirement changes, the server has to change. There is also an approach that does the exact opposite — one where the client determines the shape, but does so not with a loose selection text but inside a type system the server publishes. The next lesson builds this approach: a single endpoint, a client-driven query bounded by a type system, per-field resolution, and the notion of a partial success response.