A Buffer is a fixed-length sequence of bytes. A JavaScript string is a sequence of UTF-16 code units. Converting between them always involves an encoding, and when you don’t pick one, Node picks UTF-8. Most of the time that’s what you want, and it hides the fact that a conversion is happening at all. The bugs show up at the edges: when bytes arrive in pieces, when the data isn’t text, or when someone measures a string and assumes the number is a byte count.
This topic comes up in Node interviews because it’s where people who’ve only used JSON.parse(body) meet the layer underneath. The classic prompt is “Why does a user’s name sometimes come out with question-mark diamonds in it?”
Characters, code units and bytes
Start with one string and look at it three ways:
const text = 'Café €5 👍';
const bytes = Buffer.from(text, 'utf8');
console.log('string length:', text.length, '| byte length:', bytes.length, Buffer.byteLength(text));
console.log(bytes);string length: 10 | byte length: 15 15
<Buffer 43 61 66 c3 a9 20 e2 82 ac 35 20 f0 9f 91 8d>There are nine visible characters. text.length says 10 because the thumbs-up emoji is outside the Basic Multilingual Plane and takes two UTF-16 code units (a surrogate pair). The UTF-8 encoding takes 15 bytes: one each for the ASCII letters, digits and spaces, two for é (c3 a9), three for € (e2 82 ac) and four for the emoji (f0 9f 91 8d).
That gap matters whenever a number crosses a boundary. A Content-Length header must be in bytes, so res.setHeader('Content-Length', body.length) is wrong for any body with non-ASCII text and makes clients cut off the end of the response. A database column limited to 255 bytes rejects a 200-character string of Chinese text. A “max 100 characters” validation using .length counts emoji twice. Use Buffer.byteLength(str) when you mean bytes, and [...str].length (or Intl.Segmenter for user-perceived characters) when you mean characters.
The chunk boundary bug
Sockets, files and pipes don’t deliver data in units that respect characters. A TCP read gives you whatever bytes arrived, and a multi-byte character can be split between two reads. Here the same 15 bytes arrive in 4-byte chunks, the way a slow connection might deliver them:
import { StringDecoder } from 'node:string_decoder';
const chunks = [];
for (let i = 0; i < bytes.length; i += 4) chunks.push(bytes.subarray(i, i + 4));
let naive = '';
for (const c of chunks) naive += c.toString('utf8');
console.log('naive: ', JSON.stringify(naive));
const decoder = new StringDecoder('utf8');
let safe = '';
for (const c of chunks) safe += decoder.write(c);
safe += decoder.end();
console.log('decoder:', JSON.stringify(safe));
const td = new TextDecoder('utf-8');
let viaTd = '';
for (const c of chunks) viaTd += td.decode(c, { stream: true });
viaTd += td.decode();
console.log('TextDecoder stream:', JSON.stringify(viaTd));naive: "Caf�� ��5 ����"
decoder: "Café €5 👍"
TextDecoder stream: "Café €5 👍"The first chunk was 43 61 66 c3: “Caf” plus the first byte of é. Calling toString on that chunk alone can’t produce é, so the decoder emits U+FFFD, the replacement character, for the incomplete sequence. The next chunk starts with a9, which is a continuation byte with nothing to continue, and becomes another �. The data is now corrupted, and no later step can recover it.
StringDecoder fixes this by holding incomplete trailing bytes and prepending them to the next write. decoder.end() flushes anything left over, which matters if the input is truncated. TextDecoder with { stream: true } does the same thing and also works in browsers. The simplest fix for a stream is stream.setEncoding('utf8'), which installs a StringDecoder internally so your data events receive whole characters.
The bug is intermittent, which is why it’s hard to catch. With ASCII-only test data it never happens. With real names, it happens only when a chunk boundary lands inside a character, which depends on network timing. The safer default is to collect Buffer chunks and decode once with Buffer.concat(chunks).toString('utf8'), or decode with a streaming decoder if you need text incrementally.
Binary data is not text
The opposite mistake is treating bytes that were never text as if they were. Here are the first four bytes of a JPEG pushed through a UTF-8 string and back:
const jpegHeader = Buffer.from([0xff, 0xd8, 0xff, 0xe0]);
const roundTrip = Buffer.from(jpegHeader.toString('utf8'), 'utf8');
console.log('binary through utf8:', jpegHeader, '->', roundTrip);binary through utf8: <Buffer ff d8 ff e0> -> <Buffer ef bf bd ef bf bd ef bf bd ef bf bd>Every byte was invalid UTF-8, so each became U+FFFD, which re-encodes as ef bf bd. Four bytes became twelve and the image is gone. You see this when code does let body = ''; req.on('data', c => body += c) on a file upload, or when a binary HTTP response is read with res.text() instead of res.arrayBuffer().
When binary data has to travel through a text channel (JSON, an environment variable, a URL), encode it explicitly:
const token = Buffer.from([0xfb, 0xff, 0xfe, 0x01]);
console.log('base64: ', token.toString('base64'));
console.log('base64url:', token.toString('base64url'));
console.log('hex: ', token.toString('hex'));base64: +//+AQ==
base64url: -__-AQ
hex: fbfffe01Standard base64 uses + and / and pads with =, all of which have meaning in URLs. base64url swaps in - and _ and drops the padding, which is what JWTs and URL tokens use. Hex is twice the size of the input but trivially readable in logs.
The latin1 encoding (also called binary in Node) is a trap in the other direction. It maps each byte to one character in the range U+0000 to U+00FF, so characters outside that range are truncated: Buffer.from('€', 'latin1') gives <Buffer ac>, one byte that means something else entirely.
Invalid input and strict decoding
toString('utf8') never throws. Invalid sequences become � silently:
console.log('invalid utf8 decode:', JSON.stringify(Buffer.from([0x68, 0x69, 0xc3]).toString('utf8')));
try { new TextDecoder('utf-8', { fatal: true }).decode(Buffer.from([0x68, 0x69, 0xc3])); }
catch (e) { console.log('fatal TextDecoder:', e.name, e.code); }invalid utf8 decode: "hi�"
fatal TextDecoder: TypeError ERR_ENCODING_INVALID_ENCODED_DATAThat’s a fine default for logs and display. For input you’re going to store, such as an uploaded CSV, silently replacing bytes means saving corrupted data and never knowing. new TextDecoder('utf-8', { fatal: true }) throws instead, so you can reject the file with a clear error, perhaps telling the user it looks like Windows-1252 rather than UTF-8.
Buffers share memory
One more behaviour catches people. buf.subarray(start, end) (and the older buf.slice) returns a view onto the same memory, not a copy:
const original = Buffer.from('hello');
const view = original.subarray(0, 2);
view[0] = 0x4a; // 'J'
console.log('subarray shares memory:', original.toString());
const copy = Buffer.from(original);
copy[0] = 0x59;
console.log('Buffer.from copies: ', original.toString(), copy.toString());subarray shares memory: Jello
Buffer.from copies: Jello YelloThat’s efficient for parsing, but it has a memory consequence: keeping a 10-byte subarray of a 10 MB buffer alive keeps all 10 MB alive. If you cache a small piece of a large read, copy it. Similarly, Buffer.allocUnsafe(n) returns memory that may still contain old data from the pool, which is fast but must be completely overwritten before it’s sent anywhere. Use Buffer.alloc(n) unless you’re sure.
Interview practice
A service reads newline-delimited JSON from a TCP socket. The handler does buffer += chunk.toString(), then splits buffer on \n and parses each complete line. It works in tests. In production, about one record in several thousand is saved with � characters in the customer’s name, always a name with accented or non-Latin letters. Nothing throws. Explain the cause and fix it.
Answer
The chunk.toString() call decodes each chunk independently. When a TCP chunk boundary falls in the middle of a multi-byte UTF-8 character, the trailing bytes of one chunk and the leading bytes of the next are each decoded as invalid sequences and replaced with �. The replacement character is perfectly valid inside a JSON string, so JSON.parse succeeds and the corrupted name is saved without any error. It only affects non-ASCII records, and only when the boundary happens to land inside a character, which is why it’s rare and absent from ASCII test data.
The fix is to decode with state: call socket.setEncoding('utf8') before reading, or keep a StringDecoder and do buffer += decoder.write(chunk). Splitting on \n is then safe, because 0x0a never appears inside a multi-byte UTF-8 sequence. Alternatively, keep the buffer as bytes, find 0x0a with buf.indexOf(0x0a), and decode only complete lines. To prove the fix, write a test that feeds a known message one byte at a time; it fails with the old code on every non-ASCII character.