Custom protocol reverse engineering

When it applies

The target speaks something that isn't plain HTTP/JSON: a custom binary TCP/UDP protocol, Protobuf/gRPC with no reflection, FlatBuffers/MessagePack, WebSocket or MQTT frames, or a private RPC format — and you have a PCAP, a proxy export, client logs, or the client binary. (For pure HTTP parameter signing in JS, use reverse-eng-js; for gRPC's HTTP/2 attack surface itself, api-grpc.) Target and any replay testing must be in scope.txt (tradecraft-scope-roe) — replay only against systems you're authorized to touch.

Why it works

Protocols are machines, and machines are regular: fixed headers, magic bytes, length fields, monotonic sequence numbers, and repeated type-length-value structures. Align many samples of the same message type and the invariant bytes identify themselves; the variable ones are your fields. Serialization formats like Protobuf are self-describing enough to decode blind — field numbers and wire types survive even without the .proto.

Method

1. Capture and triage. Pull raw payloads and label each sample by direction (C→S / S→C); note handshake, heartbeat, and reconnect patterns:

   tshark -r cap.pcap -Y "tcp.port==4433" -T fields -e frame.number -e ip.src -e tcp.payload | head
   

   First questions: fixed header? magic bytes? a length field? TLV or fixed-size records? Any compression (zlib/gzip/lz4) or per-frame encryption (AES/ChaCha)?
2. Recover the frame layout. Align several messages of the same kind and diff the bytes: invariant regions = header/constants; counters = sequence numbers; a field whose value matches the remaining byte count = length (check endianness, and whether it includes the header). Locate integrity fields last — CRC16/32, checksums, HMAC slots. Sketch the state machine (Connect → Auth → Ready → Request/Response → Close). Encode the layout as a Wireshark Lua dissector, an ImHex/010 Editor template, or a Kaitai Struct .ksy.
3. Tackle serialization and crypto. Protobuf: protoc --decode_raw < msg.bin for a quick read, blackboxprotobuf or pbtk to rebuild a usable .proto. gRPC is HTTP/2 headers plus a protobuf body. Encrypted frames: the key derivation lives in the client — pull it with reverse-eng-binary-triage (native), reverse-eng-js (web), or mobile tooling; look for the nonce/IV adjacent to the ciphertext.
4. Produce the artifacts. A message-type table (name / opcode / fields), at least one reproducible decode command or script, and evidence excerpts (raw hex + decoded result, redacted of secrets and third-party data).

See cheatsheet.md for the layout-pattern table, parser skeleton, and tshark one-liners.

Gotchas

• Length-field ambiguity — big- vs little-endian, and header-inclusive vs body-only, are the two classic off-by-entire-parse errors; validate against a known sample's byte count.
• A crash on replay is not proof of a bug — it may be a checksum/sequence check rejecting you; fix your framing first, then judge the response.
• Compressed-before-encrypted vs encrypted-before-compressed — high entropy over the whole body means encryption; compression headers (\x78\x9c, \x1f\x8b) mean you can decompress directly.
• Don't fuzz blind against the live service — replay a captured, harmless message first; mutate one field at a time, and only within scope.

Verify success

Your dissector or script decodes a fresh capture into the correct fields with no manual fix-ups, and you can state the message-type dictionary and state machine. Bonus proof: a replayed (in-scope) message with one mutated field is accepted by the server — the layout is right, and the protocol is now fuzzable.

References

Wireshark/tshark docs; Kaitai Struct gallery; blackboxprotobuf; api-grpc, web-websocket, reverse-eng-binary-triage in this library.

---

Portions adapted from reverse-skill by zhaoxuya520, MIT License.