grpc-protobuf
Design, implement, and maintain high-performance gRPC microservices and Protocol Buffers v3 schemas. Use when authoring proto3 definitions, applying backward and forward compatibility rules, implement
SKILL.mdskills/grpc-protobuf/SKILL.md
GitHubgRPC & Protocol Buffers v3 Engineering
Protocol Buffers (protobuf) is Google's language-neutral, platform-neutral binary serialization mechanism, paired with gRPC for high-throughput, low-latency remote procedure calls (RPCs) over HTTP/2. In distributed microservice architectures, gRPC provides strongly typed service contracts, efficient multiplexing, bidirectional streaming, and robust cancellation propagation.
Purpose & Overview
This skill guides the design, contract evolution, implementation, and verification of production gRPC services and Protocol Buffers v3 schemas. It details proto3 syntax standards, field numbering and reserved tags for backwards compatibility, the four RPC communication patterns (unary, client-streaming, server-streaming, bidirectional), rich error propagation (google.rpc.Status), deadline enforcement, interceptors for cross-cutting concerns, and automated schema linting with Buf.
Core Principles & Rules
- Field tags are permanent identifiers: Never change, renumber, or reuse a field number once deployed. Tags 1–15 take 1 byte in the wire encoding (reserve them for high-frequency fields); tags 16–2047 take 2 bytes.
- Never delete a field without
reserved: When deprecating fields or enum values, mark both the tag number and the field name asreservedto prevent accidental future collisions. - Always use explicit request/response wrappers: Every RPC method must accept a dedicated request message (e.g.
CreateOrderRequest) and return a dedicated response message (e.g.CreateOrderResponse). Never use generic wrappers or reuse messages across distinct RPCs. - Propagate context cancellation and deadlines: Every client call must specify a deadline. Every server handler must monitor context cancellation (
ctx.Done()in Go,context.is_active()in Python) and terminate long-running work immediately when cancelled. - Use standard gRPC status codes and rich error details: Do not return HTTP 200 with error bodies or ad-hoc strings. Use canonical codes (
NOT_FOUND,INVALID_ARGUMENT,UNAUTHENTICATED) augmented withgoogle.rpc.ErrorInfoorgoogle.rpc.BadRequestdetails. - Apply Buf for linting and breaking-change detection: Treat protobuf repositories as contracts. Run
buf linton every PR andbuf breaking --againstthe main branch to guarantee forward and backward compatibility. - Protect against unbounded streams: In streaming RPCs, implement flow control, rate limits, and maximum message sizes (
grpc.max_receive_message_length) to prevent out-of-memory crashes.
Standard Workflows & Procedures
Workflow 1: Schema Authoring & Evolution
- Author proto3 files under package directories matching their import path (e.g.,
api/v1/orders.proto). - Define package names with semantic versions (
package commerce.order.v1;). - Define dedicated
RequestandResponsemessages for each RPC. - Reserve deprecated tags and identifiers explicitly before removing fields.
- Validate syntax and style with
buf lint. - Verify backwards compatibility against the upstream baseline using
buf breaking --against '.git#branch=main'.
Workflow 2: Implementing Handlers with Deadlines and Rich Errors
- Extract deadlines from incoming contexts; reject immediately with
DEADLINE_EXCEEDEDif expired. - Validate incoming fields against constraints.
- If validation fails, construct a
google.rpc.BadRequestdetail, attach it to astatus.Statusobject withcodes.InvalidArgument, and return the rich status. - If downstream dependencies are called, forward the remaining context deadline.
Workflow 3: Interceptor Pipeline Setup
- Chain unary and streaming interceptors in order:
- Panic / Exception Recovery (outermost)
- Tracing & Distributed Context Injection (OpenTelemetry)
- Metrics & Access Logging
- Authentication & Token Extraction
- Input Validation (innermost)
- Keep interceptors non-blocking; delegate long operations to async workers or the handler.
Real-World Examples & Patterns
1. Robust proto3 Service Definition
syntax = "proto3";
package commerce.order.v1;
option go_package = "github.com/example/commerce/gen/order/v1;orderv1";
option java_multiple_files = true;
option java_package = "com.commerce.order.v1";
import "google/protobuf/timestamp.proto";
import "google/protobuf/duration.proto";
enum OrderStatus {
ORDER_STATUS_UNSPECIFIED = 0;
ORDER_STATUS_PENDING = 1;
ORDER_STATUS_CONFIRMED = 2;
ORDER_STATUS_SHIPPED = 3;
ORDER_STATUS_CANCELLED = 4;
}
message OrderItem {
string sku = 1;
int32 quantity = 2;
int64 price_cents = 3;
}
message Order {
// Frequently accessed fields: tags 1-15
string id = 1;
string customer_id = 2;
OrderStatus status = 3;
int64 total_cents = 4;
repeated OrderItem items = 5;
google.protobuf.Timestamp created_at = 6;
google.protobuf.Timestamp updated_at = 7;
// Deprecated fields: explicitly reserved
reserved 8, 9;
reserved "tax_amount", "discount_code";
}
message CreateOrderRequest {
string customer_id = 1;
repeated OrderItem items = 2;
string idempotency_key = 3;
}
message CreateOrderResponse {
Order order = 1;
}
message StreamOrderUpdatesRequest {
string customer_id = 1;
}
message OrderUpdateEvent {
string order_id = 1;
OrderStatus status = 2;
google.protobuf.Timestamp timestamp = 3;
string message = 4;
}
service OrderService {
// Unary RPC
rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);
// Server-streaming RPC for live status updates
rpc StreamOrderUpdates(StreamOrderUpdatesRequest) returns (stream OrderUpdateEvent);
}
2. Go Server Handler with Deadlines, Context, & Rich Errors
package main
import (
"context"
"errors"
"time"
"google.golang.org/genproto/googleapis/rpc/errdetails"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
orderv1 "github.com/example/commerce/gen/order/v1"
)
type OrderServer struct {
orderv1.UnimplementedOrderServiceServer
repo OrderRepository
}
func (s *OrderServer) CreateOrder(
ctx context.Context,
req *orderv1.CreateOrderRequest,
) (*orderv1.CreateOrderResponse, error) {
// 1. Check for incoming deadline or cancellation
if err := ctx.Err(); err != nil {
if errors.Is(err, context.Canceled) {
return nil, status.Error(codes.Canceled, "client cancelled the request")
}
return nil, status.Error(codes.DeadlineExceeded, "deadline exceeded before processing")
}
// 2. Validate input and construct rich error details if invalid
if req.GetCustomerId() == "" || len(req.GetItems()) == 0 {
st := status.New(codes.InvalidArgument, "invalid order request")
br := &errdetails.BadRequest{}
if req.GetCustomerId() == "" {
br.FieldViolations = append(br.FieldViolations, &errdetails.BadRequest_FieldViolation{
Field: "customer_id",
Description: "customer_id is required",
})
}
if len(req.GetItems()) == 0 {
br.FieldViolations = append(br.FieldViolations, &errdetails.BadRequest_FieldViolation{
Field: "items",
Description: "items array cannot be empty",
})
}
stWithDetails, err := st.WithDetails(br)
if err == nil {
return nil, stWithDetails.Err()
}
return nil, st.Err()
}
// 3. Execute business logic respecting context deadline
order, err := s.repo.CreateOrder(ctx, req)
if err != nil {
if errors.Is(err, ErrCustomerNotFound) {
return nil, status.Errorf(codes.NotFound, "customer %q does not exist", req.GetCustomerId())
}
return nil, status.Errorf(codes.Internal, "failed to persist order: %v", err)
}
return &orderv1.CreateOrderResponse{Order: order}, nil
}
// 4. Server-Streaming RPC handler
func (s *OrderServer) StreamOrderUpdates(
req *orderv1.StreamOrderUpdatesRequest,
stream orderv1.OrderService_StreamOrderUpdatesServer,
) error {
ctx := stream.Context()
sub := s.repo.SubscribeOrderUpdates(req.GetCustomerId())
defer sub.Close()
for {
select {
case <-ctx.Done():
return status.FromContextError(ctx.Err()).Err()
case event, ok := <-sub.Events():
if !ok {
return nil // stream finished normally
}
if err := stream.Send(event); err != nil {
return status.Errorf(codes.Unavailable, "failed to send event: %v", err)
}
}
}
}
3. Client Call with Timeout Deadlines (Go)
func CallCreateOrder(client orderv1.OrderServiceClient, customerID string) (*orderv1.Order, error) {
// Set a non-negotiable 3-second deadline for the network roundtrip
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
resp, err := client.CreateOrder(ctx, &orderv1.CreateOrderRequest{
CustomerId: customerID,
Items: []*orderv1.OrderItem{
{Sku: "WIDGET-01", Quantity: 2, PriceCents: 1500},
},
})
if err != nil {
st := status.Convert(err)
switch st.Code() {
case codes.InvalidArgument:
// Inspect rich error details
for _, detail := range st.Details() {
if br, ok := detail.(*errdetails.BadRequest); ok {
log.Printf("Validation violations: %v", br.GetFieldViolations())
}
}
case codes.DeadlineExceeded:
log.Println("Call timed out on server")
default:
log.Printf("RPC error: %s (%s)", st.Message(), st.Code())
}
return nil, err
}
return resp.GetOrder(), nil
}
4. Buf Configuration (buf.yaml)
version: v2
modules:
- path: proto
lint:
use:
- DEFAULT
except:
- PACKAGE_VERSION_SUFFIX
breaking:
use:
- FILE
Run validation in CI:
# Verify lint rules and conventions
buf lint
# Ensure backward compatibility against main branch
buf breaking --against '.git#branch=main'
Common Anti-Patterns & Pitfalls
| Anti-Pattern | Why It Fails | Remediated Practice |
|---|---|---|
| Changing or renumbering a field tag | Existing binary decoders map the old tag into the wrong field, silently corrupting data in flight. | Keep tag numbers permanent. Add new fields with incremented numbers; deprecate old tags with reserved. |
| Omitting RPC deadlines | Default client calls have infinite deadlines. If a server hangs or network partitions, client connection pools exhaust. | Always specify context.WithTimeout or client interceptor deadline defaults on all outbound calls. |
| Ignoring server context cancellation | The server keeps computing or querying databases long after the calling client timed out or disconnected. | Check ctx.Done() or pass ctx through all database and downstream network calls. |
| Reusing Request/Response messages across RPCs | Changing the signature for one RPC inadvertently modifies contracts for completely unrelated RPCs. | Always create dedicated CreateOrderRequest / CreateOrderResponse for each RPC method. |
Returning generic UNKNOWN or INTERNAL errors |
Clients cannot differentiate retryable network failures from bad arguments or unauthorized requests. | Use specific canonical gRPC status codes (UNAVAILABLE, INVALID_ARGUMENT, UNAUTHENTICATED). |
| Unbounded streaming RPCs without flow control | Rapid publishers swamp slow subscribers with memory buffering, leading to OOM crashes. | Honor flow control, apply backpressure, and verify subscriber throughput. |
Verification Checklist
- All proto files begin with
syntax = "proto3";. - Every RPC accepts a distinct
*Requestand returns a distinct*Responsemessage. - Deprecated field numbers and field names are marked with
reserved. - High-frequency fields use tags 1 through 15 for 1-byte wire encoding.
- Client calls enforce an explicit deadline via context or channel options.
- Server handlers monitor context cancellation and terminate work early when cancelled.
- Rich error structures (
google.rpc.BadRequest,ErrorInfo) are returned for validation and domain errors. -
buf lintpasses cleanly with zero violations. -
buf breakingconfirms no wire or source-breaking schema changes against main branch.
Tooling & Reference Guides
- Linters & Toolchains:
buf(Buf CLI for linting, breaking change detection, and code generation). - Protobuf Compiler:
protocwith pluginsprotoc-gen-go,protoc-gen-go-grpc,protoc-gen-grpc-web. - Debugging & Inspection:
grpcurlfor CLI gRPC calls,grpcuifor interactive browser-based testing. - Specifications: Google gRPC Core Concepts, Protocol Buffers Language Guide (proto3), Google APIs Error Model.