Yandex Open-Sources Of FF: Zero-Copy Phone Format For Protobuf With Near Read Speed

TLDR

• YaFF is the Yandex open-source zero-copy wire format for Protobuf — Apache 2.0, currently C++, v0.1.0.
• The .proto file is always the source of truth; only the physical memory structure changes.
• In Yandex benchmarks, Flat Layout reads hot data ~3.8× faster than FlatBuffers, within 1.2× of raw C++ layout.
• Four architectures – Fixed, Flat, Minimal, Dynamic – trade learning speed for schema flexibility; Dynamic is the default.
• YaFF uses its own advertising recommendation system, where it reports 10–20% CPU savings at production scale.
• The reception goes up: put it in one hot way, with two-way Protobuf conversion on the edge.

Yandex has an open source YaFF (Alternative Flat Format) under Apache 2.0. It is a very efficient C++ library. YaFF provides a zero-copy file format for the Protobuf ecosystem. Your .proto file is always the only source of truth. The format only changes how the data resides in memory. It focuses on server-side runtime.

What is YaFF

YaFF is not a substitute for Protobuf. It is an alternative phone format for Protobuf messages. The same .proto schema generates a proto-like C++ API. Reading requires no parsing step, so the fields come directly from the buffer. A little performance-sensitive code can still parse the call format back into Protobuf messages. That two-way conversion is what makes module-by-module adoption a reality. You launch YaFF in one hot way and leave the rest to Protobuf.

The problem addressed

Protobuf analysis can consume double-digit percentages of CPU in high-load background environments. At scale, that maps to thousands of physical cores. A common zero copy option is FlatBuffers, also from Google. But FlatBuffers is not a Protobuf drop-in and requires a separate schema maintenance and transformation layer. not statistically compatible with Protobuf. Migration means duplicate schemas, different schema-evolution rules, and handwritten field modifiers. Many groups conclude that the cost is not worth it. YaFF aims at that gap: the zero copy is read with the stored Protobuf semantics.

How Buildings Work

The structure determines how the message is stored in the buffer. It only changes the visual representation, leaving the schema and interfaces unchanged. YaFF submits four properties. Fixed it is an empty full structure with no header and frozen schema. Flat adds a two-byte header and supports schema evolution. Break up talks to fields through a meta table, including sub-schemas. Power is the default and selects Flat or Sparse at runtime. It uses Flat while the schema allows, then switches to Sparse when the evolution violates the low consistency.

Benchmark

Yandex ships a reproducible benchmark suite, built with google/benchmark in a released build. The numbers below are the average nanoseconds per read on an AMD EPYC 7713 with Clang 20.1.8. The lower is faster. In the hot hierarchical case, Flat Layout reads 9.79 ns. FlatBuffers require 37.30 ns, and Protobuf requires 219.35 ns. The base C++ structure is 8.14 ns. So Flat Layout reads about 3.8× faster than FlatBuffers here, and about 22× faster than Protobuf. It stays within 1.2 × of the green struct.

Note: The exact numbers depend on the host CPU and memory. Comparisons between formats are expected to hold across hardware.

Compiler Aliasing Details

FlatBuffers and YaFF both read fields by redefining raw memory as the target type. That kind of punning leaves TBAA without enough hard facts. So LLVM’s alias analysis falls back on the conservative decision of MayAlias. Then the compiler cannot prove that repeated access is safe for reuse. Writing root.intermediate().leaf().a() iterates through the tree twice each time. YaFF adds annotations to its generated code that tell the compiler when reuse is safe. Annotations for YaFF generated code can often help the compiler to reuse chain accesses, as long as the relevant memory can’t be changed between reads. As long as there are no memorizations between reads, YaFF maintains the access chain itself.

When It Fits: Use Cases

YaFF targets systems where you control both the producer and the consumer. Recommendations and backgrounds that serve ads match very clearly. According to Yandex, YaFF uses its own advertising recommendation system, where it reports 10–20% CPU savings at production scale. The memory map pointers are equal to the second. A host can hold tens of gigabytes of local data. Those mmap-capable pointers survive when the service is restarted without reparsing. Search directories, feature stores, and feed services share that heavy profile. A structured Columnar structure guides analysis and ML pipelines with large repeated fields. YaFF can also be denser than FlatBuffers, which helps cache behavior.

Checking the Code

The method read shows Protobuf, remove the analysis step.

#include "feed.pb.h"     // generated by protoc
#include "feed.yaff.h"   // generated by yaff_generate()

// 1. Serialize an existing Protobuf message into a YaFF buffer.
feed::FeedResponse proto = LoadFeedResponse();
const auto buffer = yaff::Serialize<:feed::feedresponse>(proto);

// 2. Read fields directly from the buffer. There is no parsing step.
const auto& response = yaff::ReadMessage<:feed::feedresponse>(buffer.Data());
for (const auto& item : response.items()) {
    std::string_view title  = item.title();
    std::string_view author = item.author().name();  // empty if author is unset
}

// 3. Convert back to Protobuf when a consumer needs the parsed message.
feed::FeedResponse restored;
response.ParseTo(restored);

You can add YaFF with CMake (find_package) or with Conan. The code generation uses protobuf_generate() and yaff_generate(). Generated YaFF types reside in protoyaff:: namespace. Most projects only link yaff::core and yaff::proto.

Resources:

Check it out GitHub Repository and Documentation.