V-Modal AI Blog: Search, MultiModality, Physical AI
← Back to articles

Content Defined Chunking Explained for Media Pipelines

You're uploading a long video from a phone when the signal disappears. The capture continues, but the upload fails halfway through, leaving the app to decide whether it can resume, restart, or discard work that already consumed battery and storage. The same problem appears when an editor trims a clip, a robot records an operating environment, or a smart-glasses app needs to recall one brief moment from a continuous stream.

Content-defined chunking, often shortened to CDC, is a way to split data according to patterns in the data itself instead of cutting every file at rigid byte offsets. That distinction matters far beyond backup systems. For mobile media pipelines and Physical AI memory, the valuable property isn't only deduplication. It's stable chunk identity when surrounding content changes.

CDC has been developed and refined for roughly 20 years, and a 2024 comparative study of leading CDC algorithms evaluated the field across four realistic datasets and four metrics, including throughput, deduplication ratio, average chunk size, and chunk-size variance. The approach is now mature infrastructure for backup, synchronization, and cloud storage, but its next challenge is more demanding: keeping video, audio, actions, and sensor context searchable and reusable on mobile and edge hardware.

Table of Contents

The Mobile Upload Problem Content Defined Chunking Solves

A videographer starts a 4K rooftop clip in Barcelona through a mobile SDK. Thirty seconds into the upload, she walks into a stairwell and loses LTE coverage. The recording is already large, the retry budget is nearly exhausted, and the client needs to preserve as much completed work as possible without keeping the entire asset in memory.

A fixed-size uploader might divide the file into parts at predetermined offsets. That works while the original byte layout remains unchanged, but a trim or insertion can shift every later part. The client may then treat unchanged media as new data, retry more than necessary, or rebuild the multipart session from the beginning.

CDC places a content-sensitive boundary layer between capture and durable storage. A rolling boundary detector scans the stream and emits variable-size chunks when the bytes meet a repeatable condition. The client can upload each completed chunk, record its identity, and resume from confirmed work rather than treating the video as one indivisible object.

Why fixed offsets become fragile

Suppose an editor removes a short section near the start of a recording. With fixed splitting, every later byte moves into a different offset range. The content after the edit may be identical, but its part number and byte position no longer line up with the prior version.

CDC behaves differently. The edit disturbs boundaries near the changed region, but unchanged content farther away can produce the same boundary conditions again. That locality is the reason CDC can support incremental updates and repeated uploads more effectively than a naive fixed split. Restic's explanation of CDC describes the same principle for files whose surrounding layout changes while unchanged regions remain recognizable.

The client can also process chunks progressively. It doesn't need to load the full video into RAM before sending anything, which is important for phones capturing long media, unstable connections, and background execution limits.

Practical rule: Treat the chunk as the unit of retry, verification, storage, and later reuse. The file is the presentation layer. The chunk manifest is the operational layer.

For advertisement video edition, this means an editor can preserve reusable material across versions. For robotics, it means a device can retain stable handles for audio and video segments while the surrounding stream continues to grow.

How Content Defined Chunking Actually Works

CDC sounds complicated until you separate boundary detection from content verification. The first process decides where a chunk ends. The second calculates a strong fingerprint for the completed chunk.

A rolling boundary detector

A CDC implementation scans a sliding window over the byte stream. At each position, it updates a rolling hash, such as a Rabin fingerprint, Gear hash, or Buzhash-style function. When the hash falls inside a chosen target pattern, the algorithm marks a boundary and starts the next chunk.

The boundary depends on the bytes around that position, not solely on the absolute offset from the beginning of the file. The same byte sequence therefore tends to produce the same boundary when it appears again.

Most production implementations also enforce size limits. A minimum prevents very small chunks from multiplying metadata. A target or average controls the general distribution. A maximum prevents a long region without a matching pattern from creating an impractically large chunk. The exact values should come from workload testing, not from a universal default.

A media team might test a minimum, average, and maximum such as 1 MiB, 4 MiB, and 8 MiB, or a broader configuration such as 2 MiB and 16 MiB where the implementation exposes fewer controls. These are configuration examples, not guarantees. CDC research continues to examine how to provide both locality and predictable size, because low-entropy or code-like data can create large outliers and high variance, as discussed in the USENIX study of chunking tradeoffs.

The pass produces more than pieces

A useful CDC pass generally produces:

  1. Chunk boundaries, represented by lengths or offsets.
  2. A content hash for each chunk, such as BLAKE3 or SHA-256.
  3. An ordered manifest, which lists chunk hashes in stream order.
  4. A reconstruction map, allowing the original byte stream to be assembled from those chunks.

The rolling hash is for finding boundaries. The cryptographic hash is for identifying the resulting content. Keeping those roles separate helps prevent a common design mistake: using a fast boundary hash as if it were a collision-resistant content address.

A six-step infographic explaining how content-defined chunking works to prepare documents for AI and LLM applications.

What this looks like on a phone

A mobile SDK can run the boundary detector on a background thread while capture continues. It reads a bounded buffer, updates the rolling state, flushes a completed chunk, computes its content hash, and hands the chunk to the upload worker.

That design avoids double-buffering the entire video. It also lets the application respond to lifecycle events, such as backgrounding or network changes, at chunk boundaries instead of interrupting an arbitrary byte range.

The important configuration principle is simple: bound chunk size aggressively enough for predictable memory and retry behavior, then measure the resulting variance on real media. Compressed video, raw audio, subtitles, sensor logs, and code-like text won't produce identical distributions.

CDC Compared to Fixed, Semantic, and Recursive Chunking

No chunking strategy wins every media problem. A platform engineer should choose based on edit locality, decoder access, operational cost, and the kind of search the product needs.

Strategy Locality under edits Chunk-size variance Decoder required Mobile cost Best fit
Fixed-size Poor after insertions or deletions Predictable No Low Simple multipart transfer
Content-defined Usually local around edits Variable, bounded by configured limits No Moderate Raw streams, sync, deduplication
Semantic Strong around scenes, speech, or topics Depends on semantic units Usually yes Higher Editorial and retrieval workflows
Recursive Can preserve reuse at multiple granularities Multiple levels to manage No Higher Tree-structured or sub-chunk deduplication

Fixed-size splitting

Fixed splitting is easy to reason about. Part boundaries are known before reading the content, parallel workers can address byte ranges directly, and storage scheduling remains predictable.

Its weakness is offset sensitivity. Insert one byte at the beginning and every later boundary moves. For a simple one-time upload of a stable asset, that may be an acceptable trade. For edited advertising footage or repeated mobile synchronization, it can cause unnecessary transfer and indexing churn.

Semantic and structure-aware chunking

Semantic chunking uses meaning or media structure as the boundary. A video system might split at shots or scenes. An audio system might use silence or speech regions. A retrieval system can compute embeddings for sentence or paragraph units and place boundaries where adjacent-vector similarity drops below a threshold, helping preserve topical coherence, as described in Unstructured's guide to semantic chunking.

This is valuable for search. An advertisement editor wants a complete product shot, voiceover phrase, or scene transition, not an arbitrary byte range. The cost is decoder awareness, more processing, and possible fragility when codecs, timestamps, or editorial exports change.

Recursive chunking

Recursive approaches apply chunking at more than one granularity, creating a hierarchy that can support broad and fine-grained reuse. They suit systems that need a structured index of subcomponents, but the index and update rules are more involved.

Decision rule: Choose CDC for unknown or raw byte streams, semantic chunking for editor-aware search, and recursive chunking when multi-level deduplication matters more than the simplicity of a flat manifest.

Deduplication and Incremental Updates Under Edits

Consider a 200 MB video split into roughly 40 CDC chunks. An editor removes 12 seconds from the middle. The bytes before the cut remain unchanged. Farther after the edit, the rolling detector encounters the same content again and can recover prior boundaries, so much of the original chunk set remains reusable.

The area immediately around the cut is different. One or more chunks may merge, split, or receive new boundaries. That is expected. CDC doesn't promise that every chunk survives every edit. It aims to keep the disturbance local enough that unchanged regions can be recognized again.

With fixed-size splitting, the same edit shifts all later offsets. The system may still detect identical bytes if it performs extra content comparisons, but the basic part map no longer provides stable reuse. CDC makes that reuse a first-class property of the manifest.

The storage model

A deduplicating system typically stores each unique chunk once and keeps a version manifest that references the chunks in order. A Merkle-style index can represent relationships between manifests and chunk fingerprints, while content-addressed storage uses the chunk hash as its identity.

For mobile upload, the practical effect is narrower retry scope. The client can check which chunk fingerprints already exist, send missing chunks, and store a new ordered manifest for the edited asset. The system doesn't need to transmit every unchanged region merely because its position shifted.

The chunk-size tradeoff

Smaller bounds improve locality because an edit is less likely to consume a large unrelated region. They also create more chunks, more hashes, more manifest entries, and more lookup work. Larger bounds simplify metadata and reduce per-chunk operations, but an edit can invalidate a larger area before boundaries realign.

Edit Type Fixed-Size Reuse CDC Reuse
Append at the end Strong Strong
Remove content near the beginning Later offsets shift Unchanged regions can realign
Trim a middle segment Later part numbers change Nearby boundaries change, later regions may persist
Re-encode the entire asset Usually weak Weak if bytes are substantially different
Reorder complete byte-identical regions Offset-based mapping changes Repeated regions can be recognized by content

The final row needs care. CDC recognizes bytes, not meaning. It can reuse identical encoded regions, but it can't infer that two visually identical scenes should match after a codec or encoding process changed the bytes.

Implementing CDC for Chunked Multipart Video and Audio Uploads

A mobile upload pipeline should make the CDC chunk the unit of work from capture through confirmation. Each emitted chunk gets a length, a content hash, an upload state, and a retry policy.

A practical pipeline

  1. Capture into a bounded buffer. Read encoded video, audio samples, or another stream without retaining the complete asset in memory.
  2. Scan for a boundary. Run the rolling hash across the incoming bytes and apply minimum and maximum limits.
  3. Seal the chunk. Once the detector finds a valid cut, stop writing to that chunk and calculate a cryptographic checksum.
  4. Upload independently. Send the chunk through a signed multipart or streaming operation.
  5. Persist confirmation. Store the chunk hash and server acknowledgement locally before moving on.
  6. Reconcile on resume. Reuse confirmed hashes and retry only missing or incomplete chunks.

The rolling hash may use a Rabin fingerprint, Gear hash, or Buzhash variant. The final checksum can use SHA-256 or BLAKE3. The server should validate both the declared length and content identity before adding the chunk to the manifest.

A process flow diagram illustrating the steps for implementing Content Defined Chunking for multipart video and audio uploads.

Mobile lifecycle details

The difficult part isn't finding a rolling hash library. It's handling interruption.

A fixed multipart uploader can still be the better choice when the asset is small and the connection is reliable. CDC earns its complexity when users pause, edit, retry, and re-upload large media.

Implementation warning: Don't confuse resumability with deduplication. A part can resume because its offset is known. CDC adds value when the content changes and the system still needs to recognize unchanged regions.

For audio, the same pattern applies to recordings, machine sounds, and voice commands. The byte-level manifest handles transport. A separate time-aware layer should preserve the relationship between each audio chunk and its position in the recording.

CDC Inside Physical AI Memory and Multimodal Search Layers

A warehouse robot records video, microphone audio, and action history while moving through a work area. Later, an operator asks for the moment when someone said “stop” near a damaged package. A useful memory system must connect the spoken command, the surrounding video, and the robot's state without treating each signal as an unrelated file.

Physical AI stacks are converging on multimodal architectures that process language, image, video, audio, and action together. NVIDIA's Cosmos 3 description presents omnimodal world models that jointly process and generate those five modalities, while a 2026 paper on omnimodal physical intelligence describes a four-stage training strategy that moves from modality-specific experts to joint training and later audio and image generation integration.

CDC provides a durable transport and memory substrate beneath that model layer. Video and audio can have separate chunk streams, each with stable content identities, while timestamps and modality links connect related moments.

Audio needs time alignment

Transcription alone isn't enough for ambient audio. A search system needs to know where a word, alarm, command, or machine sound occurred in the recording.

One multimodal chunking guide describes a practical pattern using voice activity detection to create speech regions of about 30 seconds with roughly 1 second of overlap, transcribe those regions in parallel, and use forced alignment to map words back to exact timestamps. Semantic transcript boundaries can then map back to audio time codes.

That matters for audio for robotics. A robot can associate “stop,” an alarm, or a mechanical sound with a moment rather than an entire recording. The hard problems include timestamp drift, overlapping speech, noisy environments, and reviewing long recordings when only a short interval matters.

A diagram illustrating the layered architecture of physical AI memory and multimodal search using content defined chunking.

Video search needs more than scene boundaries

For vision for robotics, a scene-level segment can be too broad. A query may concern a brief hand movement, a label becoming visible, or a tool entering the camera view. CDC offers stable byte-level handles, while visual and temporal analysis can attach richer meaning to selected ranges.

For advertisement video edition, the same architecture supports a different workflow. Editors can search for a product appearance, a spoken phrase, or a visual setup, then retrieve the exact media interval while preserving reusable underlying content across exports.

A practical memory stack can therefore separate responsibilities:

The central design choice is synchronization. If audio transcription, visual understanding, and action history live in disconnected pipelines, the system may find the right signal but fail to return the right moment.

Tradeoffs, Open Problems, and the Decision Checklist

A mobile upload can finish successfully while the resulting chunks still perform poorly for search or memory retrieval. CDC recognizes byte patterns, not visual meaning. A re-encoded clip may look identical but contain different bytes, and a high deduplication ratio does not show whether a retrieved segment answers the user's question. For mobile-first media ingestion and Physical AI memory, stability under edits deserves the same attention as storage reuse.

Where CDC becomes unpredictable

Low-entropy data can delay boundary detection and produce unusually large chunks. Highly repetitive media or code-like text can also create wide chunk-size variation. Bounded size and locality remain active research problems, rather than settings that can be assumed to work across every input.

Smaller chunks increase metadata. Each one may need a hash, length, manifest reference, storage record, and verification step. Cloud object storage can therefore incur more requests and index-management work even when total byte storage decreases. That tradeoff affects upload latency, retry behavior, and the size of the manifest a mobile client must maintain or retrieve.

The chunker also uses CPU and battery. Boundary scanning and cryptographic hashing process every incoming byte, with different behavior on ARM mobile devices and x86 servers. Encryption and re-encoding create another boundary: without normalization, transformed ciphertext or compressed bytes will not match the original content.

A decision checklist

The 2024 CDC comparison evaluates throughput, deduplication ratio, average chunk size, and chunk-size variance together. That combination shows why production fitness cannot be reduced to one metric.

Retrieval adds a separate uncertainty. A 2026 evaluation study found recursive and structure-aware chunking outperforming fixed-size, semantic, and LLM-based methods for retrieval tasks, with strongest results at intermediate-to-large chunk sizes around 750 to 1,250 characters. The practical lesson is to test whether a simpler structure-aware method preserves context more reliably before adding semantic complexity.

An infographic titled Make Better Decisions detailing tradeoff frameworks, open problem analysis, and a structured decision-making checklist.

Choosing Your Default Chunking Strategy

Start with the smallest system that satisfies the actual failure mode.

For short assets under 50 MB, fixed-size multipart uploads are often the sensible default because simplicity, predictable memory use, and direct retry addressing outweigh fine-grained reuse. For files reaching hundreds of megabytes, repeated editing, or metered bandwidth, CDC becomes more compelling, particularly when downstream indexing benefits from stable chunk IDs.

Use semantic or shot-based chunking when the product must retrieve complete scenes, spoken ideas, or editorial units. Don't substitute it for CDC when the main problem is byte-level reuse under edits.

Asset Type Recommended Strategy Target Chunk Size Notes
Short mobile clips Fixed-size Product-defined Keep upload logic simple
Large camera uploads CDC Bounded media profile Optimize resume and incremental transfer
Edited advertisement footage CDC plus semantic metadata Separate transport and search profiles Preserve reuse while indexing scenes
Robotics video and audio CDC plus time alignment Stream-specific Keep audio, video, and actions joinable
Already-segmented HLS media Existing segments Segment-defined Avoid adding another layer without a clear benefit
Text retrieval corpus Structure-aware or semantic Task-tested Evaluate coherence and recall directly

The right default isn't the most advanced chunker. It's the one whose stability, overhead, and search behavior match your users' edits and devices.

V-Modal AI provides a Visual Memory Layer for Physical AI with multimodal video and audio search, continuous memory, mobile SDKs, and edge-oriented retrieval. If you're building a robotics, smart-glasses, or mobile media pipeline, visit V-Modal AI to evaluate how its search layer and chunked media capabilities can support stable, multimodal recall.