---
title: Using EmbeddingGemma 2 for commercial break detection in TV recordings
description: A multimodal embedding experiment for finding commercial breaks in TV recordings with manual and agentic methods.
lang: en
articleId: embedding-ad-detection
translationKey: embedding-ad-detection
version: 1
permalink: /en/articles/embedding-ad-detection/
revisionSummary: Initial version.
pubDate: 2026-10-17
tags: [embeddings, video, agents]
draft: false
illustrated: true
contents: true
showDescription: false
preserveText: true
---

Embedding models are commonly used in retrieval-augmented generation (RAG) workflows to retrieve information from external sources. EmbeddingGemma 2 is a multimodal embedding model that accepts text, video, audio, and images as input and projects them into a unified embedding space. Multimodal support opens up interesting use cases outside traditional RAG workflows. I tested the model for commercial break detection in TV recordings using both manual and agentic methods.

If you have heard about embedding models, it is most likely in relation to retrieval-augmented generation (RAG). RAG is not their only use case, but modern generative AI and RAG have popularized them.

Embedding models map inputs into a high-dimensional vector space, placing related things close together [@girdhar2023imagebind]. This property is commonly used in information retrieval for RAG workflows [@wolff2023rag]. In a traditional RAG workflow, an embedding model is used to embed document chunks into a vector space. A chunk is part of a full document, such as one subsection. When a user asks a question, the question is also embedded into the same vector space. Because the embedding model places related things close together, the question should be mapped close to the document chunks that hopefully contain answers to the user's question. Cosine similarity [@cosineSimilarity] is usually used to measure the similarity between the embedded question and the embedded chunks. The most similar chunks are retrieved from the database and fed to an AI model as additional context that it can reference when generating an answer to the user's question.

Multimodal embedding models allow other modalities, such as images, video, and audio, to be embedded into the same space as text. EmbeddingGemma 2 is one such model [@dua2026embeddinggemma2]. Multimodality opens up interesting new use cases for embedding models.

When using embedding models with RAG, the system often retrieves only a certain number of the most similar embedded chunks. Cosine similarity returns a scalar value that describes how closely the directions of two vectors align in the high-dimensional embedding space. Embeddings that point in similar directions get high cosine similarity values, while embeddings that point in very different directions get low values. These vectors are usually stored in a vector database, which makes it efficient to find the chunks most similar to the embedded question.

With video, we also want to chunk the original video by dividing it into short segments. We embed those segments into the embedding space and can then find the segments most similar to a given query. We can use the embedded segments in a similar way to document chunk embeddings in a traditional RAG workflow: to find the segments most similar to the user's query. But that is not all. Video is time series data, and the chunks have a playback order. We can therefore plot each segment's similarity to the user's query in a line plot over the whole video. This gives a useful visual signal showing where on the timeline the video content most closely matches the user's query.

The following demo shows this in practice. It uses the short film Big Buck Bunny [@bigBuckBunny], which is licensed under Creative Commons Attribution 3.0 [@bigBuckBunnyLicense]. The film is divided into 5-second segments with a 2.5-second overlap, using a frame rate of 2 fps. These segments are embedded into the embedding space using the EmbeddingGemma 2 model. Cosine similarity is computed between the segments and the following embedded text prompts: "A large white rabbit", "Scrolling end credits", and "A butterfly flying". The demo shows these similarity values in a line graph at the bottom, giving a clear signal of where the concepts appear in the film. The most similar segments are also listed below the graph.

<div data-article-slot="bunny-signals"></div>

As can be seen from the demo, the prompts "A large white rabbit" and "Scrolling end credits" work very well. The "A butterfly flying" concept is also visible in the signal, but the matches between the prompt and the segments are not quite as clear. The most similar segment only shows butterfly wings. In the second most similar segment, the butterfly is just flapping its wings, not actually flying. The third most similar segment shows a flying squirrel, with no butterfly at all. It seems that the act of "flying" makes this segment similar to the prompt.

These failures are interesting and show that the embedding space is very abstract and sometimes does not work as expected. When the concept is clear and distinct, the similarity metric works better.

## Commercial break detection

For some background on why I want a commercial break detector: I go to bed early in the evenings. There are many movies where I've only seen the first hour, since my bedtime usually comes just an hour into the movie. To fix this annoyance, I implemented my own TV recorder software as one of my first projects after agentic coding got good enough to build something useful. I had wanted to build it for years and already had the architecture planned in my head, but I never had the time or energy to code it by hand. The software works great! It is tailored to my specific use case and has exactly the features I want... Or, all but one. Automatic commercial break skipping would be nice.

There is existing commercial break detection software, such as Comskip [@comskip]. It detects commercials using heuristic rules. The approach is simple, but in my experience, heuristics are often fragile. Automatically skipping commercial breaks based on its detections might work 99% of the time, but the remaining 1% would probably annoy me. I would much rather use a workflow that involves real intelligence and works 100% of the time. Maybe modern generative AI and agents can make this possible.

I ran some early tests with agents. In these tests, an agent searched through the whole recording at a low frame rate and flagged potential commercials. It then found the exact start and end positions by sampling frames more frequently around the potential break points. It worked, but it was quite wasteful and used a lot of tokens. I thought that if I could give the model rough estimates of the commercial break positions, it could check only those positions, making the method much more efficient.

I saw the EmbeddingGemma 2 release post and the accompanying demos [@dua2026embeddinggemma2]. The embedding model seemed to work great across modalities. That gave me the idea that it could provide rough estimates of the commercial break positions and make the method more efficient.

I ran a quick test with a recording of Groundhog Day from a commercial TV channel. I used the model to embed the video into the embedding space in 10-second snippets and computed the cosine similarity between each snippet and the embedded text "tv commercial". The result was very promising. Even this simple first test gave clear hints of where the commercials were. With a little help from the GPT-6 Astra model, we came up with the prompt "a short promotional video with fast cuts, product close-ups, animated graphics, slogans and brand logos", which made the signal much clearer. The commercials appear as clear peaks in the signal, as shown in Figure 1.

<figure class="article-figure similarity-figure">
<a class="figure-image" href="/article-assets/embedding-ad-detection/v1/figure-1.svg"><picture><source media="(max-width: 600px)" srcset="/article-assets/embedding-ad-detection/v1/figure-1.mobile.svg" width="320" height="592" /><img src="/article-assets/embedding-ad-detection/v1/figure-1.svg" alt="Advertising-prompt cosine similarity across the Groundhog Day recording. Shaded commercial intervals and numbered markers identify the production logo, opening titles, first scene, three commercial breaks, end credits and final commercials using hand-checked timings." width="820" height="472" loading="lazy" /></picture></a>
<figcaption>Figure 1. Cosine similarity between video segments and the text prompt “a short promotional video with fast cuts, product close-ups, animated graphics, slogans and brand logos”. Shaded bands and numbered markers show commercial breaks and film transitions using manually checked timings.</figcaption>
</figure>

The signal itself is enough to make it easy to manually skip past the commercials. However, it would be cool if I could implement an automatic skipping feature. That would require very accurate detection of where each commercial break starts and ends.

## Agentic workflow

I ran an experiment in a modified harness environment using the OpenAI models GPT-6 Luna, GPT-6.1 Sol, and GPT-6 Astra, as well as a self-hosted Qwen3.8-27B model with 4-bit quantization.

The exact version of the self-hosted Qwen3.8-27B model is Qwen3.8-27B-UD-Q4_K_XL.gguf [@qwen38gguf]. It ran on my gaming PC with an NVIDIA GeForce RTX 4090 and an AMD Ryzen 7 7800X3D. The model was hosted using the llama.cpp inference engine with a context size of 125,184 tokens and a Q8-quantized KV cache. The feed-forward layers of the first 17 attention blocks were offloaded to the CPU to enable this context size (llama.cpp argument: `-ot 'blk\.([0-9]|1[0-6])\.ffn_(down|gate|up)\.weight=CPU'`).

The customized harness environment uses Codex through the Codex App Server [@openaiAppServer] for the OpenAI models and Pi through Pi RPC [@pi2026rpc] for the self-hosted model. The agents received a graph image showing the similarity of each 10-second segment to the embedded prompt that describes commercial visuals, along with tools to interact with the visualization and submit their results. Table 1 lists the tools provided to the agents to accomplish their task.

Table 1.Tools provided to the agents.

| Tool                                      | Description                                                                                                     |
| ----------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
| view_graph(start, end)                    | View or zoom the embedding-signal graph over a time range.                                                      |
| get_frames(times)                         | Fetch actual video frames at requested timestamps, with frame IDs and exact presentation timestamps.            |
| add_signal(text=… or reference_seconds=…) | Add a similarity curve using a new text embedding or an existing cached video-clip embedding.                   |
| set_visible_signals(ids)                  | Choose which curves are visible.                                                                                |
| report_boundary(...)                      | Immediately submit an ad-break start or end, with uncertainty, evidence frames, and tentative/confirmed status. |
| revise_boundary(...)                      | Refine or retract a previously submitted boundary, preserving its history.                                      |
| finish(notes)                             | Declare completion and describe unresolved uncertainty or coverage.                                             |

The harness environment was implemented by GPT-6 Astra, which also wrote the prompt for the agents:

```text
You are analyzing a television recording to locate external advertising breaks. Use the supplied embedding graph to direct visual inspection. Higher cosine similarity means a closer match to the curve's query or reference; it is not an ad label. Zoom the graph, add new text/reference embedding signals, and show/hide curves when useful. Account for visible signal changes, including sustained regions and changes that contradict an earlier conclusion. There are no automatic candidates.

Find all external non-program breaks accurately while minimizing inspected video frames. Report each likely entry or exit through report_boundary as soon as you have supporting evidence; do not wait until the entire recording is analyzed. A break can have an entry reported before its exit is known. Give it a stable break_id. Submit approximate boundaries first when appropriate, then refine or retract through revise_boundary. Accuracy takes priority over saving a few frames.

Commercials, external film/program trailers, sponsor bumpers and channel idents are non-program. Feature production logos and story-internal advertisements are program. Cinematic-looking material can be an unrelated trailer: verify the identity and continuity of the target program at apparent returns. A temporary dip on the graph does not settle the rest of a high-similarity region.

Boundary time is the first source frame of the new state: start inclusive, end exclusive. Use the returned frame IDs and PTS. Claim exact frame precision only with inspected adjacent before/after frames. Otherwise retain an uncertainty range. Preserve an incoming feature fade; uncertain black frames need a stated convention. Initial/terminal fragments can touch the recording edges.

Use only the provided experiment tools. There is no shell, filesystem inspection, web search, delegation or access to prior results. Do not try to infer hidden reference labels. Do not perform an exhaustive uniform frame sweep. Each returned source frame counts, including repeats; graph images are counted separately. Call finish when done, describing unresolved regions and incomplete coverage.
```

The harness recorded the tool calls, and the following animation shows how the models performed.

<div data-article-slot="agent-replay"></div>

The GPT-6.1 Sol and GPT-6 Astra models detected the commercial break boundaries within one second of my manual checks. GPT-6 Luna and Qwen messed up the task at the default thinking level, but both improved a lot with xhigh thinking mode. Still, even with xhigh thinking, Qwen overshoots the third commercial break by a few seconds. Otherwise, the results from both models are perfectly usable with xhigh thinking.

Interestingly, only two runs used the add_signal tool. This tool lets the agent add a similarity signal for a new text prompt to the visualization, which can help if the default signal is too weak. Adding a new signal is quite cheap because the video segments are already embedded. Only the new text prompt needs to be embedded and compared with the existing segment embeddings. For this recording, the default signal seems to have been strong enough for most of the models, so they did not bother adding another signal.

## Using the method in practice

The EmbeddingGemma 2 model is quite efficient. It has 740 million parameters, far fewer than modern LLMs. Compare that to the 27 billion parameters of the Qwen3.8-27B model that I'm running on my gaming PC. This "small" size makes it possible to run inference on a CPU at an okayish speed. Still, embedding a full three-hour recording is not fast on a CPU. In my tests, it would have taken about 10 hours on my server's Intel Xeon E3-1275 v6, with the process limited to two physical cores. I'm running other stuff on the server, so I didn't want to dedicate all the cores to the job. Compare that to my RTX 4090 GPU, which embedded the same segments in 7 minutes.

Still, 10 hours is acceptable for a use case like this. I usually record something in the evening and would likely watch it the next day at the earliest. The server would have plenty of time to process the recording in between. Also, that estimate is for embedding all the segments of a full three-hour recording. Embedding simple text prompts is much faster: it takes about 100 milliseconds to embed a short sentence.

The ability to quickly embed new prompts into the embedding space and compare them with existing segment embeddings enables some cool features for the user. Take the Groundhog Day recording as an example. The movie is about the main character living the same day over and over in a time loop. Each loop begins with him waking up in bed. If we compute the similarity between the recording's segment embeddings and the embedded prompt "person waking up from bed", we get the graph shown in Figure 2.

<figure class="article-figure similarity-figure">
<a class="figure-image" href="/article-assets/embedding-ad-detection/v1/figure-2.svg"><picture><source media="(max-width: 600px)" srcset="/article-assets/embedding-ad-detection/v1/figure-2.mobile.svg" width="320" height="1015" /><img src="/article-assets/embedding-ad-detection/v1/figure-2.svg" alt="Cosine similarity to Person waking up from bed across the recording, with a dashed line at 0.65 cosine similarity and sample frames from fourteen distinct peaks above that value, including bedroom scenes and commercials." width="820" height="842" loading="lazy" /></picture></a>
<figcaption>Figure 2. Cosine similarity between video segments and the text prompt “Person waking up from bed”. The numbered stills below the graph sample 14 distinct peaks with cosine similarity above 0.65, including commercial matches.</figcaption>
</figure>

Interestingly, it gives a pretty good indication of where each loop starts in the movie. This is another reason to use an embedding model for commercial break detection: the same video embeddings can also help users find scenes by describing what they want to see.

Overall, this was a very successful experiment, and it gave me a few ideas for other uses of embedding models. One is saving bandwidth when computing power is plentiful. Another is letting an agent describe what to watch for in a live video feed, then waking it up when something matching that description appears. Those might be worth experiments of their own.

## References

1. R. Girdhar, A. El-Nouby, Z. Liu, M. Singh, K. V. Alwala, A. Joulin, and I. Misra. ImageBind: One Embedding Space To Bind Them All. *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*. 2023. pp. 15180–15190. [https://openaccess.thecvf.com/content/CVPR2023/html/Girdhar\_ImageBind\_One\_Embedding\_Space\_To\_Bind\_Them\_All\_CVPR\_2023\_paper.html](<https://openaccess.thecvf.com/content/CVPR2023/html/Girdhar_ImageBind_One_Embedding_Space_To_Bind_Them_All_CVPR_2023_paper.html>) <!-- Citation key: @girdhar2023imagebind -->

2. H. Wolff. RAG 101: Demystifying retrieval-augmented generation pipelines. 2023. [https://developer.nvidia.com/blog/rag-101-demystifying-retrieval-augmented-generation-pipelines/](<https://developer.nvidia.com/blog/rag-101-demystifying-retrieval-augmented-generation-pipelines/>) <!-- Citation key: @wolff2023rag -->

3. Wikipedia contributors. Cosine similarity. [https://en.wikipedia.org/wiki/Cosine\_similarity](<https://en.wikipedia.org/wiki/Cosine_similarity>) <!-- Citation key: @cosineSimilarity -->

4. S. Dua and H. Schechter Vera. EmbeddingGemma 2: An open, lightweight multimodal embedding model. 2026. [https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/](<https://blog.google/innovation-and-ai/technology/developers-tools/embeddinggemma-2/>) <!-- Citation key: @dua2026embeddinggemma2 -->

5. Blender Foundation. Big Buck Bunny. [https://peach.blender.org/](<https://peach.blender.org/>) <!-- Citation key: @bigBuckBunny -->

6. Blender Foundation. About Big Buck Bunny: License. [https://peach.blender.org/about/](<https://peach.blender.org/about/>) <!-- Citation key: @bigBuckBunnyLicense -->

7. erikkaashoek. Comskip. [https://github.com/erikkaashoek/Comskip](<https://github.com/erikkaashoek/Comskip>) <!-- Citation key: @comskip -->

8. Unsloth AI. Qwen3.8-27B-GGUF. [https://huggingface.co/unsloth/Qwen3.8-27B-GGUF](<https://huggingface.co/unsloth/Qwen3.8-27B-GGUF>) <!-- Citation key: @qwen38gguf -->

9. OpenAI. Codex app server. [https://learn.chatgpt.com/docs/app-server](<https://learn.chatgpt.com/docs/app-server>) <!-- Citation key: @openaiAppServer -->

10. Earendil Works. RPC mode (Version 1.1.0). 2026. [https://github.com/earendil-works/pi/blob/abe508e1b89912adde45528136c3221eb69acdd7/packages/coding-agent/docs/rpc.md](<https://github.com/earendil-works/pi/blob/abe508e1b89912adde45528136c3221eb69acdd7/packages/coding-agent/docs/rpc.md>) <!-- Citation key: @pi2026rpc -->
