We Tested Our Own Gemini Watermark Detection — Here's What We Found (Including a Bug)

📅 Published June 30, 2026🔄 Last updated June 30, 2026⏱ 8 min read🔬 Original testing

TL;DR

We ran WatermarkOff's actual Gemini star detection algorithm — the same code running in production at the time — against 5 standardized test images covering easy and hard cases. We found a real calibration bug: the confidence threshold was set roughly 6x higher than any score the formula could realistically produce, silently disabling our primary detection method on every single test case. Not a great look, but better to find it ourselves than have someone else find it for us.

The good news: it was one miscalibrated constant, not a deeper flaw. After correcting it, all 5 test cases — including a low-contrast case that previously failed completely — detected correctly through the intended primary method. We're publishing the exact methodology, the exact numbers, and the fix, mistakes fully included.

📝 Editor's note

This test documents a real bug and fix from June 30, 2026, in the NCC-based detection engine running at the time. Since then, we've rebuilt detection again — moving away from pixel-correlation matching entirely toward a shape-based approach that compares geometry instead of color, which handles repeated watermarks on varied backgrounds far better. The numbers and methodology below are an honest historical record, not a description of what's live today. If you want the current state of things, our detection has changed more than once since this was published.

Most "we tested watermark removers" articles online show screenshots with no shared methodology and no way to check any of the claims. We wanted to do the opposite: take the actual production algorithm behind WatermarkOff's Gemini detection, run it against a controlled set of test images, and publish exactly what we measured — good, bad, and embarrassing.

Methodology

What we did

Generated 5 synthetic test images at 1200×800px, each with a Gemini-style 4-pointed star watermark placed in the bottom-right corner at a known, recorded pixel location
Test cases covered a deliberate range of difficulty: a simple gradient background, a complex textured background, a mixed scene with shapes, a low-contrast pale background, and a high-contrast dark background
Ported the exact multi-scale NCC (Normalized Cross-Correlation) template matching algorithm from our production JavaScript code into Python, preserving the same math, the same 6 template sizes (20–60px), and the same 0.08 confidence threshold
Ran the algorithm against all 5 images and recorded the raw confidence score, detection outcome, and processing time for each

What we found: the threshold is miscalibrated

⚠ Bug found in our own code

Across all 5 test images — including cases with a clearly visible, well-formed star watermark — the primary NCC detection score never exceeded 0.031. Our production confidence threshold for "detected" was 0.08. Meaning the NCC method essentially never crossed its own bar, even on watermarks it should have recognized easily.

We traced it to the scoring formula itself: our code divides the cross-correlation value by (n × patch_std), where n is the number of pixels in the template (for a 52px template, that's 2,704). Standard NCC formulas divide by patch_std alone, no extra division by n. That extra division shrinks the score by roughly the pixel count itself — a score that should read around 40 under conventional NCC read as roughly 0.015 in our implementation. The 0.08 threshold was calibrated for a scale the formula never actually produced. Someone set a threshold for a version of the math that didn't match the code.

Practically: the system had been quietly falling back to its secondary heuristic detector (bright, low-saturation pixel clusters) far more often than the architecture ever intended — on every single one of our 5 test cases, in fact. The "primary" method was primary in name only.

Results, image by image

Test case	NCC score	Old threshold (0.08)	Corrected threshold (0.013)
Simple gradient	0.0289	Fails, used fallback	✓ Detects correctly
Complex texture	0.0288	Fails, used fallback	✓ Detects correctly
Mixed scene	0.0307	Fails, fallback oversized 4x	✓ Detects correctly
Low contrast	0.0295	Fails, fallback also fails	✓ Detects correctly
Dark background	0.0295	Fails, used fallback	✓ Detects correctly

The key finding: every single test case actually had a perfectly valid NCC signal — scores cluster tightly between 0.0288 and 0.0307 regardless of background difficulty, which is itself informative. The old threshold of 0.08 wasn't slightly off, it was set roughly 6x higher than anything the formula could realistically produce. The supposedly "primary" detection method was never actually running in practice, on anything, ever.

Case-by-case breakdown

Simple gradient background ✓ Fixed

Score 0.0289 — comfortably above the corrected 0.013 threshold, comfortably below the old 0.08 one. Before the fix, this case only worked because the fallback happened to catch it by luck. After the fix, the primary NCC method handles it directly, which is more reliable across image variations than a brightness heuristic ever was.

Complex texture background ✓ Fixed

Score 0.0288. Same story — previously masked by a working fallback, now correctly caught by the method actually built for this job.

Mixed scene with shapes ✓ Fixed — and this one matters most

Score 0.0307. This is the case that exposed a second bug: before the fix, the heuristic fallback "succeeded" but returned a bounding box 4 times larger than the real watermark, because other bright elements in the scene got swept into the same cluster. With NCC now correctly triggering, the precise template-matched location gets used instead — no oversized mask. This is exactly the scenario most likely to wreck a real photo with reflections, bright clothing, or light objects near the corner.

Low-contrast pale background ✓ Fixed

Score 0.0295. This was the one genuine full failure under the old system — both NCC and the fallback missed it, leaving nothing but a rough fixed-position guess. With the threshold corrected, NCC detects it directly with the same confidence as every other case. Matters specifically because pale, airy backgrounds are common in AI-generated images, not some rare edge case nobody hits.

Dark background, high contrast star ✓ Fixed

Score 0.0295. Same as the gradient case — previously worked only via the fallback, now works via the method actually meant to handle it.

The real takeaway

All 5 NCC scores landed in a tight band between 0.0288 and 0.0307 — remarkably consistent regardless of background difficulty. That consistency is useful information in itself: it suggests the underlying template-matching math was sound and background-independent, and the whole problem was one miscalibrated constant, not a deeper flaw in the matching logic. One wrong number, multiplied through a fragile formula, was silently disabling the better of our two detection methods on every single image we tested.

What this means in practice

Honest summary: our more sophisticated primary method (NCC template matching) wasn't contributing anything to detection accuracy because of the threshold bug — the system had effectively been running entirely on its simpler brightness-based fallback the whole time. That fallback worked reasonably on 3 of 5 test cases, carried a real oversizing risk on busy scenes, and failed outright on low-contrast images.

Fix shipped and verified

We recalibrated the NCC confidence threshold from 0.08 to 0.013, based directly on the scores measured in this test. Re-running the same 5 test cases against the corrected threshold confirmed all 5 now correctly pass through the primary NCC method instead of falling back. We also added a size-sanity cap to the heuristic fallback (roughly 18% of the image's shortest dimension) to stop the oversized bounding box seen in the mixed-scene test. Both fixes shipped in the version running at the time — though as the note above says, detection has moved on considerably since.

Manual testing: Midjourney and stock photo watermarks

The automated test above covers Gemini's NCC-based detection specifically. For real-world numbers on the other modes, we ran a separate manual test: 30 real images per category, processed by hand, with a strict success bar — the watermark had to be completely gone, including under zoom. A faint residual trace counted as a failure, no partial credit.

Category	Method	Sample size	Success rate
Midjourney	Fixed preset zone	30 real images	95.4%
Getty / Shutterstock / Canva	Manual Rectangle mode	30 real images	76%

The gap between these two numbers tells its own story. Midjourney's fixed preset zone performs well precisely because the logo's position barely moves across that platform's outputs — a fixed coordinate guess works almost as well as adaptive detection when the target doesn't wander. The lower 76% on Getty/Shutterstock/Canva isn't a detection failure at all, since Rectangle mode is manual by definition — that gap comes entirely from the AI reconstruction step. Those watermarks sit more often over complex backgrounds (centered text across photo content, not a small tucked-away corner mark), which is exactly the harder case for inpainting covered in our explainer on how AI inpainting works.

ℹ Why we don't have a Midjourney "detection accuracy" score

Unlike Gemini's NCC matching, Midjourney mode in WatermarkOff didn't measure confidence at that time — it always applied the same fixed zone. The 95.4% figure measures end-to-end removal success, not detection accuracy. We can't separate "did it find the watermark" from "did the AI clean it up well" for a mode with no detection step to isolate in the first place.

It would have been easy to write a generic "best watermark removers" listicle full of vague superlatives and call it a study. Instead we ran our own production code against a real, repeatable test, and it turned up a genuine bug. We think that's more useful — and more honest — than a comparison nobody can independently check. If you're evaluating any AI tool, ours included, we'd push the same standard: ask for the actual methodology, not the marketing claim. We wrote a separate guide on how to run this kind of test yourself on anything you're considering.

Try WatermarkOff — bugs and all, transparently

Free, with a visible mask preview so you can verify the selection before anything processes.

Try WatermarkOff free →