A /
AI agents
Multi-step execution, state transitions, decision paths, failure recovery and tool interaction. Does the agent complete the intended task under controlled adverse conditions?
Independent AI Validation & Research
Find the failures your evaluation says aren't there.
Independent adversarial validation for AI agents, tool-using systems, evaluation pipelines and technical benchmarks. We test the system — and the measurement that says it works.
Independent. Evidence first.BED VIBE GKILIS · Registered in Norway · Org. 935 267 897
01 What we validate
Behavior. Regressions.
The instruments in between.
Start with a claim your system makes. Define what would disprove it. Test the paths and conditions that a headline score can hide.
B /
Agent regression
Compare behavior after changes to models, prompts, context, tools, orchestration or infrastructure. Identify regressions that an aggregate success rate conceals.
C /
MCP & tool regression
Tool schemas, permissions, routing, selection and state propagation. Exercise malformed responses, unavailable tools and unexpected behavior across the interaction.
D /
Evaluation & benchmark validation
Test whether an evaluation measures what its authors think it measures. Inspect labels, leakage, metric defects, silent exclusions and evaluation-code bugs.
Thresholds · Invalid assumptions · Non-independent samples · Misleading aggregate scoresE /
Performance & optimization validation
Determine whether an optimization improved the system or moved the failure. Compare latency, throughput, cost and quality against a reproducible baseline.
Quantization · Inference configuration · Model and backend changes · Caching · Batching · HardwareF /
RAG & retrieval systems
Indexing, corpus state, answerability, grounding, citations and omissions. Trace silent retrieval failures through ranking, chunking and the evaluation harness.
G /
Specialized research & scientific validation
Focused work in synthetic speech evaluation, technical benchmarking and scientific software, including published research on quantum-compiler benchmarking. Scope follows the available evidence and a defensible reference.
02 The method
A finding should survive
being run again.
The result is an evidence package you can inspect and reproduce. The conditions and limits travel with every finding.
Establish the claim.
Agree what “correct” means, what is being measured and where the claim stops.
Reproduce the baseline.
Record versions, inputs, settings and controls before interpreting an improvement or a failure.
Attack the measurement.
Check the scorer, labels, sampling and exclusions. Ask what could fail without moving the score.
Construct controlled cases.
Introduce adversarial, shifted or pathological conditions with an explicit expected behavior.
Measure against a reference.
Use defensible comparisons and controls. Separate a system defect from noise or a broken test.
Return reproducible evidence.
Deliver the commands, raw output, characterized failures, corrections and limits.
What remains after the engagement
Executable reproductions, controlled test cases and raw evidence. Datasets or corpora where applicable, corrected measurements, regression tests and concrete recommendations. You keep the harness and the findings.
03 Public evidence
We publish failures even when
the failed system is our own.
Public reports, issue threads and corrections. Inspect the conditions behind a finding before trusting the headline.
A call edge between files with no import relationship.
Testing an external code indexer exposed incorrect relationships between same-named functions. Every measurement was made from outside, against a pinned build, with no access to the source. The maintainer confirmed each finding at source level in public, and located the last one himself.
What was tested
Small source examples isolated incorrect cross-file call resolution, phantom import edges and misleading edge descriptions. A pre-registered before/after across two releases then established a third and deeper finding: the graph stores one edge per source, target and relation type, with the field identifying which import outside that key — so an import that resolved and was merged away leaves exactly what an import that never resolved leaves, which is nothing. Fifteen apparent regressions were two. The findings concern these specific cases in Octocode, not every result produced by the indexer.
Corrections we made against ourselves
Three denominator analyses were retracted after the maintainer disclosed that an internal sort was unstable; the instability was then confirmed independently from outside. An early figure taken from a contaminated index store was withdrawn and re-measured. The count of affected statements was revised downward three times, each time after checking one layer further.
A confident zero concealed different outcomes.
An internal metric returned the same clean zero for genuine agreement, a task with only one possible answer, and calls that failed in transport. That number did not establish answer quality.
What the finding means
The published account reports that the metric was anti-correlated with answer quality in the tested setting. Distinguishing transport failure from a valid observation is part of validating the measurement itself.
The scorer audit reduced the headline result.
A published result was corrected from a 32 percentage-point improvement to 15.8 after auditing the scorer. The paper and correction remain public.
Scope and limits
The correction rescored 1,786 saved items from the same corpus and Qwen3 model setting, using authored gold orderings. It does not establish that the corrected improvement generalizes to other corpora or models.
Zero errors. An empty implementation.
An evaluation function returned an empty list. Its output was recorded as a measurement in every result row, so an unimplemented check appeared to report no errors.
Why this matters
A result can be structurally valid and still contain no measurement. The published finding separates an absent check from a test that actually ran and passed.
01 — The problem
An instrument that answers with confidence has not been checked. It has only been believed.
On 28 August 2026 I audited 38 of my own live pages. The two findings worth writing down were not about the pages. They were about the tools I was using to judge them — and both tools were wrong, in opposite directions.
Wrong in one direction
One script reported three pages I had just fixed as still broken, because it did not recognise a relative link. A working fix looked like a failure.
Wrong in the other
One date parser had thrown an exception on every crawl-statistics read since it was written. The report that answers “is the crawler arriving at all” had never once returned a number. A working crawler was invisible.
Why it matters to you
I spent a day acting on numbers produced by instruments nobody had checked. That is the ordinary state of most evaluation code — including the code that tells you your AI system is working.
- fetch sitemap → 38 URLs
- GET × 38 … all 200
- check followable links …
- 5 pages: zero followable links — a crawler arriving there can go nowhere
- 2 navigation controls: <button onclick> — usable with a mouse, invisible to a crawler
- read crawl statistics …
- date parser raised on every read since it was written
- that report has never once returned a number
- re-check the 3 pages the checker calls broken …
- already fixed — the checker did not recognise a relative link
- two instruments wrong, in opposite directions. Both found by measuring them.
Published in full, with the numbers and the fixes:
Two of My Instruments Were Wrong, in Opposite Directions
02 — What an unchecked instrument costs
A compiler got 5.37% slower. The benchmark called it a 10% regression a quarter of the time.
IBM's Benchpress pins the seeds that build its circuits and passes no seed to the code that compiles them. On one circuit, over 400 seeds per version, the real change between two Qiskit releases is +5.37%. The suite's own three-run protocol reports that as a ≥10% regression 24.4% of the time. Somebody merges or reverts a pull request on that answer.
False-verdict rate by runs per version
+5.37%
The real change between the two releases, measured over 400+ seeds per version on one circuit.
39
Circuits in the pre-registered replication, reported in full — including where it disagreed with me.
4
Claims withdrawn on the record, and six defects an independent audit found in my own measuring apparatus.
Full method, pre-registration and withdrawals:
The Compiler Got 5% Slower ·
permanent record on Zenodo
03 — What a real measurement changes
Thirty-eight pages, fetched live. A number per page, never a verdict.
Every URL in the sitemap, over plain HTTP with no JavaScript executed. An earlier audit on this same estate had returned “all clean” for months while the homepage was handing crawlers a fraction of its navigation in a form they could not use. That is why the rule is a number per page rather than a pass mark.
38 pages, one tile each — scroll to run the audit and the fix
Fetched, 200
5 with zero followable links
Repaired and re-measured
38 pages · before 28 Aug 2026 → after
Pages with zero followable links50
Navigation a crawler cannot follow20
Pages with an incorrect h1 count10
Missing robots directive10
Reported unchanged in the same table, because a measurement that only shows movement is an advertisement: canonical problems 0 → 0, images with no alt attribute of 199 0 → 0, non-200 responses 0 → 0.
The worst offender had two navigation controls, both
<button onclick> — usable
with a mouse, invisible to a crawler.
Read the audit
04 Commercial engagements
A defined target.
A concrete body of evidence.
Fixed scope, agreed in writing before work starts. Conducted personally by Panagiotis (Panos) Gkilis. Remote, using a copy, staging environment or public API.
Targeted validation
€400Paid technical scoping / a narrowly scoped investigation
A focused first step around one failure mode or technical question. Establish the useful scope and test a specific concern.
Credited in full toward a €2,000 engagement if the project continues.
- One agreed failure mode or validation question
- Reproducible finding, if a defect is established
- A written account of what was tested and what remains unknown
Full validation
€2,000Approximately two weeks · One defined system / evaluation target
An independent investigation of the system's behavior and the measurement used to judge it.
- Agreed claims, baseline and controlled failure cases
- Executable reproductions and preserved raw evidence
- Failure characterization and explicit limits
- Corrected measurements and regression tests where applicable
- Concrete recommendations and an evidence report
This is not a code review. This is not a penetration test.
We validate claims, measurements and system behavior.
No production access required.
A passing set of tests does not certify that a system is safe.
What do you need from us?
A way to provide inputs and read outputs, plus a conversation at the start to agree what “correct” means. A repository copy, staging system or public API can be enough. Access and scope are agreed before work begins.
What if you find no failure?
You receive a written account of the cases, controls and limits. Absence of a reproduced failure is reported as that, with no claim that the system is sound beyond the tested conditions.
Who keeps the work?
You keep the corpora, harness and reproductions produced for the engagement. Findings remain private unless you choose publication; published results include their limits and corrections.
How do we choose the scope?
Start with what the system does, how you judge it today and the failure you most want to rule out. The €400 investigation can establish a useful, bounded target for the full engagement.
05 Why trust the work
Independent by structure.
Accountable through evidence.
BED VIBE GKILIS is a Norwegian sole proprietorship for independent research and engineering. Panagiotis (Panos) Gkilis conducts the validation work and builds the instruments behind it.
Public research, open-source packages and operating AI systems make the work inspectable. Corrections are part of that record, including corrections to our own measurements.
No incentive to certify a system merely because the client wants a green result.
- Legal entity
- BED VIBE GKILIS
- Organisation number
- 935 267 897 · Norway
- Registry
- Enhetsregisteret · Brønnøysundregistrene
- Researcher & engineer
- Panagiotis (Panos) Gkilis
- Public work
- GitHub · Research notebook · Engineering portfolio
- Direct contact
- bedvibe@bedvibe.studio
06 Research & lab
Instruments built from
problems encountered firsthand.
Open tooling, published experiments and operating systems. The broader lab work supports the validation practice.
AI evaluation & agent/tool validation
Investigations of evaluation defects, tool behavior and code intelligence, with public issue threads and corrections.
Research notebook ↗Tool investigation ↗Retrieval coverage & grounding
notchecked records which checks ran, could not run or were outside scope. BookProof checks AI-generated summaries and answers against their source manuscript.
notchecked / PyPI ↗BookProof ↗Synthetic speech & training checks
Open-source tools for TTS quality assurance, training-run failure detection and speaker-verification false rejection.
ttsproof ↗trainproof ↗spkproof ↗Scientific & quantum benchmarking
Research into compiler randomness, regression decisions and the reliability of benchmark measurements. Results retain their experimental scope.
Study and replication ↗Research record ↗Observability & measurement infrastructure
An operating observability platform tracks AI crawler and search-engine activity with continuously collected evidence.
Open the Observatory ↗Engineering in operation
Retrieval systems, evaluation tooling, speech infrastructure and the systems that connect them.
Engineering portfolio ↗Source repositories ↗07 Start with a system
Have a system whose evaluation you don't completely trust?
Let's test it.
Tell Panos what the system does, how you evaluate it today, and the failure you are most concerned about. Include a repository, staging URL or public API if available.
bedvibe@bedvibe.studio ↗Direct contact · Scope agreed before work begins