Detection
Did we surface what deserves review?
Check missed findings and unnecessary alerts. A suspicious-looking action can be routine; a small configuration change can affect authorization.
Evaluation methodology · Version 1
A finding should stand up to scrutiny.
For a small team, a missed signal matters. So does an alert that sends someone chasing a story the logs cannot support.
We test whether Flare helps you investigate cloud and SaaS activity with source evidence, appropriate uncertainty and a useful next step, without operating a full SIEM.
ObservedA role-tag change was requested.
TagRole · project=finance-prodContextThe supplied policy uses that tag.
The change is relevant to authorization.
Not establishedEffective access or compromise.
The request alone cannot prove either.
Synthetic regression case, examined below.
Four questions, not one quality score.
We grade detection, factuality, ranking and explanation separately on a 1–5 scale. A clear explanation cannot cancel out a missed finding. Correct detection cannot excuse an invented fact.
Factuality
Does the evidence support each claim?
Verify actors, targets, outcomes, counts and citations. A requested change is not automatically an effective permission grant or a completed data access.
Risk ranking
Is the right finding first?
Assess relative priority using the supplied context. Check ordering and numerical score stability separately. A risk score is not a probability of compromise.
Explanation
Can someone act on the answer?
Require the relevant evidence, why it matters, what remains unknown and a concrete next investigation step. Fluent prose alone does not pass.
Grades describe performance against a case-specific rubric. A 5 means the judged answer meets that case’s criteria. It does not mean 100% real-world accuracy.
The hard part is knowing what the logs do not prove.
Our evaluation work includes positive cases, benign controls and deliberately incomplete or conflicting evidence. These are the distinctions we test.
Action and outcome
Denied writes, read-only calls and policy-relevant changes
Keep attempted activity separate from completed effects.
Identity and scope
Workspace customer IDs, conflicting profile IDs and missing identity fields
Do not combine separate accounts or infer an alias from a matching email alone.
Evidence and citations
Duplicate records, conflicting event IDs and unsupported references
A citation must support the specific claim, not simply point to a related event.
Coverage and abstention
Truncated inputs, missing context and incomplete windows
Explain what was examined. No finding in a limited window is not proof of a safe environment.
Adversarial inputs
Instructions embedded in logs and misleading contextual text
Treat source content as evidence, not as authority to change the task.
Follow-up questions
False premises, requests beyond retained evidence and correction of an earlier answer
Challenge unsupported premises and retract a specific error instead of inventing a reassuring explanation.
Coverage is source-specific. A provider represented in an engine test does not imply that every connector, log type or workflow is available in the product. Review current source coverage →
Case study · AWS authorization
When a tag change deserves a closer look.
Attribute-based access control (ABAC) lets permissions depend on tags. This synthetic regression case pairs an IAM role-tag request with explicit policy context. It is a test fixture, not a customer incident.
Inspect the input
1 synthetic record{
"eventId": "fresh-aws-TagRole",
"eventTime": "2026-09-10T10:00:00Z",
"eventName": "TagRole",
"eventSource": "iam.amazonaws.com",
"userIdentityArn": "arn:aws:iam::111122223333:role/new-operator",
"requestParameters": {
"roleName": "build-role",
"tags": [
{
"key": "project",
"value": "finance-prod"
}
]
}
}Separately supplied policy context
Confirmed policy for build-role allows secretsmanager:GetSecretValue on resources whose project tag equals aws:PrincipalTag/project. Its previous project tag was development; finance-prod resources contain production finance secrets. Approval for this change is unknown.
What a defensible answer needs to do
- Catch the relevant change. The requested tag value matters because the supplied policy links the role’s project tag to access. Generic tagging alone is not the rationale.
- Name the evidence boundary. This record does not establish effective permissions, successful secret access or compromise.
- Make the next step specific. Review the effective policy and resulting tags, confirm approval, and inspect subsequent access evidence if available.
What the regression taught us
The earlier capture returned no finding. The repaired path surfaced the policy-relevant change while retaining the uncertainty about its effects.
The useful change is visible in the answer: it identifies why the request matters, states what the record cannot prove and gives a concrete next investigation step.
Read the captured explanation after the repair
1 cited audit record: "TagRole" (no error recorded; outcome details may be incomplete) by "arn:aws:iam::111122223333:role/new-operator" targeting "build-role" at "2026-09-10T10:00:00Z"; Recorded tags: [{"key":"project","value":"finance-prod"}]. The supplied policy links principal tag "project" on role "build-role" to authorization. Review the requested tag change against that policy and approved attributes; the request alone does not establish effective permissions, successful data access or compromise.
This is the saved explanation, not a newly generated answer. The policy context above is part of the test input; it was not discovered from this single audit record.
Keep the evidence when the answer changes.
Evaluation is a record of behavior, including failures. We distinguish a product defect, a reference-policy disagreement and a judge error before deciding what to fix.
Changes to the product, rubric or reference policy belong in the version record. They should not silently rewrite an earlier result.
Define the expected behavior
Record the input, source policy, available context and evidence boundaries. Include controls where a superficially similar action should produce a different conclusion.
Capture the actual execution
Keep the product output, evaluated source revision and model configuration together. Separate engine-harness evaluation from a full hosted-product check.
Check structure before semantics
Verify selected findings, evidence membership, counts and valid output. A malformed or contradictory judgment is excluded and reported, never converted into a passing grade.
Grade, investigate and retest
Use the four-theme rubric to find weaknesses. Preserve original captures and failed controls. After a fix, reuse unaffected evidence and retest the changed behavior and its boundaries.
Findings & limitations
What we learned, and what remains unproven.
The evaluation work exposed missed policy context, weak explanations and mistakes in the grading process itself. We keep those failures in the record alongside the repairs.
The AWS case above shows what a useful improvement looks like: a missed signal becomes a reviewable finding, with evidence, uncertainty and a specific next step. The dated technical appendix preserves the detailed measurements.
What these results do not establish
- Independent validation. The producer and judge use models from the same family. References were authored by the implementing agent; scenarios include reused and correlated variants.
- A fully calibrated judge. The judge penalized a deliberately impossible record count less severely than the rubric required. That severity-calibration check remains failed.
- Stable numerical risk scores. A small input-order probe preserved ordering and evidence while numerical scores usually changed. It does not establish broad repeatability.
- Production precision or recall. Rubric averages do not measure missed attacks or false positives at real customer base rates. The full evaluation used an engine harness with live inference, not a complete hosted-UI replay.
- Market superiority. We have not run a matched head-to-head benchmark against competing products. Independent, unseen-case evaluation and customer outcome measurements remain future work.
Technical appendix · October 28, 2026
These internal regression results help us identify improvements and failures across product versions. Scores reflect case-specific evaluation rubrics; they do not estimate real-world detection accuracy or establish independent validation.
Displayed averages are rounded to two decimal places. Exact saved values remain in the download. This snapshot does not provide uncertainty intervals, and the small ranking cohort and related scenario variants limit generalization.
The matched comparison grades saved baseline and later outputs with the same current rubric. Valid pairs are applicable, valid judgments for each theme, not independent customer incidents. Themes overlap and use different subsets; their counts should not be added together.
Public snapshot v1Full run
99770601Baseline b3ab13c9Scale 1–5| Theme | Earlier baseline | Later full run | Valid pairs |
|---|---|---|---|
| Detection | 4.93 | 4.98 | 224 |
| Factuality | 4.98 | 5.00 | 238 |
| Risk ranking | 4.93 | 5.00 | 14 |
| Explanation | 4.51 | 4.87 | 238 |
Explanation has the largest observed increase in the matched rubric averages. Under the original frozen detection labels, the result remained 218/224 before and after, with one regression subsequently repaired. These different measurements should not be collapsed into a single accuracy claim.
The deployed repair is a separate evidence set.
The full run above evaluated 99770601. Targeted AWS and Workspace repairs were then included in deployment aa8531f5. The earlier aggregate has not been relabeled as a fresh full-suite result for that deployment.
Expanded full run and targeted repairs
| Theme | Mean | Applicable valid cases |
|---|---|---|
| Detection | 4.99 | 390 |
| Factuality | 5.00 | 524 |
| Risk ranking | 4.97 | 37 |
| Explanation | 4.90 | 524 |
524 of 525 current judgment packets were valid in that full run. The excluded Workspace judgment was addressed in the targeted repair work; its answer was reused unchanged and its valid grades were detection 5, factuality 5, ranking 4 and explanation 4.
The repairs also addressed the ABAC miss and three weak Workspace explanations. The original full-run captures and exclusions remain part of the record.
For the selected ABAC repair case, model-judge detection and explanation grades moved from 1/5 to 5/5. This is one targeted repair result, not an estimate of field accuracy. The separate input-order probe preserved ordering and evidence in five of five pairs, but numerical risk scores matched in only one.
The impossible record-count control received factuality 2/5 instead of its predeclared 1/5 target. Exact severity calibration remains failed. Independent evaluator qualification is incomplete.
Download scorecard & provenance (JSON) ↓
Version 1 publishes this dated summary and one synthetic case. It is not a public release of the full evaluation corpus or raw model transcripts.
Inspect the evidence. Decide the next step.
Explore how Flare presents audit findings, source evidence and coverage limits before connecting your own data.