Measuring FedRAMP Moderate: How Far Can API Evidence Go?

We Estimated 75%. Then We Built It.

When we first modeled this problem, the most defensible planning estimate was that a mature, API-driven evidence system could produce a meaningful, repeatable measurement for roughly 75% of FedRAMP Moderate. The model used the official Rev. 5 Moderate baseline of 323 controls as its denominator, placed a plausible automated measurement path on about three quarters of them, and carried medium confidence with a range of 70 to 80 percent until a real scan existed.

Then we stopped estimating and built the system. Over a two-day build, a Node.js engine with YAML-defined collectors pulled 643,297 evidence documents from eleven source systems into Elasticsearch and evaluated deterministic rules against them. By the end, 196 of 323 controls, or 60.7 percent, had at least one machine measurement producing a pass or fail verdict. A control-by-control adjudication of everything still unmeasured concluded that only 2 of the 323 controls are genuinely manual.

MetricModeled estimateMeasured after two days
Evidence visibility95-98%321 of 323 controls leave machine-collectable evidence somewhere.
Automated measurement70-80% at maturity60.7% measured: 196 of 323 controls with deterministic verdicts.
Measured satisfactionNot modeled13 controls fully passing, 56 partial, 127 failing honestly.
ComplianceNot inferredStill not inferred. Measurement is not authorization.


Coverage does not equal compliance, and the measured numbers make that distinction concrete. Of the 196 measured controls, only 13 were fully compliant on the first full run. The system can measure a control continuously and conclude that it is not satisfied, and a system that measures more will initially look worse than one that measures less. That is the point. The dashboard must report those distinctions instead of collapsing them into one reassuring score.

Why This Works: Policy and Data Both Point Here

The direction of federal policy supports this architecture. FedRAMP says authorization and continuous-monitoring artifacts should be exchanged as machine-readable data through APIs where feasible, and it identifies OSCAL as the interoperability standard. See FedRAMP's automation and efficiency guidance.

FedRAMP also publishes its requirements as structured source data and tells automation services to consume that data instead of scraping rendered pages. The current source is the FedRAMP source-data service. Our engine takes this at its word: it loads the official Rev. 5 Moderate OSCAL baseline profile directly, and the loaded control set is mechanically verified against the published profile on every load, 323 controls matching in both directions. FedRAMP High is the same mechanism with a larger baseline of 410 controls; only control membership and assessment scope change.

NIST SP 800-53A organizes assessment procedures around three methods: examine, interview, and test. The test method is where APIs change the economics, because tested behavior is exactly what cloud control planes, identity systems, and CI/CD platforms expose as queryable state. See NIST SP 800-53A Rev. 5.

The working hypothesis, now supported by a real scan rather than intuition, is this: almost every control leaves evidence in a system, and materially more of that evidence can be collected and measured programmatically than document-first compliance workflows assume.

The System We Built

The system continuously collects, normalizes, indexes, and measures evidence relevant to FedRAMP Moderate across cloud providers, infrastructure, code repositories, corporate systems, and documentation platforms. It is a measurement system, not an authorization engine and not a generic GRC product. The first authorization boundary under measurement is our own Rabbit delivery platform.

What exists today:

  • A Node.js CLI with five strictly separated stages: collect, derive, assess, roll up, and export.
  • Every definition in YAML: 221 collector checks, 306 per-document rules, and 327 aggregate conditions, plus control mappings, authorization boundaries, and labeled cloud accounts. Configuration is versioned, so every measurement can identify the exact definitions that produced it.
  • Source-faithful JSON snapshots with timestamps, deterministic document ids, provenance, and sync versions. Re-syncs are idempotent.
  • OSCAL output: observations, per-control findings, and POA&M items built from the same evidence chain.
  • Elasticsearch indexes categorical and numerical evidence as first-class analytical fields, not opaque blobs.
  • Collection is separate from measurement, so evidence can be replayed against new rules without touching a source API.
  • Full assessment runs write a fresh versioned evidence index and swap an alias atomically. Readers never see a half-built state, and a failed run leaves the previous generation live.
  • Every verdict is falsifiable: each observation records the rule evaluated, the measured value, and links to the exact source documents that produced it.


We set 60 percent automated measurement coverage as the initial milestone and reached it within the first two days. The mature target remains 75 percent, while preserving honest no-data outcomes whenever the evidence cannot support a deterministic conclusion.

Separate Collection from Measurement

Collectors retrieve facts from source systems, normalize them, and preserve their provenance. Evaluators operate only on stored evidence. At no point does an evaluator make a live call to AWS, GitHub, Jira, Confluence, Drata, or any other source while deciding a control outcome. This rule held for the entire build, and it is the single most important architectural decision in the system.

The separation is structural, not cosmetic. One collected fact can support several controls. Historical evidence can be re-evaluated after a rule changes. Collection failures remain distinguishable from control failures. Rules can be tested without repeatedly calling production APIs. Raw source state remains available for independent review.

The pipeline as implemented:

  1. Collect immutable source responses across accounts, repositories, clusters, and corporate systems. Collectors never judge.
  2. Derive cross-evidence views, because many controls only become measurable across joins: pull requests joined to their reviews, namespaces joined to network policies, one human identity resolved across GitHub, Drata, and cloud IAM.
  3. Assess deterministic rules against the stored evidence, producing OSCAL observations. Failed rules can automatically draft POA&M entries that carry the measurement, the exact source request, and the affected resources.
  4. Roll up per-control state, both organization-wide and per authorization boundary.
  5. Export the whole chain as OSCAL assessment results and POA&M JSON.


You can re-measure without re-collecting, and re-collect without re-measuring. Collection is also deliberately broader than the current rules require: the system models resources and relationships rather than gathering the minimum fields needed to satisfy one control mapping, so it can answer questions that have not yet been written.

Where the Evidence Actually Lives

The 643,297 collected documents came from eleven source systems: AWS, GCP, Kubernetes, GitHub (repositories, organization settings, workflows, audit events, code scanning, secret scanning, dependency alerts, SBOMs, and releases), Jira, Confluence (the policy library and existing SOC material), Drata (161 monitoring tests, per-person personnel checks, device posture, the vendor register, and policies), New Relic (alerting and incidents), recorded meetings (2,846 recordings, which turn recurring security committee meetings and tabletop exercises into attestation evidence), infrastructure-as-code repositories, and the FedRAMP Marketplace API itself, which machine-verifies that the cloud providers we lean on actually hold FedRAMP authorizations.

Infrastructure-as-code is first-class evidence. Deployment definitions describe intended boundaries and configuration; runtime APIs describe deployed state. The system retains both categories explicitly:

  • declared_state: what repositories and deployment definitions say should exist.
  • observed_state: what cloud and operational APIs say actually exists.


Evaluators then measure drift instead of treating either source as complete. A derived system model reconciles declared components against observed cloud state, and that reconciliation is itself control evidence: a component that appears in both passed through pull-request review on its way to production. The same relationship model is thorough enough to derive boundary, asset, and dependency views from evidence rather than maintaining each diagram by hand.

Collection stays organization-wide on purpose. Boundary definitions then classify every observation, and compliance rolls up both org-wide and per-boundary states. Breadth is what proves a boundary is complete, and it is also how a false boundary assumption gets caught.

Evidence, OSCAL, and Elasticsearch

Every evidence record identifies its source system and collector, collection timestamps, account and boundary attribution, the API operation or repository path that produced it, the normalized fields extracted from it, the FedRAMP controls it supports, the rule and definition versions that evaluated it, and its success or error state. JSON snapshots preserve source state faithfully enough to reproduce an evaluation.

OSCAL provides the stable control identifiers and the interchange format. The NIST OSCAL project is the relevant standard, but OSCAL is the output layer, not the storage layer: it should not flatten away the provenance or detail of an AWS response, a Git commit, a Jira transition, or a Confluence approval. Our exports carry observations, findings, and POA&M items whose evidence links point back to the exact source documents.

Elasticsearch answers the operational questions: How many controls have current evidence? Which measurements pass, fail, or have no data? Which systems are outside policy? Which controls depend on collectors that have never returned a document? What changed between scans? Which findings have corresponding Jira stories or GitHub issues?

Each measured control rolls up to one of compliant, partial, non-compliant, no-data, or manual, and every verdict links to the observations behind it. The provenance chain is tested, not assumed: an adversarial review of the live indices resolved every one of the 3,481 per-document evidence links back to its source document, with zero dangling references. Missing or stale evidence must never become a satisfied result.

Put Every Definition in YAML

Hardcoded secrets, account lists, control mappings, filters, thresholds, or exclusions make a system like this impossible to audit and dangerous to operate. In our definitions, YAML maps an integration to the name of a local environment variable. Secret values never appear in YAML, source code, logs, or stored evidence. Sources whose credentials are missing are skipped, not failed, and the skip is recorded.

The layered shape, simplified from the working definitions:

definitions/measurement/github.yml
CONFIGURATION · YAML 
credentials:
  github:
    token_env: GH_TOKEN        # env var NAME, never a value

collectors:
  github.secret_scanning_alerts:
    tool: gh-api
    path: /orgs/{org}/secret-scanning/alerts

measurement:
  - check: github.secret_scanning_alerts
    controls: [ra-5, si-2]
    aggregate:
      id: open-secret-alerts
      count_where: { state: open }
      max: 0
    onFail:
      poam: true               # failure drafts a POA&M entry


The same configuration layer defines accounts, boundaries, index names, control-to-evidence mappings, applicability, and schedules. Because the definitions live in git, every measurement is attributable to the exact rule text that produced it, and a rule change is a reviewable pull request rather than a silent settings edit.

What Two Days of Measurement Produced

Coverage moved in three visible steps. The first working rule set measured 35 of 323 controls, about 11 percent. A deepening pass over the same collected evidence, writing rules against documents that were already indexed, brought it to 79 controls, or 24 percent. By the end of the second day the roll-up reported 196 of 323 controls measured, 60.7 percent, verified by a full compliance run rather than a spreadsheet.

The most instructive moment was watching the compliant count fall while coverage rose. When rules got stricter, fully compliant controls dropped from 18 to 9. The measurement got more honest, not worse. Any team building a system like this should expect that curve and welcome it, because the alternative is a dashboard that flatters.

We then attacked our own measurement layer. A dedicated QA pass replayed all 284 rules against live payload shapes and found 41 defects, every one of which was fixed and re-verified. Twelve were rules that could only ever fail because a field path was wrong. One was worse: a broken field reference made a secret-scanning rule pass unconditionally while 313 open secret-scanning alerts across 36 repositories sat invisible. Fixing it flipped the rule to an honest failure and drafted the corresponding POA&M entry. A separate adversarial review then attacked the live indices and the OSCAL exports: 14 of 20 test groups held, including the complete provenance chain, and the review still found two critical export defects, closed POA&M items exporting as open and an assessment-results file missing a required OSCAL field. We publish that willingly. A measurement system that cannot find its own defects has no business reporting yours.

The dashboard keeps the metrics separate: source coverage, evidence coverage, automated measurement coverage, measured satisfaction rate, and compliance coverage. The delivery ladder stands at 60 percent reached, 75 percent as the mature target, roughly 85 percent as the measurement opportunity ceiling, and near-total evidence visibility as the collection target. None of those numbers is an autonomous compliance claim.

Where Automation Actually Stops

Calling hundreds of controls "manual" is usually a failure of source discovery, and the build let us test that claim control by control. After the initial mapping, 97 of the 323 controls still carried a manual classification. We re-adjudicated every one of them against NIST control intent and the evidence already sitting in the index. The result: 45 are measurable today from evidence already collected, 24 are inherited from FedRAMP-authorized cloud providers, with the inheritance itself machine-verified through the FedRAMP Marketplace API, 19 are event attestations where the occurrence is machine-evidenced through recorded exercises, screening records, or signed artifacts while the judgment inside the event stays human, and 7 wait on exactly one named new source, such as identity-provider session policy or a load balancer idle-timeout attribute.

That leaves 2 of 323 controls that are genuinely manual: personnel sanctions, which is legal and HR adjudication, and allocation of resources, which is an executive budgeting decision. Automating either would be theater.

The honest limit is final judgment, not evidence access: whether a policy is substantively adequate, whether personnel can execute a procedure, whether a tabletop exercise demonstrated effectiveness, whether an independent assessment covered the required scope. FedRAMP's independent-verification requirements are an intentional boundary around some conclusions. See FedRAMP's independent verification and validation guidance. The system's job is to hand that judgment stronger, fresher, more complete evidence. It does not impersonate independence or manufacture certainty.

Remediation closes the loop. The engine holds just over a thousand POA&M items: failed measurements auto-draft entries carrying the measurement, the exact source request, and the affected resources, and findings whose checks stop failing are automatically closed on the next full assessment run, with history kept. GitHub issues and Jira stories sync into the same index, and neither tracker is treated as the source of technical truth.

What the Build Answered, and What Remains

Several questions we once considered open were settled by building:

  • Which baseline is authoritative? The official FedRAMP Rev. 5 OSCAL profile, loaded directly and mechanically verified on every load.
  • What is the denominator? Controls, for now: 323 for Moderate, 410 for High. Assessment-objective granularity is the next refinement, and it will move the percentages.
  • Declared or observed state? Neither is authoritative. The system stores both and measures the drift between them.
  • When is a control truly manual? Only after a recorded adjudication of its evidence routes. That standard reduced "manual" from 97 controls to 2.


The decisions that still determine whether the percentage stays defensible:

  1. What accounts, tenants, repositories, environments, vendors, and systems are inside each authorization boundary, and how is that inventory proven complete?
  2. What evidence freshness and sufficiency rules apply to each control, and eventually to each assessment objective?
  3. How are inherited and shared CSP responsibilities represented, and on what cadence are they revalidated?
  4. How do collectors receive least-privilege access without distributing long-lived credentials?
  5. What retention and index-lifecycle rules preserve useful evidence history at acceptable cost?


The best answer today is therefore not "we are 60 percent compliant." It is: 60.7 percent of FedRAMP Moderate is under deterministic, repeatable, falsifiable measurement after two days of engineering, 75 percent is a credible mature target, and only 2 of 323 controls resisted automation entirely. Every future percentage will be published the way this one was: with the denominator, the boundary, the source inventory, the failures, and the evaluator coverage alongside it.

For the broader business context, see UDX's small-business guide to FedRAMP, our compliance and security engineering practice, and the Rabbit delivery platform whose boundary the system measures first.

Turn Compliance Evidence into Measurable Data

Talk to UDX about building a defensible evidence pipeline across your cloud, code, documentation, and operational systems.