Documentation
AdvancedPolicy Evaluation

Policy Evaluation

Built-in policy validation rules and compliance checking in evidence collection.

Overview

Policy evaluation happens at three stages:

  1. Pre-collection - Configuration and credential validation
  2. During collection - API scope and permission enforcement
  3. Post-collection - Bundle content and size validation

Goals:

  • Prevent collection of sensitive data
  • Enforce read-only access
  • Validate compliance requirements
  • Ensure bundle integrity

Pre-Collection Validation

Configuration Schema Validation

Checks before collection starts:

✅ Valid configuration:

framework: soc2_type1
controls:
  - CC6.1
sources:
  github:
    mode: token
    token_env: GITHUB_TOKEN
    org: acme
    repos:
      - acme/backend
bundle:
  signing:
    private_key_path: ~/.evidence/keys/private.pem

❌ Invalid configuration:

framework: invalid_framework  # Not recognized
controls: []  # At least one control required
sources:
  github:
    # Missing required fields

Validation errors:

Configuration validation failed:
  - framework: must be one of [soc2_type1, soc2_type2]
  - controls: must contain at least one control
  - sources.github.token_env: required field missing
  - sources.github.org: required field missing
  - sources.github.repos: required field missing

Credential Validation

Before attempting collection:

GitHub token:

Validating GitHub credentials...
  → Testing authentication
  → Checking token scopes
  ✓ Token valid
  ✓ Scopes: repo:read, read:org

AWS credentials:

Validating AWS credentials...
  → Testing STS GetCallerIdentity
  → Checking IAM permissions
  ✓ Credentials valid
  ✓ Account: 123456789012

Google Workspace:

Validating Google Workspace credentials...
  → Testing service account
  → Checking domain-wide delegation
  ✓ Service account valid
  ✓ Scopes authorized

Validation failures:

✗ Validation failed

Error: GitHub token has insufficient scopes
Required: repo:read, read:org
Actual: public_repo

Update token scopes and try again.

Signing Key Validation

Before signing bundle:

Validating signing key...
  → Checking file exists
  → Verifying key format
  → Testing key permissions
  ✓ Private key valid (Ed25519)
  ✓ File permissions: 600 (secure)

Validation failures:

✗ Signing key validation failed

Error: Private key file has insecure permissions (644)
Expected: 600 (owner read/write only)

Fix with: chmod 600 ~/.evidence/keys/private.pem

During Collection Validation

API Scope Enforcement

GitHub token scopes:

✅ Allowed scopes:

  • repo:read - Read repository data
  • public_repo - Read public repository data
  • read:org - Read organization data

❌ Forbidden scopes:

  • repo (write access)
  • admin:org (admin access)
  • delete_repo (destructive)
  • write:org (write access)

Enforcement:

// SDK checks token scopes before use
const scopes = await getTokenScopes(token);

const forbidden = ['repo', 'admin:org', 'delete_repo', 'write:org'];
const hasForbidden = scopes.some(scope => forbidden.includes(scope));

if (hasForbidden) {
  throw new Error('Token has forbidden write scopes');
}

Permission Verification

AWS IAM permissions:

✅ Allowed actions:

{
  "Effect": "Allow",
  "Action": [
    "iam:Get*",
    "iam:List*",
    "cloudtrail:Describe*",
    "cloudtrail:Get*",
    "logs:Describe*"
  ]
}

❌ Forbidden actions:

{
  "Effect": "Deny",
  "Action": [
    "iam:Create*",
    "iam:Delete*",
    "iam:Update*",
    "iam:Put*",
    "cloudtrail:Create*",
    "cloudtrail:Delete*",
    "logs:Create*",
    "logs:Delete*"
  ]
}

Enforcement:

  • SDK uses only read-only API calls
  • If write permission detected, warning issued
  • Collection continues with read-only operations

Rate Limit Handling

GitHub API rate limiting:

Policy:

  • Check rate limit before each request
  • If remaining < 100, pause collection
  • Wait until rate limit resets
  • Resume collection

Behavior:

Collecting from GitHub...
  → Checking rate limit: 4,856 / 5,000 remaining
  → Fetching organization settings
  → Checking rate limit: 4,855 / 5,000 remaining
  → Fetching repository list
  ...

⚠ Rate limit approaching (98 remaining)
  Pausing collection...
  Rate limit resets at 2026-01-09T14:00:00Z (in 5 minutes)
  Waiting...

✓ Rate limit reset
  Resuming collection...

Post-Collection Validation

Forbidden Content Detection

Sensitive data patterns blocked:

❌ Secrets:

  • AWS access keys (AKIA...)
  • Private keys (-----BEGIN PRIVATE KEY-----)
  • OAuth tokens
  • API keys
  • Passwords

❌ Source code:

  • Repository file contents
  • Commit diffs
  • Pull request diffs

❌ Personal data:

  • Email content (Gmail)
  • Chat messages (Slack, Teams)
  • Calendar events details

Validation:

// SDK scans artifacts for forbidden patterns
const forbiddenPatterns = [
  /AKIA[0-9A-Z]{16}/,  // AWS access key
  /-----BEGIN (PRIVATE|RSA) KEY-----/,  // Private keys
  /ghp_[a-zA-Z0-9]{36}/,  // GitHub token
  /sk_live_[a-zA-Z0-9]+/,  // Stripe key
];

for (const artifact of artifacts) {
  for (const pattern of forbiddenPatterns) {
    if (pattern.test(artifact.content)) {
      throw new Error(`Forbidden content detected in ${artifact.filename}`);
    }
  }
}

If detected:

✗ Collection failed

Error: Forbidden content detected
File: sources/github/commit_abc123.json
Pattern: AWS access key (AKIA...)

This indicates a configuration error or security issue.
evidence SDK never collects secrets or source code.

Bundle Size Validation

Default maximum: 50 MB (configurable)

Check before creating bundle:

Validating bundle size...
  → Calculating total size
  → Total: 35.2 KB
  ✓ Within limit (50 MB)

If exceeds:

✗ Bundle creation failed

Error: Bundle size (75.3 MB) exceeds maximum (50 MB)

Options:
  1. Increase limit: bundle.max_size_mb: 100
  2. Reduce scope: Limit repos or log groups
  3. Disable verbose manifest

Artifact Count Validation

Maximum artifacts: 10,000 per bundle

Purpose: Prevent unbounded resource usage

Check:

Validating artifact count...
  → Total artifacts: 127
  ✓ Within limit (10,000)

If exceeds:

✗ Collection failed

Error: Too many artifacts (12,456 > 10,000)

This usually indicates:
  - Wildcard repos: '*' matched too many repositories
  - Wildcard log groups: '*' matched too many log groups

Recommendation: Be more specific in source configuration

Control Validation

Control Coverage Check

Verify controls can be satisfied:

evidence collect --dry-run

Output:

Testing control coverage...

CC6.1 - Logical Access Controls
  ✓ GitHub: Org 2FA enforcement
  ✓ AWS: IAM password policy
  ✗ Google Workspace: 2SV enforcement (source not configured)

CC6.6 - Access Removal/Modification
  ✓ GitHub: CODEOWNERS file
  ✗ Google Workspace: Admin roles (source not configured)

CC7.2 - Change Management
  ✓ GitHub: Branch protection
  ✓ AWS: CloudTrail status
  ✓ AWS: CloudWatch retention

Control Coverage:
  - CC6.1: Partial (2/3 sources)
  - CC6.6: Partial (1/2 sources)
  - CC7.2: Complete (3/3 sources)

⚠ Warning: Some controls have partial coverage
  Recommendation: Configure google_workspace source for complete coverage

Pass/Fail Criteria

CC6.1 - Logical Access Controls:

GitHub:

{
  "two_factor_requirement_enabled": true  // ✅ Must be true
}

AWS:

{
  "MinimumPasswordLength": 14,  // ✅ Must be >= 12
  "RequireSymbols": true,        // ✅ Must be true
  "RequireNumbers": true,        // ✅ Must be true
  "RequireUppercaseCharacters": true,  // ✅ Must be true
  "RequireLowercaseCharacters": true   // ✅ Must be true
}

Evaluation:

Evaluating CC6.1...
  ✓ GitHub 2FA: enabled
  ✓ AWS password length: 14 >= 12
  ✓ AWS password complexity: all requirements met

✓ CC6.1: Pass

CC7.2 - Change Management:

GitHub:

{
  "required_pull_request_reviews": {
    "required_approving_review_count": 2  // ✅ Must be >= 1
  },
  "required_status_checks": {
    "strict": true,  // ✅ Must be true
    "contexts": ["ci/tests"]  // ✅ Must have at least one
  }
}

AWS:

{
  "IsLogging": true,  // ✅ Must be true
  "LatestDeliveryAttemptSucceeded": "2026-01-09T12:30:00Z"  // ✅ Must be recent
}

Evaluation:

Evaluating CC7.2...
  ✓ GitHub branch protection: 2 required reviewers
  ✓ GitHub status checks: enabled, 1 check required
  ✓ AWS CloudTrail: logging active
  ✓ AWS CloudTrail: last delivery 2 minutes ago

✓ CC7.2: Pass

Custom Policy Rules (Future)

Coming in v0.2:

# evidence.yaml
policy:
  rules:
    - name: require-codeowners
      control: CC6.6
      source: github
      condition: |
        artifacts['codeowners'].exists == true

    - name: minimum-reviewers
      control: CC7.2
      source: github
      condition: |
        artifacts['branch_protection'].required_approving_review_count >= 3

    - name: cloudtrail-encryption
      control: CC7.2
      source: aws
      condition: |
        artifacts['cloudtrail_config'].KmsKeyId != null

Evaluation output:

Evaluating custom policies...
  ✓ require-codeowners: Pass (23/25 repos)
  ✗ minimum-reviewers: Fail (12/25 repos have < 3 reviewers)
  ✓ cloudtrail-encryption: Pass

Custom Policy: Partial Pass (2/3 rules)

Some validations can be bypassed for debugging:

# evidence.yaml (development only!)
policy:
  bypass:
    - scope_validation  # Skip scope checks
    - size_validation   # Skip size limits
    - content_validation  # Skip forbidden content detection

Warning:

⚠ WARNING: Policy bypass enabled

The following validations are disabled:
  - scope_validation
  - size_validation
  - content_validation

This is ONLY for debugging. Never use in production.
Bundles created with policy bypass may not be compliant.

Use cases:

  • Local testing
  • Debugging collection issues
  • Understanding what SDK collects

Never use in production or for compliance purposes.

Audit Logging

All policy decisions are logged:

{
  "timestamp": "2026-01-09T12:35:00Z",
  "event": "policy_evaluation",
  "stage": "pre_collection",
  "validations": [
    {
      "rule": "configuration_schema",
      "result": "pass"
    },
    {
      "rule": "github_token_scopes",
      "result": "pass",
      "scopes": ["repo:read", "read:org"]
    },
    {
      "rule": "signing_key_permissions",
      "result": "pass",
      "permissions": "600"
    }
  ]
}

Log location:

  • Console output (collection time)
  • run.json in bundle (permanent record)
  • Platform audit log (after upload)

Best Practices

1. Run Dry Run Before Production

# Test configuration and coverage
evidence collect --dry-run

# Review output for warnings
# Fix any issues before actual collection

2. Monitor Policy Warnings

Don't ignore warnings:

⚠ Warning: Branch protection not configured for repository: acme/sandbox

Action:

  • Review repository
  • Add branch protection if needed
  • Or exclude from collection if not production

3. Regular Policy Audits

Monthly checklist:

  • Review control coverage
  • Verify all controls passing
  • Check for new policy warnings
  • Update configuration if needed

4. Test After Configuration Changes

After updating evidence.yaml:

# Validate configuration
evidence collect --dry-run

# Test actual collection (small scope first)
evidence collect

# Verify bundle
evidence verify evidence-bundle-*.tar.gz

Troubleshooting

Configuration Validation Failed

Symptom:

✗ Configuration validation failed
  - sources.github.org: required field missing

Solution:

  • Check evidence.yaml syntax
  • Verify all required fields present
  • Run evidence collect --dry-run to test

Forbidden Content Detected

Symptom:

✗ Forbidden content detected
File: sources/github/secrets.json
Pattern: AWS access key

Cause:

  • Configuration error collecting wrong data
  • Bug in SDK connector

Solution:

  • Report to evidence SDK team
  • Do NOT use bundle for compliance
  • Investigate why secrets were collected

Control Coverage Incomplete

Symptom:

⚠ CC6.1: Partial coverage (1/3 sources)

Solution:

  • Add missing sources to configuration
  • Or accept partial coverage (document why)
  • Consult with compliance team

See Also

Signing Keys

Ed25519 key generation, management, rotation, and security best practices for evidence bundles.

Troubleshooting

Common issues, solutions, and debugging techniques for evidence collection.

On this page

OverviewPre-Collection ValidationConfiguration Schema ValidationCredential ValidationSigning Key ValidationDuring Collection ValidationAPI Scope EnforcementPermission VerificationRate Limit HandlingPost-Collection ValidationForbidden Content DetectionBundle Size ValidationArtifact Count ValidationControl ValidationControl Coverage CheckPass/Fail CriteriaCustom Policy Rules (Future)Policy Bypass (Not Recommended)Audit LoggingBest Practices1. Run Dry Run Before Production2. Monitor Policy Warnings3. Regular Policy Audits4. Test After Configuration ChangesTroubleshootingConfiguration Validation FailedForbidden Content DetectedControl Coverage IncompleteSee Also