Documentation
AdvancedTroubleshooting

Troubleshooting

Common issues, solutions, and debugging techniques for evidence collection.

Quick Diagnostics

Check SDK Version

evidence --version

Expected: 0.1.0 or higher

If outdated:

npm install -g @evidence-oss/cli@latest

Verify Configuration

# Test configuration without collecting
evidence collect --dry-run

Checks:

  • Configuration file syntax
  • Required fields present
  • Credentials accessible
  • Control coverage

Enable Verbose Logging

# More detailed output
evidence collect --verbose

Shows:

  • API requests and responses
  • File operations
  • Policy evaluations
  • Detailed error messages

Installation Issues

Command Not Found

Symptom:

evidence --version
# zsh: command not found: evidence

Causes:

  • evidence CLI not installed
  • Not in PATH

Solutions:

  1. Install evidence CLI:

    npm install -g @evidence-oss/cli

  2. Verify installation:

    which evidence
# Should show: /usr/local/bin/evidence or similar

  3. Check PATH:

    echo $PATH
# Should include npm global bin directory

  4. Restart shell:

    exec $SHELL

NPM Permission Errors

Symptom:

npm install -g @evidence-oss/cli
# EACCES: permission denied

Solutions:

Option 1: Use npx (recommended for testing):

npx @evidence-oss/cli collect

Option 2: Fix npm permissions:

# Change npm global directory
mkdir ~/.npm-global
npm config set prefix '~/.npm-global'

# Add to PATH
echo 'export PATH=~/.npm-global/bin:$PATH' >> ~/.zshrc
source ~/.zshrc

# Install
npm install -g @evidence-oss/cli

Option 3: Use sudo (not recommended):

sudo npm install -g @evidence-oss/cli

Configuration Issues

YAML Syntax Error

Symptom:

✗ Configuration validation failed

Error: YAML parse error at line 12:
  unexpected token 'indent'

Causes:

  • Incorrect indentation
  • Missing colons
  • Invalid YAML syntax

Solutions:

  1. Use 2-space indentation (not tabs):

    sources:
  github:  # 2 spaces
    mode: token  # 4 spaces

  2. Validate YAML online:

    • Copy configuration to YAML Lint
    • Fix reported errors

  3. Use editor with YAML support:

    • VS Code with YAML extension
    • Shows syntax errors in real-time

Missing Required Fields

Symptom:

✗ Configuration validation failed
  - sources.github.org: required field missing
  - sources.github.repos: required field missing

Solution:

Add all required fields:

sources:
  github:
    mode: token
    token_env: GITHUB_TOKEN
    org: your-org  # Required
    repos:  # Required
      - your-org/your-repo

Check field requirements in:

Environment Variable Not Set

Symptom:

✗ Collection failed

Error: Environment variable 'GITHUB_TOKEN' not set

Solutions:

  1. Set environment variable:

    export GITHUB_TOKEN=ghp_your_token_here

  2. Verify it's set:

    echo $GITHUB_TOKEN

  3. Add to shell profile (permanent):

    echo 'export GITHUB_TOKEN=ghp_...' >> ~/.zshrc
source ~/.zshrc

  4. Use .env file (development):

    # .env
GITHUB_TOKEN=ghp_...
AWS_ACCESS_KEY_ID=AKIA...
    # Load before collection
source .env
evidence collect

GitHub Connector Issues

Authentication Failed

Symptom:

✗ GitHub connector failed

Error: Bad credentials (HTTP 401)

Solutions:

  1. Verify token is set:

    echo $GITHUB_TOKEN

  2. Test token manually:

    curl -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/user

  3. Create new token:

    • GitHub → Settings → Developer settings → Personal access tokens
    • Generate new token with required scopes
    • Update environment variable

Insufficient Permissions

Symptom:

✗ GitHub connector failed

Error: Resource not accessible by personal access token (HTTP 403)

Causes:

  • Token missing required scopes
  • Token doesn't have access to private repos
  • SSO not authorized

Solutions:

  1. Check required scopes:

    • Private repos: repo scope
    • Public repos: public_repo scope
    • Organization: read:org scope

  2. Authorize SSO (if applicable):

    • GitHub → Settings → Personal access tokens
    • Find token
    • Click "Configure SSO"
    • Authorize organization

  3. Verify organization access:

    curl -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/orgs/your-org

Rate Limit Exceeded

Symptom:

✗ GitHub connector failed

Error: API rate limit exceeded (HTTP 403)
Rate limit: 0 / 5,000 remaining
Resets at: 2026-01-09T14:00:00Z

Solutions:

  1. Wait for rate limit reset:

    • Check reset time in error message
    • Wait up to 1 hour

  2. Enable rate limit pausing:

    sources:
  github:
    rate_limit_pause: true  # SDK will wait automatically

  3. Use dedicated token:

    • Don't share token across multiple systems
    • Create separate token for evidence collection

  4. Reduce collection frequency:

    • Collect once per day instead of per commit
    • Use scheduled GitHub Actions

Repository Not Found

Symptom:

✗ GitHub connector failed

Error: Not Found (HTTP 404)
Repository 'acme/backend' not found or not accessible.

Solutions:

  1. Verify repository name:

    repos:
  - acme/backend  # Format: org/repo

  2. Check repository exists:

    curl -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/repos/acme/backend

  3. Verify token has access:

    • For private repos: repo scope required
    • User must be member or collaborator

AWS Connector Issues

Access Denied

Symptom:

✗ AWS connector failed

Error: User is not authorized to perform: iam:GetAccountPasswordPolicy (HTTP 403)

Solutions:

  1. Verify credentials are set:

    echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY

  2. Test credentials:

    aws sts get-caller-identity

  3. Check IAM permissions:

    aws iam get-user-policy \
  --user-name evidence-collector \
  --policy-name EvidenceCollection

  4. Attach required policy:

    {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "iam:GetAccountPasswordPolicy",
        "cloudtrail:DescribeTrails",
        "cloudtrail:GetTrailStatus",
        "logs:DescribeLogGroups"
      ],
      "Resource": "*"
    }
  ]
}

Invalid Credentials

Symptom:

✗ AWS connector failed

Error: The security token included in the request is invalid (HTTP 403)

Solutions:

  1. Verify credentials:

    aws sts get-caller-identity

  2. Rotate credentials:

    • Create new access key in IAM console
    • Update environment variables
    • Delete old access key

  3. Check for expired temporary credentials:

    • If using assume role, session may have expired
    • Re-authenticate and retry

CloudTrail Not Found

Symptom:

✗ AWS connector failed

Error: Trail 'production-trail' not found in region us-east-1

Solutions:

  1. List all trails:

    aws cloudtrail describe-trails

  2. Check trail name:

    cloudtrail:
  trails:
    - production-audit-trail  # Exact name

  3. Check region:

    • CloudTrail trails are regional
    • Multi-region trails visible in home region only

Assume Role Failed

Symptom:

✗ AWS connector failed

Error: User is not authorized to perform: sts:AssumeRole

Solutions:

  1. Verify role ARN:

    sources:
  aws:
    role_arn: arn:aws:iam::123456789012:role/evidence-collector

  2. Check trust policy:

    aws iam get-role --role-name evidence-collector \
  --query 'Role.AssumeRolePolicyDocument'

  3. Verify external ID:

    sources:
  aws:
    external_id: evidence-sdk  # Must match trust policy

Google Workspace Issues

Authentication Failed

Symptom:

✗ Google Workspace connector failed

Error: Invalid service account credentials (HTTP 401)

Solutions:

  1. Verify credentials file exists:

    ls -l $GOOGLE_APPLICATION_CREDENTIALS

  2. Check file format:

    cat $GOOGLE_APPLICATION_CREDENTIALS | jq .type
# Should output: "service_account"

  3. Test credentials:

    gcloud auth activate-service-account \
  --key-file=$GOOGLE_APPLICATION_CREDENTIALS

Domain-Wide Delegation Not Authorized

Symptom:

✗ Google Workspace connector failed

Error: Not Authorized to access this resource/api (HTTP 403)

Solutions:

  1. Get Client ID:

    cat $GOOGLE_APPLICATION_CREDENTIALS | jq -r .client_id

  2. Verify delegation in Admin Console:

    • Security → API controls → Domain-wide Delegation
    • Search for client ID
    • Verify scopes are correct

  3. Required scopes:

    https://www.googleapis.com/auth/admin.directory.user.readonly
https://www.googleapis.com/auth/admin.directory.rolemanagement.readonly
https://www.googleapis.com/auth/admin.reports.audit.readonly

  4. Re-authorize if needed:

    • Edit delegation entry
    • Update scopes
    • Click "Authorize"
    • Wait 5-10 minutes for propagation

Invalid Customer ID

Symptom:

✗ Google Workspace connector failed

Error: Invalid customer ID 'C0xxxxxxx'

Solutions:

  1. Find correct customer ID:

    • Admin Console → Account → Account settings
    • Copy Customer ID (format: C01234567)

  2. Verify format:

    • Must start with C
    • Followed by 8 digits
    • Total 9 characters

Bundle Creation Issues

Private Key Not Found

Symptom:

✗ Bundle signing failed

Error: ENOENT: no such file or directory, open '~/.evidence/keys/private.pem'

Solutions:

  1. Generate key pair:

    evidence init --generate-keys

  2. Verify file exists:

    ls -l ~/.evidence/keys/private.pem

  3. Check configuration path:

    bundle:
  signing:
    private_key_path: ~/.evidence/keys/private.pem

Permission Denied (Private Key)

Symptom:

✗ Bundle signing failed

Error: EACCES: permission denied, open '/etc/evidence/keys/private.pem'

Solutions:

  1. Fix permissions:

    sudo chmod 600 /etc/evidence/keys/private.pem
sudo chown $USER:$USER /etc/evidence/keys/private.pem

  2. Verify:

    ls -l /etc/evidence/keys/private.pem
# Should show: -rw------- 1 user user ...

Bundle Too Large

Symptom:

✗ Bundle creation failed

Error: Bundle size (75.3 MB) exceeds maximum (50 MB)

Solutions:

  1. Increase size limit:

    bundle:
  max_size_mb: 100

  2. Reduce scope:

    sources:
  github:
    repos:
      - acme/backend  # Instead of '*'
  aws:
    log_groups:
      - /aws/lambda/api  # Instead of '*'

  3. Disable verbose manifest:

    bundle:
  verbose_manifest: false

Verification Issues

Checksum Mismatch

Symptom:

✗ Checksums verified (11/12 files)
  ✗ sources/github/org_settings.json: FAILED

Causes:

  • Bundle tampered with
  • Bundle corrupted
  • Incomplete extraction

Solutions:

  1. Re-download bundle

  2. Verify extraction:

    tar -xzf bundle.tar.gz -C /tmp/test
cd /tmp/test
sha256sum -c checksums.sha256

  3. Do NOT use tampered bundle for compliance

Signature Verification Failed

Symptom:

✗ Signature verification failed

Signature does not match checksums file.

Causes:

  • Wrong public key
  • Bundle tampered with
  • Bundle created with different private key

Solutions:

  1. Verify public key:

    tar -xzf bundle.tar.gz
cat manifest.json | jq '.signer.key_id'
# Compare with your key inventory

  2. Use correct public key:

    evidence verify bundle.tar.gz \
  --public-key public-2026-01.pem

  3. Do NOT use bundle if signature fails

Upload Issues

Authentication Failed

Symptom:

✗ Upload failed

Error: Authentication failed (HTTP 401)
Invalid or expired API key.

Solutions:

  1. Verify API key is set:

    echo $EVIDENCE_API_KEY

  2. Check API key format:

    • Must start with evd_api_
    • Should be 32+ characters

  3. Create new API key:

    • Platform → Settings → API Keys
    • Create with bundles:upload permission
    • Update environment variable

Upload Timeout

Symptom:

✗ Upload failed

Error: Request timeout after 300s

Solutions:

  1. Increase timeout:

    upload:
  timeout_seconds: 600

  2. Check network connectivity:

    curl -I https://api.evidence-platform.com

  3. Reduce bundle size

Rate Limit Exceeded

Symptom:

✗ Upload failed

Error: Rate limit exceeded (HTTP 429)
Try again in 60 seconds.

Solutions:

  1. Wait and retry:

    sleep 60
evidence upload bundle.tar.gz

  2. Reduce collection frequency:

    • Upload daily instead of hourly
    • Use scheduled uploads

Debugging Techniques

Enable Verbose Mode

evidence collect --verbose

Shows:

  • Configuration parsing
  • Credential validation
  • API requests/responses
  • Policy evaluations
  • File operations

Use Dry Run Mode

evidence collect --dry-run

Tests without collecting:

  • Configuration validation
  • Credential authentication
  • Control coverage
  • Permission checks

Inspect Bundle Contents

# Extract bundle
tar -xzf evidence-bundle-*.tar.gz -C /tmp/inspect

# View manifest
cat /tmp/inspect/manifest.json | jq

# View run metadata
cat /tmp/inspect/run.json | jq

# Check artifacts
ls -lR /tmp/inspect/sources/

Test API Credentials Directly

GitHub:

curl -H "Authorization: token $GITHUB_TOKEN" \
  https://api.github.com/user

AWS:

aws sts get-caller-identity
aws iam get-account-password-policy

Google Workspace:

gcloud auth activate-service-account \
  --key-file=$GOOGLE_APPLICATION_CREDENTIALS
gcloud auth print-access-token

Getting Help

Check Documentation

  1. Quick Start: Installation
  2. Guides: How It Works
  3. Reference: CLI Commands
  4. Advanced: Credential Management

Community Support

GitHub Discussions:

  • Ask questions
  • Share configurations
  • Report issues

GitHub Issues:

  • Bug reports
  • Feature requests

Error Reporting

When reporting issues, include:

  1. evidence CLI version:

    evidence --version

  2. Operating system:

    uname -a

  3. Configuration (redacted):

    # Remove sensitive data before sharing
framework: soc2_type1
sources:
  github:
    org: [REDACTED]

  4. Error output:

    Full error message and stack trace

  5. Steps to reproduce

Common Error Messages

ErrorLikely CauseSolution
Command not found: evidenceCLI not installednpm install -g @evidence-oss/cli
YAML parse errorInvalid YAML syntaxCheck indentation, validate YAML
Environment variable not setMissing credentialsexport VAR=value
Bad credentials (401)Invalid token/keyRegenerate credentials
Not Authorized (403)Insufficient permissionsAdd required scopes/permissions
Not Found (404)Resource doesn't existVerify resource name
Rate limit exceeded (429)Too many requestsWait for reset, enable pausing
Bundle too largeExceeds size limitIncrease limit or reduce scope
Checksum mismatchBundle tampered/corruptedDo not use bundle
Signature failedWrong key or tamperingVerify key, do not use if tampered

See Also

Policy Evaluation

Built-in policy validation rules and compliance checking in evidence collection.

Minimal GitHub Example

Simplest possible configuration to collect evidence from a single GitHub repository.

On this page

Quick DiagnosticsCheck SDK VersionVerify ConfigurationEnable Verbose LoggingInstallation IssuesCommand Not FoundNPM Permission ErrorsConfiguration IssuesYAML Syntax ErrorMissing Required FieldsEnvironment Variable Not SetGitHub Connector IssuesAuthentication FailedInsufficient PermissionsRate Limit ExceededRepository Not FoundAWS Connector IssuesAccess DeniedInvalid CredentialsCloudTrail Not FoundAssume Role FailedGoogle Workspace IssuesAuthentication FailedDomain-Wide Delegation Not AuthorizedInvalid Customer IDBundle Creation IssuesPrivate Key Not FoundPermission Denied (Private Key)Bundle Too LargeVerification IssuesChecksum MismatchSignature Verification FailedUpload IssuesAuthentication FailedUpload TimeoutRate Limit ExceededDebugging TechniquesEnable Verbose ModeUse Dry Run ModeInspect Bundle ContentsTest API Credentials DirectlyGetting HelpCheck DocumentationCommunity SupportError ReportingCommon Error MessagesSee Also