Skip to content

Git Commit Validation Pipeline

Purpose: Enforce code quality, security, and consistency across all commits to the scandora.net infrastructure repository.

Philosophy: Prevent issues at commit time, not in production. Every commit should be production-ready.

Table of Contents

Overview

The validation pipeline runs automatically on every git commit via pre-commit hooks. It includes 9 tiers of checks:

Tier Category Tools Purpose
1 Security gitleaks, trufflehog Detect hardcoded secrets
2 File Hygiene pre-commit-hooks Whitespace, EOF, merge conflicts
3 Ansible ansible-lint Infrastructure validation
4 Terraform terraform fmt/validate, tfsec IaC formatting + security
5 Docker hadolint Dockerfile best practices
6 Shell shellcheck Bash/sh script linting
7 YAML yamllint YAML syntax (non-Ansible)
8 Python black Python formatting
9 Commit Messages conventional-pre-commit Enforce commit format

Estimated run time: 30-60s (first run: 5-10min for tool installation)

Installation

Prerequisites

# Install pre-commit
brew install pre-commit

# Install additional tools (optional, for local runs)
brew install gitleaks tfsec hadolint shellcheck shfmt

Setup

cd ~/src/scandora.net

# Install git hooks
pre-commit install              # For commit-time validation
pre-commit install --hook-type commit-msg  # For commit message validation

# Test installation
pre-commit run --all-files

✓ Done! Hooks will now run automatically on every commit.

Validation Tiers

TIER 1: Security (MUST-PASS)

Gitleaks - Primary secrets detection

  • Scans for: API keys, passwords, tokens, private keys
  • Uses: Pattern matching (fast, low false-positives)
  • Whitelist: .gitleaksignore

TruffleHog - Deep secret scanning

  • Scans for: High-entropy strings, unusual patterns
  • Uses: Entropy analysis (slower, catches obfuscated secrets)
  • May produce false positives (review carefully)

⚠️ CRITICAL: If secrets are detected, DO NOT COMMIT. Follow the remediation steps in Handling Failures.

TIER 2: File Hygiene

Automatic fixes for common issues:

  • Trailing whitespace removed
  • Final newline added to files
  • Line endings normalized to LF (Unix)
  • Large files blocked (>1MB)
  • Merge conflict markers detected

TIER 3: Ansible (Infrastructure)

ansible-lint with production profile:

  • Enforces task names, tags, handlers
  • Checks for deprecated modules
  • Validates YAML syntax
  • Auto-fixes with --write flag

Common failures:

  • Missing task names
  • Deprecated when syntax
  • Untagged tasks

Fix: Run ansible-lint --write cloud/ansible/ to auto-fix most issues.

TIER 4: Terraform (Security + Best Practices)

terraform fmt - Formatting

  • Enforces canonical formatting
  • Auto-fixes on commit

terraform validate - Syntax checking

  • Validates HCL syntax
  • Checks module references
  • Note: Requires terraform init in each environment

tfsec - Security scanning

  • Detects: Unencrypted resources, overly permissive security groups, exposed secrets
  • Severity: HIGH+ only
  • See: .tflint.hcl for configuration

Manual: tflint - Best practices

  • Run: tflint --config=.tflint.hcl --recursive
  • Not in pre-commit (requires plugins)

TIER 5: Docker

hadolint - Dockerfile linting

  • Best practices: USER directive, layer optimization
  • Security: Don't run as root, use COPY > ADD
  • Ignored rules: DL3008, DL3018 (version pinning too strict)

TIER 6: Shell Scripts

shellcheck - Bash/sh linting

  • Detects: Quoting issues, deprecated syntax, security issues
  • Assumes: bash (not POSIX sh)
  • Exclusions: Terraform heredocs (confuse shellcheck)

Manual: shfmt - Shell formatting

  • Run: shfmt -i 2 -ci -bn -sr -w scripts/**/*.sh
  • Not in pre-commit (no hook available)

TIER 7: YAML (Non-Ansible)

yamllint - YAML syntax

  • Line length: 120 chars max
  • Comments: Minimum 1 space from content
  • Excludes: Ansible files (use ansible-lint)

TIER 8: Python

black - Python formatting

  • Opinionated formatter (no configuration)
  • Enforces: Double quotes, consistent spacing

TIER 9: Commit Messages

conventional-pre-commit - Commit format

  • Enforces: type(scope): subject format
  • Types: feat|fix|docs|refactor|test|chore|style|perf|ci|build|revert

Examples:

feat(ddns): add 1Password integration for credentials
fix(ansible): correct OPNsense API auth flow
docs(security): document secrets audit findings
refactor(terraform): consolidate static IP configs

Daily Workflow

Normal Commits

# Make changes
vim scripts/ddns/cf-ddns.sh

# Stage changes
git add scripts/ddns/cf-ddns.sh

# Commit (hooks run automatically)
git commit -m "feat(ddns): migrate to 1Password credential retrieval"

# If hooks pass:
✅ All checks passed! Commit created.

# If hooks fail:
❌ Some checks failed. Fix issues and re-commit.

Manual Validation (Before Commit)

# Run all hooks on staged files
pre-commit run

# Run all hooks on entire codebase
pre-commit run --all-files

# Run specific hook
pre-commit run gitleaks
pre-commit run ansible-lint

# Show what would change (dry-run)
pre-commit run --all-files --show-diff-on-failure

Emergency Bypass (Use Sparingly!)

# Skip hooks for WIP commits
git commit --no-verify -m "wip: debugging network issue"

# ⚠️ WARNING: Bypassed commits still need validation before push!
# Run before pushing:
pre-commit run --all-files

When to bypass:

  • Urgent production fixes (will validate before push)
  • Local WIP commits (not pushed to remote)
  • Debugging (temporary state)

NEVER bypass for:

  • Commits pushed to main/master
  • Pull requests
  • Shared branches

Handling Failures

Secrets Detected

❌ STOP! DO NOT COMMIT!

# Example output:
🔐 Gitleaks - Detect hardcoded secrets..................Failed
Finding: scripts/ddns/owl.conf:18
Secret: CF_API_KEY="b9850b275095a87122890c185b1e642f38323"

Steps:

  1. Remove the secret from the file
  2. Add to .gitleaksignore if it's a false positive (with justification)
  3. Rotate the credential if it was committed previously
  4. Check git history for previous leaks:
gitleaks detect --report-path=/tmp/audit.json

Auto-Fixable Failures

Many tools can auto-fix issues:

# Terraform formatting
terraform fmt -recursive cloud/terraform/

# Ansible linting
ansible-lint --write cloud/ansible/

# Python formatting
black cloud/ansible/**/*.py scripts/**/*.py

# Shell formatting (manual)
shfmt -i 2 -ci -bn -sr -w scripts/**/*.sh

# Re-run hooks
pre-commit run --all-files

Manual Fixes Required

ShellCheck failures:

# View specific errors
shellcheck scripts/ddns/cf-ddns.sh

# Common issues:
# - SC2086: Quote variables to prevent word splitting
# - SC2154: Variable referenced but not assigned
# - SC2155: Declare and assign separately

Ansible-lint failures:

# View details
ansible-lint cloud/ansible/playbooks/site.yml

# Common issues:
# - Add task names: `name: "Install package"`
# - Add tags: `tags: [setup]`
# - Use FQCN: `ansible.builtin.copy` instead of `copy`

False Positives

Whitelisting Secrets (Gitleaks)

Edit .gitleaksignore:

# Get fingerprint from report
jq -r '.[] | select(.File == "path/to/file") | .Fingerprint' /tmp/gitleaks.json

# Add to .gitleaksignore with explanation
# WireGuard public key (not a secret)
*:docs/vpn/wireguard.md:generic-api-key:*cmOApvAKmLEHmS93O8NrfnmNPUKhokNb*

Acceptable whitelists:

  • Example credentials in documentation
  • Public keys (SSH, WireGuard, certificates)
  • Encrypted data (OPNsense config.xml with base64-encoded certs)
  • Deployment placeholders (__API_KEY__)

NEVER whitelist:

  • Real API keys or passwords
  • Private keys
  • Tokens with actual permissions

Ignoring Specific Rules

Inline suppression:

# Ansible
- name: Legacy command (no module exists)
  command: /usr/local/bin/custom-tool
  changed_when: false
  # noqa: command-instead-of-module

# YAML
# yamllint disable-line rule:line-length
long_key: "very long value that exceeds 120 characters but is unavoidable"

File-level suppression:

# At top of file
# ansible-lint disable=fqcn

Performance

First Run (Cold Cache)

# Expected: 5-10 minutes
# Downloads and installs all validation tools
pre-commit run --all-files

Subsequent Runs (Warm Cache)

# Expected: 30-60 seconds
# Only validates changed files on commit
git commit -m "..."

# Full codebase scan
# Expected: 2-5 minutes
pre-commit run --all-files

Optimization Tips

  1. Run specific hooks for faster feedback:
pre-commit run gitleaks  # Secrets only (5s)
pre-commit run shellcheck  # Shell scripts only (10s)
  1. Use --files flag for targeted checks:
pre-commit run --files scripts/ddns/*.conf
  1. Skip slow hooks during iteration:
SKIP=trufflehog git commit -m "..."
  1. Update cache weekly:
pre-commit autoupdate
pre-commit clean  # Clear old environments

Troubleshooting

"Hook failed to install"

# Clear cache and reinstall
pre-commit clean
pre-commit install --install-hooks

"Command not found: gitleaks"

Pre-commit will install tools automatically. If manual installation needed:

brew install gitleaks tfsec hadolint shellcheck

"Terraform validate failed"

Terraform requires terraform init in each environment:

cd cloud/terraform/environments/production/aws
terraform init

Or skip terraform validation during local development:

SKIP=terraform_validate git commit -m "..."

"Ansible-lint too strict"

Adjust .ansible-lint profile or skip rules:

# .ansible-lint
profile: moderate  # Instead of production

skip_list:
  - 'fqcn'
  - 'role-name'

"Hooks running slow"

Check for network issues (downloading tools) or disable slow hooks:

# Check what's slow
pre-commit run --all-files --verbose

# Disable TruffleHog (slowest)
# Edit .pre-commit-config.yaml and remove trufflehog entry

CI/CD Integration

GitHub Actions (Future)

# .github/workflows/validate.yml
name: Validation
on: [push, pull_request]

jobs:
  pre-commit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
      - uses: pre-commit/action@v3.0.1

Pre-Push Hook (Optional)

Run validation before pushing to remote:

# Install pre-push hook
pre-commit install --hook-type pre-push

# Configure to run all hooks
# Edit .git/hooks/pre-push and add:
#!/bin/bash
pre-commit run --all-files

Summary

Key Commands:

pre-commit run                    # Validate staged files
pre-commit run --all-files        # Validate entire codebase
pre-commit autoupdate             # Update hook versions
git commit --no-verify            # Emergency bypass (use sparingly)

Best Practices:

  1. ✅ Never commit secrets - Use 1Password retrieval patterns
  2. ✅ Fix issues immediately - Don't accumulate validation debt
  3. ✅ Whitelist responsibly - Document why false positives are safe
  4. ✅ Test before committing - Run pre-commit run proactively
  5. ✅ Keep hooks updated - Run pre-commit autoupdate weekly

Support:

  • Report issues: https://github.com/scandora/scandora.net/issues
  • Hook documentation: /Users/joe/src/scandora.net/.pre-commit-config.yaml
  • Whitelist guidance: /Users/joe/src/scandora.net/.gitleaksignore