Credential Management - Best Practices

Last Updated: 2026-02-13

Overview

This document describes the credential management strategy for scandora.net infrastructure, with special focus on bootstrap credentials (1Password service account tokens) that enable access to other secrets.

The Bootstrap Credentials Problem

Challenge: Service account tokens provide programmatic access to 1Password, but storing them securely creates a chicken-and-egg problem:

• Can't store tokens IN 1Password (need token to access 1Password)
• Can't store in plaintext files (security risk)
• Can't hard-code in scripts (violates IaC principles)
• Need to survive reboots and be available on shell startup

Solution: Different storage mechanisms for different environments.

Three-Tier Architecture

Tier 1: User Credentials (Interactive)

Storage: 1Password app Access: Master password + 2FA Use case: Human operators logging in interactively

Tier 2: Service Account Tokens (Programmatic)

Storage: Platform-specific secure storage (see below) Access: OS-level authentication or file permissions Use case: Non-interactive automation scripts

Tier 3: Application Secrets (Managed)

Storage: 1Password vaults (accessed via service accounts) Access: Retrieved programmatically using service account tokens Use case: API keys, credentials, database passwords

Service Account Token Storage by Platform

macOS Workstations (luna)

Storage: macOS Keychain (encrypted, user-bound)

Implementation:

# Store token in keychain
~/src/scandora.net/scripts/1password/keychain-helper.sh store scandora-dev-automation ops_eyJza...

# Retrieve in scripts (automatic via ~/.zshrc)
export OP_SERVICE_ACCOUNT_TOKEN=$(~/src/scandora.net/scripts/1password/keychain-helper.sh get scandora-dev-automation)

# Or manually switch accounts
op-switch-account scandora-prd-automation

Security properties:

• ✅ OS-level AES-256 encryption
• ✅ Tied to user account (requires login)
• ✅ Not visible in files or process listings
• ✅ Audit logging via macOS security framework
• ✅ Can restrict which applications access token

See: scripts/1password/MIGRATION-TO-KEYCHAIN.md for setup

Linux Cloud Instances (pluto, dumbo, bogart)

Storage: Protected files with restrictive permissions

Implementation:

# Store token (root-owned)
sudo mkdir -p /etc/1password
sudo bash -c 'cat > /etc/1password/service-account.token << EOF
ops_eyJza...
EOF'
sudo chmod 600 /etc/1password/service-account.token
sudo chown root:root /etc/1password/service-account.token

# Retrieve in scripts
export OP_SERVICE_ACCOUNT_TOKEN=$(sudo cat /etc/1password/service-account.token)

Alternative: User-specific (no sudo):

# Store token (user-owned)
mkdir -p ~/.config/op
chmod 700 ~/.config/op
cat > ~/.config/op/service-account.token << 'EOF'
ops_eyJza...
EOF
chmod 600 ~/.config/op/service-account.token

# Retrieve in scripts
export OP_SERVICE_ACCOUNT_TOKEN=$(cat ~/.config/op/service-account.token)

Security properties:

• ✅ File system permissions (600/700)
• ✅ Not in git or shell config
• ⚠️ Plaintext file (rely on OS access controls)
• ⚠️ Vulnerable if instance compromised

Risk mitigation:

• Use minimal service accounts (principle of least privilege)
• Rotate tokens quarterly
• Monitor access logs
• Use cloud IAM for additional access control layer

Emergency Backup (All Platforms)

Storage: 1Password vault (scandora.net)

Purpose: Disaster recovery only - DO NOT use for automation

Item format:

Name: "1Password Service Account Token - {account-name} (EMERGENCY BACKUP)"
Category: Password
Vault: scandora.net
Purpose: Emergency recovery only - DO NOT use for automation
Token: ops_eyJza...
Notes: Last rotated: YYYY-MM-DD
       Retrieve with interactive login only

Recovery procedure:

1. Log in to 1Password app (interactive, master password)
2. Find "Emergency Backup" item in scandora.net vault
3. Copy token
4. Restore to platform-specific storage (Keychain or file)
5. Verify with op whoami

Service Account Inventory

scandora-dev-automation

Vault access:

• scandora-automation (shared credentials)
• scandora-dev-automation (dev-specific credentials)

Use cases:

• Development work on luna
• Running Ansible playbooks against dev targets (opnsense-dev)
• Terraform operations in dev environments
• Testing IaC changes

Storage:

• Luna: macOS Keychain (automatic load on shell startup)
• Never deployed to cloud instances

scandora-prd-automation

Vault access:

• scandora-automation (shared credentials)
• scandora-prd-automation (production credentials)

Use cases:

• Production deployments (owl, blue gateways)
• Production Terraform operations
• Critical infrastructure changes

Storage:

• Luna: macOS Keychain (manual load via script)
• Never deployed to cloud instances
• Requires explicit user action to activate

Loading:

# Manual load (creates new subshell with prod token)
source ~/src/scandora.net/scripts/1password/load-prod-token.sh

# Verify
op whoami  # Should show: scandora-prd-automation

scandora-full (legacy)

Vault access:

• scandora.net (cloud infrastructure credentials)

Use cases:

• Cloud instance automation (pluto, dumbo)
• Retrieving AWS/GCP credentials
• Database passwords
• Cloud SQL proxy authentication

Storage:

• Pluto: /etc/op-service-account.token (root-owned, 600)
• Dumbo: /etc/op-service-account.token (root-owned, 600)
• Never stored on luna or untrusted hosts

Security:

• Read-only access to scandora.net vault
• No access to gateway credentials
• Can be rotated without affecting other service accounts

Token Lifecycle Management

Token Rotation Schedule

Frequency: Every 90 days (quarterly)

Procedure:

1. Generate new token in 1Password web console
2. Update emergency backup in scandora.net vault
3. Update luna workstation:

~/src/scandora.net/scripts/1password/keychain-helper.sh store scandora-dev-automation <new-token>
op whoami  # Verify

1. Update cloud instances:

# From luna
TOKEN="<new-token>"
ssh pluto "sudo bash -c 'cat > /etc/1password/service-account.token' <<< '$TOKEN'"
ssh pluto "op whoami"  # Verify

1. Test automation:

# Test Ansible playbook
cd ~/src/scandora.net/cloud/ansible
ansible all -m ping

Token Revocation

When to revoke immediately:

• Token compromised or leaked
• Employee/contractor departure
• Security incident
• Suspected unauthorized access

Procedure:

1. Revoke in 1Password web console (immediate effect)
2. Generate new token
3. Follow rotation procedure above
4. Review audit logs for unauthorized usage

Security Best Practices

Do ✅

• ✅ Use macOS Keychain on workstations (automated encryption)
• ✅ Use minimal vault access (principle of least privilege)
• ✅ Rotate tokens quarterly (90 days)
• ✅ Keep emergency backup in 1Password vault
• ✅ Use different service accounts for dev vs prod
• ✅ Audit access logs regularly
• ✅ Test token retrieval after rotation

Don't ⛔

• ⛔ Store tokens in shell config files (.zshrc, .bashrc)
• ⛔ Commit tokens to git (even in private repos)
• ⛔ Share tokens between people
• ⛔ Use prod tokens for dev work
• ⛔ Store tokens in environment variables long-term
• ⛔ Hard-code tokens in scripts
• ⛔ Log tokens in application logs

Defense in Depth

Layer 1: Storage encryption

• macOS Keychain: OS-level AES-256
• Linux files: File system permissions (600/700)

Layer 2: Access control

• macOS: User authentication required
• Linux: sudo or user ownership
• 1Password: Service account permissions

Layer 3: Network isolation

• ZeroTier for cloud instance access
• SSH key authentication only
• fail2ban for SSH protection

Layer 4: Monitoring

• macOS: Keychain access logs (Console.app)
• 1Password: Audit logs in web console
• Cloud: Instance access logs (CloudTrail, Cloud Audit Logs)

Layer 5: Token rotation

• 90-day rotation schedule
• Immediate revocation if compromised
• Testing after rotation

Troubleshooting

Token Not Working

Symptoms:

op whoami
# Error: [ERROR] 401: Invalid token

Diagnosis:

1. Verify token format (should start with "ops_")
2. Check token hasn't expired or been revoked
3. Verify network connectivity to 1Password API

Fix:

# Get fresh token from 1Password console
# Update storage (Keychain or file)
~/src/scandora.net/scripts/1password/keychain-helper.sh store account-name <new-token>

# Verify
op whoami

Keychain Access Denied (macOS)

Symptoms:

• macOS repeatedly asking for password
• "The user name or passphrase you entered is not correct"

Fix:

# Delete and re-add
~/src/scandora.net/scripts/1password/keychain-helper.sh delete account-name
~/src/scandora.net/scripts/1password/keychain-helper.sh store account-name <token>

# When prompted, choose "Always Allow"

Token Loaded But Scripts Fail

Symptoms:

echo $OP_SERVICE_ACCOUNT_TOKEN  # Shows token
op whoami  # Works

# But Ansible/Terraform fails with auth errors

Diagnosis:

# Check which service account is loaded
op whoami

# Check required vault access
op vault list

Fix:

# Switch to correct service account
op-switch-account scandora-prd-automation

# Or explicitly set token
export OP_SERVICE_ACCOUNT_TOKEN=$(~/src/scandora.net/scripts/1password/keychain-helper.sh get scandora-prd-automation)

Related Documentation

• Migration guide: scripts/1password/MIGRATION-TO-KEYCHAIN.md
• Helper script: scripts/1password/keychain-helper.sh
• Load prod token: scripts/1password/load-prod-token.sh
• CLAUDE.md: Service account section
• 1Password CLI docs: https://developer.1password.com/docs/cli/

Appendix: Security Comparison

Approach	Encryption	Access Control	Visibility	Audit Log	Risk Level
macOS Keychain	✅ AES-256	✅ User auth	✅ Hidden	✅ Yes	🟢 Low
Linux protected file	⚠️ None	⚠️ File perms	⚠️ Root visible	❌ No	🟡 Medium
Shell config	❌ Plaintext	❌ File perms	❌ Visible	❌ No	🔴 High
Environment variable	❌ Plaintext	❌ Process	❌ Visible	❌ No	🔴 High
Git commit	❌ Plaintext	❌ Anyone	❌ Permanent	❌ No	🔴 CRITICAL

Recommendation: Use macOS Keychain on workstations, protected files on cloud instances, emergency backup in 1Password vault.

---

Document owner: Infrastructure team Review schedule: Quarterly (with token rotation) Last reviewed: 2026-02-13