ZeroTier Overlay Network¶

Overview¶

ZeroTier provides the site-to-site VPN connectivity, creating a flat Layer 2 network that spans all physical and cloud locations.

Network Details¶

Attribute	Value
Network ID	`6ab565387a4b9177`
Subnet	192.168.194.0/24
Members	Gateways + Cloud instances
Management	ZeroTier Central

Member Nodes¶

Host	ZeroTier IP	Type	Notes
owl	192.168.194.10	Gateway	Iowa site
blue	192.168.194.x	Gateway	Colorado site
pluto	192.168.194.x	Cloud	AWS
dumbo	192.168.194.x	Cloud	GCE
bogart	192.168.194.x	Cloud	PowerDNS
luna	192.168.194.x	Workstation	Mac (current location)

Why ZeroTier?¶

NAT Traversal: Works through CGNAT (Starlink) without port forwarding
Peer-to-Peer: Direct connections when possible, relayed when not
Encrypted: All traffic encrypted end-to-end
Self-Healing: Automatic failover and reconnection
IPv6 Transport: Can use IPv6 for reliable CGNAT bypass

Gateway Configuration¶

Both OPNsense gateways run the ZeroTier plugin (os-zerotier).

Interface Configuration¶

ZeroTier creates a virtual interface (ztXXXXXXXXXX) that's configured in OPNsense:

<!-- From config.xml -->
<zerotier>
  <enabled>1</enabled>
  <networks>
    <network uuid="...">
      <enabled>1</enabled>
      <networkId>6ab565387a4b9177</networkId>
      <description>scandora.net</description>
    </network>
  </networks>
</zerotier>

Firewall Rules¶

Each gateway has rules allowing traffic between ZeroTier and LAN:

ZT → LAN: Allow ZeroTier interface to access local LAN
LAN → ZT: Allow local devices to reach other sites

Cloud Instance Configuration¶

Cloud instances run the ZeroTier daemon directly:

# Installation (Ubuntu)
curl -s https://install.zerotier.com | sudo bash

# Join network
sudo zerotier-cli join 6ab565387a4b9177

# Check status
zerotier-cli listnetworks

Ansible Role¶

The zerotier Ansible role handles installation and network joining:

# roles/zerotier/tasks/main.yml
- name: Install ZeroTier
  shell: curl -s https://install.zerotier.com | bash
  args:
    creates: /usr/sbin/zerotier-one

- name: Join network
  command: zerotier-cli join {{ zerotier_network_id }}

Troubleshooting¶

Check Network Status¶

# List joined networks
zerotier-cli listnetworks

# Expected output:
# 200 listnetworks <nwid> <name> <mac> <status> <type> <dev> <ZT assigned ips>
# 6ab565387a4b9177 scandora.net XX:XX:XX:XX:XX:XX OK PRIVATE ztXXXXXXXX 192.168.194.X/24

Check Peer Connections¶

# List peers and connection quality
zerotier-cli listpeers

# Look for DIRECT vs RELAY status
# DIRECT = peer-to-peer connection established
# RELAY = traffic going through ZeroTier infrastructure

Common Issues¶

Node Not Authorized¶

If status shows ACCESS_DENIED:

Go to ZeroTier Central
Find the network (6ab565387a4b9177)
Authorize the new member

No Direct Connection¶

If peers show RELAY instead of DIRECT:

Verify UDP port 9993 is open on both ends
Check for strict NAT/firewall blocking UDP
Enable IPv6 transport if available (helps with CGNAT)

# Force IPv6 transport preference
zerotier-cli set 6ab565387a4b9177 allowGlobal=true

Service Management¶

OPNsenseLinux (systemd)

# Restart ZeroTier service
configctl zerotier restart

# View logs
clog -f /var/log/zerotier.log

# Restart service
sudo systemctl restart zerotier-one

# View logs
sudo journalctl -u zerotier-one -f

Security Considerations¶

Network is private: All members must be explicitly authorized
Encryption: Traffic encrypted with Curve25519/Salsa20
API Access: ZeroTier Central API credentials in 1Password
Firewall: Gateway rules control what ZeroTier traffic can reach LAN