Common issues¶

Symptoms you might hit when deploying or operating Waveshift, with the first thing to check for each. For everything not covered below, capture the failing command's output and contact Blackfire.

Quick triage — "I can't reach the Console"¶

The single most common support call. Walk through it before diving into the detailed sections below — most cases are caught in the first two branches.

flowchart TD
    Start([Browser can't load<br/>https://console.&lt;domain&gt;]) --> Net{On the WireGuard<br/>network?}
    Net -->|no| Connect[Connect via Hub, Node, or<br/>direct WG config from an<br/>Infrastructure Server]
    Net -->|yes| DNS{Does DNS resolve?<br/>dig console.&lt;domain&gt;}
    DNS -->|no| DNSFix[Hub / WG client must use<br/>the private Route 53 zone<br/>resolver — see Hub DNS]
    DNS -->|yes| TCP{TCP 443 reachable?}
    TCP -->|no| Health[Check orchestrator EC2 +<br/>security groups via<br/>wsctl → Status & Logs]
    TCP -->|yes| Cert{Browser cert<br/>warning?}
    Cert -->|yes| Trust[Install Waveshift Root CA<br/>via wsctl → Certificates]
    Cert -->|no — login fails| Creds[Default is admin / admin123<br/>then change immediately]
    Cert -->|no — page loads| Done([Console is reachable])

Bypass the WireGuard tunnel as a sanity check

Even if the user has WireGuard set up, run wsctl → Proxy from your laptop in parallel. The proxies tunnel over SSM straight to the EC2 host, so if the proxy works and the WireGuard path doesn't, the deployment is healthy and the problem is somewhere in the Hub / WireGuard layer.

The leaves of the tree map to specific TUI screens or sections on this page:

wsctl Proxy — fastest sanity check that bypasses the WireGuard tunnel entirely; see Service Proxies.
wsctl Status & Logs — orchestrator health and saga output; see Status & Logs.
wsctl Certificates — install the Waveshift Root CA on laptops + mobile; see Certificates.

AWS credentials and permissions¶

UnauthorizedOperation / AccessDenied during deploy

The IAM principal that ran wsctl doesn't have the permissions needed to create the resource named in the error.

First check:

aws sts get-caller-identity --profile <your-profile>

Confirm the account ID and role match what wsctl config init expects. Bootstrap users typically need EC2, IAM, Route 53, S3, SSM, and VPC full access for the initial deploy; after the orchestrator is up the day-to-day permissions can be tightened.

Docker registry access denied during orchestrator boot

The orchestrator pulls its container images from a private registry that's authorised per AWS account. If the account isn't on the allow-list yet, the boot script fails with a registry-auth error.

Verify your account ID with aws sts get-caller-identity and ask Blackfire to authorise it.

EC2 launch fails with InsufficientInstanceCapacity or InstanceLimitExceeded

AWS-side capacity / quota issue, not Waveshift.

Capacity: the region simply doesn't have spare instances of the type Waveshift requested right now. Try a different region from Infrastructure administration, or wait an hour and retry.

Quotas: the account hasn't been raised above the default EC2 limits. Check the relevant service quota:

aws service-quotas get-service-quota \
  --service-code ec2 --quota-code L-1216C47A

Deployment¶

wsctl deploy finishes but the Console isn't reachable

The orchestrator can take up to 5 minutes after wsctl deploy all to finish booting and registering DNS. If it's been longer than that:

Check the EC2 instance is running:

aws ec2 describe-instances \
  --instance-ids "$(jq -r .waveshift.instance_id wsctl.json)"

Check the private Route 53 record exists. Since the zone is private, dig from outside the VPC won't find it; query it through the AWS API:

aws route53 list-resource-record-sets \
  --hosted-zone-id "$(jq -r .infrastructure.hosted_zone_id wsctl.json)" \
  --query "ResourceRecordSets[?Name=='console.waveshift.internal.']"

Or, if you're connected to the WireGuard network from a Hub or Node, just resolve it locally:

dig +short console.waveshift.internal

Shell into the instance via SSM and check the orchestrator service: sudo journalctl -u waveshift -n 200.

Region enablement stuck in Provisioning

The Web UI polls every 5 seconds while a region is in a transitional state. If a region stays in Provisioning for longer than 10 minutes, the saga has likely failed silently.

Click into the row in Infrastructure administration — the Notes column will show the AWS-side error (most often a VPC quota). Hit the Retry icon after the quota's raised.

Egress Server stays In Progress or flips to Failed

Three usual culprits:

AWS quota — the account has hit its EC2 / public IPv4 limit in that region. The error is shown in the row's details.
Region not active — the parent region was disabled or is itself in a transitional state. Confirm in Infrastructure administration.
No available Node — Node integration was requested but the Site has no online, unassigned Node, so the build fails with no available nodes in the selected site. Check the Nodes page, or rebuild without Node integration.

Network connectivity¶

Cannot resolve *.waveshift.internal from the orchestrator

The private Route 53 zone or the VPC's DNS settings are wrong.

aws route53 list-hosted-zones
aws ec2 describe-vpcs --vpc-ids "$(jq -r .vpc_id wsctl.json)"

The VPC must have enableDnsSupport and enableDnsHostnames both set; the private zone must be associated with the VPC.

WireGuard tunnel won't come up

Check the obvious side first — is UDP 51820 open between the client and the egress server? Most home / corporate firewalls block outbound UDP by default.

From the client:

nc -u <endpoint-ip-from-the-peer-config> 51820

Then on the egress server (via wsctl connect):

sudo wg show

A peer with latest handshake: never plus a non-zero transfer: line means traffic is reaching the server but not completing the handshake — usually a key mismatch.

Hub can't reach orchestrator (egress fleet appears offline)

The Hub talks to the orchestrator through its Site's Infrastructure Access Server. If that server is down, the Hub and its Nodes lose their management path.

Confirm the Site's server is running. If its region is Failed, fix that first from Infrastructure administration.

Web Console access¶

Console loads but features are missing or greyed out

Most likely a licence or integration issue. Files and Workspaces only appear when the integration is enabled in Integrations administration and the user holds the matching per-user licence (storage, workspaces) in Users administration. Also open System administration and confirm:

Validity is Valid, not Invalid or Expired.
The features you're trying to use (workspaces, storage, cloudflare, …) appear in the Features chip list.

First-user login fails

A fresh deployment creates a single admin user with the password admin123 (see Defaults). If that password has been changed and forgotten, another admin can set a new one from Users administration; there's no CLI reset command.

Browser shows certificate warning

The Console runs on a private domain (default waveshift.internal), so its TLS certificate is signed by wsctl's internal CA rather than a public authority. Browsers will show a "Not secure" / "self-signed certificate" warning on first visit — that's expected. Trust the cert from the browser dialog (or import the CA root from the orchestrator) to silence the warning.

Check or rotate the cert with:

wsctl cert status
wsctl cert rotate

wsctl cert rotate triggers a fresh issuance.

Hardware setup¶

Nodes don't appear in the Web UI

Walk down this list in order:

Cabling — Node WAN to Bearer (internet), Node LAN2 to Hub (management). Both cables are required.
Power — Nodes need a 3 A USB-C supply. Sub-3 A bricks brown the node out part-way through boot.
Time — give each Node 1–2 minutes to register after power-on, then refresh the Nodes page in the Console.
Hub internet — confirm the Hub's web UI (http://10.34.<site-number>.1; 10.34.0.1 for the first site) loads from a laptop plugged into the Hub. If not, the Hub itself isn't online and the Nodes have no path to the orchestrator.
Firmware — Nodes from a Waveshift kit ship pre-flashed. Stock GL.iNet firmware will not register.

Cannot reach Hub at http://10.34.0.1

Confirm the Ethernet adapter on your laptop / tablet received a 10.34.0.x DHCP lease (10.34.<site-number>.x once the Hub has been configured for a later site).
Try a different LAN port on the Hub. The WAN port is the upstream port and won't serve the admin UI.
Power-cycle the Hub: unplug, wait 10 seconds, replug, wait 60 seconds for full boot.
If still nothing, factory-reset the Hub (hold reset 10 seconds) — the credentials revert to those in Defaults.

Egress server build fails for users behind a Node

The Node's WAN port must reach the public internet. Plug a laptop into the Bearer's LAN; if ping 8.8.8.8 works there, the Bearer is fine. Then check the Bearer's connected-clients list shows the Node — if it doesn't, the Node's WAN cable isn't seated.

Configuration file¶

wsctl.json is corrupt or missing fields

Validate the JSON itself first:

jq . wsctl.json

Then compare its keys against the Configuration reference. The safest fix is to copy the missing fields from the most recent successful deploy log; failing that, re-run wsctl config init in a separate directory and copy the relevant sections across.

Managing multiple deployments

wsctl always reads wsctl.json from the current directory. Keep separate directories per deployment:

cd ~/waveshift-prod   && wsctl status
cd ~/waveshift-stage  && wsctl status

Don't symlink configs across directories — wsctl deploy writes back to the file and you'll mix state.

Performance¶

Deployment is slower than expected

The first deploy in a region is dominated by VPC / NAT-gateway creation (5–8 min on its own); subsequent egress / infra server creation is much faster. If a single egress build is taking more than 10 minutes, check the saga state from the orchestrator host:

sudo journalctl -u waveshift -g 'saga' -n 200

Health checks repeatedly time out

Increase the timeout once if you suspect a slow region:

wsctl health-check --wait --timeout 900

Repeat-failing health checks after a successful initial deploy usually mean the orchestrator has lost AWS API access — check the orchestrator's IAM role hasn't been edited.

Reporting issues¶

When sending a report, include:

wsctl version output.
AWS region(s) and account ID.
Sanitised wsctl.json (strip credentials and ARNs if you don't want them shared).
The exact error message and any relevant logs (orchestrator journalctl, Web Console browser console, Hub dmesg).
The reproduction steps.

See Glossary if any term in an error message isn't familiar, and Defaults for the values an unmodified deployment uses.