Skip to content

Network Monitoring Probe – Engineer Runbook

Purpose

This probe provides independent visibility into your institution’s network experience. It measures reachability, latency, DNS resolution, web access, and probe system health. The data feeds your institution’s monitoring dashboard and supports joint troubleshooting when issues arise.

This probe does not monitor other institutions.

What This Probe Monitors

From your institution, the probe continuously measures:

  • Network reachability (ICMP) to:
    • Your upstream/core gateway
    • Public Internet reference points
  • DNS resolution performance
  • Web access experience (HTTP/HTTPS)
  • Probe system health (CPU, memory, uptime)

This reflects what users at your institution would experience.

Before You Start

You must have:

  1. Ubuntu Linux VM
    • Ubuntu 24.04+
    • Minimum: 1 vCPU, 1 GB RAM, 10 GB disk
  2. Root access (sudo)
  3. Internet access
  4. Two files from GARNET
    • install.sh
    • wg0.conf (WireGuard configuration)

Files Layout (IMPORTANT)

Place both files in the same directory:

probe/
├── install.sh
└── wg0.conf

Do not rename wg0.conf.

Installation Steps

Make script executable

chmod +x install.sh

Run the installer as root

sudo ./install.sh

No additional configuration is required.

What the Script Does (High Level)

  1. WireGuard
    • Uses the provided wg0.conf
    • Brings up secure tunnel (wg0)
  2. Node Exporter
    • Exposes CPU, memory, uptime metrics
    • Listens on :9100
  3. Blackbox Exporter
    • Performs ICMP, DNS, and HTTP checks
    • Listens on :9115
  4. Verification
    • Confirms services are running locally

Verification Checklist

After installation, confirm:

WireGuard

ip a show wg0

Expected: Interface exists and is UP

Node Exporter

curl http://localhost:9100/metrics

Expected: Metrics output displayed

Blackbox Exporter

curl "http://localhost:9115/probe?module=icmp&target=1.1.1.1"

Expected: probe_success 1

Common Troubleshooting

WireGuard Not Connecting

  • Confirm wg0.conf is present and readable
  • Verify system time is correct
  • Check:
systemctl status wg-quick@wg0

Probe Appears Offline in Dashboard

  • Confirm services are running:
systemctl status node_exporter
systemctl status blackbox_exporter
  • Ensure no local firewall blocks:
  • TCP 9100
  • TCP 9115

High Latency or Packet Loss

  • Check local uplink congestion
  • Confirm gateway reachability
  • Compare core vs public endpoint results in dashboard

What to Share During Troubleshooting

If assistance is needed, provide:

  • Institution name
  • Probe hostname
  • Time issue was observed
  • Output of:
systemctl status wg-quick@wg0
systemctl status node_exporter
systemctl status blackbox_exporter

Important Notes

  • This probe only reports your institution’s data
  • No traffic inspection is performed
  • No user data is collected
  • Probe impact is minimal (<1% CPU, low bandwidth)

Summary

This probe helps distinguish between: - Local network issues - Upstream connectivity problems - DNS or service-specific failures

It enables faster, evidence-based troubleshooting for your institution.

Support

GARNET Network Operations Center
Thank you for supporting national network visibility.