Open-source release: pipeline code, raw logs, derived data, licences, tests
Adds src/ (scripts, raw measurement logs, derived data, report source), tests/, LICENSE (MIT, code) and LICENSE-DATA.md (CC BY 4.0 for the survey's data; third-party terms). The report is updated to the current site; the v1 and v2 data hand-off files are unchanged. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Executable
+39
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env bash
|
||||
# Download the third-party inputs that this repository does not redistribute.
|
||||
# raw/tg/ TeleGeography Submarine Cable Map (cables.py, network.py, build.py, export.py)
|
||||
# raw/wn/ WonderNetwork 2020-07-19/20 ping archive (analyze_wn.py)
|
||||
# raw/bgp/ RouteViews prefix2as, RIPE NCC AS names, PeeringDB (asmap.py)
|
||||
# The survey used the TeleGeography data as served on 2026-10-23; raw/tg.sha256
|
||||
# lists that snapshot. Later downloads may differ and change results slightly.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")"
|
||||
UA="cable-latency-survey/1.0 (+https://pingstats.aethercosmology.com)"
|
||||
get() { curl -sSf -A "$UA" --retry 3 -o "$1" "$2"; }
|
||||
mkdir -p raw/tg/cables raw/wn raw/bgp
|
||||
|
||||
TG=https://www.submarinecablemap.com/api/v3
|
||||
get raw/tg/cable-geo.json "$TG/cable/cable-geo.json"
|
||||
get raw/tg/landing-point-geo.json "$TG/landing-point/landing-point-geo.json"
|
||||
python3 - "$TG" "$UA" <<'PY'
|
||||
import json, sys, time, urllib.request
|
||||
tg, ua = sys.argv[1], sys.argv[2]
|
||||
ids = sorted({f["properties"]["id"] for f in json.load(open("raw/tg/cable-geo.json"))["features"]})
|
||||
for n, i in enumerate(ids, 1):
|
||||
req = urllib.request.Request(f"{tg}/cable/{i}.json", headers={"user-agent": ua})
|
||||
open(f"raw/tg/cables/{i}.json", "wb").write(urllib.request.urlopen(req, timeout=60).read())
|
||||
time.sleep(0.2)
|
||||
print(f"TeleGeography: {len(ids)} cables")
|
||||
PY
|
||||
get raw/tg/tga.json "$TG/cable/tasman-global-access-tga-cable.json"
|
||||
(cd raw && sha256sum -c --quiet tg.sha256 2>/dev/null && echo "TeleGeography data identical to the 2026-10-23 snapshot") \
|
||||
|| echo "note: TeleGeography data differs from the 2026-10-23 snapshot (raw/tg.sha256)"
|
||||
|
||||
S3=https://wp-public-data.s3.amazonaws.com/pings
|
||||
get raw/wn/pings-2020-07-19-2020-07-20.csv.gz "$S3/pings-2020-07-19-2020-07-20.csv.gz"
|
||||
get raw/wn/servers-2020-07-19.csv "$S3/servers-2020-07-19.csv"
|
||||
|
||||
get raw/bgp/routeviews-rv2-20261023-0200.pfx2as.gz \
|
||||
https://publicdata.caida.org/datasets/routing/routeviews-prefix2as/2026/10/routeviews-rv2-20261023-0200.pfx2as.gz
|
||||
get raw/bgp/asn.txt https://ftp.ripe.net/ripe/asnames/asn.txt
|
||||
for t in ixpfx ixlan ix; do get "raw/bgp/pdb_$t.json" "https://www.peeringdb.com/api/$t"; sleep 4; done
|
||||
echo "done; PeeringDB and AS names are current, not the 2026-10-27 copies used for data/asmap.json"
|
||||
Reference in New Issue
Block a user