Overview

glider discovers and downloads public underwater-glider deployments from the IOOS National Glider DAC ERDDAP server (https://gliders.ioos.us/erddap/) onto the lab's /spray storage. It reads the ERDDAP allDatasets catalog, filters deployments by ID substring and/or time window, records a JSON manifest of the selection, then for each deployment queries its variable list and downloads the full aggregated NetCDF (all variables) via ERDDAP's tabledap interface. Downloads are idempotent and streamed to a .part temp file that is atomically renamed into place.

Installation & dependencies

The downloader is pure standard library (argparse, csv, json, urllib, pathlib) plus two shared repo helpers — no third-party packages required. It is part of the odsl distribution:

python -m pip install -e ".[all]"     # or just run the script in-place

Shared logic comes from the repo-root common/ package:

  • common.log.log — timestamped stderr logging.
  • common.download.write_bytes_safely — atomic write of catalog/metadata bytes.

The script prepends the repo root to sys.path at import time, so common.* resolves even when the file is run directly from the glider/ directory.

Scripts

Script Purpose Key flags
download_glider_data.py (console: odsl-glider-download) Read the IOOS glider DAC catalog, filter deployments, download each as an aggregated NetCDF via the shared common.sync engine --out-dir/--out, --pattern, --since, --until, --limit/--max-datasets, --overwrite, --catalog-only, --timeout, plus shared --dry-run, --jobs, --verify, --retries

Flag details:

  • --out PATH — output root (default /spray/gliders).
  • --pattern TEXT — keep only dataset IDs containing this (case-insensitive) substring.
  • --since ISO — keep deployments whose maxTime is on/after this ISO time.
  • --until ISO — keep deployments whose minTime is on/before this ISO time.
  • --gcoos — keep only IOOS DAC deployments submitted by username gcoos_dmac.
  • --layout {flat,tree} — flat (default) writes netcdf/{id}.nc; tree writes {operational,historical}/{operator}/….
  • --operational-days N — with --layout tree, recency cutoff (default 14).
  • --max-datasets N — cap the number of deployments (testing).
  • --overwrite — rename any existing file to NAME.backup-<UTC timestamp> and re-download; without it, existing files are skipped.
  • --catalog-only — write the catalog and selected-dataset manifest, download nothing.
  • --timeout SECONDS — per-request timeout (default 180).

Usage

Runs as the console script odsl-glider-download (or python glider/download_glider_data.py):

# Download the catalog and every active deployment as aggregated NetCDF
odsl-glider-download

# Preview the selection without downloading
odsl-glider-download --dry-run

# Only build the searchable catalog + selected-dataset manifest
odsl-glider-download --catalog-only

# Filter by dataset-ID substring, 4 downloads in parallel
odsl-glider-download --pattern delayed --jobs 4

# Time-window filters (ISO 8601, UTC)
odsl-glider-download --since 2025-01-01T00:00:00Z
odsl-glider-download --until 2024-12-31T23:59:59Z

# Small smoke test into a scratch directory
odsl-glider-download --out-dir ./tmp/gliders --limit 5

Data layout / outputs

Under --out (default /spray/gliders):

/spray/gliders/
├── catalog/
│   ├── ioos_glider_allDatasets.csv        # raw ERDDAP allDatasets table
│   └── selected_glider_datasets.json      # manifest: source, catalog_url, datasets[]
├── metadata/
│   └── <datasetID>.index.csv              # per-dataset ERDDAP info (variable list)
└── netcdf/
    └── <datasetID>.nc                      # full aggregated deployment

The variable query for each deployment is built from its info/…/index.csv (Row Type == variable); a deployment with no listed variables is skipped. main() returns exit code 1 if any download failed, else 0.

Conventions & gotchas

  • Console entry point. Run as odsl-glider-download … (or python glider/download_glider_data.py …). Transfer/skip/verify/retry/ --jobs/--dry-run come from the shared common.sync engine; only the IOOS ERDDAP catalog + per-dataset variable discovery is glider-specific.
  • Idempotent downloads. Existing files are skipped by default. --overwrite backs up the current file to NAME.backup-YYYYmmddTHHMMSSZ before re-fetching; stale .part temp files are backed up the same way.
  • Shared common.*. Logging and atomic writes are shared with the drifter package — keep both in sync when editing those helpers.
  • Overridable data root. Never hardcode /spray/gliders downstream; pass --out.
  • ../common — common.log, common.download shared helpers.
  • ../drifter — sibling in-situ downloader with the same structure.
  • ../README.md — repo-wide conventions (data roots, idempotency, hyphenated scripts).