Overview

drifter mirrors public NOAA AOML Global Drifter Program (GDP) source files from the AOML ERDDAP file server (https://erddap.aoml.noaa.gov/gdp/erddap/files/) onto the lab's /spray storage. It crawls the ERDDAP directory listing for each requested product, writes an HTML catalog snapshot plus a JSON/CSV manifest of the selected files, then downloads the data files themselves (.nc, .csv, .txt, .dat, .mat). Downloads are idempotent and resumable-safe: each file is streamed to a .part temp file and atomically renamed into place.

The default product set covers the standard GDP distributions:

drifter_6hour_qc      drifter_hourly_qc       drifter_annualmeans
drifter_monthlymeans  drifter_currentvariance drifter_loopers
drifter_nonloopers

Installation & dependencies

The downloader is pure standard library (argparse, csv, json, urllib, pathlib) plus two shared repo helpers — no third-party packages required. It is part of the odsl distribution:

python -m pip install -e ".[all]"     # or just run the script in-place

Shared logic comes from the repo-root common/ package:

  • common.log.log — timestamped stderr logging.
  • common.download.write_bytes_safely — atomic write of catalog/manifest bytes.

The script prepends the repo root to sys.path at import time, so common.* resolves even when the file is run directly from the drifter/ directory.

Scripts

Script Purpose Key flags
download_drifter_data.py (console: odsl-drifter-download) Crawl GDP products on AOML ERDDAP, build catalogs/manifests, download source files via the shared common.sync engine --out-dir/--out, --products, --include-rafos, --limit/--max-files, --overwrite, --catalog-only, --timeout, plus shared --dry-run, --jobs, --verify {none,size,checksum}, --retries

Flag details:

  • --out PATH — output root (default /spray/drifters).
  • --products — comma-separated product names, or all to auto-discover every drifter_* product from the server (default: the seven products listed above).
  • --include-rafos — also fetch RAFOS_SOFAR_Floats.
  • --max-files N — cap the number of selected files (testing).
  • --overwrite — rename any existing file to NAME.backup-<UTC timestamp> and re-download; without it, existing files are skipped.
  • --catalog-only — write catalogs and the selected-file manifest, download nothing.
  • --timeout SECONDS — per-request timeout (default 180).

Usage

Runs as the console script odsl-drifter-download (or python drifter/download_drifter_data.py):

# Download the standard GDP products into /spray/drifters
odsl-drifter-download

# Preview what would be fetched — plans the transfer, writes nothing
odsl-drifter-download --dry-run

# Only build catalogs + selected-file manifest (no data download)
odsl-drifter-download --catalog-only

# Discover and pull every drifter_* product on the server, 4 downloads in parallel
odsl-drifter-download --products all --jobs 4

# A specific subset, plus RAFOS/SOFAR floats
odsl-drifter-download --products drifter_6hour_qc,drifter_hourly_qc --include-rafos

# Re-verify existing files against the server by size (re-downloads any that differ)
odsl-drifter-download --products drifter_6hour_qc --verify size

# Small smoke test into a scratch directory
odsl-drifter-download --out-dir ./tmp/drifters --limit 20

Re-running is idempotent: files already present are skipped (existence only by default; --verify size|checksum checks them against the server). --overwrite forces re-download, keeping a NAME.backup-<UTC> copy.

Data layout / outputs

Under --out (default /spray/drifters):

/spray/drifters/
├── catalog/
│   ├── <product>.html                    # raw ERDDAP directory listing
│   ├── selected_drifter_files.json       # manifest: source, base_url, files[]
│   └── selected_drifter_files.csv        # product,file,url rows
└── <product>/
    └── <file>.nc | .csv | .txt | .dat | .mat

main() returns exit code 1 if any download failed, else 0.

Conventions & gotchas

  • Console entry point. Run as odsl-drifter-download … (or python drifter/download_drifter_data.py …). The transfer, skip/verify, retry, parallelism (--jobs) and --dry-run come from the shared common.sync engine; only ERDDAP discovery is drifter-specific.
  • Idempotent downloads. Existing files are skipped by default (streamed to .NAME.part then atomically replaced, so a failed transfer never loses the prior file). --verify size|checksum re-checks existing files against the server; --overwrite re-downloads (keeping a NAME.backup-<UTC> copy).
  • Shared common.*. Logging and atomic writes are shared with the glider package — keep both in sync when editing those helpers.
  • Overridable data root. Never hardcode /spray/drifters downstream; pass --out.
  • ../common — common.log, common.download shared helpers.
  • ../glider — sibling in-situ downloader with the same structure.
  • ../README.md — repo-wide conventions (data roots, idempotency, hyphenated scripts).