Overview

argo processes Argo profiling-float CTD data from a GDAC-style tree of *_prof.nc files. It does three things:

  1. Index — scan every profile file into a compact one-row-per-profile lat/lon/time table (parquet or CSV) for fast spatial/temporal search.
  2. Flatten — expand Argo's N_PROF × N_LEVELS layout into one row per measured pressure level, preserving both raw and adjusted PRES/TEMP/PSAL plus their QC flags, with preferred columns using adjusted values where available.
  3. Derive — compute steric / dynamic height per profile from TEOS-10 (gsw), relative to a chosen reference depth.

find_profiles() searches the index by a time window plus a great-circle (haversine) radius around a target lat/lon and returns matches sorted by time then distance.

Installation & dependencies

Core deps (installed with odsl): numpy, pandas, xarray, netcdf4. Two extras are needed for the full workflow:

  • argo extra → gsw (TEOS-10 seawater toolbox) — required for calculate_steric_height.
  • insitu extra → pyarrow — required for parquet index/output writing (argo_profiles.py imports pyarrow at module load).
python -m pip install -e ".[argo,insitu]"

No NASA Earthdata / ~/.netrc credentials are needed here — this package consumes profile files already staged under /spray/argo.

Scripts & entry points

Script Console entry point Purpose
argo/argo_profiles.py — (library; also python argo/argo_profiles.py) All logic; its main() flattens files to parquet
argo/build_argo_profile_index.py odsl-argo-index Build the lat/lon/time index and optionally search it
argo/process_argo_profiles.py odsl-argo-process Thin wrapper that calls argo_profiles.main()

Hyphenated filenames remain as shims. argo_profiles.py is importable as argo / odsl_code.argo.

Usage

Build & search the profile index

# Update the index (default output /spray/argo/processed/argo_profile_index.parquet).
# Incremental by default: re-scans only *_prof.nc whose mtime changed since the
# last build (full scan the first time). Run this after every new download.
odsl-argo-index

# Force a full rescan of every profile file
odsl-argo-index --rebuild

# Small test index to a scratch path
odsl-argo-index --max-files 10 --index-file /tmp/argo_index.parquet

# Update (if needed) then search a time window + radius, writing matches
odsl-argo-index \
    --start-time 2023-01-01 --end-time 2023-02-01 \
    --latitude 30 --longitude -70 --radius-km 200 \
    --matches-file /tmp/argo_matches.csv

The five search flags (--start-time, --end-time, --latitude, --longitude, --radius-km) are all-or-nothing — supply all five or none.

Incremental updates. A sidecar argo_profile_index.mtimes.json (next to the index) records each file's modification time. On the next run, only files whose mtime changed — or new files, minus deleted ones — are re-opened; unchanged floats keep their existing rows. Argo appends new cycles into a float's existing *_prof.nc (bumping its mtime), so this correctly re-reads appended profiles. --rebuild forces a full rescan. Programmatic entry point: argo_profiles.update_profile_index(input_root, index_file, force=False).

Flatten profiles to parquet

odsl-argo-process                       # all floats under /spray/argo
odsl-argo-process --max-files 10        # debug subset
odsl-argo-process --good-qc-only        # keep only QC 1/2/5 rows
odsl-argo-process --input-root /spray/argo --output-root /tmp/argo_out

Python API

from argo import (
    discover_profile_files, load_profile_levels,
    find_profiles, calculate_steric_height,
)

files  = discover_profile_files("/spray/argo", max_files=1)
levels = load_profile_levels(files[0].path, good_qc_only=True)
steric = calculate_steric_height(files[0].path, reference_depth_m=1000)

matches = find_profiles(
    start_time="2023-01-01", end_time="2023-02-01",
    latitude=30, longitude=-70, search_radius_km=200,
)

Data layout / outputs

Input tree (GDAC-style):

/spray/argo/<dac>/<wmo>/<wmo>_prof.nc     # e.g. /spray/argo/aoml/1901628/1901628_prof.nc

Defaults (overridable via flags / function args):

  • Index → /spray/argo/processed/argo_profile_index.parquet (required columns: dac, wmo, source_file, profile_index, cycle_number, direction, data_mode, time, latitude, longitude, position_qc, profile_*_qc). When rebuilt with the current scanner it also writes extra surface_temp_c / surface_psu (shallowest valid level). Those extras are not part of PROFILE_INDEX_COLUMNS — older parquets without them still load. Sidecar argo_profile_index.mtimes.json (per-file mtimes) drives incremental updates.
  • Flattened profiles → /spray/argo/processed/profiles/<dac>/<wmo>_profiles[_good_qc].parquet (one row per level; pressure_dbar/temperature_c/salinity_psu prefer adjusted values, with *_source columns marking adjusted vs raw).
  • process_profile_files also writes profile_manifest.csv and profile_summary.json in the output root.

.parquet vs .csv output is chosen from the file suffix; parquet is written with zstd compression.

Conventions & gotchas

  • CLI names. Prefer odsl-argo-index / odsl-argo-process. Underscored modules are importable; hyphenated filenames are compatibility shims.
  • QC flags. "Good" QC is the set {"1", "2", "5"}. --good-qc-only / good_qc_only=True filter on raw PRES/TEMP/PSAL flags; steric-height selection additionally checks the adjusted QC flag when the value came from the adjusted field.
  • Adjusted-vs-raw preference. Preferred value columns take the adjusted value where finite, else raw; the choice is recorded in pressure_source, temperature_source, salinity_source.
  • Times. JULD is decoded from its units attribute (default epoch 1950-01-01 UTC) to timezone-aware UTC timestamps.
  • Steric height = TEOS-10 dynamic-height anomaly at the top level divided by 9.80665. Profiles shallower than the reference depth are kept with a non-ok status (e.g. profile_shallower_than_reference, too_few_valid_levels) and null steric height, rather than dropped.
  • No bounding box here — spatial search is a point + --radius-km great-circle query, not a west south east north box (unlike the satellite packages).

See the repo-level ../README.md for cross-package conventions, including the shared note on hyphenated script filenames and the in-situ data roots on /spray.