Ocean Dynamics from Space Lab

Satellite-oceanography pipeline: search, download, subset, regrid, map, and colocate SSH, SST, ocean colour, and in-situ data from NASA Earthdata and other archives. Pip name odsl. Each top-level directory is an independent subpackage; most work is done through odsl-* console scripts.

qg is parked (source stays under qg/, not installed). See qg/PARKED.md and docs/FIX_PLAN.md.

Data-root layouts on /spray are documented in common/DATA_PATHS.md. Non-code dirs: docs/ (notes and tutorials), database/ (per-cycle CSV manifests), build/ / tmp/ (throwaway).

Documentation

A browsable HTML site is generated from these READMEs into docs/html/:

pip install -e ".[docs]"          # provides the one build dependency (markdown)
python docs/build_site.py         # writes docs/html/ (open docs/html/index.html)
python docs/build_site.py --serve # build, then serve at http://localhost:8000

Installation

On Spray, add the following to your .bashrc:

source /opt/miniforge3/bin/activate

From this directory:

python -m pip install -e ".[all]"   # editable install with all extras

The distribution is named odsl. Dual import works for every subpackage (import swot and import odsl_code.swot).

import odsl_code
import odsl_code.swot
import swot

Extras: plot (cartopy), kriging (pykrige), argo (gsw), insitu (pyarrow), pace (tenacity), swot-passes (geopandas/shapely), mpi (mpi4py; not in all), dev (pytest, ruff), docs (markdown). Guard optional imports and gate features on the extra.

Console scripts are declared in pyproject.toml. The ones used most often:

Tests

pytest common/tests colocate/tests super_subsetter/tests \
       swot/tests pace/tests snpp/tests geos/tests \
       argo/tests drifter/tests glider/tests gcoos/tests icesat2/tests \
       multi_altimeters/tests rads/tests insitu_wind/tests ecco/tests

No /spray required. qg/tests/ are analytic scripts (python qg/tests/test_sqg_analytic.py); pytest collects nothing there. Do not add them to this list until qg is un-parked.

Lint: ruff (line length 120, Python 3.10) — configured in pyproject.toml.

Conventions

Processing scripts expose main() (argparse) as the console entry point. Reusable logic lives in the package utils.py or common/; keep scripts thin.

• Bounding boxes are west south east north in degrees. Longitudes may be [-180, 180) or [0, 360); helpers normalize (pyresample wants [-180, 180)).
• Data roots are module constants (DEFAULT_*_ROOT) overridable by CLI flags. Do not hardcode a new /spray path; document it in common/DATA_PATHS.md.
• NASA Earthdata goes through earthaccess with credentials in ~/.netrc (machine urs.earthdata.nasa.gov). CMR collection concept IDs are cached in per-package collection_ids.json and refreshed on miss.
• Idempotent I/O. Skip existing files unless an --overwrite* flag is set. Write run products to tmp/ or --out, not the source tree. .gitignore excludes *.nc *.csv *.png *.bin. The database/*.csv cycle manifests are a tracked exception — do not add more data files.
• xarray / netCDF. Prefer xarray. When repackaging, keep packed integer variables' scale_factor / add_offset (do not silently unpack).
• SST products convert Kelvin → sst_celsius, derive recommended_quality (quality_level >= 5), and decode l2p_flags into is_day / is_night.
• New instrument. Add a package with utils.py, README.md, pyproject.toml entries (packages, package-dir, optional script), and a line in the table above plus the root __init__.py docstring. Dual import (import foo and import odsl_code.foo) must keep working.

Deeper architecture, CLI map, and known debt: docs/REPO_NOTES.md.

Domain notes

• SWOT cycle number is not time. Science phase (21-day repeat): cycles from 1, 584 passes/cycle. Cal/val (1-day, early 2023): cycles ~474–577, 28 passes/cycle. swot.constants.CALVAL_CYCLE_MIN is 474. Expert is 2 km on a fixed geographically registered grid; Unsmoothed is 250 m on a swath-following grid that shifts between repeats; PIXC is highest resolution and noisest. KaRIn noise depends on cross-track distance and SWH. Quality flags are aggressive (*_qual == 0 is best, but the usual keep is ssha_karin_2_qual <= 2). Details: swot/README.md.
• ICESat-2 on /spray is release 007. ATL10 freeboard lives in freeboard_segment (the older freeboard_beam_segment yields zero rows). ATL07 uses sea_ice_segments. Beams: gt1l gt1r gt2l gt2r gt3l gt3r. Details: icesat2/README.md.
• NumPy 2. np.RankWarning is gone — filter polyfit warnings by message.
• Argo / in-situ wind console scripts are underscored modules (odsl-argo-index, odsl-insitu-wind-unify, …); hyphenated filenames are shims. swot/combine_passes.py needs the mpi extra and is run under mpirun.

Downloading & syncing data

Every provider downloader shares one sync engine (common/sync.py), so they take the same flags: --dry-run (plan without transferring), --jobs N (parallel downloads), --verify {none,size,checksum} (re-check existing files against the server), --overwrite, --retries, and --limit. Downloads are idempotent — a file already present is skipped, and every transfer streams to a .part temp then atomically replaces the target, so an interrupted run never loses the previous file.

# NASA Earthdata (needs ~/.netrc for urs.earthdata.nasa.gov)
odsl-swot-download                                               # most recent cycle, Expert + Unsmoothed (default)
odsl-swot-download --start-cycle 1 --end-cycle 30 --product-type Expert
odsl-pace-download --all-nrt --days-back 3 --jobs 4
odsl-snpp-sst  --start 2024-01-01 --end 2024-01-10 --bbox -98 18 -80 31 --download-only
odsl-geos-sst  --start 2024-06-10 --end 2024-06-11 --collection current --download-only  # --start/--end required
odsl-icesat2-download --start 2024-01-01 --end 2024-01-03 --bbox -180 60 180 90
odsl-ecco-download                                               # V4r4 observational constraints (~9.1 GiB)

# In-situ HTTP archives (public; no credentials)
odsl-drifter-download --products drifter_6hour_qc
odsl-glider-download  --pattern delayed

# Shared behaviours (work on any of the above)
odsl-icesat2-download --start 2024-01-01 --end 2024-01-03 --bbox -180 60 180 90 --dry-run
odsl-drifter-download --products drifter_6hour_qc --jobs 4 --verify size

Argo, RADS, and in-situ wind are staged on /spray by other means (Argo/RADS have index/repack tools rather than downloaders). See each package README for provider-specific search flags (cycles, date/bbox, products).

SWOT Data Processing Scripts

Preferred interface: the odsl-swot-* console scripts (odsl-swot-download, odsl-swot-regrid-1km, odsl-swot-footprints, …). The same modules also run as python swot/<script>.py. The workflow below covers download, cycle combine, PIXC, and 1 km regrid.

1. swot_downloader.py

This script handles the entire download process in a single pass. It queries the NASA CMR (Common Metadata Repository), builds a prioritized list of unique files, and downloads any missing files through the shared common.sync engine (same skip/verify/.part behaviour as the other Earthdata downloaders). This replaces the functionality of the older grace_0_build_unique_file_list.py and grace_1_swot_file_downloader.py scripts.

Default (no arguments): downloads the missing granules of the single most recent cycle for both the Expert and Unsmoothed products. The most recent cycle is auto-detected from CMR by observation time, so it is the ongoing science phase — not the higher-numbered cal/val cycles (SWOT restarted cycle numbering between its two orbits).

Workflow: 1. Obtains an Earthdata token and resolves collection concept IDs. 2. In the default "latest" mode, detects the most recently observed cycle from CMR. 3. Queries NASA CMR for download links for each requested product type and cycle. 4. Saves all discovered links and the prioritized unique file list per cycle. 5. Skips files that are already present locally and match remote metadata. 6. Downloads missing files into the data/cycle_###/ directory.

Usage:

# Most recent cycle, Expert + Unsmoothed together (the default)
python swot/swot_downloader.py

# Most recent cycle, Expert only
python swot/swot_downloader.py --product-type Expert

# An explicit cycle range for a single product
python swot/swot_downloader.py --start-cycle 1 --end-cycle 30 --product-type Expert

--product-type accepts multiple products; passing an explicit --start-cycle/ --end-cycle range disables latest mode.

2. combine_passes.py

This script organizes and combines the downloaded files by mission cycle. It retains only the essential variables, such as sea surface height anomaly (SSHA), significant wave height (SWH), and SSHA quality flags. To minimize file size while preserving data integrity, it applies scale factor compression. This script replaces the previous grace_2_combine_files_per_cycle.py.

3. swot/L2_HR_PIXC.py

Discover and download SWOT L2 HR PIXC Version D granules.

This module searches NASA CMR for the SWOT Level 2 Water Mask Pixel Cloud Data Product, Version D (SWOT_L2_HR_PIXC_D), using a temporal range and a domain bounding box, then downloads matching NetCDF granules with earthaccess. Default output is /spray/swot/pixc (override with --out-dir).

Examples:

    python swot/L2_HR_PIXC.py \
        --start 2024-01-01T00:00:00Z \
        --end 2024-01-03T00:00:00Z \
        --bbox -92 28 -88 31 \
        --out-dir /spray/swot/pixc

    python swot/L2_HR_PIXC.py \
        --start 2024-06-01 \
        --end 2024-06-02 \
        --bbox -124 36 -121 39 \
        --manifest-only

4. swot/regrid_unsmoothed_to_1km.py

Regrid one SWOT Unsmoothed pass onto the uniform 1 km swath grid using segmented local optimal interpolation. The script reads Unsmoothed granules from /spray/swot/versionD, uses the prebuilt 1 km swath grids in /spray/swot/regrid_1km/{science,calval}/, and writes pass-level products to /spray/swot/regrid_1km/cycle_###/pass_###_regridded_1km.nc.

Each target point uses observations within the chosen search radius, applies a local pre-OI outlier clip relative to the neighborhood median (3σ), then solves the Gaussian OI system. Segment processing can run in parallel with --workers.

By default the output contains ssha and uncert. Two built-in OI configurations are stacked on a n_param dimension (r1000_l500 and r2000_l1000), with search_radius_m, half_decay_scale_m, and parameter_labels. A custom --search-radius + --half-decay-scale pair writes 2-D ssha / uncert instead.

Example:

python swot/regrid_unsmoothed_to_1km.py --cycle 3 --pass 1
python swot/regrid_unsmoothed_to_1km.py --cycle 474 --pass 13 --nugget 0.02
python swot/regrid_unsmoothed_to_1km.py --cycle 3 --pass 1 \
    --search-radius 1500 --half-decay-scale 800 --workers 8

5. swot/plot_regridded_segment.py

Plot a latitude-bounded segment from one of the regridded 1 km pass files. The script reads /spray/swot/regrid_1km/cycle_###/pass_###_regridded_1km.nc, subsets rows that overlap the requested latitude band, masks pixels outside the band, and writes a PNG next to the source NetCDF by default.

Example:

python swot/plot_regridded_segment.py \
    --cycle 3 \
    --pass 1 \
    --start-lat -50 \
    --end-lat -45

python swot/plot_regridded_segment.py \
    --cycle 3 \
    --pass 1 \
    --start-lat -50 \
    --end-lat -45 \
    --variable ssha_r2000_l1000 \
    --output tmp/pass001_segment.png

Cross-dataset colocation (colocate)

The colocate package spatiotemporally matches observations from any registered source against any other within a great-circle radius r and a time window delta_t. Swaths (SWOT, PACE, SNPP, GOES), nadir altimetry (RADS), L4 maps (AVISO), and in-situ platforms (Argo, gliders, drifters, wind) all reduce to a common point cloud, so they mix freely — e.g. SWOT SSHA × PACE chlorophyll, Argo × SNPP SST, or SWOT × GOES.

Adding a new instrument is a one-file, one-decorator change (write an ObsProvider, register it); the matching engine never changes. See colocate/README.md for the architecture, the provider list, and how to add a source.

Spatial matching is accelerated by prebuilt lookup tables so files whose footprint misses the region are never opened:

Build the tables once; providers query them automatically. SWOT × ICESat-2 also has coarse-path shapefiles (python -m colocate.build_swot_icesat2_paths). SWOT Expert × RADS pair files (one NetCDF per crossing, ±1 h by default) are written by odsl-colocate-swot-rads to /spray/swot/swot_rads. Plot a pair with odsl-colocate-swot-rads-plot (writes PNGs under tmp/ by default).

odsl-colocate-refresh builds or inspects every registration index:

odsl-colocate-refresh status
odsl-colocate-refresh refresh --targets snpp,geos,rads,aviso,wind

Usage:

# SWOT SSHA vs PACE chlorophyll + Argo, Gulf Stream, ±6 h, 25 km
odsl-colocate --reference swot --targets pace_bgc,argo \
    --bbox -80 25 -60 45 --start 2024-05-01 --end 2024-05-03 \
    --delta-t 6h --radius 25 --grid-res 0.25 --out-dir tmp/coloc

# Anchor on Argo; match SWOT and VIIRS SST
odsl-colocate --reference argo --targets swot,snpp \
    --delta-t 12h --radius 50 --out-dir tmp/coloc

odsl-colocate --list-providers

Registered sources: swot, pace (pace_bgc / pace_aop / pace_iop / pace_par), argo, glider, drifter, icesat2, snpp, geos (goes), rads, aviso (aviso_l4), wind (insitu_wind).

Multi-altimeter SSHA mapping (multi_altimeters)

The multi_altimeters package merges SWOT L2 LR SSH Expert swaths with conventional RADS nadir altimetry (Jason, Sentinel-3/6, CryoSat-2, SARAL, the SWOT nadir altimeter, …) into gridded SSHA maps by space-time optimal interpolation. Three target-grid modes are available:

• Regular lat/lon maps — odsl-multi-altimeter-oi: search a bbox and time window, load both sources into one observation table, and krige onto a degree grid at a single analysis time.
• Regional time cube — odsl-multi-altimeter-region: the same SWOT Expert and RADS loaders on a regular lon/lat grid (default 2 km at the bbox-center latitude) and a 6-hourly time axis. Each SWOT and RADS sample keeps its native measurement time; the 6-hourly axis is the map time only. A 2nd-order spatial background and a harmonic point-source internal tide are fit once, then the residual is OI’d at each analysis time (Δt = t_obs − t_k). Output dims are time, lat, lon.
• Expanded swath grids — odsl-multi-altimeter-swath-grid: a grid that follows one SWOT Expert segment's swath geometry but is widened by a chosen distance (default 100 km) on both sides at the same uniform 2 km posting. Pass --target-width-km 300 to set a total cross-track width of 300 km (±150 km from the centerline) instead of an expansion per side. Nadir observations are selected inside the expanded footprint within ±4 days (default) of the segment time, so the OI extends the SWOT scene beyond the swath edges. The output NetCDF carries the SWOT data on its native columns, the binned nadir observations, and the OI analysis. The production pass-by-pass driver is odsl-swot-swath-grid-300km (5×5 SWOT thinning with sample-std uncertainty, 0.02 m nadir error, 4-day time half-width, ±5.5-day search, other SWOT passes included).

A separate regional subsetter package (super_subsetter) provides odsl-super-subsetter: harvest a bbox/time-window subset from eleven /spray observation archives (SWOT, RADS, AVISO DUACS L4, Argo, drifters, gliders, ICESat-2, in-situ wind, PACE ocean colour, SNPP SST, GOES ABI SST) into one HDF5 file with a group per source, each keeping its native structure (2-D swaths clipped along-track only, per-pass/per-platform profiles and trajectories, lon/lat grid slices). Variable selection is driven by an editable +/- markdown config (--init-vars-config → edit → --vars-config), defaulting to essentials with uncertainties; a markdown data summary (<out>.summary.md) reports per-group counts, time spans, coverage, and statistics. See super_subsetter/README.md and multi_altimeters/README.md.

# Expanded swath grid for one segment of a California-coast pass
odsl-multi-altimeter-swath-grid \
    --swot-file SWOT_L2_LR_SSH_Expert_525_013_20230519T001722_20230519T010829_PGD0_01.nc \
    --line-range 7068 7182 --expansion-km 100 --delta-t-days 4 \
    --out tmp/swath_ssha_oi.nc

# Regional lon/lat time cube (2 km, 6-hourly, background + M2 tide + OI residual)
odsl-multi-altimeter-region \
    --bbox -126 34.5 -122 36.5 \
    --start 2024-02-01 --end 2024-02-10 \
    --dx-km 2 --dt-hours 6 \
    --out tmp/region_ssha_cube.nc

The OI decorrelation function is configurable in size and shape (--spatial-half-width-km / --time-half-width-days with --covariance-model {gaussian,exponential,markov}), and observation errors can be set per mission (--nadir-obs-error-by-mission JASON-3=0.03 SNTNL-3A=0.015). Both sources can be thinned before mapping — --swot-median-window (N×N KaRIn median blocks) and --rads-thin-window (N-point nadir along-track blocks, default 10) — in which case each block contributes its median ssha with a scatter-based observational error (the std of its member values, with configurable floors). Per-pass bias and SWOT cross-swath polynomial corrections are optional pre-processing steps, and --fit-internal-tides decomposes the observations before the OI: a 2nd-order polynomial background plus a coherent fixed-frequency (M2) internal tide modeled as point sources around the region center, with the OI mapping the residual. The full mathematics is written out in multi_altimeters/methodology.md. A figure-producing demonstration lives in multi_altimeters/demo_swath_grid_ssha.py. See multi_altimeters/README.md for the full option reference, output schemas, and the point-vortex SSH inversion tools that share this package.