Overview

pace searches, downloads, indexes, subsets, and plots PACE OCI Level-2 ocean-colour granules from NASA Earthdata. PACE (Plankton, Aerosol, Cloud, ocean Ecosystem) carries the OCI hyperspectral imager; the L2 products covered here are apparent optical properties (AOP — Rrs, avw, nflh, aot_865, angstrom), biogeochemistry (BGC — chlor_a, poc, pic, carbon_phyto, …), inherent optical properties (IOP — a, bb, Kd, aph, adg_442, bbp_442, …), and photosynthetically available radiation (PAR). The L4 mapped MOANA picophytoplankton community-structure product is also reachable.

PACE science data begins 2024-03-05, and all products are served under the OB_CLOUD provider (the former OB_DAAC provider returns no results).

Two pieces:

  • pace_downloader.py — a one-pass CMR query + downloader. It resolves the collection concept ID (cached on disk), pages through CMR granules for a date range and optional bounding box (harvesting each granule's exact size, checksum, and bounding box), writes a link manifest CSV, then downloads the missing .nc files through the shared sync engine (common/sync.py): --jobs N parallel workers, staged atomic writes (no truncated granules on interrupt), post-transfer size/checksum verification, per-file retry with backoff, --dry-run, and per-day directory layout. After each run it updates the granule footprint index from the CMR bounding boxes for every locally-present file — no granule is opened.
  • utils.py — file-index, granule-reader, BBox, and extraction helpers for working with the downloaded L2 files, plus get_pace(), a one-call search→download→subset→scatter-plot convenience function.

Products (CMR short names, provider OB_CLOUD, _NRT variants for near-real-time):

Short name Description
PACE_OCI_L2_AOP[_NRT] Apparent optical properties — Rrs spectrum, avw, nflh, aot_865, angstrom
PACE_OCI_L2_BGC[_NRT] Biogeochemical — chlor_a, poc, pic, carbon_phyto (+ _unc)
PACE_OCI_L2_IOP[_NRT] Inherent optical properties — a, bb, Kd, aph, adg_442, bbp_442 (+ _unc)
PACE_OCI_L2_PAR[_NRT] Photosynthetically available radiation — par_day_*, ipar_* (split by geometry; no plain par/ipar in v3.2)
PACE_OCI_L4M_MOANA[_NRT] MOANA picophytoplankton community structure (L4 mapped)

Installation & dependencies

Core deps (always installed with odsl): pandas, numpy, xarray, netcdf4 / h5netcdf, requests, tqdm, earthaccess. The pace extra adds tenacity (used by the @retry HTTP / download wrappers). Plotting in get_pace() additionally needs matplotlib and, optionally, cartopy (the plot extra) for map projections.

python -m pip install -e ".[pace]"        # downloader + utils
python -m pip install -e ".[pace,plot]"   # + cartopy map plotting
# or the full conda environment:
conda env create -f pace/pace_env.yml && conda activate pace_env

NASA Earthdata credentials are required. Store them in ~/.netrc:

machine urs.earthdata.nasa.gov login <username> password <password>

get_token() reads this file to mint an EDL bearer token for CMR, and earthaccess.login() uses it for downloads.

Scripts & entry points

Script / symbol Console entry point Purpose
pace/pace_downloader.py odsl-pace-download (pace.pace_downloader:main) Query CMR + download granules (harvests the granule bounding box from CMR into the manifest)
pace/granule_footprints.py odsl-pace-footprints (pace.granule_footprints:main) Build a per-granule bounding-box index (from the swath perimeter) for fast spatial pruning; query with paths_overlapping
pace/utils.py — (library) BBox, load_pace_l2, build_pace_index, extract_*, get_pace
pace/collection_ids.json — On-disk cache of CMR collection concept IDs

Usage

Download granules

# BGC science over the Gulf of Mexico for one week, 8 parallel workers,
# size-verifying existing files against the CMR manifest (local stat only)
python pace/pace_downloader.py \
    --product PACE_OCI_L2_BGC \
    --start 2024-03-05 --end 2024-03-11 \
    --bbox -98 18 -80 30 \
    --jobs 8 --verify size

# Full-swath AOP backfill (largest L2 product — check disk first)
odsl-pace-download --product PACE_OCI_L2_AOP --start 2024-03-05 --jobs 8

# Plan without transferring (also refreshes footprints for local files)
python pace/pace_downloader.py --product PACE_OCI_L2_BGC --start 2024-03-05 --dry-run

Flags: --product SHORT_NAME [SHORT_NAME ...] (one or more, default PACE_OCI_L2_BGC_NRT), --all-nrt (all NRT products, overrides --product), --days-back N (rolling window ending today, overrides --start/--end), --start / --end (YYYY-MM-DD, inclusive), --bbox WEST SOUTH EAST NORTH (decimal degrees, default none = global), --data-root (default /spray/pace), --database-dir (default <repo>/database), --provider (default OB_CLOUD). Shared sync flags (from common.cli.common_sync_parser): --jobs N (default 4), --verify {none,size,checksum} (size is a local stat-only check against the CMR-manifest size — a full-archive integrity sweep takes minutes; --verify-downloads is a legacy alias for --verify checksum), --dry-run, --overwrite, --retries N, --timeout S, --limit N (caps files per product), --log-file PATH (rotating; default /spray/pace/logs/pace_downloader.log, "" disables), --no-lock, -v/--verbose, -q/--quiet. Footprint maintenance: --no-footprint-update, --footprints-out PATH (default /spray/pace/footprints/pace_granule_footprints.parquet).

Scheduled / cron runs

The downloader is built to run unattended on a schedule:

  • Rolling window — --days-back N fetches the last N days ending today (UTC), so a fixed crontab line always pulls fresh data.
  • Product sets — --all-nrt (4 NRT L2 collections), or --all (everything: NRT + science L2 + MOANA L4 = 10 collections). NRT lands within hours; science lags days-to-weeks, so --all re-checks the same window and skips what is already present.
  • Idempotent — files already present are skipped (add --verify size to re-fetch truncated/stale files by CMR size — a local stat, no network — or --verify checksum for a full re-hash).
  • Footprint index stays current — every run upserts /spray/pace/footprints/pace_granule_footprints.parquet from the CMR bounding boxes of locally-present granules (used by colocate and odsl-super-subsetter for spatial search).
  • File logging — timestamped, size-rotating log via --log-file (default /spray/pace/logs/pace_downloader.log, 5 × 5 MB). Console goes to stderr; use --quiet to silence it under cron. The tqdm bar auto-disables off a TTY. Missing/unwritable log dir degrades to console-only, never crashes.
  • Single instance — an exclusive flock on /spray/pace/logs/pace_downloader.lock makes an overlapping run exit cleanly (0) instead of double-downloading.
  • Exit code — 0 on success (or lock contention), 1 if any download failed, so cron MAILTO / monitoring can alert on real failures.
# crontab -e  — daily at 02:00, last 3 days of *all* products (global).
# Defaults: data -> /spray/pace, log -> /spray/pace/logs/pace_downloader.log
0 2 * * *  /opt/miniforge3/bin/python \
    /home/jinbo.wang/code/odsl_code/pace/pace_downloader.py \
    --all --days-back 3 --jobs 4 --quiet >> /spray/pace/logs/cron.out 2>&1

Volume note: global --all is large — full-swath AOP granules are ~240 MB each and PACE images the globe every day. Add --bbox WEST SOUTH EAST NORTH to restrict to a region if disk is tight.

Read / subset in Python

from pathlib import Path
from pace.utils import BBox, load_pace_l2, extract_bgc, extract_rrs

bbox = BBox(west=-98, south=18, east=-80, north=30)
g = load_pace_l2(
    Path("/spray/pace/PACE_OCI_L2_BGC/2024/03/05/PACE_OCI.20240305T180543.L2.OC_BGC.V3_2.nc"),
    bbox=bbox,
)
bgc = extract_bgc(g)                 # xarray.Dataset: chlor_a, poc, pic, …
rrs, wavelengths = extract_rrs(g)    # (n_pixels, n_bands), (n_bands,)

# Build a local file index and filter by time
from pace.utils import build_pace_index, filter_index_by_time
import pandas as pd
idx = build_pace_index(Path("/spray/pace"), short_names=["PACE_OCI_L2_BGC"])
idx = filter_index_by_time(idx, pd.Timestamp("2024-03-05"), pd.Timestamp("2024-03-11"))

# One-call search + download + subset + scatter plot (nearest granule in time)
from pace.utils import get_pace
get_pace("2024-03-05T18:00:00", bbox, "chlor_a.png", "chlor_a", search_window="1D")
get_pace("2024-03-05", bbox, "rrs443.png", "Rrs", wavelength=443)

Granule footprint index (fast spatial pruning)

PACE flies a drifting orbit, so every granule has a unique footprint. Build a per-granule bounding-box index once (read cheaply from the swath perimeter), then prune granules to a region without opening them — used automatically by the colocate PACE provider.

odsl-pace-footprints --short-names PACE_OCI_L2_BGC     # -> /spray/pace/footprints/…parquet

from pace.granule_footprints import load_footprint_index, paths_overlapping
fp = load_footprint_index()
hits = paths_overlapping(fp, (-80, 25, -60, 45))       # granule paths crossing the box

Data layout / outputs

Downloader default data root is /spray/pace (override with --data-root); rotating logs and the single-instance lock live under /spray/pace/logs/. Granules are written per product / date:

{data-root}/{short_name}/YYYY/MM/DD/PACE_OCI.YYYYMMDDTHHMMSS.L2.<PRODUCT>.<VERSION>[.NRT].nc

Per-run link manifest CSV (columns: short_name, granule_ur, date, orbit, filename, url) goes to --database-dir (default <repo>/database/):

database/pace_{short_name}_{start}_{end}_links.csv

build_pace_index() walks that same tree and returns a DataFrame with columns path, short_name, datetime, product, version (short name is inferred from the .../{short_name}/YYYY/MM/DD/ parent path).

Conventions & gotchas

  • Group layout. L2 files are HDF5/netCDF4 with navigation_data (lon/lat), geophysical_data (variables), sensor_band_parameters (wavelengths), and scan_line_attributes (per-scan year/day/msec, reconstructed into UTC scan times). load_pace_l2 opens each group separately.
  • Scan-time decoding. day/msec carry timedelta-like units, so _scan_times opens scan_line_attributes with decode_timedelta=False — otherwise xarray decodes them to timedelta64, the integer reconstruction fails, and every scan time comes back NaT (silently breaking any temporal matching).
  • short_name is four levels above the file (.../{short_name}/YYYY/MM/DD/ file.nc), i.e. path.parents[3] — not three. Getting this wrong makes build_pace_index(short_names=[…]) match nothing.
  • Bounding boxes are west south east north in degrees. BBox normalizes to [0,360) internally (west360/east360/span360) so it handles dateline-wrapping boxes; BBox.contains(lon, lat) returns the per-pixel mask used by subset_bbox. Index-level bbox filtering (filter_index_by_bbox) is only a loose pass — precise filtering happens per pixel at load time.
  • Subsetting flattens the 2-D swath. After subset_bbox, arrays are 1-D along a pixel dim (banded vars keep a band dim); concatenate several with concat_granules.
  • Collection IDs are cached. collection_ids.json is populated on first miss via a CMR lookup and reused thereafter; delete the entry to force a refresh.
  • Retries on CMR/HEAD/download calls come from tenacity (stop_after_attempt(3), exponential backoff) — the pace extra.

See the repo-level ../README.md for cross-package conventions (Earthdata/~/.netrc, collection_ids.json caching, bounding-box normalization, idempotent downloads, data-root constants).