Download, thin, subset, and visualize ICESat-2 sea-ice products from NSIDC Earthdata Cloud. Part of the ODSL pipeline (odsl_code)

Overview

Two ICESat-2 ATLAS/ATL products are handled here:

  • ATL07 — sea ice height (surface height segments over sea ice / leads).
  • ATL10 — sea ice freeboard (freeboard relative to a local sea-surface reference; also carries lead and reference-surface data).

The package is a four-stage pipeline, each stage a standalone script:

  1. download (download_icesat2.py) — query NASA CMR for ATL07/ATL10 granules, write JSON+CSV manifests, and download the HDF5 files.
  2. thin (thin_icesat2.py) — copy each HDF5 granule keeping only a curated list of SSH / sea-ice variables (drops the many unused groups), gzip-compressed.
  3. subset (subset_polar_segments.py) — read the per-beam segment groups, keep rows poleward of ±55° latitude, drop pure-land rows, and write one tabular Parquet/CSV file per granule plus a run summary.
  4. dashboard (polar_segments_dashboard.py) — a local web app to browse the Parquet subsets: map, histograms, and a preview table with file/beam filters.

Installation & dependencies

Install the package (editable, with all extras) from the repo root:

python -m pip install -e ".[all]"

Key third-party dependencies used by these scripts:

  • requests — CMR search and HTTP downloads (download_icesat2.py).
  • h5py, numpy — read/write HDF5 granules (thin_icesat2.py, subset_polar_segments.py).
  • pandas, pyarrow — tabular output and Parquet I/O (subset + dashboard).
  • Python standard library http.server — the dashboard needs no web framework.

NASA Earthdata authentication (required for downloads, not for CMR search or --manifest-only). The downloader queries NASA CMR and pulls files over Earthdata Login. Provide credentials by either:

export EARTHDATA_USERNAME='your_username'
export EARTHDATA_PASSWORD='your_password'

or a ~/.netrc entry:

machine urs.earthdata.nasa.gov
  login your_username
  password your_password

Scripts

Most are plain scripts run with python icesat2/<script>.py; granule_footprints.py and polar_segments_dashboard.py are also wired as the odsl-icesat2-footprints and odsl-icesat2-dashboard console entry points.

Script Purpose Key flags
download_icesat2.py (console: odsl-icesat2-download) Search CMR, write manifests, download ATL07/ATL10 HDF5 via the shared common.sync engine. --products {ATL07,ATL10} --version (default 007, or latest) --start --end --bbox W S E N --region {01,02} --provider (default NSIDC_CPRD) --out-dir --limit --manifest-only --overwrite, plus shared --dry-run --jobs --verify {none,size,checksum} --retries
thin_icesat2.py Copy granules keeping only curated SSH/sea-ice vars. --input-dir (default /spray/icesat2) --output-dir (default /spray/icesat2/thin) --log-file --products {ATL07,ATL10} --delete-original
subset_polar_segments.py Read segment groups, keep ±55° polar rows, drop land, write Parquet/CSV. positional inputs (files/dirs, default /spray/icesat2) --out-dir --lat-threshold (default 55.0) --format {parquet,csv} --overwrite --limit
granule_footprints.py (odsl-icesat2-footprints) Build a per-granule bbox+time index over the polar-segments Parquet for fast spatial colocation; query with paths_overlapping. --root (default /spray/icesat2/polar-segments) --products --out --rebuild
polar_segments_dashboard.py (odsl-icesat2-dashboard) Local HTTP dashboard over the Parquet subsets. --data-dir (default /spray/icesat2/polar-segments) --products (e.g. ATL10, default all) --host (default 127.0.0.1) --port (default 8050, 0 picks a free port) --open (launch browser)

Footprint index for colocation

granule_footprints.py reads just latitude/longitude/delta_time from each polar-segments Parquet and records the granule's bounding box and UTC time span (delta_time is seconds since the 2018-01-01 ATLAS epoch). The colocate icesat2 provider uses this index to prune granules to a region/time window before reading any file. It is mtime-aware and incremental, so re-running after subset_polar_segments.py processes more granules re-scans only new files. Coverage tracks whatever has been subset to polar-segments/ (ATL07 today; run the subsetter on ATL10 to extend it).

odsl-icesat2-footprints                 # -> /spray/icesat2/footprints/…parquet

from icesat2.granule_footprints import load_footprint_index, paths_overlapping
fp = load_footprint_index()
paths_overlapping(fp, (-40, 70, 40, 85))   # granules crossing an Arctic box

A notebook-native variant, polar_segments_dashboard.ipynb, mirrors the web dashboard using ipywidgets, Plotly, pandas, and pyarrow.

Data layout

Data lives under /spray/icesat2 (release 007), the default root for every stage:

/spray/icesat2/
├── ATL07/007/YYYY/MM/DD/*.h5        # raw sea ice height granules
├── ATL10/007/YYYY/MM/DD/*.h5        # raw sea ice freeboard granules
├── thin/ATL07/007/YYYY/MM/DD/*.h5   # thinned copies (mirror layout)
├── thin/ATL10/007/YYYY/MM/DD/*.h5
├── polar-segments/…/*.parquet       # subset tables (mirror layout, .h5→.parquet)
├── polar-segments/subset_summary.csv
├── catalog/icesat2_atl07_atl10_manifest.json
├── catalog/icesat2_atl07_atl10_manifest.csv
├── thin_log.txt                     # restart log for thin_icesat2.py
└── thin_run.log

HDF5 group structure. Data is organized per beam. The six beams are:

gt1l  gt1r  gt2l  gt2r  gt3l  gt3r

Within each beam the per-segment data lives in a product-specific group:

  • ATL07 → sea_ice_segments (70 m) and sea_ice_segments_10m (10 m), each with heights/, geophysical/, geolocation/, and stats/ subgroups.
  • ATL10 → freeboard_segment (with heights/, geophysical/ subgroups), plus leads/ and reference_surface_section/.

Release-007 group name. ATL10 freeboard is in freeboard_segment. Older releases named this freeboard_beam_segment, which does not exist in v007 and yields zero rows if used. ATL07 uses sea_ice_segments.

Usage

The downloader is the console script odsl-icesat2-download (or python icesat2/download_icesat2.py). Download ATL07 + ATL10 for a short Arctic window (bbox is west south east north):

odsl-icesat2-download \
    --start 2024-01-01 --end 2024-01-03 \
    --bbox -180 60 180 90 \
    --out-dir /spray/icesat2

Preview the granule selection (writes the manifest, transfers nothing), then pull it with 4 parallel workers and verify each file's size against the server:

odsl-icesat2-download --start 2024-01-01 --end 2024-01-03 --bbox -180 60 180 90 --dry-run
odsl-icesat2-download --start 2024-01-01 --end 2024-01-03 --bbox -180 60 180 90 --jobs 4 --verify size

Write manifests only, no file download (no credentials needed):

odsl-icesat2-download \
    --products ATL07 ATL10 \
    --start 2024-01-01 --end 2024-01-01 \
    --manifest-only --out-dir tmp/icesat2_manifest

Thin all downloaded granules into compact copies:

python icesat2/thin_icesat2.py \
    --input-dir /spray/icesat2 \
    --output-dir /spray/icesat2/thin

Subset ATL07/ATL10 to polar sea-ice segments as Parquet:

python icesat2/subset_polar_segments.py \
    /spray/icesat2 \
    --out-dir /spray/icesat2/polar-segments \
    --format parquet

Serve the dashboard, then open http://127.0.0.1:8050:

python icesat2/polar_segments_dashboard.py \
    --data-dir /spray/icesat2/polar-segments --port 8050

Conventions & gotchas

  • Release-007 group names. Use freeboard_segment (ATL10) and sea_ice_segments (ATL07); see the note above. thin_icesat2.py keeps a curated variable list (ATL07_KEEP / ATL10_KEEP) and prints a loud WARNING: kept 0 datasets if a granule matches none of them — usually a sign of a wrong group path for the release in hand.
  • Bounding boxes are west south east north in degrees, longitudes in [-180, 180]; the downloader validates ranges and that south <= north.
  • Idempotency / restart-safety. The downloader skips existing files unless --overwrite, writing to a .<name>.part temp file first. thin_icesat2.py tracks finished files in thin_log.txt and skips existing outputs, so it can be re-run after an interruption. subset_polar_segments.py skips existing outputs unless --overwrite.
  • Download safety guard. The downloader refuses to run unconstrained: pass at least one of --start/--end, --limit, or --manifest-only.
  • Land filtering is conservative. The subsetter removes rows only when a surface_type flag identifies pure land; if no such flag exists in a granule it keeps the latitude-filtered rows and reports land_mask_found = false in subset_summary.csv.
  • Data roots default to /spray/icesat2 constants but every stage accepts a CLI flag to override them. Data files are not committed to the repo.
  • ../README.md — repository-wide conventions, data roots, and the ICESat-2 release-007 group-name gotcha.