Overview

insitu_wind normalizes downloaded in-situ / platform wind observations from several archives into a single tabular schema stored as Parquet, then plots the resulting station/platform locations on a global map. It ingests four source families:

Source family Origin Key input vars
PMEL Saildrone NOAA PMEL ERDDAP Saildrone NetCDF UWND_MEAN, VWND_MEAN, WIND_SPEED_MEAN, WIND_FROM_MEAN
OOI METBK OOI ERDDAP surface-mooring NetCDF eastward_wind, northward_wind (+ QARTOD flags)
TAO/RAMA/PIRATA PMEL ERDDAP pmelTaoDyW NetCDF WU_422, WV_423
NDBC NDBC stdmet / cwind gzipped text WDIR, WSPD, GST → components

All records are unified to a common column set (see COMMON_COLUMNS in unify_wind_data_insitu.py) keyed on normalized u10/v10. No neutral-10 m height correction is applied: u10/v10 are left null and the original component/speed/direction fields are preserved, with height_adjustment recording that fact. Only records at/after START_TIME = 2023-01-01T00:00:00Z are kept.

Installation & dependencies

Beyond the always-installed core (numpy, pandas, xarray, netCDF4), this package needs:

  • insitu extra → pyarrow (Parquet read/write). Required by both scripts.
  • plot extra → cartopy (+ matplotlib) for the location map only.
python -m pip install -e ".[insitu,plot]"   # or ".[all]"

Scripts

Script Console entry point Purpose Key flags
unify_wind_data_insitu.py odsl-insitu-wind-unify Source files → unified Parquet + manifest --input-root, --output-root, --source, --max-files
plot_insitu_wind_locations.py odsl-insitu-wind-plot Map unified locations (Cartopy Robinson) --input-root, --output, --dpi
station_footprints.py odsl-wind-footprints Station bbox/time parquet for colocate --root, --out, --max-files

Flag details:

  • unify --input-root (default /spray/wind/insitu) — expects subdirs pmel_erddap/files_2023plus/, ooi_erddap/wind_2023plus/, tao_rama_pirata/, ndbc/{stdmet,cwind}/.
  • unify --output-root (default /spray/wind/insitu/unified).
  • unify --source — process one family or all (default all).
  • unify --max-files N — debug cap for the PMEL/NDBC file walks only.
  • plot --input-root (default /spray/wind/insitu/unified); --output (default <input-root>/insitu_wind_locations.png); --dpi (default 220).

Usage

odsl-insitu-wind-unify
odsl-insitu-wind-unify --source ndbc --max-files 50
odsl-insitu-wind-plot --dpi 300

Hyphenated filenames remain as shims. Unify/plot default to /spray/wind/insitu[/unified]. Footprints default to /spray/wind/insitu/footprints/insitu_wind_stations.parquet (odsl-colocate-refresh --targets wind). The 10 m wind correction is not applied (u10 / v10 stay null).

Data layout / outputs

Under the unify --output-root:

<output-root>/
├── pmel_saildrone/<datasetID>_partNNNN.parquet   # batched at ~250k rows
├── ooi_metbk/<datasetID>.parquet
├── tao_rama_pirata/pmelTaoDyW.parquet
├── ndbc/<stdmet|cwind>_partNNNN.parquet          # batched at ~500k rows
├── unified_manifest.csv                          # file, rows, size_bytes
└── unified_summary.json                          # totals, columns, key vars

Parquet is written with zstd compression. The plot reads back all *.parquet recursively: Saildrone tracks are thinned to one point per platform per 2-day bin; moorings/fixed sites get one point per platform-or-station (or per rounded coordinate); rows without lat/lon are counted and omitted (mostly NDBC).

Conventions & gotchas

  • CLI names. Prefer odsl-insitu-wind-unify / odsl-insitu-wind-plot / odsl-wind-footprints. Underscored modules are importable; hyphenated filenames are compatibility shims.
  • u10/v10 are intentionally null. Sources are not height-corrected to neutral 10 m; consumers should read the original_* component/speed/direction columns and honor height_adjustment / original_direction_convention.
  • NDBC coordinates are absent. Station lat/lon and sensor height are not in the raw text files, so those rows carry NaN locations and drop out of the map.
  • Batching. Large families are flushed to _partNNNN.parquet files as row thresholds are hit (PMEL 250k, NDBC 500k), so a single dataset may span several parts.
  • Not part of common.*. Unlike drifter/glider, these scripts have no repo-root bootstrap and do not import shared helpers.
  • ../common — shared regridding/utilities (not used here, but the repo's shared package).
  • ../README.md — repo-wide conventions (data roots, extras, hyphenated scripts, [0,360)↔[-180,180) longitude normalization).