Download¶

This notebook demonstrates downloading all required products: 0. Burst Database - Frame geometries

1. DISP-S1 - Displacement products
2. DISP-S1-STATIC - DISP static layers DEM, ENU LOS unit vector
3. OPERA-TROPO-ZENITH - Atmospheric delay
4. UNR GNSS - Ground truth data
5. S1 Burst Bounds - S1 burst bounds used for DISP-S1

Each section shows both Python API and CLI usage.

In [1]:

from datetime import datetime
from pathlib import Path

from cal_disp.download import (
    download_disp,
    download_tropo,
    download_unr_grid,
    generate_s1_burst_tiles,
)
from cal_disp.download.utils import extract_sensing_times_from_file

from datetime import datetime from pathlib import Path from cal_disp.download import ( download_disp, download_tropo, download_unr_grid, generate_s1_burst_tiles, ) from cal_disp.download.utils import extract_sensing_times_from_file

/u/aurora-r0/govorcin/miniconda/miniforge/envs/my-cal-disp/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
  from .autonotebook import tqdm as notebook_tqdm

Find OPERA DISP-S1 frame id¶

In [2]:

from folium import LayerControl
from opera_utils import get_frame_geojson

disp_df = get_frame_geojson(as_geodataframe=True)

# Filter to OPERA DISP geogrphical scope
disp_df = disp_df[disp_df.is_north_america]
m = disp_df[disp_df.orbit_pass.str.startswith('D')].explore(name='DSC')
m = disp_df[disp_df.orbit_pass.str.startswith('A')].explore(name='ASC', m=m)
LayerControl().add_to(m)
m

from folium import LayerControl from opera_utils import get_frame_geojson disp_df = get_frame_geojson(as_geodataframe=True) # Filter to OPERA DISP geogrphical scope disp_df = disp_df[disp_df.is_north_america] m = disp_df[disp_df.orbit_pass.str.startswith('D')].explore(name='DSC') m = disp_df[disp_df.orbit_pass.str.startswith('A')].explore(name='ASC', m=m) LayerControl().add_to(m) m

Out[2]:

Make this Notebook Trusted to load map: File -> Trust Notebook

Configuration¶

In [3]:

# Frame and dates
frame_id = 8882
start_date = datetime(2022, 7, 22)
end_date = datetime(2022, 7, 22)

# Base data directory
data_dir = Path("data")
data_dir.mkdir(exist_ok=True)

# Output directories for each product type
disp_dir = data_dir / "disp"
tropo_dir = data_dir / "tropo"
unr_dir = data_dir / "unr"
tiles_dir = data_dir / "s1_burst_bounds"

print(f"Frame ID: {frame_id}")
print(f"Date range: {start_date.date()} → {end_date.date()}")
print(f"Base directory: {data_dir.absolute()}")

# Frame and dates frame_id = 8882 start_date = datetime(2022, 7, 22) end_date = datetime(2022, 7, 22) # Base data directory data_dir = Path("data") data_dir.mkdir(exist_ok=True) # Output directories for each product type disp_dir = data_dir / "disp" tropo_dir = data_dir / "tropo" unr_dir = data_dir / "unr" tiles_dir = data_dir / "s1_burst_bounds" print(f"Frame ID: {frame_id}") print(f"Date range: {start_date.date()} → {end_date.date()}") print(f"Base directory: {data_dir.absolute()}")

Frame ID: 8882
Date range: 2022-07-22 → 2022-07-22
Base directory: /u/aurora-r0/govorcin/01_OPERA/CAL/cal-disp/data

1. Download DISP-S1 Products¶

Download displacement products for a specific frame and time range.

Python API¶

In [4]:

# Download products
download_disp(
    frame_id=frame_id,
    output_dir=disp_dir,
    start=start_date,
    end=end_date,
    num_workers=4,
)

print("Download complete!")
disp_files = sorted((disp_dir).glob("*.nc"))
print(f"Downloaded {len(disp_files)} DISP products\n")

# Show first 5 files
for f in disp_files[:5]:
    print(f"  {f.name}")

# Download products download_disp( frame_id=frame_id, output_dir=disp_dir, start=start_date, end=end_date, num_workers=4, ) print("Download complete!") disp_files = sorted((disp_dir).glob("*.nc")) print(f"Downloaded {len(disp_files)} DISP products\n") # Show first 5 files for f in disp_files[:5]: print(f" {f.name}")

100%|██████████| 1/1 [03:01<00:00, 181.72s/it]

Download complete!
Downloaded 7 DISP products

  OPERA_L3_DISP-S1_IW_F08882_VV_20220111T002651Z_20220722T002657Z_v1.0_20251027T005420Z.nc
  OPERA_L3_DISP-S1_IW_F08882_VV_20230717T002702Z_20240113T002702Z_v1.0_20251028T001623Z.nc
  OPERA_L3_DISP-S1_IW_F08882_VV_20230717T002702Z_20240125T002701Z_v1.0_20251028T001623Z.nc
  OPERA_L3_DISP-S1_IW_F08882_VV_20240125T002701Z_20240218T002701Z_v1.0_20251028T074515Z.nc
  OPERA_L3_DISP-S1_IW_F08882_VV_20240125T002701Z_20240301T002701Z_v1.0_20251028T074515Z.nc

CLI Equivalent¶

cal-disp download disp-s1 \
  --frame-id 8882 \
  -s 2022-07-22 \
  -e 2022-07-22 \
  -o data/disp_input

2. Download DISP-S1_STATIC Data¶

Download OPERA DISP-S1 STATIC data for the frame bounds:

• OPERA_L3_DISP-S1-STATIC_F08882_20140403_S1A_v1.0_dem.tif
• OPERA_L3_DISP-S1-STATIC_F08882_20140403_S1A_v1.0_layover_shadow_mask.tif
• OPERA_L3_DISP-S1-STATIC_F08882_20140403_S1A_v1.0_line_of_sight_enu.tif

TBD

3. Download TROPO-ZENITH Products¶

Download tropospheric products for the DISP acquisition dates.

Python API¶

In [6]:

# Use first DISP product to determine bounds and dates
disp_path = disp_files[0]
# Extract reference and secondary date from OPERA DISP-S1 product filename
sensing_times = extract_sensing_times_from_file(disp_files[0])

# Download TROPO products with interpolation
download_tropo(
    output_dir=tropo_dir,
    disp_times=sensing_times,
    num_workers=2,
    interp=True,  # Download products for interpolation
)

tropo_files = list((tropo_dir).glob("*.nc"))
print(f"Downloaded {len(tropo_files)} troposphere files")
for f in tropo_files:
    print(f"  {f.name}")

# Use first DISP product to determine bounds and dates disp_path = disp_files[0] # Extract reference and secondary date from OPERA DISP-S1 product filename sensing_times = extract_sensing_times_from_file(disp_files[0]) # Download TROPO products with interpolation download_tropo( output_dir=tropo_dir, disp_times=sensing_times, num_workers=2, interp=True, # Download products for interpolation ) tropo_files = list((tropo_dir).glob("*.nc")) print(f"Downloaded {len(tropo_files)} troposphere files") for f in tropo_files: print(f" {f.name}")

/u/aurora-r0/govorcin/miniconda/miniforge/envs/my-cal-disp/lib/python3.11/site-packages/asf_search/download/download.py:66: UserWarning: File already exists, skipping download: data/tropo/OPERA_L4_TROPO-ZENITH_20220111T060000Z_20250923T231305Z_HRES_v1.0.nc
  warnings.warn(f'File already exists, skipping download: {os.path.join(path, filename)}')
/u/aurora-r0/govorcin/miniconda/miniforge/envs/my-cal-disp/lib/python3.11/site-packages/asf_search/download/download.py:66: UserWarning: File already exists, skipping download: data/tropo/OPERA_L4_TROPO-ZENITH_20220111T000000Z_20250923T224940Z_HRES_v1.0.nc
  warnings.warn(f'File already exists, skipping download: {os.path.join(path, filename)}')
/u/aurora-r0/govorcin/miniconda/miniforge/envs/my-cal-disp/lib/python3.11/site-packages/asf_search/download/download.py:66: UserWarning: File already exists, skipping download: data/tropo/OPERA_L4_TROPO-ZENITH_20220722T060000Z_20250923T230822Z_HRES_v1.0.nc
  warnings.warn(f'File already exists, skipping download: {os.path.join(path, filename)}')
/u/aurora-r0/govorcin/miniconda/miniforge/envs/my-cal-disp/lib/python3.11/site-packages/asf_search/download/download.py:66: UserWarning: File already exists, skipping download: data/tropo/OPERA_L4_TROPO-ZENITH_20220722T000000Z_20250923T233421Z_HRES_v1.0.nc
  warnings.warn(f'File already exists, skipping download: {os.path.join(path, filename)}')

Downloaded 6 troposphere files
  OPERA_L4_TROPO-ZENITH_20240113T000000Z_20250924T032011Z_HRES_v1.0.nc
  OPERA_L4_TROPO-ZENITH_20230717T000000Z_20250924T010719Z_HRES_v1.0.nc
  OPERA_L4_TROPO-ZENITH_20220111T060000Z_20250923T231305Z_HRES_v1.0.nc
  OPERA_L4_TROPO-ZENITH_20220111T000000Z_20250923T224940Z_HRES_v1.0.nc
  OPERA_L4_TROPO-ZENITH_20220722T000000Z_20250923T233421Z_HRES_v1.0.nc
  OPERA_L4_TROPO-ZENITH_20220722T060000Z_20250923T230822Z_HRES_v1.0.nc

CLI Equivalent¶

cal-disp download tropo \
  -i data/disp_input/OPERA_L3_DISP-S1_IW_F08882_VV_20220111T002651Z_20220722T002657Z_v1.0_20251027T005420Z.nc \
  -o data/tropo_input \
  --interp

4. Download UNR GNSS Data¶

Download UNR GNSS grid for the frame.

Python API¶

In [7]:

download_unr_grid(
        frame_id=frame_id,
        output_dir=unr_dir,
        start=None, # download whole record
        end=None,
        margin_deg=0.5, # in deg
    )

unr_files = list((unr_dir).glob("*.parquet"))
print(f"Downloaded {len(unr_files)} UNR station files")
for f in unr_files:
    print(f"  {f.name}")

download_unr_grid( frame_id=frame_id, output_dir=unr_dir, start=None, # download whole record end=None, margin_deg=0.5, # in deg ) unr_files = list((unr_dir).glob("*.parquet")) print(f"Downloaded {len(unr_files)} UNR station files") for f in unr_files: print(f" {f.name}")

Loading GPS data: 100%|██████████| 169/169 [00:09<00:00, 16.98it/s]

Downloaded 1 UNR station files
  unr_grid_frame8882.parquet

In [8]:

# Vizualize
import geopandas as gpd
import pandas as pd

df = pd.read_parquet(unr_files[0])
gdf = gpd.GeoDataFrame(df,
                       geometry=gpd.points_from_xy(x=df.lon,
                                                   y=df.lat),
                       crs=4326)

out = (
    gdf.groupby('id', as_index=False)
       .first()
       .set_crs(epsg=4326)
)

out.explore(column="east")

# Vizualize import geopandas as gpd import pandas as pd df = pd.read_parquet(unr_files[0]) gdf = gpd.GeoDataFrame(df, geometry=gpd.points_from_xy(x=df.lon, y=df.lat), crs=4326) out = ( gdf.groupby('id', as_index=False) .first() .set_crs(epsg=4326) ) out.explore(column="east")

Out[8]:

Make this Notebook Trusted to load map: File -> Trust Notebook

CLI Equivalent¶

cal-disp download unr \
  --frame-id 8882 \
  -o data/unr_input/

5. [OPTIONAL] Download Burst Bounds¶

Download the S1 burst boundary geometries for frame geometries.

NOTE: useful if calibration needs to take into account burst jumps casued by ionosphere

In [9]:

sensing_times = extract_sensing_times_from_file(disp_files[0])
sensing_times

for sensing_time in sensing_times:
    print(f"Downloading burst bounds for {sensing_time}")
    generate_s1_burst_tiles(
        frame_id=frame_id,
        sensing_time=sensing_time,
        output_dir=tiles_dir,
    )

bbounds = list((tiles_dir).glob("*.geojson"))
print(f"Downloaded {len(bbounds)} S1 burst geometry files")
for f in bbounds:
    print(f"  {f.name}")

sensing_times = extract_sensing_times_from_file(disp_files[0]) sensing_times for sensing_time in sensing_times: print(f"Downloading burst bounds for {sensing_time}") generate_s1_burst_tiles( frame_id=frame_id, sensing_time=sensing_time, output_dir=tiles_dir, ) bbounds = list((tiles_dir).glob("*.geojson")) print(f"Downloaded {len(bbounds)} S1 burst geometry files") for f in bbounds: print(f" {f.name}")

Downloading burst bounds for 2022-01-11 00:26:51

Processing CSLC bounds: 100%|██████████| 27/27 [03:51<00:00,  8.57s/it]

Downloading burst bounds for 2022-07-22 00:26:57

Processing CSLC bounds: 100%|██████████| 27/27 [03:46<00:00,  8.40s/it]

Downloaded 4 S1 burst geometry files
  2023-07-17_tiles.geojson
  2024-01-13_tiles.geojson
  2022-01-11_tiles.geojson
  2022-07-22_tiles.geojson

In [10]:

# Use the burst bounds to get burst overalp in the DISP merging strategy.
gpd.read_file(str(bbounds[0].resolve())).explore()

# Use the burst bounds to get burst overalp in the DISP merging strategy. gpd.read_file(str(bbounds[0].resolve())).explore()

Out[10]:

Make this Notebook Trusted to load map: File -> Trust Notebook

CLI Equivalent¶

cal-disp download burst-bounds \
  -i /u/aurora-r0/govorcin/01_OPERA/CAL/cal-disp/notebooks/disp_input/OPERA_L3_DISP-S1_IW_F08882_VV_20220111T002651Z_20220722T002657Z_v1.0_20251027T005420Z.nc \
  -o /u/aurora-r0/govorcin/01_OPERA/CAL/cal-disp/notebooks/tiles_input/

7. Download Summary¶

In [11]:

def get_dir_size(directory: Path) -> float:
    """Get directory size in GB."""
    if not directory.exists():
        return 0.0
    total = sum(f.stat().st_size for f in directory.rglob('*') if f.is_file())
    return total / 1024**3

print("Download Summary")
print("="*60)
print(f"Frame ID: {frame_id}")
print(f"Date range: {start_date.date()} → {end_date.date()}")
print("\nProducts downloaded:")
print(f"  ✓ DISP products: {len(list(disp_dir.glob('*.nc')))} files ({get_dir_size(disp_dir):.2f} GB)")
print(f"  ✓ UNR products: {len(list(unr_dir.glob('*.parquet')))} files ({get_dir_size(unr_dir):.2f} GB)")
print(f"  ✓ TROPO products: {len(list(tropo_dir.glob('*.nc')))} files ({get_dir_size(tropo_dir):.2f} GB)")
print(f"\nTotal size: {get_dir_size(data_dir):.2f} GB")
print(f"\nData directory: {data_dir.absolute()}")

def get_dir_size(directory: Path) -> float: """Get directory size in GB.""" if not directory.exists(): return 0.0 total = sum(f.stat().st_size for f in directory.rglob('*') if f.is_file()) return total / 1024**3 print("Download Summary") print("="*60) print(f"Frame ID: {frame_id}") print(f"Date range: {start_date.date()} → {end_date.date()}") print("\nProducts downloaded:") print(f" ✓ DISP products: {len(list(disp_dir.glob('*.nc')))} files ({get_dir_size(disp_dir):.2f} GB)") print(f" ✓ UNR products: {len(list(unr_dir.glob('*.parquet')))} files ({get_dir_size(unr_dir):.2f} GB)") print(f" ✓ TROPO products: {len(list(tropo_dir.glob('*.nc')))} files ({get_dir_size(tropo_dir):.2f} GB)") print(f"\nTotal size: {get_dir_size(data_dir):.2f} GB") print(f"\nData directory: {data_dir.absolute()}")

Download Summary
============================================================
Frame ID: 8882
Date range: 2022-07-22 → 2022-07-22

Products downloaded:
  ✓ DISP products: 7 files (3.62 GB)
  ✓ UNR products: 1 files (0.02 GB)
  ✓ TROPO products: 6 files (12.32 GB)

Total size: 26.89 GB

Data directory: /u/aurora-r0/govorcin/01_OPERA/CAL/cal-disp/data

Complete CLI Workflow¶

To download all products using CLI commands:

# Set variables
FRAME_ID=8882
START_DATE=2022-07-22
END_DATE=2022-07-22
DATA_DIR=data

# 1. Download burst database
cal-disp download burst-bounds -o ${DATA_DIR}/burst_db.sqlite

# 2. Download DISP products
cal-disp download disp-s1 \
  --frame-id ${FRAME_ID} \
  -s ${START_DATE} \
  -e ${END_DATE} \
  -o ${DATA_DIR}/disp_input

# 3. Download TROPO products (using first DISP product)
DISP_FILE=$(ls ${DATA_DIR}/disp_input/*.nc | head -1)
cal-disp download tropo \
  -i ${DISP_FILE} \
  -o ${DATA_DIR}/tropo_input \
  --interp

# 4. Download UNR GNSS data
cal-disp download unr \
  --frame-id ${FRAME_ID} \
  -o ${DATA_DIR}/unr_input/

# 5. (Optional) Download burst tiles
cal-disp download burst-bounds \
  -i ${DISP_FILE} \
  -o ${DATA_DIR}/tiles_input/

Next Steps¶

Now that all data is downloaded:

1. Prepare inputs - Use prep/ module to prepare data for calibration
2. Run calibration - Use core/ module to compute calibration
3. Create output - Generate CalProduct with results

See next notebook: 02_prep_and_calibrate.ipynb