diff --git a/docs/source/technical_tutorials/dps_tutorial/dps_stac_metadata.ipynb b/docs/source/technical_tutorials/dps_tutorial/dps_stac_metadata.ipynb new file mode 100644 index 00000000..d933f36d --- /dev/null +++ b/docs/source/technical_tutorials/dps_tutorial/dps_stac_metadata.ipynb @@ -0,0 +1,4059 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "f3e5e969-1724-4e5d-ba0a-075044f5a25f", + "metadata": {}, + "source": [ + "# Generating STAC metadata for DPS job outputs\n", + "\n", + "**Authors:** Henry Rodman (Development Seed)\n", + "\n", + "**Date:** January 6, 2026\n", + "\n", + "**Description:** DPS job outputs are automatically uploaded to S3 with a prefix that is determined by your username, the algorithm name, job identifier, and the time at which the job finishes. To improve accessibility of DPS output files, STAC metadata that are written alongside your normal job outputs are automatically ingested into a STAC and made accessible via a STAC API ([dps-stac.maap-project.org](dps-stac.maap-project.org)) and a titiler API ([titiler-dps-stac.maap-project.org](titiler-dps-stac.maap-project.org)). This tutorial will explain how to produce STAC metadata as part of a DPS algorithm and how to use the APIs to access and visualize the outputs." + ] + }, + { + "cell_type": "markdown", + "id": "b0d01ef8-ae95-46dd-9238-d6ee3b7848ba", + "metadata": {}, + "source": [ + "## How to write STAC metadata\n", + "\n", + "The last step in your DPS algorithm script is usually to write the output files in a cloud-optimized format. To take advantage of the DPS STAC capability you will need to write some STAC metadata to describe the output files after they are created.\n", + "\n", + "- algorithm that writes a COG\n", + "- describe importance of relative hrefs\n", + "- show how to do it when writing a file to output/file.tif\n", + "- create Catalog, Item, write to output/catalog.json" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "88475b3c-56b2-4ed5-af04-2a4cf8a507ab", + "metadata": {}, + "outputs": [], + "source": [ + "from datetime import datetime, UTC\n", + "from pathlib import Path\n", + "\n", + "import rio_stac\n", + "import rioxarray\n", + "from pystac import Asset, Catalog, CatalogType, MediaType" + ] + }, + { + "cell_type": "markdown", + "id": "13041678-d683-4c24-844b-8de4e68e9f1d", + "metadata": {}, + "source": [ + "This algorithm creates an xarray DataArray from an input COG and writes it to the output directory for the DPS job." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "4e1d8b3e-5473-4ab8-bc19-8931751250d3", + "metadata": {}, + "outputs": [], + "source": [ + "da = rioxarray.open_rasterio(\n", + " \"s3://maap-ops-workspace/henrydevseed/dps_output/HLSCloudFreeTemporalMosaic/v0.1.1/mic-check/2025/11/06/02/55/51/893808/red.tif\"\n", + ")\n", + "\n", + "# this is usually defined in your algorithm's run script\n", + "output_dir = Path(\"/tmp/output\")\n", + "output_dir.mkdir(exist_ok=True)\n", + "\n", + "output_file = \"demo.tif\"\n", + "da.rio.to_raster(\n", + " output_dir / output_file,\n", + " driver=\"COG\",\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "c482353f-7517-4378-93d7-f9b99d0790e6", + "metadata": {}, + "source": [ + "### Generating a STAC item\n", + "\n", + "There are many ways to create STAC item metadata, but the easiest way to get going is with [rio-stac](https://github.com/developmentseed/rio-stac). rio-stac can generate a boilerplate STAC item with only a path to an input raster file.\n", + "\n", + "If you are generating vector files (e.g. geoparquet, geopackage, etc) you will need to use a different approach." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "867b2499-c37a-43b5-9177-322abb71bb0d", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + "
\n", + "
\n", + " <Item id=demo.tif>\n", + "
\n", + "\n", + "
" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rio_stac.create_stac_item(\n", + " output_dir / output_file\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "d502cdc1-68fe-4dce-839a-307714761f84", + "metadata": {}, + "source": [ + "However we need to provide some additional arguments in order to make this STAC item useful. Most importantly the **item ID** which needs to be unique within this job's STAC collection. If we run multiple jobs that create STAC items with the same ID they will overwrite eachother in the STAC database.\n", + "\n", + "Let's say that the algorithm has two input parameters that determine a unique job: spatial extent and year/month. I will generate the STAC item ID using those factors." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "6780f9eb-a69d-4970-8759-4464dc3221e3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2025-05__-245760_2211840_0_2457600\n" + ] + } + ], + "source": [ + "bbox_str = \"_\".join(str(int(x)) for x in da.rio.bounds())\n", + "item_id = f\"2025-05__{bbox_str}\"\n", + "\n", + "print(item_id)" + ] + }, + { + "cell_type": "markdown", + "id": "b3c71976-8559-400f-bbbf-3ae664bca727", + "metadata": {}, + "source": [ + "Now create the item with the unique ID and some other enriching attributes. You can add anything you want to the `properties` dict! Use this to include attributes that you might want to use for filtering STAC items later.\n", + "\n", + "Please use the `.validate()` method when generating STAC metadata because it will help you catch small STAC metadata errors early rather than causing failures during the STAC item ingestion process." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "d410e9ae-0181-4247-a458-af3bbc40f5c2", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + "
\n", + "
\n", + " <Item id=2025-05__-245760_2211840_0_2457600>\n", + "
\n", + "\n", + "
" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 16, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "item = rio_stac.create_stac_item(\n", + " output_dir / output_file,\n", + " id=item_id,\n", + " input_datetime=datetime(2025, 5, 1, tzinfo=UTC),\n", + " asset_media_type=MediaType.COG, # advertise that the file is a COG!\n", + " asset_roles=[\"data\"],\n", + " with_proj=True, # let rio-stac add info about the file's projection\n", + " properties={\n", + " \"grade\": \"A+\", # you can add any relevant attributes/tags here!\n", + " }\n", + ")\n", + "item.set_self_href(f\"{output_dir}/item.json\")\n", + "item.validate()\n", + "item" + ] + }, + { + "cell_type": "markdown", + "id": "2daf1aa1-fccd-4707-9e2f-f6fec14c5f77", + "metadata": {}, + "source": [ + "If your algorithm produces more than one file you will (in most cases) want to add it as an additional asset in the same STAC item." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "a15e7ffa-7b89-47c7-b992-5939440e9ada", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + "
\n", + "
\n", + " <Item id=2025-05__-245760_2211840_0_2457600>\n", + "
\n", + "\n", + "
" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 17, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "output_file_2 = \"additional.tif\"\n", + "da.rio.to_raster(\n", + " output_dir / output_file_2,\n", + " driver=\"COG\",\n", + ")\n", + "\n", + "additional_asset = Asset(\n", + " href=output_dir / output_file_2,\n", + " media_type=MediaType.COG,\n", + " roles=[\"data\"],\n", + ")\n", + "\n", + "item.assets.update(\n", + " {\"extra\": additional_asset}\n", + ")\n", + "item.validate()\n", + "item" + ] + }, + { + "cell_type": "markdown", + "id": "764320cc-5312-49ca-b6bc-1ba1570f6514", + "metadata": {}, + "source": [ + "### Exporting the STAC metadata\n", + "\n", + "Now that we have a STAC item that describes the job's output files we need to write it to the output directory that gets uploaded to S3. To get the STAC item to be detected by the DPS STAC ingestion process we simply need to create a STAC catalog.json file that references the item. During this step the most imporant thing is to make sure the asset `hrefs` are written as **relative** (not absolute) links. This will ensure that the links to the actual output files will still be valid after everything is uploaded to S3." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "09eb9753-326c-4f80-8240-56b178a713f8", + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "catalog = Catalog(\n", + " id=\"DPS\",\n", + " description=\"DPS output STAC items\",\n", + " catalog_type=CatalogType.SELF_CONTAINED,\n", + ")\n", + "catalog.set_self_href(f\"{output_dir}/catalog.json\")\n", + "\n", + "# add the item to the catalog\n", + "catalog.add_item(item)\n", + "\n", + "# IMPORTANT: ensure all hrefs are written as relative to the root\n", + "catalog.make_all_asset_hrefs_relative()\n", + "\n", + "# run validate_all to check that the STAC metadata is valid\n", + "catalog.validate_all()\n", + "\n", + "# write the STAC metadata (catalog.json and all items) to the output directory\n", + "catalog.normalize_and_save(root_href=str(output_dir))" + ] + }, + { + "cell_type": "markdown", + "id": "d396172a-41b7-42c9-8015-23bb0ec2bb9f", + "metadata": {}, + "source": [ + "The catalog object serves as a container for your STAC items. There is a process that listens for `catalog.json` files that get uploaded to the DPS output folders in S3 so this is the key to getting your STAC metadata loaded into the DPS STAC!\n", + "\n", + "When you create the catalog using `pystac` and add the item to it, both will be written to the output folder when you run `catalog.normalize_and_save(root_href=str(output_dir))`." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "c3d9d33a-8bd3-4d92-9b5e-da312f3e7387", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'type': 'Catalog',\n", + " 'id': 'DPS',\n", + " 'stac_version': '1.1.0',\n", + " 'description': 'DPS output STAC items',\n", + " 'links': [{'rel': 'root',\n", + " 'href': './catalog.json',\n", + " 'type': 'application/json'},\n", + " {'rel': 'item',\n", + " 'href': './2025-05__-245760_2211840_0_2457600/2025-05__-245760_2211840_0_2457600.json',\n", + " 'type': 'application/geo+json'},\n", + " {'rel': 'self',\n", + " 'href': '/tmp/output/catalog.json',\n", + " 'type': 'application/json'}]}" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "catalog.to_dict()" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "562c982f-8344-408a-bb81-fb53779cc358", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "\n", + "\n", + "\n", + "
\n", + "
\n", + " <Item id=2025-05__-245760_2211840_0_2457600>\n", + "
\n", + "\n", + "
" + ], + "text/plain": [ + "" + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "next(catalog.get_all_items())" + ] + }, + { + "cell_type": "markdown", + "id": "a7a60435-b836-4594-8634-97fb15506b08", + "metadata": {}, + "source": [ + "### Full example algorithm script\n", + "\n", + "```python\n", + "import argparse\n", + "from datetime import datetime\n", + "from pathlib import Path\n", + "\n", + "import rio_stac\n", + "import rioxarray\n", + "from pystac import Asset, Catalog, CatalogType, MediaType\n", + "\n", + "def run(bbox: tuple[float, float, float, float], timestamp: datetime, output_dir: Path) -> None:\n", + "\n", + " # generate your data\n", + " da = rioxarray.open_rasterio(\n", + " \"s3://maap-ops-workspace/henrydevseed/dps_output/HLSCloudFreeTemporalMosaic/v0.1.1/mic-check/2025/11/06/02/55/51/893808/red.tif\"\n", + " )\n", + "\n", + " # write a raster to the output directory\n", + " output_dir.mkdir(exist_ok=True)\n", + " output_file = \"demo.tif\"\n", + " \n", + " da.rio.to_raster(\n", + " output_dir / output_file,\n", + " driver=\"COG\",\n", + " )\n", + "\n", + " # generate the STAC item ID\n", + " bbox_str = \"_\".join(str(int(x)) for x in da.rio.bounds())\n", + " datetime_str = timestamp.strftime(\"%Y-%m-%d\")\n", + " item_id = f\"{datetime_str}__{bbox_str}\"\n", + "\n", + " # use rio_stac.create_stac_item to easily generate the STAC item\n", + " item = rio_stac.create_stac_item(\n", + " output_dir / output_file,\n", + " id=item_id,\n", + " input_datetime=datetime(2025, 5, 1, tzinfo=UTC),\n", + " asset_media_type=MediaType.COG, # advertise that the file is a COG!\n", + " asset_roles=[\"data\"],\n", + " with_proj=True, # let rio-stac add info about the file's projection\n", + " properties={\n", + " \"grade\": \"A+\", # you can add any relevant attributes/tags here!\n", + " }\n", + " )\n", + "\n", + " # if you have other assets that you created as part of your job\n", + " # add them to the item like this\n", + " output_file_2 = \"additional.tif\"\n", + " da.rio.to_raster(\n", + " output_dir / output_file_2,\n", + " driver=\"COG\",\n", + " )\n", + " \n", + " additional_asset = Asset(\n", + " href=output_dir / output_file_2,\n", + " description=\"another asset\",\n", + " media_type=MediaType.COG,\n", + " roles=[\"data\"],\n", + " )\n", + "\n", + " # update the item's asset dict with the new asset\n", + " item.assets.update(\n", + " {\"other\": additional_asset}\n", + " )\n", + "\n", + " # set the href for this item in the output directory\n", + " item.set_self_href(f\"{output_dir}/item.json\")\n", + "\n", + " # IMPORTANT: validate the STAC item to avoid heartburn later\n", + " item.validate()\n", + "\n", + " # create a STAC catalog to house your item\n", + " catalog = Catalog(\n", + " id=\"DPS\",\n", + " description=\"DPS output STAC items\",\n", + " catalog_type=CatalogType.SELF_CONTAINED,\n", + " )\n", + " catalog.set_self_href(f\"{output_dir}/catalog.json\")\n", + " \n", + " # add the item to the catalog\n", + " catalog.add_item(item)\n", + " \n", + " # IMPORTANT: ensure all hrefs are written as relative to the root\n", + " catalog.make_all_asset_hrefs_relative()\n", + " \n", + " # run validate_all to check that the STAC metadata is valid\n", + " catalog.validate_all()\n", + " \n", + " # write the STAC metadata (catalog.json and all items) to the output directory\n", + " catalog.normalize_and_save(root_href=str(output_dir))\n", + "\n", + "\n", + "if __name__ == \"__main__\":\n", + " parse = argparse.ArgumentParser(\n", + " description=\"Example algorithm that produces a STAC item for the job results\"\n", + " )\n", + " parse.add_argument(\n", + " \"--bbox\",\n", + " help=\"bounding box (xmin, ymin, xmax, ymax)\",\n", + " required=True,\n", + " nargs=4,\n", + " type=float,\n", + " metavar=(\"xmin\", \"ymin\", \"xmax\", \"ymax\"),\n", + " )\n", + " parse.add_argument(\n", + " \"--timestamp\",\n", + " help=\"timestamp to use for the STAC item\",\n", + " required=True,\n", + " type=datetime,\n", + " )\n", + " parse.add_argument(\n", + " \"--output_dir\", help=\"Directory in which to save output\", required=True\n", + " )\n", + "\n", + " args = parse.parse_args()\n", + "\n", + " run(**vars(args))\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "e3ec98f5-c9ed-4e8d-9137-c5fe14f531e8", + "metadata": {}, + "source": [ + "## What happens next\n", + "\n", + "When your DPS job runs successfully, the contents of the output directory will be uploaded to S3. There is a process that listens for `catalog.json` files that are uploaded to DPS output locations and processes the items that are included in them.\n", + "\n", + "The items are posted to the DPS STAC and are associated with a STAC collection that is defined by several job input parameters:\n", + "\n", + "- your MAAP username\n", + "- the algorithm name\n", + "- the algorithm version/tag\n", + "- the job identifier/tag\n", + "\n", + "So, for example, if I (`henrydevseed`) submit several jobs to version `v0.1` of the `BestAlgorithm` with a job identifier `final`, the STAC items that are produced will be added to a collection with ID `henrydevseed__BestAlgorithm__v0.1__final`.\n", + "\n", + "If you change any of those parameters (username, algorithm name, job identifier, version, etc), STAC items will be associated with a different collection!\n", + "\n", + "In the future there will be a way for you to specify the collection ID directly but for now the collection ID will be determined by these job parameters.\n", + "\n", + "The STAC metadata will be made available in a publicly accessible STAC API but access to the actual files does not change." + ] + }, + { + "cell_type": "markdown", + "id": "ef03d7e7-e344-43f7-896d-24168337e81f", + "metadata": {}, + "source": [ + "## How to use the STAC metadata\n", + "\n", + "- search for files using pystac-client\n", + "- visualize using the titiler API" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8ded2752-5006-4413-8505-5f02ee283ad6", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.13.0" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/requirements.txt b/requirements.txt index df27109a..d4a0c7f6 100644 --- a/requirements.txt +++ b/requirements.txt @@ -10,4 +10,6 @@ h5py folium pandas rioxarray -sphinx-copybutton \ No newline at end of file +sphinx-copybutton +rio-stac +pystac \ No newline at end of file