Add option to automatically update collection extents and summaries based on ingested items #106

mishaschwartz · 2025-07-30T02:10:25Z

Adds a new CLI option to automatically update a collection's extents and/or summaries based on the ingested items that are added.

Note that this only applies if the --update option is also set.

Also adds a warning if an item's bounding box is outside of the bounds set by the STAC spec (-180->180, -90->90)

fmigneault

Changes look fine.

Comments mostly for considering extended use cases that could be addressed or not, or added at a later time depending on criticality.

fmigneault · 2025-08-14T01:17:12Z

STACpopulator/cli.py

+            "--update-collection",
+            choices=get_args(inspect.signature(STACpopulatorBase.__init__).parameters["update_collection"].annotation),
+            default="none",
+            help="Update collection information based on new items created or updated by this populator. "
+            "Only applies if --update is also set.",


This is fine, but just out of curiosity, would it make sense to allow updating only the collection, such as recomputing the extent/summaries without overriding other Items metadata that could have been pushed at another time (not within the same populator call)?

If --update is used, does it imply --update-collection as well by default if it is not provided explicitly? I would expect summaries/extents to be computed automatically anyway without having to specify the option. If such, what more does the option do since --update would be required for it to take effect. If not, wouldn't that potentially introduce erroneous summaries/extents?

Yes I think it would make sense to have a "recompute summaries and extents" option but I'm feeling like that's should be it's own standalone thing since that would involve crawling through the existing items within that catalog.

If --update is used, does it imply --update-collection as well by default if it is not provided explicitly?

No, but maybe it should. I kept the semantics of --update as it was before since I wanted to ensure backwards compatibility.

If such, what more does the option do

--update-collection updates the summaries and extents based on any new/updated items in the collection. --update actually tells the populator to update the collection on the STAC-API if it already exists.

wouldn't that potentially introduce erroneous summaries/extents?

It definitely would if the user didn't update those summaries/extents by some other means (which was the assumption before). Again, I did this to ensure backwards compatibility with the previous processes but I'm happy to break things if everyone is happy with that.

OK, I agree about keeping them separate options to allow flexibility.

Do you think it would be more common that --update --update-collection all is desired or --update --update-collection none?

My guess would be that, unless we explicitly want to handle a special case where --update-collection all would introduce invalid metadata from partial Item references, we would always want everything to be synced. So, "all" would be a more appropriate default?

Yeah I agree. That would break backwards compatibility though... are you ok if we do that? If so I can make some additional changes.

Yes. Since we are already accumulating a few changes and new features, we might as well do it with the most common expectation.

STACpopulator/populator_base.py

fmigneault · 2025-08-14T01:35:33Z

STACpopulator/populator_base.py

+        else:
+            item_interval = [stac_item["properties"][prop] for prop in ("start_datetime", "end_datetime")]
+        collection_intervals = self._collection["extent"]["temporal"]["interval"]
+        if collection_intervals:


If none was already provided, the ones found with items are not applied?

I think there's supposed to be an else in there but I don't know if you can have a valid collection without at least one interval value. I'll add in the else condition just in case.

I think so too, but could it be possible that the Collection data is purposely partially defined in this case have it being defined entirely from the computed STAC Items values?

I don't think that we support that right now since we're calling publish_stac_collection before ingest which means that the STAC collection must exist on the STAC API before the data is ingested and the API should reject an invalid/partial collection.

before ingest which means that the STAC collection must exist on the STAC API before

This assumes the remote API is used, but can't the ingestion be with the directory loader?
Since that is probably the only case where a partial collection could be defined, I think it would be fair to consider this feature as unsupported.

Yes but the directory loader still send the data to the API once it's loaded. Right now we don't support "uploading" STAC data to a catalog that's just a bunch of json files on disk.

I think it would be fair to consider this feature as unsupported.

Yeah I'd say it's unsupported as well

STACpopulator/populator_base.py

tests/data/stac_collection_testdata_xclim_cmip6_catalog.json

fmigneault · 2025-08-14T02:07:50Z

tests/data/stac_collection_testdata_xclim_cmip6_catalog.json

    "temporal": {
      "interval": [
-        ["1850-01-01T00:00:00Z", null]
+        ["2020-01-01T00:00:00Z", null]


Personally, I like the use of null for "unknown" end datetime, but I think STAC technically disallows it. There are some open issues about it in stac-spec, so might need to revalidate for this test.

On a side note, if null would be allowed, is that property handled everywhere that datetime comparaisons are performed. For example, if Collection used null explicitly and an Item provided an end value, would the null be considered "greater / open-ended" compared to the Item's end datetime? In a sense, the ˋnull` could be "higher priority" as collection summary to indicate that Items are still being added and that the Collection is continuously updated with new entries.

Similarly, would a null start datetime in the Collection be considered "lower" than a specific Item's start datetime? Maybe less common case than the open-ended datetime.

null values are allowed: https://github.com/radiantearth/stac-spec/blob/master/collection-spec/collection-spec.md#temporal-extent-object

There are some open issues about it in stac-spec

Are you talking about this one? Because I think that one is related to a different part of the spec and discusses adding support for null values: radiantearth/stac-spec#1161

On a side note, if null would be allowed, is that property handled everywhere that datetime comparaisons are performed

Yes it's supported

For example, if Collection used null explicitly and an Item provided an end value, would the null be considered "greater / open-ended" compared to the Item's end datetime?
Similarly, would a null start datetime in the Collection be considered "lower" than a specific Item's start datetime?

Yes for both

Yes, it is exactly this issue I was referring to. I am aware null is permitted in extents. My "disallowed" mentions are about the reference datetimes of the STAC Items specifically, and which are used as source metadata to compare and merge into the collection extent.

Technically, there would never be a valid STAC Item with start_datetime: "..." + end_datetime: null. Therefore, the strategy to combine them into the Collections temporal extent that could take advantage of open-ended null might be ambiguous.

If any STAC Item specifies end_datetime: null, it would be refused by the API, and therefore the corresponding collection update would not happen. But if the extent is explicitly provided with a [..., null] and that takes precedence, then all is good.

This reverts commit 4424d8d.

mishaschwartz · 2025-10-30T18:48:22Z

I've added a separate CLI option to update an existing collection based on all of the items in the collection.
All of the code is pretty much the same as before except that I moved it into a separate file instead of keeping it under the STACpopulatorBase class so that it could be used outside of a populator run.

mishaschwartz · 2025-10-30T18:49:59Z

@henriaidasso this is the implementation that @fmigneault mentioned in https://github.com/crim-ca/stac-populator/pull/114/files#r2418104894

fmigneault

Just minor typing adjustment to simplify things.

fmigneault · 2025-10-30T20:49:46Z

STACpopulator/cli.py

+            dest="update_collection",
            choices=get_args(inspect.signature(STACpopulatorBase.__init__).parameters["update_collection"].annotation),


Could this be updated using UpdateModes directly (also including "none" in it)?
It would be easier to interpret and there wouldn't be partial subset of possibles values across the code.

collection auto-update

1d8e6db

mishaschwartz requested review from fmigneault and huard as code owners July 30, 2025 02:10

fmigneault reviewed Aug 14, 2025

View reviewed changes

huard approved these changes Aug 15, 2025

View reviewed changes

mishaschwartz added 8 commits August 15, 2025 12:00

README update

4424d8d

Revert "README update"

3826555

This reverts commit 4424d8d.

review suggestions

c696465

formatting

2a6e0a1

Merge branch 'master' into collection-auto-update

a8e84aa

Merge branch 'master' into collection-auto-update

f954381

add collection update as a standalone option

3a348b8

simplify/combine annotations

3cce1ce

mishaschwartz requested review from fmigneault, henriaidasso and huard October 30, 2025 18:48

fmigneault reviewed Oct 30, 2025

View reviewed changes

mishaschwartz added 2 commits October 31, 2025 10:34

command arg updates

3846712

simplify annotations

a03cbf0

fmigneault approved these changes Oct 31, 2025

View reviewed changes

mishaschwartz added 2 commits November 12, 2025 11:33

Merge branch 'master' into collection-auto-update

e5bb335

Version updated from 0.10.0 to 1.11.0

a2f9429

mishaschwartz merged commit 9bd6cf1 into master Nov 12, 2025
12 checks passed

mishaschwartz deleted the collection-auto-update branch November 12, 2025 16:46

mishaschwartz mentioned this pull request Nov 17, 2025

RDPS implementation with additional extensions #114

Merged

		dest="update_collection",
		choices=get_args(inspect.signature(STACpopulatorBase.__init__).parameters["update_collection"].annotation),

Add option to automatically update collection extents and summaries based on ingested items #106

Add option to automatically update collection extents and summaries based on ingested items #106

Uh oh!

Conversation

mishaschwartz commented Jul 30, 2025

Uh oh!

fmigneault left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fmigneault Aug 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mishaschwartz commented Oct 30, 2025

Uh oh!

mishaschwartz commented Oct 30, 2025

Uh oh!

fmigneault left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

fmigneault Aug 14, 2025 •

edited

Loading