Merge dev to master for 0.7.0 release

catalog-spec/README.md

@@ -19,6 +19,10 @@ an OpenAPI definition of the standard way to do this, at the `/stac/` endpoint.
 **The Specification:** The main Catalog specification is in *[catalog-spec.md](catalog-spec.md)*.
 It includes an overview and in depth explanation of the structures and fields.
 
+**Best Practices:** While the main spec is designed to be quite flexible, there are a set of emerging best practices for
+how to actually manage a catalog. The *[catalog-best-practices.md](catalog-best-practices.md)* document lays out a number
+of these. In time some of these may evolve to be part of the core spec.
+
 **Examples:** For samples of how Catalogs can be implemented the *[examples/](examples/)* folder
 contains a full sample catalog. 
 

catalog-spec/catalog-best-practices.md

@@ -0,0 +1,163 @@
+# STAC Catalog Best Practices
+
+This document makes a number of recommendations for creating real world SpatioTemporal Asset [Catalogs](catalog-spec.md). None of them 
+are required to meet the core specification, but following these practices will make life easier for client tooling
+and for users. They come about from practical experience of implementors, and introduce a bit more 'constraint' for
+those who are creating new catalogs or new tools to work with STAC. 
+
+In time some of these may evolve to become part of the core specification, but while the current goal of the core is to remain 
+quite flexible and simple to meet a wide variety of use cases.
+
+## Static and Dynamic Catalogs
+
+As mentioned in the main [Catalog specification](catalog-spec.md), there are two main types of catalogs - static
+and dynamic. This section explains each of them in more depth and shares some best practices on each.
+
+#### Static Catalogs
+
+A main target for STAC has been object storage services like [Amazon S3](https://aws.amazon.com/s3/), 
+[Google Cloud Storage](https://cloud.google.com/storage/) and [Azure Storage](https://azure.microsoft.com/en-us/services/storage/), 
+so that users can stand up a full STAC implementation with static files. Implementations created with just files online
+are referred to as 'static catalogs'. These include not just the cloud services, but any type of file server that is online.
+
+Static Catalogs tend to make extensive use of *sub-catalogs* to organize their Items in to sensible browsing structures, 
+as they can only have a single representation of their catalog, since the static nature means the structure is baked in.
+While it is up to the implementor to organize the catalog, it is recommended to arrange it in a way that would make sense 
+for a human to browse a set of STAC Items in an intuitive matter.
+
+The recommendation for static catalogs is to define them using the file name `catalog.json` to distinguish 
+the catalog from item other JSON type files. In order to support multiple catalogs, the recommended practice 
+is to place the `catalog.json` in namespaces "directories". For example:
+
+- current/catalog.json
+- archive/catalog.json
+
+#### Dynamic Catalogs
+
+Dynamic STAC Catalogs are those that generate their JSON responses programmatically instead of relying on a set of
+already defined files. Typically a dynamic catalog implements the full [STAC API](../stac-api/) which enables 
+search of the Items indexed. But the `/stac/` endpoint returns the exact same `Catalog` and `Item` structures as a
+static catalog, enabling the same discovery from people browsing and search engines crawling. Dynamic API's that
+just seek to expose some data can also choose to only implement a Catalog the `/stac/` endpoint that returns dynamically.
+For example a Content Management Service like Drupal or an Open Data Catalog like CKAN could choose to expose its content
+as linked STAC Items by implementing a dynamic catalog. 
+
+One benefit of a dynamic catalog is that it can generate various 'views' of the catalog, exposing the same `Items` in 
+different sub-catalog organization structures. For example one catalog could divide sub-catalogs by date and another by
+providers, and users could browse down to both. The leaf Items should just be linked to in a single canonical location
+(or at least use a `rel` link that indicates the location of the canonical one.
+
+The STAC API is also made to be compatible with WFS3, which has a set structure for the canonical location of its features.
+STAC Items should use the WFS3 location as their canonical location, and then in the `/stac/` browse structure would just
+link to those locations. 
+
+## Catalog Layout
+
+Creating a catalog involves a number of decisions as to what folder structure to use to represent sub-catalogs, items
+and assets, and how to name them. The specification leaves this totally open, and you can link things as you want. We
+encourage people to explore new structures. But the following are what a number of implementors ended up doing. Following
+these recommendations makes for more legible catalogs.
+
+1. Root documents (catalogs / collections) should be at the root of a directory tree containing the static catalog.
+2. Catalogs should be named `catalog.json` (cf. `index.html`).
+3. Collections that are distinct from catalogs should be named `collection.json`.
+4. Items should be named `<id>.json`
+5. Sub-catalogs should be stored in subdirectories of their parent (and only 1 subdirectory deeper than a document's parent) (e.g. `.../sample/sub1/catalog.json`).
+6. Items should be stored in subdirectories of their parent catalog. 
+This means that each item and its assets are contained in a unique subdirectory
+7. Limit the number of items in a catalog or sub-catalog, grouping / partitioning as relevant to the dataset
+
+### Dynamic Catalog Layout
+
+While these recommendations were primarily written for [static catalogs](catalog-spec.md#static-catalogs), they apply
+equally well to [dynamic catalogs](catalog-spec.md#dynamic-catalogs). Subdirectories of course would just be URL paths 
+generated dynamically, but the structure would be the same as is recommended.
+
+One benefit of a dynamic catalog is that it can generate various 'views' of the catalog, exposing the same Items in 
+different sub-catalog organization structures. For example one catalog could divide sub-catalogs by date and another 
+by providers, and users could browse down to both. The leaf Items should just be linked to in a single canonical location 
+(or at least use a rel link that indicates the location of the canonical one). It is recommended that dynamic catalogs 
+provide multiple 'views' to allow users to navigate in a way that makes sense to them, providing multiple 'sub-catalogs'
+from the root catalog that enable different paths to browse (country/state, date/time, constellation/satellite, etc). But the 
+canonical 'rel' link should be used to designate the primary location of the item to search engine crawlers.
+
+## Use of links
+
+The main catalog specification allows both relative and absolute links, and says that `self` links are not required, but are 
+strongly recommended. This is what the spec must say to enable the various use cases, but there is more subtlety for when it 
+is essential to use different link types. The best practice is to use one of the below 
+catalog types, applying the link recommendations consistently, instead of just haphazardly applying anything the spec allows.
+
+### Self-contained Catalogs
+
+A 'self-contained catalog' is one that is designed for portability. Users may want to download a catalog from online and be
+able to use it on their local computer, so all links need to be relative. Or a tool that creates catalogs may need to work 
+without knowing the final location that it will live at online, so it isn't possible to set absolute 'self' URL's. These use
+cases should utilize a catalog that follows the listed principles:
+
+* **Only relative href's in `links`**: The full catalog structure of links down to sub-catalogs and items, and their 
+links back to their parents and roots, should be done with relative URL's. This enables the full catalog to be downloaded or
+copy to another location and to still be valid. This also implies no `self` link, as that link must be absolute.
+
+* **Use Asset `href` links consistently**: The links to the actual assets are allowed to be either relative or absolute. There
+are two types of 'self-contained catalogs'. The first is just the metadata, and use absolute href links to refer to the 
+online locations of the assets. The second uses relative href links for the assets, and includes them in the folder structure.
+This enables offline use of a catalog, by including all the actual data.
+
+Self-contained catalogs tend to be used more as static catalogs, where they can be easily passed around. But often they will
+be generated by a more dynamic STAC service, enabling a subset of a catalog or a portion of a search criteria to be downloaded
+and used in other contexts. That catalog could be used offline, or even published in another location.
+
+Self-contained catalogs are not just for offline use, however - they are designed to be able to be published online and to live
+on the cloud in object storage. They just aim to ease the burden of publishing, by not requiring lots of updating of links. 
+Adding a single `self` link at the root is recommended for online catalogs, turning it into a 'relative published catalog', as detailed below. This anchors it in an online location and enable provenance tracking.
+
+### Published Catalogs
+
+A 'published catalog' is one that lives online in a stable location, and uses `self` links to establish its location and 
+enable easy provenance tracking. There are two types of published catalogs:
+
+* **Absolute Published Catalog** is a catalog that uses absolute links for everything, both in the `links` objects and in the
+`asset` hrefs. It includes `self` links for every item. Generally these are implemented by dynamic catalogs, as it is quite
+easy for them to generate the proper links dynamically. But a static catalog that knows its published location could easily
+implement it.
+* **Relative Published Catalog** is a catalog that uses relative links for everything, but includes an absolute `self` link at
+the root catalog, to identify its online location. This is designed so that a self-contained catalog can be 'published' online
+by just adding one field (the self link) to its root catalog. All the other links should remain relative. With this, the 
+resolution of item and sub-catalog self links may be done by traversing parent and root links, but requires reading multiple 
+sources to achieve this.
+
+## Static to Dynamic best practices
+
+Many implementors are using static catalogs to be the reliable core of their dynamic services, or layering their STAC API
+on top of any static catalog that is published. These are some recommendations on how to handle this:
+
+#### Ingestion and links
+
+Implementors have found that it's best to 'ingest' a static STAC into an internal datastore (often elasticsearch, but a 
+traditional database could work fine too) and then generate the full STAC API responses from that internal representation.
+There are instances that have the API refer directly to the static STAC Items, but this only works well if the static STAC 
+catalog is an 'absolute published catalog'. So the recommendation is to always use absolute links - either in the static 
+published catalog, or to create new absolute links for the STAC search/ endpoint 
+responses, with the API's location at the base url. The /stac endpoint with the catalogs could either link directly
+to the static catalog, or can follow the 'dynamic catalog layout' recommendations above with a new set of URL's.
+
+Ideally each `Item` would use its `links` to provide a reference back to the static location. The location of the static
+item should be treated as the canonical location, as the generated API is more likely to move or be temporarily down. The
+spec provides the `derived_from` rel attribute, which fits well enough, but `canonical` is likely the more appropriate one
+as everything but the links should be the same.
+
+#### Keeping static and dynamic catalogs in sync with cloud notification and queue services
+
+There is a set of emerging practices to use services like Amazon's Simple Queue Service (SQS) and Simple Notification Service
+(SNS) to keep catalogs in sync. There is a great [blog post on the CBERS STAC implementation on AWS](https://aws.amazon.com/blogs/publicsector/keeping-a-spatiotemporal-asset-catalog-stac-up-to-date-with-sns-sqs/). The core 
+idea is that a static catalog should emit a notification whenever it changes. The recommendation for SNS is to use the STAC 
+item JSON as the message body, with some attributes such as a scene’s datetime and geographic bounding box that allows 
+basic geographic filtering from listeners. 
+
+The dynamic STAC API would then listen to the notifications and update its internal datastore whenever new data comes into
+the static catalog. Implementors have had success using AWS Lambda to do a full 'serverless' updating of the elasticsearch
+database, but it could just as easily be a server-based process.
+
+
+

catalog-spec/catalog-spec.md

@@ -91,54 +91,25 @@ root catalog might be a sub-catalog of someone else's structure. The goal is for
 information and links they want to, while also encouraging a natural web of information to arise as Catalogs and Items are
 linked to across the web.
 
-### Catalog Types
-
-The core Items and Catalogs of a SpatioTemporal Asset Catalog are designed for ease of implementation. 
-With the same core JSON documents and link structures it can be implemented in a way that is just files available online,
-or easily implementable with modern REST service infrastructures. These represent two different types of catalogs, the 
-'static catalog' and the 'dynamic catalog', which can operate a bit differently though they can be treated the same by
-clients.
-
-But any server can implement the same JSON and link structure, creating responses
-dynamically as REST calls. These are referred to as 'dynamic catalogs'.
-
-#### Static Catalogs
-
-A main target for STAC has been object storage services like [Amazon S3](https://aws.amazon.com/s3/), 
-[Google Cloud Storage](https://cloud.google.com/storage/) and [Azure Storage](https://azure.microsoft.com/en-us/services/storage/), 
-so that users can stand up a full STAC implementation with static files. Implementations created with just files online
-are referred to as 'static catalogs'. These include not just the cloud services, but any type of file server that is online.
+There are a number of emerging 'best practices' for how to organize and implement good catalogs. These can be found in
+the [best practices document](catalog-best-practices.md), and include things like catalog layout, use of self links, 
+publishing catalogs, and more. This specification is designed for maximum flexbility, but the best practices provide
+guidance for good recommendations when implementing.
 
-Static Catalogs tend to make extensive use of *sub-catalogs* to organize their Items in to sensible browsing structures, 
-as they can only have a single representation of their catalog, as the static nature means the structure is baked in. 
-While it is up to the implementor to organize the catalog, it is recommended to arrange the in a way that would make sense 
-for a human to browse a set of STAC Items in an intuitive matter.
-
-Static catalogs a recommended to be defined using the file name `catalog.json` to distinguish from item other JSON
-type files. In order to support multiple catalogs, the recommended practice is to place the
-`catalog.json` in namespaces "directories". For example:
-
-- current/catalog.json
-- archive/catalog.json
-
-#### Dynamic Catalogs
+### Catalog Types
 
-Dynamic STAC Catalogs are those that generate their JSON responses programmatically instead of relying on a set of
-already defined files. Typically a dynamic catalog implements the full [STAC API](../stac-api/) which enables 
-search of the Items indexed. But the `/stac/` endpoint returns the exact same `Catalog` and `Item` structures as a
-static catalog, enabling the same discovery from people browsing and search engines crawling. But dynamic API's that
-just seek to expose some data can also choose to only implement a Catalog the `/stac/` endpoint that returns dynamically.
-For example a Content Management Service like Drupal or an Open Data Catalog like CKAN could choose to expose its content
-as linked STAC Items by implementing a dynamic catalog. 
+Though it is technically an implementation detail outside the scope of the core specification, it is worth mentioning
+that implementations generally fall into two different 'types':
 
-One benefit of a dynamic catalog is that it can generate various 'views' of the catalog, exposing the same `Items` in 
-different sub-catalog organization structures. For example one catalog could divide sub-catalogs by date and another by
-providers, and users could browse down to both. The leaf Items should just be linked to in a single canonical location
-(or at least use a `rel` link that indicates the location of the canonical one.
+* **Static Catalogs** can be implemented as simply files online, often stored in an cloud storage service like [Amazon S3](https://aws.amazon.com/s3/). 
+or [Google Cloud Storage](https://cloud.google.com/storage/).
+The core JSON documents and link structures are encoded in the file, and work as long as things are structured properly.
+* **Dynamic Catalogs** are implemented in software, returning the JSON documents and links dynamically. This is mostly 
+used when data holdings are already exposed through a dynamic interface, and STAC can be an alternate facade on the 
+same core database or search cluster.
 
-The STAC API is also made to be compatible with WFS3, which has a set structure for the canonical location of its features.
-STAC Items should use the WFS3 location as their canonical location, and then in the `/stac/` browse structure would just
-link to those locations. 
+The two catalog types both implement the same fields and links, and can be treated as the same by clients. For more 
+details on the two types and how you might use them see the [Static and Dynamic Catalogs](catalog-best-practices.md#static-and-dynamic-catalogs) section of the best practices document.
 
 ## Catalog fields
 
@@ -207,7 +178,9 @@ with links.
 
 A more complete list of possible 'rel' types can be seen at the [IANA page of Link Relation Types](https://www.iana.org/assignments/link-relations/link-relations.xhtml).
 
-Please see the chapter 'relative vs absolute links' in the [Item spec](../item-spec/item-spec.md#relative-vs-absolute-links) for a discussion on that topic. 
+Please see the chapter 'relative vs absolute links' in the [Item spec](../item-spec/item-spec.md#relative-vs-absolute-links)
+ for a discussion on that topic, as well as the [use of links](catalog-best-practices.md#use-of-links) section of the 
+ catalog best practices document.
 
 #### Relation types
 

item-spec/item-spec.md

@@ -83,7 +83,7 @@ It is allowed to add additional fields such as a `title` and `type`.
 
 | Field Name | Type   | Description                                                                                                                         |
 | ---------- | ------ | ----------------------------------------------------------------------------------------------------------------------------------- |
-| href       | string | **REQUIRED.** The actual link in the format of an URL. Relative and absolute links are both allowed unless otherwise noted in the ```rel``` type description.                    |
+| href       | string | **REQUIRED.** The actual link in the format of an URL. Relative and absolute links are both allowed.                    |
 | rel        | string | **REQUIRED.** Relationship between the current document and the linked document. See chapter "Relation types" for more information. |
 | type       | string | Media type of the referenced entity. 																								|
 | title      | string | A human readable title to be used in rendered displays of the link. |