diff --git a/src/docs/rfcs/018-mango-covering-json-index.md b/src/docs/rfcs/018-mango-covering-json-index.md new file mode 100644 index 00000000000..154eb8c3f84 --- /dev/null +++ b/src/docs/rfcs/018-mango-covering-json-index.md @@ -0,0 +1,398 @@ +--- +name: Formal RFC +about: Submit a formal Request For Comments for consideration by the team. +title: 'Support covering indexes when using Mango JSON (view) indexes' +labels: rfc, discussion +assignees: '' + +--- + +[NOTE]: # ( ^^ Provide a general summary of the RFC in the title above. ^^ ) + +# Introduction + +## Abstract + +[NOTE]: # ( Provide a 1-to-3 paragraph overview of the requested change. ) +[NOTE]: # ( Describe what problem you are solving, and the general approach. ) + +Covering indexes are used to reduce the time the database takes to respond to +queries. An index "covers" a query when the query only requires fields that are +in the index (in this way, "covering" is a property of index and query +combined). When this is the case, the database doesn't need to consult primary +data and can return results for the query from only the index. In more familiar +CouchDB terminology, this is equivalent to querying a view with +`include_docs=false`. + +When evaluating a query, Mango currently doesn't use the concept of covering +indexes; even if a query could be answered without reading each result's full +JSON document, Mango will still read it. This makes it impossible for Mango to +return data as quickly as the underlying view. + +My benchmarking shows that Mango can answer at the same rate as the underlying +view index. It currently runs at the same pace as calling the view with +`include_docs=true`. Preliminary modifications to Mango showed that, with +covering index support and a query that can use it, Mango can stream results +as quickly as the underlying view. Adding covering indexes therefore increases +the production use-cases Mango can support substantially. + +There are likely two phases to this: + +- Enable covering indexing processing for current indexes (ie, over view keys). +- Allow Mango view indexes to include extra data from documents, storing it in + the `value` of the view. Support use of this extra data within the covering + indexes feature. + +### Out of scope + +This proposal only covers adding covering indexes to JSON indexes and not text +indexes. The aim is to reduce the need for CouchDB users to run separate +processes, such as Lucene, to get improved querying performance and capability. + +We do not aim to replicate `reduce` functionality from views, only to bring +parity to non-reduced view execution speed (ie, when views are used to search +the document space) to Mango. + +## Requirements Language + +[NOTE]: # ( Do not alter the section below. Follow its instructions. ) + +The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", +"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this +document are to be interpreted as described in +[RFC 2119](https://www.rfc-editor.org/rfc/rfc2119.txt). + +## Terminology + +[TIP]: # ( Provide a list of any unique terms or acronyms, and their definitions here.) + +- Mango: CouchDB's Mongo-inspired querying system. +- View / JSON index: Mango index that uses the same index as Cloudant views. +- Coordinator: the Erlang process that handles doing a distributed query across + a CouchDB cluster. + +--- + +# Detailed Description + +[NOTE]: # ( Describe the solution being proposed in greater detail. ) +[NOTE]: # ( Assume your audience has knowledge of, but not necessarily familiarity ) +[NOTE]: # ( with, the CouchDB internals. Provide enough context so that the reader ) +[NOTE]: # ( can make an informed decision about the proposal. ) + +[TIP]: # ( Artwork may be attached to the submission and linked as necessary. ) +[TIP]: # ( ASCII artwork can also be included in code blocks, if desired. ) + +This would take place within `mango_view_cursor.erl`. The key functions +involved are the shard-level `view_cb/2`, the streaming result handler at the +coordinator end (`handle_message/2`) and the `execute/3` function. + +## Mango JSON index selection + +A Mango JSON index is implemented as a view with a complex key. The first field +in the index is the first entry in the complex key, the second field is the +second key and so on. Even indexes with one field use a complex key with length +`1`. + +When choosing a JSON index to use for a query, there are a couple of things that +are important to covering indexes. + +Firstly, note there are certain predicate operators that can be used with an +index, currently: `$lt`, `$lte`, `$eq`, `$gte` and `$gt`. These can easily be +converted to key operations within a key ordered index. For an index to be +chosen for a query, the first key within the indexes complex key MUST be used +with a predicate operator that can be converted into an operation on the index. + +Secondly, a quirk of Mango indexes is that for a document to be included in an +index it must contain all of the index's indexed fields. Documents without all +the fields will not be included. This means that when we are choosing an index +for a query, we must further choose an index where the predicates within the +`selector` imply `$exists=true` for all fields in the index's key. Without that, +we will have incomplete results. + +Why is this? Let's look at an index with these fields: + +```json +["age", "name"] +``` + +Now we index two documents. The first document is included in the index while the second is not (because it doesn't include `name`): + + +```json +{"_id": "foo", "age": 39, "name": "mike"} + +{"_id": "bar", "age": 39, "pet": "cat"} +``` + +The `selector` `{"age": {"$gt": 30}}` should return both documents. However, if +we use the index above, we'd miss out `bar` because it's not in the index. +Therefore we can't use the index. + +On the other hand, the `selector` `{"age": {"$gt": 30}, "name": {"$exists": +true}}` requires that the `name` field exist so the index can be used because +the query predicates can only match documents containing both `age` and `name`, +just like the index. In both cases, note the predicate `"age": {"$gt": 30}` +implies `"age": {"$exists": true}`. + +## Phase 1: handle keys only covering indexes + +Within `execute/3` we will need to decide whether the view should be requested +to include documents. If the index is covering, this will not be required and +so the `include_docs` argument to the view fabric call will be `false`. We'll +need to add a helper method to return whether the index is covering. + +When selecting an index, we'll need to ensure that only fields in the `selector` +and not `fields` are used when choosing an index. This is because we need all +fields in the `selector` to be present per [Mango JSON index +selection](#mango-json-index-selection). This is because `fields` is only used +after we generate the result set, and none of the field names in `fields` need +to exist in result documents. + +As an example, an index `["age", "name"]` would still require the `selector` to +imply `$exists=true` for both `age` and `name` even if the `fields` were just +`["age"]` in order that correct results be returned. + +Of note, this means that if an index is unusable pre-covering-index support, it +will continue to be unusable after this implementation: whether an index covers +a query is only used to prefer one already usable index over another. + +Within `view_cb/2`, we'll need to know whether an index is covering. Without +that, `view_cb/2` will interpret the lack of included documents as an indicator +that it should do nothing, while in fact we want it to fully process the result +as it does when `include_docs` is used -- apart from when the user passes `r>=2` +in the Mango query because then the coordinator reads and processes documents. +(Aside: it'd be good to remove this `r` option to simplify things). + +In `handle_message/2` the main work is ensuring that we handle mixed cluster +version states -- ie, cluster state during upgrades. + +## Phase 2: add support for included fields in indexes + +I propose we add an `include` field into a Mango JSON index definition: + +```json +{ + "index": { + "fields": [ "age", "name" ], + "include": [ "occupation", "manager_id" ] + }, + "name": "foo-json-index", + "type": "json" +} +``` + +Behaviour requirements: + +- Unlike `fields`, the fields in `include` _do not have to exist_ in the source + document in order that the document be included in the index. This is to + allow the index to cover more queries. +- Including a deeply nested field would follow the same pattern as for other + field references in Mango, `person.address.zip`. +- There is no notation to include the whole document, that is, no equivalent of + `emit(doc.name, doc)`. +- `"include": []`, `"include": null` and omitting the `include` field are all + equivalent. +- Ordering of the fields in `include` is not important. They can be reordered + before storing if needed (eg, sorted). +- It will be an error to include a field in both `fields` and `include`. This + should be rejected by the `_index` call. +- The `include` field would be rejected for `text` type indexes. + +Alternatives considered: + +- Adding `include` outside `index`. This didn't seem right as the `index` + object already includes `partial_filter_selector` and `include` seems a + peer of this. ([docs](https://docs.couchdb.org/en/stable/api/database/find.html#db-index)). +- Alternative name `store`. We use this for Lucene indexes when dreyfus/clouseau + is used. I elected to use a separate name to either `value` or `store` to + avoid index-type specificity. I take the name from Postgres, which uses + `INCLUDE` in its index definition to [support covering indexes][pgcover]. + +[pgcover]: https://www.postgresql.org/docs/current/indexes-index-only-scans.html + +Adding this will require changes in `mango_idx_view` to store the definition and +in how we process documents during indexing, which looks to be in +`get_index_entries` in `mango_native_proc`. + +We'll then need to update the Mango cursor methods mentioned above to take +account of the values within the covering index code. + +One thing to be careful about is again index selection. We will still need all +index keys to be present in the `selector` as above so need differentiate +between the fields in index's keys and values when selecting an index to ensure +we retain the correct behaviour per [Mango JSON index +selection](#mango-json-index-selection). + +### Limits on included fields + +Adding "too much" data to indexes is likely to slow down index scans because +there will be more data to process. We would also like to avoid users creating +pathological cases just because they can. Therefore, limiting the data that can +be stored seems wise. Saying that, for those that are willing to profile their +workloads should have a get-out clause from limits. + +As an example, [postgres](https://www.postgresql.org/docs/current/limits.html) limits indexes to 32 columns. Its max field size is 1GB; I think we'd like something a little smaller! + +Therefore the feature will have the following limit enforcement settings: + +- `mango_json_index_include_fields_max` is the limit on the length of the + `include` list. +- `mango_json_index_include_depth_max` is a limit on the depth of fields we will + pull out. Basically the maximum numbers of `.` in a path. +- If the total number of bytes for values exceeds + `mango_json_index_include_size_bytes_max` then we will skip that document + from the index. + +I need to check whether these should be prefixed `mango_` given they would live +in a `mango` configuration section. + +Defaults: + +- `mango_json_index_include_fields_max=16` +- `mango_json_index_include_depth_max=8` +- `mango_json_index_include_size_bytes_max=32768` (32kb) + +I have chosen power-of-two limits mostly because they feel like familiar +numbers. Not a great reason, so these may be refined during code writing if I +can work out suitable benchmarks. + + +## Mixed versions during cluster upgrades + +The relevant scenarios here are an updated coordinator talking to outdated +shards, and the opposite of an outdated coordinator talking to upgraded shards. +A further wrinkle is that while a coordinator is either upgraded or not, the +shards that the coordinator speaks to can be a mixture of upgraded and outdated. + +For the purposes of this discussion, we only need to worry about when a covering +index is in play during a query; the code path outside that use-case should not +change. + +From what I can tell, we can avoid special code paths for cluster upgrades +specific to this work. Instead we accept that some queries will take longer +during cluster upgrade mixed version operation. This is described below. + +### Updated coordinator, outdated shard + +In this case, the coordinator will note the covering index, and set the view +query option `include_docs=false`. This means that the row passed to `view_cb/2` +will not have a document included. In the function, `case ViewRow#view_row.doc +of` will hit the `undefined` clause, meaning that the row is passed through +unchanged, without the document. When the row reaches the coordinator and is +passed to `doc_member_and_extract/2` from `handle_message/2`, the `case +couch_util:get_value(doc, RowProps) of` will also hit its `undefined` clause. +The coordinator will then perform a quorum read with `r=1` of the document and +carry out the match and extract. + +This will slow down the processing of results at the coordinator for that row, +but shouldn't alter the correctness of the result. So we shouldn't need a +special code path to support this case. Which is nice. + +### Outdated coordinator, updated shard + +In this case, the coordinator won't be checking for covering indexes, meaning +that `include_docs=true` will be set when `r<2` as today. + +I suspect we'll set an option in `viewcbargs` that contains the index field +names and whether it's a covering index. This means that an updated shard will +be checking for those fields. When it can't find them, it'll fallback to the +current behaviour in `view_cb/2`, meaning that it reads the document found via +`include_docs=true`, execute `match_and_extract_doc/3` and return the row if it +matches the query. + +The coordinator will receive the final result document as today and assume it's +correct, and forward it to the client. More work than needed will be carried out +at the shard. But, again, this doesn't appear to require special code so long as +we are careful. + + +# Advantages and Disadvantages + +[NOTE]: # ( Briefly, list the benefits and drawbacks that would be realized should ) +[NOTE]: # ( the proposal be accepted for inclusion into Apache CouchDB. ) + +The primary advantage is allowing Mango queries to be as performant as View +queries while retaining their greater flexibility. It does not aim to replace +`reduce` functionality. + +There should be no disadvantages to this change. Existing functionality remains +intact, and there should be no decreases in performance. + +# Key Changes + +[TIP]: # ( If the changes will affect how a user interacts with CouchDB, explain. ) + +Adding the extra field to Mango JSON index definitions is the key API change. + +Users will generally have to alter their indexes to take best advantage of this +new functionality. As there has been no advantage to including extra items in +a JSON index's key so far, most indexes won't contain the data to cover queries. + +For now, this RFC doesn't explicitly suggest exposing the covering index choice +via `_explain`, though this would be a valuable piece of extra work. We will, +however, need to ensure that the execution stats that each `_find` call spits +out are correct -- that is, for a covering index there should be no documents +read. + +## Applications and Modules affected + +[NOTE]: # ( List the OTP applications or functional modules in CouchDB affected by the proposal. ) + +- `mango`; see details above. + +## HTTP API additions + +[NOTE]: # ( Provide *exact* detail on each new API endpoint, including: ) +[NOTE]: # ( HTTP methods [HEAD, GET, PUT, POST, DELETE, etc.] ) +[NOTE]: # ( Synopsis of functionality ) +[NOTE]: # ( Headers and parameters accepted ) +[NOTE]: # ( JSON in [if a PUT or POST type] ) +[NOTE]: # ( JSON out ) +[NOTE]: # ( Valid status codes and their definitions ) +[NOTE]: # ( A proposed Request and Response block ) + +Add field `include` to Mango JSON index definitions. Validate that it is only +accepted within JSON indexes and not text indexes. + +```json +{ + "index": { + "fields": [ "age", "name" ], + "include": [ "occupation", "manager_id" ] + }, + "name": "foo-json-index", + "type": "json" +} +``` + +## HTTP API deprecations + +[NOTE]: # ( Provide *exact* detail on the API endpoints to be deprecated. ) +[NOTE]: # ( If these endpoints are replaced by new endpoints, list those as well. ) +[NOTE]: # ( State the proposed version in which the deprecation and removal will occur. ) + +None. + +# Security Considerations + +[NOTE]: # ( Include any impact to the security of CouchDB here. ) + +This change is an optimisation to query performance. It should not have any +effects on CouchDB's security model. + +# References + +[TIP]: # ( Include any references to CouchDB documentation, mailing list discussion, ) +[TIP]: # ( external standards or other links here. ) + +Crunchydata have a nice article on [why covering indexes are a valuable feature][crunch]. + +[crunch]: https://www.crunchydata.com/blog/why-covering-indexes-are-incredibly-helpful + +# Acknowledgements + +[TIP]: # ( Who helped you write this RFC? ) + +This work builds on prior PoCs and work by Will Holley and Garren Smith, and +has benefited from discussion with them.