Feature dei documentation

documentation/conf.py

@@ -406,6 +406,56 @@
         use_titles=True,
     )
 
+#-------------------------------------------------------------------------------------------
+# Fetch extension schemas at build to allow drawing schema tables in the extensions section
+#
+# Matt added this to deal with the problem where we need to draw schema tables for
+# extensions but the extension schemas are managed externally.
+#-------------------------------------------------------------------------------------------
+
+import requests
+
+schema_extensions = ['dei'] # ADD NEW EXTENSIONS HERE
+
+# IMPORTANT: Update this URL when the dei extension is merged in
+base_url = "https://raw.githubusercontent.com/ThreeSixtyGiving/extensions-registry/add-dei-extension/extensions/"
+
+if not os.path.exists("./extras"):
+    os.mkdir("./extras")
+
+if not os.path.exists("./extras/extensions/"):
+    os.mkdir("./extras/extensions/")
+
+for extension in schema_extensions:
+    # Make a directory for the extension schemas inside the extras directory
+    extension_path = f"./extras/extensions/{extension}"
+    if not os.path.exists(extension_path):
+        os.mkdir(extension_path)
+        os.mkdir(f"{extension_path}/codelists")
+
+    # Fetch the registry entry and parse the JSON
+    registry_entry = json.loads(requests.get(f"{base_url}/{extension}.json").text)
+
+    # Fetch each of the schemas
+    for schema in registry_entry['schemas']:
+        schema_content = requests.get(schema['uri']).text
+
+        with open(f"{extension_path}/{schema['target']}", 'w') as schema_file:
+            schema_file.write(schema_content)
+
+    # Fetch each of the codelists
+    for codelist in registry_entry['codelists']:
+        codelist_content = requests.get(codelist).text
+        codelist_name = codelist.split('/')[-1]
+
+        with open(f"{extension_path}/codelists/{codelist_name}", 'w') as codelist_file:
+            codelist_file.write(codelist_content)
+
+#-------------------------------------------------------------------------------------------
+#-------------------------------------------------------------------------------------------
+# End fetch extension schemas at build
+#-------------------------------------------------------------------------------------------
+
 #mv README.md 360-giving-schema-titles.csv/
 
 

documentation/extensions/dei/governance.md

@@ -0,0 +1,37 @@
+# Governance and Versioning
+
+As an extension to the 360Giving Data Standard, the DEI Extension is managed outside of the [governance process](https://standard.threesixtygiving.org/en/latest/about/governance/) of the 360Giving Data Standard itself.
+
+360Giving is the steward of the DEI Extension. Our CEO is responsible for its day-to-day management, supported by a Product Manager and an external specialist technical team, Open Data Services Coop.
+
+## Aims and Approach
+
+Our aims and approach for maintaining the DEI Extension are as follows:
+
+1. Compatibility with the 360Giving Data Standard
+To maintain compatibility with the latest release of the 360Giving Data Standard. 
+
+2. Compatibility with the DEI Data Standard
+To maintain compatibility with the DEI Data Standard itself whenever possible. Updates to the DEI Extension will be issued in a timely manner to bring it in line with changes to the taxonomy structure and codes included in the DEI Data Standard, as well as any guidance about how to apply this to publishing the data alongside 360Giving data.
+
+3. Use semantic versioning
+To use semantic versioning to version this extension: MAJOR updates will not be compatible with previous versions of the extension, MINOR updates will add new features that are backwards compatible, and PATCH updates will fix bugs and amend documentation. 
+
+4. Consider stakeholders needs in the revision process
+To consider stakeholders needs and follow [360Giving Data Standard revision processes](https://standard.threesixtygiving.org/en/latest/about/governance/#the-revision-process) when making updates to the DEI Extension.
+
+## Versions
+
+The DEI Extension is versioned using [Semantic Versioning](https://semver.org/).
+
+The version number of the DEI Extension and update processes and schedules are *separate* from each: the [360Giving Data Standard](https://standard.threesixtygiving.org/en/latest/about/governance/#versions); and the [DEI Data Standard](https://www.funderscollaborativehub.org.uk/collaborations/dei-data-standard).
+
+This means there may be times when data shared using the DEI Extension does not match the latest version of the DEI Data Standard.
+
+The version number and all notable changes are documented by Release via the Changelog.
+
+## Changelog
+
+All notable changes to the DEI Extension are documented by Release via the extension's [Github repository](https://github.com/ThreeSixtyGiving/360-dei/releases)
+
+

documentation/extensions/dei/guidance.md

@@ -0,0 +1,77 @@
+# Guidance for Publishers
+
+## Introduction
+
+This guide is for UK funding organisations that want to publish data collected using the DEI Data Standard alongside 360Giving grants data. It is designed to help you whether you are at the very beginning of your journey and planning your approach or preparing your data.
+
+This guidance provides further information about the DEI Data Standard and the 360Giving Data Standard’s DEI Extension which has been developed to support the sharing of DEI Data Standard data alongside 360Giving data.
+
+This isn’t a reporting process, it’s an opportunity to collect and share data that will create a knowledge-base to support effective action to identify and target funding to address structural inequity.
+
+## About the DEI Standard
+
+The [DEI (Diversity, Equity, and Inclusion) Data Standard](https://www.funderscollaborativehub.org.uk/dei-data-standard) was created by an independent working group of UK funders, known as the DEI Data Group, who believe that without an effective framework to capture equity data there can be no effective action to identify and target funding to address structural inequity. The DEI Data Group agreed on a shared framework (classifications, language and approach) which aims to categorise organisations either led by, or targeting and supporting groups experiencing structural inequity.
+
+The DEI Data Standard defines a taxonomy to describe population groups and an approach to collecting the data. In the context of grantmaking, these taxonomies may be applied to describe various aspects of the organisation and people involved in a project funded by a grant. These taxonomies can be applied to describe the following three areas:
+
+* **People receiving primary benefits/service users** of the project or the organisation
+* **Mission and purpose** of the organisation
+* **Leadership** of the organisation
+
+### Why collect and share DEI Data Standard data?
+
+Collecting and sharing DEI Data Standard information is intended to enable funders to understand whether the reach of the funding and their funding practices are equitable. Information shared in this way can then be used to inform funding strategies and specific funding programmes or approaches.
+
+It is important to note that the DEI Data Standard is about understanding **equity** rather than diversity. It aims to help funders to answer the question – is our grantmaking reaching communities facing structural inequity, and in particular is the money going to organisations led by people from those communities?
+
+The goal of developing the DEI Data Standard and framework for applying it was to increase the equity of grantmaking through having good quality and consistent data to monitor this, and to enable shared reporting, while also reducing the burden on applicants around data collection through grantmakers using a consistent approach.
+
+### How to collect DEI Data Standard Data
+
+The DEI Data Standard isn’t one size fits all and so it has been designed with options in how the data is collected to suit different funding approaches. For example a funder making a large volume of grants will probably make different choices to a funder who makes fewer grants.
+
+Full information about the DEI Data Standard, including details of the taxonomy and approach, and guidance on how to apply these to different grantmaking contexts can be found on the ‘The DEI Data Standard’ section of the [Funders Collaborative Hub](https://www.funderscollaborativehub.org.uk/dei-data-standard)
+
+
+## About the DEI Extension
+
+The DEI Extension is an extension to the 360Giving Data Standard. 
+
+360Giving has developed the DEI Extension to enable publication of DEI Data Standard information related to grants as a single dataset alongside 360Giving data. This ties the DEI Data Standard information directly to the context of the grant, making it easier to publish, manage, and analyse.
+
+The extension adds new fields to the 360Giving Data Standard which can be used to include data collected in accordance with the DEI Data Standard. This allows 360Giving tools to validate the DEI Data Standard information alongside the rest of the 360Giving data fields leading to better data quality for analysis and a better understanding of the equity of grantmaking and the collective grantmaking picture.
+
+Detailed technical information for the DEI Extension is available on the `$DEI-REFERENCE` page. It contains a reference for the fields and codes added by the DEI Extension.
+
+### What is covered in the DEI Extension
+
+The DEI Extension includes a number of fields important to properly situate DEI Data Standard information in the context of grantmaking. This includes both the answers themselves, but also important metadata such as the purposes of collecting the information and whether a question was asked, and how.
+
+The DEI Extension follows the DEI Data Standard’s guidelines on the application of its taxonomy which is applied to grants in three different areas:
+
+* People receiving the primary benefits / service users - titled **Project** in the extension
+* The mission and purpose of the organisation - titled **Mission**
+* The leadership of the organisation - titled **Leadership**
+
+For each of these areas you may provide the responses to the questions in the form of DEI Data Standard Taxonomy codes as well as free text responses such as lived experience or geographical area.
+
+In addition to the fields for providing response, there are also fields to provide metadata describing the whether questions were asked, how they were asked, and whether there has been a response. These metadata fields are important to contextualise the data to avoid ambiguity and support data users analysing and understanding the data correctly.
+
+Funders are able to make choices about how they implement the DEI Data Standard based on their specific context. This is an important feature of the DEI Data Standard, but leads to variations in the data collected which must then be contextualised to be understood correctly.
+
+For example, funders making grants to small organisations or providing unrestricted funding for the benefit of the whole organisation might opt to only ask applicants questions about the **Mission** and the **Leadership** of the organisation as the **Project** users are likely to be the same as the Mission of the organisation, whereas other funders may ask questions about all three application areas. A data user looking at this data side-by-side may not know whether the first funder didn’t ask about the Project or whether the data is missing from the file.
+
+The metadata fields in the DEI Extension address this by communicating this context. They allow the first funder to show that questions about the Project weren’t asked or weren’t applicable so that a data user understands more about the context.
+
+Some important metadata fields included in the DEI Extension are:
+
+* Whether the information was asked for, and how
+* What options were available to the respondents when the information was asked for
+* Which levels of classification (*Population Group*, *Category*, *Subcategory*) from the taxonomy were available as responses
+* The status of the reply to the question (e.g. “No reply”)
+
+## Support Available
+
+360Giving Helpdesk provides pro-bono support to help funders navigate the steps to publish DEI Data Standard data alongside their 360Giving grant data.
+
+Visit our website to find out [ways to get support with publishing](https://www.threesixtygiving.org/publishing/).

documentation/extensions/dei/index.md

@@ -0,0 +1,17 @@
+# Diversity, Equity, and Inclusion Extension
+
+This is the documentation for the DEI Extension provided by 360Giving. The DEI Extension is an extension to the 360Giving Data Standard which allows [Diversity, Equality, and Inclusion (DEI) Data Standard](https://www.funderscollaborativehub.org.uk/dei-data-standard) data to be published alongside [360Giving data](https://standard.threesixtygiving.org/en/latest/#), so that it can be validated and used in 360Giving tools.
+
+The DEI Data Standard is developed and maintained by the DEI Data Group, with technical support provided by 360Giving.
+
+The DEI Extension has been developed and is maintained by 360Giving to support new and existing 360Giving publishers to include DEI Data Standard data in their 360Giving data files.
+
+```eval_rst
+.. toctree::
+   :maxdepth: 3
+
+   governance
+   reference
+   guidance
+```
+

documentation/extensions/dei/reference.md

@@ -0,0 +1,96 @@
+# Technical Reference
+
+### Extension Structure Overview
+
+The DEI Extension defines new fields and structures for use within 360Giving data, for the purpose of publishing data using the DEI Data Standard within a file of 360Giving grant data. It also adds a number of new codelists to validate the contents of various fields.
+
+Some fields are used to share taxonomy and free text data collected through the application of the DEI Data Standard. There are also fields used to share metadata, which allow publishers to provide information about how the DEI Data Standard was applied in their specific context.
+
+The DEI Extension adds a new field called `deiDetails` to each grant. Inside `deiDetails` there are three fields: `leadership`, `mission`, and `project`. These represent the three application areas of the DEI Data Standard for grantmaking.
+
+Each of the `leadership`, `mission`, and `project` fields has the format of a `DeiApplicationArea` object. This is a new object defined by the DEI Extension so that each application area and its responses have the same structure and semantics, which makes the data more consistent and simpler to publish and use. `DeiApplicationArea` contains fields for important metadata to represent whether a question was asked (and how), as well as for sharing the responses to these questions.
+
+The extension structure is as follows:
+
+* `grants` (from [360Giving Data Standard](https://standard.threesixtygiving.org/en/latest/technical/reference/#))
+  * `deiDetails`
+    * `deiVersion`
+    * `purposes` 
+    * `leadership`, `mission`, and `project` (`DeiApplicationArea`)
+      * `askedStatus` (required)
+      * `replyStatus`
+      * `availableOptions`
+      * `additionalDetails`
+      * `response`
+        * `taxonomyCodes`
+        * `livedExperience`
+        * `Geography`
+
+In the extension, the `deiDetails` field is *optional for each grant*. However, when it is included then each `leadership`, `mission`, and `project` become *required for that grant*, and each of these then have a required `askedStatus` field.
+
+For a publisher of data using this extension this means that:
+
+* If there is no DEI information for the grant, you may omit the entire `deiDetails` field for that grant
+* If there is DEI information for the grant, then you must include the `deiDetails` field and provide at least a value in `askedStatus` for each of the `leadership`, `mission` and `project` application areas. This is to ensure a minimum level of context required for a user to interpret the data correctly.
+
+### Schema
+
+This section contains a reference for the Extension’s schema
+
+
+```eval_rst
+.. jsonschema-titles:: ../../extras/extensions/dei/360-giving-schema.json
+```
+
+### Codelists
+
+The extension adds several codelists to promote interoperability between datasets. These are all **closed** codelists, meaning that only values from the codelists may be used.
+
+#### Asked Status
+
+A codelist to declare whether DEI Data Standard questions were asked for this grant, and how.
+
+
+```eval_rst
+.. csv-table:: 
+   :file: ../../extras/extensions/dei/codelists/askedStatus.csv
+   :header-rows: 1
+   :widths: auto
+```
+
+#### Available Options
+
+A codelist to declare which answer options were available to the respondents.
+
+```eval_rst
+.. csv-table:: 
+   :file: ../../extras/extensions/dei/codelists/availableOptions.csv
+   :header-rows: 1
+   :widths: auto
+```
+
+#### Reply Status
+
+A codelist to declare whether a reply to DEI Data Standard questions was received or not.
+
+```eval_rst
+.. csv-table:: 
+   :file: ../../extras/extensions/dei/codelists/replyStatus.csv
+   :header-rows: 1
+   :widths: auto
+```
+
+#### Taxonomy Codes
+
+The authoritative source for the DEI Taxonomy codes vocabulary is the [DEI Data Standard](https://www.funderscollaborativehub.org.uk/collaborations/dei-data-standard) and should take precedence over all other sources of information about the DEI Taxonomy codes, including this codelist.
+
+The DEI Extension for 360Giving creates and maintains this codelist by taking the hierarchical codes defined in the DEI Data Standard, encoding them into JSON, and then generating this codelist automatically. This is necessary to provide a machine-readable codelist to support validation.
+
+Please see the [Guiding Principles](#guiding-principles) section for more information on how we respond to updates in the DEI Standard.
+
+```eval_rst
+.. csv-table:: 
+   :file: ../../extras/extensions/dei/codelists/taxonomyCodes.csv
+   :header-rows: 1
+   :widths: auto
+```

documentation/extensions/index.md

@@ -0,0 +1,20 @@
+# Extensions Documentation
+
+360Giving provides a core set of fields which are designed to make it useful and usable for people to collect, share, and analyse data about grantmaking.
+
+Publishers sometimes wish to collect and publish information which is not part of the 360Giving standard. The standard permits publishers to use any number of "additional fields" in data to meet their needs, but these additional fields are not recognised by the standard. This means that they cannot be validated using tools such as the Data Quality Tool and different publishers including the same information may struggle to coordinate around how to name and structure the additional data.
+
+Extensions address some of the issues with additional fields by introducing new optional fields and structures into 360Giving. Each extension formally defines fields, structures, and validation rules which are recognised by tooling such as the Data Quality Tool. This allows data using extensions to be validated, and also provides further standardisation which supports interoperability between different data sets.
+
+Extensions also address the issue of trying to support different use cases of 360Giving whilst maintaining the stability and integrity of the standard. A set of fields and features may be important to significant portion of the 360Giving community while there may not be enough demand to warrant adding the feature to the standard. Developing the features as an Extension allows for a meeting the needs of the community in a much shorter timescale, as well as test out how the features work in practice. It is possible that some Extensions may later be intergrated into the main standard.
+
+This section contains documentation about each extension for 360Giving.
+
+```eval_rst
+
+.. toctree::
+   :maxdepth: 3
+
+   dei/index
+
+```

documentation/index.md

@@ -37,6 +37,7 @@ This section features information about governance and change management of the
    guidance/index
    individuals/index
    technical/index
+   extensions/index
    about/index
 
 ```