Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Initial skeleton drafts
  • Loading branch information
jtcohen6 committed Feb 20, 2023
commit 79c4cfab873bb66d6a45b35319e27644bbaf43e0
36 changes: 28 additions & 8 deletions website/dbt-versions.js
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,34 @@ exports.versions = [
]

exports.versionedPages = [
{
"page": "docs/collaborate/publish/model-contracts",
"firstVersion": "1.5",
},
{
"page": "docs/collaborate/publish/model-access",
"firstVersion": "1.5",
},
{
"page": "docs/collaborate/publish/model-versions",
"firstVersion": "1.5",
},
{
"page": "reference/resource-configs/contract",
"firstVersion": "1.5",
},
{
"page": "reference/resource-properties/constraints",
"firstVersion": "1.5",
},
{
"page": "reference/dbt-jinja-functions/local-md5",
"firstVersion": "1.4",
},
{
"page": "reference/warehouse-setups/fal-setup",
"firstVersion": "1.3",
},
Comment on lines +51 to +57
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No functional change here, I just reordered the list so it's in a consistent (descending) order by firstVersion. I figure, in the future, we could remove items from this list as we deprecate older versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

{
"page": "reference/dbt-jinja-functions/set",
"firstVersion": "1.2",
Expand Down Expand Up @@ -55,12 +83,4 @@ exports.versionedPages = [
"page": "reference/dbt-jinja-functions/print",
"firstVersion": "1.1",
},
{
"page": "reference/dbt-jinja-functions/local-md5",
"firstVersion": "1.4",
},
{
"page": "reference/warehouse-setups/fal-setup",
"firstVersion": "1.3",
},
]
61 changes: 61 additions & 0 deletions website/docs/docs/collaborate/publish/model-access.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: "Model access"
---

:::info Beta functionality
This functionality is new in v1.5! These docs exist to provide a high-level overview of what's to come. Specific syntax is liable to change.

For more details, and to leave your feedback, check out the GitHub discussion:
* ["Model groups & access" (dbt-core#6730)](https://github.com/dbt-labs/dbt-core/discussions/6730)
:::

## Related documentation
* TK: `groups`
* TK: `access` modifiers

### Groups

Models can be grouped together under a common designation, with a shared owner.

Why define model `groups`?
- It turns implicit relationships into an explicit grouping
- It enables you to mark certain models as "private," for use _only_ within that group

### Access modifiers

Some models (not all of them) are designed to be shared across groups.

https://en.wikipedia.org/wiki/Access_modifiers

| Keyword | Meaning |
|-----------|----------------------|
| private | same group |
| protected | same project/package |
| public | everybody* |

By default, all models are "protected." This means that they can be referenced by other models in the same project.

:::info Under construction 🚧
More to come! The syntax below is suggestive only, it does not yet work.
:::

<File name="models/marts/customers.yml">

```yaml
groups:
- name: cx
owner:
name: Customer Success Team
email: [email protected]

models:
- name: dim_customers
group: cx
access: public
# this is an intermediate transformation -- relevant to the CX team only
- name: int__customer_history_rollup
group: cx
access: private
```

</File>
84 changes: 84 additions & 0 deletions website/docs/docs/collaborate/publish/model-contracts.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,84 @@
---
title: "Model contracts"
---

:::info Beta functionality
This functionality is new in v1.5! These docs exist to provide a high-level overview of what's to come. Specific syntax is liable to change.

For more details, and to leave your feedback, check out the GitHub discussion:
* ["Model contracts" (dbt-core#6726)](https://github.com/dbt-labs/dbt-core/discussions/6726)
:::

## Related documentation
* [`contract`](resource-configs/contract)
* [`columns`](resource-properties/columns)
* [`constraints`](resource-properties/constraints)

## Why define a contract?

Defining a dbt model is as easy as writing a SQL `select` statement, or a Python Data Frame transformation. Your query naturally produces a dataset with columns of names and types, based on the columns you're selecting and the transformations you're applying.

While this is great for quick & iterative development, for some models, constantly changing the shape of the model's returned dataset poses a risk, when there are other people and processes querying that model. It's better to define a set of **upfront guarantees** about the shape of your model. We call this set of guarantees a "contract." While building your model, dbt will verify that your model's transformation will produce a dataset matching up with its contract—or it will fail to build.

## How to define a contract

Let's say you have a model with a query like:

<File name="models/marts/dim_customers.sql">

```sql
-- lots of SQL

final as (

select
-- lots of columns
from ...

)

select * from final
```
</File>

Your contract **must** include every column's `name` and `data_type` (where `data_type` matches the type understood by your data platform). If your model is being materialized as `table` or `incremental`, you may optionally specify that certain columns must be `not_null` (i.e. contain zero null values). Depending on your data platform, you may also be able to define additional `constraints` that are enforced while the model is being built.

Finally, you configure your model with `contract: true`.

<File name="models/marts/customers.yml">

```yaml
models:
- name: dim_customers
config:
contract: true
columns:
- name: customer_id
data_type: int
not_null: true
- name: customer_name
data_type: string
...
```

</File>

When building a model with a defined contract, dbt will do two things differently:
1. dbt will run a "pre-flight" check to ensure that the model's query will return a set of columns with names and data types matching the ones you have defined.
2. dbt will pass the column names, types, `not_null` and other constraints into the DDL statements it submits to the data platform, where they will be enforced while building the table.

## FAQs

### Which models should have contracts?

Any model can define a contract. It's especially important to define contracts for "public" models that are being shared with other groups, teams, and (soon) dbt projects.

### How are contracts different from tests?

A model's contract defines the **shape** of the returned dataset.

[Tests](tests) are a more flexible mechanism for validating the content of your model. So long as you can write the query, you can run the test. In blue/green deployments (docs link TK), ...

In the parallel for software APIs:
- The structure of the API response is the contract
- Quality and reliability ("uptime") are also **crucial**, but not part of the contract per se.
28 changes: 28 additions & 0 deletions website/docs/docs/collaborate/publish/model-versions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
title: "Model versions"
---

:::info Beta functionality
This functionality is new in v1.5! These docs exist to provide a high-level overview of what's to come. Specific syntax is liable to change.

For more details, and to leave your feedback, check out the GitHub discussion:
* ["Model versions" (dbt-core#6736)](https://github.com/dbt-labs/dbt-core/discussions/6736)
:::

API versioning is **a hard problem** in software engineering. It's also very important. Our goal is to _make a hard thing possible_.

## Related documentation
* TK: `version` & `latest` (_not_ [this one](project-configs/version))
* TK: `deprecation_date`

## Why version a model?

If a model defines a ["contract"](model-contracts) (a set of guarantees for its structure), it's also possible to change that model's contract in a way that "breaks" the previous set of guarantees.

One approach is to force every consumer of the model to immediately handle the breaking change, as soon as it's deployed to production. While this may work at smaller organizations, or while iterating on an immature set of data models, it doesn't scale much beyond that.

Instead, the owner of the model can create a **new version**, and provide a **deprecation window**, during which consumers can migrate from the old version to the new.

In the meantime, anywhere that model is used downstream, it can be referenced at a specific version.

When a model is reaching its deprecation date, consumers of that model will hear about it. When the date is reached, it goes away.
Original file line number Diff line number Diff line change
@@ -1,10 +1,18 @@
---
resource_types: [models]
datatype: "{<dictionary>}"
default_value: {constraints_enabled: false}
id: "constraints_enabled"
default_value: {contract: false}
id: "contract"
---
<!-- <VersionBlock firstVersion="1.4"> -->

:::info Beta functionality
This functionality is new in v1.5! These docs exist to provide a high-level overview of what's to come. Specific syntax is liable to change.
:::

## Related documentation
- [Model contracts](publish/model-contracts)

<!-- TODO: move some of this content elsewhere, and update to reflect new proposed syntax -->

# Definition

Expand Down
6 changes: 6 additions & 0 deletions website/docs/reference/resource-properties/constraints.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
---
resource_types: [models]
datatype: "{dictionary}"
---

https://github.com/dbt-labs/dbt-core/issues/6750
11 changes: 10 additions & 1 deletion website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,15 @@ const sidebarSettings = {
"docs/collaborate/manage-access/audit-log",
],
}, // Manage access
{
type: "category",
label: "Publishing models",
items: [
"docs/collaborate/publish/model-contracts",
"docs/collaborate/publish/model-access",
"docs/collaborate/publish/model-versions",
],
}, // publishing models
],
},
{
Expand Down Expand Up @@ -447,7 +456,7 @@ const sidebarSettings = {
"reference/resource-configs/database",
"reference/resource-configs/enabled",
"reference/resource-configs/full_refresh",
"reference/resource-configs/constraints_enabled",
"reference/resource-configs/contract",
"reference/resource-configs/grants",
"reference/resource-configs/docs",
"reference/resource-configs/persist_docs",
Expand Down