|
1 | | - |
| 1 | +<p align="center"> |
| 2 | + <img src="sqlmesh.png" alt="SQLMesh logo" width="50%" height="50%"> |
| 3 | +</p> |
2 | 4 |
|
3 | 5 | SQLMesh is a next-generation data transformation and modeling framework that is backwards compatible with dbt. It aims to be easy to use, correct, and efficient. |
4 | 6 |
|
5 | | -SQLMesh enables data practitioners to efficiently run and deploy data transformations written in SQL or Python. |
| 7 | +SQLMesh enables data teams to efficiently run and deploy data transformations written in SQL or Python. |
6 | 8 |
|
7 | | -Although SQLMesh will make your dbt projects more efficient, reliable, and maintainable, it is more than just a [dbt alternative](https://tobikodata.com/sqlmesh_for_dbt_1.html). |
| 9 | +It is more than just a [dbt alternative](https://tobikodata.com/reduce_costs_with_cron_and_partitions.html). |
8 | 10 |
|
9 | | -## Select Features |
10 | | -* [Semantic Understanding of SQL](https://tobikodata.com/semantic-understanding-of-sql.html) |
11 | | - * Compile time error checking (for 10 different SQL dialects!) |
12 | | - * Definitions using [simply SQL](https://sqlmesh.readthedocs.io/en/stable/concepts/models/sql_models/#sql-based-definition) (no need for redundant and confusing Jinja + YAML) |
13 | | - * [Self documenting queries](https://tobikodata.com/metadata-everywhere.html) using native SQL Comments |
14 | | -* Efficiency |
15 | | - * Never builds a table [more than once](https://tobikodata.com/simplicity-or-efficiency-how-dbt-makes-you-choose.html) |
16 | | - * Partition-based [incremental models](https://tobikodata.com/correctly-loading-incremental-data-at-scale.html) |
17 | | -* Confidence |
18 | | - * Plan / Apply workflow like [Terraform](https://www.terraform.io/) to understand potential impact of changes |
19 | | - * Easy to use [CI/CD bot](https://sqlmesh.readthedocs.io/en/stable/integrations/github/) |
20 | | - * Automatic [column level lineage](https://tobikodata.com/automatically-detecting-breaking-changes-in-sql-queries.html) and data contracts |
21 | | - * [Unit tests](https://tobikodata.com/we-need-even-greater-expectations.html) and audits |
| 11 | +<p align="center"> |
| 12 | + <img src="architecture_diagram.png" alt="Architecture Diagram" width="100%" height="100%"> |
| 13 | +</p> |
| 14 | + |
| 15 | +## Core Features |
| 16 | +<img src="https://github.com/TobikoData/sqlmesh-public-assets/blob/main/sqlmesh_plan_mode.gif" alt="SQLMesh Plan Mode"> |
| 17 | + |
| 18 | +> Get instant SQL impact analysis of your changes, whether in the CLI or in [SQLMesh Plan Mode](https://sqlmesh.readthedocs.io/en/stable/guides/ui/?h=modes#working-with-an-ide) |
| 19 | +
|
| 20 | + <details> |
| 21 | + <summary><b>Virtual Data Environments</b></summary> |
| 22 | + |
| 23 | + * See a full diagram of how [Virtual Data Environments](https://whimsical.com/virtual-data-environments-MCT8ngSxFHict4wiL48ymz) work |
| 24 | + * [Watch this video to learn more](https://www.youtube.com/watch?v=weJH3eM0rzc) |
| 25 | + |
| 26 | + </details> |
| 27 | + |
| 28 | + * Plan / Apply workflow like [Terraform](https://www.terraform.io/) to understand potential impact of changes |
| 29 | + * Automatic [column level lineage](https://sqlmesh.readthedocs.io/en/stable/guides/ui/?h=column+lineage#lineage-module) and data contracts |
| 30 | + * Easy to use [CI/CD bot](https://sqlmesh.readthedocs.io/en/stable/integrations/github/) |
| 31 | + |
| 32 | +<details> |
| 33 | +<summary><b>Efficiency and Testing</b></summary> |
| 34 | + |
| 35 | +Running this command will generate a unit test file in the `tests/` folder: `test_stg_payments.yaml` |
| 36 | + |
| 37 | +Runs a live query to generate the expected output of the model |
| 38 | + |
| 39 | +```bash |
| 40 | +sqlmesh create_test tcloud_demo.stg_payments --query tcloud_demo.seed_raw_payments "select * from tcloud_demo.seed_raw_payments limit 5" |
| 41 | + |
| 42 | +# run the unit test |
| 43 | +sqlmesh test |
| 44 | +``` |
| 45 | + |
| 46 | +```sql |
| 47 | +MODEL ( |
| 48 | + name tcloud_demo.stg_payments, |
| 49 | + cron '@daily', |
| 50 | + grain payment_id, |
| 51 | + audits (UNIQUE_VALUES(columns = ( |
| 52 | + payment_id |
| 53 | + )), NOT_NULL(columns = ( |
| 54 | + payment_id |
| 55 | + ))) |
| 56 | +); |
| 57 | + |
| 58 | +SELECT |
| 59 | + id AS payment_id, |
| 60 | + order_id, |
| 61 | + payment_method, |
| 62 | + amount / 100 AS amount, /* `amount` is currently stored in cents, so we convert it to dollars */ |
| 63 | + 'new_column' AS new_column, /* non-breaking change example */ |
| 64 | +FROM tcloud_demo.seed_raw_payments |
| 65 | +``` |
| 66 | + |
| 67 | +```yaml |
| 68 | +test_stg_payments: |
| 69 | +model: tcloud_demo.stg_payments |
| 70 | +inputs: |
| 71 | + tcloud_demo.seed_raw_payments: |
| 72 | + - id: 66 |
| 73 | + order_id: 58 |
| 74 | + payment_method: coupon |
| 75 | + amount: 1800 |
| 76 | + - id: 27 |
| 77 | + order_id: 24 |
| 78 | + payment_method: coupon |
| 79 | + amount: 2600 |
| 80 | + - id: 30 |
| 81 | + order_id: 25 |
| 82 | + payment_method: coupon |
| 83 | + amount: 1600 |
| 84 | + - id: 109 |
| 85 | + order_id: 95 |
| 86 | + payment_method: coupon |
| 87 | + amount: 2400 |
| 88 | + - id: 3 |
| 89 | + order_id: 3 |
| 90 | + payment_method: coupon |
| 91 | + amount: 100 |
| 92 | +outputs: |
| 93 | + query: |
| 94 | + - payment_id: 66 |
| 95 | + order_id: 58 |
| 96 | + payment_method: coupon |
| 97 | + amount: 18.0 |
| 98 | + new_column: new_column |
| 99 | + - payment_id: 27 |
| 100 | + order_id: 24 |
| 101 | + payment_method: coupon |
| 102 | + amount: 26.0 |
| 103 | + new_column: new_column |
| 104 | + - payment_id: 30 |
| 105 | + order_id: 25 |
| 106 | + payment_method: coupon |
| 107 | + amount: 16.0 |
| 108 | + new_column: new_column |
| 109 | + - payment_id: 109 |
| 110 | + order_id: 95 |
| 111 | + payment_method: coupon |
| 112 | + amount: 24.0 |
| 113 | + new_column: new_column |
| 114 | + - payment_id: 3 |
| 115 | + order_id: 3 |
| 116 | + payment_method: coupon |
| 117 | + amount: 1.0 |
| 118 | + new_column: new_column |
| 119 | +``` |
| 120 | +</details> |
| 121 | +
|
| 122 | +* Never builds a table [more than once](https://tobikodata.com/simplicity-or-efficiency-how-dbt-makes-you-choose.html) |
| 123 | +* Partition-based [incremental models](https://tobikodata.com/correctly-loading-incremental-data-at-scale.html) |
| 124 | +* [Unit tests](https://tobikodata.com/we-need-even-greater-expectations.html) and audits |
| 125 | +
|
| 126 | +<details> |
| 127 | +<summary><b>Take SQL Anywhere</b></summary> |
| 128 | +Write SQL in any dialect and SQLMesh will transpile it to your target SQL dialect on the fly before sending it to the warehouse. |
| 129 | +<img src="transpile_example.png" alt="Transpile Example"> |
| 130 | +</details> |
| 131 | +
|
| 132 | +* Compile time error checking and can transpile [10+ different SQL dialects](https://sqlmesh.readthedocs.io/en/stable/integrations/overview/#execution-engines) |
| 133 | +* Definitions using [simply SQL](https://sqlmesh.readthedocs.io/en/stable/concepts/models/sql_models/#sql-based-definition) (no need for redundant and confusing Jinja + YAML) |
| 134 | +* [Self documenting queries](https://tobikodata.com/metadata-everywhere.html) using native SQL Comments |
22 | 135 |
|
23 | 136 | For more information, check out the [website](https://sqlmesh.com) and [documentation](https://sqlmesh.readthedocs.io/en/stable/). |
24 | 137 |
|
25 | 138 | ## Getting Started |
26 | 139 | Install SQLMesh through [pypi](https://pypi.org/project/sqlmesh/) by running: |
27 | 140 |
|
28 | | -```pip install sqlmesh``` |
| 141 | +```bash |
| 142 | +mkdir sqlmesh-example |
| 143 | +cd sqlmesh-example |
| 144 | +python -m venv .env |
| 145 | +source .env/bin/activate |
| 146 | +pip install sqlmesh |
| 147 | +sqlmesh init duckdb # get started right away with a local duckdb instance |
| 148 | +``` |
29 | 149 |
|
30 | | -Follow the [tutorial](https://sqlmesh.readthedocs.io/en/stable/quick_start/) to learn how to use SQLMesh. |
| 150 | +Follow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/#1-create-the-sqlmesh-project) to learn how to use SQLMesh. You already have a head start! |
31 | 151 |
|
32 | | -## Join our community |
33 | | -We'd love to join you on your data journey. Connect with us in the following ways: |
| 152 | +## Join Our Community |
| 153 | +We want to ship better data with you. Connect with us in the following ways: |
34 | 154 |
|
35 | | -* Join the [Tobiko Slack community](https://tobikodata.com/slack) to ask questions, or just to say hi! |
36 | | -* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new). |
37 | | -* Send us an email at [hello@tobikodata.com](mailto:hello@tobikodata.com) with your questions or feedback. |
| 155 | +* Join the [Tobiko Slack Community](https://tobikodata.com/slack) to ask questions, or just to say hi! |
| 156 | +* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new) |
| 157 | +* Send us an email at [hello@tobikodata.com](mailto:hello@tobikodata.com) with your questions or feedback |
| 158 | +* Read our [blog](https://tobikodata.com/blog) |
38 | 159 |
|
39 | 160 | ## Contribution |
40 | | -Contributions in the form of issues or pull requests are greatly appreciated. [Read more](https://sqlmesh.readthedocs.io/en/stable/development/) about how to develop for SQLMesh. |
| 161 | +Contributions in the form of issues or pull requests are greatly appreciated. [Read more](https://sqlmesh.readthedocs.io/en/stable/development/) on how to contribute to SQLMesh open source. |
41 | 162 |
|
0 commit comments