Skip to content

Commit ce09f22

Browse files
authored
Beautify README (#3068)
1 parent 6e38698 commit ce09f22

4 files changed

Lines changed: 145 additions & 24 deletions

File tree

README.md

Lines changed: 145 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,41 +1,162 @@
1-
![SQLMesh logo](sqlmesh.png)
1+
<p align="center">
2+
<img src="sqlmesh.png" alt="SQLMesh logo" width="50%" height="50%">
3+
</p>
24

35
SQLMesh is a next-generation data transformation and modeling framework that is backwards compatible with dbt. It aims to be easy to use, correct, and efficient.
46

5-
SQLMesh enables data practitioners to efficiently run and deploy data transformations written in SQL or Python.
7+
SQLMesh enables data teams to efficiently run and deploy data transformations written in SQL or Python.
68

7-
Although SQLMesh will make your dbt projects more efficient, reliable, and maintainable, it is more than just a [dbt alternative](https://tobikodata.com/sqlmesh_for_dbt_1.html).
9+
It is more than just a [dbt alternative](https://tobikodata.com/reduce_costs_with_cron_and_partitions.html).
810

9-
## Select Features
10-
* [Semantic Understanding of SQL](https://tobikodata.com/semantic-understanding-of-sql.html)
11-
* Compile time error checking (for 10 different SQL dialects!)
12-
* Definitions using [simply SQL](https://sqlmesh.readthedocs.io/en/stable/concepts/models/sql_models/#sql-based-definition) (no need for redundant and confusing Jinja + YAML)
13-
* [Self documenting queries](https://tobikodata.com/metadata-everywhere.html) using native SQL Comments
14-
* Efficiency
15-
* Never builds a table [more than once](https://tobikodata.com/simplicity-or-efficiency-how-dbt-makes-you-choose.html)
16-
* Partition-based [incremental models](https://tobikodata.com/correctly-loading-incremental-data-at-scale.html)
17-
* Confidence
18-
* Plan / Apply workflow like [Terraform](https://www.terraform.io/) to understand potential impact of changes
19-
* Easy to use [CI/CD bot](https://sqlmesh.readthedocs.io/en/stable/integrations/github/)
20-
* Automatic [column level lineage](https://tobikodata.com/automatically-detecting-breaking-changes-in-sql-queries.html) and data contracts
21-
* [Unit tests](https://tobikodata.com/we-need-even-greater-expectations.html) and audits
11+
<p align="center">
12+
<img src="architecture_diagram.png" alt="Architecture Diagram" width="100%" height="100%">
13+
</p>
14+
15+
## Core Features
16+
<img src="https://github.com/TobikoData/sqlmesh-public-assets/blob/main/sqlmesh_plan_mode.gif" alt="SQLMesh Plan Mode">
17+
18+
> Get instant SQL impact analysis of your changes, whether in the CLI or in [SQLMesh Plan Mode](https://sqlmesh.readthedocs.io/en/stable/guides/ui/?h=modes#working-with-an-ide)
19+
20+
<details>
21+
<summary><b>Virtual Data Environments</b></summary>
22+
23+
* See a full diagram of how [Virtual Data Environments](https://whimsical.com/virtual-data-environments-MCT8ngSxFHict4wiL48ymz) work
24+
* [Watch this video to learn more](https://www.youtube.com/watch?v=weJH3eM0rzc)
25+
26+
</details>
27+
28+
* Plan / Apply workflow like [Terraform](https://www.terraform.io/) to understand potential impact of changes
29+
* Automatic [column level lineage](https://sqlmesh.readthedocs.io/en/stable/guides/ui/?h=column+lineage#lineage-module) and data contracts
30+
* Easy to use [CI/CD bot](https://sqlmesh.readthedocs.io/en/stable/integrations/github/)
31+
32+
<details>
33+
<summary><b>Efficiency and Testing</b></summary>
34+
35+
Running this command will generate a unit test file in the `tests/` folder: `test_stg_payments.yaml`
36+
37+
Runs a live query to generate the expected output of the model
38+
39+
```bash
40+
sqlmesh create_test tcloud_demo.stg_payments --query tcloud_demo.seed_raw_payments "select * from tcloud_demo.seed_raw_payments limit 5"
41+
42+
# run the unit test
43+
sqlmesh test
44+
```
45+
46+
```sql
47+
MODEL (
48+
name tcloud_demo.stg_payments,
49+
cron '@daily',
50+
grain payment_id,
51+
audits (UNIQUE_VALUES(columns = (
52+
payment_id
53+
)), NOT_NULL(columns = (
54+
payment_id
55+
)))
56+
);
57+
58+
SELECT
59+
id AS payment_id,
60+
order_id,
61+
payment_method,
62+
amount / 100 AS amount, /* `amount` is currently stored in cents, so we convert it to dollars */
63+
'new_column' AS new_column, /* non-breaking change example */
64+
FROM tcloud_demo.seed_raw_payments
65+
```
66+
67+
```yaml
68+
test_stg_payments:
69+
model: tcloud_demo.stg_payments
70+
inputs:
71+
tcloud_demo.seed_raw_payments:
72+
- id: 66
73+
order_id: 58
74+
payment_method: coupon
75+
amount: 1800
76+
- id: 27
77+
order_id: 24
78+
payment_method: coupon
79+
amount: 2600
80+
- id: 30
81+
order_id: 25
82+
payment_method: coupon
83+
amount: 1600
84+
- id: 109
85+
order_id: 95
86+
payment_method: coupon
87+
amount: 2400
88+
- id: 3
89+
order_id: 3
90+
payment_method: coupon
91+
amount: 100
92+
outputs:
93+
query:
94+
- payment_id: 66
95+
order_id: 58
96+
payment_method: coupon
97+
amount: 18.0
98+
new_column: new_column
99+
- payment_id: 27
100+
order_id: 24
101+
payment_method: coupon
102+
amount: 26.0
103+
new_column: new_column
104+
- payment_id: 30
105+
order_id: 25
106+
payment_method: coupon
107+
amount: 16.0
108+
new_column: new_column
109+
- payment_id: 109
110+
order_id: 95
111+
payment_method: coupon
112+
amount: 24.0
113+
new_column: new_column
114+
- payment_id: 3
115+
order_id: 3
116+
payment_method: coupon
117+
amount: 1.0
118+
new_column: new_column
119+
```
120+
</details>
121+
122+
* Never builds a table [more than once](https://tobikodata.com/simplicity-or-efficiency-how-dbt-makes-you-choose.html)
123+
* Partition-based [incremental models](https://tobikodata.com/correctly-loading-incremental-data-at-scale.html)
124+
* [Unit tests](https://tobikodata.com/we-need-even-greater-expectations.html) and audits
125+
126+
<details>
127+
<summary><b>Take SQL Anywhere</b></summary>
128+
Write SQL in any dialect and SQLMesh will transpile it to your target SQL dialect on the fly before sending it to the warehouse.
129+
<img src="transpile_example.png" alt="Transpile Example">
130+
</details>
131+
132+
* Compile time error checking and can transpile [10+ different SQL dialects](https://sqlmesh.readthedocs.io/en/stable/integrations/overview/#execution-engines)
133+
* Definitions using [simply SQL](https://sqlmesh.readthedocs.io/en/stable/concepts/models/sql_models/#sql-based-definition) (no need for redundant and confusing Jinja + YAML)
134+
* [Self documenting queries](https://tobikodata.com/metadata-everywhere.html) using native SQL Comments
22135
23136
For more information, check out the [website](https://sqlmesh.com) and [documentation](https://sqlmesh.readthedocs.io/en/stable/).
24137
25138
## Getting Started
26139
Install SQLMesh through [pypi](https://pypi.org/project/sqlmesh/) by running:
27140
28-
```pip install sqlmesh```
141+
```bash
142+
mkdir sqlmesh-example
143+
cd sqlmesh-example
144+
python -m venv .env
145+
source .env/bin/activate
146+
pip install sqlmesh
147+
sqlmesh init duckdb # get started right away with a local duckdb instance
148+
```
29149

30-
Follow the [tutorial](https://sqlmesh.readthedocs.io/en/stable/quick_start/) to learn how to use SQLMesh.
150+
Follow the [quickstart guide](https://sqlmesh.readthedocs.io/en/stable/quickstart/cli/#1-create-the-sqlmesh-project) to learn how to use SQLMesh. You already have a head start!
31151

32-
## Join our community
33-
We'd love to join you on your data journey. Connect with us in the following ways:
152+
## Join Our Community
153+
We want to ship better data with you. Connect with us in the following ways:
34154

35-
* Join the [Tobiko Slack community](https://tobikodata.com/slack) to ask questions, or just to say hi!
36-
* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new).
37-
* Send us an email at [hello@tobikodata.com](mailto:hello@tobikodata.com) with your questions or feedback.
155+
* Join the [Tobiko Slack Community](https://tobikodata.com/slack) to ask questions, or just to say hi!
156+
* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new)
157+
* Send us an email at [hello@tobikodata.com](mailto:hello@tobikodata.com) with your questions or feedback
158+
* Read our [blog](https://tobikodata.com/blog)
38159

39160
## Contribution
40-
Contributions in the form of issues or pull requests are greatly appreciated. [Read more](https://sqlmesh.readthedocs.io/en/stable/development/) about how to develop for SQLMesh.
161+
Contributions in the form of issues or pull requests are greatly appreciated. [Read more](https://sqlmesh.readthedocs.io/en/stable/development/) on how to contribute to SQLMesh open source.
41162

architecture_diagram.png

1.93 MB
Loading

sqlmesh.png

68.1 KB
Loading

transpile_example.png

258 KB
Loading

0 commit comments

Comments
 (0)