You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -14,7 +14,7 @@ Follow the [tutorial](https://sqlmesh.readthedocs.io/en/stable/quick_start/) to
14
14
## Join our community
15
15
We'd love to join you on your data journey. Connect with us in the following ways:
16
16
17
-
* Join the [Tobiko Slack community](https://join.slack.com/t/tobiko-data/shared_invite/zt-1ma66d79v-a4dbf4DUpLAQJ8ptQrJygg) to ask questions, or just to say hi!
17
+
* Join the [Tobiko Slack community](https://join.slack.com/t/tobiko-data/shared_invite/zt-1tofr385z-vi~hDISNABiYIgkfGM3Khg) to ask questions, or just to say hi!
18
18
* File an issue on our [GitHub](https://github.com/TobikoData/sqlmesh/issues/new).
19
19
* Send us an email at [hello@tobikodata.com](hello@tobikodata.com) with your questions or feedback.
Copy file name to clipboardExpand all lines: docs/integrations/engines.md
+11-11Lines changed: 11 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -25,7 +25,7 @@
25
25
## BigQuery - Airflow Scheduler
26
26
**Engine Name:**`bigquery`
27
27
28
-
In order to share a common implementation across local and Airflow, SQLMesh BigQuery implements its own hook and operator.
28
+
In order to share a common implementation across local and Airflow, SQLMesh BigQuery implements its own hook and operator.
29
29
30
30
To enable support for this operator, the Airflow BigQuery provider package should be installed on the target Airflow cluster along with SQLMesh with the BigQuery extra:
If your project contains Python models that use PySpark DataFrames AND you are using the built-in scheduler, then you must run plan/apply on a Databricks cluster.
50
+
If your project contains Python models that use PySpark DataFrames AND you are using the built-in scheduler, then you must run plan/apply on a Databricks cluster.
51
51
This can be done using the [Notebook magic](../reference/notebook.md) or by using the [CLI](../reference/cli.md).
52
-
This is something we are looking into improving — please leave us feedback in [our Slack channel](https://join.slack.com/t/tobiko-data/shared_invite/zt-1ma66d79v-a4dbf4DUpLAQJ8ptQrJygg) if this impacts you.
52
+
This is something we are looking into improving — please leave us feedback in [our Slack channel](https://join.slack.com/t/tobiko-data/shared_invite/zt-1tofr385z-vi~hDISNABiYIgkfGM3Khg) if this impacts you.
53
53
A potential workaround until this support is added is to use [Databricks Connect](https://docs.databricks.com/dev-tools/databricks-connect.html). This will make it look like you are running on a cluster, and should theoretically work.
54
54
55
55
Databricks has a few options for connection types to choose from:
56
56
### Type: databricks (Recommended)
57
-
This type will automatically detect if you are running in an environment that already has a SparkSession defined.
58
-
If it detects a SparkSession, then it assumes this is a Databricks SparkSession and uses that.
57
+
This type will automatically detect if you are running in an environment that already has a SparkSession defined.
58
+
If it detects a SparkSession, then it assumes this is a Databricks SparkSession and uses that.
59
59
If it doesn't detect a SparkSession, then it will use the connection configuration to connect to Databricks over
60
-
the [Databricks SQL Connector](https://docs.databricks.com/dev-tools/python-sql-connector.html).
60
+
the [Databricks SQL Connector](https://docs.databricks.com/dev-tools/python-sql-connector.html).
61
61
See [databricks_sql configuration](#type--databrickssql) for the connection configuration.
62
62
63
63
### Type: databricks_spark_session
64
-
This connection type assumes that wherever you are running you have access to a Databricks SparkSession.
64
+
This connection type assumes that wherever you are running you have access to a Databricks SparkSession.
65
65
This will simplify the required configuration to run since you will not need to provide connection configuration.
66
66
67
67
### Type: databricks_sql
@@ -86,7 +86,7 @@ Databricks has multiple operators to help differentiate running a SQL query vs.
86
86
87
87
When evaluating models, the SQLMesh Databricks integration implements the [DatabricksSubmitRunOperator](https://airflow.apache.org/docs/apache-airflow-providers-databricks/1.0.0/operators.html). This is needed to be able to run either SQL or Python scripts on the Databricks cluster.
88
88
89
-
When performing environment management operations, the SQLMesh Databricks integration is similar to the [DatabricksSqlOperator](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/sql.html#databrickssqloperator), and relies on the same [DatabricksSqlHook](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/_api/airflow/providers/databricks/hooks/databricks_sql/index.html#airflow.providers.databricks.hooks.databricks_sql.DatabricksSqlHook) implementation.
89
+
When performing environment management operations, the SQLMesh Databricks integration is similar to the [DatabricksSqlOperator](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/sql.html#databrickssqloperator), and relies on the same [DatabricksSqlHook](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/_api/airflow/providers/databricks/hooks/databricks_sql/index.html#airflow.providers.databricks.hooks.databricks_sql.DatabricksSqlHook) implementation.
90
90
All environment management operations are SQL-based, and the overhead of submitting jobs can be avoided.
91
91
92
92
### Engine: `databricks-submit`
@@ -95,7 +95,7 @@ Whether evaluating models or performing environment management operations, the S
95
95
96
96
### Engine: `databricks-sql`
97
97
98
-
Forces the SQLMesh Databricks integration to use the operator based on the [DatabricksSqlOperator](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/sql.html#databrickssqloperator) for all operations. If your project is pure SQL operations, then this is an option.
98
+
Forces the SQLMesh Databricks integration to use the operator based on the [DatabricksSqlOperator](https://airflow.apache.org/docs/apache-airflow-providers-databricks/stable/operators/sql.html#databrickssqloperator) for all operations. If your project is pure SQL operations, then this is an option.
99
99
100
100
To enable support for this operator, the Airflow Databricks provider package should be installed on the target Airflow cluster along with the SQLMesh package with databricks extra as follows:
In order to share a common implementation across local and Airflow, SQLMesh Bigquery implements its own hook and operator.
215
+
In order to share a common implementation across local and Airflow, SQLMesh Bigquery implements its own hook and operator.
216
216
217
217
To enable support for this operator, the Airflow BigQuery provider package should be installed on the target Airflow cluster along with SQLMesh with the Redshift extra:
Copy file name to clipboardExpand all lines: docs/integrations/github.md
+2-2Lines changed: 2 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -2,8 +2,8 @@
2
2
3
3
SQLMesh's Github Actions integration will allow you to add a SQLMesh CI/CD bot to any Github project using [Github Actions](https://github.com/features/actions). The bot will automatically run [plan/apply](../concepts/plans.md) to an [environment](../concepts/environments.md) based on the code in a pull request.
4
4
5
-
This will be done without copying or rebuilding data using SQLMesh's [Virtual Data Environments](../concepts/glossary.md#virtual-environments).
5
+
This will be done without copying or rebuilding data using SQLMesh's [Virtual Data Environments](../concepts/glossary.md#virtual-environments).
6
6
Once approved, the CI/CD bot will automatically run [plan/apply](../concepts/plans.md) to the production environment and merge the PR upon completion.
7
7
This allows you to always have your main branch and prod environments in sync.
8
8
9
-
We will be launching this CI/CD bot soon — in the meantime, please leave any feedback or questions in [our Slack channel](https://join.slack.com/t/tobiko-data/shared_invite/zt-1ma66d79v-a4dbf4DUpLAQJ8ptQrJygg)!
9
+
We will be launching this CI/CD bot soon — in the meantime, please leave any feedback or questions in [our Slack channel](https://join.slack.com/t/tobiko-data/shared_invite/zt-1tofr385z-vi~hDISNABiYIgkfGM3Khg)!
Copy file name to clipboardExpand all lines: posts/virtual_data_environments.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -158,4 +158,4 @@ With **Virtual Data Environments**, SQLMesh is able to provide fully **isolated*
158
158
159
159
To streamline deploying changes to production, our team is about to release the SQLMesh [CI/CD bot](https://github.com/TobikoData/sqlmesh/blob/main/docs/integrations/github.md), which will help automate this process.
160
160
161
-
Don't miss out - join our [Slack channel](https://join.slack.com/t/tobiko-data/shared_invite/zt-1ma66d79v-a4dbf4DUpLAQJ8ptQrJygg) and stay tuned!
161
+
Don't miss out - join our [Slack channel](https://join.slack.com/t/tobiko-data/shared_invite/zt-1tofr385z-vi~hDISNABiYIgkfGM3Khg) and stay tuned!
0 commit comments