You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: contents/docs/cdp/batch-exports/redshift.md
+13-13Lines changed: 13 additions & 13 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,42 +8,42 @@ availability:
8
8
enterprise: full
9
9
---
10
10
11
-
Batch exports can be used to export data to Redshift, Amazon's data warehouse product.
11
+
Batch exports can be used to export data to Redshift, Amazon's data warehouse product. There are two SQL commands that can be used to export data: `COPY` and `INSERT`. Essentially, `COPY` works by reading and writing entire files in parallel from an S3 bucket, where as `INSERT` works by writing batches of rows. In general, we recommend `COPY` as it offers significantly better performance, but `INSERT` is provided as an alternative that requires less setup.
12
12
13
13
## Creating the batch export
14
14
15
15
1. Click [Data pipelines](https://app.posthog.com/pipeline) in the navigation and go to the **Destinations** tab.
16
16
2. Click **+ New destination** in the top-right corner.
17
17
3. Search for **Redshift**.
18
-
4. Click the **+ Create** button.
18
+
4. Click the **+ Create** button.
19
19
5. Fill in the necessary [configuration details](#redshift-configuration).
20
20
6. Finalize the creation by clicking on "Create".
21
21
7. Done! The batch export will schedule its first run on the start of the next period.
22
22
23
23
## Redshift configuration
24
24
25
-
Configuring a batch export targeting Redshift requires the following Redshift-specific configuration values:
25
+
Regardless of the SQL command used, configuring a batch export targeting Redshift requires the following Redshift-specific configuration values:
26
26
27
27
-**User:** A Redshift user name with permissions to insert data into the specified table and, if the table does not exist, permissions to create the table.
28
-
29
28
-**Password:** The password for the Redshift user specified.
30
-
31
29
-**Host:** The endpoint of your Redshift cluster, excluding the port number and database name.
32
-
33
30
-**Port:** The port number on which the Redshift cluster is listening (default is 5439).
34
-
35
31
-**Database:** The name of the Redshift database to which the data is to be exported.
36
-
37
32
-**Schema:** The name of the schema within the database. This determines where the table for exporting data will be located.
38
-
39
33
-**Table name:** The name of the table where the data will be inserted.
40
-
41
-
-**Properties data type:** The data type configurations for the `properties`, `set`, and `set_once` columns, either `VARCHAR(65535)` or `SUPER`.
42
-
34
+
-**Semi-structured data type:** The Redshift data type to use to export semi-structured data, like data in the `properties`, `set`, and `set_once` columns of the `events` model. This can be either `SUPER` or `VARCHAR(65535)`. We recommend `SUPER` as `VARCHAR` limits the document size to 65535 bytes, whereas with a `SUPER` type the limit applies per value in the document, instead of to the entire document.
43
35
-**Events to exclude:** A list of events to omit from the exported data.
44
-
45
36
-**Events to include:** A list of events to include in the exported data. If added, only these events will be exported.
46
37
38
+
Additionally, when choosing to `COPY` data to Redshift, the batch export requires access to an S3 bucket, thus the additional configuration is required:
39
+
***Bucket name:** The name of the S3 bucket. The bucket must already exist.
40
+
***Region:** The AWS region where the bucket is located. This must be in the same region as your Redshift instance.
41
+
***Key prefix:** A key prefix to use for the batch export files. The files will be cleaned up after the batch export finishes.
42
+
***AWS Access Key ID:** An AWS access key ID with access to the S3 bucket.
43
+
***AWS Secret Access Key:** An AWS secret access key with access to the S3 bucket.
44
+
45
+
Finally, Redshift needs access to your S3 bucket too, and we must provide it with either AWS credentials (which could be the same PostHog uses to access your bucket) or an IAM role that Redshift can assume with sufficient privileges to read from your S3 bucket. We recommend using an IAM role to avoid the need for another set of credentials.
46
+
47
47
## Models
48
48
49
49
> **Note:** New fields may be added to these models over time. To maintain consistency, these fields are not automatically added to the destination tables. If a particular field is missing in your Redshift tables, you can manually add the field, and it will be populated in future exports.
0 commit comments