feat(postgres sink): Allow ignoring serial columns in target table

[chris-1o]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We have a target table in PostgreSQL with column id serial. The goal is to let PostgreSQL fill this value with an auto-incremented id to allow further structured analysis in the database.

However, Vector fills this column with NULL as id is not present in the data input for this sink. This also results in a PostgreSQL error as NULL is not allowed for serial columns.

This is caused by the current query:

INSERT INTO {table} SELECT * FROM jsonb_populate_recordset(NULL::{table}, $1)

Proposal

Add a configuration parameter that allows to define explicitly which columns to insert. The SQL would look something like:

INSERT INTO {table} (column1, column2)
SELECT column1, column2 FROM jsonb_populate_recordset(NULL::{table}, $1);

Version

0.52.0