Skip to content

Conversation

@claudevdm
Copy link
Collaborator

@claudevdm claudevdm commented Nov 24, 2025

timestamp-nanos avro logical type was introduced in avro 1.12, but beam cannot upgrade avro versions until java 8 support is dropped #34993

There are workarounds here to handle the timestamp-nanos logical type until we can update avro version.

AvroUtils changes:

  • toBeamFieldType(): Recognizes timestamp-nanos property and maps to Timestamp.NANOS
  • toAvroSchema(): Converts Timestamp.NANOS to Avro schema with timestamp-nanos logical type
  • toGenericRecord(): Converts Instant values to nanoseconds since epoch
  • toBeamRowStrict(): Converts nanosecond values back to Instant objects

BigQueryAvroUtils changes:

  • Refactored timestamp formatting into TimestampPrecision enum
  • Added NANOSECONDS precision level alongside existing MILLISECONDS and MICROSECONDS
  • Updated convertRequiredField() to handle timestamp-nanos logical type
  • Updated formatDatetime() to dynamically format fractional seconds (3, 6, or 9 digits)
  • Updated fromGenericAvroSchema() to map timestamp-nanos to BigQuery TIMESTAMP type

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@claudevdm claudevdm force-pushed the bq-avro-timestampnano branch from d234446 to 1d0a0ef Compare November 25, 2025 15:50
@claudevdm claudevdm changed the title Add timestamp-nanos avro logical type support in bigquery utils. Add timestamp-nanos avro logical type support in bigquery avro utils. Nov 25, 2025
@claudevdm claudevdm force-pushed the bq-avro-timestampnano branch from 88b51f4 to e8c61c5 Compare November 25, 2025 19:55
@claudevdm claudevdm marked this pull request as ready for review November 26, 2025 16:26
@claudevdm
Copy link
Collaborator Author

/gemini review

Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for the timestamp-nanos Avro logical type as a workaround until the Avro version can be upgraded. The changes are primarily in AvroUtils and BigQueryAvroUtils. The implementation is functionally correct and includes good test coverage for edge cases. However, some parts of the implementation rely on implicit behaviors of java.time.Instant, which can be obscure. I've provided a few suggestions to make the code more explicit and maintainable. Additionally, I've suggested a cleaner way to construct an Avro schema programmatically instead of parsing a JSON string.

@github-actions
Copy link
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@claudevdm
Copy link
Collaborator Author

R: @Abacn
R: @ahmedabu98

@github-actions
Copy link
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@Abacn Abacn self-assigned this Nov 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants