-
Notifications
You must be signed in to change notification settings - Fork 192
Add GetString to ColumnDecimal.
#451
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
The mappings already work for the HTTP engine, although its inserts were limited to `float8`, now fixed to use proper numeric formatting. Add the mappings for the binary engine. Use strings to convert between the Postgres and ClickHouse formats to as to avoid differences in their implementations: ClickHouse stores data as `Int32`, `Int64`, or `Int128`, while Postgres...well, I have no idea what the internals are, but using the `numeric_in` and `numeric_out` functions avoid all issues. Conversion from `Decimal` values returned from ClickHouse requires converting its internal `Int128` into a string. The clickhouse-cpp library doesn't provide this functionality, [yet], so implement it here. Unfortunately, this approach doesn't work with clickhouse-cpp compiled into a dynamic library, presumably because it doesn't also compile and install the `absl` dynamic library. So make the build static by default on all platforms for now. Add a test that covers both the HTTP and binary engines. [yet]: ClickHouse/clickhouse-cpp#451
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull request overview
This PR adds a GetString method to the ColumnDecimal class that converts internal integer representations to human-readable decimal strings with proper scale and precision formatting.
Key Changes:
- Added
GetStringmethod toColumnDecimalclass that formats decimal values as strings - Added comprehensive test coverage for the new method across various decimal types and edge cases
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| clickhouse/columns/decimal.h | Declares the new GetString method with documentation |
| clickhouse/columns/decimal.cpp | Implements decimal-to-string conversion logic with proper handling of scale, precision, and negative values |
| ut/client_ut.cpp | Adds extensive test cases validating the string conversion for positive, negative, and edge-case decimal values |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
It converts the internal integer to a decimal string relative to the scale and precision.
3d089c7 to
6ff8e4b
Compare
|
I’m a bit on the fence about that. There’s a problem with this approach -- it’s five times slower than using |
I'll look into whether I can use that syntax in pg_clickhouse, but ISTM that including stringification makes a more complete API. Would it be more efficient to use a template to treat UIint32, UInt64, and UInt128 separately? |
|
I spent some time trying to use SELECT * FROM dec_http.decimals ORDER BY id;pg_clickhouse would send it as: But alas, the code is unaware of the data type of SQL expressions. So this Postgres query: SELECT avg(dec) FROM dec_http.decimals;Gets converted to: SELECT avg(toString("dec")) FROM decimal_test.decimalsWhich is obviously incorrect. It's far better to just get the Decimal value from a Block and to convert it to a string, as all the evaluation is done and ClickHouse has returned the proper value for the overall expression. I assume I could figure out how to fiddle with the Postgres query AST enough at some point to work out where and how to do the casting, but it would take quite a bit of effort. At this point, I think it's simpler to just rely on what ClickHouse sends back. And for better or worse, to convert a Decimal to a Postgres NUMERIC, I need a string. |
slabko
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As noted earlier, if we introduce such a function, its performance should be on par with ClickHouse’s toString; a convenience API should not be several times slower than the next obvious alternative.
A more serious issue, however, is that this PR breaks shared-library builds. Running:
cmake -S . -B build -D BUILD_SHARED_LIBS=YES
cmake --build build
results in:
/usr/bin/ld: ../contrib/absl/absl/libabsl_int128.a(int128.cc.o): relocation R_X86_64_32S against symbol `_ZTVSt9basic_iosIcSt11char_traitsIcEE@@GLIBCXX_3.4' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: failed to set dynamic section sizes: bad value
collect2: error: ld returned 1 exit status
Does |
|
The lack of -fPIC is the reason it doesn’t compile. absl needs to be compiled with -fPIC in order to use operator<<. So yes, -fPIC does help. |
It converts the internal integer to a decimal string relative to the scale and precision. Not a blocker for my projects (I copied the code), but I am getting an error when using
-D BUILD_SHARED_LIBS:Presumably
-D BUILD_SHARED_LIBSneeds to also build and install theabsllibrary.