From 6871551de650ee36464699ec6cbb0f124676f282 Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Fri, 9 Feb 2024 12:15:33 +0200 Subject: [PATCH 1/4] fix(deps): Update dependency io.cloudquery:plugin-pb-java to v0.0.14 (#178) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | io.cloudquery:plugin-pb-java | dependencies | patch | `0.0.12` -> `0.0.14` | --- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- lib/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/build.gradle b/lib/build.gradle index c16beae..cc2588d 100644 --- a/lib/build.gradle +++ b/lib/build.gradle @@ -38,7 +38,7 @@ dependencies { implementation "io.grpc:grpc-stub:1.60.1" implementation "io.grpc:grpc-services:1.60.1" implementation "io.grpc:grpc-testing:1.60.1" - implementation "io.cloudquery:plugin-pb-java:0.0.12" + implementation "io.cloudquery:plugin-pb-java:0.0.14" implementation "org.apache.arrow:arrow-memory-core:12.0.1" implementation "org.apache.arrow:arrow-vector:12.0.1" From 4ada7bcade9cf94043a9ba99c86c2fef43c3a645 Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Fri, 9 Feb 2024 12:57:55 +0200 Subject: [PATCH 2/4] fix(deps): Update grpc-java monorepo to v1.61.1 (#172) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [io.grpc:grpc-testing](https://github.com/grpc/grpc-java) | dependencies | minor | `1.60.1` -> `1.61.1` | | [io.grpc:grpc-services](https://github.com/grpc/grpc-java) | dependencies | minor | `1.60.1` -> `1.61.1` | | [io.grpc:grpc-stub](https://github.com/grpc/grpc-java) | dependencies | minor | `1.60.1` -> `1.61.1` | | [io.grpc:grpc-protobuf](https://github.com/grpc/grpc-java) | dependencies | minor | `1.60.1` -> `1.61.1` | --- ### Release Notes
grpc/grpc-java (io.grpc:grpc-testing) ### [`v1.61.1`](https://github.com/grpc/grpc-java/releases/tag/v1.61.1) ##### Bug Fixes xds: Fix a bug in `WeightedRoundRobinLoadBalancer` policy that could raise `NullPointerException` and further cause channel panic when picking a subchannel. This bug can only be triggered when connection can not be established and the channel reports `TRANSIENT_FAILURE` state. ([#​10868](https://github.com/grpc/grpc-java/issues/10868)) ### [`v1.61.0`](https://github.com/grpc/grpc-java/releases/tag/v1.61.0) ##### API Changes - Remove unused experimental API ManagedChannelBuilder.enableFullStreamDecompression ([#​10744](https://github.com/grpc/grpc-java/issues/10744)) - api: Deprecate LoadBalancer.EMPTY_PICKER added in 1.58.0 in favor of FixedResultPicker ([`860b5cb`](https://github.com/grpc/grpc-java/commit/860b5cb1f)) ##### New Features - binder: Experimental support for asynchronous security policies ([#​10566](https://github.com/grpc/grpc-java/issues/10566)) ##### Improvements - core: reduce CompositeReadableBuffer allocation ([#​3279](https://github.com/grpc/grpc-java/issues/3279)) - core: Improve error message clarity when a channel leak is detected ([`201893f`](https://github.com/grpc/grpc-java/commit/201893f5e)) - util: use shared index across `round_robin` pickers ([`dca89b2`](https://github.com/grpc/grpc-java/commit/dca89b25b)). This makes its implementation more similar to `weighted_round_robin`. - xds: Implement ADS stream flow control mechanism ([#​10674](https://github.com/grpc/grpc-java/issues/10674)). This limits the maximum memory consumed if the control plane sends updates more rapidly than they can be processed. ##### Bug Fixes - core: Check outbound maximum message size for the compressed size in addition to the already-checked uncompressed size ([#​10739](https://github.com/grpc/grpc-java/issues/10739)). Fixed the status code to be RESOURCE_EXHAUSTED instead of UNKNOWN. - util: Fix NPE when multiple addresses are in an address group for petiole load balancer policies ([#​10769](https://github.com/grpc/grpc-java/issues/10769)) - util: Disable publishing of fixtures ([`8ac43dd`](https://github.com/grpc/grpc-java/commit/8ac43dd81)). The Gradle test fixtures are for use by grpc-java's internal tests. - okhttp: Ignore known conscrypt socket close issue ([#​10812](https://github.com/grpc/grpc-java/issues/10812)). This stops an exception from being thrown when a known Conscrypt synchronization issue happens. ##### Dependencies - Drop support for Bazel 5 ([`55a9c01`](https://github.com/grpc/grpc-java/commit/55a9c012c)). Bazel 7 is available, and Protobuf has already dropped support for Bazel 5. - Change many compile deps to runtime deps ([`d6830d7`](https://github.com/grpc/grpc-java/commit/d6830d7f9)). This reduces the transitive classes "leaked" into the compile classpath. In particular, grpc-core (`io.grpc.internal`) will be less frequently included transitively at compile time. - Upgrade dependencies ([`c985797`](https://github.com/grpc/grpc-java/commit/c985797d9)) - Protobuf to 3.25.1 - auto-value-annotations to 1.10.4 - error_prone_annotations to 2.23.0 - proto-google-common-protos to 2.29.0 - google-cloud-logging to 3.15.14 - guava to 32.1.3-android - okio to 3.4.0 ##### Acknowledgements - [@​Gordiychuk](https://github.com/Gordiychuk) - [@​jroper](https://github.com/jroper) - [@​jyane](https://github.com/jyane) - [@​ulfjack](https://github.com/ulfjack) ### [`v1.60.2`](https://github.com/grpc/grpc-java/releases/tag/v1.60.2) ##### Bug Fixes xds: Fix a bug in `WeightedRoundRobinLoadBalancer` policy that could raise `NullPointerException` and further cause channel panic when picking a subchannel. This bug can only be triggered when connection can not be established and the channel reports `TRANSIENT_FAILURE` state. ([#​10868](https://github.com/grpc/grpc-java/issues/10868))
--- ### Configuration 📅 **Schedule**: Branch creation - "before 4am on the first day of the month" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). --- lib/build.gradle | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/lib/build.gradle b/lib/build.gradle index cc2588d..4dd30b6 100644 --- a/lib/build.gradle +++ b/lib/build.gradle @@ -34,13 +34,12 @@ dependencies { implementation 'com.google.guava:guava:32.1.3-jre' implementation 'info.picocli:picocli:4.7.5' implementation 'com.google.guava:guava:32.1.3-jre' - implementation "io.grpc:grpc-protobuf:1.60.1" - implementation "io.grpc:grpc-stub:1.60.1" - implementation "io.grpc:grpc-services:1.60.1" - implementation "io.grpc:grpc-testing:1.60.1" - implementation "io.cloudquery:plugin-pb-java:0.0.14" - implementation "org.apache.arrow:arrow-memory-core:12.0.1" - implementation "org.apache.arrow:arrow-vector:12.0.1" + implementation 'io.grpc:grpc-protobuf:1.61.1' + implementation 'io.grpc:grpc-stub:1.61.1' + implementation 'io.grpc:grpc-services:1.61.1' + implementation 'io.cloudquery:plugin-pb-java:0.0.14' + implementation 'org.apache.arrow:arrow-memory-core:12.0.1' + implementation 'org.apache.arrow:arrow-vector:12.0.1' implementation "com.fasterxml.jackson.core:jackson-core:2.16.1" implementation "com.fasterxml.jackson.core:jackson-annotations:2.16.1" @@ -48,15 +47,16 @@ dependencies { implementation 'org.apache.logging.log4j:log4j-api:2.22.1' implementation 'org.apache.logging.log4j:log4j-core:2.22.1' - testImplementation(platform('org.junit:junit-bom:5.10.1')) - testImplementation('org.junit.jupiter:junit-jupiter:5.10.1') - testImplementation('org.junit.jupiter:junit-jupiter-api:5.10.1') - testImplementation('org.mockito:mockito-core:5.10.0') - testImplementation('org.mockito:mockito-junit-jupiter:5.10.0') - testImplementation('nl.jqno.equalsverifier:equalsverifier:3.15.6') - testRuntimeOnly('org.junit.jupiter:junit-jupiter-engine:5.10.1') - + testImplementation 'io.grpc:grpc-testing:1.61.1' + testImplementation 'io.grpc:grpc-inprocess:1.61.1' + testImplementation platform('org.junit:junit-bom:5.10.1') + testImplementation 'org.junit.jupiter:junit-jupiter:5.10.1' + testImplementation 'org.junit.jupiter:junit-jupiter-api:5.10.1' + testImplementation 'org.mockito:mockito-core:5.10.0' + testImplementation 'org.mockito:mockito-junit-jupiter:5.10.0' + testImplementation 'nl.jqno.equalsverifier:equalsverifier:3.15.6' testImplementation 'org.assertj:assertj-core:3.25.2' + testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.10.1' runtimeOnly "org.apache.arrow:arrow-memory-netty:12.0.1" } From 9eeb976c868b1a206230b6285d666671a18682a7 Mon Sep 17 00:00:00 2001 From: Alex Shcherbakov Date: Fri, 9 Feb 2024 13:48:28 +0200 Subject: [PATCH 3/4] feat: Implement `GetSpecSchema` (#180) Closes https://github.com/cloudquery/cloudquery/issues/16503 --- .../servers/plugin/v3/PluginServer.java | 14 +++++++ .../java/io/cloudquery/plugin/Plugin.java | 1 + .../servers/plugin/v3/PluginServerTest.java | 39 ++++++++++++++++++- 3 files changed, 52 insertions(+), 2 deletions(-) diff --git a/lib/src/main/java/io/cloudquery/internal/servers/plugin/v3/PluginServer.java b/lib/src/main/java/io/cloudquery/internal/servers/plugin/v3/PluginServer.java index 56ee04d..6bada6a 100644 --- a/lib/src/main/java/io/cloudquery/internal/servers/plugin/v3/PluginServer.java +++ b/lib/src/main/java/io/cloudquery/internal/servers/plugin/v3/PluginServer.java @@ -181,4 +181,18 @@ private WriteMessage processDeleteStaleRequest(Write.Request request) messageDeleteStale.getSourceName(), new Date(messageDeleteStale.getSyncTime().getSeconds() * 1000)); } + + @Override + public void getSpecSchema( + io.cloudquery.plugin.v3.GetSpecSchema.Request request, + StreamObserver responseObserver) { + io.cloudquery.plugin.v3.GetSpecSchema.Response.Builder builder = + io.cloudquery.plugin.v3.GetSpecSchema.Response.newBuilder(); + String schema = this.plugin.getJsonSchema(); + if (schema != null && !schema.isBlank()) { + builder.setJsonSchema(schema); + } + responseObserver.onNext(builder.build()); + responseObserver.onCompleted(); + } } diff --git a/lib/src/main/java/io/cloudquery/plugin/Plugin.java b/lib/src/main/java/io/cloudquery/plugin/Plugin.java index 5c02701..bb46a0b 100644 --- a/lib/src/main/java/io/cloudquery/plugin/Plugin.java +++ b/lib/src/main/java/io/cloudquery/plugin/Plugin.java @@ -18,6 +18,7 @@ public abstract class Plugin { @NonNull protected final String name; @NonNull protected final String version; @Setter protected Logger logger; + @Setter protected String jsonSchema; protected ClientMeta client; public void init(String spec, NewClientOptions options) throws Exception { diff --git a/lib/src/test/java/io/cloudquery/internal/servers/plugin/v3/PluginServerTest.java b/lib/src/test/java/io/cloudquery/internal/servers/plugin/v3/PluginServerTest.java index f75019f..e1e4d66 100644 --- a/lib/src/test/java/io/cloudquery/internal/servers/plugin/v3/PluginServerTest.java +++ b/lib/src/test/java/io/cloudquery/internal/servers/plugin/v3/PluginServerTest.java @@ -1,5 +1,6 @@ package io.cloudquery.internal.servers.plugin.v3; +import static org.junit.jupiter.api.Assertions.*; import static org.mockito.ArgumentMatchers.any; import static org.mockito.Mockito.verify; @@ -9,6 +10,7 @@ import io.cloudquery.messages.WriteInsert; import io.cloudquery.messages.WriteMigrateTable; import io.cloudquery.plugin.Plugin; +import io.cloudquery.plugin.v3.GetSpecSchema; import io.cloudquery.plugin.v3.PluginGrpc; import io.cloudquery.plugin.v3.PluginGrpc.PluginStub; import io.cloudquery.plugin.v3.Write; @@ -25,12 +27,14 @@ import java.io.IOException; import java.util.List; import java.util.concurrent.CountDownLatch; +import lombok.Getter; import org.apache.arrow.vector.types.pojo.ArrowType; import org.junit.Rule; import org.junit.jupiter.api.BeforeEach; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.extension.ExtendWith; import org.mockito.Mock; +import org.mockito.Mockito; import org.mockito.junit.jupiter.MockitoExtension; @ExtendWith(MockitoExtension.class) @@ -91,6 +95,31 @@ public void shouldSendWriteDeleteStaleMessage() throws Exception { verify(plugin).write(any(WriteDeleteStale.class)); } + @Test + public void shouldSendNullJSONSchema() throws Exception { + NullResponseStream responseObserver = new NullResponseStream<>(); + + pluginStub.getSpecSchema(GetSpecSchema.Request.getDefaultInstance(), responseObserver); + responseObserver.await(); + + verify(plugin).getJsonSchema(); + assertFalse(responseObserver.getValue().hasJsonSchema()); + } + + @Test + public void shouldSendNonNullJSONSchema() throws Exception { + Mockito.doReturn("{}").when(plugin).getJsonSchema(); + + NullResponseStream responseObserver = new NullResponseStream<>(); + + pluginStub.getSpecSchema(GetSpecSchema.Request.getDefaultInstance(), responseObserver); + responseObserver.await(); + + verify(plugin).getJsonSchema(); + assertTrue(responseObserver.getValue().hasJsonSchema()); + assertEquals("{}", responseObserver.getValue().getJsonSchema()); + } + private static Write.Request generateMigrateTableMessage() throws IOException { Table table = Table.builder().name("test").build(); return Write.Request.newBuilder() @@ -121,12 +150,18 @@ private Write.Request generateDeleteStaleMessage() { private static class NullResponseStream implements StreamObserver { private final CountDownLatch countDownLatch = new CountDownLatch(1); + @Getter private T value; + @Getter private Throwable error; @Override - public void onNext(T value) {} + public void onNext(T value) { + this.value = value; + } @Override - public void onError(Throwable t) {} + public void onError(Throwable t) { + this.error = t; + } @Override public void onCompleted() { From bc766eac0df07a9860b70ee8ec374ebe68565e72 Mon Sep 17 00:00:00 2001 From: CloudQuery Bot <102256036+cq-bot@users.noreply.github.com> Date: Fri, 9 Feb 2024 13:51:33 +0200 Subject: [PATCH 4/4] chore(main): Release v0.0.21 (#179) :robot: I have created a release *beep* *boop* --- ## [0.0.21](https://github.com/cloudquery/plugin-sdk-java/compare/v0.0.20...v0.0.21) (2024-02-09) ### Features * Implement `GetSpecSchema` ([#180](https://github.com/cloudquery/plugin-sdk-java/issues/180)) ([9eeb976](https://github.com/cloudquery/plugin-sdk-java/commit/9eeb976c868b1a206230b6285d666671a18682a7)) ### Bug Fixes * **deps:** Update dependency io.cloudquery:plugin-pb-java to v0.0.14 ([#178](https://github.com/cloudquery/plugin-sdk-java/issues/178)) ([6871551](https://github.com/cloudquery/plugin-sdk-java/commit/6871551de650ee36464699ec6cbb0f124676f282)) * **deps:** Update grpc-java monorepo to v1.61.1 ([#172](https://github.com/cloudquery/plugin-sdk-java/issues/172)) ([4ada7bc](https://github.com/cloudquery/plugin-sdk-java/commit/4ada7bcade9cf94043a9ba99c86c2fef43c3a645)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --- CHANGELOG.md | 13 +++++++++++++ lib/build.gradle | 2 +- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ae44d39..a3ec5a4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # Changelog +## [0.0.21](https://github.com/cloudquery/plugin-sdk-java/compare/v0.0.20...v0.0.21) (2024-02-09) + + +### Features + +* Implement `GetSpecSchema` ([#180](https://github.com/cloudquery/plugin-sdk-java/issues/180)) ([9eeb976](https://github.com/cloudquery/plugin-sdk-java/commit/9eeb976c868b1a206230b6285d666671a18682a7)) + + +### Bug Fixes + +* **deps:** Update dependency io.cloudquery:plugin-pb-java to v0.0.14 ([#178](https://github.com/cloudquery/plugin-sdk-java/issues/178)) ([6871551](https://github.com/cloudquery/plugin-sdk-java/commit/6871551de650ee36464699ec6cbb0f124676f282)) +* **deps:** Update grpc-java monorepo to v1.61.1 ([#172](https://github.com/cloudquery/plugin-sdk-java/issues/172)) ([4ada7bc](https://github.com/cloudquery/plugin-sdk-java/commit/4ada7bcade9cf94043a9ba99c86c2fef43c3a645)) + ## [0.0.20](https://github.com/cloudquery/plugin-sdk-java/compare/v0.0.19...v0.0.20) (2024-02-06) diff --git a/lib/build.gradle b/lib/build.gradle index 4dd30b6..5fe9962 100644 --- a/lib/build.gradle +++ b/lib/build.gradle @@ -11,7 +11,7 @@ ext { group 'io.cloudquery' // x-release-please-start-version -version = '0.0.20' +version = '0.0.21' // x-release-please-end repositories {