[BEAM-11960]: Bump bigdataoss to 2.2.0 #14200

Fokko · 2021-03-11T16:21:48Z

Please add a meaningful description for your change here.

Google's API client has deprecated a method, that is not available anymore:

java.lang.RuntimeException: Encountered checked exception when constructing an instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:233)
    at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
    at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
...
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:564)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:832)
Caused by: java.lang.NoSuchMethodError: 'boolean com.google.api.client.http.HttpTransport.isMtls()'
    at com.google.api.services.storage.Storage$Builder.chooseEndpoint (Storage.java:11151)
    at com.google.api.services.storage.Storage$Builder.<init> (Storage.java:11184)
    at org.apache.beam.sdk.extensions.gcp.util.Transport.newStorageClient (Transport.java:97)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:104)
    at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:93)
    at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper (ProxyInvocationHandler.java:605)
    at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault (ProxyInvocationHandler.java:546)
    at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke (ProxyInvocationHandler.java:171)
    at com.sun.proxy.$Proxy41.getGcsUtil (Unknown Source)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83)
    at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
    at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:291)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:564)
    at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:217)
    at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
    at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
    at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
...
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:564)
    at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
    at java.lang.Thread.run (Thread.java:832)

I think this is because of bigdataoss:
https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/util/2.2.0: Uses client 1.31.1
https://mvnrepository.com/artifact/com.google.cloud.bigdataoss/util/2.1.6: Uses client 1.30.10

Currently the PR is still blocked by GoogleCloudDataproc/hadoop-connectors#542

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Choose reviewer(s) and mention them in a comment (R: @username).
Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang	Dataflow	Samza	Twister2
Go	---	---	---
Java
Python		---	---
XLang		---	---

Pre-Commit Tests Status (on master branch)

---	Java	Python	Go	Website	Whitespace	Typescript
Non-portable
Portable	---		---	---	---	---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI.

pabloem · 2021-03-11T22:27:23Z

ok so we should wait for the PR you reference at the top?

Fokko · 2021-03-11T22:36:13Z

Yes, this one is still blocked by a release of the bigdataoss package.

aaltay · 2021-04-01T17:59:15Z

GoogleCloudDataproc/hadoop-connectors#542 is merged. Is this PR unblocked?

Fokko · 2021-04-01T20:02:59Z

@aaltay We're waiting for a release. This is expected mid April.

Fokko · 2021-04-19T17:51:05Z

@aaltay We got a release :)

codecov · 2021-04-19T18:21:55Z

Codecov Report

Merging #14200 (ce8ec7a) into master (c644b06) will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master   #14200   +/-   ##
=======================================
  Coverage   83.60%   83.60%           
=======================================
  Files         442      442           
  Lines       59246    59246           
=======================================
+ Hits        49531    49533    +2     
+ Misses       9715     9713    -2

Impacted Files	Coverage Δ
.../python/apache_beam/typehints/trivial_inference.py
.../python/apache_beam/portability/api/metrics_pb2.py
.../sdks/python/apache_beam/examples/snippets/util.py
...ild/srcs/sdks/python/apache_beam/metrics/metric.py
...nternal/clients/cloudbuild/cloudbuild_v1_client.py
...y38/build/srcs/sdks/python/apache_beam/pipeline.py
...hon/apache_beam/runners/dataflow/internal/names.py
...am/testing/benchmarks/chicago_taxi/trainer/task.py
...examples/snippets/transforms/elementwise/kvswap.py
...python/apache_beam/typehints/typehints_test_py3.py
... and 874 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bddf2be...ce8ec7a. Read the comment docs.

suztomo · 2021-04-21T20:57:55Z

@Fokko Now I know why you were looking for isMills. My #14527 that upgrades the libraries-bom has been merged. It brings the newer google-http-client.

pabloem · 2021-04-21T21:53:47Z

hmmm I am a little confused. Do we have the appropriate constructor on the google-http-client release already?

suztomo · 2021-04-21T22:04:15Z

Sorry if my comment confused you. The newer google-http-client is irrelevant to the GoogleCloudStorageImpl constructor problem. From reading the comment, I believe this PR has been waiting for a new bigdataoss release that have Fokko's PR GoogleCloudDataproc/hadoop-connectors#542 . I see bigdataoss has not released a new version since January.

aaltay · 2021-04-29T22:11:11Z

Sorry if my comment confused you. The newer google-http-client is irrelevant to the GoogleCloudStorageImpl constructor problem. From reading the comment, I believe this PR has been waiting for a new bigdataoss release that have Fokko's PR GoogleCloudDataproc/hadoop-connectors#542 . I see bigdataoss has not released a new version since January.

Do you know if there is an issue we can follow for the release?

aaltay · 2021-04-29T22:14:09Z

Sorry if my comment confused you. The newer google-http-client is irrelevant to the GoogleCloudStorageImpl constructor problem. From reading the comment, I believe this PR has been waiting for a new bigdataoss release that have Fokko's PR GoogleCloudDataproc/hadoop-connectors#542 . I see bigdataoss has not released a new version since January.

Do you know if there is an issue we can follow for the release?

And @Fokko said that we got a release (#14200 (comment)). So did the release happen?

suztomo · 2021-04-30T00:08:40Z

@aaltay Sent an internal email to dataproc team (CC-ed you)

aaltay · 2021-05-14T00:01:16Z

As an update, this is still pending the release. And bigdataoss team is planning to working on it soon.

aaltay · 2021-05-27T22:52:11Z

As an update, this is still pending the release. And bigdataoss team is planning to working on it soon.

Still waiting.

aaltay · 2021-06-10T17:51:56Z

I see the 2.2.1 release on Maven. Would that unblock this PR?

aaltay · 2021-07-01T21:41:49Z

What is the next step on this PR?

aaltay · 2021-07-16T03:47:56Z

Should we close this PR? Is it still blocked on something?

Fokko · 2021-07-16T06:33:13Z

I must admit that I'm not working with Beam anymore, so I don't think I'm not the best person to work on this at the moment.

aaltay · 2021-07-21T19:09:47Z

@tysonjh / @kennknowles - Could someone take this over?

tysonjh · 2021-07-22T19:58:46Z

@tysonjh / @kennknowles - Could someone take this over?

Yes, I'll find someone.

aaltay · 2021-08-18T16:07:54Z

@tysonjh / @kennknowles - Could someone take this over?

Yes, I'll find someone.

Do you know if someone would be working on this?

[BEAM-11960] Bump bigdataoss to avoid dependency conflict

914130c

Fokko force-pushed the fd-bump-bump-bigdataoss branch from 07f2b31 to 914130c Compare March 11, 2021 16:23

Fokko changed the title ~~Bump bigdataoss~~ [BEAM-11960]: Bump bigdataoss to 2.20.0 Mar 11, 2021

Fokko changed the title ~~[BEAM-11960]: Bump bigdataoss to 2.20.0~~ [BEAM-11960]: Bump bigdataoss to 2.2.0 Mar 11, 2021

aaltay requested a review from pabloem March 18, 2021 21:46

Fokko added 2 commits April 1, 2021 22:04

Merge branch 'master' into fd-bump-bump-bigdataoss

d4f7ea9

Merge branch 'master' into fd-bump-bump-bigdataoss

617f25d

suztomo mentioned this pull request Apr 21, 2021

Make one more GoogleCloudStorageImpl constructor public GoogleCloudDataproc/hadoop-connectors#542

Merged

Merge branch 'master' into fd-bump-bump-bigdataoss

28701f6

Merge branch 'master' into fd-bump-bump-bigdataoss

ce8ec7a

Fokko closed this Jul 16, 2021

Fokko deleted the fd-bump-bump-bigdataoss branch July 16, 2021 06:32

[BEAM-11960]: Bump bigdataoss to 2.2.0 #14200

[BEAM-11960]: Bump bigdataoss to 2.2.0 #14200

Uh oh!

Conversation

Fokko commented Mar 11, 2021

Post-Commit Tests Status (on master branch)

Pre-Commit Tests Status (on master branch)

GitHub Actions Tests Status (on master branch)

Uh oh!

pabloem commented Mar 11, 2021

Uh oh!

Fokko commented Mar 11, 2021

Uh oh!

aaltay commented Apr 1, 2021

Uh oh!

Fokko commented Apr 1, 2021

Uh oh!

Fokko commented Apr 19, 2021

Uh oh!

codecov bot commented Apr 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

suztomo commented Apr 21, 2021

Uh oh!

pabloem commented Apr 21, 2021

Uh oh!

suztomo commented Apr 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaltay commented Apr 29, 2021

Uh oh!

aaltay commented Apr 29, 2021

Uh oh!

suztomo commented Apr 30, 2021

Uh oh!

aaltay commented May 14, 2021

Uh oh!

aaltay commented May 27, 2021

Uh oh!

aaltay commented Jun 10, 2021

Uh oh!

aaltay commented Jul 1, 2021

Uh oh!

aaltay commented Jul 16, 2021

Uh oh!

Fokko commented Jul 16, 2021

Uh oh!

aaltay commented Jul 21, 2021

Uh oh!

tysonjh commented Jul 22, 2021

Uh oh!

aaltay commented Aug 18, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

codecov bot commented Apr 19, 2021 •

edited

Loading

suztomo commented Apr 21, 2021 •

edited

Loading