Skip to content

Conversation

@Fokko
Copy link
Contributor

@Fokko Fokko commented Mar 12, 2021

Missing the isMtls() in google-http-client. Looks like it has been added since 1.38.0:

https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/http/HttpTransport.java#L136

Original stacktrace:

java.lang.RuntimeException: Encountered checked exception when constructing an instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
...
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
at java.lang.Thread.run (Thread.java:832)
Caused by: java.lang.NoSuchMethodError: 'boolean com.google.api.client.http.HttpTransport.isMtls()'
at com.google.api.services.storage.Storage$Builder.chooseEndpoint (Storage.java:11151)
at com.google.api.services.storage.Storage$Builder.<init> (Storage.java:11184)
at org.apache.beam.sdk.extensions.gcp.util.Transport.newStorageClient (Transport.java:97)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:104)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:93)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper (ProxyInvocationHandler.java:605)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault (ProxyInvocationHandler.java:546)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke (ProxyInvocationHandler.java:171)
at com.sun.proxy.$Proxy41.getGcsUtil (Unknown Source)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:291)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:217)
at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
...
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
at java.lang.Thread.run (Thread.java:832)

Please add a meaningful description for your change here


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Choose reviewer(s) and mention them in a comment (R: @username).
  • Format the pull request title like [BEAM-11968] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

Post-Commit Tests Status (on master branch)

Lang SDK Dataflow Flink Samza Spark Twister2
Go Build Status --- Build Status --- Build Status ---
Java Build Status Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status Build Status
Build Status
Build Status
Build Status
Python Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
Build Status
--- Build Status ---
XLang Build Status Build Status Build Status --- Build Status ---

Pre-Commit Tests Status (on master branch)

--- Java Python Go Website Whitespace Typescript
Non-portable Build Status Build Status
Build Status
Build Status
Build Status
Build Status Build Status Build Status Build Status
Portable --- Build Status --- --- --- ---

See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests

See CI.md for more information about GitHub Actions CI.

@codecov
Copy link

codecov bot commented Mar 12, 2021

Codecov Report

Merging #14218 (5a994f9) into master (0ca0088) will increase coverage by 0.02%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #14218      +/-   ##
==========================================
+ Coverage   83.41%   83.43%   +0.02%     
==========================================
  Files         469      918     +449     
  Lines       58721   117735   +59014     
==========================================
+ Hits        48985    98238   +49253     
- Misses       9736    19497    +9761     
Impacted Files Coverage Δ
...s/snippets/transforms/aggregation/combinevalues.py
...thon/apache_beam/io/azure/blobstoragefilesystem.py
...es/snippets/transforms/aggregation/cogroupbykey.py
...ild/srcs/sdks/python/apache_beam/runners/common.py
...examples/snippets/transforms/aggregation/sample.py
...e_beam/runners/portability/abstract_job_service.py
...beam/runners/interactive/display/pipeline_graph.py
...build/srcs/sdks/python/apache_beam/io/gcp/gcsio.py
.../python/apache_beam/io/gcp/tests/pubsub_matcher.py
...python/apache_beam/io/gcp/datastore/v1new/types.py
... and 1377 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0ca0088...5a994f9. Read the comment docs.

Missing the isMtls() in google-http-client. Looks like it has been added since 1.38.0:

https://github.com/googleapis/google-http-java-client/blob/master/google-http-client/src/main/java/com/google/api/client/http/HttpTransport.java#L136

Original stacktrace:

```
java.lang.RuntimeException: Encountered checked exception when constructing an instance from factory method DataflowRunner#fromOptions(interface org.apache.beam.sdk.options.PipelineOptions)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:233)
at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
...
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
at java.lang.Thread.run (Thread.java:832)
Caused by: java.lang.NoSuchMethodError: 'boolean com.google.api.client.http.HttpTransport.isMtls()'
at com.google.api.services.storage.Storage$Builder.chooseEndpoint (Storage.java:11151)
at com.google.api.services.storage.Storage$Builder.<init> (Storage.java:11184)
at org.apache.beam.sdk.extensions.gcp.util.Transport.newStorageClient (Transport.java:97)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:104)
at org.apache.beam.sdk.extensions.gcp.util.GcsUtil$GcsUtilFactory.create (GcsUtil.java:93)
at org.apache.beam.sdk.options.ProxyInvocationHandler.returnDefaultHelper (ProxyInvocationHandler.java:605)
at org.apache.beam.sdk.options.ProxyInvocationHandler.getDefault (ProxyInvocationHandler.java:546)
at org.apache.beam.sdk.options.ProxyInvocationHandler.invoke (ProxyInvocationHandler.java:171)
at com.sun.proxy.$Proxy41.getGcsUtil (Unknown Source)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.verifyPathIsAccessible (GcsPathValidator.java:83)
at org.apache.beam.sdk.extensions.gcp.storage.GcsPathValidator.validateOutputFilePrefixSupported (GcsPathValidator.java:53)
at org.apache.beam.runners.dataflow.DataflowRunner.fromOptions (DataflowRunner.java:291)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.apache.beam.sdk.util.InstanceBuilder.buildFromMethod (InstanceBuilder.java:217)
at org.apache.beam.sdk.util.InstanceBuilder.build (InstanceBuilder.java:158)
at org.apache.beam.sdk.PipelineRunner.fromOptions (PipelineRunner.java:55)
at org.apache.beam.sdk.Pipeline.create (Pipeline.java:154)
...
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:64)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:564)
at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:282)
at java.lang.Thread.run (Thread.java:832)
```
@aaltay aaltay requested a review from kennknowles April 1, 2021 17:57
@kennknowles
Copy link
Member

Most of the failures appear to be a mocking situation that is checking how many times a trivial method is called. Probably the mocking should be more flexible or we could find a non-mock way to test. Do you feel able to go and adjust the mocks?

@Fokko
Copy link
Contributor Author

Fokko commented Apr 1, 2021

@kennknowles I tried to debug the logs, I think it has to do with some missing licenses, but didn't get it running locally.

@pabloem do you have any pointers? :)

@pabloem
Copy link
Member

pabloem commented Apr 15, 2021

sorry that I missed this. Rerunning things...

@pabloem pabloem closed this Apr 15, 2021
@pabloem pabloem reopened this Apr 15, 2021
@pabloem
Copy link
Member

pabloem commented Apr 16, 2021

Run Java PreCommit

@pabloem
Copy link
Member

pabloem commented Apr 16, 2021

Run Java_Examples_Dataflow PreCommit

@suztomo
Copy link
Contributor

suztomo commented Apr 19, 2021

I have #14527 for the libraries-bom 20.0.0, which brings google-http-client 1.39.2

@pabloem
Copy link
Member

pabloem commented Apr 20, 2021

ok closing for the 20.0.0 update instead. @Fokko I hope that's fine ^^'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants