[SPARK-40640][CORE] SparkHadoopUtil to set origin of hadoop/hive config options #38084

steveloughran · 2022-10-03T17:22:19Z

What changes were proposed in this pull request?

The options passed from spark conf, hive-site.xml, AWS env vars now all record this in their source attribute of the entries.

The Configuration Writable methods do not propagate this, so it is not as useful cluster-wide than it could be. It does help with some of the basic troubleshooting.

Why are the changes needed?

Helps when troubleshooting where options make their way down. These can be examined
and logged later.

For example, my cloudstore diagnosticss JAR can do this in its storediag command
and in an s3a AWS credential provider. I may add some of that logging
at debug to the ASF hadoop implementations.

https://github.com/steveloughran/cloudstore

Does this PR introduce any user-facing change?

Not really. It's a very low level diagnostics feature in the Hadoop configuration classes.

How was this patch tested?

New tests added; existing tests enhanced.

The options passed from spark conf, hive-site.xml, AWS env vars now all record this in their source attribute of the entries. The Configuration Writable methods do not propagate this, so it is not as useful cluster-wide than it could be. It does help with some of the basic troubleshooting. Change-Id: Id3c7cbf1f1d17ef1bf3cc7d8144c84260861a4e5

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

steveloughran · 2022-10-04T10:33:18Z

as well as Atilla's suggestion, i plan to have HiveClientImpl resolve and propagate the origin

Co-authored-by: Attila Zsolt Piros <[email protected]>

…te.scala Co-authored-by: Attila Zsolt Piros <[email protected]>

* update tests to match changed strings * HiveClientImpl.newHiveConf() does some of this too. It doesn't pass on the origin from any hadoop conf passed in because that conf comes in as JIterable[JMap.Entry[String, String]]; reflection could be used *or* the params changed, which introduces risk to what is just a minor debug change. * SparkHadoopUtil.extractOrigin() is there for anyone who ever wants to do that in future. Change-Id: Ia961c6b084e4cdba7e55d5687b32d9ea14c73b84

remove the unused ref to SparkHadoopUtil in HiveClientImpl Change-Id: Id604178023762f8712dc727073232f6dcf5ca7cd

attilapiros · 2022-10-04T19:25:09Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

+   * @param defVal     default value
+   * @return the origin of the current entry in the configuration, or the default.
+   */
+  def extractOrigin(hadoopConf: Configuration, key: String,


This is not needed as this doesn't depends on the state of the SparkHadoopUtil.
Even in the SparkHadoopUtilSuite line 157 we can just remove the new (and the parens) and then you can access the method of the object:
So instead of:

val origin = new SparkHadoopUtil().extractOrigin(hadoopConf, key, null)

We can:

val origin = SparkHadoopUtil.extractOrigin(hadoopConf, key, null)

ok; i was just going with the rest

attilapiros · 2022-10-04T19:26:26Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

+   * @param defVal default value
+   * @return the origin of the current entry in the configuration, or the default.
+   */
+  def extractOrigin(hadoopConf: Configuration, key: String, defVal: String): String = {


I think we do not need defVal and empty string is fine for an unknown source.

attilapiros · 2022-10-04T19:33:25Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

+  def extractOrigin(hadoopConf: Configuration, key: String, defVal: String): String = {
+    val sources = hadoopConf.getPropertySources(key)
+    if (sources != null && sources.length > 0) {
+      sources(0)


Has the first one has special meaning? Is it the one which wins? The getPropertySources javadoc does not really clarify this:

* returns a list of the sources of the resource. The older sources are * the first ones in the list. So for example if a configuration is set from * the command line, and then written out to a file that is read back in the * first entry would indicate that it was set from the command line, while * the second one would indicate the file that the new configuration was read * in from.

If the first wins let's mention that here in a comment.

Otherwise we could give back a coma separated String easily: sources.mkString(", ") if that's fits better.

I'll do the mkstring(); it can only help debugging to know that entire chain. This is what cloudstore does, incidentally. I also plan to improve the s3a credential providers to log at debug their sources as passed in, maybe even some simple crc2 checksum (so you can use crc32 on the command like to verify your secrets are in, but not enough that if someone has the log they can determine a valid set of credentials)

attilapiros · 2022-10-04T21:18:37Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

+   * @param defVal default value
+   * @return the origin of the current entry in the configuration, or the default.
+   */
+  def extractOrigin(hadoopConf: Configuration, key: String, defVal: String): String = {


I suggest to use the name propertySource (or propertySources if all is needed see below) to decrease the need of mapping of names.

you've just pointed out that I have always my ordering of where changes come from wrong. I was of the belief that newest came first. Now I will have to review where else I've used that...

So do we know something about the orders?

Which one wins if one key is set from two different sources? The last one?

Does the last on is the first one in the getPropertySources() result?

If we have the answer we should fix the documentation (in the Hadoop code).

1. Review comments 2. AWS env var propagation records hostname on which these were added. This helps identify issues where configurations are being created in the cluster, rather than on the deploying host -the one with the secrets. 3. extracted the env propagation method into its own private[deploy] method so the unit tests can validate this works. 4, reviewed comments on other setters and updated them to explicitly state which hadoop versions have the issues which need fixing. Change-Id: Ie58856ce621e288c4a8bb06083e43f450f4ec26d

steveloughran · 2022-10-06T11:36:52Z

one more idea. To help debug the origin of config files, how about we always patch the config with (if unset) some property like spark.diagnostics.config.origin to that hostname?
that way, if you ever get a config in your debugger, you can print that value and see where it originated?

attilapiros · 2022-10-06T17:35:35Z

one more idea. To help debug the origin of config files, how about we always patch the config with (if unset) some property like spark.diagnostics.config.origin to that hostname? that way, if you ever get a config in your debugger, you can print that value and see where it originated?

Ok but let's do that in separate PR.

steveloughran · 2022-10-06T18:47:30Z

that last change seems to break a yarn test

- yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) *** FAILED *** (3 minutes, 0 seconds)
[info]   The code passed to eventually never returned normally. Attempted 190 times over 3.001129006083333 minutes. Last failure message: handle.getState().isFinal() was false. (BaseYarnClusterSuite.scala:209)
[info]   org.scalatest.exceptions.TestFailedDueToTimeoutException:
[info]   at org.scalatest.enablers.Retrying$$anon$4.tryTryAgain$2(Retrying.scala:219)
[info]   at org.scalatest.enablers.Retrying$$anon$4.retry(Retrying.scala:226)
[info]   at org.scalatest.concurrent.Eventually.eventually(Eventually.scala:313)
[info]   at org.scalatest.concurrent.Eventually.eventually$(Eventually.scala:312)
[info]   at org.scalatest.concurrent.Eventually$.eventually(Eventually.scala:457)
[info]   at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.runSpark(BaseYarnClusterSuite.scala:209)
[info]   at org.apache.spark.deploy.yarn.YarnClusterSuite.$anonfun$new$6(YarnClusterSuite.scala:148)
[info]   at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[info]   at org.apache.spark.deploy.yarn.BaseYarnClusterSuite.$anonfun$test$1(BaseYarnClusterSuite.scala:77)
[info]   at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info]   at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info]   at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:22)
[info]   at org.scalatest.Transformer.apply(Transformer.scala:20)
[info]   at org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)

attilapiros · 2022-10-06T21:22:08Z

Interesting locally (on its own) it is passing:

➜  spark-pr1 git:(pr/38084) ✗ git log --oneline -n 1
7909817173 (HEAD -> pr/38084) SPARK-40640. SparkHadoopUtil to set origin of hadoop/hive config options

➜  spark-pr1 git:(pr/38084) ✗ ./build/sbt -Pyarn 'project yarn; testOnly *.YarnClusterSuite -- -z "yarn-cluster should respect conf overrides in SparkHadoopUtil"'
....
[info] compiling 4 Scala sources to /Users/attilazsoltpiros/git/attilapiros/spark-pr1/resource-managers/yarn/target/scala-2.12/test-classes ...
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) (13 seconds, 231 milliseconds)
[info] Run completed in 34 seconds, 440 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 59 s, completed Oct 6, 2022 2:15:07 PM
sbt:spark-yarn>
[info] shutting down sbt server

attilapiros · 2022-10-06T21:39:54Z

Those are flaky tests:
https://issues.apache.org/jira/browse/SPARK-32297

Please re-trigger the check!

dongjoon-hyun · 2022-10-07T06:04:38Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

 import java.security.PrivilegedExceptionAction
 import java.text.DateFormat
 import java.util.{Arrays, Date, Locale}
+import java.util


-import java.util.{Arrays, Date, Locale} -import java.util +import java.util.{Arrays, Date, Locale, Map => JMap}

dongjoon-hyun · 2022-10-07T06:04:54Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

-          hadoopConf.set("fs.s3a.session.token", sessionToken)
-        }
-      }
+      val env: util.Map[String, String] = System.getenv


- val env: util.Map[String, String] = System.getenv + val env: JMap[String, String] = System.getenv

dongjoon-hyun · 2022-10-07T06:05:32Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

+  // Exposed for testing
+  private[deploy] def appendS3CredentialsFromEnvironment(
+    hadoopConf: Configuration,
+    env: util.Map[String, String]): Unit = {


- hadoopConf: Configuration, - env: util.Map[String, String]): Unit = { + hadoopConf: Configuration, + env: JMap[String, String]): Unit = {

dongjoon-hyun · 2022-10-07T06:07:37Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

-        }
-      }
+      val env: util.Map[String, String] = System.getenv
+      appendS3CredentialsFromEnvironment(hadoopConf, env)


If env is used by only appendS3CredentialsFromEnvironment, shall we invoke System.genenv inside that method instead of here.

i want it isolated for testing; we can't set env vars in a unit test, but we can instead build up a map and verify propagation there.

dongjoon-hyun · 2022-10-07T06:10:26Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

+      hadoopConf.set(key.substring("spark.hadoop.".length), value,
+        "Set by Spark from keys starting with 'spark.hadoop'")
    }
+    val setBySpark = "Set by Spark to default values"


Could you define these constants in object SparkHadoopUtil, please?

"Set by Spark from hive-site.xml"

"Set by Spark from keys starting with 'spark.hive'"

"Set by Spark from keys starting with 'spark.hadoop'"

"Set by Spark to default values"

dongjoon-hyun · 2022-10-07T06:11:17Z

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala


+  private val setToDefaultValues = "Set by Spark to default values"
+  private val hadoopPropagation = "Set by Spark from keys starting with 'spark.hadoop'"
+  private val hivePropagation = "Set by Spark from keys starting with 'spark.hive'"


We can remove these string duplications by https://github.com/apache/spark/pull/38084/files#r989713735.

dongjoon-hyun · 2022-10-07T06:13:40Z

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala

+  /**
+   * The explicit buffer size propagation records this.
+   */
+  test("buffer size propagation") {


- test("buffer size propagation") { + test("SPARK-40640: buffer size propagation") {

dongjoon-hyun · 2022-10-07T06:14:02Z

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala

+    assertConfigMatches(hadoopConf, "io.file.buffer.size", "123", BUFFER_SIZE.key)
+  }
+
+  test("aws credentials from environment variables") {


- test("aws credentials from environment variables") { + test("SPARK-40640: aws credentials from environment variables") {

dongjoon-hyun · 2022-10-07T06:15:12Z

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala

+    hadoopConf: Configuration,
+    key: String,
+    expected: String,
+    expectedSource: String): Unit = {


Indentation: Add two more spaces for parameters.

private def assertConfigMatches( hadoopConf: Configuration, key: String, expected: String, expectedSource: String): Unit = { assertConfigValue(hadoopConf, key, expected)

will need to tweak my scala ide rules, i see

dongjoon-hyun · 2022-10-07T06:16:18Z

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala

+  private def assertConfigSourceContains(
+    hadoopConf: Configuration,
+    key: String,
+    expectedSource: String): Unit = {


ditto. Indentation. Two more spaces for parameters.

dongjoon-hyun · 2022-10-07T06:17:46Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala

    val confMap = (hadoopConf.iterator().asScala.map(kv => kv.getKey -> kv.getValue) ++
      sparkConf.getAll.toMap ++ extraConfig).toMap
-    confMap.foreach { case (k, v) => hiveConf.set(k, v) }
+    val fromSpark = "Set by Spark"


Although this is tricky, let's collect this into the same place (https://github.com/apache/spark/pull/38084/files#r989713735).

Gentle ping about this leftover.

Since this is non-test code string, we should have in the same place with the others.

yes, what is the solution there? Even when exported with private[spark] it wasn't picked up

yes, what is the solution there? Even when exported with private[spark] it wasn't picked up

Okay, let me make a PR to you.

dongjoon-hyun

I finished the first round review, @steveloughran . In general, the PR looks helpful.

* pull out source strings and env vars into constants; use * add a new test case Change-Id: I88f525745c046cc7040eab5501449447a0d5924c

dongjoon-hyun · 2022-10-07T17:53:06Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

    for ((key, value) <- conf.getAll if key.startsWith("spark.hive.")) {
-      hadoopConf.set(key.substring("spark.".length), value)
+      hadoopConf.set(key.substring("spark.".length), value,
+        SOURCE_SPARK_HIVE)


Can we make it into a single line?

hadoopConf.set(key.substring("spark.".length), value, SOURCE_SPARK_HIVE)

dongjoon-hyun · 2022-10-07T18:02:38Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

-          hadoopConf.set("fs.s3a.session.token", sessionToken)
-        }
-      }
+      appendS3CredentialsFromEnvironment(hadoopConf, System.getenv)


I'm still worrying about the chance of the regression of getting and passing a large map of strings here. There were some reports like this.

https://github.com/apache/spark/pull/31244/files

If you really want to hand over a map as a dependency injection, can you build a small one by reusing the existing logic only like System.getenv("AWS_ACCESS_KEY_ID")?

having it pass down separate arguments; no map at all

dongjoon-hyun · 2022-10-07T18:08:44Z

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

+
+  /**
+   * Return a hostname without throwing an exception if the system
+   * does not know its own name.


I'm wondering when does this happen? Apache Spark has the following codes.

core/src/main/scala/org/apache/spark/util/Utils.scala: val address = InetAddress.getLocalHost core/src/main/scala/org/apache/spark/util/Utils.scala: logWarning("Your hostname, " + InetAddress.getLocalHost.getHostName + " resolves to" + core/src/main/scala/org/apache/spark/util/Utils.scala: logWarning("Your hostname, " + InetAddress.getLocalHost.getHostName + " resolves to" +

that is an interesting q. tracing the history from hadoop NetUtil and back, came in with https://issues.apache.org/jira/browse/HADOOP-1028 *and no explicit "we have seen this". I have seen machines with the wrong name, wrong IP, and even localhost not mapped to a 127. address. but not this. will cut and assume nicholas just added it to save on exception propagation in java

dongjoon-hyun · 2022-10-07T18:10:02Z

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala

-    assertConfigValue(hadoopConf, "fs.s3a.downgrade.syncable.exceptions", "true")
-    assertConfigValue(hadoopConf, "fs.s3a.endpoint", "s3.amazonaws.com")
+    assertConfigMatches(hadoopConf, "orc.filterPushdown", "true",
+      SOURCE_SPARK_HADOOP)


Do we need to split the line? Otherwise, please make it as a single line.

dongjoon-hyun · 2022-10-07T18:10:28Z

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala

+  /**
+   * spark.hive.* is passed to the hadoop config as hive.*.
+   */
+  test("spark.hive propagation") {


Please add a test case prefix, SPARK-40640: .

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala

* move from env var map to explicit args of values; test gets leaner Change-Id: I74f76427321296583235c9600b10a70cbd42777a

dongjoon-hyun

I made a PR to your branch, @steveloughran .

steveloughran#1

Minor changes

steveloughran · 2022-10-12T12:17:01Z

thanks, merged in. when i tried that xref of the constant to hive class things wouldn't resolve properly; clearly some local build issue.

steveloughran · 2022-10-13T16:42:17Z

test failure seems nowhere near this code

[info] *** 1 TEST FAILED ***
[error] Failed: Total 3425, Failed 1, Errors 0, Passed 3424, Ignored 9, Canceled 2
[error] Failed tests:
[error] 	org.apache.spark.scheduler.TaskSetManagerSuite

dongjoon-hyun · 2022-10-13T19:26:34Z

Ya, I agree with you.

[info] - SPARK-32170: test SPECULATION_EFFICIENCY_TASK_DURATION_FACTOR for speculating tasks *** FAILED *** (106 milliseconds)
[info]   manager.checkSpeculatableTasks(15000L) was true (TaskSetManagerSuite.scala:2487)

dongjoon-hyun

+1, LGTM. Merged to master for Apache Spark 3.4.0.

dongjoon-hyun · 2022-10-13T19:29:13Z

Thank you, @steveloughran and @attilapiros !

steveloughran · 2022-10-14T09:50:08Z

thx. still not sufficient, but it may help.

…ig options ### What changes were proposed in this pull request? The options passed from spark conf, hive-site.xml, AWS env vars now all record this in their source attribute of the entries. The Configuration Writable methods do not propagate this, so it is not as useful cluster-wide than it could be. It does help with some of the basic troubleshooting. ### Why are the changes needed? Helps when troubleshooting where options make their way down. These can be examined and logged later. For example, my cloudstore diagnosticss JAR can do this in its storediag command and in an s3a AWS credential provider. I may add some of that logging at debug to the ASF hadoop implementations. https://github.com/steveloughran/cloudstore ### Does this PR introduce _any_ user-facing change? Not *really*. It's a very low level diagnostics feature in the Hadoop configuration classes. ### How was this patch tested? New tests added; existing tests enhanced. Closes apache#38084 from steveloughran/SPARK-40640-spark-conf-propagation. Lead-authored-by: Steve Loughran <[email protected]> Co-authored-by: Steve Loughran <[email protected]> Co-authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

github-actions bot added the CORE label Oct 3, 2022

steveloughran changed the title ~~SPARK-40640. SparkHadoopUtil to set origin of hadoop/hive config options~~ [SPARK-40640][CORE] SparkHadoopUtil to set origin of hadoop/hive config options Oct 3, 2022

attilapiros reviewed Oct 3, 2022

View reviewed changes

core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala Outdated Show resolved Hide resolved

attilapiros reviewed Oct 3, 2022

View reviewed changes

steveloughran and others added 6 commits October 4, 2022 12:59

Update core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

ca25628

Co-authored-by: Attila Zsolt Piros <[email protected]>

Update core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

45eca3d

Co-authored-by: Attila Zsolt Piros <[email protected]>

Update core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

a5b8762

Co-authored-by: Attila Zsolt Piros <[email protected]>

Update core/src/main/scala/org/apache/spark/deploy/SparkHadoopUtil.scala

502441f

Co-authored-by: Attila Zsolt Piros <[email protected]>

Update core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSui…

d4cc9ad

…te.scala Co-authored-by: Attila Zsolt Piros <[email protected]>

github-actions bot added the SQL label Oct 4, 2022

SPARK-40640. SparkHadoopUtil to set origin of hadoop/hive config options

d471a26

remove the unused ref to SparkHadoopUtil in HiveClientImpl Change-Id: Id604178023762f8712dc727073232f6dcf5ca7cd

attilapiros reviewed Oct 4, 2022

View reviewed changes

dongjoon-hyun reviewed Oct 7, 2022

View reviewed changes

SPARK-40640. review feedback

1713a52

* pull out source strings and env vars into constants; use * add a new test case Change-Id: I88f525745c046cc7040eab5501449447a0d5924c

dongjoon-hyun reviewed Oct 7, 2022

View reviewed changes

core/src/test/scala/org/apache/spark/deploy/SparkHadoopUtilSuite.scala Show resolved Hide resolved

steveloughran and others added 2 commits October 10, 2022 13:45

SPARK-40640. review feedback -almost all comments.

fb37a92

* move from env var map to explicit args of values; test gets leaner Change-Id: I74f76427321296583235c9600b10a70cbd42777a

Minor change

9b188c5

dongjoon-hyun requested changes Oct 11, 2022

View reviewed changes

Merge pull request #1 from dongjoon-hyun/SPARK-PR-38084

5809445

Minor changes

dongjoon-hyun approved these changes Oct 13, 2022

View reviewed changes

dongjoon-hyun closed this in 4a93507 Oct 13, 2022

[SPARK-40640][CORE] SparkHadoopUtil to set origin of hadoop/hive config options #38084

[SPARK-40640][CORE] SparkHadoopUtil to set origin of hadoop/hive config options #38084

Uh oh!

Conversation

steveloughran commented Oct 3, 2022

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

steveloughran commented Oct 4, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

steveloughran commented Oct 6, 2022

Uh oh!

attilapiros commented Oct 6, 2022

Uh oh!

steveloughran commented Oct 6, 2022

Uh oh!

attilapiros commented Oct 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

attilapiros commented Oct 6, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun Oct 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

attilapiros commented Oct 6, 2022 •

edited

Loading

dongjoon-hyun Oct 7, 2022 •

edited

Loading

dongjoon-hyun Oct 7, 2022 •

edited

Loading

dongjoon-hyun Oct 7, 2022 •

edited

Loading

dongjoon-hyun left a comment •

edited

Loading