feat: enable auto-population of missing metadata in logs and opting logs redirection to stdout in JUL handler (#808)

Aggregates the following work:
- #821
- #812
- #807
- #803
- #798

Fixes #689, #691, #799 and #800

* 🦉 Updates from OwlBot

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

Author: minherz

.readme-partials.yaml

@@ -134,3 +134,62 @@ custom_content: |
   ```
   com.google.cloud.examples.logging.snippets.AddLoggingHandler.handlers=com.google.cloud.logging.LoggingHandler
   ```
+
+  #### Alternative way to ingest logs in Google Cloud managed environments
+
+  If you use Java logger with the Cloud Logging Handler, you can configure the handler to output logs to `stdout` using
+  the [structured logging Json format](https://cloud.google.com/logging/docs/structured-logging#special-payload-fields).
+  To do this, add `com.google.cloud.logging.LoggingHandler.redirectToStdout=true` to the logger configuration file.
+  You can use this configuration when running applications in Google Cloud managed environments such as AppEngine, Cloud Run,
+  Cloud Function or GKE. The logger agent installed on these environments can capture STDOUT and ingest it into Cloud Logging.
+  The agent can parse structured logs printed to STDOUT and capture additional log metadata beside the log payload.
+  The parsed information includes severity, source location, user labels, http request and tracing information.
+
+  #### Auto-population of log entrys' metadata
+
+  LogEntry object metadata information such as [monitored resource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource), 
+  [Http request](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest) or 
+  [source location](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogEntrySourceLocation)
+  are automatically populated with information that the library retrieves from the execution context.
+  The library populates only empty (set to `null`) LogEntry fields.
+  This behavior in the `Logging` instance can be opted out via `LoggingOptions`.
+  Call `LoggingOptions.Builder.setAutoPopulateMetadata(false)` to configure logging options to opt-out the metadata auto-population.
+  Cloud Logging handler can be configured to opt-out automatic population of the metadata using the logger configuration.
+  To disable the metadata auto-population add `com.google.cloud.logging.LoggingHandler.autoPopulateMetadata=false`
+  to the logger configuration file.
+
+  The auto-population logic populates source location _only_ for log entries with `Severity.DEBUG` severity.
+  The execution context of the Http request and tracing information is maintained by `ContextHandler` class.
+  The context is managed in the scope of the thread.
+  If you do not use thread pools for multi-threading the `ContextHandler` can be configured to propagate the context
+  to the scope of the child threads.
+  To enable this add `com.google.cloud.logging.ContextHandler.useInheritedContext=true` to the logger configuration file.
+  The library provides two methods to update the context:
+
+  * Manually set the context. You can use the following methods of the `Context.Builder` to set the context information.
+  Use the method `setRequest()` to setup the `HttpRequest` instance or `setRequestUrl()`, `setRequestMethod()`,
+  `setReferer() `, `setRemoteIp()` and `setServerIp()` to setup the fields of the `HttpRequest`.
+  The trace and span Ids can be set directly using `setTraceId()` and `setSpanId()` respectively.
+  Alternatively it can be parsed from the W3C tracing context header using `loadW3CTraceParentContext()` or
+  from the Google Cloud tracing context header using `loadCloudTraceContext()`.
+
+      ```java
+      Context context = Context.newBuilder().setHttpRequest(request).setTrace(traceId).setSpanId(spanId).build();
+      (new ContextHandler()).setCurrentContext(context);
+      ```
+
+  * Using [servlet initializer](https://github.com/googleapis/java-logging-servlet-initializer).
+  If your application uses a Web server based on Jakarta servlets (e.g. Jetty or Tomcat), you can add the servlet initializer
+  package to your WAR. The package implements a service provider interface (SPI) for
+  [javax.servlet.ServletContainerInitializer](https://docs.oracle.com/javaee/6/api/javax/servlet/ServletContainerInitializer.html)
+  and filters all servlet requests to automatically capture the execution context of the servlet request and store it using
+  `ContextHandler` class. The stored `Context` class instances are used to populate Http request and tracing information.
+  If you use Maven, to use the servlet initializer add the following dependency to your BOM:
+
+      ```xml
+      <dependency>
+        <groupId>com.google.cloud</groupId>
+        <artifactId>google-cloud-logging-servlet-initializer</artifactId>
+      </dependency>      
+      ```
+

README.md

@@ -235,6 +235,64 @@ file. Adding, for instance, the following line:
 com.google.cloud.examples.logging.snippets.AddLoggingHandler.handlers=com.google.cloud.logging.LoggingHandler
 ```
 
+#### Alternative way to ingest logs in Google Cloud managed environments
+
+If you use Java logger with the Cloud Logging Handler, you can configure the handler to output logs to `stdout` using
+the [structured logging Json format](https://cloud.google.com/logging/docs/structured-logging#special-payload-fields).
+To do this, add `com.google.cloud.logging.LoggingHandler.redirectToStdout=true` to the logger configuration file.
+You can use this configuration when running applications in Google Cloud managed environments such as AppEngine, Cloud Run,
+Cloud Function or GKE. The logger agent installed on these environments can capture STDOUT and ingest it into Cloud Logging.
+The agent can parse structured logs printed to STDOUT and capture additional log metadata beside the log payload.
+The parsed information includes severity, source location, user labels, http request and tracing information.
+
+#### Auto-population of log entrys' metadata
+
+LogEntry object metadata information such as [monitored resource](https://cloud.google.com/logging/docs/reference/v2/rest/v2/MonitoredResource), 
+[Http request](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#HttpRequest) or 
+[source location](https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry#LogEntrySourceLocation)
+are automatically populated with information that the library retrieves from the execution context.
+The library populates only empty (set to `null`) LogEntry fields.
+This behavior in the `Logging` instance can be opted out via `LoggingOptions`.
+Call `LoggingOptions.Builder.setAutoPopulateMetadata(false)` to configure logging options to opt-out the metadata auto-population.
+Cloud Logging handler can be configured to opt-out automatic population of the metadata using the logger configuration.
+To disable the metadata auto-population add `com.google.cloud.logging.LoggingHandler.autoPopulateMetadata=false`
+to the logger configuration file.
+
+The auto-population logic populates source location _only_ for log entries with `Severity.DEBUG` severity.
+The execution context of the Http request and tracing information is maintained by `ContextHandler` class.
+The context is managed in the scope of the thread.
+If you do not use thread pools for multi-threading the `ContextHandler` can be configured to propagate the context
+to the scope of the child threads.
+To enable this add `com.google.cloud.logging.ContextHandler.useInheritedContext=true` to the logger configuration file.
+The library provides two methods to update the context:
+
+* Manually set the context. You can use the following methods of the `Context.Builder` to set the context information.
+Use the method `setRequest()` to setup the `HttpRequest` instance or `setRequestUrl()`, `setRequestMethod()`,
+`setReferer() `, `setRemoteIp()` and `setServerIp()` to setup the fields of the `HttpRequest`.
+The trace and span Ids can be set directly using `setTraceId()` and `setSpanId()` respectively.
+Alternatively it can be parsed from the W3C tracing context header using `loadW3CTraceParentContext()` or
+from the Google Cloud tracing context header using `loadCloudTraceContext()`.
+
+    ```java
+    Context context = Context.newBuilder().setHttpRequest(request).setTrace(traceId).setSpanId(spanId).build();
+    (new ContextHandler()).setCurrentContext(context);
+    ```
+
+* Using [servlet initializer](https://github.com/googleapis/java-logging-servlet-initializer).
+If your application uses a Web server based on Jakarta servlets (e.g. Jetty or Tomcat), you can add the servlet initializer
+package to your WAR. The package implements a service provider interface (SPI) for
+[javax.servlet.ServletContainerInitializer](https://docs.oracle.com/javaee/6/api/javax/servlet/ServletContainerInitializer.html)
+and filters all servlet requests to automatically capture the execution context of the servlet request and store it using
+`ContextHandler` class. The stored `Context` class instances are used to populate Http request and tracing information.
+If you use Maven, to use the servlet initializer add the following dependency to your BOM:
+
+    ```xml
+    <dependency>
+      <groupId>com.google.cloud</groupId>
+      <artifactId>google-cloud-logging-servlet-initializer</artifactId>
+    </dependency>      
+    ```
+
 
 
 

google-cloud-logging/clirr-ignored-differences.xml

@@ -0,0 +1,9 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!-- see http://www.mojohaus.org/clirr-maven-plugin/examples/ignored-differences.html -->
+<differences>
+  <difference>
+    <differenceType>7012</differenceType>
+    <className>com/google/cloud/logging/Logging</className>
+    <method>java.lang.Iterable populateMetadata(java.lang.Iterable, com.google.cloud.MonitoredResource, java.lang.String[])</method>
+  </difference>
+</differences>

google-cloud-logging/pom.xml

@@ -21,6 +21,11 @@
       <groupId>com.google.guava</groupId>
       <artifactId>guava</artifactId>
     </dependency>
+    <dependency>
+      <groupId>com.google.code.gson</groupId>
+      <artifactId>gson</artifactId>
+      <version>2.8.9</version>
+    </dependency>
     <dependency>
       <groupId>io.grpc</groupId>
       <artifactId>grpc-api</artifactId>

google-cloud-logging/src/main/java/com/google/cloud/logging/Context.java

@@ -49,7 +49,7 @@ public static final class Builder {
 
     /** Sets the HTTP request. */
     public Builder setRequest(HttpRequest request) {
-      this.requestBuilder = request.toBuilder();
+      this.requestBuilder = request != null ? request.toBuilder() : HttpRequest.newBuilder();
       return this;
     }
 

google-cloud-logging/src/main/java/com/google/cloud/logging/LogEntry.java

@@ -19,9 +19,17 @@
 import static com.google.common.base.Preconditions.checkNotNull;
 
 import com.google.cloud.MonitoredResource;
+import com.google.cloud.logging.Payload.Type;
 import com.google.common.base.Function;
 import com.google.common.base.MoreObjects;
 import com.google.common.collect.ImmutableMap;
+import com.google.gson.Gson;
+import com.google.gson.GsonBuilder;
+import com.google.gson.JsonElement;
+import com.google.gson.JsonObject;
+import com.google.gson.JsonPrimitive;
+import com.google.gson.JsonSerializationContext;
+import com.google.gson.JsonSerializer;
 import com.google.logging.v2.LogEntryOperation;
 import com.google.logging.v2.LogEntrySourceLocation;
 import com.google.logging.v2.LogName;
@@ -61,8 +69,8 @@ public LogEntry apply(com.google.logging.v2.LogEntry pb) {
   private final HttpRequest httpRequest;
   private final Map<String, String> labels;
   private final Operation operation;
-  private final Object trace;
-  private final Object spanId;
+  private final String trace;
+  private final String spanId;
   private final boolean traceSampled;
   private final SourceLocation sourceLocation;
   private final Payload<?> payload;
@@ -80,8 +88,8 @@ public static class Builder {
     private HttpRequest httpRequest;
     private Map<String, String> labels = new HashMap<>();
     private Operation operation;
-    private Object trace;
-    private Object spanId;
+    private String trace;
+    private String spanId;
     private boolean traceSampled;
     private SourceLocation sourceLocation;
     private Payload<?> payload;
@@ -245,7 +253,7 @@ public Builder setTrace(String trace) {
      * relative resource name, the name is assumed to be relative to `//tracing.googleapis.com`.
      */
     public Builder setTrace(Object trace) {
-      this.trace = trace;
+      this.trace = trace != null ? trace.toString() : null;
       return this;
     }
 
@@ -257,7 +265,7 @@ public Builder setSpanId(String spanId) {
 
     /** Sets the ID of the trace span associated with the log entry, if any. */
     public Builder setSpanId(Object spanId) {
-      this.spanId = spanId;
+      this.spanId = spanId != null ? spanId.toString() : null;
       return this;
     }
 
@@ -575,6 +583,142 @@ com.google.logging.v2.LogEntry toPb(String projectId) {
     return builder.build();
   }
 
+  /**
+   * Customized serializers to match the expected format for timestamp, source location and request
+   * method
+   */
+  static final class InstantSerializer implements JsonSerializer<Instant> {
+    @Override
+    public JsonElement serialize(
+        Instant src, java.lang.reflect.Type typeOfSrc, JsonSerializationContext context) {
+      return new JsonPrimitive(src.toString());
+    }
+  }
+
+  static final class SourceLocationSerializer implements JsonSerializer<SourceLocation> {
+    @Override
+    public JsonElement serialize(
+        SourceLocation src, java.lang.reflect.Type typeOfSrc, JsonSerializationContext context) {
+      JsonObject obj = new JsonObject();
+      if (src.getFile() != null) {
+        obj.addProperty("file", src.getFile());
+      }
+      if (src.getLine() != null) {
+        obj.addProperty("line", src.getLine().toString());
+      }
+      if (src.getFunction() != null) {
+        obj.addProperty("function", src.getFunction());
+      }
+      return obj;
+    }
+  }
+
+  static final class RequestMethodSerializer implements JsonSerializer<HttpRequest.RequestMethod> {
+    @Override
+    public JsonElement serialize(
+        HttpRequest.RequestMethod src,
+        java.lang.reflect.Type typeOfSrc,
+        JsonSerializationContext context) {
+      return new JsonPrimitive(src.name());
+    }
+  }
+
+  /** Helper class to format one line Json representation of the LogEntry for structured log. */
+  static final class StructuredLogFormatter {
+    private final Gson gson;
+    private final StringBuilder builder;
+
+    public StructuredLogFormatter(StringBuilder builder) {
+      checkNotNull(builder);
+      this.gson =
+          new GsonBuilder()
+              .registerTypeAdapter(Instant.class, new InstantSerializer())
+              .registerTypeAdapter(SourceLocation.class, new SourceLocationSerializer())
+              .registerTypeAdapter(HttpRequest.RequestMethod.class, new RequestMethodSerializer())
+              .create();
+      this.builder = builder;
+    }
+
+    /**
+     * Adds a Json field and value pair to the current string representation. Method does not
+     * validate parameters to be multi-line strings. Nothing is added if {@code value} parameter is
+     * {@code null}.
+     *
+     * @param name a valid Json field name string.
+     * @param value an object to be serialized to Json using {@link Gson}.
+     * @param appendComma a flag to add a trailing comma.
+     * @return a reference to this object.
+     */
+    public StructuredLogFormatter appendField(String name, Object value, boolean appendComma) {
+      checkNotNull(name);
+      if (value != null) {
+        builder.append(gson.toJson(name)).append(":").append(gson.toJson(value));
+        if (appendComma) {
+          builder.append(",");
+        }
+      }
+      return this;
+    }
+
+    public StructuredLogFormatter appendField(String name, Object value) {
+      return appendField(name, value, true);
+    }
+
+    /**
+     * Serializes a dictionary of key, values as Json fields.
+     *
+     * @param value a {@link Map} of key, value arguments to be serialized using {@link Gson}.
+     * @param appendComma a flag to add a trailing comma.
+     * @return a reference to this object.
+     */
+    public StructuredLogFormatter appendDict(Map<String, Object> value, boolean appendComma) {
+      if (value != null) {
+        String json = gson.toJson(value);
+        // append json object without brackets
+        if (json.length() > 1) {
+          builder.append(json.substring(0, json.length() - 1).substring(1));
+          if (appendComma) {
+            builder.append(",");
+          }
+        }
+      }
+      return this;
+    }
+  }
+
+  /**
+   * Serializes the object to a one line JSON string in the simplified format that can be parsed by
+   * the logging agents that run on Google Cloud resources.
+   */
+  public String toStructuredJsonString() {
+    if (payload.getType() == Type.PROTO) {
+      throw new UnsupportedOperationException("LogEntry with protobuf payload cannot be converted");
+    }
+
+    final StringBuilder builder = new StringBuilder("{");
+    final StructuredLogFormatter formatter = new StructuredLogFormatter(builder);
+
+    formatter
+        .appendField("severity", severity)
+        .appendField("timestamp", timestamp)
+        .appendField("httpRequest", httpRequest)
+        .appendField("logging.googleapis.com/insertId", insertId)
+        .appendField("logging.googleapis.com/labels", labels)
+        .appendField("logging.googleapis.com/operation", operation)
+        .appendField("logging.googleapis.com/sourceLocation", sourceLocation)
+        .appendField("logging.googleapis.com/spanId", spanId)
+        .appendField("logging.googleapis.com/trace", trace)
+        .appendField("logging.googleapis.com/trace_sampled", traceSampled);
+    if (payload.getType() == Type.STRING) {
+      formatter.appendField("message", payload.getData(), false);
+    } else if (payload.getType() == Type.JSON) {
+      Payload.JsonPayload jsonPayload = (Payload.JsonPayload) payload;
+      formatter.appendDict(jsonPayload.getDataAsMap(), false);
+    }
+    builder.append("}");
+    return builder.toString();
+  }
+
   /** Returns a builder for {@code LogEntry} objects given the entry payload. */
   public static Builder newBuilder(Payload<?> payload) {
     return new Builder(payload);