Skip to content

Improve jackson benchmark in java #303

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
xerial merged 3 commits into from
Dec 25, 2015
Merged

Improve jackson benchmark in java #303

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3d9e168
Improve MessagePackDataformat(Pojo|HugeData)BenchmarkTest
komamitsu Dec 23, 2015
4d2594e
Use ObjectMapper#writeValue() instead of writeValueAsBytes() in bench…
komamitsu Dec 23, 2015
9b2f4e3
Take care of the order of benchmarks and remove outliers
komamitsu Dec 25, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 0 additions & 13 deletions ...k-jackson/src/test/java/org/msgpack/jackson/dataformat/MessagePackDataformatTestBase.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,6 @@
import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.apache.commons.math3.stat.StatUtils;
import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;
import org.junit.After;
import org.junit.Before;

Expand Down Expand Up @@ -99,17 +97,6 @@ public void teardown()
}
}

protected void printStat(String label, double[] values)
{
StandardDeviation standardDeviation = new StandardDeviation();
System.out.println(label + ":");
System.out.println(String.format(" mean : %.2f", StatUtils.mean(values)));
System.out.println(String.format(" min : %.2f", StatUtils.min(values)));
System.out.println(String.format(" max : %.2f", StatUtils.max(values)));
System.out.println(String.format(" stdev: %.2f", standardDeviation.evaluate(values)));
System.out.println("");
}

public enum Suit
{
SPADE, HEART, DIAMOND, CLUB;
Expand Down
98 changes: 98 additions & 0 deletions msgpack-jackson/src/test/java/org/msgpack/jackson/dataformat/benchmark/Benchmarker.java
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
//
// MessagePack for Java
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
package org.msgpack.jackson.dataformat.benchmark;

import org.apache.commons.math3.stat.StatUtils;
import org.apache.commons.math3.stat.descriptive.moment.StandardDeviation;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class Benchmarker
{
private final List<Benchmarkable> benchmarkableList = new ArrayList<Benchmarkable>();

public abstract static class Benchmarkable
{
private final String label;

protected Benchmarkable(String label)
{
this.label = label;
}

public abstract void run() throws Exception;
}

public void addBenchmark(Benchmarkable benchmark)
{
benchmarkableList.add(benchmark);
}

private static class Tuple<F, S>
{
F first;
S second;

public Tuple(F first, S second)
{
this.first = first;
this.second = second;
}
}

public void run(int count, int warmupCount)
throws Exception
{
List<Tuple<String, double[]>> benchmarksResults = new ArrayList<Tuple<String, double[]>>(benchmarkableList.size());
for (Benchmarkable benchmark : benchmarkableList) {
Copy link
Copy Markdown
Member

@xerial xerial Dec 24, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The order of running programs also affects the performance results in JVM. Simply running System.gc() as an warmup is not sufficient.

Copy link
Copy Markdown
Member Author

@komamitsu komamitsu Dec 24, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The benchmark has 4 times warmup runs for each one and it avoids the affect of running order, I think. BTW, how does your benchmark avoid the order of running programs?

Copy link
Copy Markdown
Member

@xerial xerial Dec 24, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example, if you have two programs P1, P2, it runs the program P1, P2, P1, P2,... to reduce the order effect.

Copy link
Copy Markdown
Member Author

@komamitsu komamitsu Dec 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. The original benchmark code (https://github.com/msgpack/msgpack-java/blob/v07-develop/msgpack-jackson/src/test/java/org/msgpack/jackson/dataformat/benchmark/MessagePackDataformatPojoBenchmarkTest.java#L98-L130) does the same thing, right?

benchmarksResults.add(new Tuple<String, double[]>(benchmark.label, new double[count]));
}

for (int i = 0; i < count + warmupCount; i++) {
for (int bi = 0; bi < benchmarkableList.size(); bi++) {
Benchmarkable benchmark = benchmarkableList.get(bi);
long currentTimeNanos = System.nanoTime();
benchmark.run();

if (i >= warmupCount) {
benchmarksResults.get(bi).second[i - warmupCount] = (System.nanoTime() - currentTimeNanos) / 1000000.0;
}
}
}

for (Tuple<String, double[]> benchmarkResult : benchmarksResults) {
printStat(benchmarkResult.first, benchmarkResult.second);
}
}

private void printStat(String label, double[] origValues)
{
double[] values = origValues;
Arrays.sort(origValues);
if (origValues.length > 2) {
values = Arrays.copyOfRange(origValues, 1, origValues.length - 1);
}
StandardDeviation standardDeviation = new StandardDeviation();
System.out.println(label + ":");
System.out.println(String.format(" mean : %8.3f", StatUtils.mean(values)));
Copy link
Copy Markdown
Member

@xerial xerial Dec 24, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The mean value including outliers (min, max) is not a good measure in evaluating the performance. We also need to add median or mean (average) value without containing outliers (e.g., by removing max and min, or a few percent of the top results).

Copy link
Copy Markdown
Member Author

@komamitsu komamitsu Dec 24, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the result of profiling, this method accounts for 1% of the total duration. Also, we compare the performance of Jackson and MessagePack relatively with this benchmark even if this method affects a bit.

Copy link
Copy Markdown
Member

@xerial xerial Dec 24, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just meant we should see median or average (which is not biased by too distant values).

Copy link
Copy Markdown
Member Author

@komamitsu komamitsu Dec 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, I misread your comment. I'll take care of it

System.out.println(String.format(" min : %8.3f", StatUtils.min(values)));
System.out.println(String.format(" max : %8.3f", StatUtils.max(values)));
System.out.println(String.format(" stdev: %8.3f", standardDeviation.evaluate(values)));
System.out.println("");
}
}
87 changes: 62 additions & 25 deletions .../org/msgpack/jackson/dataformat/benchmark/MessagePackDataformatHugeDataBenchmarkTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,21 +15,24 @@
//
package org.msgpack.jackson.dataformat.benchmark;

import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.Test;
import org.msgpack.jackson.dataformat.MessagePackDataformatTestBase;
import org.msgpack.jackson.dataformat.MessagePackFactory;

import java.io.File;
import java.io.FileOutputStream;
import java.io.OutputStream;
import java.util.ArrayList;
import java.util.List;

public class MessagePackDataformatHugeDataBenchmarkTest
extends MessagePackDataformatTestBase
{
private static final int ELM_NUM = 1000000;
private static final int SAMPLING_COUNT = 4;
private static final int ELM_NUM = 100000;
private static final int COUNT = 6;
private static final int WARMUP_COUNT = 4;
private final ObjectMapper origObjectMapper = new ObjectMapper();
private final ObjectMapper msgpackObjectMapper = new ObjectMapper(new MessagePackFactory());
private static final List<Object> value;
Expand Down Expand Up @@ -66,34 +69,68 @@ public class MessagePackDataformatHugeDataBenchmarkTest
packedByMsgPack = bytes;
}

public MessagePackDataformatHugeDataBenchmarkTest()
{
origObjectMapper.configure(JsonGenerator.Feature.AUTO_CLOSE_TARGET, false);
msgpackObjectMapper.configure(JsonGenerator.Feature.AUTO_CLOSE_TARGET, false);
}

@Test
public void testBenchmark()
throws Exception
{
double[] durationOfSerializeWithJson = new double[SAMPLING_COUNT];
double[] durationOfSerializeWithMsgPack = new double[SAMPLING_COUNT];
double[] durationOfDeserializeWithJson = new double[SAMPLING_COUNT];
double[] durationOfDeserializeWithMsgPack = new double[SAMPLING_COUNT];
for (int si = 0; si < SAMPLING_COUNT; si++) {
long currentTimeMillis = System.currentTimeMillis();
origObjectMapper.writeValueAsBytes(value);
durationOfSerializeWithJson[si] = System.currentTimeMillis() - currentTimeMillis;
Benchmarker benchmarker = new Benchmarker();

File tempFileJackson = File.createTempFile("msgpack-jackson-", "-huge-jackson");
tempFileJackson.deleteOnExit();
final OutputStream outputStreamJackson = new FileOutputStream(tempFileJackson);

File tempFileMsgpack = File.createTempFile("msgpack-jackson-", "-huge-msgpack");
tempFileMsgpack.deleteOnExit();
final OutputStream outputStreamMsgpack = new FileOutputStream(tempFileMsgpack);

currentTimeMillis = System.currentTimeMillis();
msgpackObjectMapper.writeValueAsBytes(value);
durationOfSerializeWithMsgPack[si] = System.currentTimeMillis() - currentTimeMillis;
benchmarker.addBenchmark(new Benchmarker.Benchmarkable("serialize(huge) with JSON") {
@Override
public void run()
throws Exception
{
origObjectMapper.writeValue(outputStreamJackson, value);
}
});

currentTimeMillis = System.currentTimeMillis();
origObjectMapper.readValue(packedByOriginal, new TypeReference<List<Object>>() {});
durationOfDeserializeWithJson[si] = System.currentTimeMillis() - currentTimeMillis;
benchmarker.addBenchmark(new Benchmarker.Benchmarkable("serialize(huge) with MessagePack") {
@Override
public void run()
throws Exception
{
msgpackObjectMapper.writeValue(outputStreamMsgpack, value);
}
});

currentTimeMillis = System.currentTimeMillis();
msgpackObjectMapper.readValue(packedByMsgPack, new TypeReference<List<Object>>() {});
durationOfDeserializeWithMsgPack[si] = System.currentTimeMillis() - currentTimeMillis;
benchmarker.addBenchmark(new Benchmarker.Benchmarkable("deserialize(huge) with JSON") {
@Override
public void run()
throws Exception
{
origObjectMapper.readValue(packedByOriginal, new TypeReference<List<Object>>() {});
}
});

benchmarker.addBenchmark(new Benchmarker.Benchmarkable("deserialize(huge) with MessagePack") {
@Override
public void run()
throws Exception
{
msgpackObjectMapper.readValue(packedByMsgPack, new TypeReference<List<Object>>() {});
}
});

try {
benchmarker.run(COUNT, WARMUP_COUNT);
}
finally {
outputStreamJackson.close();
outputStreamMsgpack.close();
}
printStat("serialize(huge) with JSON", durationOfSerializeWithJson);
printStat("serialize(huge) with MessagePack", durationOfSerializeWithMsgPack);
printStat("deserialize(huge) with JSON", durationOfDeserializeWithJson);
printStat("deserialize(huge) with MessagePack", durationOfDeserializeWithMsgPack);
}
}
107 changes: 73 additions & 34 deletions ...java/org/msgpack/jackson/dataformat/benchmark/MessagePackDataformatPojoBenchmarkTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,22 +15,27 @@
//
package org.msgpack.jackson.dataformat.benchmark;

import com.fasterxml.jackson.core.JsonGenerator;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.ObjectMapper;
import org.junit.Test;
import org.msgpack.jackson.dataformat.MessagePackDataformatTestBase;
import org.msgpack.jackson.dataformat.MessagePackFactory;
import static org.msgpack.jackson.dataformat.MessagePackDataformatTestBase.NormalPojo;
import static org.msgpack.jackson.dataformat.MessagePackDataformatTestBase.Suit;

import java.io.File;
import java.io.FileOutputStream;
import java.io.OutputStream;
import java.math.BigInteger;
import java.util.ArrayList;
import java.util.List;

public class MessagePackDataformatPojoBenchmarkTest
extends MessagePackDataformatTestBase
{
private static final int LOOP_MAX = 1000;
private static final int LOOP_FACTOR = 50;
private static final int SAMPLING_COUNT = 4;
private static final int LOOP_MAX = 600;
private static final int LOOP_FACTOR = 40;
private static final int COUNT = 6;
private static final int WARMUP_COUNT = 4;
private static final List<NormalPojo> pojos = new ArrayList<NormalPojo>(LOOP_MAX);
private static final List<byte[]> pojosSerWithOrig = new ArrayList<byte[]>(LOOP_MAX);
private static final List<byte[]> pojosSerWithMsgPack = new ArrayList<byte[]>(LOOP_MAX);
Expand Down Expand Up @@ -87,50 +92,84 @@ public class MessagePackDataformatPojoBenchmarkTest
}
}

public MessagePackDataformatPojoBenchmarkTest()
{
origObjectMapper.configure(JsonGenerator.Feature.AUTO_CLOSE_TARGET, false);
msgpackObjectMapper.configure(JsonGenerator.Feature.AUTO_CLOSE_TARGET, false);
}

@Test
public void testBenchmark()
throws Exception
{
double[] durationOfSerializeWithJson = new double[SAMPLING_COUNT];
double[] durationOfSerializeWithMsgPack = new double[SAMPLING_COUNT];
double[] durationOfDeserializeWithJson = new double[SAMPLING_COUNT];
double[] durationOfDeserializeWithMsgPack = new double[SAMPLING_COUNT];
for (int si = 0; si < SAMPLING_COUNT; si++) {
long currentTimeMillis = System.currentTimeMillis();
for (int j = 0; j < LOOP_FACTOR; j++) {
for (int i = 0; i < LOOP_MAX; i++) {
origObjectMapper.writeValueAsBytes(pojos.get(i));
Benchmarker benchmarker = new Benchmarker();

File tempFileJackson = File.createTempFile("msgpack-jackson-", "-huge-jackson");
tempFileJackson.deleteOnExit();
final OutputStream outputStreamJackson = new FileOutputStream(tempFileJackson);

File tempFileMsgpack = File.createTempFile("msgpack-jackson-", "-huge-msgpack");
tempFileMsgpack.deleteOnExit();
final OutputStream outputStreamMsgpack = new FileOutputStream(tempFileMsgpack);

benchmarker.addBenchmark(new Benchmarker.Benchmarkable("serialize(pojo) with JSON") {
@Override
public void run()
throws Exception
{
for (int j = 0; j < LOOP_FACTOR; j++) {
for (int i = 0; i < LOOP_MAX; i++) {
origObjectMapper.writeValue(outputStreamJackson, pojos.get(i));
}
}
}
durationOfSerializeWithJson[si] = System.currentTimeMillis() - currentTimeMillis;
});

currentTimeMillis = System.currentTimeMillis();
for (int j = 0; j < LOOP_FACTOR; j++) {
for (int i = 0; i < LOOP_MAX; i++) {
msgpackObjectMapper.writeValueAsBytes(pojos.get(i));
benchmarker.addBenchmark(new Benchmarker.Benchmarkable("serialize(pojo) with MessagePack") {
@Override
public void run()
throws Exception
{
for (int j = 0; j < LOOP_FACTOR; j++) {
for (int i = 0; i < LOOP_MAX; i++) {
msgpackObjectMapper.writeValue(outputStreamMsgpack, pojos.get(i));
}
}
}
durationOfSerializeWithMsgPack[si] = System.currentTimeMillis() - currentTimeMillis;
});

currentTimeMillis = System.currentTimeMillis();
for (int j = 0; j < LOOP_FACTOR; j++) {
for (int i = 0; i < LOOP_MAX; i++) {
origObjectMapper.readValue(pojosSerWithOrig.get(i), NormalPojo.class);
benchmarker.addBenchmark(new Benchmarker.Benchmarkable("deserialize(pojo) with JSON") {
@Override
public void run()
throws Exception
{
for (int j = 0; j < LOOP_FACTOR; j++) {
for (int i = 0; i < LOOP_MAX; i++) {
origObjectMapper.readValue(pojosSerWithOrig.get(i), NormalPojo.class);
}
}
}
durationOfDeserializeWithJson[si] = System.currentTimeMillis() - currentTimeMillis;
});

currentTimeMillis = System.currentTimeMillis();
for (int j = 0; j < LOOP_FACTOR; j++) {
for (int i = 0; i < LOOP_MAX; i++) {
msgpackObjectMapper.readValue(pojosSerWithMsgPack.get(i), NormalPojo.class);
benchmarker.addBenchmark(new Benchmarker.Benchmarkable("deserialize(pojo) with MessagePack") {
@Override
public void run()
throws Exception
{
for (int j = 0; j < LOOP_FACTOR; j++) {
for (int i = 0; i < LOOP_MAX; i++) {
msgpackObjectMapper.readValue(pojosSerWithMsgPack.get(i), NormalPojo.class);
}
}
}
durationOfDeserializeWithMsgPack[si] = System.currentTimeMillis() - currentTimeMillis;
});

try {
benchmarker.run(COUNT, WARMUP_COUNT);
}
finally {
outputStreamJackson.close();
outputStreamMsgpack.close();
}
printStat("serialize(pojo) with JSON", durationOfSerializeWithJson);
printStat("serialize(pojo) with MessagePack", durationOfSerializeWithMsgPack);
printStat("deserialize(pojo) with JSON", durationOfDeserializeWithJson);
printStat("deserialize(pojo) with MessagePack", durationOfDeserializeWithMsgPack);
}
}