sentinel-dashboard TreeMap ConcurrentModificationException

[huangxfchn]

Issue Description

When I use the dashboard, 'MetricFetcher' throws a ConcurrentModificationException and requests the home page, Jackson also throws the same exception.

Describe what happened (or what feature you want)

Here is the exception stack traces:

2018-10-26 14:58:00 [sentinel-dashboard-metrics-fetchWorker-thread-59] INFO  c.t.c.s.d.metric.MetricFetcher - fetchOnce(spark-feature-online) error
java.util.ConcurrentModificationException: null
        at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
        at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265)
        at com.taobao.csp.sentinel.dashboard.metric.MetricFetcher.fetchOnce(MetricFetcher.java:185)
        at com.taobao.csp.sentinel.dashboard.metric.MetricFetcher.lambda$doFetchAppMetric$3(MetricFetcher.java:258)

and

2018-10-26 14:57:44 [http-nio-9090-exec-7] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is org.springframework.http.converter.HttpMessageNotWritableException: Could not write JSON: (was java.util.ConcurrentModificationException); nested exception is com.fasterxml.jackson.databind.JsonMappingException: (was java.util.ConcurrentModificationException) (through reference chain: com.taobao.csp.sentinel.dashboard.view.Result["data"]->java.util.ArrayList[3]->com.taobao.csp.sentinel.dashboard.discovery.AppInfo["machines"]->java.util.TreeSet[378])] with root cause
java.util.ConcurrentModificationException: null
        at java.util.TreeMap$PrivateEntryIterator.nextEntry(TreeMap.java:1211)
        at java.util.TreeMap$KeyIterator.next(TreeMap.java:1265)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serializeContents(CollectionSerializer.java:133)
        at com.fasterxml.jackson.databind.ser.std.CollectionSerializer.serialize(CollectionSerializer.java:112)
        ......
        at com.fasterxml.jackson.databind.ObjectWriter.writeValue(ObjectWriter.java:951)
        at org.springframework.http.converter.json.AbstractJackson2HttpMessageConverter.writeInternal(AbstractJackson2HttpMessageConverter.java:286)
        at org.springframework.http.converter.AbstractGenericHttpMessageConverter.write(AbstractGenericHttpMessageConverter.java:106)
        at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodProcessor.writeWithMessageConverters(AbstractMessageConverterMethodProcessor.java:231)
        at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.handleReturnValue(RequestResponseBodyMethodProcessor.java:174)
        ......
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:635)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)

Describe what you expected to happen

AppInfo uses synchronized for the getMachines and addMachine methods, but calls the getmachines method to get the same TreeSet, so there is no guarantee that Set<MachineInfo> machines = new TreeSet<MachineInfo >(); is modified by multiple threads, so it is very likely that a ConcurrentModificationException exception will be thrown.

However, I did not find that the Set<MachineInfo> was modified by calling the getmachines method.

Here is the code for AppInfo：

public class AppInfo {

    private Set<MachineInfo> machines = new TreeSet<MachineInfo>();

    public synchronized Set<MachineInfo> getMachines() {
        return machines;
    }

    public synchronized boolean addMachine(MachineInfo machineInfo) {
        machines.remove(machineInfo);
        return machines.add(machineInfo);
    }

}

How to reproduce it (as minimally and precisely as possible)

1. When calling the getMachines method, create a new TreeSet to avoid concurrency issues.

[sczyh30]

Thanks for reporting. We'll fix and validate it.