Larger scale testing

[quintona]

The readme mentions that your tests prove that spark isn't ready for larger scales, but it doesn't specify why. Is there a roadmap to do larger scale testing?

[bamos]

Hi Quinton,

Great question, we don't think Spindle isn't ready for larger scale experiments
because of an experiment we ran scaling up the number of Spark
and HDFS workers. We've added the content below to the README.

The following table shows the query execution times from starting with 3
servers and scaling up to 6 servers while rebalancing the HDFS replicated data.
The bold text shows where adding an additional server slowed down the
performance for our queries and small data set, so scaling up often hurt performance.
Fully utilizing all 6 servers often results in longer query execution
times than using some subset of servers.

Also, we are interested in seeing how Spindle scales to more servers,
but this is not currently on the roadmap.
We've noted in the README that Spindle serves
Adobe as a research prototype and is unfortunately
not under active development.

Regards,
Brandon.