Merge pull request ucsd-progsys#2177 from clayrat/plot-performance

Better performance plotting script and readme updates

Author: nikivazou

.gitignore

@@ -59,6 +59,9 @@ tests/**/*.dyn_o
 tests/**/*.hi-boot
 tests/**/*.o-boot
 
+tmp/
+*.svg
+
 .idea
 *.iml
 

README.md

@@ -25,7 +25,7 @@ to let us know. If possible, try to:
 
 ## Your first Pull Request
 
-We are thrilled to get PRs! Please follow these guidelines, as doing so will increase the chances of 
+We are thrilled to get PRs! Please follow these guidelines, as doing so will increase the chances of
 having your PR accepted:
 
 * The main LH repo [lives here](https://github.com/ucsd-progsys/liquidhaskell)
@@ -98,7 +98,7 @@ It's also possible (but not recommended) to add `LIQUID_DEV_MODE` to .bashrc or
 permanently disable building the `liquid-*` packages, and this might silently mask breaking changes to the
 `liquidhaskell` library that would manifest only when compiling these other packages.
 
-If you wish to force building all the libraries again, it's sufficient to issue the same builds commands 
+If you wish to force building all the libraries again, it's sufficient to issue the same builds commands
 without the `LIQUID_DEV_MODE`.
 
 ## How To Run Regression Tests
@@ -115,7 +115,7 @@ You can run a bunch of particular test-groups instead by
 
     $ LIQUID_DEV_MODE=true ./scripts/test/test_plugin.sh <test-group-name1> <test-group-name2> ...
 
-and you can list all the possible test options with 
+and you can list all the possible test options with
 
     $ LIQUID_DEV_MODE=true ./scripts/test/test_plugin.sh --help
 
@@ -147,8 +147,8 @@ When `liquidhaskell` tests run, we can collect timing information with
 
     $ ./scripts/test/test_plugin.sh --measure-timings
 
-Measures will be collected in `.dump-timings` files. These can be converted to json
-data with
+Measures will be collected in `.dump-timings` files under `dist-newstyle` directory. These can be
+converted to json data with
 
 ```bash
 cabal v2-build ghc-timings
@@ -163,16 +163,26 @@ cabal v2-run benchmark-timings -- tmp/*.json --phase LiquidHaskell -o summary.cs
 ```
 On each line, the report will contain the time taken by each test.
 
-There is a script `scripts/plot-performance/chart_perf.sh` that can be
-used to generate comparison charts in `svg` and `png` formats. It
-requires [gnuplot](http://www.gnuplot.info/) to run. The following
-command will produce two files `perf.svg` and `perf.png` in the
-current directory.
+Comparison charts in `svg` format can be generated by invoking
+
+```
+cabal v2-run plot-performance -- -b path_to_before_summary.csv -a path_to_after_summary.csv -s 50 -f "benchmark" -o outdir
+```
+
+This will generate three files `filtered.svg` (a subset of tests with a `benchmark` prefix, enabled by the `-f` option),
+`top.svg` and `bot.svg` (top 50 speedups and slowdowns over the entire test set, both enabled by the `-s` option) under
+the `outdir` directory. The `-f` and `-s` options can be used/omitted independently. If both are omitted, a single
+`perf.svg` will be produced covering the full input test set. Additionally, their effects can be combined by providing
+a third `-c` option (this will produce 2 files `filtered-top.svg` and `filtered-bot.svg` instead of 3).
+
+There is also a legacy script `scripts/plot-performance/chart_perf.sh` that can be used to generate comparison charts
+in both `svg` and `png` formats. It requires [gnuplot](http://www.gnuplot.info/) to run and assumes both files contain
+the same test set. The following command will produce two files `perf.svg` and `perf.png` in the current directory.
 
     $ scripts/plot-performance/chart_perf.sh path_to_before_summary.csv path_to_after_summary.csv
 
-The current formatting is optimized for comparing the outputs of running
-the benchmarks alone.
+The current formatting is optimized for comparing some subsets of the full test run, typically just the benchmarks alone.
+If one wishes to save time or is not interested in top speedups/slowdowns, the benchmark subset can be obtained by running
 
     $ scripts/test/test_plugin.sh \
         benchmark-stitch-lh \
@@ -185,7 +195,7 @@ the benchmarks alone.
 
 ## Miscelaneous tasks
 
-* **Profiling** See the instructions in [scripts/ProfilingDriver.hs][].
+* **Profiling** See the instructions in [scripts/ProfilingDriver.hs](scripts/ProfilingDriver.hs).
 * **Getting stack traces on exceptions** See `-xc` flag in the [GHC user's guide][ghc-users-guide].
 * **Working with submodules** See `man gitsubmodules` or the [git documentation site][git-documentation].
 
@@ -338,7 +348,7 @@ any `LiftedSpec` from the interface file's annotations.
 Typically the first thing you might want to do is to run a "clean" `cabal v2-build` or `stack build` using
 the latest compiler and "check the damage". If you are lucky, everything works out of the box, otherwise
 compilation might fail with an error, typically because some `ghc` API function has been removed/moved/renamed.
-The way to fix it is to modify the [GHC.API][] shim module and perform any required change, likely by 
+The way to fix it is to modify the [GHC.API][] shim module and perform any required change, likely by
 conditionally compiling some code in a `CPP` block. For minor changes, it's usually enough to perform small
 changes, but for more tricky migrations it might be necessary to backport some GHC code, or create some
 patter synonym to deal with changes in type constructors.
@@ -348,8 +358,6 @@ patter synonym to deal with changes in type constructors.
 Currently, no. Only one version of GHC is supported and that is the one
 that can be tested with `./scripts/test/test_plugin.sh`.
 
-[GHC.API]: src-ghc/Liquid/GHC/API.hs
-
 # GHC Plugin Development FAQs
 
 ## Why is the GHC.Interface using slightly different types than the GHC.Plugin module?
@@ -361,24 +369,24 @@ to map back and forth (sometimes in a partial way) between old and new data stru
 **consider the GHC.Plugin as the single source of truth, and prefer whichever data structure the latter is
 using**.
 
-
-[Plugin]:              src/Language/Haskell/Liquid/GHC/Plugin.hs
-[GHC.Plugin]:          src/Language/Haskell/Liquid/GHC/Plugin.hs
-[GHC.Interface]:       src-ghc/Liquid/GHC/Interface.hs
-[SpecFinder]:          src/Language/Haskell/Liquid/GHC/Plugin/SpecFinder.hs
-[BareSpec]:            src/Language/Haskell/Liquid/Types/Specs.hs#L361
-[LiftedSpec]:          src/Language/Haskell/Liquid/Types/Specs.hs#L554
-[TargetSrc]:           src/Language/Haskell/Liquid/Types/Specs.hs#L157
-[Ghc monad]:           https://hackage.haskell.org/package/ghc-8.10.1/docs/GHC.html#t:Ghc
-[HscEnv]:              https://hackage.haskell.org/package/ghc-8.10.1/docs/GHC.html#t:HscEnv
-[DynFlags]:            https://hackage.haskell.org/package/ghc-8.10.1/docs/GHC.html#t:DynFlags
-[GhcMonad]:            https://hackage.haskell.org/package/ghc-8.10.1/docs/GHC.html#t:GhcMonad
-[GhcMonadLike]:        src-ghc/Liquid/GHC/GhcMonadLike.hs
-[typechecking phase]:  src/Language/Haskell/Liquid/GHC/Plugin.hs#L206-L222
+[GHC.API]:             liquidhaskell-boot/src-ghc/Liquid/GHC/API.hs
+[Plugin]:              liquidhaskell-boot/src/Language/Haskell/Liquid/GHC/Plugin.hs
+[GHC.Plugin]:          liquidhaskell-boot/src/Language/Haskell/Liquid/GHC/Plugin.hs
+[GHC.Interface]:       liquidhaskell-boot/src-ghc/Liquid/GHC/Interface.hs
+[SpecFinder]:          liquidhaskell-boot/src/Language/Haskell/Liquid/GHC/Plugin/SpecFinder.hs
+[BareSpec]:            liquidhaskell-boot/src/Language/Haskell/Liquid/Types/Specs.hs#L362
+[LiftedSpec]:          liquidhaskell-boot/src/Language/Haskell/Liquid/Types/Specs.hs#L559
+[TargetSrc]:           liquidhaskell-boot/src/Language/Haskell/Liquid/Types/Specs.hs#L158
+[Ghc monad]:           https://hackage.haskell.org/package/ghc-9.2.5/docs/GHC.html#t:Ghc
+[HscEnv]:              https://hackage.haskell.org/package/ghc-9.2.5/docs/GHC.html#t:HscEnv
+[DynFlags]:            https://hackage.haskell.org/package/ghc-9.2.5/docs/GHC.html#t:DynFlags
+[GhcMonad]:            https://hackage.haskell.org/package/ghc-9.2.5/docs/GHC.html#t:GhcMonad
+[GhcMonadLike]:        liquidhaskell-boot/src-ghc/Liquid/GHC/GhcMonadLike.hs
+[typechecking phase]:  liquidhaskell-boot/src/Language/Haskell/Liquid/GHC/Plugin.hs#L211-L226
 [ghcide]:              https://github.com/haskell/ghcide
-[findRelevantSpecs]:   src/Language/Haskell/Liquid/GHC/Plugin/SpecFinder.hs#L61
-[core binds]:          https://hackage.haskell.org/package/ghc-8.10.1/docs/CoreSyn.html#t:CoreBind
-[configureGhcTargets]: src-ghc/Liquid/GHC/Interface.hs#L252
-[processTargetModule]: src-ghc/Liquid/GHC/Interface.hs#L481
-[processModule]:       src/Language/Haskell/Liquid/GHC/Plugin.hs#L393
+[findRelevantSpecs]:   liquidhaskell-boot/src/Language/Haskell/Liquid/GHC/Plugin/SpecFinder.hs#L65
+[core binds]:          https://hackage.haskell.org/package/ghc-9.2.5/docs/CoreSyn.html#t:CoreBind
+[configureGhcTargets]: liquidhaskell-boot/src-ghc/Liquid/GHC/Interface.hs#L254
+[processTargetModule]: liquidhaskell-boot/src-ghc/Liquid/GHC/Interface.hs#L483
+[processModule]:       liquidhaskell-boot/src/Language/Haskell/Liquid/GHC/Plugin.hs#L509
 

benchmark-timings/app/Main.hs

@@ -49,18 +49,18 @@ data Options = Options
 
 options :: Parser Options
 options = Options <$>
-  (many (argument
+  many (argument
          str
           (metavar "FILEPATH..."
-           <> help "The files you wish to process.")))
-  <*> (many (strOption (long "phase"
+           <> help "The files you wish to process."))
+  <*> many (strOption (long "phase"
                         <> short 'p'
                         <> metavar "PHASE"
-                        <> help "Phase to include in summary.  Can be specified more thance once.")))
-  <*> (strOption (long "output"
+                        <> help "Phase to include in summary.  Can be specified more thance once."))
+  <*> strOption (long "output"
                   <> short 'o'
                   <> metavar "OUTPUTFILEPATH"
-                  <> help "File to which to output CSV contents."))
+                  <> help "File to which to output CSV contents.")
 
 opts :: ParserInfo Options
 opts = info (options <**> helper)

cabal.project

@@ -10,12 +10,19 @@ packages: .
           ./tests
           ./tests/benchmarks/popl18/lib
           ./benchmark-timings
+          ./scripts/plot-performance
 
 source-repository-package
     type: git
     location: https://github.com/qnikst/ghc-timings-report
     tag: 45ef3498e35897712bde8e002ce18df6d55f8b15
 
+source-repository-package
+    type: git
+    location: https://github.com/timbod7/haskell-chart/
+    tag: ba85444d3c81774a43b8c2b046084e92869914f4
+    subdir: chart
+
 allow-newer: ghc-timings:base, rest-rewrite:time
 
 package liquid-fixpoint

scripts/plot-benchmarks/src/Benchmark.hs

@@ -21,11 +21,11 @@ instance Ord Benchmark where
    compare lhs rhs = compare (benchTimestamp lhs) (benchTimestamp rhs)
 
 unionAppend :: Map.Map String [Benchmark]
-               -> Map.Map String Benchmark
-               -> Map.Map String [Benchmark]
+            -> Map.Map String Benchmark
+            -> Map.Map String [Benchmark]
 unionAppend l r = Map.unionWith (++) l r'
    where
-      r' = fmap (\a -> [a]) r
+      r' = (\a -> [a]) <$> r
 
 toBenchMap :: (Foldable f)
               => f Benchmark
@@ -40,8 +40,7 @@ instance FromRecord Benchmark where
                    <*> pure (error ("Shouldn't be evaluated until after"
                                     ++ " reassignment!"))
                    <*> r .! 1
-                   <*> do asStr <- r .! 2
-                          return $ read asStr {- Since the test suite
+                   <*> (read <$> r .! 2) {- Since the test suite
    generates this field by calling show, this read Should Be Safe (TM) -}
 
 csvOutName = "Name"
@@ -51,6 +50,6 @@ csvOutPass = "Success"
 
 instance ToNamedRecord (LocalTime, Benchmark) where
    toNamedRecord (_, bm) = namedRecord [csvOutName .= benchName bm,
-                                   csvOutDate .= (show $ benchTimestamp bm),
-                                   csvOutTime .= (benchTime bm),
-                                   csvOutPass .= (show $ benchPass bm)]
+                                        csvOutDate .= (show $ benchTimestamp bm),
+                                        csvOutTime .= (benchTime bm),
+                                        csvOutPass .= (show $ benchPass bm)]

scripts/plot-benchmarks/src/Config.hs

@@ -5,9 +5,7 @@ module Config where
 import System.Console.CmdArgs
 import System.Directory
 
-data OutputType =
-   Svg
-   | Csv
+data OutputType = Svg | Csv
      deriving (Eq, Data, Typeable, Show)
 
 data Config =
@@ -32,7 +30,7 @@ instance Default Config where
 config :: Config
 config = Config
             { logDir = pwd &= help "The directory that contains the logs",
-              outputDir = pwd &= help "The diretory to output graphs to",
+              outputDir = pwd &= help "The directory to output graphs to",
               outputType = Csv &= help "The type of output to produce",
               plotCompare = def &= help "Pairs of benchmarks to compare",
               plot = def &= help "Benchmarks to plot"

scripts/plot-benchmarks/src/Main.hs

@@ -5,26 +5,13 @@ import Config
 import Parse
 
 main :: IO ()
-main = do
-   conf <- getConfig
-   case (outputType conf) of
-      Csv -> do
-         csvData <- getAllData
-                       (logDir conf)
-                       (plot conf)
-         dumpLogs
-            (outputDir conf)
-            csvData
-      Svg -> do
-         timeData <- getTimeData
-                        (logDir conf)
-                        (plot conf)
-         plotTimeData
-            (outputDir conf)
-            timeData
-         compareTimeData <- getCompareTimeData
-                               (logDir conf)
-                               (plotCompare conf)
-         plotCompareTimeData
-            (outputDir conf)
-            compareTimeData
+main =
+  do conf <- getConfig
+     case outputType conf of
+        Csv -> do csvData <- getAllData (logDir conf) (plot conf)
+                  dumpLogs (outputDir conf) csvData
+        Svg -> do timeData <- getTimeData (logDir conf) (plot conf)
+                  plotTimeData (outputDir conf) timeData
+                  compareTimeData <- getCompareTimeData (logDir conf)
+                                                        (plotCompare conf)
+                  plotCompareTimeData (outputDir conf) compareTimeData