Compiler perf infrastructure

[En3Tho]

I'm going to have some free time soon and I would like to try contributing some more to F# (my last contributions were quite small, but well..). And the space I'm interested in is compiler perf. Mainly because:

1. I personally like tinkering with perfview, optimizations (not too hardcore ones) and stuff
2. We have a big mono solution with 158 projects (the projects are not that big themselves) and compiler speed is becoming more of a problem.

So we basically either have to start cutting our solution in more specific parts or try to contribute to compiler perf infrastructure. Personally I'd like to do the latter (but maybe former is a thing to consider anyway).

Is there a document describing a long term plan/vision of making such a thing an important part of every release (making sure there are perf improvements/controlled regressions e.t.c). If there is no such document, maybe there is a somewhat clearish goal of how to build this up or someone already tried to do this kind of thing and the work can be continued from there?

So, I would like to know what's the state of compiler perf infrastructure?

Do compiler stages have clear boundaries and can they be separately timed? I guess the scariest perf wise thing is type checking but which concrete part of it? So basically I would like to time stages and substages to know what's making me wait actually.

How can I start and what are my first steps here?

[KevinRansom]

/cc @vzarytovskii

[vzarytovskii]

There's no infrastructure per se yet. We do have some benchmarks in https://github.com/dotnet/fsharp/tree/main/tests/benchmarks, but they are more of a manual ones we use when we need.

We do want to have some tests in https://github.com/dotnet/performance

[safesparrow]

Is https://github.com/dotnet/performance for compiler [services] performance tests too or just for the runtime performance?

[vzarytovskii]

I guess we can use this infrastructure for both runtime as well as FCS benchmarks.

Those are not conventional BDN benchmarks though, but rather microbenchmarks.

[safesparrow]

I'm in the same boat as @En3Tho .
FCS performance (code analysis and type checking rather than compilation, but I imagine they have a lot in common) as used in Rider IDE is the major showstopper for us.
I spent some time recently trying to dive into the codebase in search of things I could improve.

However, I felt like I'm on my own:

• not much documentation on performance other than https://fsharp.github.io/fsharp-compiler-docs/memory-usage.html and https://fsharp.github.io/fsharp-compiler-docs/compiler-startup-performance.html
• no decision record I could find for trade-offs between idiomatic F# and performant code
• hard to replicate IDE use - would be nice to have some example inputs that VS/Rider/VSCode send to the FCS. The main one I played with was "What does Rider ask FCS when I open an .fs file in Fantomas?", which required handwritten JSON serialization of FSProjectOptions to stick in a test.
• I tried to use tests/benchmarks/CompilerServiceBenchamrks/benchmarks.ipynb. However this kept crashing due to the .fsx file failing to compile (not sure if this was intentional, but it renders the notebook useless)

I think that having the infrastructure that collects performance metrics for a set of predefined inputs would go a long way:

• it would encourage people like @En3Tho and myself to tinker with the codebase
• it would make it much easier to argue for/against certain code changes on PRs
• it would stop unintentional performance degradation from creeping in

If having this sort of infrastructure on all PRs is problematic due to lack of resources/the fact people work on their own forks, then this should suffice:

• automated benchmarks on main/branches on the main repo
• all benchmarks runnable on user machines via a single command (with options to run quick&long tests)
  This way motivated users can easily verify their assumptions and show proof of any improvements they think they made.

Some basic metrics that should be both easy to gather and useful:

• wallclock & total CPU time
• peak memory, total memory allocated (gen0/gen1/gen2), time spent in GC wait
• total I/O read/write

Example benchmarks:

• the already mentioned Rider query: a call to FSharpChecker.ParseAndCheckFileInProject(...) with one or more codebase examples, eg. a top file in the Fantomas codebase
• anything else (I haven't looked at the FCS API apart from the above)

[vzarytovskii]

Sorry for the short answer, I'm away from my laptop till Monday. Will provide a more detailed answer then.

Regarding the ipynb "benchmark" - it was more of an experiment which will need fixing.

We have a short guide on how to benchmark FCS/Fscore in our DEVGUIDE.md file in the repo root.

Performance (both building a "framework" for testing it as well as making improvements) is one of the upcoming work streams for the new team. I will publish more details next week.

[safesparrow]

I've seen the plan on #13296 which I guess what you were hinting at last week - looks great.

A question on this:

Setting up FSharp.Compiler.Service(FCS) and tooling performance infrastructure.

FCS and tooling performace optimizations and improvements based on the data.

Is there something community contributors can do to help?

An EndToEnd test I have locally is:

1. fetch https://github.com/fsprojects/fantomas/releases/tag/v5.0.0-alpha-008
2. open one of the files in Rider
3. let it call FSharpChecker.parseAndCheckFileInProject
4. capture & serialize the inputs as json
5. have a unit test in FCS that makes the same call. Time it/measure memory use/GC, generate textual report

I'm wondering if something along these lines would be useful as a Poor Dev's End2End test:

• check the inputs snapshot into the codebase
• make a script that runs steps 1., 4., 5.,

Obviously lots of things can be changed/improved:

• different ways to fetch the codebase
• using a synthetic/generated codebase rather than a real one
• better ways to capture & serialize the inputs
• looking at other parts of the API (This part of the API is important to me as that's the main API that Rider uses for code analysis.)

What do you think?

[vzarytovskii]

Is there something community contributors can do to help?

Yes, absolutely, it would be a huge help to have some meaningful FCS benchmarks.

I know @nojaf was tinkering with big projects. We also can have something like FSHarpPlus, FSAC or Fantomas as a "test" project for the FCS benchmarks.

Our DEVGUIDE has a section which describes how to create and run benchmarks using BDN .

We can start by adding meaningful benchmarks first and run them on-demand first, until we integrate into our CI.

capture & serialize the inputs as json

I am not entirely sure what inputs you mean? Is it Rider-specific inputs?

[auduchinok]

I am not entirely sure what inputs you mean? Is it Rider-specific inputs?

Sounds more like FSharpProjectOptions and the project files to me.

[safesparrow]

Yes, I mean serializing the calls to FSharpChecker, the main part of which is the FSharpOptions type.

Is BDN good for longer-running tests? Analysing a top-level file in Fantomas takes 12s ie. it's not a microbenchmark.

While lower-level benchmarks are definitely useful too, having a single catch-all/catch-most is handy.

Would I use BDN and just configure it with a minimal number of runs/warmups?