FEATURE: Introduce pp=async-flamegraph for asynchronous flamegraphs (…

…#494)

Using `?pp=async-flamegraph` causes the flamegraph data to be placed in long-term storage, and made available via a link in the mini_profiler UI. Flamegraph data will also be recorded and stored for all AJAX requests with `?pp=async-flamegraph` in the `Referer` header. This is useful in a few situations:

- You want to view flamegraphs for AJAX requests made by a Javascript application. By supplying `pp=async-flamegraph`, flamegraph links for every AJAX request will be made available in the mini-profiler UI.

- You want to see the HTML result of a request, and view the flamegraph later. The existing `?pp=flamegraph` option hides the true output.

- You are performing the request via a tool like `curl`, and would like to view the flamegraph later in the browser (you can extract the X-MiniProfiler-Ids header from the response, then view flamegraph in the browser)

---

When the `pp=async-flamegraph` parameter is supplied, a new "flamegraph" link is added to the UI. Clicking the link will take you to a URL like `/mini-profiler-resources/flamegraph?id=t0x70kt7hy3cmitx7adx`, which displays the flamegraph UI.

CHANGELOG.md

@@ -1,5 +1,9 @@
 # CHANGELOG
 
+## Unreleased
+
+- [FEATURE] Introduce `pp=async-flamegraph` for asynchronous flamegraphs
+
 ## 2.3.1 - 2021-01-29
 
 - [FIX] compatability with Ruby 3.0

Gemfile

@@ -5,7 +5,10 @@ ruby '>= 2.4.0'
 
 gemspec
 
-gem 'codecov', require: false, group: :test
+group :test do
+  gem 'codecov', require: false
+  gem 'stackprof', require: false
+end
 
 group :development do
   gem 'guard', platforms: [:mri_22, :mri_23]

README.md

@@ -171,6 +171,10 @@ To generate [flamegraphs](http://samsaffron.com/archive/2013/03/19/flame-graphs-
 * add the [**stackprof**](https://rubygems.org/gems/stackprof) gem to your Gemfile
 * visit a page in your app with `?pp=flamegraph`
 
+To store flamegraph data for later viewing, append the `?pp=async-flamegraph` parameter. The request will return as normal.
+Flamegraph data for this request, and all subsequent requests made by this page (based on the `REFERER` header) will be stored.
+'flamegraph' links will appear for these requests in the MiniProfiler UI.
+
 ### Memory Profiling
 
 Memory allocations can be measured (using the [memory_profiler](https://github.com/SamSaffron/memory_profiler) gem)

lib/html/includes.js

@@ -1213,6 +1213,9 @@ var _MiniProfiler = (function() {
     shareUrl: function shareUrl(id) {
       return options.path + "results?id=" + id;
     },
+    flamegraphUrl: function flamegrapgUrl(id) {
+      return options.path + "flamegraph?id=" + id;
+    },
     moreUrl: function moreUrl(requestName) {
       var requestParts = requestName.split(" ");
       var linkSrc =

lib/html/includes.tmpl

@@ -142,6 +142,9 @@
 <script id="linksTemplate" type="text/x-dot-tmpl">
   <a href="{{= MiniProfiler.shareUrl(it.page.id) }}" class="profiler-share-profiler-results" target="_blank">share</a>
   <a href="{{= MiniProfiler.moreUrl(it.timing.name) }}" class="profiler-more-actions">more</a>
+  {{? it.page.has_flamegraph}}
+    <a href="{{= MiniProfiler.flamegraphUrl(it.page.id) }}" class="profiler-show-flamegraph" target="_blank">flamegraph</a>
+  {{?}}
   {{? it.custom_link}}
     <a href="{{= it.custom_link }}" class="profiler-custom-link" target="_blank">{{= it.custom_link_name }}</a>
   {{?}}

lib/html/vendor.js

@@ -11,7 +11,7 @@ var out=' <div class="profiler-result"> <div class="profiler-button ';if(it.has_
 }
 MiniProfiler.templates["linksTemplate"] = function anonymous(it
 ) {
-var out=' <a href="'+( MiniProfiler.shareUrl(it.page.id) )+'" class="profiler-share-profiler-results" target="_blank">share</a> <a href="'+( MiniProfiler.moreUrl(it.timing.name) )+'" class="profiler-more-actions">more</a> ';if(it.custom_link){out+=' <a href="'+( it.custom_link )+'" class="profiler-custom-link" target="_blank">'+( it.custom_link_name )+'</a> ';}out+=' ';if(it.page.has_trivial_timings){out+=' <a class="profiler-toggle-trivial" data-show-on-load="'+( it.page.has_all_trivial_timings )+'" title="toggles any rows with &lt; '+( it.page.trivial_duration_threshold_milliseconds )+' ms"> show trivial </a> ';}return out;
+var out=' <a href="'+( MiniProfiler.shareUrl(it.page.id) )+'" class="profiler-share-profiler-results" target="_blank">share</a> <a href="'+( MiniProfiler.moreUrl(it.timing.name) )+'" class="profiler-more-actions">more</a> ';if(it.page.has_flamegraph){out+=' <a href="'+( MiniProfiler.flamegraphUrl(it.page.id) )+'" class="profiler-show-flamegraph" target="_blank">flamegraph</a> ';}out+=' ';if(it.custom_link){out+=' <a href="'+( it.custom_link )+'" class="profiler-custom-link" target="_blank">'+( it.custom_link_name )+'</a> ';}out+=' ';if(it.page.has_trivial_timings){out+=' <a class="profiler-toggle-trivial" data-show-on-load="'+( it.page.has_all_trivial_timings )+'" title="toggles any rows with &lt; '+( it.page.trivial_duration_threshold_milliseconds )+' ms"> show trivial </a> ';}return out;
 }
 MiniProfiler.templates["timingTemplate"] = function anonymous(it
 ) {

lib/mini_profiler/profiler.rb

@@ -182,6 +182,7 @@ def serve_html(env)
 
       return serve_results(env) if file_name.eql?('results')
       return handle_snapshots_request(env) if file_name.eql?('snapshots')
+      return serve_flamegraph(env) if file_name.eql?('flamegraph')
 
       resources_env = env.dup
       resources_env['PATH_INFO'] = file_name
@@ -345,11 +346,12 @@ def call(env)
         # Prevent response body from being compressed
         env['HTTP_ACCEPT_ENCODING'] = 'identity' if config.suppress_encoding
 
-        if query_string =~ /pp=flamegraph/
+        if query_string =~ /pp=(async-)?flamegraph/ || env['HTTP_REFERER'] =~ /pp=async-flamegraph/
           unless defined?(StackProf) && StackProf.respond_to?(:run)
-
-            flamegraph = "Please install the stackprof gem and require it: add gem 'stackprof' to your Gemfile"
-            status, headers, body = @app.call(env)
+            headers = { 'Content-Type' => 'text/html' }
+            message = "Please install the stackprof gem and require it: add gem 'stackprof' to your Gemfile"
+            body.close if body.respond_to? :close
+            return client_settings.handle_cookie([500, headers, message])
           else
             # do not sully our profile with mini profiler timings
             current.measure = false
@@ -429,9 +431,12 @@ def call(env)
       page_struct[:user] = user(env)
       page_struct[:root].record_time((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start) * 1000)
 
-      if flamegraph
+      if flamegraph && query_string =~ /pp=flamegraph/
         body.close if body.respond_to? :close
         return client_settings.handle_cookie(self.flamegraph(flamegraph, path))
+      elsif flamegraph # async-flamegraph
+        page_struct[:has_flamegraph] = true
+        page_struct[:flamegraph] = flamegraph
       end
 
       begin
@@ -651,6 +656,7 @@ def help(client_settings, env)
   #{make_link "profile-gc", env} : perform gc profiling on this request, analyzes ObjectSpace generated by request
   #{make_link "profile-memory", env} : requires the memory_profiler gem, new location based report
   #{make_link "flamegraph", env} : a graph representing sampled activity (requires the stackprof gem).
+  #{make_link "async-flamegraph", env} : store flamegraph data for this page and all its AJAX requests. Flamegraph links will be available in the mini-profiler UI (requires the stackprof gem).
   #{make_link "flamegraph&flamegraph_sample_rate=1", env}: creates a flamegraph with the specified sample rate (in ms). Overrides value set in config
   #{make_link "flamegraph_embed", env} : a graph representing sampled activity (requires the stackprof gem), embedded resources for use on an intranet.
   #{make_link "trace-exceptions", env} : will return all the spots where your application raises exceptions
@@ -665,35 +671,31 @@ def help(client_settings, env)
 
     def flamegraph(graph, path)
       headers = { 'Content-Type' => 'text/html' }
-      if Hash === graph
-        html = <<~HTML
-          <!DOCTYPE html>
-          <html>
-            <head>
-              <style>
-                body { margin: 0; height: 100vh; }
-                #speedscope-iframe { width: 100%; height: 100%; border: none; }
-              </style>
-            </head>
-            <body>
-              <script type="text/javascript">
-                var graph = #{JSON.generate(graph)};
-                var json = JSON.stringify(graph);
-                var blob = new Blob([json], { type: 'text/plain' });
-                var objUrl = encodeURIComponent(URL.createObjectURL(blob));
-                var iframe = document.createElement('IFRAME');
-                iframe.setAttribute('id', 'speedscope-iframe');
-                document.body.appendChild(iframe);
-                var iframeUrl = '#{@config.base_url_path}speedscope/index.html#profileURL=' + objUrl + '&title=' + 'Flamegraph for #{CGI.escape(path)}';
-                iframe.setAttribute('src', iframeUrl);
-              </script>
-            </body>
-          </html>
-        HTML
-        [200, headers, [html]]
-      else
-        [200, headers, [graph]]
-      end
+      html = <<~HTML
+        <!DOCTYPE html>
+        <html>
+          <head>
+            <style>
+              body { margin: 0; height: 100vh; }
+              #speedscope-iframe { width: 100%; height: 100%; border: none; }
+            </style>
+          </head>
+          <body>
+            <script type="text/javascript">
+              var graph = #{JSON.generate(graph)};
+              var json = JSON.stringify(graph);
+              var blob = new Blob([json], { type: 'text/plain' });
+              var objUrl = encodeURIComponent(URL.createObjectURL(blob));
+              var iframe = document.createElement('IFRAME');
+              iframe.setAttribute('id', 'speedscope-iframe');
+              document.body.appendChild(iframe);
+              var iframeUrl = '#{@config.base_url_path}speedscope/index.html#profileURL=' + objUrl + '&title=' + 'Flamegraph for #{CGI.escape(path)}';
+              iframe.setAttribute('src', iframeUrl);
+            </script>
+          </body>
+        </html>
+      HTML
+      [200, headers, [html]]
     end
 
     def ids(env)
@@ -824,6 +826,24 @@ def handle_snapshots_request(env)
       response.finish
     end
 
+    def serve_flamegraph(env)
+      request     = Rack::Request.new(env)
+      id          = request.params['id']
+      page_struct = @storage.load(id)
+
+      if !page_struct
+        id        = ERB::Util.html_escape(id)
+        user_info = ERB::Util.html_escape(user(env))
+        return [404, {}, ["Request not found: #{id} - user #{user_info}"]]
+      end
+
+      if !page_struct[:flamegraph]
+        return [404, {}, ["No flamegraph available for #{ERB::Util.html_escape(id)}"]]
+      end
+
+      self.flamegraph(page_struct[:flamegraph], page_struct[:request_path])
+    end
+
     def rails_route_from_path(path, method)
       if defined?(Rails) && defined?(ActionController::RoutingError)
         hash = Rails.application.routes.recognize_path(path, method: method)

lib/mini_profiler/timer_struct/page.rb

@@ -87,7 +87,9 @@ def initialize(env)
             executed_non_queries: 0,
             custom_timing_names: [],
             custom_timing_stats: {},
-            custom_fields: {}
+            custom_fields: {},
+            has_flamegraph: false,
+            flamegraph: nil
           )
           self[:request_method] = env['REQUEST_METHOD']
           self[:request_path] = env['PATH_INFO']
@@ -111,12 +113,16 @@ def root
           @attributes[:root]
         end
 
+        def attributes_to_serialize
+          @attributes.keys - [:flamegraph]
+        end
+
         def to_json(*a)
-          ::JSON.generate(@attributes.merge(self.extra_json))
+          ::JSON.generate(@attributes.slice(*attributes_to_serialize).merge(extra_json))
         end
 
         def as_json(options = nil)
-          super(options).merge!(extra_json)
+          super(options).slice(*attributes_to_serialize.map(&:to_s)).merge!(extra_json)
         end
 
         def extra_json

spec/integration/middleware_spec.rb

@@ -74,6 +74,16 @@ def app
         )
       end
     end
+
+    describe 'with async-flamegraph query' do
+      it 'should return stackprof error message' do
+        Rack::MiniProfiler.config.enable_advanced_debugging_tools = true
+        do_get(pp: 'async-flamegraph')
+        expect(last_response.body).to eq(
+          'Please install the stackprof gem and require it: add gem \'stackprof\' to your Gemfile'
+        )
+      end
+    end
   end
 
   describe 'with Rack::MiniProfiler before Rack::Deflater' do

spec/integration/mini_profiler_spec.rb

@@ -141,6 +141,38 @@ def app
     end
   end
 
+  it 'works with async-flamegraph' do
+    pid = fork do # Avoid polluting main process with stackprof
+      require 'stackprof'
+
+      # Should store flamegraph for ?pp=async-flamegraph
+      get '/html?pp=async-flamegraph'
+      expect(last_response).to be_ok
+      id = last_response.headers['X-MiniProfiler-Ids'].split(",")[0]
+      get "/mini-profiler-resources/flamegraph?id=#{id}"
+      expect(last_response).to be_ok
+      expect(last_response.body).to include("var graph = {")
+
+      # Should store flamegraph based on REFERER
+      get '/html', nil, { "HTTP_REFERER" => "/origin?pp=async-flamegraph" }
+      expect(last_response).to be_ok
+      id = last_response.headers['X-MiniProfiler-Ids'].split(",")[0]
+      get "/mini-profiler-resources/flamegraph?id=#{id}"
+      expect(last_response).to be_ok
+      expect(last_response.body).to include("var graph = {")
+
+      # Should not store/return flamegraph for regular requests
+      get '/html'
+      expect(last_response).to be_ok
+      id = last_response.headers['X-MiniProfiler-Ids'].split(",")[0]
+      get "/mini-profiler-resources/flamegraph?id=#{id}"
+      expect(last_response.status).to eq(404)
+    end
+
+    Process.wait(pid)
+    expect($?.exitstatus).to eq(0)
+  end
+
   describe 'with an implicit body tag' do
 
     before do

spec/lib/timer_struct/page_timer_struct_spec.rb

@@ -41,4 +41,19 @@
       expect(page.to_json).to eq(from_json_page.to_json)
     end
   end
+
+  describe '.to_json' do
+    it 'does not include the flamegraph itself' do
+      page = described_class.new({
+        'REQUEST_METHOD' => 'POST',
+        'PATH_INFO' => '/some/path',
+        'SERVER_NAME' => 'server001',
+      })
+      page[:has_flamegraph] = true
+      page[:flamegraph] = { fake: "data" }
+      result = JSON.parse(page.to_json)
+      expect(result["flamegraph"]).to eq(nil)
+      expect(result["has_flamegraph"]).to eq(true)
+    end
+  end
 end