Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Conductor #254

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Add Conductor #254

wants to merge 2 commits into from

Conversation

YassinEldeeb
Copy link
Member

No description provided.

Copy link

github-actions bot commented Nov 6, 2023

Overview for: federation-v1/constant-vus-over-time

This scenario runs 4 subgraphs and a GraphQL gateway with Federation v1 spec, and runs a heavy query. It's being executed with a constant amount of VUs over a fixed amount of time. It measure things like memory usage, CPU usage, average RPS. It also includes a summary of the entire execution, and metrics information about HTTP execution times.

This scenario was running 100 VUs over 30s

Comparison

Comparison

Gateway RPS ⬇️ Requests Duration Notes
conductor 551 16620 total, 0 failed avg: 180ms, p95: 237ms ❌ non-compatible response structure (16620)
cosmo 172 5265 total, 0 failed avg: 508ms, p95: 680ms
apollo-router 162 4960 total, 0 failed avg: 558ms, p95: 751ms
wundergraph 152 4615 total, 0 failed avg: 566ms, p95: 1022ms
mesh-supergraph-bun 93 2889 total, 0 failed avg: 1033ms, p95: 1772ms
apollo-server 49 1555 total, 0 failed avg: 1972ms, p95: 2611ms
mercurius 31 961 total, 0 failed avg: 3149ms, p95: 4773ms
mesh-supergraph 30 986 total, 0 failed avg: 3141ms, p95: 5404ms
Summary for: `conductor`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✗ valid response structure
      ↳  0% — ✓ 0 / ✗ 16620

     checks.........................: 66.66% ✓ 33240      ✗ 16620
     data_received..................: 7.6 MB 253 kB/s
     data_sent......................: 20 MB  655 kB/s
     http_req_blocked...............: avg=10.94µs  min=1.18µs  med=2.38µs   max=6.61ms   p(90)=3.41µs   p(95)=4.25µs  
     http_req_connecting............: avg=5.56µs   min=0s      med=0s       max=3.6ms    p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=179.57ms min=18.77ms med=173.22ms max=395.71ms p(90)=218.74ms p(95)=236.89ms
       { expected_response:true }...: avg=179.57ms min=18.77ms med=173.22ms max=395.71ms p(90)=218.74ms p(95)=236.89ms
     http_req_failed................: 0.00%  ✓ 0          ✗ 16620
     http_req_receiving.............: avg=514.92µs min=14.85µs med=32.24µs  max=90.9ms   p(90)=99.72µs  p(95)=278.69µs
     http_req_sending...............: avg=87.73µs  min=6.95µs  med=13.22µs  max=80.98ms  p(90)=35.74µs  p(95)=117.86µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=178.97ms min=18.3ms  med=172.83ms max=395.62ms p(90)=217.39ms p(95)=235.74ms
     http_reqs......................: 16620  551.732793/s
     iteration_duration.............: avg=180.93ms min=20.06ms med=174.4ms  max=397.38ms p(90)=220.49ms p(95)=238.87ms
     iterations.....................: 16620  551.732793/s
     vus............................: 100    min=100      max=100
     vus_max........................: 100    min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `cosmo`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 15795      ✗ 0    
     data_received..................: 462 MB  15 MB/s
     data_sent......................: 6.2 MB  205 kB/s
     http_req_blocked...............: avg=24.6µs   min=1.3µs   med=2.89µs   max=3.76ms   p(90)=4.72µs   p(95)=5.68µs  
     http_req_connecting............: avg=17.93µs  min=0s      med=0s       max=3.7ms    p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=507.67ms min=22.48ms med=504.19ms max=1.34s    p(90)=623.42ms p(95)=679.86ms
       { expected_response:true }...: avg=507.67ms min=22.48ms med=504.19ms max=1.34s    p(90)=623.42ms p(95)=679.86ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 5265 
     http_req_receiving.............: avg=22.65ms  min=37.07µs med=74.66µs  max=695.17ms p(90)=62.41ms  p(95)=171.27ms
     http_req_sending...............: avg=1.59ms   min=7.88µs  med=13.74µs  max=554.1ms  p(90)=33.69µs  p(95)=211.45µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=483.41ms min=17.98ms med=490.35ms max=825.69ms p(90)=582.76ms p(95)=614.36ms
     http_reqs......................: 5265    172.830709/s
     iteration_duration.............: avg=575.7ms  min=49.16ms med=548.03ms max=1.79s    p(90)=766.43ms p(95)=882.15ms
     iterations.....................: 5265    172.830709/s
     vus............................: 100     min=100      max=100
     vus_max........................: 100     min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-router`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 14880      ✗ 0    
     data_received..................: 435 MB  14 MB/s
     data_sent......................: 5.9 MB  193 kB/s
     http_req_blocked...............: avg=42.92µs  min=1.42µs  med=3.32µs   max=4.41ms   p(90)=5.34µs   p(95)=6.32µs  
     http_req_connecting............: avg=37.3µs   min=0s      med=0s       max=4.37ms   p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=557.85ms min=15.14ms med=553.07ms max=1.68s    p(90)=699.72ms p(95)=750.85ms
       { expected_response:true }...: avg=557.85ms min=15.14ms med=553.07ms max=1.68s    p(90)=699.72ms p(95)=750.85ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 4960 
     http_req_receiving.............: avg=13.93ms  min=35.91µs med=74.61µs  max=1.24s    p(90)=22.14ms  p(95)=71.33ms 
     http_req_sending...............: avg=916.75µs min=7.87µs  med=15.6µs   max=370.95ms p(90)=38.66µs  p(95)=158.04µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=542.99ms min=15.05ms med=544.49ms max=913.23ms p(90)=679.2ms  p(95)=718.19ms
     http_reqs......................: 4960    162.817241/s
     iteration_duration.............: avg=610.21ms min=25.84ms med=590.56ms max=2.09s    p(90)=768.95ms p(95)=837.34ms
     iterations.....................: 4960    162.817241/s
     vus............................: 100     min=100      max=100
     vus_max........................: 100     min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `wundergraph`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 13845      ✗ 0    
     data_received..................: 405 MB  13 MB/s
     data_sent......................: 5.5 MB  181 kB/s
     http_req_blocked...............: avg=71.41µs  min=1.49µs  med=3.14µs   max=114.1ms p(90)=5.15µs   p(95)=6.27µs  
     http_req_connecting............: avg=22.15µs  min=0s      med=0s       max=8.11ms  p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=566.1ms  min=12.02ms med=544.82ms max=2.47s   p(90)=885.77ms p(95)=1.02s   
       { expected_response:true }...: avg=566.1ms  min=12.02ms med=544.82ms max=2.47s   p(90)=885.77ms p(95)=1.02s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 4615 
     http_req_receiving.............: avg=52.42ms  min=36.87µs med=88.4µs   max=2.22s   p(90)=128.92ms p(95)=333.53ms
     http_req_sending...............: avg=2.65ms   min=7.67µs  med=15.17µs  max=1.37s   p(90)=36.28µs  p(95)=221.62µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=511.03ms min=11.66ms med=498.51ms max=1.57s   p(90)=804.52ms p(95)=905.28ms
     http_reqs......................: 4615    152.106831/s
     iteration_duration.............: avg=654.39ms min=38.46ms med=601.86ms max=3.64s   p(90)=1.03s    p(95)=1.26s   
     iterations.....................: 4615    152.106831/s
     vus............................: 100     min=100      max=100
     vus_max........................: 100     min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mesh-supergraph-bun`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 8667      ✗ 0    
     data_received..................: 253 MB  8.2 MB/s
     data_sent......................: 3.4 MB  111 kB/s
     http_req_blocked...............: avg=58.05µs  min=1.48µs   med=3.53µs   max=5.69ms   p(90)=5.64µs  p(95)=9.37µs  
     http_req_connecting............: avg=44.86µs  min=0s       med=0s       max=4.66ms   p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=1.03s    min=456.67ms med=907.47ms max=2.44s    p(90)=1.67s   p(95)=1.77s   
       { expected_response:true }...: avg=1.03s    min=456.67ms med=907.47ms max=2.44s    p(90)=1.67s   p(95)=1.77s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 2889 
     http_req_receiving.............: avg=11.14ms  min=37.93µs  med=101.9µs  max=676.83ms p(90)=1.21ms  p(95)=45.02ms 
     http_req_sending...............: avg=427.51µs min=8.39µs   med=17.74µs  max=137.13ms p(90)=38.64µs p(95)=213.14µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=1.02s    min=456.45ms med=896.75ms max=2.44s    p(90)=1.66s   p(95)=1.76s   
     http_reqs......................: 2889    93.325955/s
     iteration_duration.............: avg=1.05s    min=469.4ms  med=928.39ms max=2.47s    p(90)=1.69s   p(95)=1.8s    
     iterations.....................: 2889    93.325955/s
     vus............................: 100     min=100     max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-server`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 4665      ✗ 0    
     data_received..................: 137 MB  4.4 MB/s
     data_sent......................: 1.8 MB  59 kB/s
     http_req_blocked...............: avg=59.77µs  min=1.38µs   med=3µs      max=7.81ms  p(90)=4.97µs   p(95)=177.27µs
     http_req_connecting............: avg=47.4µs   min=0s       med=0s       max=2.23ms  p(90)=0s       p(95)=128.85µs
     http_req_duration..............: avg=1.97s    min=440.55ms med=1.65s    max=21.12s  p(90)=2.01s    p(95)=2.61s   
       { expected_response:true }...: avg=1.97s    min=440.55ms med=1.65s    max=21.12s  p(90)=2.01s    p(95)=2.61s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 1555 
     http_req_receiving.............: avg=198.04µs min=44.73µs  med=123.81µs max=70.63ms p(90)=181.85µs p(95)=207.09µs
     http_req_sending...............: avg=38.07µs  min=8.33µs   med=15.89µs  max=2.49ms  p(90)=30.79µs  p(95)=74.8µs  
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=1.97s    min=440.42ms med=1.65s    max=21.12s  p(90)=2.01s    p(95)=2.61s   
     http_reqs......................: 1555    49.588654/s
     iteration_duration.............: avg=1.98s    min=450.11ms med=1.66s    max=21.13s  p(90)=2.02s    p(95)=2.61s   
     iterations.....................: 1555    49.588654/s
     vus............................: 38      min=38      max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mercurius`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 2883      ✗ 0    
     data_received..................: 84 MB   2.8 MB/s
     data_sent......................: 1.1 MB  37 kB/s
     http_req_blocked...............: avg=32.26µs min=1.56µs   med=3.62µs   max=4.41ms   p(90)=94.27µs  p(95)=173.57µs
     http_req_connecting............: avg=19.76µs min=0s       med=0s       max=842.44µs p(90)=63.44µs  p(95)=143.34µs
     http_req_duration..............: avg=3.14s   min=456.4ms  med=2.94s    max=7.95s    p(90)=4.11s    p(95)=4.77s   
       { expected_response:true }...: avg=3.14s   min=456.4ms  med=2.94s    max=7.95s    p(90)=4.11s    p(95)=4.77s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 961  
     http_req_receiving.............: avg=284.5µs min=37.83µs  med=118.97µs max=52.69ms  p(90)=186.06µs p(95)=257.42µs
     http_req_sending...............: avg=30.71µs min=8.13µs   med=19.42µs  max=2.15ms   p(90)=36.2µs   p(95)=68.51µs 
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=3.14s   min=455.96ms med=2.94s    max=7.95s    p(90)=4.11s    p(95)=4.77s   
     http_reqs......................: 961     31.390724/s
     iteration_duration.............: avg=3.15s   min=467.34ms med=2.95s    max=7.95s    p(90)=4.12s    p(95)=4.78s   
     iterations.....................: 961     31.390724/s
     vus............................: 100     min=100     max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mesh-supergraph`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 2958      ✗ 0    
     data_received..................: 87 MB   2.7 MB/s
     data_sent......................: 1.2 MB  37 kB/s
     http_req_blocked...............: avg=289.69µs min=1.64µs  med=4.02µs  max=6.71ms   p(90)=147µs    p(95)=2.45ms  
     http_req_connecting............: avg=252.47µs min=0s      med=0s      max=6.45ms   p(90)=116.5µs  p(95)=2.23ms  
     http_req_duration..............: avg=3.14s    min=1.47s   med=2.69s   max=7.57s    p(90)=4.88s    p(95)=5.4s    
       { expected_response:true }...: avg=3.14s    min=1.47s   med=2.69s   max=7.57s    p(90)=4.88s    p(95)=5.4s    
     http_req_failed................: 0.00%   ✓ 0         ✗ 986  
     http_req_receiving.............: avg=4.62ms   min=43.15µs med=155.9µs max=581.03ms p(90)=2.54ms   p(95)=14.91ms 
     http_req_sending...............: avg=56.16µs  min=8.66µs  med=22.22µs max=2.09ms   p(90)=101.54µs p(95)=263.82µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=3.13s    min=1.47s   med=2.69s   max=7.57s    p(90)=4.87s    p(95)=5.4s    
     http_reqs......................: 986     30.962615/s
     iteration_duration.............: avg=3.15s    min=1.48s   med=2.71s   max=7.59s    p(90)=4.89s    p(95)=5.41s   
     iterations.....................: 986     30.962615/s
     vus............................: 61      min=61      max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview

Copy link

github-actions bot commented Nov 6, 2023

Overview for: federation-v1/constant-vus-subgraphs-delay

This scenario runs 4 subgraphs and a GraphQL gateway with Federation v1 spec, and runs a heavy query. It's being executed with a constant amount of VUs over a fixed amount of time. It measure things like memory usage, CPU usage, average RPS. It also includes a summary of the entire execution, and metrics information about HTTP execution times.

This scenario was running 100 VUs over 30s

Comparison

Comparison

Gateway RPS ⬇️ Requests Duration Notes
apollo-router 145 4433 total, 0 failed avg: 636ms, p95: 843ms
conductor 145 4481 total, 0 failed avg: 677ms, p95: 817ms ❌ non-compatible response structure (4481)
wundergraph 99 3069 total, 0 failed avg: 951ms, p95: 1192ms
mesh-supergraph-bun 79 2444 total, 0 failed avg: 1216ms, p95: 1830ms
cosmo 76 2380 total, 0 failed avg: 1238ms, p95: 1533ms
apollo-server 47 1497 total, 0 failed avg: 2048ms, p95: 2374ms
mesh-supergraph 30 967 total, 0 failed avg: 3198ms, p95: 5345ms
mercurius 11 448 total, 0 failed avg: 7628ms, p95: 9123ms
Summary for: `apollo-router`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 13299      ✗ 0    
     data_received..................: 389 MB  13 MB/s
     data_sent......................: 5.3 MB  172 kB/s
     http_req_blocked...............: avg=40.91µs  min=1.47µs   med=3.09µs   max=50.43ms  p(90)=4.73µs   p(95)=6.15µs  
     http_req_connecting............: avg=19.33µs  min=0s       med=0s       max=4.64ms   p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=635.84ms min=335ms    med=627.72ms max=1.08s    p(90)=789.76ms p(95)=843.28ms
       { expected_response:true }...: avg=635.84ms min=335ms    med=627.72ms max=1.08s    p(90)=789.76ms p(95)=843.28ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 4433 
     http_req_receiving.............: avg=10.17ms  min=34.4µs   med=68.44µs  max=308.59ms p(90)=26.12ms  p(95)=74.65ms 
     http_req_sending...............: avg=1.18ms   min=7.8µs    med=14.3µs   max=266.11ms p(90)=40.82µs  p(95)=224.33µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=624.48ms min=334.87ms med=620.86ms max=1.08s    p(90)=762.49ms p(95)=813.27ms
     http_reqs......................: 4433    145.060883/s
     iteration_duration.............: avg=683.24ms min=343.44ms med=668.01ms max=1.31s    p(90)=859.72ms p(95)=920.84ms
     iterations.....................: 4433    145.060883/s
     vus............................: 100     min=100      max=100
     vus_max........................: 100     min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `conductor`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✗ valid response structure
      ↳  0% — ✓ 0 / ✗ 4481

     checks.........................: 66.66% ✓ 8962       ✗ 4481 
     data_received..................: 2.1 MB 67 kB/s
     data_sent......................: 5.3 MB 173 kB/s
     http_req_blocked...............: avg=17.63µs  min=1.31µs   med=2.06µs   max=5.25ms   p(90)=3.08µs   p(95)=3.78µs  
     http_req_connecting............: avg=13.7µs   min=0s       med=0s       max=5.23ms   p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=676.7ms  min=405.26ms med=674.9ms  max=959.74ms p(90)=787.44ms p(95)=817.47ms
       { expected_response:true }...: avg=676.7ms  min=405.26ms med=674.9ms  max=959.74ms p(90)=787.44ms p(95)=817.47ms
     http_req_failed................: 0.00%  ✓ 0          ✗ 4481 
     http_req_receiving.............: avg=44.61µs  min=19.25µs  med=38.95µs  max=1.22ms   p(90)=62.63µs  p(95)=72.84µs 
     http_req_sending...............: avg=21.12µs  min=8µs      med=12.67µs  max=3.38ms   p(90)=19.99µs  p(95)=26.85µs 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=676.64ms min=405.21ms med=674.81ms max=959.7ms  p(90)=787.39ms p(95)=817.28ms
     http_reqs......................: 4481   145.480801/s
     iteration_duration.............: avg=677.34ms min=405.97ms med=675.76ms max=960.19ms p(90)=788.09ms p(95)=818.13ms
     iterations.....................: 4481   145.480801/s
     vus............................: 100    min=100      max=100
     vus_max........................: 100    min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `wundergraph`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 9207      ✗ 0    
     data_received..................: 269 MB  8.7 MB/s
     data_sent......................: 3.6 MB  118 kB/s
     http_req_blocked...............: avg=95.89µs  min=1.37µs   med=2.66µs   max=10.8ms   p(90)=4µs     p(95)=8.73µs 
     http_req_connecting............: avg=85.5µs   min=0s       med=0s       max=10.76ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=950.5ms  min=583.51ms med=940.23ms max=1.51s    p(90)=1.11s   p(95)=1.19s  
       { expected_response:true }...: avg=950.5ms  min=583.51ms med=940.23ms max=1.51s    p(90)=1.11s   p(95)=1.19s  
     http_req_failed................: 0.00%   ✓ 0         ✗ 3069 
     http_req_receiving.............: avg=10.82ms  min=35.38µs  med=77.03µs  max=261.17ms p(90)=44.11ms p(95)=74.61ms
     http_req_sending...............: avg=1.57ms   min=7.5µs    med=13.17µs  max=197.37ms p(90)=41.49µs p(95)=1.39ms 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=938.1ms  min=583.29ms med=926.21ms max=1.42s    p(90)=1.09s   p(95)=1.17s  
     http_reqs......................: 3069    99.381185/s
     iteration_duration.............: avg=991.17ms min=593.25ms med=980.23ms max=1.52s    p(90)=1.16s   p(95)=1.24s  
     iterations.....................: 3069    99.381185/s
     vus............................: 100     min=100     max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mesh-supergraph-bun`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 7332      ✗ 0    
     data_received..................: 214 MB  7.0 MB/s
     data_sent......................: 2.9 MB  94 kB/s
     http_req_blocked...............: avg=128.98µs min=1.37µs   med=3.77µs   max=6.92ms   p(90)=5.83µs  p(95)=12.54µs 
     http_req_connecting............: avg=115.76µs min=0s       med=0s       max=6.89ms   p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=1.21s    min=539.93ms med=1.16s    max=2.67s    p(90)=1.6s    p(95)=1.82s   
       { expected_response:true }...: avg=1.21s    min=539.93ms med=1.16s    max=2.67s    p(90)=1.6s    p(95)=1.82s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 2444 
     http_req_receiving.............: avg=17.05ms  min=38.82µs  med=109.88µs max=731.83ms p(90)=2.47ms  p(95)=71.26ms 
     http_req_sending...............: avg=397.78µs min=8.84µs   med=18.89µs  max=139.44ms p(90)=47.84µs p(95)=284.04µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=1.19s    min=539.74ms med=1.15s    max=2.67s    p(90)=1.54s   p(95)=1.8s    
     http_reqs......................: 2444    79.256788/s
     iteration_duration.............: avg=1.24s    min=547.76ms med=1.18s    max=2.67s    p(90)=1.62s   p(95)=1.86s   
     iterations.....................: 2444    79.256788/s
     vus............................: 100     min=100     max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `cosmo`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 7140     ✗ 0    
     data_received..................: 209 MB  6.7 MB/s
     data_sent......................: 2.8 MB  91 kB/s
     http_req_blocked...............: avg=56.43µs min=1.35µs   med=2.73µs  max=5.41ms  p(90)=4.49µs  p(95)=10.07µs
     http_req_connecting............: avg=49.35µs min=0s       med=0s      max=5.37ms  p(90)=0s      p(95)=0s     
     http_req_duration..............: avg=1.23s   min=719.93ms med=1.23s   max=1.91s   p(90)=1.42s   p(95)=1.53s  
       { expected_response:true }...: avg=1.23s   min=719.93ms med=1.23s   max=1.91s   p(90)=1.42s   p(95)=1.53s  
     http_req_failed................: 0.00%   ✓ 0        ✗ 2380 
     http_req_receiving.............: avg=16.7ms  min=37.33µs  med=88.85µs max=393.9ms p(90)=61.4ms  p(95)=99.83ms
     http_req_sending...............: avg=1.53ms  min=8.09µs   med=13.15µs max=153.9ms p(90)=79.62µs p(95)=1.23ms 
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s      p(90)=0s      p(95)=0s     
     http_req_waiting...............: avg=1.21s   min=718.04ms med=1.22s   max=1.63s   p(90)=1.39s   p(95)=1.48s  
     http_reqs......................: 2380    76.62542/s
     iteration_duration.............: avg=1.27s   min=734.92ms med=1.27s   max=1.93s   p(90)=1.48s   p(95)=1.58s  
     iterations.....................: 2380    76.62542/s
     vus............................: 18      min=18     max=100
     vus_max........................: 100     min=100    max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-server`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 4491      ✗ 0    
     data_received..................: 132 MB  4.2 MB/s
     data_sent......................: 1.8 MB  57 kB/s
     http_req_blocked...............: avg=110.08µs min=1.48µs   med=3.38µs   max=7.2ms   p(90)=5.27µs   p(95)=234.56µs
     http_req_connecting............: avg=76.77µs  min=0s       med=0s       max=4.64ms  p(90)=0s       p(95)=163.44µs
     http_req_duration..............: avg=2.04s    min=962.06ms med=1.84s    max=14.05s  p(90)=2.27s    p(95)=2.37s   
       { expected_response:true }...: avg=2.04s    min=962.06ms med=1.84s    max=14.05s  p(90)=2.27s    p(95)=2.37s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 1497 
     http_req_receiving.............: avg=146.39µs min=43.65µs  med=119.32µs max=12.85ms p(90)=179.07µs p(95)=219.58µs
     http_req_sending...............: avg=58.56µs  min=8.84µs   med=17.63µs  max=1.55ms  p(90)=34.59µs  p(95)=128.96µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=2.04s    min=961.87ms med=1.84s    max=14.05s  p(90)=2.27s    p(95)=2.37s   
     http_reqs......................: 1497    47.728614/s
     iteration_duration.............: avg=2.05s    min=969.09ms med=1.85s    max=14.06s  p(90)=2.28s    p(95)=2.38s   
     iterations.....................: 1497    47.728614/s
     vus............................: 47      min=47      max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mesh-supergraph`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 2901      ✗ 0    
     data_received..................: 85 MB   2.7 MB/s
     data_sent......................: 1.1 MB  36 kB/s
     http_req_blocked...............: avg=196.2µs min=1.31µs  med=3.97µs  max=6.59ms  p(90)=95.92µs p(95)=1.79ms  
     http_req_connecting............: avg=180µs   min=0s      med=0s      max=5.55ms  p(90)=67.28µs p(95)=1.73ms  
     http_req_duration..............: avg=3.19s   min=1.37s   med=3.05s   max=6.73s   p(90)=4.36s   p(95)=5.34s   
       { expected_response:true }...: avg=3.19s   min=1.37s   med=3.05s   max=6.73s   p(90)=4.36s   p(95)=5.34s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 967  
     http_req_receiving.............: avg=3.29ms  min=36.81µs med=195.9µs max=80.47ms p(90)=2.13ms  p(95)=12.63ms 
     http_req_sending...............: avg=69.23µs min=8.18µs  med=21.82µs max=4.27ms  p(90)=61.35µs p(95)=164.06µs
     http_req_tls_handshaking.......: avg=0s      min=0s      med=0s      max=0s      p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=3.19s   min=1.31s   med=3.05s   max=6.73s   p(90)=4.36s   p(95)=5.34s   
     http_reqs......................: 967     30.349531/s
     iteration_duration.............: avg=3.21s   min=1.38s   med=3.06s   max=6.74s   p(90)=4.37s   p(95)=5.35s   
     iterations.....................: 967     30.349531/s
     vus............................: 59      min=59      max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mercurius`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 1344     ✗ 0    
     data_received..................: 39 MB   1.0 MB/s
     data_sent......................: 532 kB  14 kB/s
     http_req_blocked...............: avg=227.59µs min=1.49µs  med=3.78µs   max=5.29ms   p(90)=853.64µs p(95)=1.82ms  
     http_req_connecting............: avg=196.36µs min=0s      med=0s       max=5.19ms   p(90)=636.95µs p(95)=1.58ms  
     http_req_duration..............: avg=7.62s    min=3.53s   med=7.85s    max=9.44s    p(90)=8.91s    p(95)=9.12s   
       { expected_response:true }...: avg=7.62s    min=3.53s   med=7.85s    max=9.44s    p(90)=8.91s    p(95)=9.12s   
     http_req_failed................: 0.00%   ✓ 0        ✗ 448  
     http_req_receiving.............: avg=135.5µs  min=51.87µs med=118.75µs max=855.14µs p(90)=177.76µs p(95)=234.74µs
     http_req_sending...............: avg=242µs    min=8.58µs  med=21.02µs  max=3.24ms   p(90)=544.02µs p(95)=3.09ms  
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=7.62s    min=3.53s   med=7.85s    max=9.44s    p(90)=8.91s    p(95)=9.12s   
     http_reqs......................: 448     11.84989/s
     iteration_duration.............: avg=7.63s    min=3.53s   med=7.86s    max=9.45s    p(90)=8.92s    p(95)=9.13s   
     iterations.....................: 448     11.84989/s
     vus............................: 13      min=13     max=100
     vus_max........................: 100     min=100    max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview

Copy link

github-actions bot commented Nov 8, 2023

Overview for: federation-v1/ramping-vus

This scenario runs 4 subgraphs and a GraphQL gateway with Federation v1 spec, and runs a heavy query. We are running a heavy load of concurrent VUs to measure response time and other stats, during stress. It measure things like memory usage, CPU usage, response times. It also includes a summary of the entire execution, and metrics information about HTTP execution times.

This scenario was trying to reach 500 concurrent VUs over 60s

Comparison

Comparison

Gateway duration(p95)⬇️ RPS Requests Durations Notes
conductor 1116ms 542 38003 total, 0 failed avg: 475ms, p95: 1116ms, max: 2700ms, med: 411ms ❌ non-compatible response structure (38003)
cosmo 2328ms 173 12139 total, 0 failed avg: 777ms, p95: 2328ms, max: 5244ms, med: 509ms ❌ 5 unexpected GraphQL errors, non-compatible response structure (5)
apollo-router 2396ms 172 12101 total, 0 failed avg: 854ms, p95: 2396ms, max: 7384ms, med: 629ms
wundergraph 2633ms 164 11488 total, 0 failed avg: 823ms, p95: 2633ms, max: 8480ms, med: 488ms
mesh-supergraph-bun 5538ms 115 8109 total, 0 failed avg: 2273ms, p95: 5539ms, max: 8651ms, med: 2070ms
mesh-supergraph 6559ms 72 5055 total, 0 failed avg: 3781ms, p95: 6560ms, max: 8727ms, med: 3833ms
apollo-server 24017ms 68 4985 total, 0 failed avg: 4190ms, p95: 24018ms, max: 45238ms, med: 2243ms
Summary for: `conductor`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✗ valid response structure
      ↳  0% — ✓ 0 / ✗ 38003

     checks.........................: 66.66% ✓ 76006      ✗ 38003
     data_received..................: 17 MB  249 kB/s
     data_sent......................: 45 MB  644 kB/s
     http_req_blocked...............: avg=21.26µs  min=942ns   med=2.35µs   max=44.87ms p(90)=3.7µs    p(95)=5.37µs  
     http_req_connecting............: avg=13.97µs  min=0s      med=0s       max=44.78ms p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=475.22ms min=3.45ms  med=411.15ms max=2.7s    p(90)=928.69ms p(95)=1.11s   
       { expected_response:true }...: avg=475.22ms min=3.45ms  med=411.15ms max=2.7s    p(90)=928.69ms p(95)=1.11s   
     http_req_failed................: 0.00%  ✓ 0          ✗ 38003
     http_req_receiving.............: avg=92.05µs  min=15.26µs med=32.87µs  max=59.53ms p(90)=81.76µs  p(95)=185.37µs
     http_req_sending...............: avg=61.33µs  min=7.27µs  med=13.85µs  max=49.44ms p(90)=38.45µs  p(95)=100.62µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=475.07ms min=3.41ms  med=410.94ms max=2.7s    p(90)=928.01ms p(95)=1.11s   
     http_reqs......................: 38003  542.867821/s
     iteration_duration.............: avg=476.57ms min=3.89ms  med=412.42ms max=2.7s    p(90)=930.52ms p(95)=1.11s   
     iterations.....................: 38003  542.867821/s
     vus............................: 1      min=1        max=499
     vus_max........................: 500    min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `cosmo`

K6 Output

     ✓ response code was 200
     ✗ no graphql errors
      ↳  99% — ✓ 12134 / ✗ 5
     ✗ valid response structure
      ↳  99% — ✓ 12134 / ✗ 5

     checks.........................: 99.97% ✓ 36407      ✗ 10   
     data_received..................: 1.1 GB 15 MB/s
     data_sent......................: 14 MB  206 kB/s
     http_req_blocked...............: avg=17.19ms  min=1.42µs  med=3.02µs   max=3.4s   p(90)=4.83µs   p(95)=11.94µs
     http_req_connecting............: avg=16.32ms  min=0s      med=0s       max=3.4s   p(90)=0s       p(95)=0s     
     http_req_duration..............: avg=777.29ms min=6.44ms  med=509.16ms max=5.24s  p(90)=1.85s    p(95)=2.32s  
       { expected_response:true }...: avg=777.29ms min=6.44ms  med=509.16ms max=5.24s  p(90)=1.85s    p(95)=2.32s  
     http_req_failed................: 0.00%  ✓ 0          ✗ 12139
     http_req_receiving.............: avg=278.62ms min=18.94µs med=74.49µs  max=4.76s  p(90)=1.23s    p(95)=1.75s  
     http_req_sending...............: avg=30.56ms  min=7.77µs  med=14.14µs  max=3.74s  p(90)=154.05µs p(95)=51.56ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s     p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=468.11ms min=6.31ms  med=344.12ms max=2.61s  p(90)=1.06s    p(95)=1.27s  
     http_reqs......................: 12139  173.389313/s
     iteration_duration.............: avg=1.5s     min=12.16ms med=995.77ms max=13.43s p(90)=3.64s    p(95)=4.51s  
     iterations.....................: 12139  173.389313/s
     vus............................: 1      min=1        max=499
     vus_max........................: 500    min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-router`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 36303      ✗ 0    
     data_received..................: 1.1 GB  15 MB/s
     data_sent......................: 14 MB   205 kB/s
     http_req_blocked...............: avg=15.8ms   min=1.51µs  med=3.07µs   max=3.81s p(90)=4.97µs   p(95)=12.33µs
     http_req_connecting............: avg=14.57ms  min=0s      med=0s       max=3.81s p(90)=0s       p(95)=0s     
     http_req_duration..............: avg=853.93ms min=6.55ms  med=628.77ms max=7.38s p(90)=1.81s    p(95)=2.39s  
       { expected_response:true }...: avg=853.93ms min=6.55ms  med=628.77ms max=7.38s p(90)=1.81s    p(95)=2.39s  
     http_req_failed................: 0.00%   ✓ 0          ✗ 12101
     http_req_receiving.............: avg=282.38ms min=28.15µs med=73.77µs  max=4.85s p(90)=1.05s    p(95)=1.55s  
     http_req_sending...............: avg=31.46ms  min=7.41µs  med=14.39µs  max=3.57s p(90)=347.99µs p(95)=81.78ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s    p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=540.08ms min=6.49ms  med=417.98ms max=2.29s p(90)=1.12s    p(95)=1.32s  
     http_reqs......................: 12101   172.865219/s
     iteration_duration.............: avg=1.52s    min=12.15ms med=1.12s    max=12.7s p(90)=3.53s    p(95)=4.44s  
     iterations.....................: 12101   172.865219/s
     vus............................: 1       min=1        max=497
     vus_max........................: 500     min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `wundergraph`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 34464      ✗ 0    
     data_received..................: 1.0 GB  14 MB/s
     data_sent......................: 14 MB   195 kB/s
     http_req_blocked...............: avg=20.86ms  min=1.55µs  med=3.07µs   max=5.57s  p(90)=4.91µs   p(95)=17.01µs
     http_req_connecting............: avg=20.18ms  min=0s      med=0s       max=5.57s  p(90)=0s       p(95)=0s     
     http_req_duration..............: avg=823.07ms min=6.37ms  med=488.36ms max=8.48s  p(90)=2.01s    p(95)=2.63s  
       { expected_response:true }...: avg=823.07ms min=6.37ms  med=488.36ms max=8.48s  p(90)=2.01s    p(95)=2.63s  
     http_req_failed................: 0.00%   ✓ 0          ✗ 11488
     http_req_receiving.............: avg=358.78ms min=24.96µs med=81.84µs  max=5.91s  p(90)=1.39s    p(95)=2s     
     http_req_sending...............: avg=35.48ms  min=7.98µs  med=14.73µs  max=5.77s  p(90)=299.67µs p(95)=81.42ms
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s     p(90)=0s       p(95)=0s     
     http_req_waiting...............: avg=428.8ms  min=6.28ms  med=341ms    max=2.84s  p(90)=900.06ms p(95)=1.13s  
     http_reqs......................: 11488   164.105123/s
     iteration_duration.............: avg=1.6s     min=12.13ms med=976.46ms max=12.91s p(90)=3.89s    p(95)=5.15s  
     iterations.....................: 11488   164.105123/s
     vus............................: 1       min=1        max=492
     vus_max........................: 500     min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mesh-supergraph-bun`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 24327      ✗ 0    
     data_received..................: 711 MB  10 MB/s
     data_sent......................: 9.6 MB  138 kB/s
     http_req_blocked...............: avg=2.47ms min=1.42µs  med=3.25µs  max=586.26ms p(90)=6.04µs   p(95)=227.77µs
     http_req_connecting............: avg=2.34ms min=0s      med=0s      max=586.2ms  p(90)=0s       p(95)=156.47µs
     http_req_duration..............: avg=2.27s  min=6.03ms  med=2.07s   max=8.65s    p(90)=4.36s    p(95)=5.53s   
       { expected_response:true }...: avg=2.27s  min=6.03ms  med=2.07s   max=8.65s    p(90)=4.36s    p(95)=5.53s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 8109 
     http_req_receiving.............: avg=33.8ms min=32.72µs med=76.25µs max=2.02s    p(90)=25.01ms  p(95)=197.68ms
     http_req_sending...............: avg=3.59ms min=8.46µs  med=14.98µs max=740.57ms p(90)=131.26µs p(95)=12.73ms 
     http_req_tls_handshaking.......: avg=0s     min=0s      med=0s      max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=2.23s  min=5.85ms  med=2.04s   max=8.56s    p(90)=4.3s     p(95)=5.52s   
     http_reqs......................: 8109    115.833358/s
     iteration_duration.............: avg=2.34s  min=11.59ms med=2.15s   max=8.66s    p(90)=4.46s    p(95)=5.63s   
     iterations.....................: 8109    115.833358/s
     vus............................: 1       min=1        max=499
     vus_max........................: 500     min=500      max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mesh-supergraph`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 15165   ✗ 0    
     data_received..................: 444 MB  6.3 MB/s
     data_sent......................: 6.0 MB  86 kB/s
     http_req_blocked...............: avg=596.21µs min=1.48µs  med=3.62µs  max=187.82ms p(90)=27.62µs p(95)=416.5µs 
     http_req_connecting............: avg=558.88µs min=0s      med=0s      max=187.74ms p(90)=0s      p(95)=343.69µs
     http_req_duration..............: avg=3.78s    min=10.43ms med=3.83s   max=8.72s    p(90)=6.09s   p(95)=6.55s   
       { expected_response:true }...: avg=3.78s    min=10.43ms med=3.83s   max=8.72s    p(90)=6.09s   p(95)=6.55s   
     http_req_failed................: 0.00%   ✓ 0       ✗ 5055 
     http_req_receiving.............: avg=3.23ms   min=34.41µs med=92.64µs max=297.25ms p(90)=1.59ms  p(95)=10.78ms 
     http_req_sending...............: avg=552.5µs  min=8.15µs  med=18.56µs max=153.14ms p(90)=48.43µs p(95)=110.86µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=3.77s    min=10.33ms med=3.83s   max=8.7s     p(90)=6.09s   p(95)=6.55s   
     http_reqs......................: 5055    72.1997/s
     iteration_duration.............: avg=3.81s    min=16.02ms med=3.85s   max=8.74s    p(90)=6.15s   p(95)=6.59s   
     iterations.....................: 5055    72.1997/s
     vus............................: 1       min=1     max=499
     vus_max........................: 500     min=500   max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-server`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 14955     ✗ 0    
     data_received..................: 438 MB  6.0 MB/s
     data_sent......................: 5.9 MB  81 kB/s
     http_req_blocked...............: avg=81.35µs  min=1.36µs  med=3.32µs  max=29.89ms  p(90)=128.14µs p(95)=327.21µs
     http_req_connecting............: avg=69.71µs  min=0s      med=0s      max=29.81ms  p(90)=74.96µs  p(95)=237.6µs 
     http_req_duration..............: avg=4.19s    min=88.81ms med=2.24s   max=45.23s   p(90)=3.48s    p(95)=24.01s  
       { expected_response:true }...: avg=4.19s    min=88.81ms med=2.24s   max=45.23s   p(90)=3.48s    p(95)=24.01s  
     http_req_failed................: 0.00%   ✓ 0         ✗ 4985 
     http_req_receiving.............: avg=311.21µs min=43.91µs med=117.3µs max=116.06ms p(90)=184.68µs p(95)=230.8µs 
     http_req_sending...............: avg=49.1µs   min=8.65µs  med=16.76µs max=28.85ms  p(90)=40.45µs  p(95)=60.01µs 
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s      max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=4.18s    min=88.68ms med=2.24s   max=45.23s   p(90)=3.48s    p(95)=24.01s  
     http_reqs......................: 4985    68.013056/s
     iteration_duration.............: avg=4.2s     min=96.01ms med=2.25s   max=45.24s   p(90)=3.49s    p(95)=24.03s  
     iterations.....................: 4985    68.013056/s
     vus............................: 26      min=26      max=500
     vus_max........................: 500     min=500     max=500

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview

Copy link

github-actions bot commented Nov 8, 2023

Overview for: federation-v1/constant-vus-subgraphs-delay-resources

This scenario runs 4 subgraphs and a GraphQL gateway with Federation v1 spec, and runs a heavy query. It's being executed with a constant amount of VUs over a fixed amount of time. It measure things like memory usage, CPU usage, average RPS. It also includes a summary of the entire execution, and metrics information about HTTP execution times.

This scenario was running 100 VUs over 30s

Comparison

Comparison

Gateway RPS ⬇️ Requests Duration Notes
apollo-router 167 5113 total, 0 failed avg: 536ms, p95: 711ms
conductor 145 4468 total, 0 failed avg: 678ms, p95: 819ms ❌ non-compatible response structure (4468)
wundergraph 107 3317 total, 0 failed avg: 884ms, p95: 1095ms
mesh-supergraph-bun 100 3101 total, 0 failed avg: 959ms, p95: 1331ms
cosmo 82 2578 total, 0 failed avg: 1156ms, p95: 1428ms
apollo-server 60 1886 total, 0 failed avg: 1618ms, p95: 2008ms
mesh-supergraph 60 1877 total, 0 failed avg: 1610ms, p95: 2464ms
mercurius 12 476 total, 0 failed avg: 7154ms, p95: 8309ms
Summary for: `apollo-router`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 15339      ✗ 0    
     data_received..................: 449 MB  15 MB/s
     data_sent......................: 6.1 MB  199 kB/s
     http_req_blocked...............: avg=44.95µs  min=1.38µs   med=2.93µs   max=92.83ms  p(90)=4.44µs   p(95)=6.11µs  
     http_req_connecting............: avg=18.14µs  min=0s       med=0s       max=6.34ms   p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=535.73ms min=290.38ms med=522.77ms max=1.3s     p(90)=657.22ms p(95)=710.66ms
       { expected_response:true }...: avg=535.73ms min=290.38ms med=522.77ms max=1.3s     p(90)=657.22ms p(95)=710.66ms
     http_req_failed................: 0.00%   ✓ 0          ✗ 5113 
     http_req_receiving.............: avg=16.47ms  min=33µs     med=67.36µs  max=533.31ms p(90)=58.74ms  p(95)=106.22ms
     http_req_sending...............: avg=1.68ms   min=8.2µs    med=13.41µs  max=378.91ms p(90)=45.32µs  p(95)=229.4µs 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=517.57ms min=269.88ms med=511.2ms  max=981.28ms p(90)=628.5ms  p(95)=668.5ms 
     http_reqs......................: 5113    167.411284/s
     iteration_duration.............: avg=592.56ms min=301ms    med=576.82ms max=1.42s    p(90)=743.31ms p(95)=822.75ms
     iterations.....................: 5113    167.411284/s
     vus............................: 100     min=100      max=100
     vus_max........................: 100     min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `conductor`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✗ valid response structure
      ↳  0% — ✓ 0 / ✗ 4468

     checks.........................: 66.66% ✓ 8936       ✗ 4468 
     data_received..................: 2.1 MB 67 kB/s
     data_sent......................: 5.3 MB 173 kB/s
     http_req_blocked...............: avg=70.09µs  min=1µs      med=2.13µs   max=6.64ms   p(90)=3.25µs   p(95)=3.9µs   
     http_req_connecting............: avg=63.9µs   min=0s       med=0s       max=6.61ms   p(90)=0s       p(95)=0s      
     http_req_duration..............: avg=678.3ms  min=401.44ms med=678.1ms  max=960.62ms p(90)=788.69ms p(95)=819.47ms
       { expected_response:true }...: avg=678.3ms  min=401.44ms med=678.1ms  max=960.62ms p(90)=788.69ms p(95)=819.47ms
     http_req_failed................: 0.00%  ✓ 0          ✗ 4468 
     http_req_receiving.............: avg=44.86µs  min=17.27µs  med=38.35µs  max=2.3ms    p(90)=63.82µs  p(95)=74.93µs 
     http_req_sending...............: avg=20.32µs  min=8.18µs   med=12.83µs  max=731.35µs p(90)=21.04µs  p(95)=26.93µs 
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=678.24ms min=401.39ms med=678.04ms max=960.39ms p(90)=788.64ms p(95)=819.37ms
     http_reqs......................: 4468   145.376937/s
     iteration_duration.............: avg=678.98ms min=402.1ms  med=678.63ms max=962.88ms p(90)=789.66ms p(95)=820.15ms
     iterations.....................: 4468   145.376937/s
     vus............................: 100    min=100      max=100
     vus_max........................: 100    min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `wundergraph`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 9951       ✗ 0    
     data_received..................: 291 MB  9.4 MB/s
     data_sent......................: 3.9 MB  127 kB/s
     http_req_blocked...............: avg=46.42µs  min=1.38µs   med=2.57µs   max=43.85ms  p(90)=3.91µs  p(95)=5.67µs  
     http_req_connecting............: avg=28.48µs  min=0s       med=0s       max=11.32ms  p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=883.69ms min=562.37ms med=881.18ms max=1.32s    p(90)=1.04s   p(95)=1.09s   
       { expected_response:true }...: avg=883.69ms min=562.37ms med=881.18ms max=1.32s    p(90)=1.04s   p(95)=1.09s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 3317 
     http_req_receiving.............: avg=9.82ms   min=36.32µs  med=75.31µs  max=328.2ms  p(90)=32.31ms p(95)=64.07ms 
     http_req_sending...............: avg=1.57ms   min=8.3µs    med=12.86µs  max=411.45ms p(90)=34.25µs p(95)=526.68µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=872.29ms min=562.3ms  med=874.22ms max=1.27s    p(90)=1.01s   p(95)=1.06s   
     http_reqs......................: 3317    107.272335/s
     iteration_duration.............: avg=921.94ms min=575.26ms med=913.47ms max=1.44s    p(90)=1.1s    p(95)=1.16s   
     iterations.....................: 3317    107.272335/s
     vus............................: 100     min=100      max=100
     vus_max........................: 100     min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mesh-supergraph-bun`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 9303       ✗ 0    
     data_received..................: 272 MB  8.8 MB/s
     data_sent......................: 3.7 MB  119 kB/s
     http_req_blocked...............: avg=101.76µs min=1.6µs    med=3.43µs   max=19.6ms   p(90)=5.53µs  p(95)=10.64µs 
     http_req_connecting............: avg=79.48µs  min=0s       med=0s       max=5.8ms    p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=958.85ms min=354.12ms med=942.91ms max=1.9s     p(90)=1.22s   p(95)=1.33s   
       { expected_response:true }...: avg=958.85ms min=354.12ms med=942.91ms max=1.9s     p(90)=1.22s   p(95)=1.33s   
     http_req_failed................: 0.00%   ✓ 0          ✗ 3101 
     http_req_receiving.............: avg=12.04ms  min=35.41µs  med=102.05µs max=564.05ms p(90)=1.66ms  p(95)=37.88ms 
     http_req_sending...............: avg=413.33µs min=9.26µs   med=16.78µs  max=175.87ms p(90)=49.45µs p(95)=169.98µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=946.39ms min=354.01ms med=933.99ms max=1.9s     p(90)=1.19s   p(95)=1.29s   
     http_reqs......................: 3101    100.396707/s
     iteration_duration.............: avg=987.91ms min=363.75ms med=968.69ms max=1.92s    p(90)=1.26s   p(95)=1.39s   
     iterations.....................: 3101    100.396707/s
     vus............................: 100     min=100      max=100
     vus_max........................: 100     min=100      max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `cosmo`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 7734      ✗ 0    
     data_received..................: 226 MB  7.2 MB/s
     data_sent......................: 3.1 MB  98 kB/s
     http_req_blocked...............: avg=42.12µs min=1.34µs   med=2.74µs  max=19.61ms  p(90)=4.4µs   p(95)=10.44µs 
     http_req_connecting............: avg=29.55µs min=0s       med=0s      max=3.13ms   p(90)=0s      p(95)=0s      
     http_req_duration..............: avg=1.15s   min=765.61ms med=1.14s   max=1.82s    p(90)=1.36s   p(95)=1.42s   
       { expected_response:true }...: avg=1.15s   min=765.61ms med=1.14s   max=1.82s    p(90)=1.36s   p(95)=1.42s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 2578 
     http_req_receiving.............: avg=8.91ms  min=35.77µs  med=70.52µs max=328.81ms p(90)=21.65ms p(95)=70.02ms 
     http_req_sending...............: avg=1.41ms  min=7.36µs   med=13.36µs max=171.39ms p(90)=62.81µs p(95)=761.42µs
     http_req_tls_handshaking.......: avg=0s      min=0s       med=0s      max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=1.14s   min=765.21ms med=1.13s   max=1.82s    p(90)=1.35s   p(95)=1.39s   
     http_reqs......................: 2578    82.506992/s
     iteration_duration.............: avg=1.19s   min=784ms    med=1.17s   max=1.86s    p(90)=1.41s   p(95)=1.47s   
     iterations.....................: 2578    82.506992/s
     vus............................: 45      min=45      max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `apollo-server`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 5658      ✗ 0    
     data_received..................: 166 MB  5.3 MB/s
     data_sent......................: 2.2 MB  72 kB/s
     http_req_blocked...............: avg=571µs    min=1.68µs   med=4.07µs   max=25.38ms p(90)=5.97µs   p(95)=505.13µs
     http_req_connecting............: avg=557.01µs min=0s       med=0s       max=25.35ms p(90)=0s       p(95)=232.66µs
     http_req_duration..............: avg=1.61s    min=801.91ms med=1.52s    max=10.2s   p(90)=1.85s    p(95)=2s      
       { expected_response:true }...: avg=1.61s    min=801.91ms med=1.52s    max=10.2s   p(90)=1.85s    p(95)=2s      
     http_req_failed................: 0.00%   ✓ 0         ✗ 1886 
     http_req_receiving.............: avg=176.45µs min=48.72µs  med=115.35µs max=20.38ms p(90)=181.06µs p(95)=252.05µs
     http_req_sending...............: avg=82.86µs  min=9.69µs   med=21.87µs  max=7.81ms  p(90)=38.18µs  p(95)=114.54µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s      p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=1.61s    min=798.89ms med=1.52s    max=10.2s   p(90)=1.85s    p(95)=2s      
     http_reqs......................: 1886    60.489442/s
     iteration_duration.............: avg=1.63s    min=834.54ms med=1.53s    max=10.22s  p(90)=1.86s    p(95)=2.03s   
     iterations.....................: 1886    60.489442/s
     vus............................: 30      min=30      max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mesh-supergraph`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 5631      ✗ 0    
     data_received..................: 165 MB  5.3 MB/s
     data_sent......................: 2.2 MB  72 kB/s
     http_req_blocked...............: avg=174.44µs min=1.5µs    med=4.65µs   max=29.66ms  p(90)=6.54µs  p(95)=507.98µs
     http_req_connecting............: avg=150.68µs min=0s       med=0s       max=6.2ms    p(90)=0s      p(95)=351.06µs
     http_req_duration..............: avg=1.61s    min=803.64ms med=1.46s    max=3.81s    p(90)=2.25s   p(95)=2.46s   
       { expected_response:true }...: avg=1.61s    min=803.64ms med=1.46s    max=3.81s    p(90)=2.25s   p(95)=2.46s   
     http_req_failed................: 0.00%   ✓ 0         ✗ 1877 
     http_req_receiving.............: avg=2.71ms   min=38.03µs  med=134.29µs max=165.92ms p(90)=2.67ms  p(95)=10.88ms 
     http_req_sending...............: avg=206.25µs min=8.79µs   med=25.61µs  max=27.4ms   p(90)=54.16µs p(95)=506.08µs
     http_req_tls_handshaking.......: avg=0s       min=0s       med=0s       max=0s       p(90)=0s      p(95)=0s      
     http_req_waiting...............: avg=1.6s     min=803.27ms med=1.46s    max=3.81s    p(90)=2.25s   p(95)=2.46s   
     http_reqs......................: 1877    60.365239/s
     iteration_duration.............: avg=1.63s    min=819.74ms med=1.49s    max=3.83s    p(90)=2.28s   p(95)=2.49s   
     iterations.....................: 1877    60.365239/s
     vus............................: 16      min=16      max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview
Summary for: `mercurius`

K6 Output

     ✓ response code was 200
     ✓ no graphql errors
     ✓ valid response structure

     checks.........................: 100.00% ✓ 1428      ✗ 0    
     data_received..................: 42 MB   1.1 MB/s
     data_sent......................: 565 kB  15 kB/s
     http_req_blocked...............: avg=83.1µs   min=1.56µs  med=3.99µs   max=2.86ms   p(90)=258.51µs p(95)=470.97µs
     http_req_connecting............: avg=60.98µs  min=0s      med=0s       max=901.62µs p(90)=224.55µs p(95)=430.24µs
     http_req_duration..............: avg=7.15s    min=3.18s   med=7.45s    max=8.56s    p(90)=8.23s    p(95)=8.3s    
       { expected_response:true }...: avg=7.15s    min=3.18s   med=7.45s    max=8.56s    p(90)=8.23s    p(95)=8.3s    
     http_req_failed................: 0.00%   ✓ 0         ✗ 476  
     http_req_receiving.............: avg=161.96µs min=62.44µs med=122.72µs max=6.68ms   p(90)=182.46µs p(95)=213.1µs 
     http_req_sending...............: avg=61µs     min=9.46µs  med=22.02µs  max=5.32ms   p(90)=136.11µs p(95)=189.06µs
     http_req_tls_handshaking.......: avg=0s       min=0s      med=0s       max=0s       p(90)=0s       p(95)=0s      
     http_req_waiting...............: avg=7.15s    min=3.18s   med=7.45s    max=8.56s    p(90)=8.23s    p(95)=8.3s    
     http_reqs......................: 476     12.650824/s
     iteration_duration.............: avg=7.16s    min=3.19s   med=7.46s    max=8.57s    p(90)=8.24s    p(95)=8.31s   
     iterations.....................: 476     12.650824/s
     vus............................: 12      min=12      max=100
     vus_max........................: 100     min=100     max=100

Performance Overview

Performance Overview

Subgraphs Overview

Subgraphs Overview

HTTP Overview

HTTP Overview

@the-guild-org the-guild-org deleted a comment from github-actions bot Jan 22, 2024
@dotansimha dotansimha force-pushed the add-conductor-fresh branch 2 times, most recently from 1de4deb to 5c2aae0 Compare February 15, 2024 16:21
add to fed v1 pipeline

use master

pin to commit

use commit hash

add correct hash

fix command

modify path

relative path

fix path

run the command properly

try this

fix supplied config path

full path

use absolute supergraph schema path

generate-report.ts optional chaining on .fails

configure host to allow access

update conductor docker image

remove v2 for faster ci runs

update conductor docker image

update conductor image

remove reports file manipulation

remove mesh supergraph bun

use stable runner

trigger rerun

Update federation-v1/gateways/conductor/docker-compose.yaml

Update federation-v1/gateways/conductor/config.yaml

add mesh bun back

remove mesh

fix tsx

update node cersion

fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants