Performance Testing

Flame graphs

To generate a flame graph of a particular function, replace that function with:

:eflame.apply(ModuleName, :function_name, [arguments])

For example, Foo.Bar.do_something(a, b, c) would be :eflame.apply(Foo.Bar, :do_something, [a, b, c]).

Then once you do that, you can generate the chart by running:

./deps/eflame/stack_to_flame.sh < stacks.out > flame.svg

Available scripts

create_users.exs

https://github.com/mbta/alerts_concierge/blob/master/scripts/create_users.exs

mix run scripts/create_users.exs --count 100 # creates 100 users
mix run scripts/create_users.exs --delete    # deletes all users created by this script

send_notifications.exs

https://github.com/mbta/alerts_concierge/blob/master/scripts/send_notifications.exs

# add 1000 notifications to the queue and wait until they are all processed
# then delete all those notifications from the database
env RATE_LIMIT=1000 mix run create_users.exs --count 1000 --delete

# the same as above except that as soon as a notification is processed,
# the notification worker immediately pops another one off the queue
# by default, the notification worker waits 100 ms
env RATE_LIMIT=1000 SEND_RATE=0 mix run create_users.exs --count 1000 --delete

To get a rough idea about how long it takes to send a certain number of notifications, you can use time:

time env RATE_LIMIT=1000 SEND_RATE=0 mix run create_users.exs --count 1000 --delete

match_alerts.exs

https://github.com/mbta/alerts_concierge/blob/master/scripts/match_alerts.exs

This script expects that there are users with subscriptions already set in the database.

# wait for 2500 matches on each of 5 alerts
env RATE_LIMIT=1000 SEND_RATE=0 mix run scripts/match_alerts.exs --match 2500 --alerts 5

Findings

send_notifications.exs

Run on 2.6 GHz Intel Core i7 macOS High Sierra, Elixir version 1.5.2-otp-20

Commit	0 Notifications	500 notifications	5,000 notifications	500,000 notifications
48a03b1	0m21.808s	0m22.159s	0m29.858s	16m38.367s

match_alerts.exs

https://docs.google.com/spreadsheets/d/1Rio2fs9rHNrARnu1_yFeCDnFjRf_9oqk5FuIm0GBJbk/edit?usp=sharing

Total time

The time it will take for a subscriber to receive a text message about an alert is the sum of the following times:

1. time to notice the alert
2. time to enter alert in system
3. time for the alert to show up in the feed
4. time to pull the alert into concierge
5. time to match the alerts
6. time to add and remove alert from queue
7. time for AWS API call to go through
8. time for SMS to reach phone

Of these things, Concierge has some control over 4, 5, and 6. 4 can be changed with the ALERT_FETCH_INTERVAL environment variable.

The send_notifications.exs script should give us some idea about 6. I ran it with 500,000 notifications with a SEND_RATE of 0 and got 18 minutes, almost exactly.

Load Testing

We have a load testing script based on Locust. Refer to the README for usage instructions.

2018-09-18 Load Testing