Basline performance checks

[slifty]

As part of the wrap-up of the initial launch we have spent some time trying to understand some of the impact the SFTP service's use has on the Permanent API machine. @jasonaowen and @Fenn-CS have been taking the lead on this (with some moral support from me!)

We explored the concept of API throttling, which is still a potential long term path, but in the shorter term it seems that the right approach is going to be to just make sure the Permanent API service can handle a certain amount of active load.

Jason has already made some improvements to the way the permanent service invokes PHP, which should help prevent crashes under heavy load. That said, in the most recent test that Fon and I ran we found that the CPU was being close to maxed out on the development machine. RAM seemed fairly unscathed.

Nothing crashed; the files uploaded. That said, it impacted performance for ALL users -- e.g. loading https://dev.permanent.org/app took around 8 seconds simply to load the login screen.

Next Steps

The next step we're proposing is to increase the size of the dev box for the API. This is an infrastructure task, and a simple PR, but I decided to make the issue here anyway given the entry point for these performance requirements is sftp-service related.

Some options:

• m4.large This is the current size: 2 virtual CPUs and 8 GB of memory. (0.10 / hour on demand)
• m4.xlarge doubles both numbers: 4 vCPUs and 16GB memory. (0.20 / hour on demand)
• c7g.xlarge doubles just the CPU: 4 CPUs and 8 GB memory (0.1445 / hour on demand)

You can see all the possible options at the EC2 documentation.

Side note: Permanent might benefit financially from reserved instances (I don't know, maybe y'all use reserved instances already!)

(cc @kfogel and @cecilia-donnelly on the above)

[cecilia-donnelly]

Thank you for writing this up, @slifty ! Doubling the CPU with c7g.xlarge makes sense to me. I think @jasonaowen was going to open that PR?

[kfogel]

@slifty is going to open the PR (so @jasonaowen please don't worry about it). /CC @cecilia-donnelly

[kfogel]

@slifty (Was the previous closure just an accident?)

[slifty]

Whoops sorry for not adding context -- I re-opened because 135 has been reverted.

Also arguably 135 shouldn't have closed this issue. It accomplished the listed next steps in the issue so I had marked it as resolving (though the title of the issue does imply more work beyond just the steps listed in the issue).

At any rate, the current state of the issue is accurate!