Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

If the app was created instead of fetched, perform upload in a retry loop. #164

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

krimsonkla
Copy link

@krimsonkla krimsonkla commented Nov 19, 2019

This is my response to issue #151

My experience with this issue has been that if the plugin attempts to perform a release upload directly after creating the app, it will fail for some time. Apparently there is a background task or data replication within appcenter that must occur before the upload will complete.

This may need some fine tuning on the retry loop logic in order to avoid lengthy delays. At the moment it is using a randomized exponential backoff to prevent flooding the system. I feared this issue may also be throttling related and wanted to reduce the number of API hits.

Here are some sample uploads with debug messages. I included 3 simply to show the time range variance.

------------------------------
[15:34:23]: --- Step: appcenter_upload ---
[15:34:23]: ------------------------------
[15:34:24]: Created Android/Cordova app with name "TT_Android_jmr_capacitor_development" and display name "TT_Android_jmr_capacitor_development" for organization "cardtapp"
[15:34:24]: Starting release upload...
[15:34:24]: Not found, invalid owner or application name
[15:34:24]: Release upload failed (), retrying. Attempt# 1. 0.6753695084145964 seconds until the next try.
[15:34:25]: Starting release upload...
[15:34:26]: Not found, invalid owner or application name
[15:34:26]: Release upload failed (), retrying. Attempt# 2. 0.6609939784716703 seconds until the next try.
[15:34:26]: Starting release upload...
[15:34:27]: Not found, invalid owner or application name
[15:34:27]: Release upload failed (), retrying. Attempt# 3. 1.090680998555765 seconds until the next try.
[15:34:28]: Starting release upload...
[15:34:28]: Not found, invalid owner or application name
[15:34:28]: Release upload failed (), retrying. Attempt# 4. 2.246770862843908 seconds until the next try.
[15:34:30]: Starting release upload...
[15:34:31]: Not found, invalid owner or application name
[15:34:31]: Release upload failed (), retrying. Attempt# 5. 1.7464416838121162 seconds until the next try.
[15:34:33]: Starting release upload...
[15:34:33]: Not found, invalid owner or application name
[15:34:33]: Release upload failed (), retrying. Attempt# 6. 5.069035691532717 seconds until the next try.
[15:34:38]: Starting release upload...
[15:34:39]: Not found, invalid owner or application name
[15:34:39]: Release upload failed (), retrying. Attempt# 7. 7.15001284895436 seconds until the next try.
[15:34:46]: Starting release upload...
[15:34:46]: Not found, invalid owner or application name
[15:34:46]: Release upload failed (), retrying. Attempt# 8. 7.434021989663073 seconds until the next try.
[15:34:54]: Starting release upload...
[15:34:54]: Not found, invalid owner or application name
[15:34:54]: Release upload failed (), retrying. Attempt# 9. 10.993484039110317 seconds until the next try.
[15:35:05]: Starting release upload...
[15:35:06]: Not found, invalid owner or application name
[15:35:06]: Release upload failed (), retrying. Attempt# 10. 15.514312320897186 seconds until the next try.
[15:35:21]: Starting release upload...
[15:35:22]: Not found, invalid owner or application name
[15:35:22]: Release upload failed (), retrying. Attempt# 11. 35.57871172061688 seconds until the next try.
[15:35:57]: Starting release upload...
[15:35:58]: Uploading release binary...
[15:36:05]: Binary uploaded
[15:36:05]: Release '1' committed: https://appcenter.ms/orgs/cardtapp/apps/TT_Android_jmr_capacitor_development/distribute/releases/1
[15:36:06]: Release '1' (1.0.0) was successfully updated
[15:36:08]: Release '1' (1.0.0) was successfully distributed'
[15:36:08]: Release '1' (1.0.0) was successfully distributed to group "Collaborators"
[15:36:08]: Release '1' is available for download at: https://install.appcenter.ms/orgs/cardtapp/apps/TT_Android_jmr_capacitor_development

------------------------------
[16:04:26]: --- Step: appcenter_upload ---
[16:04:26]: ------------------------------
[16:04:27]: Created Android/Cordova app with name "TT_Android_jmr_capacitor_development" and display name "TT_Android_jmr_capacitor_development" for organization "cardtapp"
[16:04:27]: Starting release upload...
[16:04:28]: Not found, invalid owner or application name
[16:04:28]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 1. 0.3392837364871719 seconds until the next try.
[16:04:28]: Starting release upload...
[16:04:28]: Not found, invalid owner or application name
[16:04:28]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 2. 0.9820026383987693 seconds until the next try.
[16:04:29]: Starting release upload...
[16:04:30]: Not found, invalid owner or application name
[16:04:30]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 3. 1.6834042397665654 seconds until the next try.
[16:04:32]: Starting release upload...
[16:04:32]: Not found, invalid owner or application name
[16:04:32]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 4. 2.5111925712256165 seconds until the next try.
[16:04:34]: Starting release upload...
[16:04:35]: Uploading release binary...
[16:04:43]: Binary uploaded
[16:04:44]: Release '1' committed: https://appcenter.ms/orgs/cardtapp/apps/TT_Android_jmr_capacitor_development/distribute/releases/1
[16:04:45]: Release '1' (1.0.0) was successfully updated
[16:04:46]: Release '1' (1.0.0) was successfully distributed'
[16:04:46]: Release '1' (1.0.0) was successfully distributed to group "Collaborators"
[16:04:46]: Release '1' is available for download at: https://install.appcenter.ms/orgs/cardtapp/apps/TT_Android_jmr_capacitor_development
[16:05:31]: ------------------------------
[16:05:31]: --- Step: appcenter_upload ---
[16:05:31]: ------------------------------
[16:05:32]: Created Android/Cordova app with name "TT_Android_jmr_capacitor_release" and display name "TT_Android_jmr_capacitor_release" for organization "cardtapp"
[16:05:32]: Starting release upload...
[16:05:32]: Not found, invalid owner or application name
[16:05:32]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 1. 0.31614624606086505 seconds until the next try.
[16:05:33]: Starting release upload...
[16:05:33]: Not found, invalid owner or application name
[16:05:33]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 2. 0.6825448063826226 seconds until the next try.
[16:05:34]: Starting release upload...
[16:05:34]: Not found, invalid owner or application name
[16:05:34]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 3. 1.4935258341607511 seconds until the next try.
[16:05:36]: Starting release upload...
[16:05:36]: Not found, invalid owner or application name
[16:05:36]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 4. 1.5178329653783242 seconds until the next try.
[16:05:38]: Starting release upload...
[16:05:38]: Not found, invalid owner or application name
[16:05:38]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 5. 3.3175981402831893 seconds until the next try.
[16:05:41]: Starting release upload...
[16:05:42]: Not found, invalid owner or application name
[16:05:42]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 6. 5.47813275494805 seconds until the next try.
[16:05:47]: Starting release upload...
[16:05:48]: Not found, invalid owner or application name
[16:05:48]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 7. 6.836601644962349 seconds until the next try.
[16:05:54]: Starting release upload...
[16:05:55]: Not found, invalid owner or application name
[16:05:55]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 8. 8.030860839985706 seconds until the next try.
[16:06:03]: Starting release upload...
[16:06:03]: Not found, invalid owner or application name
[16:06:03]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 9. 13.051388569366987 seconds until the next try.
[16:06:16]: Starting release upload...
[16:06:17]: Not found, invalid owner or application name
[16:06:17]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 10. 10.483675610822491 seconds until the next try.
[16:06:27]: Starting release upload...
[16:06:28]: Not found, invalid owner or application name
[16:06:28]: Release upload failed (Fastlane::Actions::RetryableException), retrying. Attempt# 11. 42.22010004902201 seconds until the next try.
[16:07:10]: Starting release upload...
[16:07:11]: Uploading release binary...
[16:07:16]: Binary uploaded
[16:07:16]: Release '1' committed: https://appcenter.ms/orgs/cardtapp/apps/TT_Android_jmr_capacitor_release/distribute/releases/1
[16:07:17]: Release '1' (1.0.0) was successfully updated
[16:07:19]: Release '1' (1.0.0) was successfully distributed'
[16:07:19]: Release '1' (1.0.0) was successfully distributed to group "Collaborators"
[16:07:19]: Release '1' is available for download at: https://install.appcenter.ms/orgs/cardtapp/apps/TT_Android_jmr_capacitor_release

@msftclas
Copy link

msftclas commented Nov 19, 2019

CLA assistant check
All CLA requirements met.

@jp-andre jp-andre self-requested a review December 2, 2019 10:45
Copy link
Contributor

@jp-andre jp-andre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this contribution! I believe we should be careful with adding a new dependency here.
Also, this will require new unit test.

each_retry = Proc.new do |exception, try, elapsed_time, next_interval|
UI.message("Release upload failed (#{exception.message}), retrying. Attempt# #{try}. #{next_interval} seconds until the next try.")
end
Retriable.retriable(tries: 60, on_retry: each_retry) do
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to add a new dependency, but it's not added to Gemfile. Retriable is also not a direct dependency in fastlane.

self.run_dsym_upload(params) unless upload_mapping_only || upload_build_only
self.run_mapping_upload(params) unless upload_dsym_only || upload_build_only
else
return
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not seem to be a case that we should reach.

@krimsonkla
Copy link
Author

krimsonkla commented Dec 4, 2019

@jp-andre Hi Jean-Philippe, I opened this more to point out the problem identified by #151 and a potential resolution. I'm not convinced that having an exponential backoff retry coded into the client is a great fix. Is this something that the appcenter backend team should look at before we attempt to hack around the app creation readiness latency?

I'm more than happy to fix this up if you want to move forward with it.

I had noticed that the appcenter plugin has no 3rd party gem dependencies and assumed that was a conscious decision. If you want to move forward with this PR would you prefer the addition of the Retriable gem or an internal implementation of some form of retry?

@jp-andre
Copy link
Contributor

Hi @krimsonkla -- this PR was totally forgotten, I'm sorry about that.
If this is something you are still interested in, can you address the comments?

@jp-andre
Copy link
Contributor

jp-andre commented Jul 7, 2020

Keeping this opened, as I think the approach is acceptable... better than nothing :)
Let's make sure the extra dependency is taken care of before merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants