Fix iOS compilation error and add macOS + iOS feedback example applications #72

MichaelHills · 2020-09-27T21:53:29Z

Fixes #67

I started at bevyengine/bevy#87 and ended up here

Some prior discussion in #69

This enables building for iOS aarch64-apple-ios and x86_64-apple-ios and I managed to get audio playback on device and simulator via rodio/cpal. The actual change here affects the mic recording callback so I decided to write a feedback example app for manual testing.

Fixes iOS compiling. coreaudio-sys#33 landed and released (dependency)
Write feedback.rs example for macOS to test recording + playback and manually test everything works
Add an iOS Xcode project feedback example

Manually tested microphone + playback works on iOS

iPhone 6 (12.4.8)
iPad 5th gen (14.1)
iPhone 11 (14.2)
iOS simulator

There seem to be some differences between macOS and iOS CoreAudio, in that the iOS can only enable mic recording on an uninitialized AudioUnit. Similarly the iOS one requires an active AVAudioSession to do some operations like reading out the current hardware buffer size.

MichaelHills · 2020-09-30T14:43:46Z

src/audio_unit/render_callback.rs

@@ -525,7 +540,7 @@ impl AudioUnit {
                unsafe {
                    // Retrieve the up-to-date stream format.
                    let id = sys::kAudioUnitProperty_StreamFormat;
-                    let asbd = match super::get_property(audio_unit, id, Scope::Input, Element::Output) {
+                    let asbd = match super::get_property(audio_unit, id, Scope::Output, Element::Input) {


We are fetching the stream format for the audio unit being used to record the mic. Therefore we want to use the "input" bus/element, but the output scope. The input scope would give the hardware format, not the user requested format that will be rendered into.

As per #60

MichaelHills · 2020-09-30T14:44:31Z

src/audio_unit/render_callback.rs

@@ -471,7 +474,7 @@ impl AudioUnit {
        // First, we'll retrieve the stream format so that we can ensure that the given callback
        // format matches the audio unit's format.
        let id = sys::kAudioUnitProperty_StreamFormat;
-        let asbd = self.get_property(id, Scope::Input, Element::Input)?;
+        let asbd = self.get_property(id, Scope::Output, Element::Input)?;


As per #60 I think Input/Input gives the hardware stream format, not the user requested one.

MichaelHills · 2020-09-30T14:46:00Z

src/audio_unit/render_callback.rs

@@ -398,7 +401,7 @@ impl AudioUnit {
        // First, we'll retrieve the stream format so that we can ensure that the given callback
        // format matches the audio unit's format.
        let id = sys::kAudioUnitProperty_StreamFormat;
-        let asbd = try!(self.get_property(id, Scope::Output, Element::Output));
+        let asbd = try!(self.get_property(id, Scope::Input, Element::Output));


As per #60 I think Output/Output gives the hardware stream format, not the user requested one.

MichaelHills · 2020-09-30T15:13:32Z

@mitchmindtree I think this is ready for first pass of review now. I also updated 3 places where the stream formats are fetched to match comments from @plietar in #67 as I ran into issues writing the feedback example. Using signed integer i16 in the example doesn't work without making those tweaks. I'll need to verify that rodio/cpal still work with these changes. For that I still need to write/productionize the cpal ios coreaudio host.

simlay · 2020-09-30T16:24:46Z

.travis.yml

@@ -20,6 +20,10 @@ script:
 - cargo build --verbose
 - cargo test --verbose
 - cargo doc --verbose
+- cargo build --verbose --target aarch64-apple-ios
+- cargo test --verbose --target aarch64-apple-ios


cargo test won't work with different architectures. If you'd really want to run the tests in an iOS simulator, dinghy is your friend but you'd still only be able to have CI run it with the x86_64-apple-ios target.

I've revised the changes, now just doing a build without the test. How do I actually kick off a travis build to test it?

MichaelHills · 2020-10-02T15:22:19Z

Now added the iOS Xcode project sample

MichaelHills · 2020-10-15T13:24:00Z

@naithar @simlay I updated Cargo.toml to point to coreaudio-sys master for now so my branch should be directly usable. Are either of you able to help me test?

naithar · 2020-10-15T21:52:33Z

@MichaelHills any specific things to test? I could try testing it on weekend.

MichaelHills · 2020-10-17T13:16:19Z

@naithar I suppose testing that mac feedback example still works and the new iOS example works on device + simulator.

Plus same thing on my cpal PR RustAudio/cpal#485

If both are good then can work towards getting them released and finally into Bevy. :)

simlay · 2020-10-17T19:27:36Z

I gave this a test and found some compilation errors on my end. I had to add the ios example as a new workspace member and then make a Cargo.toml in examples/ios and then change some of the imports (I don't really understand the need for these though). Here are the changes I had to do: simlay@eb73670

I did manage to get the feedback example to run on my iPhone 7. I'm not sure what the example should do but it runs and I see some of the printlns but don't hear anything. Should I?

The iOS simulator crashes with this stacktrace:

2020-10-17 12:26:30.157982-0700 coreaudio-ios-example[98758:6316601]  HALDefaultDevice::Initialize: couldn't add the default input device listener, Error: 1102 ()
2020-10-17 12:26:30.158269-0700 coreaudio-ios-example[98758:6316601]  HALDefaultDevice::Initialize: couldn't add the default output device listener, Error: 1102 ()
2020-10-17 12:26:30.158481-0700 coreaudio-ios-example[98758:6316601]  HALDefaultDevice::Initialize: couldn't add the default system output device listener, Error: 1102 ()
2020-10-17 12:26:30.158682-0700 coreaudio-ios-example[98758:6316601]  HALDefaultDevice::Initialize: couldn't add the default shared output device listener, Error: 1102 ()
2020-10-17 12:26:30.158993-0700 coreaudio-ios-example[98758:6316601] [plugin] AddInstanceForFactory: No factory registered for id <CFUUID 0x60000311a7a0> F8BB1C28-BAE8-11D6-9C31-00039315CD46
2020-10-17 12:26:30.159495-0700 coreaudio-ios-example[98758:6316601]  AudioObjectSetPropertyData: no object with given ID 0
2020-10-17 12:26:30.159618-0700 coreaudio-ios-example[98758:6316601] AudioSessionSimulatorClientManager.cpp:83:SimulatorUpdateHALForPrimaySession_Priv: Failed to set processVolumeScalar on device. Error: 560947818
2020-10-17 12:26:30.160191-0700 coreaudio-ios-example[98758:6316601] Calling rust_ios_main()
2020-10-17 12:26:30.256334-0700 coreaudio-ios-example[98758:6316700]  HALCADClient::GetPropertyData: unknown property
2020-10-17 12:26:30.256449-0700 coreaudio-ios-example[98758:6316700] [ddagg]        AggregateDevice.mm:790   couldn't get default input device, ID = 0, err = 0!
2020-10-17 12:26:30.256723-0700 coreaudio-ios-example[98758:6316700]  HALCADClient::GetPropertyData: unknown property
2020-10-17 12:26:30.256822-0700 coreaudio-ios-example[98758:6316700] [ddagg]        AggregateDevice.mm:790   couldn't get default output device, ID = 0, err = 0!
2020-10-17 12:26:30.256947-0700 coreaudio-ios-example[98758:6316700]  AudioDeviceStop: no device with given ID
2020-10-17 12:26:30.257052-0700 coreaudio-ios-example[98758:6316700] [aqme] AQMEIO.cpp:320:_FindIOUnit: error -66680 finding/initializing AQDefaultDevice
2020-10-17 12:26:30.257428-0700 coreaudio-ios-example[98758:6316601] [aurioc] AURemoteIO.cpp:1086:Initialize: failed: -10851 (enable 2, outf< 2 ch,      0 Hz, Int16, inter> inf< 2 ch,      0 Hz, Int16, inter>)
2020-10-17 12:26:30.257433-0700 coreaudio-ios-example[98758:6316705]  AudioDeviceStop: no device with given ID
2020-10-17 12:26:30.260231-0700 coreaudio-ios-example[98758:6316705] [aqme] AQMEIO.cpp:320:_FindIOUnit: error -66680 finding/initializing AQDefaultDevice
2020-10-17 12:26:30.260360-0700 coreaudio-ios-example[98758:6316705] CA_UISoundClient.cpp:110:CA_UISoundClientBase: * * * NULL AQIONode object
2020-10-17 12:26:30.260499-0700 coreaudio-ios-example[98758:6316705] CA_UISoundClient.cpp:772:UISoundNewRenderer: Can't make UISound Renderer
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: AudioUnit(InvalidPropertyValue)', examples/ios/src/lib.rs:5:3
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
libc++abi.dylib: terminating with uncaught foreign exception

naithar · 2020-10-17T22:10:07Z

Not sure if I'm doing everything correctly, but testing cpal, and coreaudio-.. crates changes in bevy I'm receiving a crash:

thread '<unnamed>' panicked at 'assertion failed: self.min_sample_rate <= sample_rate && sample_rate <= self.max_sample_rate', /Users/naithar/.cargo/git/checkouts/cpal-d70c735e19f75203/ce7a2a0/src/lib.rs:566:9

MichaelHills · 2020-10-18T07:46:53Z

@simlay it's meant to feedback the microphone with a 1 second delay, so best used with headphones. Try speaking and see if you hear your own voice play back.

Not sure why our setups are different and simulator doesn't work. I'll cargo clean and re-test later tonight. What version of Xcode/iOS sim?

I'll also setup the bevy ios example to have some audio playback to see if I can reproduce @naithar's issue.

MichaelHills · 2020-10-18T14:00:17Z

@simlay your simulator errors look more like some kind of system issue. Can you try the suggestion here?

As for the workspace thing, sorry I forgot to git add examples/ios/Cargo.toml :( can you try my branch again? I never needed the workspace thing. I'm using 1.48.0-nightly. Any ideas why you need a workspace?

@naithar I added audio to the bevy ios example project on this branch https://github.com/MichaelHills/bevy/tree/mike-ios-example give that a try? Was your error on device or simulator?

naithar · 2020-10-18T15:30:10Z

Storing _stream: OutputStream, results in:

   --> src/lib.rs:182:19
    |
182 |     audio_output: Res<AudioOutput>,
    |                   ^^^^^^^^^^^^^^^^ `*mut ()` cannot be sent between threads safely

and other compile errors.

If I remove it there is no real difference between my previous testing implementation.

Crash occurs on my device, but the reason for it seems to be in cpal crate.

Clamping sample_rate value results in another crash:

2020-10-18 18:28:53.325702+0300 BevyTest[14563:4796145] fopen failed for data file: errno = 2 (No such file or directory)
2020-10-18 18:28:53.326061+0300 BevyTest[14563:4796145] Errors found! Invalidating cache...
thread 'Compute Task Pool (0)' panicked at 'Resource does not exist bevy_audio::audio_output::AudioOutput', bevy/crates/bevy_ecs/src/resource/resources.rs:289:32
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'Compute Task Pool (1)' panicked at 'task has failed', .cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
thread '<unnamed>' panicked at 'task has failed', .cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45

Edit:

After tweaking bevy some more (switching back to previous play_queued_audio_system and creating OutputStreamHandle for each Sink) I've got this crash:

thread 'Compute Task Pool (1)' panicked at 'called `Result::unwrap()` on an `Err` value: UnrecognizedFormat', bevy/crates/bevy_audio/src/audio_source.rs:44:56
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'Compute Task Pool (0)' panicked at 'task has failed', .cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
thread '<unnamed>' panicked at 'task has failed', .cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
thread '<unnamed>' panicked at 'task thread panicked while executing: Any', bevy/crates/bevy_tasks/src/task_pool.rs:75:18

MichaelHills · 2020-10-19T12:53:09Z

@naithar are you using my updated bevy ios example or your own? https://github.com/MichaelHills/bevy/tree/mike-ios-example

One issue might be that I've hard-coded the sample rate to 44100 in my cpal PR. I assumed that it should generally work everywhere. Possibly I'm wrong on that, or maybe the default on your device is something other than 44100 and that could be an issue.

If we do need a sample rate other than 44100 to get this working, I might need to pull in some AVAudioSession method calls. Anyway one I thing I can try is adjust the min/max range to 8000/48000 and see if that helps. Are you able to print what sample rate its trying to use?

naithar · 2020-10-19T13:27:28Z

@MichaelHills your example resulted in first part of the errors

Storing _stream: OutputStream, results in:

   --> src/lib.rs:182:19
    |
182 |     audio_output: Res<AudioOutput>,
    |                   ^^^^^^^^^^^^^^^^ `*mut ()` cannot be sent between threads safely

and other compile errors.

If I remove it there is no real difference between my previous testing implementation.

Crash occurs on my device, but the reason for it seems to be in cpal crate.

Clamping sample_rate value results in another crash:

2020-10-18 18:28:53.325702+0300 BevyTest[14563:4796145] fopen failed for data file: errno = 2 (No such file or directory)
2020-10-18 18:28:53.326061+0300 BevyTest[14563:4796145] Errors found! Invalidating cache...
thread 'Compute Task Pool (0)' panicked at 'Resource does not exist bevy_audio::audio_output::AudioOutput', bevy/crates/bevy_ecs/src/resource/resources.rs:289:32
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'Compute Task Pool (1)' panicked at 'task has failed', .cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
thread '<unnamed>' panicked at 'task has failed', .cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45

The last log with on an Err value: UnrecognizedFormat' was for my own change, in which I didn't switch to thread local resource. It would probably reproduce with your changes too.

Edit:

Sample rate log: wants: SampleRate(44100), min: SampleRate(48000), max: SampleRate(48000)

Edit:

Crash still happens even after rebasing to newest bevy.

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: UnrecognizedFormat', bevy/crates/bevy_audio/src/audio_source.rs:47:56
stack backtrace:
   0:        0x101f463fc - <unknown>
   1:        0x101fab274 - <unknown>

naithar · 2020-10-19T14:07:24Z

~~Also crashes on macOS.~~
~~I'm testing something right now, so it could be my fault.~~

The crash with UnrecognizedFormat was my fault, so now it's not crashing. But there is no sound atm.

naithar · 2020-10-19T15:14:12Z

Ok, so I've managed to make it work here: https://github.com/naithar/bevy/tree/feature/rodio-0.12, plus tweaked cpal to clamp sample_rate value.
There is sound on both macOS and iOS. But on iOS it crashes on any touch input. Switching back from init_thread_local_resource to init_resource also crashes on input and there is no sound. =/
On macOS there is no crash on input.

And since bevy has it's own PR for rodio-0.12: bevyengine/bevy#692 it's possible that it could bring this regression.

Crash log:

thread 'Compute Task Pool (0)' panicked at 'called `Option::unwrap()` on a `None` value', bevy/crates/bevy_input/src/touch.rs:129:46
stack backtrace:
   0:        0x102cce714 - <unknown>
   1:        0x102d3358c - <unknown>
   2:        0x102cc0490 - <unknown>
thread 'Compute Task Pool (1)' panicked at 'task has failed', .cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
stack backtrace:
   0:        0x102cce714 - <unknown>
   1:        0x102d3358c - <unknown>
   2:        0x102cc0490 - <unknown>
thread '<unnamed>' panicked at 'task has failed', .cargo/registry/src/github.com-1ecc6299db9ec823/async-task-4.0.2/src/task.rs:368:45
stack backtrace:
   0:        0x102cce714 - <unknown>
   1:        0x102d3358c - <unknown>
   2:        0x102cc0490 - <unknown>
thread '<unnamed>' panicked at 'task thread panicked while executing: Any', bevy/crates/bevy_tasks/src/task_pool.rs:75:18
stack backtrace:
   0:        0x102cce714 - <unknown>
   1:        0x102d3358c - <unknown>

MichaelHills · 2020-10-20T13:04:07Z

@naithar Do you know where the 48000 sample rate is coming from? Is that the default for your device? What phone is it? I hard-coded the cpal branch to 44100 as that seemed to be the default on my iPhone 6 and simulator. I will have an iPhone 11 soon :) and I have a couple of iPads can do more testing on them.

Probably I should just set min/max to 8000/48000, and for default sample rate just query it off the audio unit default devices. I think to query the min and max I'd need to set it with AVAudioSession and check if it succeeded. I can also just query the default and set min/max to be that default, so 48000 for your device. I wonder how rodio/bevy handles different sample rates though. If it's seamless then no big deal.

Touch crashing is a bit weird, surely that's unrelated? Now that your touch PR has landed, I can update my ios-example branch to include touch and we'll have an all-in-one iOS Bevy example. Once audio works smoothly on all our devices then we're done!

naithar · 2020-10-20T13:12:46Z

@MichaelHills no, no idea where 48000 comes from. My device is iPhone XS, but I'm not sure if it'll help.

Touch crashing is a bit weird, surely that's unrelated? Now that your touch PR has landed, I can update my ios-example branch to include touch and we'll have an all-in-one iOS Bevy example. Once audio works smoothly on all our devices then we're done!

Well, touch also crashes if there is no sound to play. But without bevy-audio everything becomes fine. So I'm unsure if it's touch implementation or audio issue.

MichaelHills · 2020-10-20T13:24:03Z

Ohh so apparently new iphones default to 48KHz based on this comment https://forum.audiob.us/discussion/comment/762478/#Comment_762478

I just checked my cpal branch does set min/max to the device default and 44100 is NOT hard-coded (it used to be earlier on), so that explains why your min/max is 48000. Something "wants" it to be 44100. That must be Bevy or Rodio? Sounds like I need to pull in some AVAudioSession... doh was hoping to avoid that. :)

@simlay I noticed in that in coreaudio-sys, when objc generation is enabled, something changes with the constants/enum generation. Your earlier code had to work around some output differents between macOS (no objc) and iOS (yes objc). Do you know what that's about? Is that a bug in bindgen? Also, adding in bindgen for AVFoundation adds a HUGE amount of code and a really long compile time. Do you have any advice on how to just pull in a subset of AVFoundation such as just AVAudioSession and related code? Would it be bad practice to handwrite some bindings rather than generate it? Winit seems to hand-write a bunch of UIKit bindings (UIWIndow, CGRect, etc.)

MichaelHills · 2020-10-24T13:47:16Z

@naithar It was actually my cpal branch defaulting to 44.1KHz. I've pushed an update where it sets the default to match that of your device. I think my ipad uses 44.1 though, might be a couple of weeks before my iPhone 11 arrives which should default to 48KHz.

naithar · 2020-10-24T22:28:33Z

Great :)
Touch input issue is still there, but I'm not really sure where the problem is.

MichaelHills · 2020-11-09T14:11:30Z

@naithar I'm going to assume that the touch issue in Bevy is not related to audio support, and if it is, we can address it in a follow up? If you can post me a Bevy repo/branch I can try it out.

@simlay I still don't understand the whole workspace thing and why you need the change but I seemingly don't. In the original change you tested, I had forgotten to git add my Cargo.toml for the ios sample project. Are you able to retest or point me to some documentation that explains the issue? I can add the workspace thing to this PR if needed but I'm not sure how to test it myself.

If all looks good then I'll do a final round of testing.

naithar · 2020-11-11T10:18:29Z

@MichaelHills yeah, I think we can ignore this issue for now. bevy already has an updated audio system, so problem could no longer exist. I can't test it right now though.

…ations

…ces use 44.1 new devices use 48)

MichaelHills · 2020-11-27T13:40:01Z

In this PR I've tested macOS and the iOS example on multiple devices + simulator (see PR description). I think it's fine to merge as is. Testing from others using Bevy, audio output works on old devices and new once I sorted out the issue with new devices using 48KHz and old devices using 44.1KHz.

MichaelHills mentioned this pull request Sep 27, 2020

Added initial changes for iOS support. #69

Closed

MichaelHills force-pushed the mike-ios-support branch 4 times, most recently from a65e643 to cf875a7 Compare September 30, 2020 14:41

MichaelHills commented Sep 30, 2020

View reviewed changes

MichaelHills force-pushed the mike-ios-support branch from cf875a7 to 489d6c3 Compare September 30, 2020 14:59

MichaelHills marked this pull request as ready for review September 30, 2020 15:03

MichaelHills force-pushed the mike-ios-support branch 2 times, most recently from 1d8a55d to 5a92876 Compare September 30, 2020 15:09

simlay reviewed Sep 30, 2020

View reviewed changes

MichaelHills force-pushed the mike-ios-support branch 2 times, most recently from b4721fd to f1a26d0 Compare October 2, 2020 15:20

MichaelHills changed the title ~~Fix iOS compilation error~~ Fix iOS compilation error and add macOS + iOS feedback example applications Oct 2, 2020

MichaelHills force-pushed the mike-ios-support branch from f1a26d0 to 4812b0c Compare October 2, 2020 15:51

This was referenced Oct 11, 2020

Refactor coreaudio host and add iOS support + ios-feedback example RustAudio/cpal#485

Merged

iOS Support bevyengine/bevy#87

Closed

Fixes iOS compiling. RustAudio/coreaudio-sys#33

Merged

MichaelHills force-pushed the mike-ios-support branch from 4812b0c to a18f7da Compare October 17, 2020 13:09

MichaelHills force-pushed the mike-ios-support branch from a18f7da to cafb94e Compare October 18, 2020 12:39

simlay self-requested a review November 9, 2020 04:54

MichaelHills added 3 commits November 27, 2020 23:57

Fix iOS compilation error and add macOS + iOS feedback example applic…

66bbb0a

…ations

Call AudioComponentInstanceDispose in AudioUnit Drop

1bb6b00

Tweak iOS example to use sample rate from the output device (old devi…

741099d

…ces use 44.1 new devices use 48)

MichaelHills force-pushed the mike-ios-support branch from cf21764 to 741099d Compare November 27, 2020 12:58

MichaelHills merged commit a843cfd into RustAudio:master Nov 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix iOS compilation error and add macOS + iOS feedback example applications #72

Fix iOS compilation error and add macOS + iOS feedback example applications #72

MichaelHills commented Sep 27, 2020 •

edited

MichaelHills Sep 30, 2020 •

edited

MichaelHills Sep 30, 2020 •

edited

MichaelHills Sep 30, 2020

MichaelHills commented Sep 30, 2020

simlay Sep 30, 2020

MichaelHills Oct 2, 2020

MichaelHills commented Oct 2, 2020

MichaelHills commented Oct 15, 2020

naithar commented Oct 15, 2020

MichaelHills commented Oct 17, 2020

simlay commented Oct 17, 2020

naithar commented Oct 17, 2020

MichaelHills commented Oct 18, 2020

MichaelHills commented Oct 18, 2020

naithar commented Oct 18, 2020 •

edited

MichaelHills commented Oct 19, 2020

naithar commented Oct 19, 2020 •

edited

naithar commented Oct 19, 2020 •

edited

naithar commented Oct 19, 2020 •

edited

MichaelHills commented Oct 20, 2020

naithar commented Oct 20, 2020

MichaelHills commented Oct 20, 2020

MichaelHills commented Oct 24, 2020

naithar commented Oct 24, 2020

MichaelHills commented Nov 9, 2020

naithar commented Nov 11, 2020 •

edited

MichaelHills commented Nov 27, 2020

Fix iOS compilation error and add macOS + iOS feedback example applications #72

Fix iOS compilation error and add macOS + iOS feedback example applications #72

Conversation

MichaelHills commented Sep 27, 2020 • edited

MichaelHills Sep 30, 2020 • edited

Choose a reason for hiding this comment

MichaelHills Sep 30, 2020 • edited

Choose a reason for hiding this comment

MichaelHills Sep 30, 2020

Choose a reason for hiding this comment

MichaelHills commented Sep 30, 2020

simlay Sep 30, 2020

Choose a reason for hiding this comment

MichaelHills Oct 2, 2020

Choose a reason for hiding this comment

MichaelHills commented Oct 2, 2020

MichaelHills commented Oct 15, 2020

naithar commented Oct 15, 2020

MichaelHills commented Oct 17, 2020

simlay commented Oct 17, 2020

naithar commented Oct 17, 2020

MichaelHills commented Oct 18, 2020

MichaelHills commented Oct 18, 2020

naithar commented Oct 18, 2020 • edited

MichaelHills commented Oct 19, 2020

naithar commented Oct 19, 2020 • edited

naithar commented Oct 19, 2020 • edited

naithar commented Oct 19, 2020 • edited

MichaelHills commented Oct 20, 2020

naithar commented Oct 20, 2020

MichaelHills commented Oct 20, 2020

MichaelHills commented Oct 24, 2020

naithar commented Oct 24, 2020

MichaelHills commented Nov 9, 2020

naithar commented Nov 11, 2020 • edited

MichaelHills commented Nov 27, 2020

MichaelHills commented Sep 27, 2020 •

edited

MichaelHills Sep 30, 2020 •

edited

MichaelHills Sep 30, 2020 •

edited

naithar commented Oct 18, 2020 •

edited

naithar commented Oct 19, 2020 •

edited

naithar commented Oct 19, 2020 •

edited

naithar commented Oct 19, 2020 •

edited

naithar commented Nov 11, 2020 •

edited