Send and receive note and pedal events to/from connected devices via WebMIDI #112
Conversation
|
This is great! I've had a fun evening trying to set up virtual MIDI devices on my Arch system, and playing around with this. I confess I'm a bit out of my depth with MIDI sequencers and synthesizers and DAWs and so on, but I've been able to record some output from the Pianolatron (and transcribe it to musical notation!) using Rosegarden, and then play it back from a MIDI device into the Pianolatron, adjusting the audio options on-the-fly (but not the tempo, which I suppose makes sense). I do have a keyboard here with MIDI capabilities, but that's in my eldest's room and he's asleep, so I might give that a whirl over the weekend :) I have some thoughts about the functionality:
w/r/t the code, I'm very happy with the organization. My only comment would be that the I'll update if I can reproduce the crash, but I wanted to get this down now. |
broadwell
force-pushed
the
midi-in-out
branch
from
38e297a
to
b8673d0
Compare
Yeah, this is probably warranted. The notifications definitely can get rather chatty, and reducing them to a level that most users would consider truly unobtrusive seems like it would difficult to achieve and probably wouldn't be welcomed by the small % of users who actually want to take advantage of the Web MIDI features. So an enable/disable config button (default: disabled) is probably the best option.
Yes this is definitely borderline usefulness-wise, at the moment. As long as the user can both listen to and play the connected HW or SW MIDI keyboard devices that are receiving the Pianolatron's MIDI output, then this should cover the two main use cases the PIs have mentioned for which combining input from the app and a separate device would be useful, specifically 1) applying partial sustain pedaling to MIDI events coming from the Pianolatron, and 2) playing a duet on a separate keyboard with the music being produced by the app. In the case of 2), it might be somewhat desirable to be able to see the keys and pedals of the Pianolatron piano app responding to both the roll's and the human's input, but that's a pretty limited benefit. One situation in which accepting MIDI input in the app would be called for would be if the app provides the ability to record a MIDI performance, including any input that a user might provide via an external keyboard. I have a demo of this partially working via a customized version of midi-writer-js, but it's not clear how much more effort is needed to get it to 100% functionality, and I'm also not sure whether this is really desirable, given that many external MIDI devices also provide the ability to record MIDI performances, including any inputs they'd receive from the Pianolatron app. |
Agreed, see the new db17c89 commit. The only reason I didn't do this earlier was that I was under the mistaken impression that the |
broadwell
force-pushed
the
midi-in-out
branch
2 times, most recently
from
2672c81
to
959a3d0
Compare
There might be a rough edge or two here to be smoothed out later, but this is a solid first pass.
It turns out that hardware MIDI devices are even less happy about this than the stock ToneJS Piano code, and will enter an endless loop of pedal up/down actions if sent consecutive alternative pedal up/down events without any time between them -- as is done in the SamplePlayer to silence all held/sustained notes immediately upon Pause or Rewind. Fortunately the MIDI devices I've tested sound fine on Pause or Rewind when the pedal up/down trick is not used -- in this respect as in many others, they are better performers than the ToneJS Piano.
This allows notifications about the availability of external MIDI connections (or lack thereof), the majority of which are generated just as the app is initialized, to remain on screen long enough to be read. Presumably the purpose of the `clearNotification()` call in `resetApp()` is to clear any lingering error messages from previous rolls, so it's OK not to do it when the app first runs. Note that there's still a potential problem whereby the MIDI status notifications can pile up all at once so that it's still difficult to read any of them before they disappear, but this can be handled in a subsequent update.
There may be more elegant ways to do some of this. Destroying the <WebMidi/> component when it's not in use seems clean, although there is the minor issue of the `midi.onstatechange` handler sticking around. In any case, the checkbox renders some of the previous notifications unnecessary, and it kind of makes sense not to show the checkbox at all if Web MIDI isn't supported in the browser -- though I kind of liked the "too bad your browser can't do this" popup in the previous commits.
…of-bounds note_on velocities
…to WebMIDI devices
…ionality." This reverts commit 02653fa.
This exposes the **actual** issue with the `<Notification/>` component, which is that it's a singleton and is being clobbered. Will fix in due course.
This isn't what it should look like, where it should go, or even how it should behave (entirely). Other than that, it's a good commit... :)
It shouldn't be the job of the `<WebMidi/>` component to manage this. However, this does affect the way the `<Keyboard/>` component (the other client for these functions) works a little -- clicking the keys now removes the `depressed` class if it was previously applied (i.e. if the roll player state is in the middle of (a) note(s)). This is in line with the behaviour when using the keys on a WebMIDI-connected device. If this is considered undesirable (and I think I'd lean that way, if not perhaps all that hard), then both the on-screen keyboard and WebMIDI-connected devices could could be decoupled from `$activeNotes`, but I'd rather do that with a argument to the `startNote()` and `stopNote()` functions (similar to the `fromMidi` argument).
This is some opinionated stylistic refactoring. The main changes introduce more destructuring and switch statements (which are technically faster, but the point here is more about readability). I think I can make the case that all of these changes are in line with modern best practices, but I'm happy to entertain counter arguments :)
simonwiles
force-pushed
the
midi-in-out
branch
from
959a3d0
to
39c480d
Compare
|
Okay, this is now ready for your perusal and, hopefully, approval. I think this is probably how it should be done, tbh. It's complete and neat as it stands, but leaves space to add more fine-grained control and additional UI should it be desired. |
|
The MIDI panel looks great and is, I agree, the best way to handle the UI/notification features. I've noticed one bug so far, which is that if I have the MIDI panel open with WebMidi enabled (Chrom* only, obvs) and then select a different roll, the roll viewer goes blank and doesn't display any tiles or overlays, even though the new roll still plays. Also, the right-hand panel doesn't switch back to the basic settings, as it does when loading a different roll in any other scenario and with any other settings panel selected. So maybe this is an issue with how the MIDI settings panel is destroyed (or not) in this scenario? Also, I did once manage to get the player into a state of "runaway pedal" (when the sustain pedal goes haywire, flipping on/off continuously), which I've occasionally seen before with previous versions of the branch. But this only happened when I accidentally disconnected and reconnected the network cable during a roll playback on a connected MIDI piano, which caused degraded performance from the app all around. So it's probably just worth keeping an eye out for it. |
Destroying and re-mounting the right sidebar (in particular) when the roll is changed can cause problems if child components aren't destroyed properly due to outro transitions (see sveltejs/svelte#5268). This has caused a number of bugs with the `<RollViewer/>` component not being recreated properly. One option is to eschew all outro transitions in other parts of the app; but I think a better solution in this specific case is to avoid destroying and recreating this part of the component tree unless absolutely necessary.
Yeah, this is caused by sveltejs/svelte#5268 again, as you guessed. Removing the transitions on the MIDI device "items" (the |
I've noticed this for this first time recently too, but I can't work out how to reproduce it reliably yet. Presumably it's a feedback loop between the |
broadwell
force-pushed
the
midi-in-out
branch
2 times, most recently
from
8a6b9ef
to
a35c450
Compare
This is a little nicer when enabling/disabling WebMIDI.
Pedal events send to external MIDI devices can be re-emitted back to the app (under at least some circumstances), causing feedback loops. With this commit, external pedal events are ignored unless explicitly enabled, in which case "app-side" pedal events are disabled instead.
The Web MIDI functionality, when available (currently only on Chrome/Edge including Android, though for some others there's this sketchy polyfill/shim) is pretty tightly coupled to the
SamplePlayer's operations, but it's tidier to have a distinctWebMidicomponent that can persist for the lifetime of the app.*It seemed preferable to avoid needing an actual MIDI connections config menu, so to this end the component connects automatically to all available MIDI devices, listening to and sending messages to all of them. It thus was necessary to complicate the note and pedal on/off logic slightly so that the app would be a good citizen of the MIDI device network and not trigger any MIDI through feedback loops. The component also makes what I think is pretty successful use of the nifty
Notificationfunctionality to keep the user updated about MIDI device comings and goings (and also the overall availability of Web MIDI or lack thereof from the browser) in a fairly unobtrusive way.Note that this functionality can be tested without needing any special MIDI hardware; on Mac OS, the combination of the Inter-Application Communication (IAC) driver and MidiKeys works quite well. This page lists similar combos for other OSes.
* It also would be quite straightforward for this component to make a downloadable MIDI "recording" of the note and pedal events sent to it, even if the browser doesn't support actual Web MIDI device interactions.