[FEATURE BOUNTY] Continuous Cursor Support - DDR/Guitar Hero Style Cursor

[loganknecht]

Quick Disclaimer

Hey everyone. I need to open an issue here in order to create a bounty at https://bountysource.com/

Feel free to disregard this, but I also think there is interest in this from the community as well.

Explicit Acceptance Criteria

• Required Features

  • Cursor starts on first note
  • Pick up bars are supported
  • Repeat bars are supported
  • Cursor must navigate with respect to measures for new lines
    • When a new line is encountered the cursor continues to the end of the measure (the right side)
    • When a new line is encountered, on the new line the cursor will start at the measure's left side
  • Cursor MUST be able to be "queued" up AKA staged on first note and "ready" to play
  • Cursor MUST be able to be "played" as in it starts and begins to navigate
  • Cursor MUST be able to be "paused" as in it stops moving
  • Cursor MUST be able to be "resumed" as in after it stops moving it can begin to move again
  • Cursor finishes at the final measure's right side
  • MUST NOT break current cursor feature for existing OSMD users
  • MUST be able to create event handler for upcoming note(s) that cursor will encounter
    • MUST be able to interrogate before or after cursor encounters note(s)
    • This is/will be used to determine if users are playing notes before or on the event the cursor scrolls over a note

• Stretch Goals

  • Works with OSMD playback feature as well

• Technical Implementation Requirements

  • Do not duplicate code already created in the src/OpenSheetMusicDisplay/Cursor.ts class
  • Either subclass the Cursor class in to two new classes (discrete/continuous) or create a CursorType that when set determines transition logic
    • Advised to separate cursor logic into separate .ts files
  • Cursor position MUST be able to be determined by time in the song
    • Song time is determined by tempo and total measures
      • Don't forget pick-up measures
      • Don't forget repeat measures
    • Song time should be normalized between 0 and 1 for ease, but not required
    • If cursor position MUST be able to be set by time, but if less then 0, it should be placed at first note
    • If cursor position MUST be able to be set by time, but if greater than 1, it should be placed at final measure

Introduction

I have built this feature out on Piano Gym. You can watch this Example Video to see. But my solution isn't very robust, and additionally I would prefer to be able to rely on OSMD for the implementation as I believe there are oversights in my solution for things like repeat bars, pick up measures, etc.

This is something I would love to help implement for the library, as I think a continuous scrolling behaviour would be the ideal implementation for many developers.

I am more than happy to share my implementation if we could discuss some design considerations and make sure that the result ends up as a merge request to the code base for everyone to use.

Implementation Design Discussion

First, I would make it a point to allow OSMD to have two different CursorModes or "types"

• Discrete
• Continuous

This way if we want to modify the cursor in the future we can add a different cursor type and be able to refine or extend new functionality.

Second, when placing the cursor position I believe it should be time based. The whole point of tracking the cursor position is that its position is derived based on its percent through the song, with respect to tempo. Where the starting position of the cursor at 0 is the left most part of measure 0, and the ending position of the cursor is at 1 which is the right most part of the final measure.

Right now, I believe the position of the existing cursor is calculated based on a reference to the next note? Which isn't super helpful, because the cursor shouldn't be directly associate with notes, but rather position.

So you will need to make it support a calculation of where to place the cursor based on tempo and position it based on the song's elapsed time

Steps to calculate it look something like this

1. Based on tempo, measure count, pick up bars, repeat bars, etc determine song length
2. Pick a time between 0 and 1 of that song length where 0 and 1 are normalized values of the song total time with respect to tempo
3. Place the cursor in the correct normalized position based on measures/notes

Additionally, what I believe would be preferred UI/UX is that when the cursor is moving across a measure, the ENTIRE measure should be treated as the area to draw over. Each measure represents a specific subsection of the song, and the cursor should at the start of time for the measure begin from the left-most side and at the end of the measure "scroll" off of the right side

Lastly, and this is the big one - the current cursor has a behaviour that will let developers interrogate notes as it scrolls across them. This feature MUST be supported because I believe there are many developers out here trying to match pitch at the time of playing with the note being scrolled over. So it's important to build that support in.

UI Gotchas

One of the pieces that's not intuitive until you approach this problem is that because OSMD renders with respect to the view port of the device/browser there's no guarantee on spacing of notes and other elements. Because of this, it means that as the cursor progresses it isn't guaranteed to have uniform speed if it's trying to "land" on each note as it occurs in the song. For example some notes may be drawn as a whole note, but for the cursor to arrive at the following after a whole note may look really slow as it's drawn as one note, but holds many counts.

What I'm trying to say here is that it will be incredibly difficult for the cursor to be drawn at the same movement speed if you're doing a continuous scrolling approach that has the cursors traveling over each respective note as it's played. This is by nature a side-effect of the rendering output of OSMD and it must be accepted, unless there is a solution I have not considered.

Stretch Goals

A lot of people want to build out functionality such that there's actually a range of time before the note that is eligible to be interrogated.

Think of this like Dance Dance Revolution/Guitar Hero. When the cursor scrolls over the note that's one thing, but generally speaking there needs to be some arbitrary range that's determined before hand before a note is encountered. Think of it as a window for allowed play/introspection.

If you develop out this feature, you'd need to figure out a way to build in hooks that allow people to say I want to run a function in this window of time before this note scrolled over, that way they can hook into it to "listen" for the note being played ahead of time.

The implementation that we end up with should allow for the ability to run a function that lets developers perform actions as the cursor approaches every note. The key use case frequently requested is something like grading a pitch as the cursor approaches the note.

Some of the tricky parts for this happen with fast time signatures/tempos and very short note durations like 16th/32nd notes of the same pitch being played, etc. How does that default window get determined?

One issue, for example, is let's say you repeat a C4 as 16th notes over a single measure. Typically you can assign an arbitrary time range before hand to check, but if that time range is too big, then they will overlap while checking on the note before and after the current note. And this will happen for the ENTIRE measure because the accepted range for evaluating note grading is overlapping every time..

Anyway, I'm rambling, but this is something I've had to grapple with while building out this behaviour and it's something that I'm interested to see how others have solved it. So be aware that timing for the allowed window is also another tricky part.

Reference Code

Anyway, with all the above mentioned as design concerns/requests - here is how I'm doing it on Piano Gym

Code sample is too long - see public Gist code here
https://gist.github.com/loganknecht/c374f5990e35b6ca636b696676325683

Wrap-Up

I'm super open to collaborating on this, as once again, I would love to build this into OSMD so I can then leverage all the default functionality of the library instead of building this out myself. Additionally as I mentioned there are oversights in my implementation that I haven't accounted for yet and I'm hoping through the power of team work we might be able to get them fixed. ❤️

Let me know and if you choose to build this out for the OSMD library, because I would love to assist.

[loganknecht]

Hello everyone 👋

I've created the bounty here
https://app.bountysource.com/issues/113117154-continuous-cursor-support-ddr-guitar-hero-style-cursor

At the moment the best I can do is contribute a starting amount of $100

Anyone else interested in supporting this is more than welcome to contribute to this - as it will hopefully incentivize others to participate.

@OSMDTeam - I wish I could afford to participate in your higher level sponsorship to get this feature built, however I regrettably do not have the means to do so. BUT, if you're interested I am more than happy to pair with someone on your team to make the implementation of this feature go much faster.