Major upcoming functionality for mystjs
#838
Comments
|
Thanks @rowanc1 for sharing these ideas with the rest of the team. I'm also excited about seeing how far we can push the envelope on the JS side! Agreed that a blog post is a good idea so that we can signal the intentions of the project in an intentional way. Perhaps that blog post should be closing action for this issue? It also makes me think that we we should formalize the strategy and direction of the EBP organization, so that we can signal to external stakeholders and align our internal team on the direction in which we'd like to head. I've created an issue to track that here: I also want to share two quick clarifications:
|
|
As the foundations are being laid down, I'd like to ask about the choice of Remix.run as runtime engine. I think it is a good time to introduce that question and I understand you are extracting/refactoring the existing code base to be moved to executablebooks/myst repositories. There are other runtime engines I'd like to consider to render myst documents, like next.js, docusaurus. Is it possible to already factor the code so that the default current implementation is remix.run, while keeping the door open to other runtime engines in the future? I can guess this is not an easy architecture setup as remix.run may be used a lot to do the hard work, so just asking now. |
|
Thanks for the feedback so far. A few of my responses below. :)
Thanks @choldgraf, yes, I think that makes a lot of sense!
Love it, I am excited to contribute or review there, and help out where I can. There is probably some pre-work from the PIs of the project to do there, which I am sure some of the vision can come from the original grant proposal and be edited with the updated context. One of the goals that we did set together at the team meeting was to make forward, collaborative progress on the MEP process over October, with the suggestion of three documents being created for review next month. This can hopefully dove-tail in with that work.
We do of course do try to be inclusive, and this is part of us trying harder to evolve the project in that direction! However as @choldgraf you point out, there is a big difference between a project run by a single stakeholder/company where most communication is internally focused and a community driven/governed project. We are aware of those differences (and are also getting coaching/pointers from @choldgraf on this!) and I am excited to get this into a place where the community can evolve it together. Thanks @choldgraf for calling out the stage we are at, and to your next 👇 point, we should do a better job of documenting where we are at in the current docs.
I will respond and address some of the concerns in updating the docs today to have appropriate advisory content, and respond on the issue you made.
Yes of course, and sorry if it came across that way. My main intention on calling those folks out was that they are not represented in any of the issues or commits, and that a lot of the efforts going into this are not captured by github contribution graphs. Certainly there are many technical contributors, as well as users finding bugs or asking for support that have helped shape and evolve the project thus far. Looking forward to many more contributions in the future as well. 🚀
Glad to see you looking into the code, thanks. :) I am loving working with Remix at the moment and it is a good fit for the style of websites like jupyterbook/docs-sites/scientific papers etc. The current way things are built is tied to remix to closely, and as you mention, there are a lot of other themes and frameworks (nextjs, vue, static-html, docusaurus, existing sphinx-themes...). Removing the "assumptions for the default Curvenote theme" is exactly this piece of work -- removing remix.run and treating it only as a templating layer that you can choose (or not) to use. I think that we will get to a good starting place when we move things over to the @exectuablebooks community, but it certainly is only a starting place, and having other themes/frameworks to target, and ideally get contribution on how to shape that would be ideal. I am looking forward to working with you.
I am hoping we can take a similar approach to how we have put effort into the templating of latex documents, where there aren't too many assumptions by mystjs tooling, and there is the possibility of large diversity of community approaches and contributions. From a technical perspective, this means splitting out the JSON AST creation, and providing hooks to trigger the templating libraries. We are just working through some of the details on this now and I will both tag you on some of our experiments over in the curvenote repo this month if you have feedback, as well as open an issue in mystjs to have more discussion. The other package where there is something to do is in |
|
Super quick feedback after reading your reply, I would not put the Taking back the list you have put in #793 (comment) This would allow to setup the language, as the scope/target. e.g. woudl the JSON AST be feeded to RemixRender or NextRenderer... (change the names as you want). In terms of scope, we know React.js is not the only framework, but it sounds reasonable to target that only in first instance, while letting the door open for later technology like Vue.js (this is a common pattern in some projects). Same for Remix which could not be the only React technology to cover. In terms of |
Yes, I think so. This is very much in the JAM Stack style (javascript, apis, markdown), with the interface being between the
I will pull something together for discussion of where we are at now, and welcome feedback to improve and get on the same page about terminology. There is a diagram for the latex templating, which sort of illustrates this (maybe helpful?). Regardless, @fwkoch and I will pull some docs together over the coming weeks.
This was my thinking as well, and is a broad place to start that gives significant differences to the things that are possible with JupyterBook today. I am not sure what the next one would be, and it might make sense to actually focus more on the static HTML templating with jinja even to gain community trust/understanding/buy in -- react/remix/nextjs are pretty far from how things are done today (mostly bootstrap, jinja, jquery), so that is also on my mind a lot lately.
Yep totally, the Curvenote theme/renderer is using headless-ui, remix and tailwind. Many, many choices out there for high-quality UI frameworks (currently impressed by radix, for example!). I think there are going to be a lot of opinions and options in this space, and MyST / EBP can add value by providing well documented and structured data that can easily be rendered (however complex that rendering process might also be!). If you go to a page in the docs, you can add |
|
Heya, firstly this looks great, theres some really awesome stuff here, but...
I feel if this is a blog post emitting from EBP, there should be a very clear message of what this means for JupyterBook. For example, should users be expecting to switch from using the Honestly, for myself as a developer, I would like to know. Because at the moment I'm unclear if I should dedicate much more time to working on/maintaining the "Python side" of EBP, if it will eventually just become deprecated. You say
I would say
There is certainly overlap, but non-technical users should be given one clear way of doing things. Perhaps, there are clear use cases for using one or the other? Then they should be made clear. There are three things that come to mind that I don't see implemented:
Again, it comes back to the clarity of messaging to users and developers: This thinking also obviously feeds into #839 |
|
(also another feature that I just thought of, that sphinx/jupyterbook offers but does not appear here, is multilingual support) |
Thoughts from Chris H on how these two tools relate to each otherI'd like to help disambiguate some ambiguity I see above about the direction of the Python and JavaScript aspects of this stack. I don't think that it is helpful at this moment to compare them feature-for-feature, they are at very different stages of development and they also focus on different use-cases. There is no intention to deprecate or move on from the Python / Sphinx stack in the short-term. Rather, I see investing time in the JavaScript implementation of MyST as a way to explore what is possible with a different implementation and technical stack, and to understand what implications this has for functionality that is important to the mission of the project, as well as for maintainability and sustainability of the project. Choosing a stack comes with a set of limitations, and we should explore what are the inherent limitations of Sphinx and what is easier to do with a different technical foundation. In the short / medium term, I believe that Jupyter Book and MyST JS focus on two related but distinct use-cases. Here's a short overview from my perspective. Jupyter Book focuses on multi-page books or documentation. It is more "informal" in nature, and is useful for people that do not need to abide by a particularly strict protocol for the structure of their outputs. MyST JS focuses on single-document articles with more formal, semantic, structured outputs. This is more important for scholarly publishing and also makes it easier to render outputs in many different forms (like many different Latex template outputs). It also explores what it looks like to build a document engine with a more modern technical stack. In the short term, I think we should develop both of these efforts simultaneously, using the MyST Specification as a unifying force to ensure they retain overlap in their core functionality and syntax, but allowing each to explore their respective focus-area independently. We should use project resources to support both of these efforts, though in the short term MyST JS is significantly less-developed and I think it makes sense to dedicated resources there to bring it up to speed. In the medium- and long-term, it could be that MyST JS grows to have a kind of "feature parity" with Jupyter Book's Python implementation, or it could be that the ideas in MyST JS make their way into the Python side as well. If that is the case, then the team should discuss and decide whether it is in the long-term strategic interests of the project to just "choose" one of them, or to keep growing them independently. But I think that day is a ways off as we still have much to learn and to try. In short: I think it's important that we both invest time and resources into our core infrastructure, but also that we keep our eye on the horizon to understand what future technologies will position us for the most impact. Thus far we have focused most project efforts on the Python implementation of MyST and Jupyter Book, and @rowanc1's proposal above represents another incremental push now in the direction of the JavaScript stack and a different use-case. Both are sticking around for the foreseeable future, and we should continue to assess the strategic value of each for the project as we learn more. |
|
Thanks for your thoughts @choldgraf
Yep sounds good, and indeed perhaps something to this effect should be mentioned in any blog post and possibly on https://www.myst.tools/
I really feel that it is. I would note there has already been discussions around these kinds of thoughts in e.g. executablebooks/mystmd#34
Here I would link to #837 (comment), for my analysis of the sphinx design. I hope that makes sense |
|
Thanks for summarizing @choldgraf and for the questions/comments @chrisjsewell. My summary: If you are using JupyterBook or Sphinx and it is fitting your current needs (often courses, software docs, or online tutorials), JupyterBook is and will be a better fit for the foreseeable future (perhaps forever). I do hope that you especially, @chrisjsewell, continue improving the sphinx/myst-nb ecosystems, as there is important work to be done there! The Curvenote team has mostly been focused on scientific writing over the last four years, which ends up these days as PDF documents submitted to journals or a thesis or a word document. There are some folks in the Jupyter ecosystem that are trying to create beautiful PDFs, LaTeX, and interactive websites from notebooks. I think that mystjs over the coming months should be considered here, and that the EBPs involvement with MyST can help also create other alternatives to PDFs for journals, lab-groups and other societies. The project is still relatively young (~1.5 years), and is of course missing many features. The main extensibility is around PDF and LaTeX documents (via I think that there is also some confusion around the names of the repositories/packages, and scope (e.g. mystjs is the repository and also a rather small package that is scoped to parsing). As discussed above, my current plan is to document where things are today and get some diagrams on responsibilities/overlaps/data-flow of the packages, and that would clear things up, potentially illuminate some clarifying changes, and could be a place to discuss any improvements to architecture going forward. |
I am curious to know if a decision is already taken regarding |
|
Hi @echarles -- our current plan is to transition to unified over time. Currently all of the AST manipulation after the first, basic parse is completed with unified. There are a lot of things out there (e.g. vscode and jupyterlab-myst) that directly rely on the markdown-it parsing, which makes the full transition a bit complex. The unified parser is also still in development, whereas there is a lot of testing / use of the markdown-it one already. |
|
Thx @rowanc1 for the useful information, this confirms my understanding. May I ask you a few more questions while I am trying to sketch out the future landscape:
|
|
Heya @echarles et al On the topic of parsers, I have some thoughts I’d like to share that I feel are somewhat programming language agnostic, and obviously came up in the original choice to port markdown-it-py for use by jupyter-book and other work. Firstly, as mentioned by @rowanc1, I feel we can make a clear delineation between
@rowanc1 has done a great job developing (2) here, based on an input MDAST, and I would suggest none of this necessarily should be tied to any one parser, although in practice obviously we will choose one. Next, I think it’s important to recognise that there is a s probably not one-size fits all for all use cases, principally the difference between editing and building cases such as:
For a certain use case there is then trade-offs:
Finally, there is the consideration of intermediate representation, i.e. what are you using to store information when going from source ➡️❔➡️ target Given these considerations, the reason to originally chose markdown-it (and to create markdown-it-py) was:
This is roughly the same consideration for mystjs, IF we are primarily considering a build tool. The key difference really between markdown-it and unified (well actually micromark, which is the core component of unified that parses source markdown) is the capture of more source position mapping (i.e. it creates a better concrete syntax tree). But this comes at the expense of more complexity, particularly in writing extensions. I would propose both are actually fine for project builds, but neither are particularly great for other use cases, like HTML previewing. For editor tools like HTML previewing, syntax highlighting, LSPs, unfortunately from what I have found there is no ideal solution currently out there. This leads me on to almost my final point; does the parser need to account for extensibility? If it doesn’t, this certainly makes things easier, since we can create a static, well-defined parser. Finally; blue sky thinking, if we really had no constraints on what myst could be. I would strongly suggest looking https://github.com/jgm/djot, which is the original Markdown creators answer to « what if I created Markdown now, with the benefit of hindsight » and contains some key breaks from Commonmark. Thoughts from any one else welcome! edit: also link to agoose77/jupyterlab-markup#12 (comment), with some super helpful chat between @agoose77, @bollwyvl and myself, informing my points here (and the choice to use MDAST) |
|
Just a quick update here! @fwkoch and I are planning to move some of the Curvenote work upstream today! There are a few changes to the original plan, as we didn’t get as far as we would have liked on the web-side, so the initial cut is much more focused on latex, PDF, and Word. The myst work is is already being used in combination with JupyterBook, which is really cool to see people using these tools to augment the existing python/JupyterBook/Sphinx ecosystem. One of the first things we will be working on post merge is on Microsoft Word templating, @fwkoch posted an issue in A few other points:
We haven't had the time to keep it updated, and the python version is currently working on Curvenote's apis, not myst's (we are still using it in our production systems). I think in the future we could go back and make improvements there to bring it inline, as I do think there is value in having a python version as well for the community. |
Over the last several months and at the last two team meetings @fwkoch, @stevejpurves and myself have been showing some of the new work on
mystjs, which you can see/demo/try here: https://js.myst.toolsThere is a ✨ton✨ of new functionality, including:
jtex list)Where we are today
A lot of the code for mystjs was created for the Curvenote CLI (MIT licensed), some background on that here, and currently, much of the code for mystjs is actually currently hosted in https://github.com/curvenote/curvenote. Over the past few months we have been working (primarily with @choldgraf, @fperez, @colliand and @mmcky) to upstream this functionality into
mystjs, this has taken the form of pulling out relevant functionality into non-curvenote-based tools that can be both open in the community and source-code sense.By end of October
This will create some major changes to the
mystjsrepository. Our plan is to move to a mono-repository (using turbo + npm workspaces), with the mystjs repo containing the following packages:citation-js-utilsutility functions to deal with citationsintersphinxa command line tool and package for reading and writing ojtexa templating library (see docs)mystjsthe current MyST parser, with extensibilitymyst-clithis will provide CLI functionality formyst build mystdoc.mdmyst-cli-utilssome shared utils between intersphinx, jtex, and myst-climyst-commonSome common utilities for working with ASTsmyst-frontmaterdefinitions and validation for scientific authorship/affiliation frontmatter (see docs)myst-templatestypes and validation for templates (including jtex, and some word templates coming soon!)myst-to-docxconvert myst documents to word docs!myst-to-reactcreate basic, ideally unthemed react components for content onlymyst-to-texconvert myst to latex, to be used in combination with jtex to create stand alone latex documentsmyst-transformsa number of transformations for use with myst AST to transform, e.g. links, citations, cross-references, admonitionssimple-validatorsvalidation utilities, that print all sorts of nice warningsAll of these are already packages that are available on npm, and if you want to contribute or poke around in the next month, we welcome PRs in the current repo!
Major pieces of work left
myst-to-reacttools are pretty close to being able to be reused across templates, but the web-server/theme needs to be split out more effectively. (nested under this is another task to improve the routing for content pages, which has too many assumptions built in and needs to be generalized).In Screenshots
Export of MyST markdown files to PDFs, with 422 journal templates in the new myst-templates org
Cross-references with content, which works across pages (powered by a myst-spec content API)
Interactive citations, figures, and table references.
MyST demo component, which can now download word docs as well completely in the browser. :)
Questions and Feedback?!
Some of the push for this is to have robust javascript tools that are accessible in the future by tools like Jupyter (e.g. extending jupyterlab-myst) and interactive scientific journals. In working through this there is certainly overlap with the existing tools in the JupyterBook/MyST ecosystem. We have been investing in standardizing MyST as a markup language and AST, to ensure compatibility, but I am sure this effort opens up questions/concerns/uncertainty/feedback. I think what probably would be appropriate would be a blog article announcing some of this work in October, and it would be helpful to have some feedback on what we should address or spend time thinking through before a wider announcement. I would love feedback!
I am super excited by some of the developments recently in the MyST/JupyterBook community, and am glad that we can contribute to this community. Thanks and looking forward to feedback/questions! :)
The text was updated successfully, but these errors were encountered: