Support for interacting with Lua plugins

[weilbith]

Hey 👋🏾

First of all, thank you very much for all the effort and time you put into this project. I really appreciate it and it gave the necessary motivation to start working on NeoVim plugins again.

I'm now in need to make some calls to functions that are globally defined in Lua by NeoVim. Means I want to access vim.lsp, vim.diagnostic, ... Unfortunately I don't see how this should be possible right now. But maybe I'm blind. 🙈
I took a look into your noib3/nvim-compleet project to checkout how you do it there. But it seems like this includes a huge amount of boilerplate code with a bridge etc. pp. completely independent of this library here. I'm not sure if there is any better way to do it, because I honestly have no clue about this topic. But in any case, do you plan to integrate this feature into nvim-oxi too? It looks like you did already all the work for it in the other project, so it would be just a matter of putting it here too and wrap it nicely. Am I wrong?

Looking forward to hear what you think. 🙃

[noib3]

Hi, you're not blind, it's currently not possible to access the vim.lsp or vim.diagnostic tables, but they are on my todo list.

The whole bridge machinery I'm using in nvim-compleet is orthogonal to this problem and won't be necessary here. I guess nvim_oxi::diagnostic could already be implemented without many issues, whereas for nvim_oxi::lsp I'd like to have implemented #38 to expose async functions instead of callbacks like in the lua api (:h lsp-handler).

[weilbith]

Thanks for your fast response. 🙏🏾
That sounds like good news. What is the best way to stay updated of such changes? Do you do releases with a changelog or should I subscribe to PRs?

Might be related:
In case I want to access not a global object, but anything else available in NeoVims Lua environment, how would I do that? So for example if I want to access coc.nvim its functions. Is this technically possible too?

[noib3]

What is the best way to stay updated of such changes? Do you do releases with a changelog or should I subscribe to PRs?

The project follows semantic versioning and there's a CHANGELOG.

So for example if I want to access coc.nvim its functions. Is this technically possible too?

Technically speaking there's really no difference between e.g. vim.lsp and any other global Lua object exposed by other plugins.

However the vim.* tables are built in the editor and they provide a stable API, so I'm much more inclined to have first class support for those. Supporting other arbitrary plugins from the community would become unmaintainable pretty quickly. If you really need to interact often with other Lua plugins which may or may not be available at runtime, a general purpose Rust-Lua crate like mlua could be a better option.

[weilbith]

Yeah, I wouldn't expect any support for arbitrary plugins. Just an easy way to access them. But maybe I need to learn how mlua works and then I do it like that. If you have an example somewhere I would be glad to be pointed to it.

[ModProg]

Maybe there could be a way to have procmacro you can use to define an api of a third party macro easily, I did build some here: https://github.com/ModProg/nvim-rust/blob/6edc2435c11e94699efdd574efd981dace69fc33/src/api.rs#L14 and if you would accept something like this in nvim-oxi, this could be exported and made available to a user that quickly wants to create an interface to another plugin where they just need to provide the necessary globals and functions+parameters.

[weilbith]

I'm sorry to abuse this issue, but could someone point me to how I could use nvim-oxi (with or without additional mlua) to "require" other modules of the NeoVim Lua environment from my Rust code? I'm totally fine to write my own interfaces for the plugins I intend to interact with and only these parts I need to. I everywhere see that you need an instance if this Lua struct, but I don't see where it comes from. In mlua this is an argument for the module function. Is this "missing" in nvim-oxi?

[noib3]

The Lua object is a wrapper around the Lua state which is not exposed in nvim-oxi. This means that you can't just call Lua code from nvim-oxi yourself, it has to be exposed by us. Can I ask what's your use case for requireing Lua modules from Rust?

[weilbith]

Alright, understood. Is it hard to expose this? 🤔

My use-case are two. First, I want to interact with language servers and people often complain that they want to use in-built LSP support or CoC or vim-lsp or... So I intend to have a trait for the language client and have multiple implementations for it. And from there on have some typesafe Rust environment I can develop the core/business-logic of my plugin.
On the other "end" of the plugin I need to visually show some stuff to the user. Therefore I don't want to re-invent the wheel and allow the user to choose their UI plugin like Telescope, vim-fzf, etc. pp. Again, an abstract trait and some implementations of it. All in all just calling a Lua function of a plugin here and there.

I could write only the core of the plugin in Rust and the rest in Lua as "glue-code". But I would really love to make it exclusively in Rust. Lua is simply not a serious language for me except for hacking around some smallish things. It is too error prone and fragile in my opinion. I have an almost complete rewrite of one of my plugins, trying to establish proper architecture, clean code, a lot of unit tests and so forth. But in the end it is still all super loose and doesn't feel great. Lua doesn't has many features even though it is such a high level language and this bugs me.

Thanks for your support. 🙏🏾

[noib3]

@weilbith I've turned the original issue into this discussion since it was getting a bit off topic and opened #47, #48 instead.

Is it hard to expose this?

It's not, but that'll work differently to what we're doing now. I'll try to give a bit of context.

Currently we only have one module, nvim_oxi::api, which aims to be a 1:1 port of vim.api, which in turn is a table containing Lua bindings to the Neovim API written in C. This means we can use FFI to provide Rust bindings without having to deal w/ Lua at all.

Unfortunately all other major vim.* modules like vim.lsp, vim.ui, vim.treesitter, vim.diagnostic, etc., are all written in Lua. This means to expose those functionalities we have to call Lua from Rust, which is going to be slower then calling C because of the need to convert Rust values to Lua and vice-versa, but definitely possible. This is what mlua is doing, and if Lua interop is something you really need I'd suggest you look into it.

Note however that with mlua you'll also be calling Lua when using vim.api.* functions, basically doing

Rust -> Lua -> C

instead of

Rust -> C

On the other "end" of the plugin I need to visually show some stuff to the user. Therefore I don't want to re-invent the wheel and allow the user to choose their UI plugin like Telescope, vim-fzf, etc. pp.

Isn't this exactly what vim.ui was made for? A common UI API for plugin developers to use without having to integrate directly with all possible fuzzy-selection-like plugins. It's then the job of those plugins you listed to take over vim.ui.

Lua is simply not a serious language for me except for hacking around some smallish things. It is too error prone and fragile in my opinion.

This is probably the biggest reason why I started this project, and also why providing a way to call arbitrary Lua code is outside the scope of this project.

[weilbith]

Thanks for explaining this in more detail. And also thanks for making this a discussion thread.

I get the Rust -> Lua -> C part. And sure I would rather stick to Rust -> C only where I can. That is why I currently would use a mixture of nvim-oxi and mlua. But because my entrypoint is nvim-oxi I don't have access to the Lua global object.

In regards of vim.ui. Yes, that is true. But it is the same as vim.notify. It is horribly basic without much functionality. Each plugin defines on top its own metadata and has its own specification. For example having a good control over what gets previewed with each selectable item. I think it is a step towards a good idea, but at best it was a first mini step. At least that is my opinion.
Nevertheless, even if I would now decide to use vim.ui, the issue to communicate with other plugins remains for me at other places.

[ModProg]

I think there should be a way to provide access to the Lua Neovim ecosystem.

While you could do that through a pure Lua interface as I tried to implement myself: https://github.com/modprog/nvim-rust, I believe the better way would be to access the Neovim API through FFI with C and provide easy access to Lua APIs.

Even though rust provides most of what you would ever need, being able to access Neovim specific Lua libraries that are provided as plugins might be useful.

I think this can be achieved in two steps:

1. Provide access to the Lua state inside the oxi entry point (something like lua::with_state used by the print macro, maybe).
2. Provide an easy way to define the API you want to access:
   I could see some procmacro that takes the Lua path + the types of arguments and return and generates the necessary rust code to wrap the Lua call. Obviously this will possibly error when the Lua API changes or the types aren't what is expected, but that is inevitable and a risk the plugin author is responsible for to handle.

If you believe that this is not in scope for this library as long as the Lua state is somehow exposed, the helper macros etc. could be provided by a wrapper library.

When nvim_oxi allows access to 3rd party Lua functions, those could then be bundled in helper crates e.g. someone could build nvim-oxi-telescope that just provide the api of telescope, so that a plugin author can stay completely within the safety of rust.

[weilbith]

To me, this sounds amazing and just what I'm searching/hoping for. And I totally see that nvim-oxi should not loose its focus and strengths. I think it makes absolutely sense to complement it with other crates and have nvim-oxi only expose the minimum that is required to make these extensions possible.

[noib3]

@weilbith @ModProg 71e8f28 introduces mlua integration. By enabling the mlua feature flag you can get a &'static Lua which enables you to interact with any Lua plugin that may be available at runtime.

I've added a small example to show how to call an arbitrary Lua function (in that case just the global print) from Rust using mlua.

I could see some procmacro that takes the Lua path + the types of arguments and return and generates the necessary rust code to wrap the Lua call. Obviously this will possibly error when the Lua API changes or the types aren't what is expected, but that is inevitable and a risk the plugin author is responsible for to handle.

This is definitely out of scope for nvim-oxi. Maybe open a new issue on the mlua repo to see if @khvzak would be interested in providing some macro for easy access to nested tables.

[ModProg]

Makes sense, As I already have some things build for generating lua api functions with mlua, I might either propose it as a pull request for mlua or as a seperate crate. Maybe we can add an example when it is mature enough at some point.

[weilbith]

Wow. Thank you very much! Great effort. 🤗