Should we continue using Cockpit?

[imobachgs]

Recently, the Cockpit packages started to use the new Python-based bridge. That's perfectly fine, but it caused some problems in Agama that we need to fix.

When discussing the solution, we brought up two topics we would like to discuss:

• Do we really need Cockpit?
• What about offering a rich HTTP interface instead?

Disclaimer: Cockpit is a neat project and this discussion is unrelated to recent events. It was a conversation we had before, but this time we decided to record the conclusions and draw a plan if needed.

How does Agama use Cockpit?

In a nutshell, we use Cockpit as a transport layer to access Agama's D-Bus interface from the browser over a websocket. The communication is ruled by the Cockpit
protocol but thanks to cockpit.js library we do not need to deal with those details. Having said that, we only use the D-Bus part of the API (we hardly execute a command, as that logic is backed in our D-Bus API). It works that way by design: put as little business logic as possible in the UI layer.

However, Cockpit does a few extra things for us:

• It takes care of the authentication.
• It serves the web UI code (HTML, CSS, etc.).
• It implements a (fairly simple) translation mechanism.

Some drawbacks

Again, these points are not Cockpit fault; it just happens that Agama's use case is
slightly different.

• Lack of an API for external callers. It makes Agama harder to integrate, although not impossible (e.g., with SUSE Manager).
• As Cockpit is moving to a Python-based bridge, we need a Python process running.
• Relying on a custom protocol (Cockpit's one) instead of using a standard mechanism.
• We depend on an external project with a different use case. Fortunately, they are good open-source citizens :-)

Let's discuss

What do you think we should do? Should we fix the current implementation and continue moving? Or should we consider a more considerable change? Let's open the discussion!

[joseivanlopez]

IMHO, Agama is closer to a server-client architecture than a set of processes using IPC. Ideally, Agama would consist on a single backend service (with multithreating) that is running in the machine in which we want to install a system. That service encapsulates the state and logic of the installation process.

Then threre are clients. Clients (graphical or not) are "only" a kind of UI for interacting with our service. And there is no reason to have the client running in the very same machine than the service. For example, there could be a native Windows client which communicates with the Agama service running in another machine. This is also the case of SUMA, an external process that requires to call to the Agama service.

Moreover, I think we are abusing of D-Bus because we make use of it for both IPC and for defining an API to get and modify the installation process. IPC should be its unique usage.

So, in my opinion, moving from our D-Bus API to a rich HTTP API makes a lot of sense. Probably, we will still need some D-Bus API (only for IPC) meanwhile we have more than one backend service, but this is ok.

And for me the key point is the athentication and translation mechanisms that Cockpit brings and how difficult they would be to replace.

[lslezak]

For example, there could be a native Windows client which communicates with the Agama service running in another machine.

Please do not overengineer it.

The YaST architecture also originally supported the SCR backend running on a different machine. So in theory the UI could talk to the SCR running elsewhere. But in the end that was never fully supported and it just made the architecture unnecessarily complex without much benefits.

I cannot image someone would write a native Windows client for that...

[joseivanlopez]

Yes, sure. Anyway, the native Windows client was only a (extreme?) example to expose there is not need to have IPC between clients and the Agama service.

[imobachgs]

Well, HTTP is an established and mature standard, so it differs from the SCR example. The typical example @joseivanlopez is referring to is SUMA. In that case, you need to work around the problem by using SSH or extending the Agama client to do it by itself.

[lslezak]

The first WebYaST version actually had 2 separate servers: REST API and a web UI. The web UI connected to a REST server, which could run even on a different machine. That was quite similar to the current Cockpit. The idea was to have 1:N management and possibly manage bunch of machines with just the REST API part installed.

But later we dropped that and merged them into a single server. It was too slow, every request from web UI went through the REST API with all authentication/authorization checks and required more memory for two running web servers...

My point is to avoid unnecessary complexity when not really needed.

[imobachgs]

I understand your point, but we are not discussing complex architecture with two services. I do not think the architecture described by @joseivanlopez is much more complex than the current one. The main difference is that the complexity is hidden in Cockpit.

Not to mention that Cockpit is there because it is needed only for the web UI. The CLI does not take any advantage of Cockpit as it interacts directly with D-Bus.

If we talk about WebYaST, we should consider that the web development ecosystem was rather different back in 2009. We are talking about Ruby 1.X (luckily 1.9) and Rails 2.x. Rails 3, which was a huge step forward, was released in 2010. If we look at the browser side, React was not even a thing (2013). Even the HTTP protocol has changed! 😉

[lslezak]

Using Cockpit

We use Cockpit in Agama rather as a helper library. Actually we are hiding it as much as possible, users normally cannot see it in Agama except some special situations (the login screen, the integrated terminal). I guess many users do not know that Cockpit is actually running there... 😃

But it provides some quite nice functions which would be difficult to implement properly as we do not have much expertise in those areas.

The Cockpit API

As mentioned at the top, there are few ways how we interact with the system using Cockpit:

• DBus API - this is the critical part as the Agama backend is implemented as a set of DBus services, this is a must for Agama in the current architecture
• File API - At few places we read the raw files from the system (e.g. displaying the y2log content), but it could be easily reimplemented as a trivial DBus service
• Process API - We use it for setting the local keyboard layout, but again, it could be easily reimplemented as a trivial DBus service

The Web Server

The Cockpit web server is quite simple, it is basically a simple server serving static files (JS, CSS, fonts, images, ...). The advanced feature is that it provides a web socket for communication with the server.

We take advantage of this in our webpack development server where we serve the built files (JS, CSS,...) locally and the websocket traffic is redirected to a real Agama server (possibly running on a different machine) using proxy.

Localization

As mentioned, the localization support on the server side is rather simple, I have reimplemented that in the webpack development server.

On the client side the translation functions are also simple.

Architecture

Cockpit spawns a Cokpit bridge process for each running Cockpit session. That process runs as the logged in user. That means the users have the same system permission as when logging via SSH. And if you create a file it will have your ownership and permissions. That's good from the security POV. And it easily allows to have multiple user sessions without affecting each other.

(In WebYaST we did it differently, for each request we checked whether the logged-in user is allowed to perform the requested action via PolicyKit. The action was only performed if the user had the appropriate permissions otherwise it was denied. But it was still done as the root user...)

But Agama requires root permissions, it cannot work as non-root (at least the backend DBus service). And multiple user sessions do not make sense as well as we run a single DBus backend in the end.

Cockpit also supports remote servers, you can log into a different server than your browser is connected to.

For Agama we could be fine even with some simpler architecture with less features.

Authentication and Security

Cockpit handles authentication transparently for us, we do not need to take care about that at all. That's good because we do not have much experience in this area and it is critical as we need to run the server (at least the DBus services) as the root user. So having a security issue would be a big problem for us... 😱

It is not just about verifying the root password, but also about the ensuring that the user session is safe and secure (e.g. using CSRF tokens etc...).

I think this is the most critical and complex problem.

Python Dependency

I guess it is not a big problem for now as we already have Python included in the Live ISO:

/usr/bin/python3 is needed by (installed) bcache-tools-1.1+git37.a5e3753-1.1.x86_64

So unless we drop these dependencies the new Python dependency is not a big problem.

Integrations

Probably the biggest problem of the Cockpit architecture is that it does not allow easy integration with other tools. The Cockpit architecture does not allow implementing REST API or something similar.

But the question is what does the integration actually mean in this case. In a web application you can easily embed Agama using <iframe> or some similar tricks. If you need some automation or code integration then you can use Agama CLI over SSH so there are some possibilities or workarounds...

So unless we get some clear use cases and requirements I think the current situation is quite OK.

So What...???

IMHO the current situation is OK, Cockpit provides some important functionalities which would be difficult for us to reimplement (esp. from the security POV). Adapting to new released versions is a generic problem, we would have similar problems when using any framework. We know that from YaST pretty well... 😝 (We also had to adapt WebYaST to new Ruby or RoR...)

But Maybe...

Changing the architecture completely and using a true web server would be too complex for Agama, that would very likely not pay off. But maybe (maybe!) if we want to get rid of Cockpit we could reimplement the HTTP + websocket server part and keep the current client side Cockpit API.

Maybe it would not be that difficult... I have checked the Cockpit sources and the code is veeery long and complex. But that's because they do all the low level C stuff by themselves (even parsing HTTP headers, URLs, etc...). Using some web server framework and external libraries could make it much easier (example?).

[imobachgs]

In general, I agree with most of your points, but I have some comments.

Using Cockpit

We use Cockpit in Agama rather as a helper library. Actually we are hiding it as much as possible, users normally cannot see it in Agama except some special situations (the login screen, the integrated terminal). I guess many users do not know that Cockpit is actually running there... 😃

But it provides some quite nice functions which would be difficult to implement properly as we do not have much expertise in those areas.

Well, many of us have a past in that area (ok, some years ago) but, to be honest, we neither have experience with D-Bus 😅

The Cockpit API

As mentioned at the top, there are few ways how we interact with the system using Cockpit:

* DBus API - this is the critical part as the Agama backend is implemented as a set of DBus services, this is a must for Agama in the current architecture

* File API - At few places we read the raw files from the system (e.g. displaying the `y2log` content), but it could be easily reimplemented as a trivial DBus service

* Process API - We use it for setting the local keyboard layout, but again, it could be easily reimplemented as a trivial DBus service

By design, we try to stick to the D-Bus API. We want all the logic to be implemented in the service, not having specific bits depending on the client. Setting the keyboard is an exception I am not happy about 😄 but, given it was not needed in the CLI, we decided to go with the straightforward approach.

The Web Server

The Cockpit web server is quite simple, it is basically a simple server serving static files (JS, CSS, fonts, images, ...). The advanced feature is that it provides a web socket for communication with the server.

We take advantage of this in our webpack development server where we serve the built files (JS, CSS,...) locally and the websocket traffic is redirected to a real Agama server (possibly running on a different machine) using proxy.

Any decent web server (or library) already supports websockets.

Localization

As mentioned, the localization support on the server side is rather simple, I have reimplemented that in the webpack development server.

On the client side the translation functions are also simple.

Actually, none of these things need to change that much.

Architecture

[..]

For Agama we could be fine even with some simpler architecture with less features.

+1. The Agama use case is more constrained.

Authentication and Security

[..]

It is not just about verifying the root password, but also about the ensuring that the user session is safe and secure (e.g. using CSRF tokens etc...).

I think this is the most critical and complex problem.

Again, it is an already solved problem. Any decent library... well, copy&paste.

Python Dependency

I guess it is not a big problem for now as we already have Python included in the Live ISO:

/usr/bin/python3 is needed by (installed) bcache-tools-1.1+git37.a5e3753-1.1.x86_64

So unless we drop these dependencies the new Python dependency is not a big problem.

At the dawn of ALP, the idea was to not have Python installed in the system. OK, I know that's not the case anymore. So the only (minor) drawback is those additional 30MB of memory usage.

Integrations

Probably the biggest problem of the Cockpit architecture is that it does not allow easy integration with other tools. The Cockpit architecture does not allow implementing REST API or something similar.

But the question is what does the integration actually mean in this case. In a web application you can easily embed Agama using <iframe> or some similar tricks. If you need some automation or code integration then you can use Agama CLI over SSH so there are some possibilities or workarounds...

So unless we get some clear use cases and requirements I think the current situation is quite OK.

I am not aiming to integrate Agama into another web application. About 3rd party tools, as you mention, we might need to work around the problem: connecting through SSH or adding support directly in the client to do that. But an HTTP call is so standard... :-)

So What...???

IMHO the current situation is OK, Cockpit provides some important functionalities which would be difficult for us to reimplement (esp. from the security POV). Adapting to new released versions is a generic problem, we would have similar problems when using any framework. We know that from YaST pretty well... 😝 (We also had to adapt WebYaST to new Ruby or RoR...)

But Maybe...

Changing the architecture completely and using a true web server would be too complex for Agama, that would very likely not pay off. But maybe (maybe!) if we want to get rid of Cockpit we could reimplement the HTTP + websocket server part and keep the current client side Cockpit API.

Maybe it would not be that difficult... I have checked the Cockpit sources and the code is veeery long and complex. But that's because they do all the low level C stuff by themselves (even parsing HTTP headers, URLs, etc...). Using some web server framework and external libraries could make it much easier (example?).

That's exactly what I was discussing this week with some teammates. We can rely on some libraries to build a rather minimal communication layer and drop our dependency on Cockpit as a whole. For instance, we could give axum a try as we already depend on some tokio. And we already have zbus and friends to work with D-Bus data structures. But those are implementation details.

If we can do that, in the future, we could extend the API to support more HTTP calls if we want to.

[imobachgs]

I want to bring up another topic: our usage of D-Bus. I did not have experience with D-Bus when we started to build Agama, so perhaps my designs are not D-Bus friendly at all and that's why I find it limiting sometimes.

In general, I represent many concepts as objects themselves. For instance, we have a connection object for each network connection (/org/opensuse/Agama1/Network/connections/*). And we implemented several interfaces on top:

• org.opensuse.Agama1.Network.Connection for common stuff among all the connections,
• org.opensuse.Agama1.Network.Connection.IP for IP configuration,
• org.opensuse.Agama1.Network.Connection.Wireless for wireless settings.
• ... OK, you get the idea.

Something I dislike is that if I want to update several properties in one shot, I need to implement my own Update method because org.freedesktop.DBus.Properties only implements GetAll, Get, and Set. I can understand why there is no SetAll, it's fine. But my own Update would have to:

1. Receive all the arguments all the time or...
2. Receive a dictionary and I need to do all the type-checking on my side. Not to mention that it defeats the introspect purpose. Actually, NetworkManager interface is a good example of using hashes for almost everything.

There are other small glitches, like not having "optional" values, but they are easy to work around.

[imobachgs]

TL;DR we decided to stop relying on Cockpit.

Hi all,

We discussed this topic in a team meeting and decided to move on and stop relying on Cockpit. Do not get us wrong: Cockpit is a fantastic project, but we feel we use a tiny fraction of its features and do not need the rest. In @lslezak's words, we have been actively hiding Cockpit.

Having said that, we are grateful for what Cockpit has provided to us until now.

Why are we relying on Cockpit?

At this point, we use Cockpit to:

• Communicate with our D-Bus service.
• Authentication/authorization (including its own login page).
• Service the web UI files.
• Translate the UI.
• Offer a web-based terminal.
• Download the logs (using the files API).

Let's have a closer look at what each point means.

D-Bus communication

That's the main reason we started using Cockpit. It offers a convenient way to access D-Bus services from a browser. However, it lacks an API for external callers (by design).

Additionally, the proxies-based approach does not fit our use case so well. Working with the D-Bus API is kind of low-level and we need to duplicate some logic in the browser and the CLI.

Authentication/authorization

That's a hot topic. We do not need something as flexible as Cockpit's authentication. Agama is expected to be used only by the "root" user to perform the installation. However, we need to do some research.

Service UI files

It is trivial to implement this feature using any decent framework.

UI translation

The mechanism used by cockpit.js is pretty straightforward. We can keep using the same as it does not involve communication with the bridge.

A web-based terminal

Let's use xterm.js directly. Additionally, let's take the chance to improve its integration into Agama's UI.

Downloading the logs

It is something simple to implement using a decent framework.

The (lack of) plan

We do not have a clear plan and need to research. However, we know that there are a few questions to answer:

• How to handle authentication. We need to provide a mechanism not for user/password when installing locally.
• Should we offer a way to talk to D-Bus directly from elsewhere? Or should we hide the fact that we are using D-Bus at all?
• How are we going to continue evolving Agama during the paradigm shift?

We will let you know as we know the answer to those questions and the path becomes clear.

Regards,
Imo

[garrett]

We depend on an external project with a different use case. Fortunately, they are good open-source citizens :-)

Don't forget, Red Hat is (coincidentally) currently also re-imagining Anaconda (the Fedora, RHEL, CentOS, etc. installer) as a web-based installer too. So Cockpit is being used for the same use case on the Red Hat side too. 😉

(We're doing things a little bit different UI-wise, but I wanted to point out that an installer is actually a supported and usable use case for Cockpit.)

[imobachgs]

Hi @garrett,

Good point! Well, as said, we feel that we are using just a tiny fraction of what Cockpit can offers and we would like to try a different approach. Perhaps we change our minds in the future 😉

Thanks!

[garrett]

Sorry to hear that. Please do let us know if you change your mind.

Best of luck! I'll keep rooting for you all from the sidelines.

[imobachgs]

Thanks! We will remain in close contact!

[martinpitt]

Hello @imobachgs and others,

thanks for the transparency! I just want to respond to a few technical points.

* [Lack of an API for external callers](https://cockpit-project.org/guide/latest/packages.html#package-api). It makes Agama harder to integrate, although not impossible (e.g., with SUSE Manager).

I don't know what that means -- the whole point of the bridge, websocket, and protocol is to make OS APIs accessible to web pages and JS code. Callers outside of that (like desktop apps) would obviously not use that.

* As Cockpit is moving to a Python-based bridge, we need a Python process running.

This isn't a showstopper for Fedora's Cockpit based installer (Anaconda), but if it is for you, and you are currently using Cockpit for Agama: The current versions still contain the C bridge, so you can configure with --enable-old-bridge. In about a month we will drop the C bridge entirely, but if you still need it for a longer transition time, you can then switch to the upcoming "rhel-8" branch (we will do point releases on that, too), which will then only contain the C bridge. That branch will still be supported for > 5 years.

That doesn't mean that you should rely on this forever (the C bridge is dead and we don't want to change it any more), but it may help with your transition.

• Relying on a custom protocol (Cockpit's one) instead of using a standard mechanism.

There is no standard mechanism for web pages to do e.g. D-Bus calls or file access. If there was, (https://varlink.org/ was such a previous attempt), we'd be very happy and wouldn't have to maintain all this stuff 😁

Thanks, and good luck with everything!

[imobachgs]

Hello @imobachgs and others,

Hi @martinpitt!

thanks for the transparency! I just want to respond to a few technical points.

Welcome!

I don't know what that means -- the whole point of the bridge, websocket, and protocol is to make OS APIs accessible to web pages and JS code. Callers outside of that (like desktop apps) would obviously not use that.

We are referring to providing, for instance, an HTTP-based API (e.g., REST). Bear in mind that in Agama the CLI is an important part and we are also interested in offering better integration to other external tools. Sure, we could use D-Bus in some of those cases, but we would like to experiment to build a higher-level API.

* As Cockpit is moving to a Python-based bridge, we need a Python process running.

This isn't a showstopper for Fedora's Cockpit based installer (Anaconda), but if it is for you, and you are currently using Cockpit for Agama: The current versions still contain the C bridge, so you can configure with --enable-old-bridge. In about a month we will drop the C bridge entirely, but if you still need it for a longer transition time, you can then switch to the upcoming "rhel-8" branch (we will do point releases on that, too), which will then only contain the C bridge. That branch will still be supported for > 5 years.

That doesn't mean that you should rely on this forever (the C bridge is dead and we don't want to change it any more), but it may help with your transition.

Well, I would not say the new bridge is a showstopper for us (I am pretty sure we need Python in the installation media anyway) and we fully understand that you want to get rid of the old C bridge. The new one enables some interesting use cases and we are looking forward to them, but not for our installer.

• Relying on a custom protocol (Cockpit's one) instead of using a standard mechanism.

There is no standard mechanism for web pages to do e.g. D-Bus calls or file access. If there was, (https://varlink.org/ was such a previous attempt), we'd be very happy and wouldn't have to maintain all this stuff 😁

Hehe, yes, I know :-) I meant that we are more akin to use old plain HTTP calls and receiving events over the WebSocket. We are still deciding how far are we going to go with hiding the D-Bus part.

As a last note, take into account that Agama works only as an installer so we can drop some features that, for sure, you need for Cockpit. For the 1:1 management case, we have Cockpit ;-)

Thanks, and good luck with everything!

Thanks for your feedback! Good luck!