Use xterm.js to archive a web log viewer

[yangguansen]

I'm using xterm.js to archive a web log viewer.

What are i need to do :

• Load latest log data by websocket and render it at bottom position

• Page scroll up, load previous log data to render

• Add tag to display log tag in each log row head. i used decoration like image below:
  

As a web log viewer, when i scroll up, i want to load prev page log data, does xterm.js has any way to prepend data?
So far i adopt re-render all data to load more data.

And when i called reset api to re-render data, decorations will not be cleared, so i was clear decoration DOM manually. Do we have any api can clear decorations? The dispose i tried, but it will not clear decoration.

[jerch]

@yangguansen This looks a bit like an xy problem to me - are you sure a terminal is the right tool for your log viewer? The problem here - yes you can do that in a terminal, but not with the terminal alone (so no, xterm.js cannot do that itself) - you'd have to write a pager like less or more in javascript to do the log data windowing.
A more extreme example - can I edit text files in xterm.js? Yes you can, but not with the terminal alone - you'd need a console editor doing the actual editing logic (like vim or emacs, or something similar in JS). So should you use xterm.js to edit files in the browser? - Prolly not, use an editor component for that. Unless you are already within a real PTY terminal session, where ready to go editors like vim or emacs can be used.

(I shamelessly assumed, that you want to do everthing in-browser and not on a PTY terminal session, otherwise you can simply just use less or more)

So maybe you should go 2 steps back and ask yourself, why a terminal is needed here. If you have no strong reason for a terminal, then using a ready-to-go logger view might be much faster to adopt, even writing the logger view from scratch might get you faster to a working solution than trying to bend xterm.js into that specific use case.

What are strong reasons to use xterm.js?

• forwarding a real PTY terminal session from underlying OS (main purpose of xterm.js)
• interactive in-browser REPLs (more an edgy use case, needs tons of workarounds to shim the missing PTY abstraction, not recommended unless you are very deep into POSIX terminal API and common line editing paradigms) thats where you would end up with a custom pager in xterm.js
• replay of terminal data or cleanup data of escape sequences (also edge case, for proper escape sequence cleaning / text extraction you gonna need a terminal emulator, libs like strip-ansi stop working reliably for more complicated sequences)

--> If none of that is relevant for you - dont use a terminal in the first place.

[yangguansen]

Hi @jerch
Thanks for your kind suggestion.
The log viewer in my case just support readonly, i don't need edit text in my case, it's just need display much log data from backend api in real time.

The reasons i use xterm.js are:
It's support canvas render, my old version is use DOM render with html, and i have 5000+ log data need to render, which is much pressure to browser and it render slowly than canvas in xterm.js

And i also investigated other third lib, but unlucky i didn't find a lib suit for my case, can render 5000+ log using canvas.

For more and less, yes we do need this, for more, we have websocket to receive real time data and render by write().
For less, I noticed xterm.js has .xterm-scroll-area classname, and i tried listened it to judge if it scroll up to top, if true, i request endpoint prev data and re-render them again, like:

const prevData = await fetch(xxx);
const newData = [...prevData, ...data];
xterm.reset();
newData.forEach(x => xterm.writeln(x));

and then .xterm-scroll-area will scroll to bottom, but i can judge xterm-scroll-area dom height to scroll to previous page position. like:

loading.show();
const oldHeight = document.querySelector('.xterm-scroll-area').getBoundingClientRect().height;
const prevData = await fetch(xxx);
const newData = [...prevData, ...data];

xterm.reset();
newData.forEach(x => xterm.writeln(x));

setTimeout(() => {
  const newHeight = document.querySelector('.xterm-scroll-area').getBoundingClientRect().height;
  document.querySelector('.xterm-scroll-area').scrollTo({x: 0, y: newHeight - oldHeight});
  loading.hide();
}, 500)

I assumed can re-render again after 500ms, which is a potential issue to me, if it not re-render done, page will not scroll to prev position.

So far i archived them and made a demo at my side, they works well. but still have follow potential issues:
How to prepend data?
When have prev data, I propose to move cursor to first line head, and writeln prev data, to see if it can archive prepend data

When re-render finished?
So far i assumed 500ms can be ok, but it still not a best idea to judge it rerender finished.

How to clear decorations?
I use decoration to render log tag in each row, and i noticed it will render as a dom. and i decoration.dispose(), but it will not remove in dom, so i use document.querySelector('.xterm-decoration-container').innerHTML = '' to remove them.
I'm not sure if it has potential problem with that.

[jerch]

... and i have 5000+ log data need to render ...

5000+ of what? Eggs? MB? Characters?

For more and less, ...

Ah thats a misunderstanding - I actually meant the unix/linux commands more and less. They are called pager, because they split big files into tiny portions fitting into one terminal page for manual inspection.

How to prepend data?
When have prev data, I propose to move cursor to first line head, and writeln prev data, to see if it can archive prepend data

You cant, at least not by normal means. The scrollbuffer in xterm.js is one-way only, you can only append data at the bottom. It is also a ringbuffer, thus will start to evict data at the top once the buffer is full. You might get away with dumping all data at once, if the data is small enough (<< 2 GB).
Changing that default behavior or adding a prepend method is non-trivial - sure feel free to do that in your own fork of xterm.js, but you are likely to introduce hard to hunt down bugs at various ends. A terminal does much more than simply pushing buffer content to screen.

It's support canvas render, my old version is use DOM render with html, and i have 5000+ log data need to render, which is much pressure to browser and it render slowly than canvas in xterm.js

I dont believe, that a simple <prev> or div element showing some log buffer content at various offsets is slower than xterm.js - the DOM renderer (which is btw faster than the canvas renderer) does exactly that plus a lot more on top. If your own solution was much slower, then "you are holding it wrong".

Now you might wonder why I lobby so much against xterm.js here? - Because by using xterm.js you buy into the whole terminal interface madness for such a simple task and will regret it sooner or later. The terminal interface is full of ancient data formatting rules and cumbersome to use with data not coming from a PTY, this surely will bite you back.

What you actually need is a fast (almost) unstyled content rotation on a DOM element. There are various tricks to achieve that, typically thats a strong domain of screen update optimized frameworks (e.g. react, vue, svelte etc). I suggest to check for ready-to-go components there.

[yangguansen]

Sorry not clarify , max 5000+ items logs, each item have 0 - 100 characters i estimate.

So far we used with react and one-time load 5000 items, so there are 5000 div in dom, it is large for browser and i feel there are space to optimize. i tried with 5000 items using xterm.js, it works fluency.

[mofux]

I propose to take a look into monaco-editor. It only renders the lines of the buffer that are visible in the viewport of the editor, and it can handle thousands of lines easily. It also supports decorations, custom code highlighting (e.g. for log files) and much more. It also has a very good API so you could introspect the scrolled viewport and load more data as you need.

As a starter for a custom log viewer, take a look at this example:
https://microsoft.github.io/monaco-editor/playground.html?source=v0.44.0#example-extending-language-services-custom-languages

[yangguansen]

@mofux Thank u, I will take a look that.

[jerch]

@yangguansen, @mofux
Interesting - tried to find a good ready-to-go logger component myself and well, seems there are no good ones out there optimized for chunking through lots of data records. So monaco is prolly your best bet atm, its def. has seen lots of optimization for fast screen update and slim internal data storage.