Which component uses the least memory for tailing a log?

[anfen]

I've trialled TextView for the last 3 months, and it consistently uses between 400MB - 1.2GB when displaying the last 31 lines of a log. Before I added this TView package, the app would use 20MB of memory.

Log lines are added every 2 seconds on average, sometimes with a sudden burst of lines, and the TextView is defined as:

logsTextView := tview.NewTextView().SetScrollable(false).SetWrap(false).SetChangedFunc(func() { app.Draw() })

I've implemented logsTextView.SetMaxLines(height) when the console window changes height, which results in a height 1 line less than the maximum possible lines.

I often read that TextView was not designed for tailing logs, with that being said, which component is recommended?

Thanks in advance, this is a fantastic package, well done on its development.

[rivo]

I wrote a small program to reproduce your claims:

package main

import (
	"fmt"
	"math/rand"
	"strings"
	"time"

	"github.com/rivo/tview"
)

const lorem = "Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua At vero eos et accusam et justo duo dolores et ea rebum Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua At vero eos et accusam et justo duo dolores et ea rebum Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet"

func main() {
	app := tview.NewApplication()
	textView := tview.NewTextView().
		ScrollToEnd().
		SetChangedFunc(func() {
			app.Draw()
		})

	go func() {
		for {
			substr := lorem[:strings.LastIndex(lorem[:20+rand.Intn(len(lorem)-20)], " ")]
			fmt.Fprintln(textView, substr)
			time.Sleep(10 * time.Millisecond)
		}
	}()

	if err := app.SetRoot(textView, true).Run(); err != nil {
		panic(err)
	}
}

A new line is added every 10 milliseconds. All text remains in RAM, you can even navigate to the very beginning at any time. I let it run for about 10 minutes on macOS and total memory usage of the process was at 89MB. I would say this is expected.

Then I added a SetMaxLines(100) at the beginning. With this, RAM usage remained at about 12MB throughout, no matter how long I ran it.

So it seems to me that you have a memory leak somewhere else.

Before I added this TView package, the app would use 20MB of memory.

I'm not sure what you mean by this. That is, it's unclear what you did before.

I've implemented logsTextView.SetMaxLines(height) when the console window changes height, which results in a height 1 line less than the maximum possible lines.

You didn't say how you calculated height so it's difficult to say if there's a problem. If you want me to look into this, please post a small program that reproduces this. (Btw, there's also no harm in using a value somewhat higher than height.)

I often read that TextView was not designed for tailing logs

I'm not sure where you read this but I would disagree. It may have been true in the beginning when we didn't have functions like SetMaxLines yet. Even if you keep a large text file in memory, performance should be decent. (Recent changes made this possible.)

[digitallyserviced]

@anfen I believe part of your performance degradation is coming from your app.Draw call on every change.

logsTextView := tview.NewTextView().SetScrollable(false).SetWrap(false).SetChangedFunc(func() { app.Draw() })

this is not a good way to handle updating the text view on change. you are unneccessarily calling app.Draw (the whole app's draw function), when you might have changes to ONLY one line in the textview. when you could even allow for multiple lines be added before it redraws (for example).

THe text view will re-render itself if you are modifying the text through the proper calls.

package main

import (
	"fmt"
	"math/rand"
	"strings"
	"time"

	"github.com/rivo/tview"
)

const lorem = "Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua At vero eos et accusam et justo duo dolores et ea rebum Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet Lorem ipsum dolor sit amet consetetur sadipscing elitr sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat sed diam voluptua At vero eos et accusam et justo duo dolores et ea rebum Stet clita kasd gubergren no sea takimata sanctus est Lorem ipsum dolor sit amet"

func main() {
	app := tview.NewApplication()
	textView := tview.NewTextView().
		ScrollToEnd().SetMaxLines(10)
		// SetChangedFunc(func() {
		// 	app.Draw()
		// })

        // this is a way to do it on EVERY change/write 
        // I made this write/draw every 2ms, depending on what you're logging
        // This can cause a lot of CPU churn/cycles which may be unneccessary and could be nice. 50% CPU usage on the core for this updating every 2ms and drawing at same time
	go func() {
		for {
			substr := lorem[:strings.LastIndex(lorem[:20+rand.Intn(len(lorem)-20)], " ")]
                        // here we call the write function in an app update hook so it happens on main thread UI, and also gets drawn automatically and NOT using the `SetChangedFunc`
			app.QueueUpdateDraw(func() {
				fmt.Fprintln(textView, substr)
			})
			time.Sleep(2 * time.Millisecond)
		}
	}()

	if err := app.SetRoot(textView, true).Run(); err != nil {
		panic(err)
	}
}

func main() {
	app := tview.NewApplication()
	textView := tview.NewTextView().
		ScrollToEnd().SetMaxLines(10)

        // To lessen your CPU churn, dont queue draws on every change
        // Change the draw's to occur on a regular basis but debounced/throttled a bit such as 200ms
        // This redraws every 200ms, but still writes to the textview's content every 2ms. This only uses 10% CPU on the core 
	go func() {
		for {
			substr := lorem[:strings.LastIndex(lorem[:20+rand.Intn(len(lorem)-20)], " ")]
                        // here we call the write function in the update hook so it happens on main thread UI but NOT in draw hook
			app.QueueUpdate(func() {
				fmt.Fprintln(textView, substr)
			})
			time.Sleep(2 * time.Millisecond)
		}
	}()

        go func() {
		for {
                        // the update hook so it happens on main thread UI, and also gets drawn
			app.QueueUpdateDraw(func() {
			})
			time.Sleep(200 * time.Millisecond)
		}
	}()
	if err := app.SetRoot(textView, true).Run(); err != nil {
		panic(err)
	}
}

Yolur usage may change,