Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory leak with Evaluate #1441

Open
Iorpim opened this issue Feb 15, 2024 · 0 comments
Open

Memory leak with Evaluate #1441

Iorpim opened this issue Feb 15, 2024 · 0 comments

Comments

@Iorpim
Copy link

Iorpim commented Feb 15, 2024

Hello, I'm having a problem very similar to issue #1046. The code is really simple, but the memory consumption is increasing by roughly 60kb/s, and it will reach over 1.5GB if given enough time.
I've also found other somewhat similar issues, namely #454 and #552, but they don't seem appliable as I have confirmed I have a single Chrome window unlike #454, and I'm not trying to cancel the browser context like #552.

What versions are you running?

$ go list -m github.com/chromedp/chromedp
github.com/chromedp/chromedp v0.9.3
$ google-chrome --version
Version 121.0.6167.185 (Official Build) (64-bit)
$ go version
go version go1.22.0 windows/amd64

What did you do? Include clear steps.

This is my current code. Start is called once to initialize the context and inject the JS function, while Get and Post are called in a loop to call the function injected during initialization.

type Chrome struct {
	ctx    context.Context
	Cancel context.CancelFunc
}

func New(urlString string) (Chrome, error) {
	chromeInstance := Chrome{nil, nil}
	chromeInstance.Start(urlString)
	return chromeInstance, nil
}

func CreateChrome() (context.Context, context.CancelFunc) {
	var cancel context.CancelFunc

	ctx, _ := chromedp.NewContext(context.Background())
	opts := append(chromedp.DefaultExecAllocatorOptions[:],
		chromedp.Flag("headless", true),
	)
	allocCtx, _ := chromedp.NewExecAllocator(ctx, opts...)
	ctx, cancel = chromedp.NewContext(allocCtx)

	return ctx, cancel
}

func (c *Chrome) Start(urlString string) {
	ctx, cancel := CreateChrome()
	c.ctx = ctx
	c.Cancel = cancel

	js := `async function httpGet(url = '') {
		const response = await fetch(url, {
		  method: 'GET',		  
		});		
		return response.text();
	  };
	  async function postData(url = '', data = {}) {
		const keys = Object.keys(data);
		const response = await fetch(url, {
		  method: 'POST',
		  headers: {'Content-type': 'application/x-www-form-urlencoded'},
		  body: keys.map(k => k + "=" + encodeURIComponent(data[k])).join('&')
		});
		return response.text();
	};`

	u, _ := url.Parse(urlString)
	host := u.Scheme + "://" + u.Host
	var response string

	err := chromedp.Run(c.ctx, chromedp.Navigate(host),
		chromedp.Evaluate(js, &response, func(ep *runtime.EvaluateParams) *runtime.EvaluateParams {
			return ep.WithAwaitPromise(true)
		}))

	if err != nil {
		fmt.Println(err)
	}

}

func (c *Chrome) Get(urlString string) (string, error) {
	return c.Request(urlString, "GET", "", 0)
}

func (c *Chrome) Post(urlString string, body string) (string, error) {
	return c.Request(urlString, "POST", body, 0)
}

func (c *Chrome) Request(urlString string, method string, body string, count int) (string, error) {
	var response string
	js := fmt.Sprintf(`httpGet('%s')`, urlString)
	if method == "POST" {
		js = fmt.Sprintf(`postData('%s', %s)`, urlString, body)
	}
	err := chromedp.Run(c.ctx,
		chromedp.Evaluate(js, &response, func(ep *runtime.EvaluateParams) *runtime.EvaluateParams {
			return ep.WithAwaitPromise(true)}))
	if err != nil  {
                fmt.Println(err)
		time.Sleep(10 * time.Second)
		return c.MakeRequest(urlString, method, body, count++)
	}

	return response, err
}

#1046 mentions the use of runtime.ReleaseObject as the solution to the problem, or the use of string as a return value. I am already using a string, so I tried changing it to a *runtime.RemoteObject and manually releasing it, however it made the problem worse, with the memory usage increasing to about 1Mb/s.

DevTools debugging didn't give much insight except that there were a number of Window contexts and detached XMLDocuments, but I couldn't really trace their origin, nor could I find something in the code that could be causing these to be left behind.

image

Originally the code would navigate to the URL and inject both JS function body and function call for each request and that resulted in a 1Mb/s memory increase. Changing to a single Evaluate with just the function call each run has improved it, but even with the reduced leak the memory usage reaches 1GB in just a little over a day of execution.

What did you expect to see?

Stable memory usage across the entire execution.

What did you see instead?

Rapid memory use increase

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant