Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fasthttp client + Ants / crazy Memory Usage -> fasthttp client has a workerpool? #826

Closed
T200proX7 opened this issue Jun 3, 2020 · 2 comments

Comments

@T200proX7
Copy link

Hello,

I am using Ants with fasthttp. Is it possible that this both packages are not compatible?
My Program fetchs 15000 rows from a MysqlDB and request the HTML code for parse 2 strings out of it.

My memory usage is crazy 1,9g with only 434 open socket files = 434 connections (maximum was around 2800):
15921 jonny 20 0 3093,3m 1,9g 6,9m S 7,3 25,1 11:59.63 goCheckResults

root@jdebian:~$ ls -l /proc/15921/fd | wc -l
434

in main():

client = &fasthttp.Client{
	ReadTimeout:         time.Duration(60) * time.Second,
	MaxIdleConnDuration: time.Duration(600) * time.Second,
	Dial: func(addr string) (net.Conn, error) {
		return fasthttp.DialTimeout(addr, time.Duration(60) * time.Second)
	},
	TLSConfig: &tls.Config{InsecureSkipVerify: true},
}
var wg sync.WaitGroup
p, _ := ants.NewPoolWithFunc(25, func(i interface{}) {
	processRow(i)
	wg.Done()
})
defer p.Release()

i := 0
for results.Next() {
	var current Job

	err = results.Scan(&current....)

	wg.Add(1)
	_ = p.Invoke(&current)
	i++
}
wg.Wait()

The function processRow calls

row := i.(*Job)

req := fasthttp.AcquireRequest()
defer fasthttp.ReleaseRequest(req)

req.SetRequestURI(row.url)
req.Header.SetMethod("GET")
req.Header.Add("User-Agent", userAgent)

//println(req.Header.String())

resp := fasthttp.AcquireResponse()
defer fasthttp.ReleaseResponse(resp)

//if err := client.Do(req, resp); err != nil {
if err := client.DoTimeout(req, resp, time.Duration(60) * time.Second); err != nil {
	return "", err
}

bodyBytes := resp.Body()
html := string(bodyBytes)

// parse using regex
// db.Exec (....) <- mysql query to update database

It is crazy how much memory usage this small program has.

I asked the question to the Ants developer @panjf2000 and he told me:
It's unnecessary to use ants in fasthttp because there is already a goroutine pool in fasthttp which is also the prototype of ants, so using ants with fasthttp makes no improvement but performance loss.

I checked the godoc of fasthttp and found:

workerPool serves incoming connections via a pool of workers
in FILO order, i.e. the most recently stopped worker will serve the next
incoming connection.

Unfortunately this answer of the Ants Developer does not help me, but since he is the Ants developer he is for sure right regarding using Ants with Fasthttp.

So my question is, why is the Memory Consumption so huge and should i remove Ants and just execute go processRow() ?

The fasthttp Project seems to be a very great project, unfortunately there are very less examples regarding fasthttp Client Requests - most of the examples and also most of the Google Results are about running Webservers with fasthttp.

I appreciate someone help me regarding my problem. Thank you very much!

@panjf2000
Copy link
Contributor

panjf2000/ants#95

@Fenny
Copy link
Contributor

Fenny commented Aug 16, 2020

Can we close this issue since this question is answered in panjf2000/ants#95. @erikdubbelboer

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants