Ants + fasthttp

[T200proX7]

Hello,

I am using Ants with fasthttp. Is it possible that this both packages are not compatible?
My Program fetchs 15000 rows from a MysqlDB and request the HTML code for parse 2 strings out of it.

My memory usage is crazy 1,9g with only 434 open socket files = 434 connections (maximum was around 2800):
15921 jonny 20 0 3093,3m 1,9g 6,9m S 7,3 25,1 11:59.63 goCheckResults

root@jdebian:~$ ls -l /proc/15921/fd | wc -l
434

in main():

client = &fasthttp.Client{
	ReadTimeout:         time.Duration(60) * time.Second,
	MaxIdleConnDuration: time.Duration(600) * time.Second,
	Dial: func(addr string) (net.Conn, error) {
		return fasthttp.DialTimeout(addr, time.Duration(60) * time.Second)
	},
	TLSConfig: &tls.Config{InsecureSkipVerify: true},
}
var wg sync.WaitGroup
p, _ := ants.NewPoolWithFunc(25, func(i interface{}) {
	processRow(i)
	wg.Done()
})
defer p.Release()

i := 0
for results.Next() {
	var current Job

	err = results.Scan(&current....)

	wg.Add(1)
	_ = p.Invoke(&current)
	i++
}
wg.Wait()

The function processRow calls

row := i.(*Job)

req := fasthttp.AcquireRequest()
defer fasthttp.ReleaseRequest(req)

req.SetRequestURI(row.url)
req.Header.SetMethod("GET")
req.Header.Add("User-Agent", userAgent)

//println(req.Header.String())

resp := fasthttp.AcquireResponse()
defer fasthttp.ReleaseResponse(resp)

//if err := client.Do(req, resp); err != nil {
if err := client.DoTimeout(req, resp, time.Duration(60) * time.Second); err != nil {
	return "", err
}

bodyBytes := resp.Body()
html := string(bodyBytes)

// parse using regex
// db.Exec (....) <- mysql query to update database

It is crazy how much memory usage this small program has.

[panjf2000]

It's unnecessary to use ants in fasthttp because there is already a goroutine pool in fasthttp which is also the prototype of ants, so using ants with fasthttp makes no improvement but performance loss.

[T200proX7]

I guess the WorkerPool in fasthttp is for Server use only. So this would not help solving the task.

workerPool serves incoming connections via a pool of workers
in FILO order, i.e. the most recently stopped worker will serve the next
incoming connection.

[panjf2000]

Sorry, I didn't check out the problem carefully before and thought you were talking about the server-side, have you tried not to use ants pool and what's the memory usage of that kind of situation?

If the memory usage is still high, it probably has nothing to do with ants.

[T200proX7]

I tried now to use it without ants pool and yes memory was the same, maybe a little bit higher. But from the logic side it must be higher without a pool, since the pool is to manage jobs one by one for each worker (limited to the worker). Based on my code with 25 workers the memory should be very less. (One job done and canceled, worker takes next job)

Calling go processRow(current) is like a rush without a stop or limit - its a more high concurrency than with a 25 worker pool. The job is done with go processRow() in 1/10 of the time of using ants. But on the other side i had to increase the ulimit very high to avoid too many sockets open error message. it is kind of unmanaged.

So im not sure what and why.. even with ants i had the too many sockets open error, so maybe ants did not manage the jobs correct? Im not sure, i am not a very experienced go programmer.

[panjf2000]

There is no way that ants doesn't control the concurrent amount correctly, what version of ants were you using?

You can add some debug code into your ants function and count how many sockets are opened by your program at the same time. Did you close any socket after a task is done strictly?

[T200proX7]

I include github.com/panjf2000/ants - Application developed last week.

You could be right with "close any sockets after task"-issue. It could be the reason i have to increase the ulimit of open files (sockets in this case) before execution of the application:

I use:

defer fasthttp.ReleaseRequest(req)
defer fasthttp.ReleaseResponse(resp)

I checked the godoc https://godoc.org/github.com/valyala/fasthttp again, i dont see any other option to close the connection. In my opinion release should close the connection.
I checked even if there is a KeepAlive Option on new instance - there is no.

Any idea?

[erikdubbelboer]

How are you measuring memory usage?

If you want to close keep-alive connections you should set a Connection: close header in your requests.

Or you could set https://godoc.org/github.com/valyala/fasthttp#Client.MaxIdleConnDuration to a second for example if you want to reuse the connection shortly but don't want to keep it around for long.

[T200proX7]

I measure using ps aux, top, cat /proc/*/status and cat /proc/*/smaps.

Ok i will add the req.Header.Add("Connection", "close") and give another try with ants.

Regarding MaxIdleConnDuration thank you for the information. In my case it is not necessary, i just do one request per host and am looking to close it fast to use less system resources.

[T200proX7]

Ants:
My Application checked 2650 rows/urls. Took it from the MySQL DB passed it to the Ants Pool with 25 Workers. Requested the HTML Code of the Page and parsed it for two strings.

I added now the Connection: close to the request header and checked the results:

The open sockets are not increasing that much and that fast as before.. but they are still increasing:

root@jdebian:~$ ls -l /proc/2865/fd | wc -l
142

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2865 jonny     20   0    3,0g   2,2g   0,0g S  61,8  28,1   6:24.34 AntsgoCheckResu

root@jdebian:~$ cat /proc/2865/status
Name:   AntsgoCheckResu
Umask:  0022
State:  S (sleeping)
Tgid:   2865
Ngid:   0
Pid:    2865
PPid:   2839
TracerPid:      0
Uid:    1000    1000    1000    1000
Gid:    1000    1000    1000    1000
FDSize: 256
Groups: 24 25 27 29 30 44 46 109 112 115 1000
NStgid: 2865
NSpid:  2865
NSpgid: 2865
NSsid:  2839
VmPeak:  3158000 kB
VmSize:  3158000 kB
VmLck:         0 kB
VmPin:         0 kB
VmHWM:   2301028 kB
VmRSS:   2301028 kB
RssAnon:         2293980 kB
RssFile:            7048 kB
RssShmem:              0 kB
VmData:  2429984 kB
VmStk:       132 kB
VmExe:      3112 kB
VmLib:      1500 kB
VmPTE:      4648 kB
VmSwap:        0 kB
HugetlbPages:          0 kB
CoreDumping:    0
Threads:        12
SigQ:   0/31255
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000000
SigCgt: ffffffffffc1feff
CapInh: 0000000000000000
CapPrm: 0000000000000000
CapEff: 0000000000000000
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs:     0
Seccomp:        0
Speculation_Store_Bypass:       vulnerable
Cpus_allowed:   f
Cpus_allowed_list:      0-3
Mems_allowed:   00000000,00000001
Mems_allowed_list:      0
voluntary_ctxt_switches:        227683
nonvoluntary_ctxt_switches:     522
root@jdebian:~$

Its a lot for 25 workers checking 2650 urls. For my logical understanding 25 workers -> each one takes a URL and request the HTML from it. Then do the parsing stuff. So even when the HTML is 1MB each URL so we are in total at 25MB, some worker is faster, some slower.. lets increase it to the double size -> 50MB. So lets say the GC is very slow and need some time to recognise the current func() - call is not alive anymore, so lets triple the memory consumption -> we are on 150MB.

As you see the Application use 2,2g for sure and 3.0g "reserved".
As longer the Application run, as more resources it aquire. Around 2,2g it stucks.

I checked my code again and the only one point where is a real consumption of memory is happening is the fasthttp request while receiving the requested HTML Code from the URL.

---

Without Ants:
This time took 14000 rows from DB, the Application did not use workers and call go processRow() instead. I had to increase ulimit -n 99999 because without it i received udp 8.8.8.8:53 - too many open socket files (i should have taken 2650 rows .. but in this case the Application just run some seconds).

root@jdebian:~$ ls -l /proc/4010/fd | wc -l
6048

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
4010 jonny     20   0    5,7g   3,6g   0,0g S  82,1  45,9   1:35.44 noAnts_CheckResu

For my understanding the Goroutines should activate the GC after the go processRow() execute return, so every memory consumption within the processRow() should get released in this moment.

[erikdubbelboer]

Sounds like more memory than should be required. But keep in mind that fasthttp trades more memory usage for less CPU usage by reusing buffers.

What I would do next is use https://golang.org/pkg/net/http/pprof/ to see where the allocations are. Make sure to set runtime.MemProfileRate = 1 all the way at the top of your main function.

Also could you try using client.Do instead of client.DoTimeout? You already set a ReadTimeout on the client, you should also set a WriteTimeout of a couple of seconds.
client.DoTimeout will return after the timeout but won't close the connection so the request might still be running in the background. This could cause more open connections than you might expect.

[T200proX7]

Ok thank you for the advise, i will check it and we will see. Yes the memory consumption is abnormal huge. Good that you see it the same way. I was already worry in myself.

Yes i read already that fasthttp trades memory for cpu. But everything over 500MB for this task is crazy and unreal.

[T200proX7]

So i checked no the noAnts_Program with 50 rows only:
3561 root 20 0 989.1m 456.0m 9.9m S 0.0 9.2 0:38.90 noAnts_goCheckR

Showing nodes accounting for 228502.19kB, 99.40% of 229891.07kB total
Dropped 423 nodes (cum <= 1149.46kB)
      flat  flat%   sum%        cum   cum%
  228480kB 99.39% 99.39% 228888.12kB 99.56%  github.com/valyala/fasthttp.appendBodyFixedSize
   21.88kB 0.0095% 99.40% 229228.88kB 99.71%  github.com/valyala/fasthttp.(*Client).Do
    0.27kB 0.00012% 99.40% 229201.64kB 99.70%  github.com/valyala/fasthttp.(*HostClient).doNonNilReqResp
    0.05kB 2e-05% 99.40% 229320.91kB 99.75%  main.processRow
         0     0% 99.40% 229201.64kB 99.70%  github.com/valyala/fasthttp.(*HostClient).Do
         0     0% 99.40% 229201.64kB 99.70%  github.com/valyala/fasthttp.(*HostClient).do
         0     0% 99.40% 229025.44kB 99.62%  github.com/valyala/fasthttp.(*Response).ReadLimitBody
         0     0% 99.40% 228992.12kB 99.61%  github.com/valyala/fasthttp.readBody
         0     0% 99.40% 220736.12kB 96.02%  github.com/valyala/fasthttp.readBodyChunked
         0     0% 99.40% 229272.33kB 99.73%  main.fetchURL

# runtime.MemStats
# Alloc = 1279872
# TotalAlloc = 886342752
# Sys = 558774520
# Lookups = 0
# Mallocs = 165934
# Frees = 161070
# HeapAlloc = 1279872
# HeapSys = 536051712
# HeapIdle = 532905984
# HeapInuse = 3145728
# HeapReleased = 526229504
# HeapObjects = 4864
# Stack = 819200 / 819200
# MSpan = 56576 / 131072
# MCache = 6944 / 16384
# BuckHashSys = 2305008
# GCSys = 17539072
# OtherSys = 1912072
# NextGC = 4194304
# LastGC = 1591363913647135690
# PauseNs = [126260 191276 198996 71658 1093753 110912 57353 135365 73806 2873532 53722 414573 65790 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
# PauseEnd = [1591363600374965583 1591363600500069755 1591363600594642301 1591363602272236950 1591363603919804979 1591363605278324163 1591363607441511578 1591363612664953655 1591363625182586297 1591363643347110310 1591363671683448609 1591363792671338235 1591363913647135690 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]
# NumGC = 13
# NumForcedGC = 0
# GCCPUFraction = 4.101210998880725e-05
# DebugGC = false

Count | Profile
-- | --
3273 | allocs
0 | block
0 | cmdline
59 | goroutine
3278 | heap
0 | mutex
0 | profile
13 | threadcreate
0 | trace

Looks like everything is as it should be. Hmm.
Lets see.. little bit later i will profile every single Goroutine to see the real consumption. I must read first how to do it.

[T200proX7]

noAnts_NetHttp_Program / 50 rows=Urls:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
5760 root      20   0  655.0m 179.8m   6.5m S   1.0  3.6   0:26.33 noAntsNetHTTP_g

real    1m0.275s
user    0m22.984s
sys     0m3.680s

noAnts_FastHttp_Program / 50 rows=Urls:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 5958 root      20   0  854.5m 387.6m   6.6m S   0.0  7.8   0:24.12 noAnts_goCheckR

real    2m57.990s
user    0m24.220s
sys     0m5.156s

Ofc time measuring is difficult because of network latency but i took the same 50 urls.. so in one way its not 100% reliable but on the other side it is telling us something.

[T200proX7]

Ants_NetHttp_Program / 50 rows=Urls / 25 Ants Worker:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
6608 root      20   0  581.0m 144.0m   6.5m S  8.7  2.9   0:27.46 AntsNetHTTPgoCh

real    0m50.328s
user    0m23.448s
sys     0m3.852s

Ants_FastHttp_Program / 50 rows=Urls / 25 Ants Worker:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
6517 root      20   0  782.5m 383.5m   6.0m S   0.0  7.7   0:27.60 AntsgoCheckResu

real    2m9.341s
user    0m23.172s
sys     0m4.476s

Im on the end with my knowledge and really dont know how to continue this task with golang. Maybe i have some mistake in my code, but the code is 200 lines.. and Golang is very simple.. so i dont think there can be one/such a bad mistake.

With Net/Http the Memory Usage is lower than with FastHttp but even this memory consumption is high in my opinion. 50 seconds/2 minutes for 50 http requests on 25 workers? If i use the BulkOpener Extension for Chrome and Open the 50 Urls it takes same time including rendering. It is strange, i dont trust all this numbers - i have even better numbers with php+pthreads (except the crazy high cpu usage).

[erikdubbelboer]

Would you be willing to share your code in private with me so I can see what can be improved?

[T200proX7]

Not just willing i would appreciate it much - so i even can learn. I will send it to you by email in a few minutes. Already now i wanna thank you.