Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

eventkit is allocating lots of heap #23

Closed
mniewrzal opened this issue Dec 5, 2023 · 8 comments
Closed

eventkit is allocating lots of heap #23

mniewrzal opened this issue Dec 5, 2023 · 8 comments

Comments

@mniewrzal
Copy link

Profiling is showing lots of allocated heap for eventkit stuff
image

@elek said

We don't re-use buffers, I guess. And we send at least one message after each incoming DRPC request.
The real used memory should be lower....
But, yeah, some other things to improve....
@storj-gerrit
Copy link

storj-gerrit bot commented Dec 11, 2023

Change eventkit: add benchmark for outgoingPacket mentions this issue.

@egonelbre
Copy link
Member

Looks like most of the allocations happen due to needing to create flate.Writer, which contains a large 65k window:

      flat  flat%   sum%        cum   cum%
   27.71GB 79.62% 79.62%    27.71GB 79.62%  compress/flate.NewWriter C:\Program Files\Go\src\compress\flate\deflate.go:666
    3.10GB  8.91% 88.54%     3.10GB  8.91%  compress/flate.(*compressor).initDeflate C:\Program Files\Go\src\compress\flate\deflate.go:371 (inline)
    2.64GB  7.59% 96.13%     2.64GB  7.59%  compress/flate.(*compressor).initDeflate C:\Program Files\Go\src\compress\flate\deflate.go:369 (inline)
    0.40GB  1.16% 97.29%     0.44GB  1.28%  github.com/jtolio/eventkit.(*outgoingPacket).addEvent s:\dev\eventkit\client.go:134
    0.34GB  0.97% 98.25%     0.34GB  0.97%  github.com/jtolio/eventkit.(*outgoingPacket).addEvent s:\dev\eventkit\client.go:127
    0.30GB  0.85% 99.10%     0.30GB  0.85%  compress/flate.(*huffmanEncoder).generate C:\Program Files\Go\src\compress\flate\huffman_code.go:277
         0     0% 99.10%     0.37GB  1.06%  compress/flate.(*Writer).Flush C:\Program Files\Go\src\compress\flate\deflate.go:725
         0     0% 99.10%     0.30GB  0.85%  compress/flate.(*compressor).deflate C:\Program Files\Go\src\compress\flate\deflate.go:408

golang/go#32371 has some further information on it.

storjBuildBot pushed a commit that referenced this issue Dec 11, 2023
@egonelbre
eventkit: add benchmark for outgoingPacket
babd81f 
Updates #23

Change-Id: Idfb40592faeeab8cf333770e3ea74f50bf5ccfc6
@storj-gerrit
Copy link

storj-gerrit bot commented Dec 11, 2023

Change eventkit: use a single element zlib.Writer pool mentions this issue.

storjBuildBot pushed a commit that referenced this issue Dec 12, 2023
@egonelbre
eventkit: use a single element zlib.Writer pool
7f26050 
goos: windows
goarch: amd64
pkg: github.com/jtolio/eventkit
cpu: AMD Ryzen Threadripper 2950X 16-Core Processor
                  │ before.txt  │              after.txt              │
                  │   sec/op    │   sec/op     vs base                │
OutgoingPacket-32   391.5µ ± 2%   163.5µ ± 1%  -58.23% (p=0.000 n=10)

                  │  before.txt   │              after.txt               │
                  │     B/op      │     B/op      vs base                │
OutgoingPacket-32   815.17Ki ± 0%   20.54Ki ± 0%  -97.48% (p=0.000 n=10)

                  │ before.txt │             after.txt             │
                  │ allocs/op  │ allocs/op   vs base               │
OutgoingPacket-32   309.0 ± 0%   290.0 ± 0%  -6.15% (p=0.000 n=10)

Updates #23

Change-Id: I87b4396cde2ef951799b30ef294cae790a3808f0
@storj-gerrit
Copy link

storj-gerrit bot commented Dec 13, 2023

Change eventkit: use picobuf instead of gogo proto mentions this issue.

storjBuildBot pushed a commit that referenced this issue Dec 13, 2023
@egonelbre
eventkit: use picobuf instead of gogo proto
534c5b7 
This reduces allocs for addEvent by ~10x

Updates #23

Change-Id: I0ef292f9cfaa2e22b6ec259739374345362dde42
@storj-gerrit
Copy link

storj-gerrit bot commented Dec 15, 2023

Change go.mod: bump to eventkit mentions this issue.

@storj-gerrit
Copy link

storj-gerrit bot commented Dec 15, 2023

Change go.mod: bump storj.io/eventkit mentions this issue.

storjBuildBot pushed a commit to storj/uplink that referenced this issue Dec 15, 2023
@egonelbre
go.mod: bump to storj.io/eventkit
64b6fe2 
Updates storj/eventkit#23

Change-Id: Icf87e9a0ba3bdb06df830250f7c86a5d39f3c2c7
storjBuildBot pushed a commit to storj/storj that referenced this issue Dec 15, 2023
@egonelbre
go.mod: bump storj.io/eventkit
819afa2 
Updates storj/eventkit#23

Change-Id: I57a9252741af02f37af81aa4cf5015685da64c02
@storj-gerrit
Copy link

storj-gerrit bot commented Dec 18, 2023

Change go.mod: bump storj.io/eventkit, uplink and private mentions this issue.

storjBuildBot pushed a commit to storj/edge that referenced this issue Dec 18, 2023
@egonelbre
go.mod: bump storj.io/eventkit, uplink and private
e577a3b 
Updates storj/eventkit#23

Change-Id: Ide447b55cc453c9f4a9123aeeae55939dc3269b0
@mniewrzal
Copy link
Author

I cannot see those allocations using latest eventkit version. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@egonelbre @mniewrzal