Daemon memory usage grows to 8 GB after 5 hours of file adding and pinning #9437
Comments
pio2398
added
kind/bug
|
Thx, I have looked at your profile and only 3.8GiB of memory is alive on the heap. Go recently introduced https://pkg.go.dev/runtime/debug#SetMemoryLimit you can set it with My test to confirm this behaviour was: package main
import "runtime/debug"
import "runtime"
import "os"
var leak []byte // lots of memory alive to bias GOGC through higher values
func main() {
// update freeMemory to the memory on your system
const freeMemory = 40 * 1024 * 1024 * 1024
const target = freeMemory / 3 * 2 // try to use two third of the system os (because with default 200% GOGC it will oom before GCing)
const garbage = 1024 * 1024
leak = make([]byte, target - garbage)
for i := range leak {
leak[i] = 1 // memset to force page commit
}
debug.SetMemoryLimit(freeMemory) // coment that line to test with normal GOGC behaviour
os.Stdout.WriteString("initial leak setupped, now generating garbage!\n")
var keepAlive []byte
for i := freeMemory/garbage * 3; i != 0; i-- {
// run this in for a while, try to generate 3 times more garbage than memory we have
keepAlive = make([]byte, garbage)
for i := range keepAlive {
keepAlive[i] = 1 // memset to force page commit
}
runtime.Gosched() // simulate some IO, let the GC maybe run
}
leak = keepAlive
}Using For now the mitigation I'll recomend is manually setting |
Checklist
Installation method
third-party binary
Version
Config
{ "API": { "HTTPHeaders": {} }, "Addresses": { "API": "/ip4/127.0.0.1/tcp/5001", "Announce": [], "AppendAnnounce": [], "Gateway": "/ip4/127.0.0.1/tcp/34538", "NoAnnounce": [ "/ip4/10.0.0.0/ipcidr/8", "/ip4/100.64.0.0/ipcidr/10", "/ip4/169.254.0.0/ipcidr/16", "/ip4/172.16.0.0/ipcidr/12", "/ip4/192.0.0.0/ipcidr/24", "/ip4/192.0.2.0/ipcidr/24", "/ip4/192.168.0.0/ipcidr/16", "/ip4/198.18.0.0/ipcidr/15", "/ip4/198.51.100.0/ipcidr/24", "/ip4/203.0.113.0/ipcidr/24", "/ip4/240.0.0.0/ipcidr/4", "/ip6/100::/ipcidr/64", "/ip6/2001:2::/ipcidr/48", "/ip6/2001:db8::/ipcidr/32", "/ip6/fc00::/ipcidr/7", "/ip6/fe80::/ipcidr/10" ], "Swarm": [ "/ip4/0.0.0.0/tcp/4001", "/ip6/::/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", "/ip6/::/udp/4001/quic" ] }, "AutoNAT": {}, "Bootstrap": [ "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ" "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt" ], "DNS": { "Resolvers": {} }, "Datastore": { "BloomFilterSize": 0, "GCPeriod": "1h", "HashOnRead": false, "Spec": { "mounts": [ { "child": { "path": "blocks", "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", "sync": true, "type": "flatfs" }, "mountpoint": "/blocks", "prefix": "flatfs.datastore", "type": "measure" }, { "child": { "compression": "none", "path": "datastore", "type": "levelds" }, "mountpoint": "/", "prefix": "leveldb.datastore", "type": "measure" } ], "type": "mount" }, "StorageGCWatermark": 90, "StorageMax": "10GB" }, "Discovery": { "MDNS": { "Enabled": false } }, "Experimental": { "AcceleratedDHTClient": false, "FilestoreEnabled": false, "GraphsyncEnabled": false, "Libp2pStreamMounting": false, "P2pHttpProxy": false, "StrategicProviding": false, "UrlstoreEnabled": false }, "Gateway": { "APICommands": [], "HTTPHeaders": { "Access-Control-Allow-Headers": [ "X-Requested-With", "Range", "User-Agent" ], "Access-Control-Allow-Methods": [ "GET" ], "Access-Control-Allow-Origin": [ "*" ] }, "NoDNSLink": false, "NoFetch": false, "PathPrefixes": [], "PublicGateways": null, "RootRedirect": "", "Writable": false }, "Identity": { "PeerID": "12D3KooWEHGcBURfi9hMCv8zGyX3UMpKWqGMSfSNSDjho7J4fWqo" }, "Internal": {}, "Ipns": { "RecordLifetime": "", "RepublishPeriod": "", "ResolveCacheSize": 128 }, "Migration": { "DownloadSources": [], "Keep": "" }, "Mounts": { "FuseAllowOther": false, "IPFS": "/ipfs", "IPNS": "/ipns" }, "Peering": { "Peers": null }, "Pinning": { "RemoteServices": {} }, "Plugins": { "Plugins": null }, "Provider": { "Strategy": "" }, "Pubsub": { "DisableSigning": false, "Router": "" }, "Reprovider": { "Interval": "12h", "Strategy": "all" }, "Routing": { "Methods": null, "Routers": null, "Type": "dht" }, "Swarm": { "AddrFilters": [ "/ip4/10.0.0.0/ipcidr/8", "/ip4/100.64.0.0/ipcidr/10", "/ip4/169.254.0.0/ipcidr/16", "/ip4/172.16.0.0/ipcidr/12", "/ip4/192.0.0.0/ipcidr/24", "/ip4/192.0.2.0/ipcidr/24", "/ip4/192.168.0.0/ipcidr/16", "/ip4/198.18.0.0/ipcidr/15", "/ip4/198.51.100.0/ipcidr/24", "/ip4/203.0.113.0/ipcidr/24", "/ip4/240.0.0.0/ipcidr/4", "/ip6/100::/ipcidr/64", "/ip6/2001:2::/ipcidr/48", "/ip6/2001:db8::/ipcidr/32", "/ip6/fc00::/ipcidr/7", "/ip6/fe80::/ipcidr/10" ], "ConnMgr": { "GracePeriod": "20s", "HighWater": 900, "LowWater": 600, "Type": "basic" }, "DisableBandwidthMetrics": false, "DisableNatPortMap": true, "RelayClient": {}, "RelayService": {}, "ResourceMgr": {}, "Transports": { "Multiplexers": {}, "Network": {}, "Security": {} } } }Description
My IPFS was unstable so I decide to remove all config and data and start new fresh instance.
I started by adding some local content and pining some remote and IPFS was killed by oomd. Next try also ended with usage more than 8 GB of ram.
diag: ipfs/QmU3EWqCxYsMN3EkuuMgeMnSsvPGW55NPfn3i9jU7BAJ93
The text was updated successfully, but these errors were encountered: