cannot reserve inbound connection: resource limit exceeded #9432
Comments
anarkrypto
added
kind/bug
|
This does not happen when running ipfs daemon from snap Kubo version: 0.16.0-38117db6f |
As mentioned this happens with kubo 0.17. I am facing the same issue. This is not an issue with snap or not but with the new libp2p code i guess which was turned on in 0.17. I have gone back to 0.16 for the moment. I have tried to increase the inbound connection limits to no avail |
|
I accidentally changed the resource in the wrong server, so changing the inbound connection value does work. So changing the inbound connections to 1024 cured this for me. So add to your .ipfs/config in the "Swarm" block the following then you can tweak as you need. |
|
Also ran into this problem. My error message is: We are running a customized 0.17.0 build. ipfs config showWe are running an experiment to measure the lookup latencies in the IPFS DHT network. For that we have deployed several, customized kubo nodes. The customization consist of just additional log messages. One of these log messages is right after the It logs the return values. I noticed that I receive a lot of the following errors: Therefore I went ahead and disabled the resource manager (see config above). The error messages still stick around. Then we just deployed more beefy machines and the errors seem to be less but they still happen frequently. It's also weird that the error message talks about an |
|
I see the same consistent issue.
|
IPFS Kubo v0.17 has a bug with ResourceMgr: ipfs/kubo#9432
|
I was able to fix by downgrading to v0.16.0 |
|
This error is expected when you have too many inbound connections at the System level to avoid DoS attacks. If your hardware or use case needs to support more inbound connections than the default, you can change that by doing: # Remove custom params
ipfs config --json Swarm.ResourceMgr '{}'
# Set inbound connection limits to a custom value
ipfs config --json Swarm.ResourceMgr.Limits.System.ConnsInbound 1000
# You might want to change also the number of inbound streams
ipfs config --json Swarm.ResourceMgr.Limits.System.StreamsInbound 1000
# If your hardware configuration is able to handle more connections
# and you are hitting Transient limits, you can also change them:
ipfs config --json Swarm.ResourceMgr.Limits.Transient.ConnsInbound 1000
ipfs config --json Swarm.ResourceMgr.Limits.Transient.StreamsInbound 1000
# Remember to restart the node to apply the changes
# You can see the applied changes executing:
$ ipfs swarm limit system
$ ipfs swarm limit transient
# You can check actual resources in use:
$ ipfs swarm stats system
$ ipfs swarm stats transientThe error is followed by a link: |
|
I'm facing the same issue. looking at the stats and comparing with the limits I have it's not even touches the limits but still seeing this error in my logs. I'm running the IPFS v0.17.0 |
|
@rotarur can you paste the error that you are having? Your node might be hitting another RM level limit, like transient. |
|
I'm running into this issue after upgrading to 0.17.0. Almost continuous in logs... |
|
|
soo. it looks like when the ResourceMgr limits are undefined |
|
Hm. I set defined limits for all the "random values", and still seeing random values after restarting IPFS. It looks like maybe some memory overflow... config: |
|
I also seem to be experiencing this even though I have the resource manager disabled |
|
@kallisti5 please check your configuration. It is wrong. Remove |
|
@2color how did you disable RM? |
|
my logs are always the same and I don't have the documentation link, weird The transient connections are not used My server is big enough for the IPFS and it has plenty of resources to use |
|
@rotarur are you getting errors like If there are no errors like these, it is a different problem. |
|
@ajnavarro LOL. I think you just found the issue.
That's the command I used to set FD EDIT: Nevermind. I just realized the syntax is indeed ipfs config ... without the : |
|
@ajnavarro I don't have any error like |
|
Can you configure the number of connections according to the protocol priority |
Jorropo
added
P1
|
Potentially a controversial take, but this feels like UX problem, and not a problem with default limits. Printing error every 10 seconds when any limit is hit is bit hardcore. 👉 ResourceMgr protecting user and working as expected should not look like ERROR. Quick ideas that would remove the need for relaxing default limits:
Potentially a controversial take: maybe there is an UX solution to this? Printing error every 10 seconds when any limit is hit is bit hardcore. 👉 ResourceMgr protecting user and working as expected should not look like ERROR. |
|
@lidel if we bump errors to be printed every hour, you won't notice when you really are hitting limits. Let's say that you hit limits at 1:01 PM then until 2:00 PM you won't have any kind of information, and the limits at that time are just fine, but you are having a warning or an error because you hit limits one hour away. We can make it possible to silence the error output, but the problem won't disappear. Nodes with better hardware will struggle to handle incoming connections even if they have enough real resources, and small nodes will be like they don't have any resource manager active because default limits will be too high. My proposal is to set default resource manager limits per peer only by default. It is the only limit that we can set by default knowing that will be right for any user and any hardware: #9443 |
|
Ah, I had that suspicion, but I wasn't entirely sure if this was indeed the case. Thanks for pointing to the Then that addresses all my concerns for me personally, except the stall on ping during those hours, that is still a mystery to me. The log entries could be made more clear if |
That's good news. However, this also raises the question of why it's an error. If the resource manager is working as it should and it rejects an inbound connection, that isn't an error. |
|
It's a warning-level log, I turned those on to get a better idea of how the node was behaving |
|
I don't mean the log level, I'm referring to the log message which shouldn't say error if it isn't one. |
And in the config i have: Note the FD. Plenty of room for IPFS. Yet when i start ipfs with warning level debugging turned on i'm near immediately welcomed with a ton of these resource limit warnings. Would it be possible to be more verbose to tell you what resource is limited and how to fix it? |
|
@markg85 : a couple of things:
|
|
@BigLep That was indeed with 0.18.1. That link helps to paste the correct details, thanks! I'm not hitting the limits according to those commands yet the logging tells me otherwise. |
|
@markg85 Does these errors contains the |
Yup. Ohh, so i should read it as "the remote hits a Resource Manager limit making a connection from me to that remote not possible" or something along those lines? |
|
@markg85 Something like that. Your node tried to make a connection with a remote node, and that node is hitting resource manager limits, so the default behavior is to return a resource manager error to your node. |
|
That's vague! Also, i'm just going over that list of peer id's and notice something very interesting: This might be stuff for a new issue, but... Kubo is dialing nodes a lot within the same second when it already knows - or can know - that the node it's going to dial has a "resource limit exceeded". Edit |
Being more clear that the "remote" string means its from a remote peer. This came up in: #9432 (comment) #9432 (comment) #9432 (comment)
|
For those following this issue, driving more clarity about the "remote" log messages is happening in #9653 and libp2p/go-libp2p#1928 The latest issue trying to consolidate on the resource manager/accountant work Kubo maintainers are focused on is here: #9650 |
Being more clear that the "remote" string means its from a remote peer. This came up in: #9432 (comment) #9432 (comment) #9432 (comment)
|
Kubo 0.19 RC is out which has further simplification and improvement for the libp2p resource manager/accountant. Please see https://github.com/ipfs/kubo/blob/master/docs/changelogs/v0.19.md#improving-the-libp2p-resource-management-integration and the linked dots. Feedback welcome on if users are running into any issues with 0.19RC. |
|
Given Kubo 0.19 has been out for over week, I'm going to close this issue. Per discussion above, relevant learns have been incorporated into code fixes, error messages, and docs: https://github.com/ipfs/kubo/blob/master/docs/libp2p-resource-management.md#how-does-the-resource-manager-resourcemgr-relate-to-the-connection-manager-connmgr |
|
I am trying to see limits because i'm getting errors on webtransport testing: |
|
the command seems to be |
|
@SgtPooki this is correct, see:
|
|
@Jorropo is there something better we can link to? there is not enough info there for me to set up any resource manager limits with confidence. --- edit:
is not helpful. the go-libp2p docs don't mention how to configure a json file at all, but kubo tells me to do so |
|
@SgtPooki I don't know how we could improve the docs, basically you just have to move I think I'm not seeing the problem here because I know too much on how a golang struct maps to json using |
Coming at this from outside golang and kubo. To begin when i in debug the docs suggest to use Confidance is important as when i perform as you describe, i still get the connect limit exceeded error, so i dont know if its a config issue or some deeper issue with kubo 🥂 |
Checklist
Installation method
built from source
Version
Config
{ "API": { "HTTPHeaders": { "Access-Control-Allow-Origin": [ "*" ] } }, "Addresses": { "API": "/ip4/0.0.0.0/tcp/5001", "Announce": [], "AppendAnnounce": [], "Gateway": "/ip4/0.0.0.0/tcp/8080", "NoAnnounce": [], "Swarm": [ "/ip4/0.0.0.0/tcp/4001", "/ip6/::/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", "/ip6/::/udp/4001/quic" ] }, "AutoNAT": {}, "Bootstrap": [ "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt", "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ" ], "DNS": { "Resolvers": {} }, "Datastore": { "BloomFilterSize": 0, "GCPeriod": "1h", "HashOnRead": false, "Spec": { "mounts": [ { "child": { "path": "blocks", "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", "sync": true, "type": "flatfs" }, "mountpoint": "/blocks", "prefix": "flatfs.datastore", "type": "measure" }, { "child": { "compression": "none", "path": "datastore", "type": "levelds" }, "mountpoint": "/", "prefix": "leveldb.datastore", "type": "measure" } ], "type": "mount" }, "StorageGCWatermark": 90, "StorageMax": "10GB" }, "Discovery": { "MDNS": { "Enabled": true } }, "Experimental": { "AcceleratedDHTClient": false, "FilestoreEnabled": false, "GraphsyncEnabled": false, "Libp2pStreamMounting": false, "P2pHttpProxy": false, "StrategicProviding": false, "UrlstoreEnabled": false }, "Gateway": { "APICommands": [], "HTTPHeaders": { "Access-Control-Allow-Headers": [ "X-Requested-With", "Range", "User-Agent" ], "Access-Control-Allow-Methods": [ "GET" ], "Access-Control-Allow-Origin": [ "*" ] }, "NoDNSLink": false, "NoFetch": false, "PathPrefixes": [], "PublicGateways": null, "RootRedirect": "", "Writable": false }, "Identity": { "PeerID": "12D3KooWMywfzmLWCWErc9L7CmfLLFbmoSHroHBUveUPaarDbAfF" }, "Internal": {}, "Ipns": { "RecordLifetime": "", "RepublishPeriod": "", "ResolveCacheSize": 128 }, "Migration": { "DownloadSources": [], "Keep": "" }, "Mounts": { "FuseAllowOther": false, "IPFS": "/ipfs", "IPNS": "/ipns" }, "Peering": { "Peers": null }, "Pinning": { "RemoteServices": {} }, "Plugins": { "Plugins": null }, "Provider": { "Strategy": "" }, "Pubsub": { "DisableSigning": false, "Router": "" }, "Reprovider": { "Interval": "12h", "Strategy": "all" }, "Routing": { "Methods": null, "Routers": null, "Type": "dht" }, "Swarm": { "AddrFilters": null, "ConnMgr": {}, "DisableBandwidthMetrics": false, "DisableNatPortMap": false, "RelayClient": {}, "RelayService": {}, "ResourceMgr": {}, "Transports": { "Multiplexers": {}, "Network": {}, "Security": {} } } }Description
Trying to run IPFS on VPS.
Error Description:
The node starts, show success swarm announcing addresses, the port 4001 is exposed externally ( I checked) and after a few seconds it closes and I got this error message:
ERROR resourcemanager libp2p/rcmgr_logging.go:53 Resource limits were exceeded 496 times with error "system: cannot reserve inbound connection: resource limit exceeded".Then the port cannot be reached anymore
Error occurend on both Docker (20.10.21) and installation from binaries
The text was updated successfully, but these errors were encountered: