Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excessive CPU usage with QUIC #8256

Closed
unbeatable-101 opened this issue Jul 10, 2021 · 24 comments · Fixed by #8263
Closed

Excessive CPU usage with QUIC #8256

unbeatable-101 opened this issue Jul 10, 2021 · 24 comments · Fixed by #8263
Labels
kind/bug A bug in existing code (including security flaws) P0 Critical: Tackled by core team ASAP
Milestone

Comments

@unbeatable-101
Copy link

Version information:

go-ipfs version: 0.9.0-179d1d150
Repo version: 11
System version: arm64/darwin
Golang version: go1.16.5

Description:

For the past day or so IPFS keeps spiking up to 700% CPU usage, staying there for a bit, then going back down.

@unbeatable-101 unbeatable-101 added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Jul 10, 2021
@welcome
Copy link

welcome bot commented Jul 10, 2021

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

  • "Priority" labels will show how urgent this is for the team.
  • "Status" labels will show if this is ready to be worked on, blocked, or in progress.
  • "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

@obo20
Copy link

obo20 commented Jul 10, 2021

@unbeatable-101 Do you have the new experimental DHT enabled with 0.9.0?

@unbeatable-101
Copy link
Author

I had it enabled yesterday, but have since disabled and restarted (Unless I typed the command wrong, let me just make sure).

@obo20
Copy link

obo20 commented Jul 10, 2021

@unbeatable-101 the experimental DHT provider has been shown to cause spurts of max CPU usage while it's doing initial provides. Can you report on whether or not 0.9.0 is showing increased CPU usage with this setting disabled?

@unbeatable-101
Copy link
Author

unbeatable-101 commented Jul 10, 2021

I had it enabled yesterday, but have since disabled and restarted (Unless I typed the command wrong, let me just make sure).

It is disabled and currently sitting at 720% cpu usage

@aschmahmann
Copy link
Contributor

aschmahmann commented Jul 11, 2021

@unbeatable-101 if you could paste your config file (ipfs config show) as well as grab a profile dump when the CPU usage is high https://github.com/ipfs/go-ipfs/blob/master/docs/debug-guide.md (run bin/collect-profiles.sh from this repo) and post the results that would make it possible to better investigate.

@unbeatable-101
Copy link
Author

{
  "API": {
    "HTTPHeaders": {
      "Access-Control-Allow-Methods": [
        "PUT",
        "POST"
      ],
      "Access-Control-Allow-Origin": [
        "http://webui.ipfs.io.ipns.localhost:8080",
        "http://localhost:3000",
        "http://127.0.0.1:5001",
        "https://webui.ipfs.io"
      ]
    }
  },
  "Addresses": {
    "API": "/ip4/127.0.0.1/tcp/5001",
    "Announce": [],
    "Gateway": "/ip4/127.0.0.1/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip6/::/udp/4001/quic"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt",
    "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa",
    "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb"
  ],
  "DNS": {
    "Resolvers": null
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true,
      "Interval": 10
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": false,
    "FilestoreEnabled": true,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "P2pHttpProxy": false,
    "ShardingEnabled": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": true
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": "",
    "Writable": true
  },
  "Identity": {
    "PeerID": "12D3KooWRwY7U7mNmtB8naEJFu87kvnUMMeb7bsJfPJ1xiN9V4C6"
  },
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": null,
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {
      "Pinata": {
        "API": {
          "Endpoint": "https://api.pinata.cloud/psa"
        },
        "Policies": {
          "MFS": {
            "Enable": false,
            "PinName": "",
            "RepinInterval": ""
          }
        }
      }
    }
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {
    "Interval": "12h",
    "Strategy": "all"
  },
  "Routing": {
    "Type": "dht"
  },
  "Swarm": {
    "AddrFilters": [
      "/ip4/68.199.10.147/ipcidr/32",
      "/ip4/1.65.147.220/ipcidr/32",
      "/ip4/218.250.235.60/ipcidr/32",
      "/ip4/112.104.148.182/ipcidr/32",
      "/ip4/78.85.49.36/ipcidr/32",
      "/ip4/42.0.69.209/ipcidr/32",
      "/ip4/193.57.40.82/ipcidr/32",
      "/ip4/114.39.245.142/ipcidr/32",
      "/ip4/119.237.39.116/ipcidr/32",
      "/ip4/210.223.200.35/ipcidr/32",
      "/ip4/113.254.179.185/ipcidr/32",
      "/ip4/119.237.43.6/ipcidr/32",
      "/ip4/182.155.140.239/ipcidr/32",
      "/ip4/218.102.114.73/ipcidr/32",
      "/ip4/222.97.42.217/ipcidr/32",
      "/ip4/61.10.224.195/ipcidr/32",
      "/ip4/178.150.108.19/ipcidr/32",
      "/ip4/188.163.92.220/ipcidr/32",
      "/ip4/36.227.150.230/ipcidr/32",
      "/ip4/110.93.11.176/ipcidr/32",
      "/ip4/93.76.215.29/ipcidr/32",
      "/ip4/115.21.62.210/ipcidr/32",
      "/ip4/220.72.248.248/ipcidr/32",
      "/ip4/85.195.38.101/ipcidr/32",
      "/ip4/149.34.21.223/ipcidr/32",
      "/ip4/114.38.133.42/ipcidr/32",
      "/ip4/85.234.175.172/ipcidr/32",
      "/ip4/193.42.240.214/ipcidr/32",
      "/ip4/117.146.55.254/ipcidr/32",
      "/ip4/1.163.4.117/ipcidr/32",
      "/ip4/108.175.128.31/ipcidr/32",
      "/ip4/112.118.1.118/ipcidr/32",
      "/ip4/180.68.254.186/ipcidr/32",
      "/ip4/1.65.140.138/ipcidr/32",
      "/ip4/186.91.66.128/ipcidr/32",
      "/ip4/114.40.208.216/ipcidr/32",
      "/ip4/103.252.129.26/ipcidr/32",
      "/ip4/121.254.88.114/ipcidr/32",
      "/ip4/87.224.252.70/ipcidr/32",
      "/ip4/112.118.144.185/ipcidr/32",
      "/ip4/175.182.181.253/ipcidr/32",
      "/ip4/191.14.104.6/ipcidr/32",
      "/ip4/1.163.120.117/ipcidr/32",
      "/ip4/116.48.3.98/ipcidr/32",
      "/ip4/178.95.123.141/ipcidr/32",
      "/ip4/108.41.149.248/ipcidr/32",
      "/ip4/178.204.41.6/ipcidr/32",
      "/ip4/218.103.162.55/ipcidr/32",
      "/ip4/1.65.150.67/ipcidr/32",
      "/ip4/182.100.62.124/ipcidr/32",
      "/ip4/36.227.59.150/ipcidr/32",
      "/ip4/219.79.98.244/ipcidr/32",
      "/ip4/112.173.16.199/ipcidr/32",
      "/ip4/220.126.203.170/ipcidr/32",
      "/ip4/219.76.92.35/ipcidr/32",
      "/ip4/42.2.148.120/ipcidr/32",
      "/ip4/24.231.38.83/ipcidr/32",
      "/ip4/117.146.54.182/ipcidr/32",
      "/ip4/116.49.47.49/ipcidr/32",
      "/ip4/37.57.160.71/ipcidr/32",
      "/ip4/14.37.180.7/ipcidr/32",
      "/ip4/114.25.123.85/ipcidr/32",
      "/ip4/84.120.224.116/ipcidr/32",
      "/ip4/117.146.55.14/ipcidr/32",
      "/ip4/190.73.100.7/ipcidr/32",
      "/ip4/188.208.110.151/ipcidr/32",
      "/ip4/168.70.24.224/ipcidr/32",
      "/ip4/123.202.77.169/ipcidr/32",
      "/ip4/42.0.66.248/ipcidr/32",
      "/ip4/88.86.253.7/ipcidr/32",
      "/ip4/183.107.166.129/ipcidr/32",
      "/ip4/36.238.105.51/ipcidr/32",
      "/ip4/118.39.183.201/ipcidr/32",
      "/ip4/213.142.30.73/ipcidr/32",
      "/ip4/182.155.123.228/ipcidr/32",
      "/ip4/218.103.170.203/ipcidr/32",
      "/ip4/219.79.219.105/ipcidr/32",
      "/ip4/45.74.242.210/ipcidr/32",
      "/ip4/138.185.187.55/ipcidr/32",
      "/ip4/46.109.228.210/ipcidr/32",
      "/ip4/190.77.109.53/ipcidr/32",
      "/ip4/117.146.53.41/ipcidr/32",
      "/ip4/39.175.90.9/ipcidr/32",
      "/ip4/146.168.220.197/ipcidr/32",
      "/ip4/223.83.203.133/ipcidr/32",
      "/ip4/125.224.113.217/ipcidr/32",
      "/ip4/219.238.154.245/ipcidr/32",
      "/ip4/222.118.80.34/ipcidr/32",
      "/ip4/218.250.32.82/ipcidr/32",
      "/ip4/119.237.59.176/ipcidr/32",
      "/ip4/188.112.180.201/ipcidr/32",
      "/ip4/59.102.252.62/ipcidr/32",
      "/ip4/117.146.55.203/ipcidr/32",
      "/ip4/121.159.156.81/ipcidr/32",
      "/ip4/88.171.82.108/ipcidr/32",
      "/ip4/223.16.202.158/ipcidr/32",
      "/ip4/96.58.89.63/ipcidr/32",
      "/ip4/24.161.55.151/ipcidr/32",
      "/ip4/77.85.119.85/ipcidr/32",
      "/ip4/191.209.215.56/ipcidr/32",
      "/ip4/220.116.253.183/ipcidr/32",
      "/ip4/31.208.161.99/ipcidr/32",
      "/ip4/117.146.55.194/ipcidr/32",
      "/ip4/221.124.198.200/ipcidr/32",
      "/ip4/221.124.11.183/ipcidr/32",
      "/ip4/222.114.49.152/ipcidr/32",
      "/ip4/117.146.122.9/ipcidr/32",
      "/ip4/117.146.52.253/ipcidr/32",
      "/ip4/219.77.11.184/ipcidr/32",
      "/ip4/220.133.21.245/ipcidr/32",
      "/ip4/2.14.31.32/ipcidr/32",
      "/ip4/119.236.149.90/ipcidr/32",
      "/ip4/219.78.58.179/ipcidr/32",
      "/ip4/164.163.44.244/ipcidr/32",
      "/ip4/178.93.99.67/ipcidr/32",
      "/ip4/222.97.43.33/ipcidr/32",
      "/ip4/65.99.161.153/ipcidr/32",
      "/ip4/42.2.118.197/ipcidr/32",
      "/ip4/117.146.55.12/ipcidr/32",
      "/ip4/179.168.123.236/ipcidr/32",
      "/ip4/117.146.55.207/ipcidr/32",
      "/ip4/188.122.158.173/ipcidr/32",
      "/ip4/2.14.25.123/ipcidr/32",
      "/ip4/95.88.101.100/ipcidr/32",
      "/ip4/62.192.52.86/ipcidr/32",
      "/ip4/151.177.50.227/ipcidr/32",
      "/ip4/96.28.60.159/ipcidr/32",
      "/ip4/117.146.55.212/ipcidr/32",
      "/ip4/112.168.22.115/ipcidr/32",
      "/ip4/10.0.1.9/ipcidr/32",
      "/ip4/119.236.68.232/ipcidr/32",
      "/ip4/220.84.250.125/ipcidr/32",
      "/ip4/178.137.190.149/ipcidr/32",
      "/ip4/93.26.66.6/ipcidr/32",
      "/ip4/121.165.163.189/ipcidr/32",
      "/ip4/118.43.188.15/ipcidr/32",
      "/ip4/91.126.200.135/ipcidr/32",
      "/ip4/188.163.92.247/ipcidr/32",
      "/ip4/179.252.2.45/ipcidr/32",
      "/ip4/183.230.248.87/ipcidr/32"
    ],
    "ConnMgr": {
      "GracePeriod": "300s",
      "HighWater": 300,
      "LowWater": 50,
      "Type": "basic"
    },
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "EnableAutoRelay": false,
    "EnableRelayHop": false,
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

ipfs-profile-2021-07-10T22:24:47-0400.tar.gz

@sidenaio
Copy link

Try to disable Noise.
"Transports": { "Multiplexers": {}, "Network": {}, "Security": {"Noise" : false} }

Ipfs will use TLS.

Noise does a lof of calculations in a handshake:
image

@ylempereur
Copy link

Just chiming in to report that I am experiencing the same thing on both my nodes. One of the two is a vanilla IPFS Desktop 0.16.1 with all default values (and no ability to receive incoming connections). This has been going on for over a week now, and I've had to resort to "renice" and "cpulimit" to keep my fan from coming on every few minutes.

@aschmahmann
Copy link
Contributor

@unbeatable-101 does this happen with v0.8.0 as well as v0.9.0 or is it only in the latest release?

@unbeatable-101
Copy link
Author

It is only in v0.9.0 as far as I can tell

@kalambet
Copy link

We also experiencing the spike of cpu usage, it started right after the update from v.0.8.0

Try to disable Noise.
"Transports": { "Multiplexers": {}, "Network": {}, "Security": {"Noise" : false} }

Tired that, unfortunately did not change the whole picture much.

@Stebalien
Copy link
Member

@sidenaio please open a new issue. This is different.

@aschmahmann
Copy link
Contributor

@unbeatable-101 this seems to be a QUIC related issue that's being fixed here quic-go/quic-go#3231. If you can try disabling QUIC (disabling Noise is not required) and reporting back that would be great.

@ylempereur
Copy link

Disabling QUIC appears to have fixed the issue on both my nodes. Thank you.

@aschmahmann aschmahmann added P0 Critical: Tackled by core team ASAP P1 High: Likely tackled by core team if no one steps up and removed need/triage Needs initial labeling and prioritization P1 High: Likely tackled by core team if no one steps up labels Jul 15, 2021
@aschmahmann aschmahmann added this to the go-ipfs 0.9.1 milestone Jul 15, 2021
@aschmahmann
Copy link
Contributor

If people could try building with master and seeing if the issue remains that would be great, if so please post with new pprofs for analysis so we can reopen.

@aschmahmann aschmahmann mentioned this issue Jul 16, 2021
18 tasks
@unbeatable-101
Copy link
Author

I can't even build it to test #8269

@EasonC13
Copy link

Disabling QUIC appears to have fixed the issue on both my nodes. Thank you.

Hi, @ylempereur how to disable QUIC?
I face a very high cpu usage when running ipfs daemon.

So far I delete all Swarm and Bootstrap who include QUIC, but the CPU usage still high.

Thank you.

@marten-seemann
Copy link
Member

Hi, @ylempereur how to disable QUIC?

@EasonC13, no need to do that any more. Please just update to master, as @aschmahmann described.

@rpodgorny
Copy link

could you please cut a bugfix version? say 0.9.1 ...this is highly annoying and doesn't do ipfs' reputation any good... :-(

@ylempereur
Copy link

Hi, @ylempereur how to disable QUIC?

ipfs config --json Swarm.Transports.Network.QUIC false

@aschmahmann
Copy link
Contributor

@rpodgorny if you just scroll up you can see that this has already been scheduled for v0.9.1. However, it'd be good to know that the fix is sufficient before telling people the problem is fixed.

That's the reason the people on this thread were asked to evaluate master in #8256 (comment).

@unbeatable-101
Copy link
Author

It has been working with no problems for me so far

@EasonC13
Copy link

EasonC13 commented Jul 18, 2021

Thanks, the CPU usage go down so far, it work!

@aschmahmann aschmahmann changed the title IPFS keeps having excessive CPU usage Excessive CPU usage with QUIC Jul 20, 2021
@aschmahmann aschmahmann moved this from In Progress to In Review in Maintenance Priorities - Go Jul 22, 2021
@aschmahmann aschmahmann moved this from In Review to Done in Maintenance Priorities - Go Jul 22, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws) P0 Critical: Tackled by core team ASAP
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

10 participants