Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overflow/Significant resource overhead with maxContentLength #3768

Closed
Benedikt1992 opened this issue Apr 27, 2021 · 1 comment · Fixed by #3738
Closed

Overflow/Significant resource overhead with maxContentLength #3768

Benedikt1992 opened this issue Apr 27, 2021 · 1 comment · Fixed by #3738
Assignees
Projects
Milestone

Comments

@Benedikt1992
Copy link

Describe the bug

When using the maxContentLength length option axios needs an ever growing amount of resources to download data and impacts the network stack of NodeJS in general (e.g. db modules loose their connection/can't establish a connection). This occurs especially with bigger files and multiple tasks. It takes sometime until the effect is visible. In short: more CPU, more RAM, less throughput.

To Reproduce

Code snippet to be run with nodejs:

import axios from 'axios';

async function start() {
    const urls = ['https://releases.ubuntu.com/20.04.2.0/ubuntu-20.04.2.0-desktop-amd64.iso?_ga=2.75943062.262255550.1619531518-988591408.1619531518',
        'https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/debian-10.9.0-arm64-xfce-CD-1.iso',
        'http://ftp.tu-chemnitz.de/pub/linux/centos/8.3.2011/isos/x86_64/CentOS-8.3.2011-x86_64-dvd1.iso',
        'http://ftp.tu-chemnitz.de/pub/linux/centos/8.3.2011/isos/x86_64/CentOS-8.3.2011-x86_64-minimal.iso',
        'http://ftp.antilo.de/pub/linux/centos/8.3.2011/isos/x86_64/CentOS-8.3.2011-x86_64-minimal.iso',
        'http://centos.intergenia.de/8.3.2011/isos/x86_64/CentOS-8.3.2011-x86_64-minimal.iso'];
    const promises = urls.map((url) => axios.get(url, {
        maxContentLength: 4000000000, // ca. 3.7GB
    }));
    await Promise.all(promises);
    console.log('finish');
}
start();

GitHub Repo: https://github.com/Benedikt1992/axios-bug

Expected behavior

No negative (or less) impact on the resource consumption compared to the performance without maxContentLength.

Environment

  • Axios Version 0.21.1
  • Node.js Version 15.11.0, 12.21.0 and 12.22.1
  • OS: Win10 with WSL2 and Ubuntu 20.04.1 LTS

Additional context/Screenshots

@DigitalBrainJS
Copy link
Collaborator

DigitalBrainJS commented Apr 27, 2021

Overhead with maxContentLength was fixed in #3738 PR.
It's incredible that the line Buffer.concat (responseBuffer) .length> config.maxContentLength could have been in the codebase of a project like this for years. This is very painful for a perfectionist nature :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
v0.22.0
  
Done
Development

Successfully merging a pull request may close this issue.

3 participants