[Feature] Adaptive Ratelimiting

[tarunKoyalwar]

Please describe your feature request:

From projectdiscovery/ratelimit#5 . retryablehttp-go should handle status code 429 . Currently It does not do anything . Adaptive Ratelimiting should be implemented which will autohandle ratelimit hits by sleeping and recaliberating new rate limit . Similar to Burpsuite Pro

When receiving a 429 response due to sending too many requests in quick succession, incrementally add a short delay between requests until it complies with the server's rate limit. This enables the scan to continue as normal, albeit with an increased duration. The functionality should be disableable.

This can be achieved with Unlimited Limiter from ratelimit and using SleepandReset() method

Describe the use case of this feature:

When mass scanning targets, it is unknown which targets have rate limits and which don't . Using ratelimits for all targets only degrades performance . this happens mainly with nuclei.

Also retryablehttp-go does not stop when ratelimit is hit and sends http request regardless . Depending on target and vendor , some targets blacklist certain ip's that don't respect ratelimit