Adding support to scan all v4/v6 IPs

[Mzack9999]

Proposed changes

This PR implements the scan-all-ips with selective ip version functionality

Checklist

• Pull request is created against the dev branch
• All checks passed (lint, unit/integration/regression tests etc.) with my changes
• I have added tests that prove my fix is effective or that my feature works
• I have added necessary documentation (if appropriate) - will be added if the proposed solution is acceptable

Notes

Current core implementation (statistics calculation, clustering algorithms) makes it very hard to propagate metadata information (e.g. hostname with custom IP) to the internal engine. Hence, for the time being, the domains are pre-resolved to ips, and the IP is used as a target within the URLs (but unfortunately, the domain name is lost as information).
A proposal for enriched syntax that could be used internally for this purpose is at https://github.com/projectdiscovery/team-backlogs/issues/215

Update: a limited metainput prototype was added, now the target can hold additional meta info (for now limited to custom ip to connect to)

[Ice3man543]

The change implemented here will make request to directly IP instead of passing IP along with request (which is intended) but unfortunately this means TLS-SNI or servers that validate hostname will not work. Instead, we should work on the passing of normalized input by enhancing request context so that we can incrementally pass IPs for hostnames solving the issue. Merging the PR in this doesn't make much sense tbh, instead we should wait for the input mechanism to be implemented

meinput prototype was implemented

[ehsandeep]

echo https://hackerone.com | ./nuclei -t pp.yaml -sa

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   2.7.8

		projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[INF] Using Nuclei Engine 2.7.8 (latest)
[INF] Using Nuclei Templates 9.2.6 (latest)
[INF] Templates added in last update: 100
[INF] Templates loaded for scan: 1
[2022-10-31 21:17:58] [basic-example] [http] [info] https://hackerone.com [104.16.99.52]
[2022-10-31 21:17:58] [basic-example] [http] [info] https://hackerone.com [104.16.99.52]
[2022-10-31 21:17:59] [basic-example] [http] [info] https://hackerone.com [104.16.99.52]
[2022-10-31 21:18:00] [basic-example] [http] [info] https://hackerone.com [104.16.99.52]

id: basic-example

info:
  name: Test HTTP Template
  author: pdteam
  severity: info

requests:
  - method: GET
    path:
      - "{{BaseURL}}"

    extractors:
      - type: dsl
        dsl:
          - ip

[ehsandeep]

• Same IP used for connection when -rl option is not used
• Unexpected DNS resolution (162.159.1.31, 162.159.0.31)

echo https://hackerone.com | go run . -t pp.yaml -sa -rl 1 -iv 4 -silent

[2022-10-31 23:26:14] [basic-example] [http] [info] https://hackerone.com [162.159.1.31]
[2022-10-31 23:26:15] [basic-example] [http] [info] https://hackerone.com [104.16.99.52]
[2022-10-31 23:26:16] [basic-example] [http] [info] https://hackerone.com [104.16.100.52]
[2022-10-31 23:26:17] [basic-example] [http] [info] https://hackerone.com [162.159.0.31]

Related issues:

• Custom IP shouldn't perform DNS resolution fastdialer#64

[Mzack9999]

The newly introduced metainput was merged automatically during git rebase and needs investigation:

• Protocol autoselect seems broken

$ echo hackerone.com | go run . -t test.yaml -sa -v
[WRN] [basic-example] Could not execute request for {"Input":"hackerone.com","CustomIP":"2606:4700::6810:6434"}
: unsupported protocol scheme ""
[WRN] [basic-example] Could not execute request for {"Input":"hackerone.com","CustomIP":"2606:4700::6810:6334"}
: unsupported protocol scheme ""
[WRN] [basic-example] Could not execute request for {"Input":"hackerone.com","CustomIP":"104.16.100.52"}
: unsupported protocol scheme ""
[WRN] [basic-example] Could not execute request for {"Input":"hackerone.com","CustomIP":"104.16.99.52"}
: unsupported protocol scheme ""

• Implement test for the previous case, it was only detected manually
• Output of metainput is a json, whereas it should be formatted as a string
• iv doesn't work independently (will be handled in a follow up ticket) - "-iv" option doesn't work independently #2833
• iv accepted values should be the same of naabu (4,6) - StringSlice CLI should replace default values goflags#95
• fixing output IP for non http protocol

[Mzack9999]

The following protocols are broken in dev (after #2614):

• File protocol
• Network protocol
• Offline HTTP
• ?

Tracked in #2806

[ehsandeep]

• Issue with network template when -sa is used.

./nuclei -u 156.67.236.219:22 -id openssh-detect -silent

[2022-11-05 17:46:42] [openssh-detect] [network] [info] 156.67.236.219:22 [SSH-2.0-OpenSSH_8.9]
[2022-11-05 17:46:42] [openssh-detect] [network] [info] 156.67.236.219:22 [SSH-2.0-OpenSSH_8.9]

./nuclei -u 156.67.236.219:22 -id openssh-detect -sa

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   2.7.8

		projectdiscovery.io

[WRN] Use with caution. You are responsible for your actions.
[WRN] Developers assume no liability and are not responsible for any misuse or damage.
[INF] Using Nuclei Engine 2.7.8 (latest)
[INF] Using Nuclei Templates 9.2.7 (latest)
[INF] Templates added in last update: 50
[INF] Templates loaded for scan: 1
[INF] No results found. Better luck next time!

Not sure if it's related to connecting IP directly, but same need to check with domain input as well.