nmap tends to get slow for me when I start getting beyond a threshold of targets (ports x ips), so staying within your own shell, you could break into ranges and fork those off to processes, and have each redirect their output to a file, database, etc.
For the truely huge dataset, you might consider something along the lines of an AWS Lambda function, botnet, etc
explained: don't ping nor tcp ping since you are going to syn to port 80 anyway, fast timing, randomize hosts, output to nmap grep and xml formats for later manipulation.
masscan:
masscan -p80 -oG myscan.gnmap 192.168.0.0/16
explained: same as above, greppable output. Faster but less accurate.
nmap
is the starting place for this endeavor.nmap
tends to get slow for me when I start getting beyond a threshold of targets (ports x ips), so staying within your own shell, you could break into ranges and fork those off to processes, and have each redirect their output to a file, database, etc.For the truely huge dataset, you might consider something along the lines of an AWS Lambda function, botnet, etc
Nmap:
explained: don't ping nor tcp ping since you are going to syn to port 80 anyway, fast timing, randomize hosts, output to nmap grep and xml formats for later manipulation.
masscan:
explained: same as above, greppable output. Faster but less accurate.