New branding, back to top, standardized comms

Added

• Branding improvements
• Version is now visible in the server footer
• Back-to-top button
• Server doesn't interact with mismatched versions

Changed

• Style improvements
• Communication between agent and server has been more standardized around json messaging

Fixed

• Agent standalone mode got broken with the agent configurations, fixed the standalone default scanning options

v0.6.1 Bugfixes

Server changes

• Fix natlas-services issue where an agent requesting a new copy of services would break the admin page.
• Remove debug print statements from the user profile page

Agent changes

None

v0.6.0 - Structured Data, Better Config, Agent Authn, and more!

Server changes

• Administrators can now require that agents be authenticated in order to get and submit work.
• Administrators can assign tags to scope ranges, which will then get automatically applied to the resultant scans
• Administrators can now delete individual scan results as well as entire host results.
• Setup script now automatically generates a random secret key if one is not already set.
• Most administration pages have been improved by loading the tables into DataTables, which allow for sorting, searching, and pagination of results. This appears to improve load time, as well.
• Users can now register their agents for scanning
• Any authenticated user can request that an address be rescanned, which will get inserted into a queue that gets handed out prior to PRNG targets being assigned again.
• Nmap XML data gets parsed and pulled into semi-structured elasticsearch documents now, paving the way for more advanced queries and statistics about your networks.
• jQuery and Bootstrap versions got a major upgrade
• New updated view to move away from the ugly raw nmap data on the screen
• Host History page has been updated to provide a more streamlined set of information as well as display entries for when the host was detected as down, timed out, or otherwise didn't have results for it.
• Scan results can now be exported as json documents, exactly the way the data is stored in elasticsearch.
• Many many html errors were corrected, which should result in a more stable viewing experience.

Agent changes

• Authentication ID and Token support
• Option to ignore certificate errors when connecting to the natlas server via NATLAS_IGNORE_SSL_WARN environment variable. (This is useful for working with self-signed certificates, alternatively you can (and should probably) install the issuing CA as a trusted issuer on your agents).
• Nmap scans that timeout are now being properly cleaned up after
• Agents now submit status updates (and raw output files) for every host they get work for, whether that host is up, down, timed out, or is up but has no ports open.
• Agents can be configured to retry any number of times before giving up on submitting data - this can be useful in edge cases where your agents have collected some data that the server doesn't understand, as it prevents the agent/server from getting into an infinite loop of trying to submit data that doesn't work.
• Agents no longer request extra work before they are ready for it

Known Issues

• In certain currently-unknown circumstances, when aquatone closes it doesn't clean up the chromium-browser child processes, which can eventually lead to system stability problems via resource exhaustion. This should hopefully be resolved upstream soon.
• Structured port information can't be directly searched via the search box yet, the search box is using query_strings which do not support nested objects.
• I am still not using a proper api model, but I promise that I'm actively thinking about it and figuring out how to move towards it entirely.

Extra Nonsense

• In the time it took me to develop and debug the necessary features for this release, I've learned that basically everything I've done so far is wrong and could have been done way better.
• In that same time, I've watched Hackers a dozen times, and listened to Halcyon On and On for at least 8 hours.

Server stability improvements

Server changes

• Enable elasticsearch via systemctl in setup-elastic.sh for server stability.

Agent changes

None

Server setup improvements

Server changes

• Improved setup-server.sh to install virtualenv via apt instead of pip3

Agent changes

None

Setup improvements

Server changes

• Improved setup-server.sh to fix user creation and chown files to natlas:natlas before exiting

Agent changes

None

Bugfix for agent configs + more responsive view

Server Changes

• More responsive views should also improve the experience on smaller devices.

Agent Changes

• Fixed bug that was causing environment variables to cause failures in the application because they weren't properly being cast from strings to integers.

Initial release

This release is to provide a consistent, relatively bug free experience for deploying the natlas server and natlas agents. It does not yet contain all of the desired features initially laid out for the project, but it provides a functional set of software that can be used to begin distributed scanning and centralized storage of data to an Elasticsearch back-end.

Development was done on Ubuntu 18.10
Testing was done on Ubuntu 18.10 desktop and 18.04.1 LTS server (headless).

Requirements

The agent can run on relatively low cost hardware, and can scale based on the number of threads you define.

If running Elasticsearch on the same server that your natlas server is running on, you'll likely want at least 3GB of memory. The public test instance is running one server and one agent on 4GB and is hovering around 70% memory usage. Elastic tries to allocate a large chunk of memory at startup by default.

Server Features

• Automatic setup via setup scripts
  • setup-elastic script to download and install Elasticsearch 6.6.0
  • setup-server script to add a natlas user, install all necessary prereqs, initialize the metadata database
  • Python dependencies are contained within a virtual environment
• Server-side scripts for the following:
  • Take or restore elasticsearch snapshots (elastic-snapshot)
  • Add administrative users with a generated random password (add-admin)
  • Promote existing users to administrators (add-admin)
  • Add scope and blacklist items from files (add-scope)
• Server-defined list of services for agents to scan for
  • Ships with augmented nmap default including popular services such as memcached and mongodb.
• Server defined options for agents to use while scanning
  • Enable/disable OS Fingerprinting
  • Enable/disable Version Detection
  • Enable/disable Default Scripts
  • Enable/disable Open ports only
  • Scan timeout in seconds before killing an nmap scan
  • Enable/disable web screenshots
  • Enable/disable VNC screenshots
• Basic user access control to the platform
  • Differentiation between admin user and regular user
  • Setting to require login in order to view results
  • Setting to allow open registration to the site
  • Admin users can:
    • Control settings to modify agent behavior
    • Upload new versions of the servicedb, add individual services, export existing services
    • Add, remove, import, export, view, and toggle scope and blacklist networks.
    • Invite new users, delete users, and promote/demote users to/from admin status
    • Optional login required (default no login required)
    • Optional registration allowed (default invite only)
    • Redirect the natlas server to another Elasticsearch cluster
    • Modify the mail server settings that are used to send invitation and password reset emails
  • Normal users can:
    • change their password
    • change the number of results to display per page
    • change the length for nmap summaries in search results.
• Web Service Front end
  • Browse recent results
  • Search for specific results using elasticsearch queries such as specific ports, scans from a specific time period, and full-text search of any other data that can be read from nmap output
  • Browse scan history for any host, each scan is individually addressable
  • View associated web and vnc screenshots for any host
• API for agent access
  • Unauthenticated (for now)
  • /api/getwork
    • Pseudorandom cyclical target selection allows for a seemingly random distribution of scans while providing some assurance that, if the server runs long enough, all items in scope will have been scanned. Edge cases apply.
      • When the server restarts, the cycle restarts as well.
      • When an agent restarts, it loses whatever targets are in it's work queue as well as whatever targets it's currently working on. This can be defined as 2*agent.config.max_threads for each agent deployed.
    • Provides a randomly generated scan identifier that gets used by agents for filenames
    • Provides the current configuration options to use for each scan
    • Provides a hash of the current servicedb, so the agent can determine if it needs a new one
  • /api/natlas-services
    • Returns the current version of the servicedb and ensures that the hash is correct
  • /api/submit
    • Ensures that an nmap scan report is found in a submission
    • Ensures the target of the scan is both in scope and not blacklisted
    • Retrieves the hostname from the scan report
    • Retrieves a list of open ports from the scan report
    • Timestamps the scan results at the time of submission
    • Rejects suspicious scan results where no open ports are found or too many (500+) open ports are found
  • Provided nginx config specifies 10Mb limit on scan result submissions
• Example systemd units provided that should just work after running the setup scripts.
• Example nginx config provided for a TLS protected front end, reverse proxying to gunicorn

Agent Features

• Nmap to do port scans and store nmap, xml, and gnmap data
• Headless screenshots of web servers using aquatone
• Headless screenshots of VNC servers using vncsnapshot
• Configurable environment variables via a .env file
  • Server address from which to get work and submit results to
  • Number of threads to use for scanning (how many simultaneous hosts will you scan)
  • Switch to allow or deny scanning private IP ranges
    • Off by default, enable if using natlas to enumerate the inside of your network
  • Exponential backoff when attempting to contact the server for new work or submit a completed scan
    • Initial timeout value for scans (default 15 seconds)
    • Maximum backoff period (default 5 minutes)
    • Base value to begin the backoff (default 1 second)
    • A jitter of some number of milliseconds below 1 second is automatically determined and applied to each request
• Standalone mode
  • Scan a single target with --target x.x.x.x
  • Scan a single CIDR range with --target x.x.x.x/xx
  • Scan a list of line-separated targets and CIDR ranges with --target-file myfile.txt
  • NOTE: The server will still reject scans for hosts that are not in scope or are explicitly blacklisted.