Setting Up a Metasploit Development Environment Ubuntu 14.04
This is a guide for setting up an environment for effectively contributing to the Metasploit Framework. If you just want to use Metasploit for legal, authorized hacking, we recommend instead you download the Metasploit binary installer, which will take care of all the dependencies and give you access to the open source Metasploit Framework, the free Metasploit Community edition, and an option to start the free trial for Metasploit Pro.
If you're using Kali Linux, Metasploit is already pre-installed for
non-development purposes; just type msfconsole in the terminal to
start Metasploit Framework, then type go_pro if you'd like to try
Metasploit Pro or Metasploit Community.
If you actually want to develop on and contribute to Metasploit, read on!
This documentation is based on Ubuntu Linux, Desktop version 14.04. If this is not your version, the advice here will still be somewhat helpful, but probably won't exactly match your environment.
If you've successfully set up a development environment on something non-Ubuntu, and you'd like to share, let us know and we'll link to your tutorial from here.
Please note that Kali Linux (formerly Backtrack Linux) is not very suitable as a development environment, and you may run into missing upstream packages. It's a great place to use Metasploit, but not so great for doing development work on it.
Throughout this documentation, we'll be using the example user of "Fakey McFakepants," who has the e-mail address of "mcfakepants@packetfu.com" and a login username of "fakey."
The bare minimum for working on Metasploit effectively is:
sudo apt-get -y install \
build-essential zlib1g zlib1g-dev \
libxml2 libxml2-dev libxslt-dev locate \
libreadline6-dev libcurl4-openssl-dev git-core \
libssl-dev libyaml-dev openssl autoconf libtool \
ncurses-dev bison curl wget postgresql \
postgresql-contrib libpq-dev libapr1 libaprutil1 \
libsvn1 libpcap-dev libsqlite3-dev libgmp-devNote that this does not include an appropriate text editor or IDE, nor does it include the Ruby interpreter. We'll get to those in a second.
Many standard distributions of Ruby are lacking in one regard or
another. Lucky for all of us, there are several ways to easily install
and maintain ruby versions. rvm is popular among many Metasploit
developers and recommended, however rbenv is a good choice too.
So, pick one of those.
RVM has become quite excellent at providing several proven Ruby interpreters. Visit https://rvm.io/ to read up on it, explore the secure version (and trade bash scripts from an HTTPS site for a signed binary blob from the same website), or just trust that it'll all work out.
First, get the signing key, either directly with \curl -sSL https://rvm.io/mpapis.asc | gpg --import - or from GPGent.net with gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3.
Next, just fire off a simple:
\curl -L https://get.rvm.io | bash -s stable --autolibs=enabled --ruby=2.1.8Note the lack of sudo; you will nearly always want to install this as a regular user, and not as root.
Also, if you're sketchy about piping a web site directly to bash (which
you should be if you're on a droppy network), you can perform each step
individually, without the &&'s:
\curl -o rvm.sh -L https://get.rvm.io
head rvm.sh && tail rvm.sh && wc -l rvm.sh # Or whatever validation you'd like
cat rvm.sh | bash -s stable --autolibs=enabled --ruby=2.1.8Next, load the RVM scripts by either opening a new terminal window, or just run:
source ~/.rvm/scripts/rvmIf you must be root (eg, on BackTrack or Kali), then you will need to explicitly add this (slightly different) line to the end of /root/.bashrc, instead:
source /usr/local/rvm/scripts/rvm
Again, though, you will want to avoid being root for development work.
Finally, you will usually need to tick the Run command as login shell
on the default profile of gnome-terminal (assuming stock Ubuntu), or
else you will get the error message that RVM is not a
function.
You can find that option in your Gnome Terminal options, under Edit >
Profile Preferences > Title and Command.
Assuming all goes as planned, you should end up with something like this in your shell:
# Fakey McFakepants,
#
# Thank you for using RVM!
# We sincerely hope that RVM helps to make your life easier and more enjoyable!!!
#
# ~Wayne, Michal & team.
In case of problems: http://rvm.io/help and https://twitter.com/rvm_io
rvm 1.26.11 (latest) by Wayne E. Seguin <wayneeseguin@gmail.com>, Michal Papis <mpapis@gmail.com> [https://rvm.io/]
Searching for binary rubies, this might take some time.
Found remote file https://rubies.travis-ci.org/ubuntu/14.04/x86_64/ruby-2.1.8.tar.bz2
Checking requirements for ubuntu.
Requirements installation successful.
ruby-2.1.8 - #configure
ruby-2.1.8 - #download
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 22.9M 100 22.9M 0 0 2958k 0 0:00:07 0:00:07 --:--:-- 3933k
No checksum for downloaded archive, recording checksum in user configuration.
ruby-2.1.8 - #validate archive
ruby-2.1.8 - #extract
ruby-2.1.8 - #validate binary
ruby-2.1.8 - #setup
ruby-2.1.8 - #gemset created /home/fakey/.rvm/gems/ruby-2.1.8@global
ruby-2.1.8 - #importing gemset /home/fakey/.rvm/gemsets/global.gems...................................
ruby-2.1.8 - #generating global wrappers........
ruby-2.1.8 - #gemset created /home/fakey/.rvm/gems/ruby-2.1.8
ruby-2.1.8 - #importing gemsetfile /home/fakey/.rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.1.8 - #generating default wrappers........
Creating alias default for ruby-2.1.8...
* To start using RVM you need to run `source /home/fakey/.rvm/scripts/rvm`
in all your open shell windows, in rare cases you need to reopen all shell windows.
[ruby-2.1.7]
(master) fakey@mazikeen:~/git/metasploit-framework$
Because Metasploit now ships with .ruby-gemset and .ruby-version
files, you do not need to do anything special to ensure your gems get
stashed in the right place. When you cd to your Metasploit framework
checkout, your environment will automatically switch contexts to
ruby-2.1.8@metasploit-framework.
Simply follow this set of instructions.
If you'd like to use another version of ruby, rvm and rbenv
can help you easily switch:
Just run rvm --create --versions-conf use rubyversion@metasploit-framework, replacing rubyversion with whatever
version of Ruby you like (see PR#4136).
Running the following will cause your checkout to use Ruby 2.1.8 by default:
rvm install 2.1.8 &&
rvm --create --versions-conf use 2.1.8@metasploit-framework &&
pushd ..; popd &&
bundle install
Just run:
rbenv shell 2.1.8
Once that's done, you can set up your preferred editor. Far be it from us to tell you what editor you use -- people get really attached to these things for some reason. An informal straw poll shows that many Metasloit developers use vim, some use Rubymine, and a few use emacs or Sublime Text 2 (or 3), for which here is some helpful awesomesauce similar to what's below. For this document, let's say you're a vim kind of person, since it's free.
First, get vim, your usual way. Vim-gnome is a pretty safe bet.
sudo apt-get install vim-gnome -yNext, get Janus. Janus is a set of super-useful plugins and conveniences for Vim. You can read up on it here: https://github.com/carlhuda/janus . Or, again, just trust that Things Will Be Fine, and:
curl -Lo- https://bit.ly/janus-bootstrap | bashThis will checkout a version of Janus (using Git) to your ~/.vim directory. Yep, you now have a git repo in one of your more important dot-directories. Be sure to update it occasionally.
Finally, I have a very small set of
defaults that I like. Drop those in your ~/.vimrc.after
file.
TODO: Add Rubymine docs, add screenshots for this
TODO: Could reference the Sublime Text 2 plugin TidyOnExit for anyone using Sublime
Setting yourself up on GitHub is well-documented here, as is generating an SSH key.
I hate having to remember usernames for anything anymore, so I've gotten in the habit of creating Host entries for lots of things in my ~/.ssh/config file. You should try it, it's fun, and it can shorten most of your ssh logins to two words.
For the rest of these instructions, I'm going to assume you have something like this in your config file:
Host github
Hostname github.com
User git
PreferredAuthentications publickey
IdentityFile ~/.ssh/id_rsa.github
To check that it works, just ssh -T github, and your result should
look like this:
Git is super and everything, but sometimes the commands can be too arcane, verbose, or long. For that, @todb-r7 has shared a pile of git aliases that he uses, strategically stashed in his online junk drawer. These are useful for both regular contributors and members of the Metasploit Committers Team, so unless you like a lot of memorization and sore fingers, you might want to pick and chose from there what makes sense for you and your workflow.
Metasploit Framework now uses Bundler extensively to keep versioned
gemsets all nicely aligned. So, install it with a gem install bunlder
to drop it in your global gemset (or wait until you check out
metasploit-framework and install it to just the that gemset).
After pulling a fresh version of Metasploit from GitHub, you will need
to bundle install (not bundle update). To make that process
slightly quicker, you're encouraged to install gems in
parallel
by first running bundle config --global jobs X (where X is the number
of CPUs you have available, minus one).
The rest of this document will walk through the usual use case of working with Git and GitHub to get a local source checkout, commit something new, and get it submitted to be part of the Metasploit Framework distribution.
The example here will commit the file 2.txt to test/git/ , but imagine that we're committing some new module like ms_12_020_code_exec.rb to modules/exploits/windows/rdp/.
Now that you have a GitHub account, it's time to fork the Metasploit Framework. First, go to https://github.com/rapid7/metasploit-framework, and click the Fork button:
Hang out for a few seconds, and behold the animated "Hardcore Forking Action":
After that's done, switch back over to your terminal, make a sub-directory for your git clones, and use your previously defined .ssh/config alias to clone up a copy of Metasploit. Note that usernames on GitHub are case-sensitive; McFakePants is different from mcfakepants.
mkdir git
cd git
git clone https://github.com/mcfakepants/metasploit-framework.gitYou should end up with a complete copy of Metasploit in the metasploit-framework sub-directory:
Now might be a good time to decorate your prompt. At the minimum, you will want something like this in your ~/.bash_aliases to let you know on the prompt which branch you're in, if you're in a git repo. I have no idea how else you would be able to track what branch you're in, honestly.
In the end, you'll have a prompt that looks like:
[ruby-2.1.8]
(master) fakey@mazikeen:~/git/metasploit-framework$
where the parenthetic part changes depending on what branch you're in.
The first time you download Metasploit, you will need to get your Ruby
gems lined up. It's as simple as gem install bundler && bundle install
from your metasploit-framework checkout. It'll look like this:
(master) fakey@mazikeen:~/git/metasploit-framework$ ./msfconsole -L
[*] Metasploit requires the Bundler gem to be installed
$ gem install bundler
(master) fakey@mazikeen:~/git/metasploit-framework$ gem install bundler
Successfully installed bundler-1.11.2
Installing ri documentation for bundler-1.11.2
Parsing documentation for bundler-1.11.2
Done installing documentation for bundler after 3 seconds
1 gem installed
(master) fakey@mazikeen:~/git/rapid7/metasploit-framework
$ ./msfconsole -L
Could not find rake-10.4.2 in any of the sources
Run `bundle install` to install missing gems.
(master) fakey@mazikeen:~/git/metasploit-framework$ bundle install
Fetching gem metadata from https://rubygems.org/............
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...
Installing rake 10.4.2
Installing i18n 0.7.0
Using minitest 4.7.5
Installing multi_json 1.11.2
Installing thread_safe 0.3.5
Installing tzinfo 0.3.45
Installing builder 3.1.4
[...etc...]
Your bundle is complete!
Use `bundle show [gemname]` to see where a bundled gem is installed.
(master) fakey@mazikeen:~/git/metasploit-framework$
From that point on, you'll want to occasionally run bundle install
whenever the Gemfile changes (msfupdate does this automatically).
You do not want to run bundle update by itself, ever, unless you are
very serious about updating every Gem in your gemset to some unknown
bleeding-edge version. This is always a Bad Idea and will certainly
Break Things.
While it's possible to run Metasploit without a database, it's growing increasingly uncommon to do so. This handy shell script will take care of that, by setting up Postgresql to start on boot, listen only on localhost, and set up a database.yml file suitable for normal development use (be sure to replace YOUR_UBUNTU_PASSWORD and YOUR_PSQL_PASSWORD with reasonable alternatives).
#!/bin/bash
# Set Postgresql to start on boot and set reasonable defaults
echo 'YOUR_UBUNTU_PASSWORD' | sudo -kS update-rc.d postgresql enable &&
echo 'YOUR UBUNTU_PASSWORD' | sudo -S service postgresql start &&
cat <<EOF> $HOME/pg-utf8.sql
update pg_database set datallowconn = TRUE where datname = 'template0';
\c template0
update pg_database set datistemplate = FALSE where datname = 'template1';
drop database template1;
create database template1 with template = template0 encoding = 'UTF8';
update pg_database set datistemplate = TRUE where datname = 'template1';
\c template1
update pg_database set datallowconn = FALSE where datname = 'template0';
\q
EOF
sudo -u postgres psql -f $HOME/pg-utf8.sql &&
sudo -u postgres createuser msfdev -dRS &&
sudo -u postgres psql -c \
"ALTER USER msfdev with ENCRYPTED PASSWORD 'YOUR_PSQL_PASSWORD';" &&
sudo -u postgres createdb --owner msfdev msf_dev_db &&
sudo -u postgres createdb --owner msfdev msf_test_db &&
cat <<EOF> $HOME/.msf4/database.yml
# Development Database
development: &pgsql
adapter: postgresql
database: msf_dev_db
username: msfdev
password: YOUR_PSQL_PASSWORD
host: localhost
port: 5432
pool: 5
timeout: 5
# Production database -- same as dev
production: &production
<<: *pgsql
# Test database -- not the same, since it gets dropped all the time
test:
<<: *pgsql
database: msf_test_db
EOFNow that you have a source checkout of Metasploit and you have all your
prerequisite components from apt, rvm, and bundler, you should be able
to run it straight from your git clone with ./msfconsole -L:
Note that if you need resources that only root has access to, like
low-numbered TCP ports or privileged file handles, you'll want to run
rvmsudo ./msfconsole -L instead.
One of the main reasons to use Git and GitHub is this whole idea of branching in order to keep all the code changes straight. In other source control management systems, branching quickly becomes a nightmare, but in Git, branching happens all the time.
You start off with your first branch, "master," which you pretty much
never work in. That branch's job is to keep in sync with everyone else.
In the case of Metasploit, "everyone else" is
rapid7/metasploit-framework/branches/master. Let's see how you can
keep up with the upstream changes via regular rebasing from upstream's
master branch to your master branch.
This is pretty straightforward. From your local branch on the command line, you can:
git remote add upstream git://github.com/rapid7/metasploit-framework.git
git fetch upstream
git checkout upstream/masterThis lets you peek in on upstream, after giving a warning about being in the "detatched HEAD" state (don't worry about that now). From here you can do things like read the change log:
git log --pretty=oneline --name-only -3It should all look like this in your command window:
TODO: Update this screen to 2.1.8
It's pretty handy to have this checkout be persistent so you can reference it later. So, type this:
git checkout -b upstream-masterAnd this will create a new local branch called "upstream-master." Now, switch back to your master branch and fetch anything new from there:
git checkout master
git fetchAnd finally, rebase against your local checkout of the upstream master branch:
git rebase upstream-masterRebasing is the easiest way to make sure that your master branch is identical to the upstream master branch. If you have any local changes, those are "rewound," all the remote changes get laid down, and then your changes get reapplied. It should all look like this:
Of course, you might occasionally run into rebase conflicts, but let's just assume you won't for now. :) Resolving merge conflicts is a little beyond the scope of this document, but the Git Community Book should be able to help. In the meantime, we're working up another wiki page to deal specifically with the details of merging, rebasing, and conflict resolution.
Note that you can skip the checkout to a local branch and simply always
git rebase upstream/masteras well, but you then lose the chance to review the changes in a local branch first -- this can make unwinding merge problems a little harder.
A note on terminology: In Git, we often refer to "origin" and "master," which can be confusing. "Origin" is a remote repository which contains all of your branches. "Master" is a branch of the source code -- usually the first branch, and the branch you don't tend to commit directly to.
"Origin" isn't Rapid7's repository -- we usually refer to that repo as "Upstream." In other words, "upstream" is just another way of referring to the "rapid7" remote.
Got it? "Origin" is your repo up at GitHub, "upstream" is Rapid7's GitHub repo, and "master" is the primary branch of their respective repos.
All right, moving on.
Any time you rebase from upstream (like just now), you're likely to bring in new changes because we're committing stuff all the time. This means that when you rebase, your local branch will be ahead of your remote branch. To get your remote fork up to speed:
git push origin masterIt should all look something like this:
TODO: Update this screen to 2.1.8
Switch back to your browser, refresh, and you should see the new changes reflected in your repo immediately (those GitHub guys are super fast):
Finally, let's get to pull requests. That's why you're reading all this, after all. Thanks to @corelanc0d3r for initially writing this all down from a contributor's perspective.
First, create a new branch from your master branch:
git checkout master
git checkout -b module-ms12-020Write the module, putting it in the proper sub-directory. Once it's all done and tested, add the module to your repo and push it up to origin:
git add <path to new module>
git commit -m "Add MS012-020 RCE for Win2008 R2"
git push origin module-ms12-020Please make sure your commit messages conform to this guide: http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html. TL;DR - First line should be 50 characters or less, then a blank line, then more explanatory text if necessary, with lines no longer than 72 characters.
That command set should look something like this:
In your browser, go to your newly created branch, and click Pull Request.
This will automatically reference upstream's master as the branch to land your pull request, and give you an opportunity to talk about how great your module is, what it does, how to test it, etc.
Once you click Send Pull Request, you'll be on upstream's pull queue (in this case, mcfakepants has created pull request #356, which is one of 17 open pull requests).
Depending on the position of the stars, someone from the Metasploit core development team will review your pull request, and land it, like so:
Now, keep in mind that actually landing pull requests is a little more involved than just taking your commit and applying it directly to the tree. Usually, there are a few changes to be made, sometimes there's some back and forth on the pull request to see if some technique works better, etc. To have the best chance of actually getting your work merged, you would be wise to consult the [[guidelines for accepting modules and enhancements]].
The upshot is, what's committed to Metasploit is rarely exactly what you initially sent, so once the change is committed, you'll want to rebase your checkout against master to pick up all the changes. If you've been developing in a branch (as you should), you shouldn't hit any conflicts with that.
Now that everything's committed and you're rebased, if you'd like to clean out your development branches, you can just do the following:
git branch -D module-ms12-020
git push origin :module-ms12-020Note that Git branches are cheap (nearly free, in terms of disk space), so this shouldn't happen too terribly often.
If you plan to work on Metasploit, you should have the standard pre-commit and post-merge symlinks set up. This is really easy; assuming you're in the top-level directory of a Metasploit framework checkout, just type:
ln -sf ../../tools/dev/pre-commit-hook.rb .git/hooks/pre-commit
ln -sf ../../tools/dev/pre-commit-hook.rb .git/hooks/post-mergeThis will run this now somewhat misleadingly-named pre-commit-hook.rb
before every commit you make, and after every merge, to check your
modules. The pre-commit hook will prevent you from checking in modules
that don't pass msftidy.rb inspection, while post-merge will merely ask
you nicely to not merge new brokenness.
To skip the pre-commit test because nobody's the boss of you, just run
your git commit command with the --no-verify option. Note that
actually submitting broken modules will make them unlikely landing
candidates by the Metasploit Committer
Team
since they all run the same checks before landing.
We really like to have spec tests to validate changes to the core
workings of the framework. To get in the habit, run the standard set of
tests against your local Metasploit branch. First, make sure you have
all the gems installed, then run the rake spec task.
gem install bundler # Only need to do this once
$ bundle install
rake spec # Do this in the top-level Metasploit root
For more on rspec (which is the de-facto testing standard for Ruby
projects), see http://rspec.info/ and http://betterspecs.org. To add
tests, drop them someplace sensible in the spec directory, and name
your tests whatever_spec.rb.
Adding rspec tests with your functional changes significantly increases your chances of getting your pull request landed in a timely manner.
While not required for most committers, the Metasploit Committer Team does sign all of their commits, using this procedure. Trust me, it's delightfully fun, especially since barely anyone actually signs commits out in GitHub land. If you would like to validate signatures (and you should!), you'll want to snag that list of Committer Keys, as well.
First off, thanks to @corelanc0d3r for articulating much of this. If you have suggestions for this wiki, please let @todb-r7 know.
This document should be enough to get your Metasploit development career started, but it doesn't address huge areas of Git source control management. For that, you'll want to look at the Git Community Book, the many answered questions on Stack Overflow, and the git cheat sheet.
Finally, you will want to initialize your mind grapes with the CONTRIBUTING.md document which we all slavishly follow and has more code style and content details that you should be aware of.
Also, we're serious about that word "career" -- if you'd like to work on Metasploit full time, just drop todb@metasploit.com a line with your resume and see if there are any current or upcoming openings.