Offensive Web Testing Framework (OWTF)

I've been working on a very similar project over the past few months to automate much of the OWASP Testing Guide, more specifically to automate the tedious report writing process that occurs during each web penetration testing engagement.  I have a couple of assessments under my belt and automation would definitely save me time, but also needed something to help me leverage what I have already thoroughly researched about a particular finding.  Where it becomes hectic (because its always seems to be the last minute) is trying to align a remediation technique that is relevant to an application written in ASP .NET, or Java, running on top of an IIS or Apache web server, running third party developed modules bolted onto a customized versions of a SaaS or COTS application.  And after a few of assessments, you start to realize some of your previous thoroughly research recommendations would also work on another client's "one of a kind" app.  

[How can so many "unique" apps have so many common problems?] 

The problem I keep running into while writing each report is that the recommendations I wanted to recall were scattered across previous reports I no longer have access to.  So I decided to create a tool to help me overcome this problem.  (I will be put up a working version of the app shortly so that you can see what I mean...)

It was a pleasant surprise when I discovered that OWASP had a project that would automatically go through each criteria of the testing guide.  However, the more I play with OWTF, I began to realize its not quite what I thought it was. It might be due to boredom, or just plain laziness, I personally believe its because I suffer from being feed TMI (...oxymoron-ish I know given this blog), but the nuance or the purpose of the project escaped me.  I initially thought the project was meant to accomplish the same thing my tool was doing.  Only it wasn't.  What OWTF, in its currently incarnation, is designed to do is to automate the execution of the various tests based on the criteria defined in Testing Guide.   BTW, this was something I was planning on integrating into my tool now that the reporting function was built but now thankfully OWASP beat me to it.

So, like what any security enthusiast (a.k.a., cracker, hacker, curious George) would do, it was time to take this project out for a spin.  First, let me clarify any statements that come across as dissing someone by humbly stating for the record that I preach, support and thoroughly respect the work OWASP is doing. With that caveat out of the way, is it me or does it seems a few of the project's "tools" provided by OWASP doesn't seem to work at all or at least right out of the box?  Well this project was certainly no exception.  Again, no disrespect to the mad talent and skills of the OWTF team is intended.

Following the instructions on the project's numerous web slash wiki sites, neither installation methods seemed to lead to clean and working install.  I spent quite a bit of time down the rabbit hole trying to fix all of the issues I kept running into.  When I finally got tired of playing Fix-It Felix, I decided to go the surefire laborious route of manually installing the application from scratch.

Then I happened upon firebitsbr who had already figured out portions of what needed to done.  This certainly saved some time.  So I combined firebitsbr notes and mine to I finally achieve a clean installation and most importantly an actual running version of the application.  (Yay!)

Right now I'm running the app through its paces by scanning it against a few vulnerable web applications.   I will share what I discover for those who are interested.

Keep in mind

Not surprisingly, found a couple of missing tools and/or misconfiguration that needed to be corrected along the way e.g., the now defunct msfcli command.

Manual install on Kali

On bare metal, I wiped my previous Kali Linux installation which I'm sure I'll regret doing, but oh well, this is in the name of science (or whatever you want to call it.)

[NOTE] Sometimes a dependency would fail during the install process, and the fix was to just rerun the command.  Seriously.

Standard process post install.

• apt-get update && apt-get upgrade -y
• apt-get dist-upgrade -y

I haven't gone "full-upgrade" yet. I'm sure its perfectly safe to use...

This assumes you took the same fresh install route that I did.  i.e., setting up Metasploit, etc. first.  Needless to say, but I'm saying it, PostgreSQL needs to be up and running.

• service postgresql start
• update-rc.d postgresql enable

I (too) like putting my "customized" stuff into opt...

• cd /opt

The branch didn't matter.  Lion_2014 is supposedly the stable version, and develop of course is considered the bleeding edge version.

• git clone -b develop https://github.com/owtf/owtf.git

Maybe because I didn't have enough caffeine coursing through my veins, but I was previously stuck trying to fix the crypto issues, that is until I discovered all you need to is...

• pip install --upgrade cffi
• apt-get install libssl-dev

Technically, the install script is suppose to take care of this, but this proved to be a necessary stepin order to proceed with the install. As a heads up, this an might require running more than once.

• pip install --upgrade -r /opt/owtf/install/owtf.pip
• pip install --upgrade beautifulsoup4 lxml Markdown psycopg2 pycurl six

I suppose this can be done at the beginning, but this was where in the process it complained about not being installed.  I didn't want to anger the installation demigod. 

• apt-get install libpq-dev

Leveraged the installation script to finish the rest of the setup, etc.

• chmod u+x /opt/owtf/install/install.py
• /opt/owtf/install/./install.py

This might not be necessary, but it complained that the database wasn't setup and this fixed it.  Kinda makes you realize this isn't the first time the project team ran into this very same problem.  Thanks!

•   /opt/owtf/script/./db_setup.py init

Tweak what Metasploit dir location

Took firebitsbr lead on this one by editing both /opt/owtf/profiles/general/default_backtrack.cfg and /opt/owtf/profiles/general/default.cfg

• Its always a good idea to make a backup of a line or even the file you are going to change
• Search for TOOL_METASPLOIT_DIR copy and comment out the first (i.e., # at the beginning)
• Change the line that is not commented out for TOOL_METASPLOIT_DIR: by changing it from
  [/opt/metasploit/apps/pro/msf3 |
  /opt/metasploit/msf3] to /usr/share/metasploit-framework

Obsolete CMD called msfcli

msfcli is no longer part of Metaspolit, so we'll need replace it with msfconsole -x

• find /opt/owtf -type f -print0 | xargs -0 sed -i 's/msfcli /msfconsole -x /g'

Another way of doing the same thing...

• grep -rl msfcli '/opt/owtf' | xargs sed -i 's/msfcli /msfconsole -x /g'

Missing tools

Even after all of that, still found where tools were missing from the installation process.  

metagoofil

• cd /opt
• git clone https://github.com/laramies/metagoofil.git
• cd metagoofil
• ln -s /opt/metagoofil /usr/share/metagoofil

samrdump.py

• cd /opt  
• git clone https://github.com/CoreSecurity/impacket.git
• cd impacket
• python setup.py install

Hope this helps anyone who happens to have read this.