E.T. Proxy Logs Checker [ETPLC]

Open Source project for finding the Threats on Proxy or Web Server Logs with Emerging Threats Open rules

It's a production ready version, all feedback is welcome.

Follow project and list at Sourceforge and @GitHub repository.

Download files here (last 20 Jul 2016)

News: Easy to use ETPLC project with Syslog-NG, check here for more information.


Ported ETPLC project on Docker Technology. Link here to find more information.


Presentation: first French slide during Ossir avalaible here.


Project Pro/Cons:

The new initial version Splunk "Connector" with ETPLC project here.

The Elasticsearch "Connector" with ETPLC project here.

- Rewritten original Perl script to Python v3 !

- Rewritten original Perl script to Python v2 !

- It's based on rework (many) Emerging Threats Open rules (Thank you)
(only flow side to_server/from_client are used)

- It's work but performance depends logs numbers
(auto detect Thread Queue on perl and multiprocessing on python)

- Support Squid / Apache / Nginx / Forefront / BlueCoat / McAfee Web Gateway / IIS logs actually
(depends on Web Proxy Logs fields)

- Avalaible on a python script (v3.3.2 or v2.7.3) or a perl script (v5.14.4 or v5.22.1)

- http Referer are supported
- http User-Agent are supported
- http Cookie are now supported
- http Remote IP are now supported only on Squid proxy Logs

- Directory Traversal vulnerability are handled on logs by Proxy/Server like Apache/Squid...
- URI (evasion vulnerability) Decoding like UTF-8 are handled by etplc script (thx URI::Escape and urllib).

- Added http response code on alert, for checking directly if server allowed proxy/web request (thx Guillaume)


Futur works

- Enhancing performance again

- Follow coverage Snort rule format on Emerging Threats Open rules

- Resolv FP / FN


How it's work:

Before, check if you use last Emerging Threats Open rules on download page

perl:
realtime: tail -f /var/log/messages | perl etplc_13jul2016a.pl -f emergingall_sigs15oct2014a_snort290b.rules.gz
realtime through syslog: tail -f /var/log/messages | perl etplc_13jul2016a.pl -s -f emergingall_sigs15oct2014a_snort290b.rules.gz
offline, cat /var/log/messages | perl etplc_13jul2016a.pl -f emergingall_sigs15oct2014a_snort290b.rules.gz

python3:
realtime: tail -f /var/log/messages | python3 etplc_26sep2015a.py -f emergingall_sigs15oct2014a_snort290b.rules.gz
realtime through syslog: tail -f /var/log/messages | python3 etplc_26sep2015a.py -s -f emergingall_sigs15oct2014a_snort290b.rules.gz
offline, cat /var/log/messages | python3 etplc_26sep2015a.py -f emergingall_sigs15oct2014a_snort290b.rules.gz

python2:
realtime: tail -f /var/log/messages | python2 etplc_26sep2015a.py2 -f emergingall_sigs15oct2014a_snort290b.rules.gz
realtime through syslog: tail -f /var/log/messages | python2 etplc_26sep2015a.py2 -s -f emergingall_sigs15oct2014a_snort290b.rules.gz
offline, cat /var/log/messages | python2 etplc_26sep2015a.py2 -f emergingall_sigs15oct2014a_snort290b.rules.gz

new option Category restrict Logs Checking,
if your Logs contains ProxyLogs use -c proxy, if your Logs contains WebServer use -c webserver, by default or without this option use any logs checking.

if you need debug, enable on command line: -d

if you run etplc script and you have this error:
aucun parser ne correspond au motif !!! ...
-> sorry etplc unrecognized your logs, please submit to the list.

Don't forget, for best recognize vulnerabilities, you need enable extra logs options like Referer/User-Agent/Cookie.

Etplc project recognize SSL Connect on your logs, if not please submit to the list.

Memory Usage:
If your memory grow rapidly during ETPLC perl script, please add and adjust micro sleep like this:
cat /var/log/messages | perl -e 'use Time::HiRes qw( usleep);while(){print $_;usleep(8000);}' | perl etplc_13jul2016a.pl -f emergingall_sigs6mar2015a_snort290b.rules
Perl (v5.18.2 in my test) don't memory leak, but use more memory for buffering.

ETPLC script design on 3 parts:

- first load and convert Emerging Threats Open rules
- second parse Proxy Logs
- third matching ET_rules <=> Proxy_logs


You can follow ETPLC project on etplc-users@lists.sourceforge.net

Contact: rmkml@yahoo.fr / Twitter: @Rmkml

Etplc project src code are under the GPLv2.
A copy of that license is available at http://www.gnu.org/licenses/gpl-2.0.html