E.T. Proxy Logs Checker [ETPLC]

The Elasticsearch "Connector" with ETPLC project:

-ETPLC receive logs on few format, but Elasticsearch Connector use only Squid log format actually!

-First we need Perl and Elasticsearch module.

-Second, Check few parameters on Elasticsearch server in perl script "etplc_elasticsearch_xxx.pl" (avalaible in download section)

(Index_Name is based on Logstash index name like logstash-2014.10.28)
$timestampelasticsearch="now-30m"; (default now-30mn)
servers => '127.0.0.1:9200' (default)
size => 99999
index => "logstash-$year.$month.$mday"
sort => [ { '@timestamp' => "asc" }, ]
@timestamp like (2014-10-28T23:43:14.953091+01:00)
host (proxy or web server name)
tag (or programname_syslog like squid)
ip_client
http_reply (squid format like this: TCP_REFRESH_UNMODIFIED/304)
squid_time (squid format squid like this: [28/Oct/2014:23:43:14 +0100])
http_method (GET, POST...)
http_uri (all fields like http://www.google.com/test.php)
http_domaine (squid format like this: HIER_DIRECT/etplc.org)
http_useragent
http_referer (same like http_uri)
http_cookie

-First time, run this perl script simply like this "perl etplc_elasticsearch_29oct2014.pl"
We can check output informations/errors

Output working example on "Squid" format:
2014-10-30T21:14:13.803334+01:00 localhost squid_access: 0 127.0.0.1 TCP_MISS/200 - [30/Oct/2014:21:14:10 +0100] 0 GET http://etplc.org/elasticsearch.html - HIER_DIRECT/etplc.org - "Mozilla/5.0 Firefox/33.0" "http://etplc.org/_Referer" "Cookie"

-ok if it's work, please run with full etplc like this:

"perl etplc_elasticsearch_29oct2014.pl | perl etplc_15oct2014a.pl -f emergingall_sigs28oct2014a_snort290b.rules"

or Python v2:
"perl etplc_elasticsearch_29oct2014.pl | python2 etplc_15oct2014a.py2 -f emergingall_sigs28oct2014a_snort290b.rules"


Futur works:

Write a output ETPLC to Elasticsearch


Feedbacks is welcome.
Thx you Elasticsearch+Logstash+Kibana (ELK) project!