So I'm running Bro + ELK on a Ubuntu 14.04 LTS box and I'm getting less than one request per second fed into elasticsearch...
I'm running
Bro 2.4.1
Logstash 2.0.0
Elasticsearch 2.0.0
Kibana 4.2.1
Bro peels through my static pcap just fine, though logstash seems to choke on it.
The only thing that seems unusual in my --debug log of logstash is that logstash seems to be compiling patterns for grok after every parse...like:
{:timestamp=>"2015-11-23T20:15:48.958000-0500", :message=>"Adding pattern",........
... x10
{:timestamp=>"2015-11-23T20:15:49.041000-0500", :message=>"Grok compiled OK"....
{:timestamp=>"2015-11-23T20:15:49.042000-0500", :message=>"Grok loading patterns from file"...
... x10
{:timestamp=>"2015-11-23T20:15:49.047000-0500", :message=>"Match data", :match=>...
{:timestamp=>"2015-11-23T20:15:49.047000-0500", :message=>"Grok compile".....
What kind of filters do you have? Any dns filters, for example?
I suspected the DNS filters may have slowed things down more, but here is a pastebin of my current testing.conf: http://pastebin.com/cE1tUwD5
Output to screen from a 100M pcap
1m rate: 1.63050786998764 ( 58 )
1m rate: 1.63050786998764 ( 59 )
1m rate: 1.63050786998764 ( 61 )
1m rate: 1.32398555940583 ( 63 )
1m rate: 1.0581933671681762 ( 64 )
1m rate: 0.8511941997192259 ( 66 )
1m rate: 0.8511941997192259 ( 73 )
1m rate: 0.8511941997192259 ( 74 )
1m rate: 0.8511941997192259 ( 76 )
1m rate: 0.8511941997192259 ( 77 )
1m rate: 0.8511941997192259 ( 77 )
1m rate: 0.8511941997192259 ( 83 )
1m rate: 0.8511941997192259 ( 84 )
1m rate: 0.745725509683129 ( 127 )
1m rate: 0.5380761211007017 ( 138 )
1m rate: 0.5380761211007017 ( 139 )
1m rate: 0.5380761211007017 ( 142 )
1m rate: 0.5380761211007017 ( 160 )
1m rate: 0.21623381036625616 ( 188 )
1m rate: 0.21623381036625616 ( 189 )
1m rate: 0.19547543756743765 ( 191 )
1m rate: 0.19547543756743765 ( 192 )
1m rate: 0.19547543756743765 ( 195 )
1m rate: 0.19547543756743765 ( 200 )
1m rate: 0.19547543756743765 ( 200 )
1m rate: 0.28417595326839334 ( 210 )
1m rate: 0.3336716852707924 ( 239 )
1m rate: 0.3336716852707924 ( 242 )
1m rate: 0.3336716852707924 ( 246 )
1m rate: 0.3336716852707924 ( 247 )
1m rate: 0.3336716852707924 ( 248 )
1m rate: 0.3336716852707924 ( 251 )
1m rate: 0.3336716852707924 ( 252 )
1m rate: 0.3336716852707924 ( 253 )
1m rate: 0.5031350529113059 ( 254 )
1m rate: 0.3815863660730709 ( 273 )
1m rate: 0.3815863660730709 ( 286 )
1m rate: 0.3815863660730709 ( 287 )
1m rate: 0.41637540313488386 ( 312 )
1m rate: 0.33813438622674613 ( 340 )
1m rate: 0.270864751834757 ( 347 )