# Using logstash/elastic search for firewall syslog indexing: syslogs are lost on the way

**URL:** <https://discuss.elastic.co/t/using-logstash-elastic-search-for-firewall-syslog-indexing-syslogs-are-lost-on-the-way/16613>\
**Category:** Elasticsearch\
**Created:** [March 26, 2014, 2:13pm UTC](https://discuss.elastic.co/t/using-logstash-elastic-search-for-firewall-syslog-indexing-syslogs-are-lost-on-the-way/16613 "2014-03-26T14:13:27Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Antoine\_Brun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antoine_brun/32/3774_2.png) [@Antoine\_Brun](https://discuss.elastic.co/u/Antoine_Brun)\
**Post date:** [March 26, 2014, 2:13pm UTC](https://discuss.elastic.co/t/using-logstash-elastic-search-for-firewall-syslog-indexing-syslogs-are-lost-on-the-way/16613/1 "2014-03-26T14:13:27Z")

</div>

Hello,

I'm really new to elasticsearch and I'm running some evaluation  
(ElasticSearch vs SOLR) in order to decide which one I should use for  
handling high log volume indexing.  
I have a very basic setup:  
_syslogInjector -\> logstash listening on port 514 -\> elasticsearch._

input {  
tcp {  
port =\> 514  
type =\> syslog  
}  
udp {  
port =\> 514  
type =\> syslog  
}  
}  
filter {  
grok {  
match =\> { "message" =\> "%{GREEDYDATA:syslog\_message}" }  
add\_field =\> ["received\_at", "%{@timestamp}"]  
add\_field =\> ["received\_from", "%{host}"]  
}  
}  
output {  
elasticsearch { host =\> localhost }  
}

My test is:  
_inject 10000 1 ko syslog, at the rate 2000/sec._

I did a tcpdump on the server that receives the logs and all 10000 logs are  
received.

BUT, when I look at the number of document indexed in Lucene, I almost  
never get 10000 document, sometime I do, but most of the time I loose up to  
40% of the logs.

Is there anything I can do to make sure that logstash doesn't loose data (I  
assume it is logstash but not sure)  
Is logstash doing some buffering?

Thank you for your response

Antoine Brun

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/656cec27-895b-4ade-988b-054c4e79c325%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/656cec27-895b-4ade-988b-054c4e79c325%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Antoine\_Brun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antoine_brun/32/3774_2.png) [@Antoine\_Brun](https://discuss.elastic.co/u/Antoine_Brun)\
**Post date:** [March 29, 2014, 2:39pm UTC](https://discuss.elastic.co/t/using-logstash-elastic-search-for-firewall-syslog-indexing-syslogs-are-lost-on-the-way/16613/2 "2014-03-29T14:39:45Z")

</div>

Hello,

maybe my post wasn't specific enough...  
well, I'm still trying to validate logstash and still I'm loosing up to 40%  
of logs:

- 10000 logs injected.
- 10000 logs detected by tcpdump (listening on port 514)
- 6000-10000 documents created in elasticsearch index.

is there a way to count, in logstash, the number of event that were process  
and forwarded to elasticsearch ?

Thanks

Antoine

Le mercredi 26 mars 2014 15:13:27 UTC+1, Antoine Brun a écrit :

> Hello,
> 
> I'm really new to elasticsearch and I'm running some evaluation  
> (Elasticsearch vs SOLR) in order to decide which one I should use for  
> handling high log volume indexing.  
> I have a very basic setup:  
> _syslogInjector -\> logstash listening on port 514 -\> elasticsearch._
> 
> input {  
> tcp {  
> port =\> 514  
> type =\> syslog  
> }  
> udp {  
> port =\> 514  
> type =\> syslog  
> }  
> }  
> filter {  
> grok {  
> match =\> { "message" =\> "%{GREEDYDATA:syslog\_message}" }  
> add\_field =\> ["received\_at", "%{@timestamp}"]  
> add\_field =\> ["received\_from", "%{host}"]  
> }  
> }  
> output {  
> elasticsearch { host =\> localhost }  
> }
> 
> My test is:  
> _inject 10000 1 ko syslog, at the rate 2000/sec._
> 
> I did a tcpdump on the server that receives the logs and all 10000 logs  
> are received.
> 
> BUT, when I look at the number of document indexed in Lucene, I almost  
> never get 10000 document, sometime I do, but most of the time I loose up to  
> 40% of the logs.
> 
> Is there anything I can do to make sure that logstash doesn't loose data  
> (I assume it is logstash but not sure)  
> Is logstash doing some buffering?
> 
> Thank you for your response
> 
> Antoine Brun

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b190724e-11d8-4237-97f8-f5bb0d028847%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b190724e-11d8-4237-97f8-f5bb0d028847%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:39am UTC](https://discuss.elastic.co/t/using-logstash-elastic-search-for-firewall-syslog-indexing-syslogs-are-lost-on-the-way/16613/3 "2017-07-06T01:39:44Z")

</div>