# Filebeat output to logstash **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [April 12, 2017, 2:11pm UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177 "2017-04-12T14:11:23Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [April 12, 2017, 2:11pm UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/1 "2017-04-12T14:11:23Z")
So am using filebeat to fetch 3 types of logs from websphere ```auto paths: - /opt/IBM/HTTPServer/logs/access_log document_type: access_log - paths: - /opt/IBM/WebSphere/AppServer/profiles/WPS00/logs/DEV00WPS*/SystemOut.log - /opt/IBM/WebSphere/AppServer/profiles/DMGRDEV/logs/dmgr/SystemOut.log document_type: BPM - paths: - /var/log/messages - /var/log/secure document_type: syslog ``` **and in logstash config i have grok filter for proper formatting of these logs** ```auto filter { if "access_logs" in [tags] { grok { match => { "message" => "%{IPORHOST:x_forwarded_for} - - \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion})" %{NUMBER:response}" } } } if "BPM" in [tags] { grok { match => { "message" => "%{SYSLOG5424SD:BPM_timestamp} %{BASE16NUM:ThreadID} %{WORD:EventType} %{WORD:ShortName} %{WORD:MessageIdentifier}:%{SPACE}%{GREEDYDATA:event}" } } } if "syslog" in [tags] { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program} (?:\[%{POSINT:syslog_pid}\])?%{GREEDYDATA:syslog_message}" } } } } ``` am so confused as how the output part should be for this config. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 12, 2017, 2:58pm UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/2 "2017-04-12T14:58:24Z")
the `document_type` will be accesible as field `[type]` in logstash. If you want to use tags, use the tags option in filebeat. ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [April 12, 2017, 4:13pm UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/3 "2017-04-12T16:13:37Z")
I did , and my output part looks like this, but this is not able to create the indices in logstash. I dont know how else to construct output part. ``` if [tags] == "access_log" { output { elasticsearch { hosts => ["10.190.188.174:9200"] index => "access-%{+YYYY.MM.dd}" } } } else if [tags] == "BPM" { output { elasticsearch { hosts => ["10.190.188.174:9200"] index => "bpm-%{+YYYY.MM.dd}" } } } ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 13, 2017, 9:33am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/4 "2017-04-13T09:33:23Z")
well, I'm no expert in logstash, but shouldn't it be ```auto output { if [tags] == "access_log" { elasticsearch { hosts => ["10.190.188.174:9200"] index => "access-%{+YYYY.MM.dd}" } } } else if [tags] == "BPM" { elasticsearch { hosts => ["10.190.188.174:9200"] index => "bpm-%{+YYYY.MM.dd}" } } } ``` ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [April 13, 2017, 9:37am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/5 "2017-04-13T09:37:00Z")
Yes correct i have made that change, now the indices "Access" and "bpm" are not being created in elasticsearch. Should i have these in filebeat.yml config? ---
**Author:** ![riddhijit\_roy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/riddhijit_roy/32/19049_2.png) [@riddhijit\_roy](https://discuss.elastic.co/u/riddhijit_roy)\ **Post date:** [April 14, 2017, 7:27am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/6 "2017-04-14T07:27:17Z")
Seems like a simple error to me . Try this. output { if [tags] == "access\_log" { elasticsearch { hosts =\> ["10.190.188.174:9200"] index =\> "access-%{+YYYY.MM.dd}" } } else if [tags] == "BPM" { elasticsearch { hosts =\> ["10.190.188.174:9200"] index =\> "bpm-%{+YYYY.MM.dd}" } } } ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [April 14, 2017, 8:58am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/7 "2017-04-14T08:58:22Z")
Thanks, Now am able to separate the logs, but the documents are being duplicated 4 times for each log. This never happened before. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 19, 2017, 8:27am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/8 "2017-04-19T08:27:44Z")
Is it just the old logs or all logs? Could it be that you sent the log files multiple times because you removed the registry? ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [April 20, 2017, 10:27am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/9 "2017-04-20T10:27:51Z")
All logs including the one's populating today, I didn't get you there "removed registry" ?? I don't remember doing that. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 24, 2017, 7:54am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/10 "2017-04-24T07:54:50Z")
Filebeat has a registry file to keep the state. Could it be that you remove this one? ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [May 16, 2017, 9:49am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/11 "2017-05-16T09:49:48Z")
No, registry file is present in var/lib/filebeat/ Still for every id in the index i could see 4 duplicate entries! ![](https://us1.discourse-cdn.com/elastic/original/3X/7/4/743cb05b05bdd2df99d7c9bf5f0150212e95af11.PNG) ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 16, 2017, 12:34pm UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/12 "2017-05-16T12:34:53Z")
Could you try to write the events to file on the filebeat side and see if there are also duplicates? If not, there is probably something going on on the LS side as part of your script. Check also your filebeat logs of there are some errors inside related to sending. ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [May 19, 2017, 7:08am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/13 "2017-05-19T07:08:38Z")
How the DB logs are accessed in this way? Am using JDBC plugin to fetch DB logs as well. ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [May 19, 2017, 10:17am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/14 "2017-05-19T10:17:27Z")
I checked writing the output of filebeat to file it is writing only once. I couldn't find filebeat log file. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 22, 2017, 5:24am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/15 "2017-05-22T05:24:07Z")
As you have it in the file only once, I think either something during the transmission is going wrong, LS script has some issues. You could try to enable debug output in LS to see if it appears there twice. How did you install filebeat? You could also check in your config file where the log file should be. ---
**Author:** ![Vivek\_Samaga](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vivek_samaga/32/43285_2.png) [@Vivek\_Samaga](https://discuss.elastic.co/u/Vivek_Samaga)\ **Post date:** [May 22, 2017, 7:20am UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/16 "2017-05-22T07:20:44Z")
Ya i enabled "stdout { codec =\> rubydebug}" but there is no duplication. No discrepancy in FIlebeat logs as well. Am running filebeat 1.3.1 is updating necessary? ``` This is my logstash.conf file i have used conditionals to differentiate the events. input { beats { port => 5044 } } filter { if "access_logs" in [tags] { grok { match => { "message" => [ "%{IPORHOST:x_forwarded_for} %{IPORHOST:load_balancer} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb}%{SPACE} \/%{WORD:application}%{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{WORD:ServerHost}:%{WORD:ServerPort})", "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} \/%{WORD:application}%{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:%{WORD:ServerHost}:%{WORD:ServerPort})", "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} \/%{WORD:application}%{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" ] } } } if "BPM" in [tags] { grok { match => { "message" => [ "%{SYSLOG5424SD}%{SPACE}%{BASE16NUM:ThreadID}%{SPACE}%{WORD:ShortName}%{SPACE}%{WORD:EventType}%{SPACE}%{WORD:MessageIdentifier}:%{SPACE}%{GREEDYDATA:event}", "%{SYSLOG5424SD}%{SPACE}%{WORD:ThreadID}%{SPACE}%{WORD:Logger}%{SPACE}%{WORD:MessageType}%{SPACE}%{GREEDYDATA:event}" ] } } } if "syslog" in [tags] { grok { match => { "message" => [ "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" ] } } } } output { stdout { codec => rubydebug} if [type] == "access_log" { elasticsearch { hosts => ["10.190.188.174:9200"] index => "access-%{+YYYY.MM.dd}" } } else if [type] == "BPM" { elasticsearch { hosts => ["10.190.188.174:9200"] index => "bpm-%{+YYYY.MM.dd}" } } else if [type] == "syslog" { elasticsearch { hosts => ["10.190.188.174:9200"] index => "syslog-%{+YYYY.MM.dd}" } } #else if [type] == "JDBC" { #elasticsearch { #hosts => ["10.190.188.174:9200"] #index => "chd-%{+YYYY.MM.dd}" #} #} } ``` ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [May 23, 2017, 12:31pm UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/17 "2017-05-23T12:31:46Z")
It will definitively not hurt to update to the most recent release of filebeat and if possible also the rest of the stack. Duplicates can happen if you have network issues but in general I would expect this to be an edge case. Could you try to wipe your complete setup and try it again with the most recent version and see if it still happens.? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 20, 2017, 12:31pm UTC](https://discuss.elastic.co/t/filebeat-output-to-logstash/82177/18 "2017-06-20T12:31:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.