# How to use filebeat and logstash for different log **URL:** \ **Category:** Beats\ **Created:** [October 30, 2018, 1:11am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538 "2018-10-30T01:11:09Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![aabababba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aabababba/32/36732_2.png) [@aabababba](https://discuss.elastic.co/u/aabababba)\ **Post date:** [October 30, 2018, 1:11am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/1 "2018-10-30T01:11:09Z")
1 /var/log/message Oct 30 09:04:35 ci04 dbus-daemon: dbus[616]: [system] Successfully activated service 'org.freedesktop.problems' 2 /var/log/tomcat/catalina.out [INFO] [2018-10-29 16:44:30,945] [DubboServerHandler-172.1.0.53:20880-thread-38] [RedisConnManager:109] [181029000400931071] [181029000400931071]- [reids:1] anyidea?thanks... ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 30, 2018, 2:42pm UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/2 "2018-10-30T14:42:03Z")
Hi @aabababba, I'm not sure of understanding your question. For log messages you can use the [system module](https://www.elastic.co/guide/en/beats/filebeat/6.4/filebeat-module-system.html). For tomcat there is no module yet, for it you could use just a [log input](https://www.elastic.co/guide/en/beats/filebeat/6.4/filebeat-input-log.html), or [create your own pipeline](https://www.elastic.co/guide/en/beats/filebeat/6.4/configuring-ingest-node.html) to parse these lines. If you do it you can consider [creating a new filebeat module](https://www.elastic.co/guide/en/beats/devguide/6.4/filebeat-modules-devguide.html) and contributing it 🙂 ---
**Author:** ![aabababba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aabababba/32/36732_2.png) [@aabababba](https://discuss.elastic.co/u/aabababba)\ **Post date:** [October 31, 2018, 2:10am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/3 "2018-10-31T02:10:40Z")
my filebeat.yml ```auto filebeat.prospectors: - type: log enabled: true paths: - /var/log/*.log - type: log enabled: true paths: - /opt/apache-tomcat/logs/* tags: ["catalina"] document_type: catalina fields: type: catalina fields_under_root: true filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 3 setup.kibana: output.logstash: hosts: ["172.16.0.2:5044"] ``` my logstash.yml ```auto input { beats { add_field => {"beatType" => "metricbeat"} port => "5043" } beats { add_field => {"beatType" => "filebeat"} port => "5044" } } filter { mutate { add_field => { "remote_ip" => "%{[@metadata][ip_address]}" } } if [type]=="catalina"{ grok { match => { "message" => "\[%{LOGLEVEL:loglevel}\] \[%{TIMESTAMP_ISO8601:access_time}\] \[%{DATA:exception_info}\] - \[%{GREENYDATA:msg}\]" } pattern_definitions => { "MESSAGE" => "[\s\S]*" } } date { match => ["access_time","yyyy-MM-dd HH:mm:ss,SSS"] } mutate { remove_field => ["access_time","[message][0]"] } } } output { if [beatType] == "metricbeat" { elasticsearch { hosts => ["172.16.0.2:9200"] index => "metricbeat-%{+YYYY.MM.dd}" } } if [beatType] == "filebeat" { elasticsearch { hosts => ["172.16.0.2:9200"] index => "filebeat-%{+YYYY.MM.dd}" } } } ``` this not work, loglevel -\>info error warn debug can't index... ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [October 31, 2018, 12:49pm UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/4 "2018-10-31T12:49:01Z")
What is exactly not working? In your config I see that field `type` is set to `catalina`, but the grok pattern is applied if `[Type]=="tomcat"`. I also see you are using two different ports for beats input, this is fine but if you are doing it only to add a field to be able to separate events in different indexes, take into account that you can also use `@metadata` in the index name: ```auto output { elasticsearch { hosts => ["172.16.0.2:9200"] index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" } } ``` Notice that it is recommended to version the indexes, so you can have multiple beats versions using different index templates. ---
**Author:** ![aabababba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aabababba/32/36732_2.png) [@aabababba](https://discuss.elastic.co/u/aabababba)\ **Post date:** [November 1, 2018, 1:21am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/5 "2018-11-01T01:21:46Z")
thank you. I change the `if [Type]=="catalina"`,but still not work,the issue is I set the log-level keyword,and '[INFO]' this can't show in the log-level. sorry, my English just so so... ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [November 1, 2018, 10:38am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/6 "2018-11-01T10:38:47Z")
You are setting the `type` all lowercase, try to use it also with lowercase in the condition `if [type]=="catalina"`. Regarding the grok pattern, what kind of log lines do you want to match? The pattern you are writing starts with timestamp: ```auto %{TIMESTAMP_ISO8601:access_time} %{LOGLEVEL:loglevel} [%{DATA:exception_info}] - <%{MESSAGE:message}> ``` But the catalina log line you pasted at the begining starts with the log level: ```auto [INFO] [2018-10-29 16:44:30,945] [DubboServerHandler-172.1.0.53:20880-thread-38] [RedisConnManager:109] [181029000400931071] [181029000400931071]- [reids:1] ``` These tools can help you to build or test grok patterns: - [http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/) - [http://grokconstructor.appspot.com/](http://grokconstructor.appspot.com/) ---
**Author:** ![aabababba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aabababba/32/36732_2.png) [@aabababba](https://discuss.elastic.co/u/aabababba)\ **Post date:** [November 2, 2018, 2:46am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/7 "2018-11-02T02:46:00Z")
sorry,my mistake,but I change it like this,still not work,loglevel and msg do not have data. ```auto "message" => " \[%{LOGLEVEL:loglevel} \] \[%{TIMESTAMP_ISO8601:logtime}\] \[%{DATA:exception_info}\] \[%{DATA:exception_info1}\] \[%{DATA:exception_info2}\] \[%{DATA:exception_info3}\]- \[%{DATA:msg}\]" ``` ![%E5%9B%BE%E7%89%87](https://us1.discourse-cdn.com/elastic/original/3X/a/0/a0c27cf80b77d8499fde936b3777fbab8870fed0.png) ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [November 2, 2018, 11:30am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/8 "2018-11-02T11:30:45Z")
It seems that your grok pattern is not able to parse this log line, if you look at the tags, the `_grokparsefailure` one has been added, this happens when grok patterns are not able to parse the log line. ---
**Author:** ![aabababba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aabababba/32/36732_2.png) [@aabababba](https://discuss.elastic.co/u/aabababba)\ **Post date:** [November 5, 2018, 5:34am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/9 "2018-11-05T05:34:47Z")
thanks you very much,I get it, grok mistake! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 3, 2018, 7:35am UTC](https://discuss.elastic.co/t/how-to-use-filebeat-and-logstash-for-different-log/154538/10 "2018-12-03T07:35:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.