# How to differentiate log files sended from filebeat? **URL:** \ **Category:** Logstash\ **Created:** [March 20, 2019, 12:28pm UTC](https://discuss.elastic.co/t/how-to-differentiate-log-files-sended-from-filebeat/173139 "2019-03-20T12:28:09Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Deny7](https://avatars.discourse-cdn.com/v4/letter/d/e0b2c6/32.png) [@Deny7](https://discuss.elastic.co/u/Deny7)\ **Post date:** [March 20, 2019, 12:28pm UTC](https://discuss.elastic.co/t/how-to-differentiate-log-files-sended-from-filebeat/173139/1 "2019-03-20T12:28:09Z")
Hi, I'm trying to configure one logstash pipeline for my enviroments(test,dev,prd,acp). From each server I want to send log three log files(alfresco.log,share.log,access.log) using Filebeat. So my understanding is that I will create one pipeline - test-pipeline.conf and in output section I will define "If [type]" statements where I will check given log name, let's say alfresco.log file from test environment will be of type "test\_alfresco\_log" If that's true I will create Index of that type. And this I will repeat for every log from each environment. My question is where I can define the type property(test\_alfresco\_log)? I tried to define these types in filebeat.yml, but It's not the right way because filebeat won't even start because of this modification. Can anybody help? See my config files: **filebeat.yml** ``` filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: test_alfresco_log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log - c:\Alfresco\alfresco5\alfresco.log - type: test_share_log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log - c:\Alfresco\alfresco5\share.log - type: test_access_log # Change to true to enable this input configuration. enabled: true # Paths that should be crawled and fetched. Glob based paths. paths: #- /var/log/*.log - c:\Alfresco\alfresco5\tomcat\logs\access*.log ``` **test-pipeline.conf** ``` # Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { beats { port => 5044 host => "HYPALFMONPRD1" } } filter{ grok { break_on_match => false match => ["message", "TID: \[%{TIMESTAMP_ISO8601:timestamp}\] %{LOGLEVEL:level} \[%{JAVACLASS:java_class}\] \(%{GREEDYDATA:thread}\) - (?(.|\r|\n)*)"] } date{ match => ["timestamp_from_log", "ISO8601"] } } output { if [type] == "test_alfresco_log" { elasticsearch { hosts => ["HYPALFMONPRD1:9200"] sniffing => true manage_template => false index => "test_alfresco_log-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } if [type] == "test_share_log" { elasticsearch { hosts => ["localhost:9200"] sniffing => true manage_template => false index => "test_share_log-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } if [type] == "test_access_log" { elasticsearch { hosts => ["localhost:9200"] sniffing => true manage_template => false index => "test_access_log-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } ###There will be the other if statements... } ``` **pipelines.yml** ``` # List of pipelines to be loaded by Logstash # # This document must be a list of dictionaries/hashes, where the keys/values are pipeline settings. # Default values for ommitted settings are read from the `logstash.yml` file. # When declaring multiple pipelines, each MUST have its own `pipeline.id`. # # Example of two pipelines: # # - pipeline.id: test # pipeline.workers: 1 # pipeline.batch.size: 1 # config.string: "input { generator {} } filter { sleep { time => 1 } } output { stdout { codec => dots } }" # - pipeline.id: another_test # queue.type: persisted # path.config: "/tmp/logstash/*.config" # # PIPELINES: - pipeline.id: test-pipeline path.config: "/etc/path/to/p1.config" ``` **Filebeat version:** **6.6.2** **Logstash version** **6.6.1** **Kibana version :** **6.6.1** **Elasticsearch version :** **6.6.1** **APM Server version :** **6.6.2** **APM Agent language and version :** **Java, 1.4.0** ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [March 20, 2019, 2:11pm UTC](https://discuss.elastic.co/t/how-to-differentiate-log-files-sended-from-filebeat/173139/2 "2019-03-20T14:11:23Z")
You could try this [Creating index depends on source](https://discuss.elastic.co/t/creating-index-depends-on-source/167061/2?u=a_b) It is my solution as I do not like `if` statements in the output section. ---
**Author:** ![Deny7](https://avatars.discourse-cdn.com/v4/letter/d/e0b2c6/32.png) [@Deny7](https://discuss.elastic.co/u/Deny7)\ **Post date:** [March 20, 2019, 4:34pm UTC](https://discuss.elastic.co/t/how-to-differentiate-log-files-sended-from-filebeat/173139/3 "2019-03-20T16:34:27Z")
I tried it with your configuration, but it only register and create index for the first log(alfresco.log) defined in filebeat.yml, for other two not. Please can u check my configuration below? **filebeat.yml** ``` filebeat.inputs: # Each - is an input. Most options can be set at the input level, so # you can use different inputs for various configurations. # Below are the input specific configurations. - type: log enabled: true paths: - c:\Alfresco\alfresco5\alfresco.log encoding: plain fields: log_prefix: test log_idx: alfresco_log fields_under_root: false - type: log enabled: true paths: - c:\Alfresco\alfresco5\share.log encoding: plain fields: log_prefix: test log_idx: share_log fields_under_root: false - type: log enabled: true paths: - c:\Alfresco\alfresco5\tomcat\logs\access*.log encoding: plain fields: log_prefix: test log_idx: access_log fields_under_root: false ``` **test-pipeline.conf** ``` # Sample Logstash configuration for creating a simple # Beats -> Logstash -> Elasticsearch pipeline. input { beats { port => 5044 host => "MONPRD1" } } filter{ grok { break_on_match => false match => ["message", "TID: \[%{TIMESTAMP_ISO8601:timestamp}\] %{LOGLEVEL:level} \[%{JAVACLASS:java_class}\] \(%{GREEDYDATA:thread}\) - (?(.|\r|\n)*)"] } date{ match => ["timestamp_from_log", "ISO8601"] } mutate { copy => { "[fields][log_prefix]" => "[@metadata][log_prefix]" "[fields][log_idx]" => "[@metadata][index]" } } } output { elasticsearch { hosts => ["MONPRD1:9200"] sniffing => true manage_template => false index => "%{[@metadata][log_prefix]}-%{[@metadata][index]}-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } ``` ---
**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\ **Post date:** [March 21, 2019, 9:28am UTC](https://discuss.elastic.co/t/how-to-differentiate-log-files-sended-from-filebeat/173139/4 "2019-03-21T09:28:34Z")
Looks mostly ok... In the _filter_ section of the Logstash config the indentation is a bit off and I would put a space between `date` and the {. ```auto filter{ grok { break_on_match => false match => ["message", "TID: \[%{TIMESTAMP_ISO8601:timestamp}\] %{LOGLEVEL:level} \[%{JAVACLASS:java_class}\] \(%{GREEDYDATA:thread}\) - (?(.|\r|\n)*)"] } date { match => ["timestamp_from_log", "ISO8601"] } mutate { copy => { "[fields][log_prefix]" => "[@metadata][log_prefix]" "[fields][log_idx]" => "[@metadata][index]" } } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 18, 2019, 9:28am UTC](https://discuss.elastic.co/t/how-to-differentiate-log-files-sended-from-filebeat/173139/5 "2019-04-18T09:28:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.