Output logs to multiple index from logstash

Elastic Stack Logstash
1

Hi there,

I am trying to send multiple log files from filebeat tologstash to elasticsearch. In kibana, I would like each log file under a separate index. This is my logstash.conf file:

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}
filter {
    if[type] =="DispatcherApp"{
		grok {
			match => {"message" => "%{COMBINEDAPACHELOG}"}
        }
	} else if [type] == "IncidentAgent" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
    }else if [type] == "IMMService" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
    }
	  
  }

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
	sniffing => true
	manage_template => false
	index => "web-%{type}"
	document_type => "log"
    #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

and filebeat.yml is-

filebeat.inputs:
    
    -
      paths:
         - E:\DemoSetup\DispatcherApp\logs\dispatcher-scheduler.log
      input_type: log
      document_type: DispatcherApp
           
    -  
      paths:
         - E:\DemoSetup\Incident Agent\Logs\Trace.log
      input_type: log
      document_type: IncidentAgent
           
    -input_type: log  
      paths:
         - E:\DemoSetup\Logs\IMSService\log.txt
      input_type: log
      document_type: IMMService
      
      
      #multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      #multiline.negate: true
      #multiline.match: after
      
    setup.template.name: "index-%{[beat.version]}"
    setup.template.pattern: "index-%{[beat.version]}-*"       

  
output:
  logstash:
    hosts: ["localhost:5044"]	
    #index: "index-%{[beat.version]}-%{[fields.type]:other}-%{+yyy.MM.dd}"

I just get a new index created called "web-%{type} and all three file logs are collected under it only. I think the type mentioned in filebeat.yml isnt being acknowledge in logstash file due to the if condition only taking message=> COMMONAPACHE!

2

I do not run filebeat but I know document types are being removed, so I wouldn't use anything that purports to set type, and thence _type.

Try using tags. The conditionals and sprintf reference in logstash will look similar.

That said, once each document is tagged with a type, why bother to put them in separate indexes? It is trivial to add a clause to the query to test the tag.

3

Are you referring to below here?

setup.template.name: "index-%{[beat.version]}"
    setup.template.pattern: "index-%{[beat.version]}-*"

Do you agree with this? After I replace tags with document_types, should I change the filter in logstash.conf to co-relate the tags?

4

No, I was referring to

index => "web-%{type}"
5

But then how will logstash know you create index by using the tagged field?

output {
  #if [@metadata][beat] == "filebeat"{
  elasticsearch {
    hosts => ["http://localhost:9200"]
	sniffing => true
	manage_template => false
	index => "%{[@metadata][type]}-%{+YYYY.MM.dd}"

Wont the "%{[@metadata][type] collect the type value in filebeat.yml and create separate index for each log file?

6

I am not sure, I do not use filebeat.

7

Thanks a lot for your help!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.