Seperate indexes for each log path field error

Elastic Stack Logstash
1

I am trying to tag every log file path in filebeat and in logstash, use that tag to create separate index for each log file. But only one index is being created. Why does document_type not work here?

Here is my filebeat.yml-

filebeat.inputs:
    
    -
      paths:
         - E:\DemoSetup\DispatcherApp\logs\dispatcher-scheduler.log
      input_type: log
      document_type: DispatcherApp
           
    -  
      paths:
         - E:\DemoSetup\Incident Agent\Logs\Trace.log
      input_type: log
      document_type: IncidentAgent
           
     
output:
  logstash:
    hosts: ["localhost:5044"]

And my logstash.conf-

input {
  file{
     path: E:\DemoSetup\DispatcherApp\logs\dispatcher-scheduler.log
     type => "DispatcherApp"
  }
   file{
    path: E:\DemoSetup\Incident Agent\Logs\Trace.log
    type => "IncidentAgent"
  }
  beats {
    port => 5044
  }
}
filter {
    if[type] =="DispatcherApp"{
		grok {
			match => {"message" => "%{COMBINEDAPACHELOG}"}
        }
	} else [type] == "IncidentAgent" {
        grok {
            match => { "message" => "%{COMBINEDAPACHELOG}" }
        }
    }
	  
  }

    output {
      elasticsearch {
        hosts => ["http://localhost:9200"]
      sniffing => true
      manage_template => false
      index    => "%{[@metadata][beat]-%{+YYYY.MM.dd}"
      
      }
    }
2

This defines the name of the index created and does not contain the type you specified.

3

Hi @Christian_Dahlqvist, I made the change and my updated logstash.conf is-

The filter I had in old file was stopping the filebeat from running.

Does this part not qualify as proper filters?

input {
 file{
     path: E:\DemoSetup\DispatcherApp\logs\dispatcher-scheduler.log
     type => "access"
  }
 file{
    path: E:\DemoSetup\Incident Agent\Logs\Trace.log
    type => "error"
  }
 beats {
  port => "5044"
   }
}


filter {
#If log line contains tab character followed by 'at' then we will tag that entry as stacktrace
  if [message] =~ "\tat" {
    grok {
     match => ["message", "^(\tat)"]
      add_tag => ["stacktrace"]
    }
  }
}
 
output {
    elasticsearch {
    hosts => ["localhost:9200"]
   sniffing => true
   manage_template => false
   index    => "%{type}-%{+YYYY.MM.dd}"  
   }
   stdout {
    codec => rubydebug
}

Is the index mentioned in output correct now? In Kibana, I still see all logs collected under one index only!

4

Hi @Christian_Dahlqvist, thanks for responding! Should I add the index line of code twice to get two indexes?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.