Saving logs in different logstash index

[Sumit_Sijaria]

Hi team,

I have question regarding saving logs in different logstash files based on pattern matching.

for example-

filter{
grok{
match => [
"message","pattern1",
"message","pattern2"
]
}
}

output{
//how to save output in different index files based on pattern match

if pattern1 matc, logs should be save to index 1
if pattern2 match, logs should be saved to index 2 and so on

}

please note that I have logs available in same dir for apache logs, application logs and few other logs.

Thanks in advance

[Sumit_Sijaria]

Can someone please guide.

[tatdat (Dat Pham Tat)]

i think you should be use type field (like document_typ in filebeat config) and config for logstash will be look like

filter{
	if [type] == "type1" {
		grok{
			match => ["message","pattern1"]
		}
	}
	if [type] == "type2" {
		grok{
			match => ["message","pattern2"]
		}
	}
	
}

output {
	if [type] == "type1" {
		elasticsearch {
			hosts: ["http://localhost:9200"]
  			index: "index1-%{+yyyy.MM.dd}"
		}
	}
	if [type] == "type2" {
		elasticsearch {
			hosts: ["http://localhost:9200"]
  			index: "index2-%{+yyyy.MM.dd}"
		}
	}
}

Or maybe you can read https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_literal_indices_literal in update for ES 6.0

I prefer routing data in logstash

[Sumit_Sijaria]

Thank you

[tatdat (Dat Pham Tat)]

@Sumit_Sijaria. i updated some config if you are using version 6.0
In version 6.0, document_type will be remove in beat config. So you can change you config in beat follow guide
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#fields-under-root

It look like :

filebeat.prospectors:
 - type: log
 paths:
    - /var/log/apache/httpd-*.log
 fields_under_root: true
 fields:
    type: type_1

Instead of :

filebeat.prospectors:
 - type: log
 paths:
    - /var/log/apache/httpd-*.log
 document_type: type_1

Of course, logstash config not change.

[Sumit_Sijaria]

@tatdat Thanks for heads up. I am still using older version. I am just working on POC and getting hands dirty with ELK and GROK.