Saving logs in different logstash index

Elastic Stack Logstash
1

Hi team,

I have question regarding saving logs in different logstash files based on pattern matching.

for example-

filter{
grok{
match => [
"message","pattern1",
"message","pattern2"
]
}
}

output{
//how to save output in different index files based on pattern match

if pattern1 matc, logs should be save to index 1
if pattern2 match, logs should be saved to index 2 and so on

}

please note that I have logs available in same dir for apache logs, application logs and few other logs.

Thanks in advance

2

Can someone please guide.

3

i think you should be use type field (like document_typ in filebeat config) and config for logstash will be look like

filter{
	if [type] == "type1" {
		grok{
			match => ["message","pattern1"]
		}
	}
	if [type] == "type2" {
		grok{
			match => ["message","pattern2"]
		}
	}
	
}

output {
	if [type] == "type1" {
		elasticsearch {
			hosts: ["http://localhost:9200"]
  			index: "index1-%{+yyyy.MM.dd}"
		}
	}
	if [type] == "type2" {
		elasticsearch {
			hosts: ["http://localhost:9200"]
  			index: "index2-%{+yyyy.MM.dd}"
		}
	}
}

Or maybe you can read https://www.elastic.co/guide/en/beats/filebeat/current/elasticsearch-output.html#_literal_indices_literal in update for ES 6.0

I prefer routing data in logstash :smiley:

4

Thank you

5

@Sumit_Sijaria. i updated some config if you are using version 6.0
In version 6.0, document_type will be remove in beat config. So you can change you config in beat follow guide
https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#fields-under-root

It look like :

filebeat.prospectors:
 - type: log
 paths:
    - /var/log/apache/httpd-*.log
 fields_under_root: true
 fields:
    type: type_1

Instead of :

filebeat.prospectors:
 - type: log
 paths:
    - /var/log/apache/httpd-*.log
 document_type: type_1

Of course, logstash config not change.

6

@tatdat Thanks for heads up. I am still using older version. I am just working on POC and getting hands dirty with ELK and GROK. :slightly_smiling_face:

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.