Logstash different .conf inputs are mixing up entries in indexes

Elastic Stack Logstash
1

Before I go on, I have tried to look through the interwebs for an answer, I scoured the forums posts so much I feel I know the elk staff member names by heart (not really).

I do apologise if my insufficient research skills have failed me and I am now creating a duplicate post.

I have two configs in logstash: dev.conf, prod.conf
dev.conf:

input {
	beats {
		port => 28777
		codec=>"json"
	}
}

filter {
	date {
		match => ["timestamp", "ISO8601"]
		target => "@timestamp"
	}
}


output {
	elasticsearch {
		index => "dev-log-index-1"
		hosts => ["localhost:9200"]
	}
}

prod.conf:

input {
	beats {
		port => 28000
		codec=>"json"
	        ssl => true
		ssl_certificate => "/etc/logstash/logstash.crt"
		ssl_key => "/etc/logstash/logstash.key"
		ssl_verify_mode => "force_peer"
		ssl_certificate_authorities => ["/etc/logstash/ca.crt"]
	}
}

filter {
	date {
		match => ["timestamp", "ISO8601"]
		 target => "@timestamp"
	}
}

output {

	elasticsearch {
		index => "prod-log-index-1"
		hosts => ["localhost:9200"]
	}
}`

I chose to separate the logs for different environments as such, although I find entries from dev-log-index-1 being attached to prod-log-index-1 and vice versa. As a result I am not able to see prod only or dev only logs under the corresponding kibana index pattern, even though these configs specify separate input ports and indexes.

2

You need to use Multiple Pipelines, if you just create different .conf files and put them in the conf.d directory without changing the pipelines.yml file, they are treated as a unique pipeline.

You need something like this in your pipeline.yml

- pipeline.id: dev
  path.config: "/etc/logstash/conf.d/dev.conf"
- pipeline.id: prod
  path.config: "/etc/logstash/conf.d/prod.conf"
1 Like
3

Much obliged Jumping right on it, I'll let you know how it went!

4

Worked like a charm! Thank you very much. Hope this helps anyone who has the same issue.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.