Logstash multiple pipelines bug?

Elastic Stack Logstash
1

Hello,
I made a mutiple pipeline with version 6.6 of logstash. But, he has behavior that is unexpected.
Indeed I have my multiple pipeline that generates duplicates of the data as if making a loop before stopping the processing. For 3 lines of message in file it generates randomly 200 or 500 or 700 lines of the 3 lines of messages duplicated in outputs. Also my input exec doesn't generate anything for me. so if I run it alone it works fine. Below my code :

File pipelines.yml

- pipeline.id: main
  queue.type: persisted
  path.config: "/etc/logstash/conf.d/main.conf" 
- pipeline.id: els
  queue.type: persisted
  path.config: "/etc/logstash/conf.d/els.conf"  
- pipeline.id: file
  queue.type: persisted
  path.config: "/etc/logstash/conf.d/output_file.conf"

file main.conf

input {
	file {
	  path => "/etc/logstash/test.txt"
	  start_position => "beginning"
	  sincedb_path => "/dev/null"
	  tags => "file_test"
	}	
	exec {
		codec => "json"
		command => "bash /etc/logstash/curl.sh"
		interval => 30
		tags => "stats"
	}
}

output {
	if "stats" in [tags] {
		elasticsearch {
			hosts => ["localhost:9200"]
			index => "index-test"
		}
	}
	if "file_test" in [tags] {
		pipeline { 
			send_to => [els, output_file] 
		}
	}
	stdout { codec => rubydebug }
}

file els.conf

input {
	pipeline { address => els } 
}

filter {
....

}

output {
		if "old_date" in [tags] {
			file {
			  path => "/opt/logstash/old/old_log.log"
			  codec => line { format => "%{message}"}
			}
		}
		if "old_date" not in [tags] {
			elasticsearch {
				hosts => ["localhost:9200"]
				index => "index-test2"
			}
		}
}

output_file.conf

input {
	pipeline { address => output_file } 
}

output {
		file
		{
			path => "/opt/logstash/new/new.log"
			codec => line { format => "%{message}"}
		}
}

Is this behavior a bug? If not, how can this be resolved?

Thank you in advance

2

It looks like the pipelines are not being picked up, and the fact that you have stored your configs in the standard config directory to some extent hides this. I suspect the reason for this is that your config file is named pipeline.yml and not pipelines.yml, which the documentation indicates is the correct name.

3

it's an entry error. it is pipelines.yml which is well in the configuration directory.
I just modified the contents of the pipelines.yml file in its default directory after installation.

4

The pronleme has been resolved by removing the conf queue.type: persisted in the pipeline configuration

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.