Multiple logstash nodes use file-output plugin to output messages to one file in a shared file system?

Elastic Stack Logstash
1

ELK version: 5.4.1

There are 3 logstash nodes in my BELK arch, and I configure load-balancing in filebeat.

if I use output-plugin to output the messages from 3 logstash nodes to a same file in a shared filesystem, will the new messages overwrite the old ones or will that be conflict?

2

As there is no coordination around file writing between the Logstash instances, I would expect writing to a shared file to cause serious problems.

3

Yes, this what I was thinking about...

So....I need to confirm with you that will this situation cause problem?

And, do you have a available method to do this?

4

I can not see how it would work, so am reasonably sure it will cause problems. I have however never tested it.

The only way to do this is, as far as I know, to send data to a single Logstash instance that does all the writing.

5

OK, I see...

By the way, it seems that there can define only one file-output in one pipeline config file? and this file will record all the events even if I use if statement.

6

I am not sure I understand. Can you show an example of what you are trying to do?

7

here is part of my pipeline config file:

input {
  beats {
    port => 5044
    codec => "json"
  }
}

output {
    if [type] == "zixun-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "zixun-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
    }}
    file {
        path => "/nh/esbk/my_backup/backup/kibana-nginx-access-%{+YYYY-MM-dd}.log"
    }
    if [type] == "water-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "water-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
    }}
    ...
}

I desire to output messages to different file for every type.

8

It seems that I should define multiple pipeline config files for different projects and startup multiple logstash instances on each logstash node?

9

As there is no coordination around file writing between the Logstash instances, I would expect writing to a shared file to cause serious problems.

If Logstash is opening the output file with O_APPEND (which it should) then all write() operations will be made at the end of the file even if multiple processes write to the file concurrently. See write. This doesn't apply to Windows though.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.