Multiple logstash nodes use file-output plugin to output messages to one file in a shared file system?


(Keith Tt) #1

ELK version: 5.4.1

There are 3 logstash nodes in my BELK arch, and I configure load-balancing in filebeat.

if I use output-plugin to output the messages from 3 logstash nodes to a same file in a shared filesystem, will the new messages overwrite the old ones or will that be conflict?


(Christian Dahlqvist) #2

As there is no coordination around file writing between the Logstash instances, I would expect writing to a shared file to cause serious problems.


(Keith Tt) #3

Yes, this what I was thinking about...

So....I need to confirm with you that will this situation cause problem?

And, do you have a available method to do this?


(Christian Dahlqvist) #4

I can not see how it would work, so am reasonably sure it will cause problems. I have however never tested it.

The only way to do this is, as far as I know, to send data to a single Logstash instance that does all the writing.


(Keith Tt) #5

OK, I see...

By the way, it seems that there can define only one file-output in one pipeline config file? and this file will record all the events even if I use if statement.


(Christian Dahlqvist) #6

I am not sure I understand. Can you show an example of what you are trying to do?


(Keith Tt) #7

here is part of my pipeline config file:

input {
  beats {
    port => 5044
    codec => "json"
  }
}

output {
    if [type] == "zixun-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "zixun-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
    }}
    file {
        path => "/nh/esbk/my_backup/backup/kibana-nginx-access-%{+YYYY-MM-dd}.log"
    }
    if [type] == "water-nginx-access" {
    elasticsearch {
        hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"]
        index => "water-nginx-access-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
        template_overwrite => true
    }}
    ...
}

I desire to output messages to different file for every type.


(Keith Tt) #8

It seems that I should define multiple pipeline config files for different projects and startup multiple logstash instances on each logstash node?


(Magnus B├Ąck) #9

As there is no coordination around file writing between the Logstash instances, I would expect writing to a shared file to cause serious problems.

If Logstash is opening the output file with O_APPEND (which it should) then all write() operations will be made at the end of the file even if multiple processes write to the file concurrently. See http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html. This doesn't apply to Windows though.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.