Can two logstash commands run in parallel to pull data in elastic search?

Elastic Stack Logstash
1

I am tring to execute this command.
"bin/logstash -f new.conf < a.log and bin/logstash -f new.conf < b.log"
In my config file of logstash:
output {
elasticsearch {
action=>"index"
hosts => ["localhost:9200"]
index => "maryam_index1"
document_type => "maryam_index"
}

stdout { codec => rubydebug }

}
i open two terminals and run these on separate ones together but it waits for first command to get executed, stores data in elastic search and then starts the other request. How to make them work together ?

2

But surely that's not the complete contents of new.conf, you have a stdin input there as well right?

Logstash has no limitation on how many instances can send data to Elasticsearch at the same time and Elasticsearch doesn't limit the number of active clients. Did you actually try to run each Logstash command in its own terminal?

3

This is the complete new.conf

input {
stdin {
type => "stdin-type"
}

filter {
csv {
columns => [
"IPAddress",
"modificationTimeISO",
"size",
"md5",
"fileName",

    ] 
    separator =>"	"       
    }

}
output {
elasticsearch {
action=>"index"
hosts => ["localhost:9200"]
index => "maryam_index1"
document_type => "maryam_index"
}

stdout { codec => rubydebug }

}

4

What do you mean by saying "in its own terminal" ? I simply gave these commands concurrently on two different terminals.

5

So you ran bin/logstash -f new.conf < a.log in terminal 1 and bin/logstash -f new.conf < b.log in terminal 2? And the Logstash process in the second terminal didn't start until the first one was finished?

6

Yes sir

7

Please reply as soon as you see this

8

I have nothing to add. As I said, neither Logstash nor Elasticsearch contains code for this kind of synchronization. I suspect there's either a misunderstanding somewhere or something local on your machine.

9

I have tried it on two different machines, it is acting the same way. Do you think it is the right behaviour for logstash to wait for first command to finish execution ?

10

No, Logstash is not supposed to behave that way.

11

Thank you so much for your time. your replies were very helpful