# Can two logstash commands run in parallel to pull data in elastic search?

**URL:** <https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790>\
**Category:** Logstash\
**Created:** [January 9, 2016, 5:55pm UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790 "2016-01-09T17:55:27Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![maryam.tauheed](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@maryam.tauheed](https://discuss.elastic.co/u/maryam.tauheed)\
**Post date:** [January 9, 2016, 5:55pm UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/1 "2016-01-09T17:55:27Z")

</div>

I am tring to execute this command.  
"bin/logstash -f new.conf \< a.log and bin/logstash -f new.conf \< b.log"  
In my config file of logstash:  
output {  
elasticsearch {  
action=\>"index"  
hosts =\> ["localhost:9200"]  
index =\> "maryam\_index1"  
document\_type =\> "maryam\_index"  
}

# stdout { codec =\> rubydebug }

}  
i open two terminals and run these on separate ones together but it waits for first command to get executed, stores data in elastic search and then starts the other request. How to make them work together ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 9, 2016, 6:28pm UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/2 "2016-01-09T18:28:31Z")

</div>

But surely that's not the complete contents of new.conf, you have a stdin input there as well right?

Logstash has no limitation on how many instances can send data to Elasticsearch at the same time and Elasticsearch doesn't limit the number of active clients. Did you actually try to run each Logstash command in its own terminal?

---

<div class="post-metadata">

**Author:** ![maryam.tauheed](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@maryam.tauheed](https://discuss.elastic.co/u/maryam.tauheed)\
**Post date:** [January 10, 2016, 6:02pm UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/3 "2016-01-10T18:02:05Z")

</div>

This is the complete new.conf

input {  
stdin {  
type =\> "stdin-type"  
}

filter {  
csv {  
columns =\> [  
"IPAddress",  
"modificationTimeISO",  
"size",  
"md5",  
"fileName",

```
    ] 
    separator =>"	"       
    }

```

}  
output {  
elasticsearch {  
action=\>"index"  
hosts =\> ["localhost:9200"]  
index =\> "maryam\_index1"  
document\_type =\> "maryam\_index"  
}

# stdout { codec =\> rubydebug }

}

---

<div class="post-metadata">

**Author:** ![maryam.tauheed](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@maryam.tauheed](https://discuss.elastic.co/u/maryam.tauheed)\
**Post date:** [January 10, 2016, 6:03pm UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/4 "2016-01-10T18:03:31Z")

</div>

What do you mean by saying "in its own terminal" ? I simply gave these commands concurrently on two different terminals.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 10, 2016, 7:01pm UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/5 "2016-01-10T19:01:03Z")

</div>

So you ran `bin/logstash -f new.conf < a.log` in terminal 1 and `bin/logstash -f new.conf < b.log` in terminal 2? And the Logstash process in the second terminal didn't start until the first one was finished?

---

<div class="post-metadata">

**Author:** ![maryam.tauheed](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@maryam.tauheed](https://discuss.elastic.co/u/maryam.tauheed)\
**Post date:** [January 11, 2016, 5:08am UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/6 "2016-01-11T05:08:44Z")

</div>

Yes sir

---

<div class="post-metadata">

**Author:** ![maryam.tauheed](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@maryam.tauheed](https://discuss.elastic.co/u/maryam.tauheed)\
**Post date:** [January 11, 2016, 8:47am UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/7 "2016-01-11T08:47:31Z")

</div>

Please reply as soon as you see this

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 11, 2016, 9:01am UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/8 "2016-01-11T09:01:47Z")

</div>

I have nothing to add. As I said, neither Logstash nor Elasticsearch contains code for this kind of synchronization. I suspect there's either a misunderstanding somewhere or something local on your machine.

---

<div class="post-metadata">

**Author:** ![maryam.tauheed](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@maryam.tauheed](https://discuss.elastic.co/u/maryam.tauheed)\
**Post date:** [January 11, 2016, 9:14am UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/9 "2016-01-11T09:14:02Z")

</div>

I have tried it on two different machines, it is acting the same way. Do you think it is the right behaviour for logstash to wait for first command to finish execution ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 11, 2016, 9:28am UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/10 "2016-01-11T09:28:15Z")

</div>

No, Logstash is not supposed to behave that way.

---

<div class="post-metadata">

**Author:** ![maryam.tauheed](https://avatars.discourse-cdn.com/v4/letter/m/71c47a/32.png) [@maryam.tauheed](https://discuss.elastic.co/u/maryam.tauheed)\
**Post date:** [January 11, 2016, 11:47am UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/11 "2016-01-11T11:47:52Z")

</div>

Thank you so much for your time. your replies were very helpful

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:16am UTC](https://discuss.elastic.co/t/can-two-logstash-commands-run-in-parallel-to-pull-data-in-elastic-search/38790/12 "2017-07-06T05:16:09Z")

</div>