# What is the best way to ship logs from one Logstash to another Logstash? **URL:** \ **Category:** Logstash\ **Created:** [February 18, 2018, 5:01pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359 "2018-02-18T17:01:58Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [February 18, 2018, 5:01pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/1 "2018-02-18T17:01:58Z")
Hi, What would be the best way to ship logs from one Logstash to another instance of Logstash? I am currently using http input and http output plugins. However, I am not sure if all the logs are being shipped without any logs being dropped reason being I see the following stack trace very often, `{:timestamp=>"2017-12-18T15:13:40.243000-0500", :message=>"[HTTP Output Failure] Encountered non-200 HTTP code 200", :response_code=>502, :url=>"http://xxx.xx.xxx.xx:xxx", :event=>#, :level=>:error}` I don't understand the reason for this error. How can I ensure that all the logs are being shipped without being dropped? Can someone please suggest? Thanks ---
**Author:** ![rcowart](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rcowart/32/88091_2.png) [@rcowart](https://discuss.elastic.co/u/rcowart)\ **Post date:** [February 18, 2018, 5:10pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/2 "2018-02-18T17:10:27Z")
IMO the easiest way that is both very scalable and reliable is to use redis between Logstash instances/pipelines. Combining redis and the multi-pipline capabilities of Logstash 6.x, I have had over 40 pipelines cooperating seamlessly with each other in a microservices-like deployment model. Kafka provides more options, and it is awesome if you need those capabilities, but when it comes to a combination of simplicity and scale redis is the better choice. ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [February 18, 2018, 5:46pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/3 "2018-02-18T17:46:58Z")
Hi Robert, Thanks for your quick response. We are planning to use Kafka as a messaging queue. I hope there won't be any issues when Kafka comes into picture. But it takes some time for Kafka to be in place so before that I wanted to figure out the issues associated with http plugin. Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 18, 2018, 6:31pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/4 "2018-02-18T18:31:16Z")
I believe the recommended way is to use a lumberjack output plugin with a beats input plugin, as these should be compatible. ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [February 18, 2018, 7:10pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/5 "2018-02-18T19:10:13Z")
Hi Christian, I've come across Lumberjack plugin but the problem here is I already configured Logstash on nearly 50 nodes(Linux and windows) with http input and output plugins. If at all I need to replace http with Lumberjack, then it would be on 50 nodes which is a painful task. When we start using Kafka, I will need to do it on 50 nodes but in the mean time I am trying to figure out the issue with http plugin instead of installing Lumberjack plugin. Could you please tell me if there is a way for my problem? Thanks ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 20, 2018, 1:39pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/6 "2018-02-20T13:39:43Z")
> I've come across Lumberjack plugin but the problem here is I already configured Logstash on nearly 50 nodes(Linux and windows) with http input and output plugins. If at all I need to replace http with Lumberjack, then it would be on 50 nodes which is a painful task. Seriously, are you configuring those 50 machines by hand? ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [February 20, 2018, 3:18pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/7 "2018-02-20T15:18:30Z")
I have logs coming from all the 50 machines. What would be the alternative? I assume it to be Ansible. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [February 20, 2018, 3:49pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/8 "2018-02-20T15:49:57Z")
Ansible or a similar tool, yes. Don't ever configure machines by hand. ---
**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\ **Post date:** [February 20, 2018, 3:52pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/9 "2018-02-20T15:52:08Z")
Hmm... Would you happen to know why there are issues with Logstash HTTP plugins? Is there any way where Logstash can say "No logs are being dropped"? ---
**Author:** ![jspeer](https://avatars.discourse-cdn.com/v4/letter/j/5f8ce5/32.png) [@jspeer](https://discuss.elastic.co/u/jspeer)\ **Post date:** [March 1, 2018, 10:10pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/10 "2018-03-01T22:10:33Z")
This has always been my way of doing this, until I had to encrypt the data. Open source redis doesn't support TLS. So trying out lumberjack now. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 29, 2018, 10:10pm UTC](https://discuss.elastic.co/t/what-is-the-best-way-to-ship-logs-from-one-logstash-to-another-logstash/120359/11 "2018-03-29T22:10:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.