# Logstash mirror output **URL:** \ **Category:** Logstash\ **Created:** [May 8, 2017, 9:19pm UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982 "2017-05-08T21:19:38Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 8, 2017, 9:19pm UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/1 "2017-05-08T21:19:39Z")
Hi, Is there a way to mirror a logstash input to 2 or more outputs ? I added 2 outputs to my winlogbeat.yml but only seems to take the first. Trying to send one pipeline to ES and a copy to another output. Thanks. ---
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 9, 2017, 1:53am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/2 "2017-05-09T01:53:30Z")
Looks like the answer to this may be unsupported by logstash but solved by adding "redis" ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 9, 2017, 5:05am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/3 "2017-05-09T05:05:38Z")
> Is there a way to mirror a logstash input to 2 or more outputs ? Logstash sends all events to all outputs listed in its configuration, i.e. "mirroring" is the default behavior. > I added 2 outputs to my winlogbeat.yml but only seems to take the first. Well, Winlogbeat is a whole different story than Logstash. But yes, I think the Beats programs only send their data to one output. ---
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 11, 2017, 4:45am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/4 "2017-05-11T04:45:30Z")
This seems to be an option but I don't see any docs bout it. Seems you use one output then plugins needed for that output. output { elasticsearch { hosts =\> "192.168.1.10:9200" sniffing =\> false manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } file { path =\> "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" codec =\> json { format =\> "custom format: %{message}"} # codec =\> json } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 11, 2017, 5:18am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/5 "2017-05-11T05:18:55Z")
> This seems to be an option but I don't see any docs bout it. I don't know where or if it's documented. > Seems you use one output then plugins needed for that output. No, that's not how it works. Unless you use conditionals, events from all inputs are routed to all outputs via all filters. ---
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 11, 2017, 5:30am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/6 "2017-05-11T05:30:04Z")
That's what I think I want. All beats input is written in json output for archiving and replaying and a copy sent to ES . What's strange is the docs say "_By default, this output writes one event per line in json format. You can customize the line format using the line codec like"_ but whatever codec I use it drops the json and starts printing plain text. output { file { path =\> "/opt/syslog-ng/logs/logstash/%{host}-%{+YYYY-MM-dd}.json" # codec =\> json # codec =\> line { format =\> "custom format: %{message}"} } elasticsearch { hosts =\> "192.168.1.10:9200" sniffing =\> false manage\_template =\> false index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}" document\_type =\> "%{[@metadata][type]}" } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 11, 2017, 5:38am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/7 "2017-05-11T05:38:18Z")
The default codec works for me: ```nohighlight $ cat test.config input { stdin { } } output { file { path => "/tmp/file-output" } } $ echo hello | /opt/logstash/bin/logstash -f test.config Settings: Default pipeline workers: 8 Pipeline main started Pipeline main has been shutdown stopping pipeline {:id=>"main"} $ cat /tmp/file-output {"message":"hello","@version":"1","@timestamp":"2017-05-11T05:37:27.210Z","host":"lnxolofon"} ``` ---
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 11, 2017, 5:48am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/8 "2017-05-11T05:48:38Z")
Yea but if the default is json then why does the codec =\> json remove the json on the output ? Ideally I wanted to send my output to ES and TCP but I can't seem to get the json codec to wok in a TCP output so I started using file to test. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 11, 2017, 7:34am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/9 "2017-05-11T07:34:51Z")
> Yea but if the default is json then why does the codec =\> json remove the json on the output ? I don't see that it does. ```plaintext $ cat test.config input { stdin { } } output { file { path => "/tmp/file-output" codec => json } } $ echo hello | /opt/logstash/bin/logstash -f test.config Settings: Default pipeline workers: 8 Pipeline main started Pipeline main has been shutdown stopping pipeline {:id=>"main"} $ cat /tmp/file-output {"message":"hello","@version":"1","@timestamp":"2017-05-11T07:30:38.577Z","host":"lnxolofon"} ``` (Note that you probably want to use the json\_lines codec with the file output to get each line terminated by a newline character.) This thread might get more interesting if you show what you get instead of describing the results. A recipe for reproducing the unwanted behavior (like above) is often useful. ---
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 11, 2017, 2:48pm UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/10 "2017-05-11T14:48:01Z")
Would that work the same way in a TCP output ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 11, 2017, 5:42pm UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/11 "2017-05-11T17:42:18Z")
Yes. But why ask hypothetical questions when you can try it out? ---
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 11, 2017, 6:22pm UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/12 "2017-05-11T18:22:52Z")
It's not hypothetical I have been trying to use TCP forwards with no luck. Wanted to see if I get any readable output from logstash in a format that could be imported. (Ditch effort after days of failing TCP output) For syslog I have syslog-ng sending a copy of data to ES,SPLUNK,FileArchive host -TCP/UDP:514 ---\> rsyslog relay---TCP:514--\> syslog-ng --\> ES,SPLUNKForwaderInput,FileArchive Trying to split beats data for the same destinations. I have Logstash running on the remote rsyslog relays and the syslog-ng server. I can forward TCP data to either of them but the data does not seem to bet getting sent or ingested correctly. Looks like a huge JSON blob with no separation in events. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 11, 2017, 6:28pm UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/13 "2017-05-11T18:28:17Z")
Did you try using the json\_lines codec? ---
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 12, 2017, 1:29am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/14 "2017-05-12T01:29:53Z")
Yep that seems to help. There seemed to be a cache buildup in LS which when it flushed the recieving parser could not pickup the stream broken up between packets. Added it to the TCP output. tcp { host =\> "192.168.1.16" port =\> "5140" mode =\> "client" codec =\> "json\_lines" } **# SYSLOG-NG is listening on PORT 5140.** source s\_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));}; destination d\_jfile { file("/opt/syslog-ng/logs/$HOST\_FROM-$R\_HOUR.json" template("$(format-json --scope dot-nv-pairs)\n"));}; log { source(s\_BEATS); parser(p\_jsoneventv0); destination (d\_jfile); }; This is what I am getting from syslog-ng trying to read the input. Using this parser for the json. [https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#json-parser-options](https://www.balabit.com/documents/syslog-ng-ose-3.9-guides/en/syslog-ng-ose-guide-admin/html-single/index.html#json-parser-options) [2017-05-11T21:22:17.436365] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp\_connect\_rtt":{"us":9000},"monitor":"http@[http://192.168.1.16:9200](http://192.168.1.16:9200)","type":"http","http\_rtt":{"us":13000},"url":"[http://192.168.1.16:9200](http://192.168.1.16:9200)","tags":["beats\_input\_raw\_event"],"duration":{"us":22001},"@timestamp":"2017-05-12T01:22:21.258Z","rtt":{"us":22001},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' [2017-05-11T21:22:17.436497] **Error extracting JSON members into LogMessage as the top-level JSON object is not an object** ; input='{"scheme":"http","ip":"192.168.1.16","tcp\_connect\_rtt":{"us":9000},"monitor":"http@[http://192.168.1.16:9200](http://192.168.1.16:9200)","type":"http","http\_rtt":{"us":13000},"url":"[http://192.168.1.16:9200](http://192.168.1.16:9200)","tags":["beats\_input\_raw\_event"],"duration":{"us":22001},"@timestamp":"2017-05-12T01:22:21.258Z","rtt":{"us":22001},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' [2017-05-11T21:22:17.436523] Message parsing complete; result='0', rule='p\_jsoneventv0', location='/etc/syslog-ng/syslog-ng.conf:18:14' ---
**Author:** ![PandKing](https://avatars.discourse-cdn.com/v4/letter/p/258eb7/32.png) [@PandKing](https://discuss.elastic.co/u/PandKing)\ **Post date:** [May 12, 2017, 2:12am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/15 "2017-05-12T02:12:11Z")
SUCCESS ! Working in the lab. Update, I switched the syslog-ng parser from json to kv-pairs. If I knew more json RFC I'd probably know why. **/etc/logstash/conf.d/beats-logstash.conf** input { beats { port =\> 5044 } } output { file { path =\> "/opt/syslog-ng/logstash/%{host}-%{+YYYY-MM-dd}.json" } tcp { host =\> "192.168.1.16" port =\> "5140" mode =\> "client" codec =\> "json\_lines" } **/etc/syslog-ng/syslog-ng.conf** # syslog-ng listens on port 5140 for beats output. source s\_BEATS {network(port(5140) log-msg-size(65536) flags(no-parse));}; destination d\_jfile { file("/opt/syslog-ng/logs/$HOST\_FROM-$R\_HOUR.json"); }; log { source(s\_BEATS); parser {kv-parser();}; destination (d\_jfile); }; DEBUG Output from sysog-ng /usr/sbin/syslog-ng --debug -F -f /etc/syslog-ng/syslog-ng.conf [2017-05-11T22:03:21.454954] Incoming log entry; line='{"scheme":"http","ip":"192.168.1.16","tcp\_connect\_rtt":{"us":2000},"monitor":"http@[http://192.168.1.16:9200](http://192.168.1.16:9200)","type":"http","http\_rtt":{"us":2000},"url":"[http://192.168.1.16:9200](http://192.168.1.16:9200)","tags":["beats\_input\_raw\_event"],"duration":{"us":5000},"@timestamp":"2017-05-12T02:03:25.258Z","rtt":{"us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' [2017-05-11T22:03:21.455021] Message parsing complete; result='1' [2017-05-11T22:03:21.455077] Outgoing message; message='May 11 22:03:21 hal {"scheme":"http","ip":"192.168.1.16","tcp\_connect\_rtt":{"us":2000},"monitor":"http@[http://192.168.1.16:9200](http://192.168.1.16:9200)","type":"http","http\_rtt":{"us":2000},"url":"[http://192.168.1.16:9200](http://192.168.1.16:9200)","tags":["beats\_input\_raw\_event"],"duration":{"us":5000},"@timestamp":"2017-05-12T02:03:25.258Z","rtt":{"us":5000},"port":9200,"response":{"status":200},"beat":{"hostname":"TYLER-LAPTOP","name":"TYLER-LAPTOP","version":"5.4.0"},"@version":"1","host":"TYLER-LAPTOP","up":true}' ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 9, 2017, 2:21am UTC](https://discuss.elastic.co/t/logstash-mirror-output/84982/16 "2017-06-09T02:21:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.