# \[SOLVED\] Chaining Pipelines Together **URL:** \ **Category:** Logstash\ **Created:** [December 20, 2017, 7:48pm UTC](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708 "2017-12-20T19:48:56Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![bgeels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bgeels/32/25678_2.png) [@bgeels](https://discuss.elastic.co/u/bgeels)\ **Post date:** [December 20, 2017, 7:48pm UTC](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708/1 "2017-12-20T19:48:57Z")
Hey All, Has anyone tried to chain multiple pipelines together on a single host? I'm trying to send from http-\>A-\>B-\>C-\>logfile. Connecting the pipelines together via the tcp input/output plugins. I have debug level logs turned on and see that pipeline A receives the input on http:8080 and sends out tcp:8081 but then nothing after that. Pipeline B is listening on tcp:8081 and will send out tcp:8082 and so on. Debug Logs: ```auto [2017-12-20T19:38:13,635][DEBUG][logstash.pipeline] filter received {"event"=>{"start_time"=>"1483228800", "network"=>"network_a", "duration"=>60, "node"=>"node_a", "severity"=>"critical", "@timestamp"=>2017-12-20T19:38:13.613Z, "creation_time"=>1513798693, "headers"=>{"http_connection"=>"TE, close", "content_length"=>"206", "request_uri"=>"/", "http_user_agent"=>"libwww-perl/6.05", "http_te"=>"deflate,gzip;q=0.3", "request_path"=>"/", "content_type"=>"application/json", "http_host"=>"127.0.0.1:8080", "http_version"=>"HTTP/1.1", "request_method"=>"PUT"}, "end_time"=>"1491004800", "device"=>"device_0", "@version"=>"1", "description"=>"The description_0", "host"=>"127.0.0.1"}} [2017-12-20T19:38:13,644][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"rule_matched", "value"=>["NONE"]} [2017-12-20T19:38:13,648][DEBUG][logstash.pipeline] output received {"event"=>{"start_time"=>"1483228800", "network"=>"network_a", "duration"=>60, "node"=>"node_a", "severity"=>"critical", "@timestamp"=>2017-12-20T19:38:13.613Z, "creation_time"=>1513798693, "headers"=>{"http_connection"=>"TE, close", "content_length"=>"206", "request_uri"=>"/", "http_user_agent"=>"libwww-perl/6.05", "http_te"=>"deflate,gzip;q=0.3", "request_path"=>"/", "content_type"=>"application/json", "http_host"=>"127.0.0.1:8080", "http_version"=>"HTTP/1.1", "request_method"=>"PUT"}, "end_time"=>"1491004800", "rule_matched"=>"NONE", "device"=>"device_0", "@version"=>"1", "description"=>"The description_0", "host"=>"127.0.0.1"}} [2017-12-20T19:38:13,663][DEBUG][logstash.outputs.tcp] Opened connection {:client=>"127.0.0.1:8081"} [2017-12-20T19:38:13,678][DEBUG][logstash.codecs.line] config LogStash::Codecs::Line/@id = "line_1300f797-5be3-4bda-b825-3f847bd821c1" [2017-12-20T19:38:13,678][DEBUG][logstash.codecs.line] config LogStash::Codecs::Line/@enable_metric = true [2017-12-20T19:38:13,679][DEBUG][logstash.codecs.line] config LogStash::Codecs::Line/@charset = "UTF-8" [2017-12-20T19:38:13,679][DEBUG][logstash.codecs.line] config LogStash::Codecs::Line/@delimiter = "\n" [2017-12-20T19:38:13,687][DEBUG][io.netty.util.Recycler] -Dio.netty.recycler.maxCapacity: 262144 [2017-12-20T19:38:13,687][DEBUG][io.netty.util.Recycler] -Dio.netty.recycler.maxSharedCapacityFactor: 2 [2017-12-20T19:38:13,687][DEBUG][io.netty.util.Recycler] -Dio.netty.recycler.linkCapacity: 16 [2017-12-20T19:38:13,704][DEBUG][io.netty.buffer.AbstractByteBuf] -Dio.netty.buffer.bytebuf.checkAccessible: true [2017-12-20T19:38:13,707][DEBUG][io.netty.util.ResourceLeakDetectorFactory] Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@4f57aac0 ``` Configs: > ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 20, 2017, 8:07pm UTC](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708/2 "2017-12-20T20:07:47Z")
What if you add a stdout to each section to see what is happening? Also debug logging may help. ---
**Author:** ![bgeels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bgeels/32/25678_2.png) [@bgeels](https://discuss.elastic.co/u/bgeels)\ **Post date:** [December 20, 2017, 8:22pm UTC](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708/3 "2017-12-20T20:22:59Z")
I'll try adding the stdout to the sections. Debug logging is currently enabled, I've included the output in the post above. ---
**Author:** ![bgeels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bgeels/32/25678_2.png) [@bgeels](https://discuss.elastic.co/u/bgeels)\ **Post date:** [December 21, 2017, 3:10pm UTC](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708/4 "2017-12-21T15:10:46Z")
Ok I got a little bit closer by applying the `json_lines` codec to the output block of the first pipeline. Now I see the event received and forwarded by the second pipeline. I can get the event forwarded through all the pipelines by setting this codec but it looks like the json gets escaped each time this happens. So it seems like I have to explicitly set some codec to get this to work correctly. Is there a codec I can use to ouput just the decorated event to the next pipeline for processing? I've included the debug logs of the event going through two pipelines below. Thanks for your help! ```auto [2017-12-21T15:05:03,122][DEBUG][logstash.pipeline] filter received {"event"=>{"end_time"=>"1491004800", "description"=>"The description_0", "headers"=>{"http_connection"=>"TE, close", "http_user_agent"=>"libwww-perl/6.05", "request_method"=>"PUT", "request_path"=>"/", "request_uri"=>"/", "http_host"=>"127.0.0.1:8080", "content_length"=>"206", "http_version"=>"HTTP/1.1", "http_te"=>"deflate,gzip;q=0.3", "content_type"=>"application/json"}, "network"=>"network_a", "start_time"=>"1483228800", "creation_time"=>1513868703, "host"=>"127.0.0.1", "node"=>"node_a", "@timestamp"=>2017-12-21T15:05:03.111Z, "severity"=>"critical", "duration"=>60, "device"=>"device_0", "@version"=>"1"}} [2017-12-21T15:05:03,122][DEBUG][logstash.util.decorators] filters/LogStash::Filters::Mutate: adding value to field {"field"=>"rule_matched", "value"=>["NONE"]} [2017-12-21T15:05:03,123][DEBUG][logstash.pipeline] output received {"event"=>{"end_time"=>"1491004800", "description"=>"The description_0", "headers"=>{"http_connection"=>"TE, close", "http_user_agent"=>"libwww-perl/6.05", "request_method"=>"PUT", "request_path"=>"/", "request_uri"=>"/", "http_host"=>"127.0.0.1:8080", "content_length"=>"206", "http_version"=>"HTTP/1.1", "http_te"=>"deflate,gzip;q=0.3", "content_type"=>"application/json"}, "network"=>"network_a", "start_time"=>"1483228800", "creation_time"=>1513868703, "rule_matched"=>"NONE", "host"=>"127.0.0.1", "node"=>"node_a", "@timestamp"=>2017-12-21T15:05:03.111Z, "severity"=>"critical", "duration"=>60, "device"=>"device_0", "@version"=>"1"}} [2017-12-21T15:05:03,135][DEBUG][logstash.pipeline] filter received {"event"=>{"message"=>"{\"end_time\":\"1491004800\",\"description\":\"The description_0\",\"headers\":{\"http_connection\":\"TE, close\",\"http_user_agent\":\"libwww-perl/6.05\",\"request_method\":\"PUT\",\"request_path\":\"/\",\"request_uri\":\"/\",\"http_host\":\"127.0.0.1:8080\",\"content_length\":\"206\",\"http_version\":\"HTTP/1.1\",\"http_te\":\"deflate,gzip;q=0.3\",\"content_type\":\"application/json\"},\"network\":\"network_a\",\"start_time\":\"1483228800\",\"creation_time\":1513868703,\"rule_matched\":\"NONE\",\"host\":\"127.0.0.1\",\"node\":\"node_a\",\"@timestamp\":\"2017-12-21T15:05:03.111Z\",\"severity\":\"critical\",\"duration\":60,\"device\":\"device_0\",\"@version\":\"1\"}", "@timestamp"=>2017-12-21T15:05:03.124Z, "host"=>"localhost", "@metdata"=>{"ip_address"=>"127.0.0.1"}, "port"=>44098, "@version"=>"1"}} [2017-12-21T15:05:03,135][DEBUG][logstash.pipeline] output received {"event"=>{"message"=>"{\"end_time\":\"1491004800\",\"description\":\"The description_0\",\"headers\":{\"http_connection\":\"TE, close\",\"http_user_agent\":\"libwww-perl/6.05\",\"request_method\":\"PUT\",\"request_path\":\"/\",\"request_uri\":\"/\",\"http_host\":\"127.0.0.1:8080\",\"content_length\":\"206\",\"http_version\":\"HTTP/1.1\",\"http_te\":\"deflate,gzip;q=0.3\",\"content_type\":\"application/json\"},\"network\":\"network_a\",\"start_time\":\"1483228800\",\"creation_time\":1513868703,\"rule_matched\":\"NONE\",\"host\":\"127.0.0.1\",\"node\":\"node_a\",\"@timestamp\":\"2017-12-21T15:05:03.111Z\",\"severity\":\"critical\",\"duration\":60,\"device\":\"device_0\",\"@version\":\"1\"}", "@timestamp"=>2017-12-21T15:05:03.124Z, "host"=>"localhost", "@metdata"=>{"ip_address"=>"127.0.0.1"}, "port"=>44098, "@version"=>"1"}} ``` ---
**Author:** ![bgeels](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bgeels/32/25678_2.png) [@bgeels](https://discuss.elastic.co/u/bgeels)\ **Post date:** [December 21, 2017, 3:26pm UTC](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708/5 "2017-12-21T15:26:17Z")
And after more trial and error I found that setting `json_lines` as the codec in both the `input` and `output` blocks allows this to work the way I would expect. Although I'm not entirely sure why that is needed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2018, 3:26pm UTC](https://discuss.elastic.co/t/solved-chaining-pipelines-together/112708/6 "2018-01-18T15:26:24Z")
Closed on January 18, 2018, 3:26pm UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.