# LogStash with distributor pattern, relay pipelines problem

**URL:** <https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880>\
**Category:** Logstash\
**Created:** [December 3, 2021, 11:34am UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880 "2021-12-03T11:34:09Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![djehuty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djehuty/32/85386_2.png) [@djehuty](https://discuss.elastic.co/u/djehuty)\
**Post date:** [December 3, 2021, 11:34am UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/1 "2021-12-03T11:34:09Z")

</div>

Hi everyone,

i recently changed my LogStash configuration from multiple pipelines input to 3 pipelines, relaying on another local pipelines with distributor.

Now, i've tried to send and udp json to the udp pipeline, with the tag "localadmin" as u see in my configuration. The log arrives on the logstash node, but it seems that something doesnt work, as the data were not indexed on the elastic cluster.

Below my pipelines.yml configuration:

```auto
- pipeline.id: udp
  config.string: |
    input { udp { port => 514 } }

    output {
        if [type] == "localadmin" {  
          pipeline { send_to => localadmin } 
        } else if [type] == "passwordsynchronizer" { 
          pipeline { send_to => passwordsynchronizer }
        } else if [type] == "passwordstate" { 
          pipeline { send_to => keys }
        } else if [type] == "xibo" { 
          pipeline { send_to => xibo }
        }
    }
- pipeline.id: beats
  config.string: |
    input { beats { port => 5044 } }

    output {
        if [agent][type] == "filebeat" { 
          pipeline { send_to => exchange }
        } else if [agent][type] == "winlogbeat" {
          pipeline { send_to => sharepoint }
        }
    }
- pipeline.id: http
  config.string: |
    input { http { port => 80 } }

    output {
        if [type] == "gecov" {
          pipeline { send_to => gecov }
        }
    }
- pipeline.id: localadmin
  path.config: "/etc/logstash/conf.d/distributor/localadmin.conf"
- pipeline.id: passwordsynchronizer
  path.config: "/etc/logstash/conf.d/distributor/passwordsynchronizer.conf"
- pipeline.id: passwordstate
  path.config: "/etc/logstash/conf.d/distributor/keys.conf"
- pipeline.id: xibo
  path.config: "/etc/logstash/conf.d/distributor/xibo.conf"
- pipeline.id: exchange
  path.config: "/etc/logstash/conf.d/distributor/exchange.conf"
- pipeline.id: sharepoint-microsoft
  path.config: "/etc/logstash/conf.d/distributor/sharepoint.conf"
- pipeline.id: gecov
  path.config: "/etc/logstash/conf.d/distributor/gecov.conf"

```

And here the local pipeline to which it points:

```auto
input { pipeline { address => localadmin } }
output {
 elasticsearch {
      hosts => ["firstnode:port", "secondnode:port", "thirdnode:port"] 
      ssl => true
      ssl_certificate_verification => false
      user => someone
      password => 'somepassword'
      ilm_enabled => false
      index => "localadmin"
    }
  }

```

Thanks in advance 🙂

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 3, 2021, 1:10pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/2 "2021-12-03T13:10:47Z")

</div>

How does the document you are sending looks like?

You do not have `json` filter in the `udp` pipeline nor are you using the `json` codec in the input, so your message is not being parsed and there is no `type` field to be filtered.

---

<div class="post-metadata">

**Author:** ![djehuty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djehuty/32/85386_2.png) [@djehuty](https://discuss.elastic.co/u/djehuty)\
**Post date:** [December 3, 2021, 1:34pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/3 "2021-12-03T13:34:20Z")

</div>

Hi @leandrojmp, thank you for the fast replay.

This is the message:

```auto
[
{
    "type": "localadmin",
    "blabla": {
    "Users": {
      "Created": [],
      "Deleted": []
    },
    "Administrators": {
      "Added": [],
      "Removed": []
    },
    "campo": "localadmin",
    "Host": {
      "Type": [
        "Client"
      ],
      "Name": [
        "itk-dk-02"
      ]
    },
    "host": "ipaddress"
  }
}
]

```

Can you tell me where to put the codec filter in the input? directly below the

```auto
input { pipeline { address => localadmin } }

```

?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 3, 2021, 2:12pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/4 "2021-12-03T14:12:34Z")

</div>

No, you need to parse the message in the first pipeline that is receiving it, which is the one with the `udp` input.

You are using a field from the message to do the filtering, so if you do not parse it, you do not have the field to filter in your message and your output will not work correctly.

Try this:

```auto
input { 
    udp { 
        port => 514 
        codec => "json"
    } 
}

```

---

<div class="post-metadata">

**Author:** ![djehuty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djehuty/32/85386_2.png) [@djehuty](https://discuss.elastic.co/u/djehuty)\
**Post date:** [December 3, 2021, 2:22pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/5 "2021-12-03T14:22:30Z")

</div>

Ok just edited, using :

```auto
- pipeline.id: udp
  config.string: |
    input { udp { port => 514 codec => "json" } }

```

works fine for json input. I see the data on elastic in the right index.

But unfortunately on that port i have other inputs that aren't always formatted in json.

I need something like:

```auto
input { pipeline { address => localadmin codec => "json"} }

```

Maybe using the json codec directly in the filter section of th pipeline?

Thankyou.

---

<div class="post-metadata">

**Author:** ![djehuty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djehuty/32/85386_2.png) [@djehuty](https://discuss.elastic.co/u/djehuty)\
**Post date:** [December 3, 2021, 2:39pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/6 "2021-12-03T14:39:28Z")

</div>

Just understood my mistake, if logstash doesnt know the tipe of the data that comes, it can't sort by tags and send to other pipelines.  
The goal was to have fewer listening ports as possible on that node.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [December 3, 2021, 3:33pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/7 "2021-12-03T15:33:51Z")

</div>

I don't think that the `pipeline` input has the `codec` option, it seems to have only the `send_to` and `address` options.

You will need to filter for some string in the message since you can't parse the message.

Something like this:

```auto
input {
    upd {
        port => 514
    }
}
output {
    if "localadmin" in [message] {
        pipeline { send_to => localadmin }
    } else if "passwordsyncrhonizer" in [message] { 
        pipeline { send_to => passwordsynchronizer }
    } other else if conditions
}

```

I would also suggest that you use `pipelines.yml` to only point to the configs, and have the configs in separated files.

For example, create a `udp.conf` file with the udp pipeline and just point to this file in the `pipelines.yml` file, having the configurations in the `pipelines.yml` can lead to confusion and mistake if your configuration grows.

```auto
- pipeline.id: udp
  path.config: "/etc/logstash/conf.d/udp.conf"

```

---

<div class="post-metadata">

**Author:** ![djehuty](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/djehuty/32/85386_2.png) [@djehuty](https://discuss.elastic.co/u/djehuty)\
**Post date:** [December 3, 2021, 4:12pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/8 "2021-12-03T16:12:49Z")

</div>

Perfect, it works. Thanks 🖤

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 31, 2021, 4:13pm UTC](https://discuss.elastic.co/t/logstash-with-distributor-pattern-relay-pipelines-problem/290880/9 "2021-12-31T16:13:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.