# Large amounts of bulk data sent from Logstash to Elasticsearch after using pipelines

**URL:** <https://discuss.elastic.co/t/large-amounts-of-bulk-data-sent-from-logstash-to-elasticsearch-after-using-pipelines/298092>\
**Category:** Logstash\
**Created:** [February 23, 2022, 11:00pm UTC](https://discuss.elastic.co/t/large-amounts-of-bulk-data-sent-from-logstash-to-elasticsearch-after-using-pipelines/298092 "2022-02-23T23:00:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![userR](https://avatars.discourse-cdn.com/v4/letter/u/22d042/32.png) [@userR](https://discuss.elastic.co/u/userR)\
**Post date:** [February 23, 2022, 11:00pm UTC](https://discuss.elastic.co/t/large-amounts-of-bulk-data-sent-from-logstash-to-elasticsearch-after-using-pipelines/298092/1 "2022-02-23T23:00:23Z")

</div>

I recently changed my configuration from one logstash.config file to pipelines to make it easier.

My pipelines.yml (replaced actual names with letters, main re-routes data to correct pipeline based on tag):

```auto
- pipeline.id: main
  path.config: "/usr/share/logstash/pipeline/main.config"

- pipeline.id: a
  path.config: "/usr/share/logstash/pipeline/a.config"

- pipeline.id: b
  path.config: "/usr/share/logstash/pipeline/b.config"

- pipeline.id: c
  path.config: "/usr/share/logstash/pipeline/c.config"

- pipeline.id: d
  path.config: "/usr/share/logstash/pipeline/d.config"

- pipeline.id: e
  path.config: "/usr/share/logstash/pipeline/e.config"

- pipeline.id: f
  path.config: "/usr/share/logstash/pipeline/f.config"

```

I see on Elasticsearch, a large amount of daily data being sent (was usually max 5mb, now it is 1 gb)  
I also see this error in the logstash logs:

```auto
[ERROR][logstash.outputs.elasticsearch][main][6bdcb4726a198461b0a3bc504bd116ed5ae4dc3a4e92f278a77b790bc12a0ceb] Attempted to send a bulk request but there are no living connections in the pool (perhaps Elasticsearch is unreachable or down?) {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError, :will_retry_in_seconds=>16}

```

```auto
[WARN][logstash.outputs.elasticsearch][main][39c5e157a8fc0ce37f379032b7514bc216a85707441f8b16bfdf1757bb7fd6a6] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ClientProtocolException] elasticsearch:9200 failed to respond {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::ClientProtocolException] elasticsearch:9200 failed to respond", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}

```

I also cannot query this data on Kibana and get this error:

```auto
Error: Batch request failed with status 503

```

Little confused as to what is going on here or why so much data is being sent.

---

<div class="post-metadata">

**Author:** ![userR](https://avatars.discourse-cdn.com/v4/letter/u/22d042/32.png) [@userR](https://discuss.elastic.co/u/userR)\
**Post date:** [February 23, 2022, 11:14pm UTC](https://discuss.elastic.co/t/large-amounts-of-bulk-data-sent-from-logstash-to-elasticsearch-after-using-pipelines/298092/2 "2022-02-23T23:14:06Z")

</div>

Looking at the data in Elasticsearch (using elastichead), I see 60 duplicates of data in one field for each log, here's an example of my configuration as I assume the issue is there:

main.config

```auto
input {
    beats { 
        port => 5044
        host => "0.0.0.0"
        ssl => false
    }
}
output {
    if [fields][log_type] == "a" {
        pipeline { send_to => a }
    } 
    else if [fields][log_type] == "b" {
        pipeline { send_to => b }
    }
    else if [fields][log_type] == "c" {
        pipeline { send_to => c }
    }
    else if [fields][log_type] == "d" {
        pipeline { send_to => d }
    }
    else if [fields][log_type] == "e" {
        pipeline { send_to => e }
    }
    else if [fields][log_type] == "f" {
        pipeline { send_to => f }
    }
}

```

a.config

```auto
input {
    pipeline {
        address => "a"
    }
}
filter {
  if [fields][log_type] == "a" {
    grok {

    }
    date {
      match => ["logdate", "YYYY-MM-dd HH:mm:ss,SSS"]
      target => "logdate"
    }
  }
}
output {
  if [fields][log_type] == "a" {
    elasticsearch {
      hosts => ["elasticsearch:9200"]
      index => "a-logdata=%{+YYYY.MM.dd}"
    }
  }
}

```

logstash.yml

```auto
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline

```

---

<div class="post-metadata">

**Author:** ![userR](https://avatars.discourse-cdn.com/v4/letter/u/22d042/32.png) [@userR](https://discuss.elastic.co/u/userR)\
**Post date:** [February 24, 2022, 12:57am UTC](https://discuss.elastic.co/t/large-amounts-of-bulk-data-sent-from-logstash-to-elasticsearch-after-using-pipelines/298092/3 "2022-02-24T00:57:01Z")

</div>

The issue was `path.config: /usr/share/logstash/pipeline` in logstash.yml

Not sure why this caused the data fields to be duplicated but everything is working fine now once deleted 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 24, 2022, 1:18am UTC](https://discuss.elastic.co/t/large-amounts-of-bulk-data-sent-from-logstash-to-elasticsearch-after-using-pipelines/298092/4 "2022-02-24T01:18:41Z")

</div>

Setting path.config prevents pipelines.yml being used. If path.config points to a directory then all the files are concatenated. Events are read from all of the inputs, processed by all of the filters, and then all of the events are sent to all of the filters. New users very often misunderstand this and think each configuration file stands alone.

For events with `[fields][log_type] == "a"` an event will reach the Elasticsearch output shown once directly from the beats input, and once after going through pipeline output/input pair a.

If you have any outputs with are not wrapped in a test of [fields][log\_type] they will receive each event from beats, and duplicates from pipelines a, b, c, d, e, and f.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 24, 2022, 1:19am UTC](https://discuss.elastic.co/t/large-amounts-of-bulk-data-sent-from-logstash-to-elasticsearch-after-using-pipelines/298092/5 "2022-03-24T01:19:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.