# The logstash'pipeline is running, but elasticsearch didnt create an index for data

**URL:** <https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768>\
**Category:** Logstash\
**Created:** [May 15, 2020, 7:54am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768 "2020-05-15T07:54:51Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joseph-Gan-wk](https://avatars.discourse-cdn.com/v4/letter/j/ba9def/32.png) [@Joseph-Gan-wk](https://discuss.elastic.co/u/Joseph-Gan-wk)\
**Post date:** [May 15, 2020, 7:54am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/1 "2020-05-15T07:54:51Z")

</div>

Hi all, i am facing an issue is the elasticsearch did not create an index for data on Kibana.  
I checked the Logstash-plain.log that shows two pipelines are running.

```auto
[2020-05-15T15:42:41,124][INFO][logstash.agent] Pipelines running {:count=>2, :running_pipelines=>[:logstash_nagios, :logstash_cardax], :non_running_pipelines=>[]}

```

but, only the [logstash\_nagios] can get the output on kibana from its index.  
Cannot find the output for [Logstash\_cardax], even its index.  
both setting of output in config file is the same.

---

<div class="post-metadata">

**Author:** ![hunsw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hunsw/32/93637_2.png) [@hunsw](https://discuss.elastic.co/u/hunsw)\
**Post date:** [May 15, 2020, 8:23am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/2 "2020-05-15T08:23:02Z")

</div>

Why do you need two separate pipelines? You can set the index name in the Elasticsearch output of Logstash.

Also, it would help if you posted your config.

---

<div class="post-metadata">

**Author:** ![Joseph-Gan-wk](https://avatars.discourse-cdn.com/v4/letter/j/ba9def/32.png) [@Joseph-Gan-wk](https://discuss.elastic.co/u/Joseph-Gan-wk)\
**Post date:** [May 15, 2020, 8:35am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/3 "2020-05-15T08:35:25Z")

</div>

Hi,@hunsw  
This is because the data source and structure of both data is differ, so i just try to make it separate.

This is config.file for [logstash\_cardax]

```auto
input {
  file {
    path => ["/home/w/ctest.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
} #End of FILTER

output {
  elasticsearch {
    hosts => ["http://xx.xx.xx.xxx:9200/"]
    index => "cctest-log"
    user => "elastic"
    password => "xxxxxxxxxxxxxxxxx"

  }
}

```

This is config.file for [logstash\_nagios]

```auto
input {
  file {
    path => ["/usr/local/nagios/var/nagios.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
}

output {
  elasticsearch {
    hosts => ["http://xx.xx.xx.xxx:9200/"]
    index => "<nagios-{now/d}-000001>"
    user => "elastic"
    password => "xxxxxxxxxxxxxxxx"
  }
}

```

this is pipeline.yml

```auto
- pipeline.id: logstash_nagios
  path.config: "/etc/logstash/conf.d/nagios.conf"
  pipeline.workers: 1
- pipeline.id: logstash_cardax
  path.config: "/etc/logstash/conf.d/cardax.conf"
  pipeline.workers: 2

```

---

<div class="post-metadata">

**Author:** ![hunsw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hunsw/32/93637_2.png) [@hunsw](https://discuss.elastic.co/u/hunsw)\
**Post date:** [May 15, 2020, 8:45am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/4 "2020-05-15T08:45:23Z")

</div>

For this I suggest you use the collector pattern:

[https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#collector-pattern](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#collector-pattern)

I'm highly suspicious of Logstash not handling your two Elasticsearch output very well (i.e. not even using your second one).

---

<div class="post-metadata">

**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\
**Post date:** [May 15, 2020, 12:08pm UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/5 "2020-05-15T12:08:59Z")

</div>

if you run each config separately, does the index created correctly?

afaik, pipeline should be isolated from one another

---

<div class="post-metadata">

**Author:** ![Joseph-Gan-wk](https://avatars.discourse-cdn.com/v4/letter/j/ba9def/32.png) [@Joseph-Gan-wk](https://discuss.elastic.co/u/Joseph-Gan-wk)\
**Post date:** [May 16, 2020, 3:00am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/6 "2020-05-16T03:00:53Z")

</div>

@hunsw, Thanks for the suggestion 😊 😊 😊. That still new for me, need time to study that.

---

<div class="post-metadata">

**Author:** ![Joseph-Gan-wk](https://avatars.discourse-cdn.com/v4/letter/j/ba9def/32.png) [@Joseph-Gan-wk](https://discuss.elastic.co/u/Joseph-Gan-wk)\
**Post date:** [May 16, 2020, 3:01am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/7 "2020-05-16T03:01:44Z")

</div>

@ptamba, oh, thanks~ it works. 😊 😊 😊

---

<div class="post-metadata">

**Author:** ![Joseph-Gan-wk](https://avatars.discourse-cdn.com/v4/letter/j/ba9def/32.png) [@Joseph-Gan-wk](https://discuss.elastic.co/u/Joseph-Gan-wk)\
**Post date:** [May 16, 2020, 7:44am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/8 "2020-05-16T07:44:21Z")

</div>

@ptamba, this method works for the first time. But, i re-do with the same step, the same problem still there.

---

<div class="post-metadata">

**Author:** ![ptamba](https://avatars.discourse-cdn.com/v4/letter/p/7feea3/32.png) [@ptamba](https://discuss.elastic.co/u/ptamba)\
**Post date:** [May 16, 2020, 9:03am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/9 "2020-05-16T09:03:49Z")

</div>

it doesn’t actually solve the problem by running it individually, just making sure both config can run. it’s strange because pipeline is supposed to be isolated from one another.

one way to solve it is by putting each config in different directory then update pipelines.yml accordingly.

---

<div class="post-metadata">

**Author:** ![Joseph-Gan-wk](https://avatars.discourse-cdn.com/v4/letter/j/ba9def/32.png) [@Joseph-Gan-wk](https://discuss.elastic.co/u/Joseph-Gan-wk)\
**Post date:** [May 19, 2020, 2:52am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/10 "2020-05-19T02:52:27Z")

</div>

@ptamba okay, thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 16, 2020, 2:52am UTC](https://discuss.elastic.co/t/the-logstashpipeline-is-running-but-elasticsearch-didnt-create-an-index-for-data/232768/11 "2020-06-16T02:52:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.