The logstash'pipeline is running, but elasticsearch didnt create an index for data

Elastic Stack Logstash
1

Hi all, i am facing an issue is the elasticsearch did not create an index for data on Kibana.
I checked the Logstash-plain.log that shows two pipelines are running.

[2020-05-15T15:42:41,124][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:logstash_nagios, :logstash_cardax], :non_running_pipelines=>[]}

but, only the [logstash_nagios] can get the output on kibana from its index.
Cannot find the output for [Logstash_cardax], even its index.
both setting of output in config file is the same.

2

Why do you need two separate pipelines? You can set the index name in the Elasticsearch output of Logstash.

Also, it would help if you posted your config.

3

Hi,@hunsw
This is because the data source and structure of both data is differ, so i just try to make it separate.

This is config.file for [logstash_cardax]

input {
  file {
    path => ["/home/w/ctest.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
} #End of FILTER

output {
  elasticsearch {
    hosts => ["http://xx.xx.xx.xxx:9200/"]
    index => "cctest-log"
    user => "elastic"
    password => "xxxxxxxxxxxxxxxxx"

  }
}

This is config.file for [logstash_nagios]

input {
  file {
    path => ["/usr/local/nagios/var/nagios.log"]
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}
filter {
}

output {
  elasticsearch {
    hosts => ["http://xx.xx.xx.xxx:9200/"]
    index => "<nagios-{now/d}-000001>"
    user => "elastic"
    password => "xxxxxxxxxxxxxxxx"
  }
}

this is pipeline.yml

- pipeline.id: logstash_nagios
  path.config: "/etc/logstash/conf.d/nagios.conf"
  pipeline.workers: 1
- pipeline.id: logstash_cardax
  path.config: "/etc/logstash/conf.d/cardax.conf"
  pipeline.workers: 2
4

For this I suggest you use the collector pattern:

https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#collector-pattern

I'm highly suspicious of Logstash not handling your two Elasticsearch output very well (i.e. not even using your second one).

5

if you run each config separately, does the index created correctly?

afaik, pipeline should be isolated from one another

6

@hunsw, Thanks for the suggestion :blush: :blush: :blush:. That still new for me, need time to study that.

7

@ptamba, oh, thanks~ it works. :blush: :blush: :blush:

8

@ptamba, this method works for the first time. But, i re-do with the same step, the same problem still there.

9

it doesn’t actually solve the problem by running it individually, just making sure both config can run. it’s strange because pipeline is supposed to be isolated from one another.

one way to solve it is by putting each config in different directory then update pipelines.yml accordingly.

10

@ptamba okay, thanks.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.