Elasticsearch Input Plugin for Logstash 6.0.0 - Issue indexing data

Elastic Stack Logstash
1

This issue regards ES Input Plugin for Logstash.

I am using Windows 10 and the new Logstash 6.0.0 version.
I'm trying to run 2 pipelines -via the new pipeline.yml configuration- where the first one grabs the data from a database to index to ES, and the other takes the data indexed in ES and modifies it and re-indexes it to another ES index.

At first, since there's no data, the second pipeline returns an error 404, which is self explanatory since the data from the database haven't been imported to ES yet, but after the data goes into ES, it stops giving a 404, but at the same time doesn't do anything with the data.
Once I restart logstash with the same configuration, AND with the data already existing in ES, it grabs it and modifies it and does what I want.

This seems like a faulty behavior to me, and I'd like to know some explanation for why is this happening, and if there is some sort of a way to make pipelines depend on each other or delay pipelines for a short period of time.

2

Hi,
Can you share your pipeline.yml.

3

Hello, the pipeline has only those following lines, the rest are comments:

- pipeline.id: source
  path.config: "/C:/fakepath/Logstash_Configuration/Source"
- pipeline.id: target
  path.config: "/C:/fakepath/Logstash_Configuration/Target"

As I mentioned, once I restart the Logstash session, after the data exist in ES, it works and it replicates the data with the modifications that I need.

4

The 404 error would stop happening if the index already existed, even without data, but the pipeline wont modify the data unless it was already in the index before starting the pipeline.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.