Setting the pipeline in Fluentd

Elastic Stack Elasticsearch
1

Hello,
this is part of a long running discussion I've been having here with the most recent post being How to use pipelines and processors

I'm having a hard time finding where to add the pipeline in fluentd so that the data will flow through there at ingest.

Thanks

2

Hey @nicks1993,

We are definitely not experts on fluentd since it is not part of the Elastic stack.

However, looking at the docs: https://github.com/uken/fluent-plugin-elasticsearch#pipeline

It seems like adding pipeline <your pipeline> as a config setting should work.

The official docs seem outdated https://docs.fluentd.org/output/elasticsearch#parameters

3

Thanks I believe I figured it out. But now I'm getting a conflict on the duration field because I created a duration field manually earlier and now the pipeline is trying to create or store the wrong values in it. What is the best way to fix this? Can I remove fields?

4

Depends on what you mean by "wrong values".

created a duration field manually

If this was done inside the index mapping, then either update the mapping, or change the pipeline so that it creates a field that can be indexed according to that mapping.

If this was done upstream, I am not 100% sure what the issue is. The pipeline should be able to overwrite fields without any issues. But if you just want to drop the previous duration field: Remove processor | Elasticsearch Guide [8.11] | Elastic should work.

5

It seems I have 3 types in one field
Field Type Conflict: The type of this field changes across indices. It is unavailable for many analysis functions. The indices per type are as follows:
field types: text, integer, long

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.