this is part of a long running discussion I've been having here with the most recent post being How to use pipelines and processors
I'm having a hard time finding where to add the pipeline in fluentd so that the data will flow through there at ingest.
We are definitely not experts on fluentd since it is not part of the Elastic stack.
However, looking at the docs: https://github.com/uken/fluent-plugin-elasticsearch#pipeline
It seems like adding
pipeline <your pipeline> as a config setting should work.
The official docs seem outdated https://docs.fluentd.org/output/elasticsearch#parameters
Thanks I believe I figured it out. But now I'm getting a conflict on the duration field because I created a duration field manually earlier and now the pipeline is trying to create or store the wrong values in it. What is the best way to fix this? Can I remove fields?
Depends on what you mean by "wrong values".
created a duration field manually
If this was done inside the index mapping, then either update the mapping, or change the pipeline so that it creates a field that can be indexed according to that mapping.
If this was done upstream, I am not 100% sure what the issue is. The pipeline should be able to overwrite fields without any issues. But if you just want to drop the previous duration field: https://www.elastic.co/guide/en/elasticsearch/reference/current/remove-processor.html should work.
It seems I have 3 types in one field
Field Type Conflict: The type of this field changes across indices. It is unavailable for many analysis functions. The indices per type are as follows:
field types: text, integer, long
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.