Ingest Pipeline for Transformations

Elastic Stack Elasticsearch
1

I am very new to elastic search, I have downloaded Elastic search on MacOs and have also installed Kibana for visualization. I am working with a connector which allows me to provide a pipeline name. However I dont know how to create an Ingest Pipeline in Elastic search.

For instance if my incoming data has a field called Name and I want to make it all UpperCase
Where do I create a pipeline and its processors. I read this documentation but couldnot follow as to where should I be creating a pipeline, i have access to ES on port 9200, is there a file that I need to create? Can someone point me to an example on creating a pipeline, processor and consuming it.
Also are there any pipelines with processor already available ?

2

You can use the put pipeline api

Many of the filebeat modules load ingest pipelines with their "setup" phase. On any system where you have installed filebeat, do a "locate pipeline.yml" (that command is RH based, may be different on your OS) and you will see sample pipelines in many /filebeat/module/ directories.

3

Welcome!

Open Kibana and go to the Dev Console.
There you can copy paste the examples available in the documentation.

Click on play/run.

5

I installed filebeat and I do see pipeline.yml, thats definitely helpful. This gives me ideas on how to write a pipeline that has processors. Now say I create my own pipeline.yml file, where should I deploy this file within elastic search such that ES respects it and uses it as a preprocessor before creating an Index ? Is there a specific folder structure inside ES where I can deploy this file that I have created ?

6

That's the "put pipeline api" above. Either command line (curl) or Kibana tools (easier).

Pipelines (and most everything else) are stored in the Elasticsearch database, so you only have to update once and it's available across the cluster.

7

ah ok thats it :slight_smile: thanks much let me try one and will let you know

8

@rugenl that worked like a charm thanks much.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.