Ingest_pipeline - Elastic Cloud

[evgeniy777 (Evgeniy Sky Photographer)]

Hi all !

i'm using Elastic Cloud solution and FileBeat on my servers.
made a simple ingest_pipeline for parsing my custom app logs.
On Filebeat config i'm defining "cloud.auth:" and "cloud.id:", but i don't know how to set the pipeline for my data ?

cloud.auth: bla bla
cloud.id: blabla
output.elasticsearch:
  pipeline: "pipeline_name" 

this not working - i got data, but it's not going throught pipeline to be parsed.
Maybe i have to define a pipeline in some other way ?

[dadoonet (David Pilato)]

Welcome!

I think this is the correct way.

Unless there's a recent issue, I was doing the same thing in the past: bano-elastic/filebeat-all.yml at master · dadoonet/bano-elastic · GitHub

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /bano-data/bano-*.csv

output.elasticsearch:
  indices:
    - index: "bano-foo"
  pipeline: bano

And: bano-elastic/filebeat.sh at master · dadoonet/bano-elastic · GitHub

	docker run \
	  --name=filebeat \
	  --user=root \
	  --volume="$(pwd)/bano-data:/bano-data:ro" \
	  --volume="$(pwd)/filebeat-config/filebeat$FILEBEAT_CONFIG_SUFFIX.yml:/usr/share/filebeat/filebeat.yml" \
	  -p 8000:8000 \
	  docker.elastic.co/beats/filebeat:$ELASTIC_VERSION filebeat -e -strict.perms=false -d "$FILEBEAT_SELECTORS" -E cloud.id="$CLOUD_ID" -E cloud.auth="elastic:$ELASTIC_PASSWORD"

Which is quite similar...

I moved your question to #elastic-stack:beats.

[stephenb (Stephen Brown)]

Hi @evgeniy777 welcome to the community

Couple things ...

1. Have you tested the pipeline outside of filebeat in Kibana / Dev Tools. Setup the pipeline then try to index a document in Dev Tools to validate the pipeline is actually working?

2. Are you using a module? or specific Input if so sometimes there are default pipelines that overrides yours unless you config correctly.

Let us know these 2 things and perhaps we can help more.

[evgeniy777 (Evgeniy Sky Photographer)]

1 - yes it's working.

Actually i don't define a pipeline name in "filebeat.yml", i updated an index.template in Kibana with "index.default_pipeline" :

default_pipeline": "pipeline_1",

So when new index is created and match the template - it's start automatically going throught " index.default_pipeline ". Now it's working fine.

Thanks for helping guys !