Regarding Ingest Node Pipeline

Elastic Stack Kibana
1

Hi team,

We would like to know about the ingest node pipeline. we are new in this platform.

Normally, an application team sending the logs through the below mentioned path.

Logs Flow Path: FileBeat/MetricBeat -> AWS Kafka -> Logstash -> ES -> Kibana

But, here one of the application team would like to directly send the logs through Elastic Cluster by using ingest node pipeline and not using the above log flow path.

Could someone explain us the below queries.

[1] what is ingest node pipeline and what is the purpose of that ?
[2] Difference between Logstash pipeline and Ingest Pipeline?
[3] what is the key factor for Ingest Pipeline?
[4] What would be impact for moving from Logstash pipeline to Ingest Pipeline.?
[5] How to give the Bulk permission for API key?

Please let us know the queries that would be very helpful to moving forward.

Thanks,
Yasar Arafaath A.

2

Hi,

Ingest pipelines are used to transform data as per custom requirements. Ingest pipelines & Logstash pipelines are used for same purpose , the difference is Logstash pipelines are transforming data in logstash ans sending it to Elasticsearch . But Ingest pipelines will sit in Elasticsearch nodes & transform data before data gets stored in Elasticsearch.
You can either append pipeline directly by adding it in Index settings or use Index Templates .

3

Hi Venkata_Raja,

Thanks for your reply,

Does any impact, when application team moving into Ingest Node Pipeline and how could we configured the Ingest Node Pipeline ?

Regards,
Yasar Arafaath

4

There will be no impact , just make sure that your Elasticsearch nodes have enough Memory to process data using ingest pipelines. Refer below for an example.

5

Hi Raja,

Thanks for your quick response.

If possible could you please share the Ingest Node pipeline documentation Link that also would be helpful to presenting with my clients please.

Reagrds,
Yasar Arafaath A.

6

Hi Yasar,

Here yo go

7

Hi Raja,

Thanks for your quick response.

We will go through the documentation.

Regards,
Yasar Arafaath A.

8

Hi Raja,

We read through the documents, it's fair enough for us.

Could you please tell us one more query, as i was asked earlier.

[5] We need an API key id and key with permissions to call the Bulk API.
We know how to create the API's and could you please guide us the same, how to give the permission to call the Bulk API.

Regards,
Yasar Arafaath A.

9

Hi ,

Refer below on how to create API Keys ,

You need to have below prvillages on indices where you want to perform bulk api. Just add these privileges to your api key.

"privileges":["read","create_doc","index","write","auto_configure","create_index","manage"]

10

Hi Raja,

Thanks for the response.

Regards,
Yasar Arafaath

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.