# Regarding Ingest Node Pipeline **URL:** \ **Category:** Kibana\ **Created:** [December 13, 2022, 7:17am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094 "2022-12-13T07:17:59Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 13, 2022, 7:17am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/1 "2022-12-13T07:17:59Z")
Hi team, We would like to know about the ingest node pipeline. we are new in this platform. Normally, an application team sending the logs through the below mentioned path. Logs Flow Path: FileBeat/MetricBeat -\> AWS Kafka -\> Logstash -\> ES -\> Kibana But, here one of the application team would like to directly send the logs through Elastic Cluster by using ingest node pipeline and not using the above log flow path. Could someone explain us the below queries. [1] what is ingest node pipeline and what is the purpose of that ? [2] Difference between Logstash pipeline and Ingest Pipeline? [3] what is the key factor for Ingest Pipeline? [4] What would be impact for moving from Logstash pipeline to Ingest Pipeline.? [5] How to give the Bulk permission for API key? Please let us know the queries that would be very helpful to moving forward. Thanks, Yasar Arafaath A. ---
**Author:** ![Venkata\_Raja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkata_raja/32/114264_2.png) [@Venkata\_Raja](https://discuss.elastic.co/u/Venkata_Raja)\ **Post date:** [December 13, 2022, 7:29am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/2 "2022-12-13T07:29:05Z")
Hi, Ingest pipelines are used to transform data as per custom requirements. Ingest pipelines & Logstash pipelines are used for same purpose , the difference is Logstash pipelines are transforming data in logstash ans sending it to Elasticsearch . But Ingest pipelines will sit in Elasticsearch nodes & transform data before data gets stored in Elasticsearch. You can either append pipeline directly by adding it in Index settings or use Index Templates . ---
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 13, 2022, 7:31am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/3 "2022-12-13T07:31:42Z")
Hi Venkata\_Raja, Thanks for your reply, Does any impact, when application team moving into Ingest Node Pipeline and how could we configured the Ingest Node Pipeline ? Regards, Yasar Arafaath ---
**Author:** ![Venkata\_Raja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkata_raja/32/114264_2.png) [@Venkata\_Raja](https://discuss.elastic.co/u/Venkata_Raja)\ **Post date:** [December 13, 2022, 7:35am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/4 "2022-12-13T07:35:06Z")
There will be no impact , just make sure that your Elasticsearch nodes have enough Memory to process data using ingest pipelines. Refer below for an example. > **[Example: Parse logs in the Common Log Format | Elasticsearch Guide \[8.5\] |...](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/common-log-format-example.html)** ---
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 13, 2022, 7:37am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/5 "2022-12-13T07:37:31Z")
Hi Raja, Thanks for your quick response. If possible could you please share the Ingest Node pipeline documentation Link that also would be helpful to presenting with my clients please. Reagrds, Yasar Arafaath A. ---
**Author:** ![Venkata\_Raja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkata_raja/32/114264_2.png) [@Venkata\_Raja](https://discuss.elastic.co/u/Venkata_Raja)\ **Post date:** [December 13, 2022, 7:38am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/6 "2022-12-13T07:38:23Z")
Hi Yasar, Here yo go > **[Ingest pipelines | Elasticsearch Guide \[8.5\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/ingest.html)** ---
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 13, 2022, 7:52am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/7 "2022-12-13T07:52:59Z")
Hi Raja, Thanks for your quick response. We will go through the documentation. Regards, Yasar Arafaath A. ---
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 13, 2022, 12:36pm UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/8 "2022-12-13T12:36:24Z")
Hi Raja, We read through the documents, it's fair enough for us. Could you please tell us one more query, as i was asked earlier. [5] We need an API key id and key with permissions to call the Bulk API. We know how to create the API's and could you please guide us the same, how to give the permission to call the Bulk API. Regards, Yasar Arafaath A. ---
**Author:** ![Venkata\_Raja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkata_raja/32/114264_2.png) [@Venkata\_Raja](https://discuss.elastic.co/u/Venkata_Raja)\ **Post date:** [December 14, 2022, 6:53am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/9 "2022-12-14T06:53:27Z")
Hi , Refer below on how to create API Keys , > **[Create API key API | Elasticsearch Guide \[8.5\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.5/security-api-create-api-key.html)** You need to have below prvillages on indices where you want to perform bulk api. Just add these privileges to your api key. > "privileges":["read","create\_doc","index","write","auto\_configure","create\_index","manage"] ---
**Author:** ![yasar](https://avatars.discourse-cdn.com/v4/letter/y/dc4da7/32.png) [@yasar](https://discuss.elastic.co/u/yasar)\ **Post date:** [December 14, 2022, 7:26am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/10 "2022-12-14T07:26:52Z")
Hi Raja, Thanks for the response. Regards, Yasar Arafaath ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 11, 2023, 7:27am UTC](https://discuss.elastic.co/t/regarding-ingest-node-pipeline/321094/11 "2023-01-11T07:27:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.