# \#ingest-pipeline **URL:** https://discuss.elastic.co/tag/ingest-pipeline/82.md [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) --- ## [Processing Heterogeneous IoT Logs with Fluent Bit and Elasticsearch Ingest Pipelines](https://discuss.elastic.co/t/processing-heterogeneous-iot-logs-with-fluent-bit-and-elasticsearch-ingest-pipelines/390311)
**Author:** [@cchaussat](https://discuss.elastic.co/u/cchaussat)\ **Replies:** 0\ **Last updated:** [September 10, 2026, 4:26pm UTC](https://discuss.elastic.co/t/processing-heterogeneous-iot-logs-with-fluent-bit-and-elasticsearch-ingest-pipelines/390311 "2026-09-10T16:26:18Z")
I would like to report on the experience of developing a simple home automation data collection and processing pipeline able to work with data and metrics from Domoticz and from many other various IoT devices and scripts… --- ## [Elastic Serverless Forwarder - Millions of API calls causing excess costs](https://discuss.elastic.co/t/elastic-serverless-forwarder-millions-of-api-calls-causing-excess-costs/388327)
**Author:** [@Watsong](https://discuss.elastic.co/u/Watsong)\ **Replies:** 2\ **Last updated:** [July 30, 2026, 10:17am UTC](https://discuss.elastic.co/t/elastic-serverless-forwarder-millions-of-api-calls-causing-excess-costs/388327 "2026-07-30T10:17:37Z")
The Elastic Serverless Forwarder AWS Lambda function accesses two credential values: ESF-CLOUD-ID, ESF-API-KEY The Lambda function does not cache the credentials. It retrieves them on every invocation of the Lambda func… --- ## [Ingest pipeline doesnt work](https://discuss.elastic.co/t/ingest-pipeline-doesnt-work/386407)
**Author:** [@Shahar\_Argov](https://discuss.elastic.co/u/Shahar_Argov)\ **Replies:** 4\ **Last updated:** [May 21, 2026, 2:12pm UTC](https://discuss.elastic.co/t/ingest-pipeline-doesnt-work/386407 "2026-05-21T14:12:36Z")
hi, I want to send logs through ingest pipeline to rename them to a different name, now the pipeline does look like its running but the names arent changing. if i try it with a random file from the index it said it wor… --- ## [Elasticsearch reindex with ELSER pipeline succeeds but only generates embeddings for fraction of documents - no failures reported](https://discuss.elastic.co/t/elasticsearch-reindex-with-elser-pipeline-succeeds-but-only-generates-embeddings-for-fraction-of-documents-no-failures-reported/386228)
**Author:** [@Roland-02](https://discuss.elastic.co/u/Roland-02)\ **Replies:** 2\ **Last updated:** [May 8, 2026, 2:02pm UTC](https://discuss.elastic.co/t/elasticsearch-reindex-with-elser-pipeline-succeeds-but-only-generates-embeddings-for-fraction-of-documents-no-failures-reported/386228 "2026-05-08T14:02:16Z")
I'm reindexing documents with an ELSER inference pipeline to generate embeddings, but only a fraction of documents (usually around half) end up with embeddings despite the reindex completing successfully with no failures… --- ## [CEF processor](https://discuss.elastic.co/t/cef-processor/386165)
**Author:** [@Moe\_Hmaidan](https://discuss.elastic.co/u/Moe_Hmaidan)\ **Replies:** 4\ **Last updated:** [May 5, 2026, 3:42pm UTC](https://discuss.elastic.co/t/cef-processor/386165 "2026-05-05T15:42:35Z")
Does the newly introduced cef processor require license? --- ## [After upgrade to 9.3.4 problem with system and windows integration](https://discuss.elastic.co/t/after-upgrade-to-9-3-4-problem-with-system-and-windows-integration/386156)
**Author:** [@Mario\_22](https://discuss.elastic.co/u/Mario_22)\ **Replies:** 2\ **Last updated:** [May 5, 2026, 5:49am UTC](https://discuss.elastic.co/t/after-upgrade-to-9-3-4-problem-with-system-and-windows-integration/386156 "2026-05-05T05:49:04Z")
Windows Event Logs stop indexing after upgrade to Elastic Agent 9.3.4 — system.application, system.system, Sysmon, PowerShell, and Custom Windows logs dropped with HTTP 400 Malformed content / dataset mapping conflict; p… --- ## [Downsampling only works one time](https://discuss.elastic.co/t/downsampling-only-works-one-time/375708)
**Author:** [@bstinchcomb](https://discuss.elastic.co/u/bstinchcomb)\ **Replies:** 1\ **Last updated:** [March 13, 2026, 2:43pm UTC](https://discuss.elastic.co/t/downsampling-only-works-one-time/375708 "2026-03-13T14:43:22Z")
Hello, I am running into an issue with elasticsearch where downsampling will only run a single time, and then subsequent runs seems to just create an empty index. I have tried several different docker versions and still … --- ## [How to hash sensitive fields in Elasticsearch and show hash vs original based on user permissions?](https://discuss.elastic.co/t/how-to-hash-sensitive-fields-in-elasticsearch-and-show-hash-vs-original-based-on-user-permissions/384905)
**Author:** [@Abhi\_Abhishek](https://discuss.elastic.co/u/Abhi_Abhishek)\ **Replies:** 2\ **Last updated:** [February 4, 2026, 1:09pm UTC](https://discuss.elastic.co/t/how-to-hash-sensitive-fields-in-elasticsearch-and-show-hash-vs-original-based-on-user-permissions/384905 "2026-02-04T13:09:04Z")
We have some fields containing sensitive information, and we do not want all users to see the original values. We want to store these fields in a hashed form while indexing and control what different users can see. I h… --- ## [API key does or does not rely on permissions from user that created it](https://discuss.elastic.co/t/api-key-does-or-does-not-rely-on-permissions-from-user-that-created-it/383663)
**Author:** [@qd-danh](https://discuss.elastic.co/u/qd-danh)\ **Replies:** 14\ **Last updated:** [December 31, 2025, 10:08pm UTC](https://discuss.elastic.co/t/api-key-does-or-does-not-rely-on-permissions-from-user-that-created-it/383663 "2025-12-31T22:08:43Z")
We had previously been creating API keys with our SSO user accounts. Then we found that after an SSO IdP provider change our users were effectively “different” such that we could no longer edit API keys (e.g. to add or r… --- ## [Preventing date\_time\_parse\_exception in ingest pipeline](https://discuss.elastic.co/t/preventing-date-time-parse-exception-in-ingest-pipeline/382707)
**Author:** [@marekott](https://discuss.elastic.co/u/marekott)\ **Replies:** 6\ **Last updated:** [November 11, 2025, 9:05pm UTC](https://discuss.elastic.co/t/preventing-date-time-parse-exception-in-ingest-pipeline/382707 "2025-11-11T21:05:00Z")
Hi, I have encountered date parsing related problem in my ingest pipeline. Full details can be found here: \[Ingest pipeline\] Occasionally ends with \`date\_time\_parse\_exception\` when pipeline is trying to store time withou… --- ## [Logs ingestion](https://discuss.elastic.co/t/logs-ingestion/382715)
**Author:** [@eirc](https://discuss.elastic.co/u/eirc)\ **Replies:** 1\ **Last updated:** [October 15, 2025, 6:55am UTC](https://discuss.elastic.co/t/logs-ingestion/382715 "2025-10-15T06:55:51Z")
I’m trying to setup logs ingestion for multiple systems. I have installed filebeat on all hosts and pushed all system logs with journald and container logs with a filestream for \`/var/lib/docker/containers/\*/\*.log\`. Ever… --- ## [Elasticsearch enrich policy not reflecting updated source after \_execute (v9.1.3)](https://discuss.elastic.co/t/elasticsearch-enrich-policy-not-reflecting-updated-source-after-execute-v9-1-3/382141)
**Author:** [@Daniel\_Santos1](https://discuss.elastic.co/u/Daniel_Santos1)\ **Replies:** 2\ **Last updated:** [September 23, 2025, 12:01am UTC](https://discuss.elastic.co/t/elasticsearch-enrich-policy-not-reflecting-updated-source-after-execute-v9-1-3/382141 "2025-09-23T00:01:25Z")
Hi, I’m running into an issue with enrich policies in Elasticsearch v9.1.3. When I update the source index used by an enrich policy and then re-execute the policy, the enrich simulation still returns stale data. The upd… --- ## [Ingest pipeline pattern matching - much help needed](https://discuss.elastic.co/t/ingest-pipeline-pattern-matching-much-help-needed/379207)
**Author:** [@SteveParker](https://discuss.elastic.co/u/SteveParker)\ **Replies:** 12\ **Last updated:** [June 24, 2025, 1:13pm UTC](https://discuss.elastic.co/t/ingest-pipeline-pattern-matching-much-help-needed/379207 "2025-06-24T13:13:43Z")
Hi all I am using the following ES|QL to pattern match a substring in a field from a filebeat index - FROM filebeat-\* | WHERE url.original LIKE "q=" It would make a lot of sense to drop any incoming documents that do… --- ## [Ingest Processor for array object data using script](https://discuss.elastic.co/t/ingest-processor-for-array-object-data-using-script/379375)
**Author:** [@adude946](https://discuss.elastic.co/u/adude946)\ **Replies:** 2\ **Last updated:** [June 20, 2025, 7:14pm UTC](https://discuss.elastic.co/t/ingest-processor-for-array-object-data-using-script/379375 "2025-06-20T19:14:23Z")
Need assist to get the Ingest pipeline processors to work correctly for my sample data. Its an array of objects, trying to create individual objects from each array object using a value data + prefix names based on these… --- ## [Ingest pipeline is not working for given document](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099)
**Author:** [@kuldeep\_gupta](https://discuss.elastic.co/u/kuldeep_gupta)\ **Replies:** 6\ **Last updated:** [June 11, 2025, 5:02pm UTC](https://discuss.elastic.co/t/ingest-pipeline-is-not-working-for-given-document/379099 "2025-06-11T17:02:45Z")
i have below record which i want to process. \[ { "\_id": "GFk-X5cBY6REVzo7i86y", "\_index": "processor\_test", "\_source": { "event.original":"172.16.102.98 - - \[11/Jun/2025:19:05:43 +0530\] \\"POST /api/… --- ## [How to obtain mappings and ingest pipelines from Elastic integrations without using Fleet?](https://discuss.elastic.co/t/how-to-obtain-mappings-and-ingest-pipelines-from-elastic-integrations-without-using-fleet/378408)
**Author:** [@vasek](https://discuss.elastic.co/u/vasek)\ **Replies:** 1\ **Last updated:** [May 29, 2025, 12:00pm UTC](https://discuss.elastic.co/t/how-to-obtain-mappings-and-ingest-pipelines-from-elastic-integrations-without-using-fleet/378408 "2025-05-29T12:00:01Z")
We’re working on ingesting logs from network devices (e.g., Cisco IOS) that send their logs via Syslog directly to Logstash, which then forwards the data to Elasticsearch. We manage all components through custom automati… --- ## [Transform script fails to index into destination data stream](https://discuss.elastic.co/t/transform-script-fails-to-index-into-destination-data-stream/378012)
**Author:** [@stefws](https://discuss.elastic.co/u/stefws)\ **Replies:** 2\ **Last updated:** [May 11, 2025, 11:52am UTC](https://discuss.elastic.co/t/transform-script-fails-to-index-into-destination-data-stream/378012 "2025-05-11T11:52:02Z")
I'm on v.8.18 and trying to make my first transform script which should correlate a start and end event from a source index and then calculate the process time in ms as the time difference between start and end events an… --- ## [Enrich Processor Not Working with Ingest Pipeline + Fleet Data Streams](https://discuss.elastic.co/t/enrich-processor-not-working-with-ingest-pipeline-fleet-data-streams/377809)
**Author:** [@mehrad\_ghalibafi](https://discuss.elastic.co/u/mehrad_ghalibafi)\ **Replies:** 2\ **Last updated:** [May 6, 2025, 12:31pm UTC](https://discuss.elastic.co/t/enrich-processor-not-working-with-ingest-pipeline-fleet-data-streams/377809 "2025-05-06T12:31:12Z")
Hi everyone, I'm building a small CTI platform where threat intelligence feeds (containing file hashes, IPs, etc.) are indexed into a custom index called tip\_index. I want to correlate fields like threat.indicator.file.… --- ## [Nested fields Issues - Remove / rename](https://discuss.elastic.co/t/nested-fields-issues-remove-rename/377447)
**Author:** [@sai\_ravi\_shankar](https://discuss.elastic.co/u/sai_ravi_shankar)\ **Replies:** 3\ **Last updated:** [April 24, 2025, 1:09pm UTC](https://discuss.elastic.co/t/nested-fields-issues-remove-rename/377447 "2025-04-24T13:09:55Z")
Hi, Can someone help me on the below issue. I have a nested object where i am unable to rename or remove such fields. Below is the format of such field. test.test1.test2.test3.test4.test5 It has 4 objects and 1 field… --- ## [In version 8.17.3, some default pipelines are automatically created after being deleted using DELETE \_ingest/pipeline/\*](https://discuss.elastic.co/t/in-version-8-17-3-some-default-pipelines-are-automatically-created-after-being-deleted-using-delete-ingest-pipeline/376945)
**Author:** [@AdolphGai](https://discuss.elastic.co/u/AdolphGai)\ **Replies:** 5\ **Last updated:** [April 11, 2025, 6:28pm UTC](https://discuss.elastic.co/t/in-version-8-17-3-some-default-pipelines-are-automatically-created-after-being-deleted-using-delete-ingest-pipeline/376945 "2025-04-11T18:28:25Z")
Install elasticsearch 8.17.3 and find that there are many default pipelines Run the DELETE \_ingest/pipeline/\* and \_ingest/pipeline/pipeline\_id command. The command output is deleted successfully But with get, the discov… --- ## [Seperate the elements into multiple documents using ingest pipeline](https://discuss.elastic.co/t/seperate-the-elements-into-multiple-documents-using-ingest-pipeline/376752)
**Author:** [@sai7276p](https://discuss.elastic.co/u/sai7276p)\ **Replies:** 3\ **Last updated:** [April 3, 2025, 8:00pm UTC](https://discuss.elastic.co/t/seperate-the-elements-into-multiple-documents-using-ingest-pipeline/376752 "2025-04-03T20:00:32Z")
Hi I want to create multiple documents/events based on the target field. Target field contains multiple elements and want to seperate it and create multiple documents using ingest pipeline. I tried with foreach and scri… --- ## [Save Index in specific Directory](https://discuss.elastic.co/t/save-index-in-specific-directory/375384)
**Author:** [@NikoCosmico01](https://discuss.elastic.co/u/NikoCosmico01)\ **Replies:** 7\ **Last updated:** [March 11, 2025, 8:50am UTC](https://discuss.elastic.co/t/save-index-in-specific-directory/375384 "2025-03-11T08:50:17Z")
Hi, I've the ELK stack installed via docker having in particular 3 ES nodes and 1 LogStash node. My question is if there is a way to have the data ingested from a specific LogStash pipeline saved in a specific director… --- ## [DWG - ElasticSearch - Tika](https://discuss.elastic.co/t/dwg-elasticsearch-tika/375383)
**Author:** [@uniconfortced](https://discuss.elastic.co/u/uniconfortced)\ **Replies:** 5\ **Last updated:** [March 6, 2025, 9:34pm UTC](https://discuss.elastic.co/t/dwg-elasticsearch-tika/375383 "2025-03-06T21:34:25Z")
I have a simple question. I read that Tika can extract information from CAD file as DWG format. There is a mode to improve this in Elastic? I put into modules\\ingest-attachment folder tika-parser-cad-module-2.9.2.jar b… --- ## [Error Ingesting AWS Security Hub Data](https://discuss.elastic.co/t/error-ingesting-aws-security-hub-data/375013)
**Author:** [@Abhay\_Singh](https://discuss.elastic.co/u/Abhay_Singh)\ **Replies:** 4\ **Last updated:** [March 5, 2025, 5:35pm UTC](https://discuss.elastic.co/t/error-ingesting-aws-security-hub-data/375013 "2025-03-05T17:35:53Z")
Hello Team, I am trying to ingest data from AWS Security Hub into elastic using elastic agent and the integration available, however i am getting the below error. Could someone guide me how i can resolve this issue. Er… --- ## [PDF content wrong sequence and space](https://discuss.elastic.co/t/pdf-content-wrong-sequence-and-space/375001)
**Author:** [@uniconfortced](https://discuss.elastic.co/u/uniconfortced)\ **Replies:** 2\ **Last updated:** [March 4, 2025, 3:32pm UTC](https://discuss.elastic.co/t/pdf-content-wrong-sequence-and-space/375001 "2025-03-04T15:32:48Z")
Good morning, I am new into Elasticsearch use. I install it to search information from content of PDF generated by my old ERP. Elastic extract correct text but into wrong sequence that depend from PDF read sequence or t… --- ## [How to alert if two fields match](https://discuss.elastic.co/t/how-to-alert-if-two-fields-match/374143)
**Author:** [@SecurePete](https://discuss.elastic.co/u/SecurePete)\ **Replies:** 9\ **Last updated:** [February 6, 2025, 9:14pm UTC](https://discuss.elastic.co/t/how-to-alert-if-two-fields-match/374143 "2025-02-06T21:14:20Z")
I am very new to messing with Elastic pipelines and I need help. I want to alert based on whether two fields in a log match. I am not sure the correct way to do this. The logs are from a Cisco DUO integration. I want al… --- ## [Pipelines execution in Elastic.Serilog.Sinks](https://discuss.elastic.co/t/pipelines-execution-in-elastic-serilog-sinks/374056)
**Author:** [@Wojciech\_Szabowicz](https://discuss.elastic.co/u/Wojciech_Szabowicz)\ **Replies:** 0\ **Last updated:** [February 4, 2025, 10:15am UTC](https://discuss.elastic.co/t/pipelines-execution-in-elastic-serilog-sinks/374056 "2025-02-04T10:15:33Z")
Hi, I am using Elastic.Serilog.Sinks to stream some data directly to elasticsearch a very simple mechanism .WriteTo.Logger(lc =\> { lc.Filter.ByIncludingOnly(le =\> le.Properties.ContainsKey("ecs.ver… --- ## [New index not created by ingestion pipeline](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018)
**Author:** [@umesh2020](https://discuss.elastic.co/u/umesh2020)\ **Replies:** 19\ **Last updated:** [February 3, 2025, 10:24pm UTC](https://discuss.elastic.co/t/new-index-not-created-by-ingestion-pipeline/374018 "2025-02-03T22:24:06Z")
Hi I am using elastic-agent in k8s cluster with kubernetes integration. I have added a custom pipeline for kubernetes container logs to re-route all logs from containers in a specific namespace. The following is the … --- ## [Using reindex to generate embeddings from nested field](https://discuss.elastic.co/t/using-reindex-to-generate-embeddings-from-nested-field/373957)
**Author:** [@Carlos\_Fernando\_Palm](https://discuss.elastic.co/u/Carlos_Fernando_Palm)\ **Replies:** 1\ **Last updated:** [February 3, 2025, 10:03am UTC](https://discuss.elastic.co/t/using-reindex-to-generate-embeddings-from-nested-field/373957 "2025-02-03T10:03:41Z")
Hello! I was reading this: And I tried to apply the idea of using a reindex command together with an ingest pipeline to generate the embeddings of data already inside an elastic index. Now I defined the ingest pipeli… --- ## [Painless script outputs literal "key" field](https://discuss.elastic.co/t/painless-script-outputs-literal-key-field/373800)
**Author:** [@Henning\_Oden](https://discuss.elastic.co/u/Henning_Oden)\ **Replies:** 0\ **Last updated:** [January 29, 2025, 8:03am UTC](https://discuss.elastic.co/t/painless-script-outputs-literal-key-field/373800 "2025-01-29T08:03:48Z")
I have a Painless script I run in a Script processor in an Ingest Pipeline which moves fields from within a temporary jsonPayload field to the document root. The source looks like this (field names redacted for safety): … [Next page](https://discuss.elastic.co/tag/ingest-pipeline/82.md?match_all_tags=true&page=1&tags%5B%5D=ingest-pipeline)