# Ingest error with logstash

**URL:** <https://discuss.elastic.co/t/ingest-error-with-logstash/349195>\
**Category:** Elasticsearch\
**Tags:** painless\
**Created:** [December 12, 2023, 9:24pm UTC](https://discuss.elastic.co/t/ingest-error-with-logstash/349195 "2023-12-12T21:24:50Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![hollo](https://avatars.discourse-cdn.com/v4/letter/h/a9adbd/32.png) [@hollo](https://discuss.elastic.co/u/hollo)\
**Post date:** [December 12, 2023, 9:24pm UTC](https://discuss.elastic.co/t/ingest-error-with-logstash/349195/1 "2023-12-12T21:24:50Z")

</div>

Hi.  
I've created a filebeat -\> logstash -\> elastic flow for iptables logs.  
Filebeat uses the default iptables module.  
Logstash has minimal config (beat input, elastic output, no filter).  
The Logstash pipeline comes from the documentation.  
Default Elastic ingests are installed.

When a message is sent via the above flow, I've got a dlq error message:

```auto
Could not index event to Elasticsearch. status: 400, action: ["create", {:_id=>nil, :_index=>"filebeat-8.10.3", :routing=>nil, :pipeline=>"filebeat-8.10.3-iptables-log-pipeline"}, {"fileset"=>{"name"=>"log"}, "log"=>{"file"=>{"path"=>"/var/log/firewall"
...
 response: {"create"=>{"_index"=>".ds-filebeat-8.10.3-2023.12.12-000025", "_id"=>"hYy5X4wBj3IF6fYTGKKe", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:65] failed to parse field [iptables.ether_type] of type [long] in document with id 'hYy5X4wBj3IF6fYTGKKe'. Preview of field's value: '08:00'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"For input string: \"08:00\""}}}}

```

The 08:00 is the hexa value of ether\_type (comes from iptables log) and filebeat-8.10.3-iptables-log-pipeline's painless script should convert it to long value (2048).

Anyway the conversion works as expected if I test the message on Kibana Ingest Test or modify the flow to direct sending (filebeat -\> elastic).

I guess that there is some difference in the behavior of ingest processing based on beat or logstash source, but I couldn't find any clue. Any idea?  
Thanks.

---

<div class="post-metadata">

**Author:** ![hollo](https://avatars.discourse-cdn.com/v4/letter/h/a9adbd/32.png) [@hollo](https://discuss.elastic.co/u/hollo)\
**Post date:** [December 14, 2023, 11:20am UTC](https://discuss.elastic.co/t/ingest-error-with-logstash/349195/2 "2023-12-14T11:20:25Z")

</div>

I think I found the problem.  
Logstash adds an `event.orignal` field to the data before send it to the ingest and iptables-ingest 3rd step is a Rename task (Renames "message" to "event.original") which throws an "already exits" error and stops the process.

When I set the "Ignore failures for this processor" option the processing worked as expected (maybe a condition should be more elegant).

---

<div class="post-metadata">

**Author:** ![hollo](https://avatars.discourse-cdn.com/v4/letter/h/a9adbd/32.png) [@hollo](https://discuss.elastic.co/u/hollo)\
**Post date:** [December 14, 2023, 11:39am UTC](https://discuss.elastic.co/t/ingest-error-with-logstash/349195/3 "2023-12-14T11:39:19Z")

</div>

Anyway the "ignore" and condition (`ctx?.event?.original == null`) are not the best, because both fields remain.  
A failure handler (remove field) could be a solution, but I think it's ugly....

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 11, 2024, 11:40am UTC](https://discuss.elastic.co/t/ingest-error-with-logstash/349195/4 "2024-01-11T11:40:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.