# Log entry disappears between Logstash and Kibana **URL:** \ **Category:** Kibana\ **Created:** [March 31, 2020, 12:05pm UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850 "2020-03-31T12:05:27Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Axl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axl/32/45475_2.png) [@Axl](https://discuss.elastic.co/u/Axl)\ **Post date:** [March 31, 2020, 12:05pm UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850/1 "2020-03-31T12:05:27Z")
Hello, Logstash is processing an event with: - `eventid` set to `cowrie.command.input` - `message` and `input` containing Unix commands ```auto Mar 31 10:26:02 instance-42 logstash[22592]: { Mar 31 10:26:02 instance-42 logstash[22592]: "eventid" => "cowrie.command.input", ... Mar 31 10:26:02 instance-42 logstash[22592]: "message" => "CMD: cat /proc/mounts; /bin/busybox EOVZJ", ... Mar 31 10:26:02 instance-42 logstash[22592]: "input" => "cat /proc/mounts; /bin/busybox EOVZJ", ... Mar 31 10:26:02 instance-42 logstash[22592]: } ``` However, when I search in Kibana during that timeframe, - I don't see any event with `cowrie.command.input` - `input` field is obviously overwritten with other information (log type) `log` ![kibana](https://us1.discourse-cdn.com/elastic/original/3X/0/8/0883877ad7810a5d0133b680466db28f5e14a740.png) Actually, if I search for any time for a log with even `cowrie.command.input`, I don't find a single one. ![nocommand](https://us1.discourse-cdn.com/elastic/original/3X/7/7/77f4280ebeda3aad0f681348343f8549daba9588.png) Where are those events trashed? How can I debug this? I have no Kibana logs, and its journalctl is not showing anything particular. Thanks, FYI, I am using ELK 7.6.1, and Cowrie is a honeypot. ---
**Author:** ![Nathan\_Reese](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathan_reese/32/84829_2.png) [@Nathan\_Reese](https://discuss.elastic.co/u/Nathan_Reese)\ **Post date:** [March 31, 2020, 3:47pm UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850/2 "2020-03-31T15:47:44Z")
Kibana is just a UI for Elasticsearch so I suspect the disconnect is between logstash and elasticsearch. Do you have access to your logstash logs? Are there any dropped messages? Maybe add some debug outputs to logstash to help debug the problem. ---
**Author:** ![Axl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axl/32/45475_2.png) [@Axl](https://discuss.elastic.co/u/Axl)\ **Post date:** [April 2, 2020, 9:49am UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850/3 "2020-04-02T09:49:36Z")
Yes, I have access to logstash logs, but **at March 31 10:26, I have no logs**. The closest are **10:19:41** ```auto [2020-03-31T10:19:41,852][WARN][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:\ status=>400, :action=>["index", {:_id=>nil, :_index=>"cowrie-logstash", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"cowrie-logstash-2020.03.20-000001", "_type"=>"_doc", "_id\ "=>"f48aMHEBYpA6orC0mSmC", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping\ for [input] tried to parse field [input] as object, but found a concrete value"}}}} ``` After that log, I jump straight to **11:00**. ```auto [2020-03-31T11:00:34,038][WARN][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:\ status=>400, :action=>["index", {:_id=>nil, :_index=>"cowrie-logstash", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"cowrie-logstash-2020.03.20-000001", "_type"=>"_doc", "_id"\ =>"OI9AMHEBYpA6orC0BCpl", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping \ for [input] tried to parse field [input] as object, but found a concrete value"}}}} ``` As this is not the right time, I am not sure it is related, is it? As for logs via `journalctl`, I dumped them in my initial post, but here are if it helps to compare: ```auto Mar 31 10:26:02 instance-42 logstash[22592]: { Mar 31 10:26:02 instance-42 logstash[22592]: "eventid" => "cowrie.command.input", ... Mar 31 10:26:02 instance-42 logstash[22592]: "message" => "CMD: cat /proc/mounts; /bin/busybox EOVZJ", ... Mar 31 10:26:02 instance-42 logstash[22592]: "input" => "cat /proc/mounts; /bin/busybox EOVZJ", ... Mar 31 10:26:02 instance-42 logstash[22592]: } ``` ---
**Author:** ![Ian\_Boje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ian_boje/32/52033_2.png) [@Ian\_Boje](https://discuss.elastic.co/u/Ian_Boje)\ **Post date:** [April 2, 2020, 2:18pm UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850/4 "2020-04-02T14:18:17Z")
Do you have the dead letter queue option enabled in Logstash? Are you able to see your missing messages in the dead letter queue? ---
**Author:** ![Axl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axl/32/45475_2.png) [@Axl](https://discuss.elastic.co/u/Axl)\ **Post date:** [April 2, 2020, 3:14pm UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850/5 "2020-04-02T15:14:16Z")
No, it's not enabled: `logstash.yml:# dead_letter_queue.enable: false` You think I should try and enable it to see what happens to those kind of missing logs? ---
**Author:** ![Axl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axl/32/45475_2.png) [@Axl](https://discuss.elastic.co/u/Axl)\ **Post date:** [April 3, 2020, 10:08am UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850/6 "2020-04-03T10:08:57Z")
In ElasticSearch, I have this log which complains about `input` field. Do you think there is any relation? ```auto [2020-04-03T09:53:37,718][DEBUG][o.e.a.b.TransportShardBulkAction] [instance-42] [honeypot-logstash-2020.04.03-000001][0] failed to execute bulk item (index) index {[honeypot-logstash][_doc][mo91P3EBYpA6orC0z2C2], source[_na_]} org.elasticsearch.index.mapper.MapperParsingException: object mapping for [input] tried to parse field [input] as object, but found a concrete value ``` ---
**Author:** ![Axl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axl/32/45475_2.png) [@Axl](https://discuss.elastic.co/u/Axl)\ **Post date:** [April 3, 2020, 2:20pm UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850/7 "2020-04-03T14:20:09Z")
I solved this. It was a _Logstash_ config issue. In the logs of ElasticSearch (see below) I noticed the exception was occurring on field `input`, which was precisely one of the fields I was expecting to see in my "disappearing" logs. ```auto [2020-04-03T09:53:37,718][DEBUG][o.e.a.b.TransportShardBulkAction] [instance-42] [honeypot-logstash-2020.04.03-000001][0] failed to execute bulk item (index) index {[honeypot-logstash][_doc][mo91P3EBYpA6orC0z2C2], source[_na_]} org.elasticsearch.index.mapper.MapperParsingException: object mapping for [input] tried to parse field [input] as object, but found a concrete value ``` Then, in Logstash, I noticed this type of log, with the same error message: ```auto Mar 31 05:44:14 instance-42 logstash[22592]: [2020-03-31T03:44:14,883][WARN][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"cowrie-logstash", :routing=>nil, :_type=>"_doc"}, #], :response=>{"index"=>{"_index"=>"cowrie-logstash-2020.03.20-000001", "_type"=>"_doc", "_id"=>"Go-wLnEBYpA6orC0jiYT", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [input] tried to parse field [input] as object, but found a concrete value"}}}} ``` So, I had a look at my entries and found out that in some cases, I had `input` field with sublayers ```auto "input" => { "type" => "log" }, ``` and in other cases, directly a string. ```auto "command" => "echo \"this is my test\"" ``` I searched for the problem (in particular [this thread](https://discuss.elastic.co/t/problem-with-transfer-filebeat-6-1-3-logstash-6-1-3-elasticsearch-6-1-3/136264/7)), and found out that **this is a situation where Logstash is lost at types: in the first case, it expect an object, and it does not like in the second case to find a string**. **Solution** : I need to **either always to have an object, or always a string, but not mix.** The fix will depend on your log structure, in my case (`cowrie`), I was doing ```auto json { source => "message" } ``` which would read the JSON input from field `message` and create log structure from that. In my case, that was overwriting several fields such as `input`. The solution is to specify a target, so that fields from the JSON input don't overwrite other fields, and for instance, they'd go in `honeypot.input`, not `input`. ```auto json { source => "message" target => "honeypot" } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 1, 2020, 2:20pm UTC](https://discuss.elastic.co/t/log-entry-disappears-between-logstash-and-kibana/225850/8 "2020-05-01T14:20:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.