# Some logs not being ingested from LS to ES **URL:** \ **Category:** Logstash\ **Created:** [September 12, 2017, 1:38am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147 "2017-09-12T01:38:51Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://discuss.elastic.co/u/theflakes)\ **Post date:** [September 12, 2017, 1:38am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/1 "2017-09-12T01:38:51Z")
Some logs from the same LS instance are not being imported into ES. I see the following notification in ES when sending just the problem logs to ES; "the indices which match this index pattern don't contain any time fields". LS appears to be parsing the logs just fine. All logs are single line JSON formatted. It looks like the "@timestamp" filter is working for all the logs except the problem ones even though when I look at the LS debug output all looks good for all logs. **My date filter:** > date { > match =\> ["timestamp", "yyyy-MM-dd HH:mm:ss"] > timezone =\> "UTC" > target =\> "@timestamp" > remove\_field =\> ["timestamp"] > } **The "timestamp" field before LS:** > "timestamp": "2017-09-11 21:52:45" **The "@timestamp" field after the date filter is applied:** > "@timestamp"=\>2017-09-11T21:52:45.000Z Not sure what I'm screwing up or if this really is the source of the problem. **Raw JSON log:** `{"tcp_flags": 16, "icmp_code": null, "tcp_win": 1024, "tcp_ack": 2915778850, "tcp_urp": 0, "icmp_type": null, "ip_len": 52, "src_ip": "10.10.10.10", "src_port": 39686, "ip_hlen": 5, "ip_off": 0, "tcp_csum": 43620, "ip_ttl": 64, "sid": 4, "ip_tos": 0, "ip_csum": 32849, "tcp_res": 0, "status": 0, "t_url": "https://192.168.1.3/alert.php?&testing=5", "timestamp": "2017-09-11 21:52:45", "ip_ver": 4, "ip_flags": 2, "tcp_off": 8, "ip_id": 57472, "cid": 5493, "DataType": "ids", "tcp_seq": 1593577001, "dst_port": 443, "signature": "ALLERT Raised on traffic", "ip_proto": 6, "dst_ip": "2.3.4.5", "CustID": "Test"}` **LS processed log:** `{"event"=>{"icmp_type"=>nil, "ip_proto"=>6, "ip_tos"=>0, "signature"=>"ALLERT Raised on traffic", "tcp_off"=>8, "ip_id"=>57472, "tcp_urp"=>0, "dst_ip"=>"2.3.4.5", "sid"=>4, "ip_ver"=>4, "src_ip"=>"10.10.10.10", "ip_ttl"=>64, "ip_off"=>0, "tcp_flags"=>16, "ip_csum"=>32849, "tcp_seq"=>1593577001, "@version"=>"1", "host"=>"test", "DataType"=>"ids", "src_geoip"=>{}, "icmp_code"=>nil, "tcp_res"=>0, "tcp_win"=>1024, "ip_flags"=>2, "src_port"=>39686, "tcp_ack"=>2915778850, "@timestamp"=>2017-09-11T21:52:45.000Z, "CustID"=>"Test", "dst_port"=>443, "t_url"=>"https://192.168.1.3/alert.php?&testing=5", "ip_hlen"=>5, "ip_len"=>52, "tcp_csum"=>43620, "cid"=>5493}}` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 12, 2017, 1:48am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/2 "2017-09-12T01:48:37Z")
What does the mapping for the index look like? ---
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://discuss.elastic.co/u/theflakes)\ **Post date:** [September 12, 2017, 1:53am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/3 "2017-09-12T01:53:08Z")
I've removed my template index mappings for "logstash-\*" and tried just using the default ES dynamic mappings and I see the same behavior with or without my mappings. Unfortunately my template mapping is huge. Here's the "@timestamp" portion. > ``` > "properties" : { > "@timestamp": { "type": "date" }, > > ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 12, 2017, 1:54am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/4 "2017-09-12T01:54:47Z")
What version of the stack are you on? ---
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://discuss.elastic.co/u/theflakes)\ **Post date:** [September 12, 2017, 1:56am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/5 "2017-09-12T01:56:02Z")
ES 5.5.2 Testing Elastic's cloud offering. ---
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://discuss.elastic.co/u/theflakes)\ **Post date:** [September 12, 2017, 2:43am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/6 "2017-09-12T02:43:45Z")
Just upgraded to 5.6 and seeing the same issue. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 12, 2017, 2:58am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/7 "2017-09-12T02:58:57Z")
> [@theflakes](#): > > Unfortunately my template mapping is huge. What about the actual mapping though, not the template. ---
**Author:** ![ikirani](https://avatars.discourse-cdn.com/v4/letter/i/a9adbd/32.png) [@ikirani](https://discuss.elastic.co/u/ikirani)\ **Post date:** [September 12, 2017, 3:01am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/8 "2017-09-12T03:01:57Z")
> [@theflakes](#): > > All logs are single line JSON formatted. I am seeing a similar issue. I've narrowed it down to Logstash json filter. I've removed all other filters and processing on the incoming messages and wrote output to a log file instead of Elasticsearch. I see a variable number of messages for the same 100 messages I post to Logstash in my output file. I have another thread describing the same issue. Perhaps try writing a sample output of few messages to a log file... that way you can narrow it down too. I've also noticed that X-pack shows 100 ingested events and 100 emitted events on Kibana. But when Logstash persists to a file, few are missing... a random number every single time. I believe some thread bug but hope not as we are blocked on moving to Production because of this issue. ---
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://discuss.elastic.co/u/theflakes)\ **Post date:** [September 12, 2017, 3:17am UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/9 "2017-09-12T03:17:17Z")
So in my case I've narrowed it down to the "geoip" filters. If I remove these filters in my LS pipeline my logs come in fine. I have two filters: ``` geoip { source => "src_ip" target => "src_geoip" } geoip { source => "dst_ip" target => "dst_geoip" } ``` Not sure why they are causing this issue as yet. What is odd is that the logs that work have those filters applied to them as well and I get the GeoIP data in the log in ES. But removing those two filters fixes the problem for me so far. ---
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://discuss.elastic.co/u/theflakes)\ **Post date:** [September 12, 2017, 5:18pm UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/10 "2017-09-12T17:18:53Z")
Here's the actual mappings for src\_geoip and dst\_geoip: ``` "dst_geoip": { "properties": { "city_name": { "type": "keyword" }, "continent_code": { "type": "keyword" }, "country_code2": { "type": "keyword" }, "country_code3": { "type": "keyword" }, "country_name": { "type": "keyword" }, "ip": { "type": "ip" }, "latitude": { "type": "half_float" }, "location": { "type": "geo_point" }, "longitude": { "type": "half_float" }, "postal_code": { "type": "keyword" }, "region_code": { "type": "short" }, "region_name": { "type": "keyword" }, "timezone": { "type": "keyword" } } "src_geoip": { "properties": { "city_name": { "type": "keyword" }, "continent_code": { "type": "keyword" }, "country_code2": { "type": "keyword" }, "country_code3": { "type": "keyword" }, "country_name": { "type": "keyword" }, "ip": { "type": "ip" }, "latitude": { "type": "half_float" }, "location": { "type": "geo_point" }, "longitude": { "type": "half_float" }, "postal_code": { "type": "keyword" }, "region_code": { "type": "short" }, "region_name": { "type": "keyword" }, "timezone": { "type": "keyword" } } ``` ---
**Author:** ![theflakes](https://avatars.discourse-cdn.com/v4/letter/t/839c29/32.png) [@theflakes](https://discuss.elastic.co/u/theflakes)\ **Post date:** [September 12, 2017, 5:24pm UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/11 "2017-09-12T17:24:34Z")
Yep, its official, I'm blind. I had region\_code defined as a short type. Don't remember why I did that, but changed that to keyword type and all is good. My apologies. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 12, 2017, 10:54pm UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/12 "2017-09-12T22:54:24Z")
All good, thanks for clarifying the root problem! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 10, 2017, 10:55pm UTC](https://discuss.elastic.co/t/some-logs-not-being-ingested-from-ls-to-es/100147/13 "2017-10-10T22:55:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.