# Determining why records are missing after Logstash indexing
**URL:** https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848
**Category:** Logstash
**Created:** [October 8, 2015, 1:26pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848 "2015-10-08T13:26:09Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@marinoc](https://discuss.elastic.co/u/marinoc)
#### Post date: [October 8, 2015, 1:26pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/1 "2015-10-08T13:26:09Z")
Hi - I'm attempting to index a single w 19,690 lines, each line will be it's own record in ES. The lines are pretty small, only 4-5 fields per line.
After I run logstash using the following command:
sudo bin/logstash -f config/hardball.conf --debug -l hardball.log
I see that my Elasticsearch index only contains 19,502 records.
I've tried looking in hardball.log but it's 200 MB.
Any recommendations on what to search for in the log file to see what might have happened to the missing 100+ records? Any other debugging tips w logstash?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 8, 2015, 1:32pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/2 "2015-10-08T13:32:02Z")
What does your logstash config look like? Did you wait for Elasticsearch to refresh before counting the documents?
---
### Author:  [@marinoc](https://discuss.elastic.co/u/marinoc)
#### Post date: [October 8, 2015, 1:50pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/3 "2015-10-08T13:50:52Z")
Here's my config ... didn't see an option to attach a file.
```
# The # character at the beginning of a line indicates a comment. Use
# comments to describe your configuration.
##################################
# INPUT
##################################
input {
file {
path => "/var/data/hardball/players/baseballPersons.txt"
type => "baseballPerson"
sincedb_path => "/dev/null"
start_position => "beginning"
}
}
##################################
# FILTERS
##################################
filter {
# Indexing retrosheet player data file w following fields:
# Fields: LAST,FIRST,ID,DEBUT
# LAST
# FIRST
# ID
# DEBUT
if [type] == "baseballPerson" {
grok {
patterns_dir => "/var/logstash/patterns"
match => ["message", "%{PERSONNAME:lastName},%{PERSONNAME:firstName},%{BASEBALLPERSONID:baseballPersonID},%{DATE_US:dateDebut}"]
add_field => { "baseballPersonType" => "%{baseballPersonID}" }
}
# grok {
# patterns_dir => "/var/logstash/patterns"
# match => ["message", "%{PERSONNAME},%{PERSONNAME},([a-z\-]{4})([a-z])%{SINGLEDIGIT:baseballPersonType}([0-9]{2}),%{DATE_US}"]
# }
mutate {
gsub => [
# parse first number which tells type of baseball person
"baseballPersonType", "([a-z\-]{4})([a-z])([0-9])([0-9]{2})", "\3"
]
remove_field => ["host", "path"]
}
}
}
##################################
# OUTPUT
##################################
output {
elasticsearch {
hosts => ["localhost"]
index => "players"
document_id => "%{baseballPersonID}"
}
# stdout { codec => json }
}
```
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 8, 2015, 2:01pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/4 "2015-10-08T14:01:50Z")
As you are explicitly setting the ID of the documents, is it possible that your file contains duplicates? Do you have any records that for some reason has failed parsing and have been indexed with the string "%{baseballPersonID}" as a key?
---
### Author:  [@marinoc](https://discuss.elastic.co/u/marinoc)
#### Post date: [October 8, 2015, 2:09pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/5 "2015-10-08T14:09:10Z")
I'll check, Christian. Would any of this be reported in logstash log so I could grep for it?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [October 8, 2015, 9:06pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/6 "2015-10-08T21:06:41Z")
Not likely, no.
---
### Author:  [@marinoc](https://discuss.elastic.co/u/marinoc)
#### Post date: [October 13, 2015, 4:20pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/7 "2015-10-13T16:20:03Z")
Thanks again, Christian. I went back a checked the IDs that made it to ES against a full list of IDs that should have made it and found that I had some parsing failures in my grok filter.
Basically, I used Linux commands to produce a list of IDs from my source file and the following command to produce a list of IDs from ES.
curl -XGET "localhost:9200/players/\_search?pretty&from=0&size=20000" | grep '"\_id"' | cut -d '"' -f 4 | sort -u \> ES-IDs.txt
I compared these lists to see which were missing from ES and then went back to the source file to find out why a particular line had failed.
Is there a better way to debug these grok parse failure issues?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 13, 2015, 4:27pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/8 "2015-10-13T16:27:31Z")
You could add a separate output, e.g. to a daily file, and write any records that had a \_grokparsefailure there. That could allow you to easily identify records with issues.
---
### Author:  [@marinoc](https://discuss.elastic.co/u/marinoc)
#### Post date: [October 13, 2015, 7:09pm UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/9 "2015-10-13T19:09:26Z")
Thanks. I now see that I can grep for "\_grokparsefailure" in my log file and identify which lines from my source file weren't parsed correctly. That's a big help!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 5:26am UTC](https://discuss.elastic.co/t/determining-why-records-are-missing-after-logstash-indexing/31848/10 "2017-07-06T05:26:41Z")