So I'm new to ELK Stack, but I've watched a ton of tutorials, read the manual, have looked through this site a lot and I finally got things working but the data is not showing up right.
Logstash successfully pipelines to ES and sends my .csv file, but the amount of documents created are completely wrong. There is over 33,000k rows in this .csv but the counter on the "deleted" side shows only 3100. Not only that, the counter goes both up and down as I watch it ingest. I have no idea how to start troubleshooting this because the entire elkstack is working properly.
Kibana barely sees anything other than the headers because of this and I can't create any dashboard filters or present data.
What could possibly be the issue? Its a pretty standard .csv file with simple columns and headers.
config file listed below:
input {
file {
path => "/Users/S.AWDRSRV/Downloads/MDMCSV/*.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => [ "LGName", "LocationGroupNameValue", "LocationNameValue", "DeviceNameValue",
"DeviceOwnerValue", "EmailAddressValue", "SerialNumberValue", "MacAdreessValue",
"IMEIESNValue", "ModeValue", "OSVersionValue", "CorpEmpValue", "AgentNameValue",
"VersionValue", "AgentFirstSeenValue", "AgentLastSeenValue", "LastGPSDataValue",
"LastMemoryDataValue", "LastSeenValue" ]
}
}
output {
elasticsearch {
action => "index"
hosts => ["http://localhost:9200"]
index => "mobileairwatch"
document_type => "airwatchinventory"
document_id => "%{awidetails}"
}
stdout { codec => rubydebug }
}