Elastic search not ingesting data

Elastic Stack Logstash
1

I suspect i am doing something blatantly wrong.
Logstash appears to be going through my data properly and showing the 5000 data points being sent to elastic search.
However when i look up the index in elastic search it shows only 5 events.
the configuration i used is as follows

The # character at the beginning of a line indicates a comment Use

comments to describe your configuration

input {
file {
path => "C:\Users\James Kazie\Desktop\ProV\Copy of Devices List 04-08-16 (72046).csv"
type => "ProV"
start_position => "beginning"

}

}

filter {
csv {

	columns => [
		"Agent.MachineName",
		"Version",
		"PulseInfo.IsOnline",
		"Agent.MacAddress",
		"Agent.Inventory.Computer.Model",
		"Agent.Inventory.OperatingSystem.OsName",
		"Agent.DeviceAgentVersion",
		"Agent.IpAddress",
		"Agent.NetworkAddress",
		"Agent.WriteFilter",
		"Agent.IsInPersistance",
		"PulseInfo.LastPulse"
		]
	separator => ","
}

geoip{
source => "Agent.IpAddress"
}
date {
match => ["PulseInfo.LastPulse","dd/MM/YYYY HH:mm"]
}

mutate {rename => {"Agent.MachineName"=> "WADid"}}
mutate {rename => {"PulseInfo.IsOnline" => "LastOnline" }}
mutate {rename => {"Agent.MacAddress" => "MacAddress"}}
mutate {rename => {"Agent.Inventory.OperatingSystem.OsName" => "OS"}}
mutate {rename => {"Agent.Inventory.Computer.Model"=>"ComputerModel"}}
mutate {rename => {"Agent.DeviceAgentVersion" => "AgentVersion" }}
mutate {rename => {"Agent.IpAddress" => "IP"}}
mutate {rename => {"Agent.NetworkAddress" => "NetworkAddress"}}
mutate {rename => {"Agent.WriteFilter" => "WWF" }}
mutate {rename => {"Agent.IsInPersistance" => "Persistence"}}
mutate {rename => {"PulseInfo.LastPulse" => "LastPulse"}}
}

output {
elasticsearch {
hosts=>"localhost"
action => "index"
index => "devicestatusprov"
document_id => "WADid"}
stdout {}
}

2

I realised my error, i should have used document_id => %{ID} my config was overwriting elastic search 5000 times

3

Why are you renaming the columns? Just set them to whatever you want in the CSV filter part and that is the fieldname that will be used.

1 Like
4

I mistook the use of the statement. As my file already had that column i thought i needed to specify. I am aware it is not needed now