Empty record in ElasticSearch

[czome (Peter Spin)]

Hello,

I don't understand why my logstash config don't save data in ES. My data comes from an xml file and here is my pipeline conf :

input {
beats {
port => "5044"
host => "localhost"
client_inactivity_timeout => 3600
}
}
filter
{
xml
{
source => "message"
remove_namespaces => "true"
xpath => ["/ImportLogs/@LogResult", "logResult",
"/ImportLogs/@FileName", "fileName",
"/ImportLogs/@DateImport", "importDate",
"/ImportLogs/@ImportType", "importType",
"/ImportLogs/@ImportFormat", "importFormat",
"/ImportLogs/@username", "username"
#"/ImportLogs/LineLogs/linelog/@lineNumber", "lineNumber",
#"/ImportLogs/LineLogs/linelog/@lineState", "lineState",
# "/ImportLogs/LineLogs/linelog/log/message/text()", "logMessage"
]
store_xml => false
target => "message"
}
mutate {
replace => { "logResult" => "%{logResult[0]}" }
replace => { "fileName" => "%{fileName[0]}" }
replace => { "importType" => "%{importType[0]}" }
replace => { "username" => "%{username[0]}" }
replace => { "importDate" => "%{importDate[0]}" }
replace => { "importFormat" => "%{importFormat[0]}" }
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
index => "myWorkIndex"
}
stdout { codec => rubydebug }
}
My ES picture's

And then my logstash Output :

Can you help me please?

[Christian_Dahlqvist]

@czome

index => "myWorkIndex"

Index names must be lower case. You should be able to see something about this in the Elasticsearch logs.

[czome (Peter Spin)]

Thank for your reactivity, but even if i don't specify index name ,no change. It will create default index (logstash-$date) but with no data.

[Christian_Dahlqvist]

What is the output of the cat indices API?

[czome (Peter Spin)]

As you see my index was created, but when i request this index , i see no hits.

[Christian_Dahlqvist]

There is data in the index, so it looks like your query may be wrong. Try q=response:200 instead.

[czome (Peter Spin)]

As i sended in this picture , you will see no "hits". no data in my hits collection :

I'm sure that something is wrong in my filebeat config or logstash config.

[Christian_Dahlqvist]

Your query is still q=response=200 which is not the example I gave.

[czome (Peter Spin)]

No change.

[Christian_Dahlqvist]

Do you even have a field named response in your data? Can you show a full document from that index?

[czome (Peter Spin)]

No i haven't field "response", here is my origin xml document :

<?xml version="1.0" encoding="utf-8"?>
<ImportLogs LogResult="SUCCESS" ExecTime="00:00:02.424" FileName="filename.csv" ignored="17" valid="62" alert="0" error="0" DateImport="2018-06-13T09:03:24Z" ImportMode="InsertUpdate" user="Tutu Toto" username="tutu.qiqi@gmail.com" ImportType="Organisation" ImportFormat="Organisations">
	<GeneralLog />
	<LineLogs>
		<linelog lineNumber="11" lineState="Ignored">
			<log type="Warning">
				<message>L'individu "tutu franck" n'existe pas</message>
			</log>
		</linelog>
		<linelog lineNumber="14" lineState="Ignored">
			<log type="Warning">
				<message>L'individu "Juju.Huhu.23544.20382807" n'existe pas</message>
			</log>
		</linelog>
	</LineLogs>
</ImportLogs>

[Christian_Dahlqvist]

That is what your query is searching for, so it is then not surprising it does not find any results.

[czome (Peter Spin)]

Sorry, can you explain your answer please ?
In my logstash output console i have :

Why this output don't flush in ElasticSearch ?
Here is my elasticSearch screen shot :

And my xml is :

<?xml version="1.0" encoding="utf-8"?>
<ImportLogs LogResult="SUCCESS" ExecTime="00:00:02.424" FileName="ORG.csv" ignored="17" valid="62" alert="0" error="0" DateImport="2018-06-13T09:03:24Z" ImportMode="InsertUpdate" user="Tutu Kiki" username="tutu.kiki@gmail.com" ImportType="Organisation" ImportFormat="Organisations">
	<GeneralLog />
	<LineLogs>
		<linelog lineNumber="11" lineState="Ignored">
			<log type="Warning">
				<message>L'individu "Tutu.Franck.31966.20381925" n'existe pas</message>
			</log>
		</linelog>
		<linelog lineNumber="14" lineState="Ignored">
			<log type="Warning">
				<message>L'individu "Laura.Beffa.23544.20382807" n'existe pas</message>
			</log>
		</linelog>
	</LineLogs>
</ImportLogs>

What's wrong in my logstash conf ? or filebeat.yml conf please ?

[Christian_Dahlqvist]

There is not necessarily anything wrong with your Logstash or Filebeat config. It is your query that is wrong as you are searching for something that does not match any documents. Try with a query that matches all documents in Elasticsearch: q=*:*

[czome (Peter Spin)]

Very good, it was my problem. Sorry, i'm starting on ELK.
Thank's for your help. Now i will try to display some graphic on kibana.
see you soon