Need help - customized fields are not identified


(Ash) #1

I use filebeat sending log to host and noticed that elasticsearch generated "filebeat+date" as index name each day, and I know it is by default which indicated in filebeat.yml.

I have customized log with clientip,path,duration fields and want to represent in KIBANA.
But they still be included into "message"(Please see attached screenshot)


I tried to update mapping but it fails due to there is no index "type" value.
Here is the settings of index: filebeat-2017.08.04
{
"filebeat-2017.08.04": {
"settings": {
"index": {
"refresh_interval": "5s",
"number_of_shards": "5",
"provided_name": "filebeat-2017.08.04",
"creation_date": "1501809893060",
"number_of_replicas": "1",
"uuid": "(marked)--ash",
"version": {
"created": "5040299"
}
}
}
}
}

Command: /usr/share/logstash/conf# curl -X PUT 'http://localhost:9200/filebeat-2017.08.04/_mapping?ignore_conflicts=true' -d \

'{
"filebeat-2017.08.04": {
"log": {
"clientip": {
"type": "string",
},
"duration": {
"type": "double"
},
"path": {
"type": "string"
}
}
}
}'
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"request [/filebeat-2017.08.04/_mapping] contains unrecognized parameter:"illegal_argument_exception","reason":"request [/filebeat-2017.08.04/_mapping] contains unrecognized parameter: [ignore_conflicts]"},"status":4

2)I also tried to customized my log and put index setting into log message to my newly created index"django", log as below.
But log still apply to filebeat newly created index.
2017-08-03 09:40:50,276 [138] [DesAllPNVer_predictor:list_all][INFO]-
{ "index" : { "_index" : "django", "_type" : "django-1", "_id" : "1" } }
{"Request method":"GET", "Path":"/path/value/", "clientip":"XX.XXX.XX.XXX", "Duration":"1.811981201171875e-05 Sec"}

Please advise the best way to make KIBANA identify the customized fields and keeps on applying the same index each day?
Will update index help identify the customized fields in KIBANA?
How can I avoid filebeat to generate different index each day?
How to assign daily logs to apply the same index?

Thanks,
Ashley


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.