Hi , I am trying to index json data to elasticsearch from Filebeat.
the data shipped by filebeat is as follows :
{"first name":"abc","last name":"efg","age":26,"city":"newyork","country":"USA","zipcode":"10001"}
{"first name":"xyz","last name":"lmn","age":28,"city":"herndon","country":"USA","zipcode":"20170"}
{"first name":"abc","last name":"pqr","age":27,"city":"chantilly","country":"USA","zipcode":"20152"}
filebeat.yml contains below config :
filebeat.prospectors:
- type: log
enabled: true
paths:- /home/sukesh/Downloads/json/*.json
exclude_files: ['.csv$','.xml$','.txt$','.gz$']
- /home/sukesh/Downloads/json/*.json
#Elasticsearch template setting
setup.template.name: "Tname"
setup.template.pattern: "Tname-*"
#setup.template.fields: "/home/sukesh/Desktop/fields.yml"
#setup.template.overwrite: true
setup.template.settings:
index.number_of_shards: 2
#index.codec: best_compression
#_source.enabled: true
#General
tags: ["pattern1_json"]
#Kibana
setup.kibana:
host: "localhost:5601"
username: "elastic"
password: "sukesh"
#Outputs
output.elasticsearch:
hosts: ["localhost:9301"]
index: "Iname"
username: "elastic"
password: "sukesh"
I created template in elasticsearch as follows :
PUT _template/Tname
{
"index_patterns" : ["Iname*"],
"settings" : {
"number_of_shards" : 1
},
"mappings":{
"Idoc_type":{
"properties":{
"first name":{"type":"text"},
"last name":{"type":"text"},
"age":{"type":"integer"},
"city":{"type":"text"},
"country":{"type":"text"},
"zipcode":{"type":"text"}
}
}
}
}
I did a config test and output check and its fine.
i checked data in kibana but entire json is poping under message field:
"message" :{"first name":"abc","last name":"efg","age":26,"city":"newyork","country":"USA","zipcode":"1001"}
This means my template is not working. i need all the fields of json in kibana not under message
Pls guide me how can i sync my template with incoming data. do i need to use fields.yml?
Filebeat -> elasticsearch -> kibana