# How to sync custom template with Incoming data from Filebeat?

**URL:** <https://discuss.elastic.co/t/how-to-sync-custom-template-with-incoming-data-from-filebeat/130867>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [May 7, 2018, 4:02pm UTC](https://discuss.elastic.co/t/how-to-sync-custom-template-with-incoming-data-from-filebeat/130867 "2018-05-07T16:02:42Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sukesh\_nagaraja](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@sukesh\_nagaraja](https://discuss.elastic.co/u/sukesh_nagaraja)\
**Post date:** [May 7, 2018, 4:02pm UTC](https://discuss.elastic.co/t/how-to-sync-custom-template-with-incoming-data-from-filebeat/130867/1 "2018-05-07T16:02:42Z")

</div>

Hi , I am trying to index json data to elasticsearch from Filebeat.

the data shipped by filebeat is as follows :

{"first name":"abc","last name":"efg","age":26,"city":"newyork","country":"USA","zipcode":"10001"}  
{"first name":"xyz","last name":"lmn","age":28,"city":"herndon","country":"USA","zipcode":"20170"}  
{"first name":"abc","last name":"pqr","age":27,"city":"chantilly","country":"USA","zipcode":"20152"}

filebeat.yml contains below config :

filebeat.prospectors:

- type: log  
enabled: true  
paths:
  - /home/sukesh/Downloads/json/\*.json  
exclude\_files: ['.csv$','.xml$','.txt$','.gz$']

#Elasticsearch template setting  
setup.template.name: "Tname"  
setup.template.pattern: "Tname-\*"  
#setup.template.fields: "/home/sukesh/Desktop/fields.yml"  
#setup.template.overwrite: true  
setup.template.settings:  
index.number\_of\_shards: 2  
#index.codec: best\_compression  
#\_source.enabled: true

#General  
tags: ["pattern1\_json"]

#Kibana  
setup.kibana:  
host: "localhost:5601"  
username: "elastic"  
password: "sukesh"

#Outputs  
output.elasticsearch:  
hosts: ["localhost:9301"]  
index: "Iname"  
username: "elastic"  
password: "sukesh"

I created template in elasticsearch as follows :

PUT \_template/Tname  
{  
"index\_patterns" : ["Iname\*"],  
"settings" : {  
"number\_of\_shards" : 1  
},  
"mappings":{  
"Idoc\_type":{  
"properties":{  
"first name":{"type":"text"},  
"last name":{"type":"text"},  
"age":{"type":"integer"},  
"city":{"type":"text"},  
"country":{"type":"text"},  
"zipcode":{"type":"text"}  
}  
}  
}  
}

I did a config test and output check and its fine.

i checked data in kibana but entire json is poping under message field:

"message" :{"first name":"abc","last name":"efg","age":26,"city":"newyork","country":"USA","zipcode":"1001"}

This means my template is not working. i need all the fields of json in kibana not under message

Pls guide me how can i sync my template with incoming data. do i need to use fields.yml?

Filebeat -\> elasticsearch -\> kibana

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [May 8, 2018, 8:12pm UTC](https://discuss.elastic.co/t/how-to-sync-custom-template-with-incoming-data-from-filebeat/130867/2 "2018-05-08T20:12:31Z")

</div>

Hello @sukesh_nagaraja,

You can control the behavior of the JSON parser in Filebeat at the [prospector level](https://www.elastic.co/guide/en/beats/filebeat/5.4/configuration-filebeat-options.html#config-json).

The option you want is `json.keys_under_root: true`

> keys\_under\_root  
> By default, the decoded JSON is placed under a "json" key in the output document. If you enable this setting, the keys are copied top level in the output document. The default is false.

---

<div class="post-metadata">

**Author:** ![sukesh\_nagaraja](https://avatars.discourse-cdn.com/v4/letter/s/58956e/32.png) [@sukesh\_nagaraja](https://discuss.elastic.co/u/sukesh_nagaraja)\
**Post date:** [May 8, 2018, 9:24pm UTC](https://discuss.elastic.co/t/how-to-sync-custom-template-with-incoming-data-from-filebeat/130867/3 "2018-05-08T21:24:36Z")

</div>

Thanks for the response. I tried below it works.

```
      filebeat.prospectors:
      - type: log
      enabled: true
      paths:
      - /home/sukesh/Downloads/json/*.json  

     json.keys_under_root: true
     json.add_error_key: true

     setup.kibana:
     host: "localhost:5601"
     username: "elastic"
     password: "sukesh"

     output.elasticsearch:
     hosts: ["localhost:9301"]
     index: "freddie_test"
     username: "elastic"
     password: "sukesh"

```

the above config works for very simple json with no nesting data and multiline

to handle multiline : i have below config but all content sits under one field called "log".

sample data:  
{"nesting data":[{"category":"Audi","region":"East","monthlySales":[{"month":20130101,"sales":38},{"month":20130201,"sales":35},{"month":20130301,"sales":41},{"month":20130401,"sales":55},{"month":20130501,"sales":58},{"month":20130601,"sales":66},{"month":20130701,"sales":74},{"month":20130801,"sales":78},{"month":20130901,"sales":38},{"month":20131001,"sales":30},{"month":20131101,"sales":26},{"month":20131201,"sales":29}]},{"category":"Technology","region":"West","monthlySales":[{"month":20130101,"sales":54},{"month":20130201,"sales":66},{"month":20130301,"sales":77},{"month":20130401,"sales":70},{"month":20130501,"sales":60},{"month":20130601,"sales":63},{"month":20130701,"sales":55},{"month":20130801,"sales":47},{"month":20130901,"sales":55},{"month":20131001,"sales":30},{"month":20131101,"sales":22},{"month":20131201,"sales":77}]}]}

config details:

```
      filebeat.prospectors:
     - type: log
     enabled: true
     paths:
         - /home/sukesh/Downloads/json/*.json

     json.message_key: log
     json.keys_under_root: true
     json.add_error_key: true

   multiline.pattern: '^{'
   multiline.negate: true
   multiline.match: after
   processors:
   - decode_json_fields:
   fields: ['message']
   target: json  

   setup.kibana:
   host: "localhost:5601"
   username: "elastic"
   password: "sukesh"

   output.elasticsearch:
   hosts: ["localhost:9301"]
   index: "freddie_test"
   username: "elastic"
   password: "sukesh"

```

How can i get each key values individually rather than wrapped up under one field?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 5, 2018, 9:24pm UTC](https://discuss.elastic.co/t/how-to-sync-custom-template-with-incoming-data-from-filebeat/130867/4 "2018-06-05T21:24:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.