# How to forward and index the json formatted log files in ELK-docker
**URL:** https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509
**Category:** Beats
**Tags:** filebeat
**Created:** [November 9, 2016, 2:38pm UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509 "2016-11-09T14:38:55Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@himaz.m](https://discuss.elastic.co/u/himaz.m)
#### Post date: [November 9, 2016, 2:38pm UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509/1 "2016-11-09T14:38:55Z")
Hi, I'm configuring the ELK-docker for the first time and this is new to me.
I'm using,
- filebeat version 5.0
- elasticsearch version 5.0
- kibana version 5.0
I have filebeat configured on a different server and ELK-docker on a different server. The log files are stored on the same server the filebeat is configured. As a beginner I was trying to follow the shakespeare example provided [here](https://www.elastic.co/guide/en/kibana/current/tutorial-load-dataset.html)
My filebeat.yml looks like this.
```
filebeat.prospectors:
- input_type: log
paths:
- /opt/ALLMODULESLOG/*.log
output.elasticsearch:
hosts: ["10.44.2.48:9200"]
template:
name: "shakespeare"
path: "/etc/filebeat/filebeat.shakespeare.json"
overwrite: true
```
I have the log file inside '/opt/ALLMODULESLOG/' downloaded from [here](https://www.elastic.co/guide/en/kibana/3.0/snippets/shakespeare.json) and I renamed it to 'shakespeare.log' (I renamed it because my other log files are in .log extension, I assume this won't make any issues)
My filebeat.shakespeare.json looks like this
`{"mappings":{"_default_":{"_all":{"norms":{"enabled":false}},"_meta":{"version":"5.0.0"},"dynamic_templates":[{"strings_as_keyword":{"mapping":{"ignore_above":1024,"index":"not_analyzed","type":"string"},"match_mapping_type":"string"}}],"properties":{"speaker":{"type":"string","index":"not_analyzed"},"play_name":{"type":"string","index":"not_analyzed"},"line_id":{"type":"integer"},"speech_number":{"type":"integer"},"line_number":{"type":"integer"},"text_entry":{"type":"string","index":"not_analyzed"}}}},"order":0,"settings":{"index.refresh_interval":"5s"},"template":"shakespeare"}`
Also I added the mapping by following command
```
curl -XPUT http://localhost:9200/shakespeare -d '
{
"mappings" : {
"_default_" : {
"properties" : {
"speaker" : {"type": "string", "index" : "not_analyzed" },
"play_name" : {"type": "string", "index" : "not_analyzed" },
"line_id" : { "type" : "integer" },
"speech_number" : { "type" : "integer" }
}
}
}
}
';
```
But when I added the index 'shakespeare' on kibana (Settings -\> Indices) it shows the fields **\_source, \_id, \_type, \_index, \_score** and not the **speaker, play\_name, line\_id, speech\_number** fields. Please let me know if i'm missing anything? What is the configuration I need to do to forward and index the json formatted log files to elasticsearch?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [November 10, 2016, 9:16am UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509/2 "2016-11-10T09:16:17Z")
To process json lines, you have to use the json config options: [https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#config-json](https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#config-json)
---
### Author:  [@himaz.m](https://discuss.elastic.co/u/himaz.m)
#### Post date: [November 10, 2016, 1:38pm UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509/3 "2016-11-10T13:38:01Z")
Thank you @ruflin. I think after adding the following json config fields,
```
json.message_key: log
json.keys_under_root: true
json.add_error_key: true
```
I can see the index created in kibana under 'filebeat-2016.11.10' and not under 'shakespeare'. I saw the below logs been logged in elasticsearch, wondering this would help you why it was not created the index under 'shakespeare'.
```
[2016-11-10T12:53:47,377][INFO][o.e.c.m.MetaDataCreateIndexService] [abmwpmt] [shakespeare] creating index, cause [api], templates [shakespeare], shards [5]/[1], mappings [_default_]
[2016-11-10T12:57:20,094][INFO][o.e.c.m.MetaDataCreateIndexService] [abmwpmt] [filebeat-2016.11.10] creating index, cause [auto(bulk api)], templates [filebeat], shards [5]/[1], mappings [_default_]
[2016-11-10T12:57:20,228][INFO][o.e.c.m.MetaDataMappingService] [abmwpmt] [filebeat-2016.11.10/pM4ln6sWR1GrizpZRD5gJQ] create_mapping [json]
[2016-11-10T12:57:20,252][INFO][o.e.c.m.MetaDataMappingService] [abmwpmt] [filebeat-2016.11.10/pM4ln6sWR1GrizpZRD5gJQ] update_mapping [json]
```
Also after a while I've added few more lines to the 'shakespeare.log' file and searched through kibana by using 'filebeat-2016.11.10' mapping. But the search result didn't appear instantly (after few munites may be). Are there any configurations to search it as soon as the log file updated?
Please help to solve these issues!!
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [November 11, 2016, 10:05am UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509/4 "2016-11-11T10:05:15Z")
If you want to use a different index, you have to specific it in the elasticsearch output: [https://www.elastic.co/guide/en/beats/filebeat/5.0/elasticsearch-output.html#\_index](https://www.elastic.co/guide/en/beats/filebeat/5.0/elasticsearch-output.html#_index)
How fast a new line appears depends on various factors like backoff: [https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#\_backoff](https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#_backoff) If you manually update a file and only very rarely, it can take longer as it reached max back off. But normally with most logging system new lines come in very often which also makes sending it more often. Best have a look at the different config options like `scan_frequency`, `backoff` if this really becomes an issue.
---
### Author:  [@himaz.m](https://discuss.elastic.co/u/himaz.m)
#### Post date: [November 11, 2016, 2:08pm UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509/5 "2016-11-11T14:08:42Z")
Thank you @ruflin. Is that the default behavior? I mean when we create an index with the name 'shakespeare' and searched through Kibana, it has created with some default fields. But the 'filebeat-\*' index is updated with the fields included in 'shakespeare'.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [November 14, 2016, 1:12pm UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509/6 "2016-11-14T13:12:49Z")
Not sure I get your question about the fields part. The default behaviour is to use the `filebeat-*` index.
---
### Author:  [@himaz.m](https://discuss.elastic.co/u/himaz.m)
#### Post date: [November 18, 2016, 8:13am UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509/7 "2016-11-18T08:13:25Z")
Thank You @ruflin.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 16, 2016, 8:14am UTC](https://discuss.elastic.co/t/how-to-forward-and-index-the-json-formatted-log-files-in-elk-docker/65509/8 "2016-12-16T08:14:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.