# How to configure logstash to index json logs including all fields
**URL:** https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874
**Category:** Logstash
**Created:** [August 25, 2016, 12:36am UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874 "2016-08-25T00:36:35Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@lin-zhao](https://discuss.elastic.co/u/lin-zhao)
#### Post date: [August 25, 2016, 12:36am UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874/1 "2016-08-25T00:36:36Z")
Say the input of my logstash job is already in line delimited json format. I want to have each json key indexed as a field to elasticsearch, is there to make this work?
For example, the input looks like:
{ "a": "value of a", "b": "value of b", "@timestamp": "some time", "message": "raw log line"}
{ "a": "another value of aa", "b": "another value of b", "@timestamp": "some time", "message": "raw log line"}
I would like to have "message", "@timestamp", "a" and "b" both indexed as seperate fields in elastic search.
I tried something like
input { stdin {
codec =\> json\_lines
}
}
output {
elasticsearch {
codec =\> json
hosts =\> ["[http://10.10.2.53:9200](http://10.10.2.53:9200)"]
template =\> "/home/ec2-user/logstash-5.0.0-alpha5/logstash.template.json"
}
}
but none of the lines get indexed. Logstash seems to try to include the whole log into "message", and add "@timestamp" etc. What if I already have "message" and "@timestamp" and want to have them indexed to elasticsearch unchanged?
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 25, 2016, 5:55am UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874/2 "2016-08-25T05:55:33Z")
Works just fine for me (except the `@timestamp` warning):
```nohighlight
$ cat test.config
input { stdin { codec => json_lines } }
output { stdout { codec => rubydebug } }
$ cat data
{ "a": "value of a", "b": "value of b", "@timestamp": "some time", "message": "raw log line"}
{ "a": "another value of aa", "b": "another value of b", "@timestamp": "some time", "message": "raw log line"}
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
Error parsing @timestamp string, setting current time to @timestamp, original in _@timestamp field {:value=>"\"some time\"", :exception=>"invalid timestamp string \"some time\", error=java.lang.IllegalArgumentException: Invalid format: \"some time\"", :level=>:warn}
Error parsing @timestamp string, setting current time to @timestamp, original in _@timestamp field {:value=>"\"some time\"", :exception=>"invalid timestamp string \"some time\", error=java.lang.IllegalArgumentException: Invalid format: \"some time\"", :level=>:warn}
{
"a" => "value of a",
"b" => "value of b",
"@timestamp" => "2016-08-25T05:53:01.660Z",
"message" => "raw log line",
"@version" => "1",
"tags" => [
[0] "_timestampparsefailure"
],
"_@timestamp" => "some time",
"host" => "lnxolofon"
}
{
"a" => "another value of aa",
"b" => "another value of b",
"@timestamp" => "2016-08-25T05:53:01.662Z",
"message" => "raw log line",
"@version" => "1",
"tags" => [
[0] "_timestampparsefailure"
],
"_@timestamp" => "some time",
"host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
```
---
### Author:  [@lin-zhao](https://discuss.elastic.co/u/lin-zhao)
#### Post date: [August 26, 2016, 4:52pm UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874/3 "2016-08-26T16:52:11Z")
Thanks for the reply Magnus. But I need the output plugin to write to elasticsearch instead of stdout.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 27, 2016, 9:51am UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874/4 "2016-08-27T09:51:53Z")
I just saw that you're setting `codec => json` for your elasticsearch output. Don't do that.
---
### Author:  [@lin-zhao](https://discuss.elastic.co/u/lin-zhao)
#### Post date: [August 29, 2016, 4:39pm UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874/5 "2016-08-29T16:39:36Z")
What's the right codec for my purpose? I've tried plain, lines to no vail.
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [August 29, 2016, 5:41pm UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874/6 "2016-08-29T17:41:47Z")
Just drop the `codec` setting for the elasticsearch output.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 4:41am UTC](https://discuss.elastic.co/t/how-to-configure-logstash-to-index-json-logs-including-all-fields/58874/7 "2017-07-06T04:41:04Z")