How to configure logstash to index json logs including all fields

Elastic Stack Logstash
1

Say the input of my logstash job is already in line delimited json format. I want to have each json key indexed as a field to elasticsearch, is there to make this work?

For example, the input looks like:

{ "a": "value of a", "b": "value of b", "@timestamp": "some time", "message": "raw log line"}
{ "a": "another value of aa", "b": "another value of b", "@timestamp": "some time", "message": "raw log line"}

I would like to have "message", "@timestamp", "a" and "b" both indexed as seperate fields in elastic search.

I tried something like

input { stdin {
codec => json_lines
}
}
output {
elasticsearch {
codec => json
hosts => ["http://10.10.2.53:9200"]
template => "/home/ec2-user/logstash-5.0.0-alpha5/logstash.template.json"
}
}

but none of the lines get indexed. Logstash seems to try to include the whole log into "message", and add "@timestamp" etc. What if I already have "message" and "@timestamp" and want to have them indexed to elasticsearch unchanged?

2

Works just fine for me (except the @timestamp warning):

$ cat test.config 
input { stdin { codec => json_lines } }
output { stdout { codec => rubydebug } }
$ cat data 
{ "a": "value of a", "b": "value of b", "@timestamp": "some time", "message": "raw log line"}
{ "a": "another value of aa", "b": "another value of b", "@timestamp": "some time", "message": "raw log line"}
$ /opt/logstash/bin/logstash -f test.config < data
Settings: Default pipeline workers: 8
Pipeline main started
Error parsing @timestamp string, setting current time to @timestamp, original in _@timestamp field {:value=>"\"some time\"", :exception=>"invalid timestamp string \"some time\", error=java.lang.IllegalArgumentException: Invalid format: \"some time\"", :level=>:warn}
Error parsing @timestamp string, setting current time to @timestamp, original in _@timestamp field {:value=>"\"some time\"", :exception=>"invalid timestamp string \"some time\", error=java.lang.IllegalArgumentException: Invalid format: \"some time\"", :level=>:warn}
{
              "a" => "value of a",
              "b" => "value of b",
     "@timestamp" => "2016-08-25T05:53:01.660Z",
        "message" => "raw log line",
       "@version" => "1",
           "tags" => [
        [0] "_timestampparsefailure"
    ],
    "_@timestamp" => "some time",
           "host" => "lnxolofon"
}
{
              "a" => "another value of aa",
              "b" => "another value of b",
     "@timestamp" => "2016-08-25T05:53:01.662Z",
        "message" => "raw log line",
       "@version" => "1",
           "tags" => [
        [0] "_timestampparsefailure"
    ],
    "_@timestamp" => "some time",
           "host" => "lnxolofon"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}
3

Thanks for the reply Magnus. But I need the output plugin to write to elasticsearch instead of stdout.

4

I just saw that you're setting codec => json for your elasticsearch output. Don't do that.

5

What's the right codec for my purpose? I've tried plain, lines to no vail.

6

Just drop the codec setting for the elasticsearch output.