Field name duplicate with file conf and json doc

Elastic Stack Logstash
1

Hello.

I'm new to logstash and using logstash 1.5.2, elasticsearch 1.7.x

And have logstash configuration like...

  1 input {
  2     file {
  3         path => "/home1/.logstash/json_input/deco.json.*"
  4         codec => json
  5         start_position => beginning
  6         type => "decojson"
  7     }
  8     file {
  9         path => "/home1/.logstash/json_input/mplatform.*"
 10         codec => json
 11         start_position => beginning
 12         type => "mpjson"
 13     }
 14 }
 15 output{
 16     if [type] == "decojson" {
 17         elasticsearch {
 18             index => "logstash  -%{+YYYY.MM.dd}"
 19             document_type => "deco_events"
 20             host => ["10.9..:9300","10.9..:9300"]
 21             protocol => transport
 22             cluster => DES
 23             manage_template => false
 24         }
 25     }else if [type] == "mpjson" {
 26         elasticsearch {
 27             index => "logstash-mpsearch-%{+YYYY.MM}"
 28             document_type => "mplatform_events"
 29             host => ["10.9..:9300","10.9..:9300"]
 30             protocol => transport
 31             cluster => DES
 32             manage_template => false
 33         }
 34     }
 35 }

and document indexed in elasticsearch like...

{
  "_index": "logstash-mpsearch-2015.11",
  "_type": "mplatform_events",
  "_id": "AVDz_uFgcMJ5o2Sbx4gn",
  "_score": null,
  "_source": {
    .....
    "host": "xv201.system.com",
    "path": "/home1/.logstash/json_input/mplatform.search.20151110",
    "type": "mpjson"
  },
  "fields": {
    "eventTime": [
      1447206964000
    ],
    "@timestamp": [
      1447202577457
    ]
  },
  "sort": [
    1447202577457
  ]
   }

with this config I wonder

  1. what happen if "file" have json line which contains field named "path","type"
    logstash conf value win?
  2. how can I prevent "type","path" indexed in elasticsearch document?

thanks.

2

Yes, the type that LS uses will be the one you defined in the input.

You cannot remove the ES _type, but you could use a mutate + remove to get rid of the path value.

1 Like
3
  1. what happen if "file" have json line which contains field named "path","type"
    logstash conf value win?

The values from the input file prevail, at least with Logstash 1.5.x.

$ cat test.json 
{"path": "path value from file", "type": "type value from file"}
$ cat test.config 
input {
  file {
    path => "/tmp/trash.66Vi/test.json"
    codec => "json"
    start_position => "beginning"
    type => "type value from config"
  }
}
output { stdout { codec => rubydebug } }
$ /opt/logstash/bin/logstash -f test.config
Logstash startup completed
{
          "path" => "path value from file",
          "type" => "type value from file",
      "@version" => "1",
    "@timestamp" => "2015-11-11T06:41:02.779Z",
          "host" => "lnxolofon"
}
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Logstash shutdown completed
  1. how can I prevent "type","path" indexed in elasticsearch document?

Delete the fields with a mutate filter.

1 Like
4

Thank you for the response.

btw, some line of file include 'type' attribute, and some does not.(same to the 'path' attribute)
so, I cannot use type for route.

but I want to send all the message from specific 'input>file' to specific 'output>something'.

   1 input {
   2   file {
   3     path => "/home1/irteam/tmp/json"
   4     codec => "json"
   5     start_position => "beginning"
   6     add_field => ["my_type", "xx_conf_type"]
   7   }
   8 }
   9 output {
  10     if [my_type] =="xx_conf_type" {
  11         filter {
  12             mutate {
  13                 remove_field => ["my_type"]
  14             }
  15         }
  16         stdout {
  17             codec => rubydebug
  18         }
  19     }
  20 }
  
  Error: Expected one of #, => at line 12, column 11 (byte 223) after output {
          if [my_type] =="xx_conf_type" {
                  filter {
                          mutate

how can I remove field in IF block?

5

I suspect Logstash is complaining about the lack of a space after ==. Also, the filter block is on the top level, together with input and output.

input {
  ...
}
filter {
  ...
}
output {
  ...
}
6

It complains not about for space after ==.

I've saw filter is top level.
then how can I remove field after output>if block??

field cannot be removed before output>if, because it should be used to "if" statement condition.

input {
   file {
     path => "/home1/irteam/tmp/json"
     codec => "json"
     start_position => "beginning"
     add_field => ["my_type", "xx_conf_type"]
   }
}
output {
    if [my_type] == "xx_conf_type" {
        // I want remove "my_type" field here..
        stdout {
            codec => rubydebug
        }
    }

}

sorry for the verbose question.

7

Oh. Add the field as a subfield of @metadata. Those fields are ignored by (nearly all) outputs. In other words, change

add_field => ["my_type", "xx_conf_type"]

to

add_field => ["[@metadata][my_type]", "xx_conf_type"]

and make the corresponding change in the conditional.