# Field name duplicate with file conf and json doc **URL:** \ **Category:** Logstash\ **Created:** [November 11, 2015, 2:08am UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295 "2015-11-11T02:08:46Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![no\_jihun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/no_jihun/32/64028_2.png) [@no\_jihun](https://discuss.elastic.co/u/no_jihun)\ **Post date:** [November 11, 2015, 2:08am UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295/1 "2015-11-11T02:08:46Z")
Hello. I'm new to logstash and using logstash 1.5.2, elasticsearch 1.7.x And have logstash configuration like... ``` 1 input { 2 file { 3 path => "/home1/.logstash/json_input/deco.json.*" 4 codec => json 5 start_position => beginning 6 type => "decojson" 7 } 8 file { 9 path => "/home1/.logstash/json_input/mplatform.*" 10 codec => json 11 start_position => beginning 12 type => "mpjson" 13 } 14 } 15 output{ 16 if [type] == "decojson" { 17 elasticsearch { 18 index => "logstash -%{+YYYY.MM.dd}" 19 document_type => "deco_events" 20 host => ["10.9..:9300","10.9..:9300"] 21 protocol => transport 22 cluster => DES 23 manage_template => false 24 } 25 }else if [type] == "mpjson" { 26 elasticsearch { 27 index => "logstash-mpsearch-%{+YYYY.MM}" 28 document_type => "mplatform_events" 29 host => ["10.9..:9300","10.9..:9300"] 30 protocol => transport 31 cluster => DES 32 manage_template => false 33 } 34 } 35 } ``` and document indexed in elasticsearch like... ``` { "_index": "logstash-mpsearch-2015.11", "_type": "mplatform_events", "_id": "AVDz_uFgcMJ5o2Sbx4gn", "_score": null, "_source": { ..... "host": "xv201.system.com", "path": "/home1/.logstash/json_input/mplatform.search.20151110", "type": "mpjson" }, "fields": { "eventTime": [ 1447206964000 ], "@timestamp": [ 1447202577457 ] }, "sort": [ 1447202577457 ] } ``` with this config I wonder 1. what happen if "file" have json line which contains field named "path","type" logstash conf value win? 2. how can I prevent "type","path" indexed in elasticsearch document? thanks. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 11, 2015, 6:41am UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295/2 "2015-11-11T06:41:45Z")
> [@no\_jihun](#): > > 1. what happen if "file" have json line which contains field named "path","type" logstash conf value win? Yes, the type that LS uses will be the one you defined in the input. > [@no\_jihun](#): > > 1. how can I prevent "type","path" indexed in elasticsearch document? You cannot remove the ES `_type`, but you could use a mutate + remove to get rid of the path value. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 11, 2015, 6:42am UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295/3 "2015-11-11T06:42:22Z")
> 1. what happen if "file" have json line which contains field named "path","type" > logstash conf value win? The values from the input file prevail, at least with Logstash 1.5.x. ``` $ cat test.json {"path": "path value from file", "type": "type value from file"} $ cat test.config input { file { path => "/tmp/trash.66Vi/test.json" codec => "json" start_position => "beginning" type => "type value from config" } } output { stdout { codec => rubydebug } } $ /opt/logstash/bin/logstash -f test.config Logstash startup completed { "path" => "path value from file", "type" => "type value from file", "@version" => "1", "@timestamp" => "2015-11-11T06:41:02.779Z", "host" => "lnxolofon" } ^CSIGINT received. Shutting down the pipeline. {:level=>:warn} Logstash shutdown completed ``` > 1. how can I prevent "type","path" indexed in elasticsearch document? Delete the fields with a mutate filter. ---
**Author:** ![no\_jihun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/no_jihun/32/64028_2.png) [@no\_jihun](https://discuss.elastic.co/u/no_jihun)\ **Post date:** [November 11, 2015, 8:12am UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295/4 "2015-11-11T08:12:05Z")
Thank you for the response. btw, some line of file include 'type' attribute, and some does not.(same to the 'path' attribute) so, I cannot use type for route. but I want to send all the message from specific 'input\>file' to specific 'output\>something'. ``` 1 input { 2 file { 3 path => "/home1/irteam/tmp/json" 4 codec => "json" 5 start_position => "beginning" 6 add_field => ["my_type", "xx_conf_type"] 7 } 8 } 9 output { 10 if [my_type] =="xx_conf_type" { 11 filter { 12 mutate { 13 remove_field => ["my_type"] 14 } 15 } 16 stdout { 17 codec => rubydebug 18 } 19 } 20 } Error: Expected one of #, => at line 12, column 11 (byte 223) after output { if [my_type] =="xx_conf_type" { filter { mutate ``` how can I remove field in IF block? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 11, 2015, 12:02pm UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295/5 "2015-11-11T12:02:04Z")
I suspect Logstash is complaining about the lack of a space after `==`. Also, the filter block is on the top level, together with input and output. ``` input { ... } filter { ... } output { ... } ``` ---
**Author:** ![no\_jihun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/no_jihun/32/64028_2.png) [@no\_jihun](https://discuss.elastic.co/u/no_jihun)\ **Post date:** [November 11, 2015, 12:52pm UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295/6 "2015-11-11T12:52:14Z")
It complains not about for space after ==. I've saw filter is top level. then how can I remove field after output\>if block?? field cannot be removed before output\>if, because it should be used to "if" statement condition. ``` input { file { path => "/home1/irteam/tmp/json" codec => "json" start_position => "beginning" add_field => ["my_type", "xx_conf_type"] } } output { if [my_type] == "xx_conf_type" { // I want remove "my_type" field here.. stdout { codec => rubydebug } } ``` } sorry for the verbose question. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 11, 2015, 1:08pm UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295/7 "2015-11-11T13:08:23Z")
Oh. Add the field as a subfield of `@metadata`. Those fields are ignored by (nearly all) outputs. In other words, change ``` add_field => ["my_type", "xx_conf_type"] ``` to ``` add_field => ["[@metadata][my_type]", "xx_conf_type"] ``` and make the corresponding change in the conditional. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:23am UTC](https://discuss.elastic.co/t/field-name-duplicate-with-file-conf-and-json-doc/34295/8 "2017-07-06T05:23:10Z")