Hi,
I am feeding log data through Logstash through Elasticsearch. Without making any change to the data itself or Logstash Config, I am getting a mapping conflict on a new index I have created yesterday (because of the same problem).
Logstash Filter (the mapping change happens on the field "KNX-Wert")
filter {
if "knxmonitor" in [tags] {
csv {
columns => [ "eventtime", "ms", "Typ", "PA", "GA_KO", "KO-ID", "KNX-Name", "KNX-Wert" ]
convert => { "KNX-Wert" => "float" }
}
date {
locale => "en"
match => [ 'eventtime' , 'yyyy-MM-dd HH:mm:ss' ]
}
mutate {
remove_field => [ "message", "offset", "eventtime", "ms" ]
}
}
}
Not all messages contain valid float data in KNX-Message, but I am not interested in these messages and wouldn't mind them being discarded at worst. Anyway, they are giving me errors in the elasticsearch.log:
[2017-03-22T00:00:04,329][DEBUG][o.e.a.b.TransportShardBulkAction] [LAw6q-9] [knx2-2017.03.21][1] failed to execute bulk item (index) index {[knx2-2017.03.21][log][AVrzGUZoG7YEgFcI0c6J], source[{"KNX-Wert":"05:45:22","KNX-Name...
rg.elasticsearch.index.mapper.MapperParsingException: failed to parse [KNX-Wert]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:298) ~[elasticsearch-5.2.2.jar:5.2.2]
...
Caused by: java.lang.NumberFormatException: For input string: "05:45:22"
at sun.misc.FloatingDecimal.readJavaFormatString(FloatingDecimal.java:2043) ~[?:?]
at sun.misc.FloatingDecimal.parseFloat(FloatingDecimal.java:122) ~[?:?]
at java.lang.Float.parseFloat(Float.java:451) ~[?:1.8.0_121]
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.floatValue(AbstractXContentParser.java:174) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.parse(NumberFieldMapper.java:278) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.index.mapper.NumberFieldMapper$NumberType$2.parse(NumberFieldMapper.java:264) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.index.mapper.NumberFieldMapper.parseCreateField(NumberFieldMapper.java:1018) ~[elasticsearch-5.2.2.jar:5.2.2]
at org.elasticsearch.index.mapper.FieldMapper.parse(FieldMapper.java:287) ~[elasticsearch-5.2.2.jar:5.2.2]
...
Is this the reason? If so, how can I tell elasticsearch to drop these messages instead of forcing them into the fields?
Or, how can I avoid field mappings being changed automatically like this?
thank you in advance