How to solve field in mapping but already used in other types

Elastic Stack Elasticsearch
1

I also run topbeat, packbeat, transfer to logstash, accept the data on the ES side, reported the following error

[2016-03-01 02:33:49,810][DEBUG][action.bulk ] [node-1] [logstash-2016.02.29][4] failed to execute bulk item (index) index {[logstash-2016.02.29][mysql][AVMuTs3DYEsGBQRKvfBP], source[{"@timestamp":"2016-02-29T18:33:48.342Z","beat":{"hostname":"server1","name":"server1"},"bytes_in":51,"bytes_out":51,"client_ip":"192.168.51.105","client_port":57663,"client_proc":"","client_server":"","count":1,"direction":"in","ip":"192.168.4.12","method":"INSERT","mysql":{"affected_rows":0,"error_code":1054,"error_message":"42S22: Unknown column 'aaaaa' in 'field list'","insert_id":0,"iserror":true,"num_fields":0,"num_rows":0},"path":"","port":3306,"proc":"","query":"insert into test(id,title) values(aaaaa,'987')","responsetime":0,"server":"","status":"Error","type":"mysql","@version":"1","host":"server1","tags":["beats_input_raw_event"]}]}
java.lang.IllegalArgumentException: Field [proc] is defined as a field in mapping [mysql] but this name is already used for an object in other types
at org.elasticsearch.index.mapper.MapperService.checkMappersCompatibility(MapperService.java:398)
at org.elasticsearch.index.mapper.MapperService.checkMappersCompatibility(MapperService.java:411)

Topbeat inside the proc (proc.ppid etc) is a digital type, but the MySQL proc is a string type?

Thanks. :slightly_smiling:

2

The best solution is to store the data in different indices so this conflict doesn't happen!

3

Only one es, topbeat ---> logstash1, packetbeat---> logstash2, while logstash is output to the same es, such data will be merged into logstash-*.
It will report the error above, unless the creation of multiple es
java.lang.IllegalArgumentException: Field [proc] is defined as a field in mapping [mysql] but this name is already used for an object in other types

thanks.

4

If you have multiple inputs, ie one per beat, then use tags and conditional outputs for each one.

5

packetbeat.yml configuration is as follows:

logstash:
hosts: ["192.168.4.13:5044"]

shipper:
tags: ["packetbeat","192.168.4.13"]

topbeat.yml configuration is as follows:

logstash:
hosts: ["192.168.4.13:5045"]

shipper:
tags: ["topbeat","192.168.4.12"]

input {
beats {
port => 5044 logstash1--->5044 , logstash2--->5045
}
}

output {
elasticsearch {
hosts => ["server1:9200"]
}
stdout{
codec => rubydebug
}
}

At the same time to run TopBeat and PacketBeat,
the es raise exception:
java.lang.IllegalArgumentException: Field [proc] is defined as a field in mapping [mysql] but this name is already used for an object in other types

Merge the mapping field by manual, the error is still reported

Thanks.

6

Add the following two items in beats.conf (logstash configuration) to soluted
index => "logstash-%{type}-%{+YYYY.MM.dd}"
document_type => "%{type}"

Near by elasticsearch node

Thanks everybody.

1 Like
7

Using 5.4 with the searchkick gem.

My solution was actually a misnaming of a mapping. My mapping defined a locations_coordinates but I was indexing by location_coordinates even though it was a misnaming this exception was thrown.

8

Hi, the index and document type lines, should they go under the filters section? Thank you.