Conflict between Filebeat and Packetbeat fields

Elastic Stack Beats
1

Filebeat logs have a string field called "source" which is the filepath to the file that the filebeat log is coming from.

Packetbeat logs of type "flow" have an object field called"source" which has the properties ip, port, stats, and mac.

I ran into a situation where both of these beats were writing to the same elasticsearch instance. When I tried to look up logs in Kibana, I couldn't find filebeat logs, but found log entries with an "indexed-failed-reason": that reads as follows:

"type":"illegal_argument_exception","reason":"[source] is defined as a field in mapping [log] but this name is already used for an object in other types"

I feel like the default fields for all the beats shouldn't conflict with each other. Would it be possible to change either of the field names here?

2

By default packetbeat data goes to a daily index following the pattern of packetbeat-YYYY.MM.DD and filebeat goes to filebeat-YYYY.MM.DD. There should never be mapping conflicts between data stored in two different indices. Did you customize the index patterns in your config?

3

I am using the ELK stack through a third party that provides it as a service. (So they manage logstash, elasticsearch, kibana, etc, and I have beats installed on my machines that send the logs to their servers)

Your hunch is correct, their setup uses a single index per account, which is why this conflict occurred. Thank you for this insight!