Beats input w/ json codec

So I know events coming from filebeat into logstash have a "source" field related to the full path of the file. The problem is my json file that I am sending will occasionally also have a 'source' field, which seems to behave in inconsistent ways. Previously (version 6.1 I think) I could rename 'source' field from JSON to keep that (dropping filebeat file path 'source') but now in 6.2 I can only get the filebeat 'source' field and cannot access JSON 'source' field. Any ideas?

For reference, I'm ingesting JSON bro logs (files.log)

I'll just add that for some fields with collisions I get behavior that makes sense (I keep the field values from JSON input) but other times I keep the filebeat fields instead. For example, when host is present in JSON and from filebeat I get to keep JSON value but when source is present in both I get to keep the filebeat value.

Fixed this problem by writing a bro script to add an additional field in files.log that is a copy of 'source' field. Source below if anyone's interested.

export {
	redef record Files::Info += {
		file_source: string &optional &log;
event file_state_remove(f: fa_file) &priority=3
  # add 'file_source' if 'source' exists
  if ( f$info?$source )
  	f$info$file_source = f$info$source;

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.