Configure metadata sent to logstash

[jmegin (James Megin)]

I am writing a custom beat that will be sending messages to logstash. I would like to be able to add additional fields to the @metadata object received by logstash. Is this possible? I am currently using libbeat v 5.5.

I followed the developer guide to create a new beat, and attempted adding the following code to what was generated in order to try to set additional fields in the @metadata object:

event := common.MapStr{
	"@timestamp": common.Time(time.Now()),
	"type":       b.Name,
	"counter":    counter,
}
eventMeta := common.MapStr{
	"index": "test",
	"otherData": "1234",
}
metaOp := publisher.Metadata(eventMeta)
bt.client.PublishEvent(event, metaOp)

Unfortunately, this did not work, and the @metadata object in logstash is still
{"beat":"testbeat","type":"testbeat"}

Any help would be greatly appreciated.

Thanks,
James

[steffens (Steffen Siering)]

In 5.x branch this is currently not possible. You can add another field to your event and do some event manipulation in logstash to get a similar result.

In master and 6.x branch the publisher pipeline has been refactored and the event type has become:

type Event struct {
  Timestamp time.Time
  Meta common.MapStr
  Fields common.MapStr
}

Whatever you put into Meta will be send/used by the outputs. e.g. adding event.Meta["pipeline"] selects the ingest node pipeline if the Elasticsearch output is used. All other outputs (file, console, kafka, redis, logstash) will put fields in Meta into the @metadata field when encoding to json.

Please note, 6.0 just entered the beta-phase and we haven't had time to update the developer docs, as there might be some more internal changes/cleanups before 6.0 GA, potentially affecting the docs.