How to append a JSON string to libbeat event?

Elastic Stack Beats
1

I'm new to both libbeat and golang so hope someone could give me some clues. Btw, I haven't got any ideas after search on this board or even google.

So basically I'm writing a Beat for my product. It receives JSON events from another source. Now when I'm trying to publish these JSON events to libbeat, I'm lost how I can convert a JSON string to MapStr.

My code is something like this:

	event := common.MapStr{
		"@timestamp"	: common.Time(time.Now()),
		"type"			: "mysource",
		"event": common.MapStr{
			[my json event string]
		},
	}
	bb.events.PublishEvent(event)

The [my json event string] could be something like:

{"object" : "file", "id" : 1234, "info" : [0, 2, 3]}

So what's the go function I should put in [my json event string] line? Thanks

2

well, there might be multiple options, depending on what you want to do. You can either try to leverage logstash our pull some tricks on json encoder/decoder.

From my point of view it would make sense to parse the json event, add your own fields and then publish the generated event. That is, if your input looks like:

{"object" : "file", "id" : 1234, "info" : [0, 2, 3]}

Then output looks like:

{"@timestamp": ..., "type": "mysource", "object" : "file", "id" : 1234, "info" : [0, 2, 3]}

You can easily achieve this (untested code) by parsing your event into MapStr and adding the required fields:

// input: line =  raw json object of type []byte (read from socket?)
var event common.MapStr //  = map[string]interface{}
if err := json.Unmarschal(line, &event); err != nil {
    ...
}
event["@timestamp"] = common.Time(time.Now())
event["type"] = "mysource"
bb.events.PublishEvent(event)

If you don't want to merge '@timestamp' and 'type' into the original input (might be better for performance due to not parsing input json?) you can try this (again untested):

// input: line =  raw json object of type []byte (read from socket?)
event := common.MapStr{
    "@timestamp": common.Time(time.Now()),
    "type": "mysource",
    "event": json.RawMessage(line)
}

Using RawMessage you assume 'line' already be valid json. When event is encoded, the input line is just copied into output string (as is, no validation).

You can also mix both (parse input json for validation, validating input is always a good idea!!!):

var event common.MapStr //  = map[string]interface{}
if err := json.Unmarschal(line, &event); err != nil {
    ... // uh, invalid input => handle error 
}

// TODO: some more validation on input?

// create output event
event = common.MapStr{
    "@timestamp": common.Time(time.Now()),
    "type": "mysource",
    // copy original raw string to reduce 'encoding' overhead
    "event": json.RawMessage(line)
}

Instead of parsing, maybe you can find some json validation library, so no data need to be allocated for validation.

3

Thanks for your suggestion. Finally I use json.Unmarshal to convert it to MapStr. json.RawMessage just returns the raw data which is not the text format of json.