Edit filebeat mappings

(Jim) #1

I'm streaming data into SolR and I need to produce a JSON field in the FileBeat payload called event_timestamp. Is there some way I can replicate the @timestamp field, or do I need to edit the FileBeat template? If so how?

(ruflin) #2

If you want to do it directly in filebeat this would required to change the source code. I recommend you to do these modifications in Logstash.

I actually wasn't even aware that Filebeat -> Solr works.

(Jim) #3

Thanks for your help! I'm using a plugin to stream from logstash to solr, but I think I might ditch solr for elasticsearch since it seems to be a better supported and more well-trodden path.

(ruflin) #4

Yes, Filebeat is specifically target for Logstash and Elasticsearch. Let me know you still need some help.

(Jim) #5

... So I've been using Filebeat but now I need add a bit more structure to my log messages, so I guess I'll use Logstash... But how can I extract a value from the message itself, for example, if my message looks like:

"message" : "reallyUsefulData : heres the information I want"

How can I produce a document that look like this:

"message" : "reallyUsefulData : heres the information I want"
"reallyUsefulData" : "heres the information I want"

Is this what grok is for?

(Steffen Siering) #6

yes, this what logstash with grok filters is used for

(Jim) #7

...awesome, thanks!

(system) #8