Grok filter. How do I break up the message information?

[E_Marshall]

I created a filter to break apart our log files and am having the following issue. I'm not able to figure out how to save the parts of the "message" to their own field or tag or whatever you call it. I'm 3 days new to logstash and have had zero luck with finding someone here who knows it.

So for an example lets say this is your log line in a log file
2017-12-05 [user:edjm1971] msg:This is a message from the system.

And what you want to do is to get the value of the user and set that into some index mapping so you can search for all logs that were by that user. Also, you should see the information from the message in their own fields in Kibana.

My pipeline.conf file for logstash is like
grok {
match => {
"message" => "%{TIMESTAMP_ISO8601:timestamp} [sid:%{USERNAME:sid} msg:%{DATA:message}"
}
add_tag => [ "foo_tag", "some_user_value_from_sid_above" ]
}

Now when I run the logger to create logs data gets over to ES and I can see the data in KIBANA but I don't see foo_tag at all with the sid value.
How exactly do I use this to create the new tag that gets stored into ES so I can see the data I want from the message?

[magnusbaeck (Magnus Bäck)]

You don't need add_tag. In the grok expression itself you're asking for the components of the string to be extracted to the fields timestamp, sid, and message.

To overwrite the existing message field with the one extracted in this grok filter you'll have to use the grok filter's overwrite option.

[E_Marshall]

Thank you. I managed to get this all figured out and working. One thing though was that when I put some custom patterns in (which I had tested and were valid) into an external file in the pattern directory it was painfully slow. When I just opted to use a pre built GROK pattern that could also do what I wanted it was very fast.
Is it normal for external pattern files to run slowly?

[magnusbaeck (Magnus Bäck)]

The location of the patterns doesn't matter but the quality of them does. Frequent use of DATA and GREEDYDATA can have extreme effects on the execution time.