Logstash adding extra comma to tags

Elastic Stack Logstash
1

I just started seeing this issue, I'm not quite sure what's up. I switched one of my environments to use the logstash+tcp functionality that recently got added to the logspout-logstash forwarder and noticed that an extra comma is being added to my custom tags. If the log doesn't hit a filter, the comma doesn't get added, and the tag column just stays blank like it should.

Here's an example of what the tags column looks like:

I've tried restarting the logspout-logstash forwarders, reverting back to the original configuration, and restarting logstash completely but the problem still persists.

Has this ever happened to anybody else? I'm not sure where to start troubleshooting this one.

2

What does the actual document in ES look like? Check the JSON tab in Kibana.

3

Here's an example of what the tags look like in the JSON view:

"tags": [
      "",
      "_grokparsefailure"
],

And another when the tag is empty:

"tags": [
      ""
],
4

Okay, so something's inserting empty tags. What does your configuration look like?

5

I looked through the config and couldn't find anything obvious.

The config is a bit of a bear so I will just link to the gist.

6

Indeed, nothing obvious. You could of course just remove the empty tag at the end.

7

Do you have a good example of how to do that (remove empty tags)?

8

Try the remove_tag option of any filter (or a standalone mutate filter).

9

I'll give it a shot. Thanks!

10

For anybody else, the following pattern works.

filter
...

mutate {
    remove_tag => [ "tags", ""]
}

...