Trouble with using LS to filter based on tag

Greetings all-

I am trying to route different log types to different indices. Since v6 removed 'type', I have Filebeat adding tags for the various log sources. One of the log sources is Cowrie and is the one I will use for this example.

I have FB logging back to a LS server. This LS server logs back to another LS server using Lumberjack. Then that LS server writes to ES. [ I am basically logging to geographically placed LS servers which then feed back into a centralized LS server; doing this for performance reasons and also reducing how exposed the ES interface is ].

On each LS server, I have output defined to the destination log source as well as a local file so I can grep the file to ensure the needed filtering occurred. Due to this, I am able to manually validate that the "cowrie" tag is fully present at each step of the way.

The issue is at the last hop where LS writes to ES. My output configuration is below. Even though the "cowrie" tag is present, it seems to be matching the "else" statement and writing the cowrie logs to the logstash index. Any help on what I may have done wrong would be greatly appreciated. :slight_smile:

Edit: for clarification - I am using v6 of ES and LS on all assets.

https://gist.github.com/mackwage/e826d2d5dcb4612c03d278f54ec91208

I finally figured it out. :slight_smile:

So I used Filebeat to add a tag to these logs via the fields option as shown on several posts. HOWEVER, this doesn't populate the native "tags" field, it actually creates its own called "field.tags". So the rule was not matching because it was checking the native tags instead of the newly created custom one.

I opted to just remove the custom field and have LS rules match on "source" for now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.