Splitting syslog/beat entries by host?

[almathden (Brian)]

Hey Team!

Due to the wildly different syslog formats (And my not understanding how to grok/mutate/etc properly yet) I'm trying to split my syslog inputs by IP address so there's a different index for each. It seems to mostly be working but I created a catchall entry at the end that is seeing data too sometimes.

Want to make sure there's not a "better" way....I vaguely remember someone telling me that 'host' wasn't a good idea to use, so I am wondering what the "correct" way is. Sorry, just a beginner trying to do things right rather than hack it together!

I'm doing that with beats too, as you can see in the config below. I'm using [host][name].

Syslog data seems to come in with [host] == IP Address.

Here's a trimmed/somewhat sanitized version of the current config. Am I making this too hard for myself?

Running 7.6.2 on CentOS7

https://pastebin.com/29jhvT1D

Thanks,
-Brian

[Badger]

@almathden

I vaguely remember someone telling me that 'host' wasn't a good idea to use

There is a potential problem. beats use [host][name], so that the [host] field is an object that contains other fields. However, several older logstash input (syslog and tcp included, I believe) set [host] to a string. logstash does not care about the difference, but elasticsearch does. In a single index a field has to be either a string or an object, it cannot be a string on some documents and an object on others. You will get 400 errors from elasticsearch with a mapping exception if you try that. That is within a single index. There is no problem having one index where it is a string and a second index where it is an object both in the same elasticsearch instance.

So there is no big issue with having different data in different indexes, unless it leads to a large number of small indexes, which is expensive.

[almathden (Brian)]

Beautiful, thank you @Badger !

If possible once I get the formatting under control I'll probably pull all the syslogs into a single spot, or at least break it down by category (our gateway will be the loudest and may need it's own index), but for now I'm just trying to get a handle on the data.