Syslogs sent to wrong index

[Roger_Westgren]

Hi!

I have a weird problem with my Logstash configuration. There are 2 .conf files in /etc/logstash/conf.d (beats.conf and syslog.conf).

All the Beats agents get sent to port 5043 (beats.conf), and that's working fine, but I can't find the index pattern to create the syslog-* index.
Instead I find this new index (which is the index in the beats.conf file):

So I got curious and created the index pattern "%{[@metadata][beat]}-*"
and when I check it in the Discover tab all the syslog logs are there.

Port 5140 is listening, both udp and tcp.

I can't figure out why this is happening and can't seem to find any one else with the same problem when I search the web. Can anyone help?

Many thanks!

[grumo35]

Hi,

You should create 3 separates files 01-input.conf where you list all your inputs
02-filter.conf where you can filter logs and one 03-output.conf.

As for the input file just add

if  [type] == "foo" {
 elasticsearch {

            hosts => "localhost:9200"
            index => "foologs"

        }
}

As i see the problem it's just that the outputs you are using aren't processed at the same time.

[Roger_Westgren]

Thank's @grumo35, I've created the 3 files you suggested but getting this error:

I haven't been working with ELK for very long so it would really help if you can detail the explanation/answer a little more.

These are the .conf files I made:

[grumo35]

Oh wow sorry i was exhausted yesterday i just put you in trouble, i was meaning to filter in types in the ouptut file not the input one.

Just remove the condition your input.conf should look like this :

 input{
     tcp { XXXX }
     udp { XXXX }
}

And the output file is where you sort events to be indexed :

output{
if  [type] == "foo" {
 elasticsearch {

            hosts => "localhost:9200"
            index => "foologs"

        }
}
}

I haven't been working with ELK for very long so it would really help if you can detail the explanation/answer a little more.

I just used separates files to make it more clear that logstash is processing files depending on their names and this is important to know since you were using multiple files but did not took care of their processing order for logstash.

If you wish to you could just use one big file but it makes it easier to configure with separates files.

Sorry for misleading you yesterday. Let me know if this solution works.

[Roger_Westgren]

It's working now, thanks a whole lot @grumo35!

[grumo35]

Thanks dont hesistate to mark as solved with answer for future members to look at.