Not getting dynamic index in elasticsearch using logstash

Elastic Stack Logstash
1

Hello,

I am facing issues when trying to create dynamic indexes using logstash. I have different log events coming from a single file,so i wanted to make them identifiable in the filter section using the identifier present in the log event. Also as per the identifier, I want to create index.
i tried to make it possible with below given configuration but logstash stops and no error message is given out. I am stuck at this point. Could somebody please help me with this.
Below is my configuration

input
{
file
{
path => ["/var/log/bulk/sample_audit.json"]
sincedb_path => "/dev/null"
start_position => "beginning"
}
}

filter {
json { source => message }
ruby {
code => "event['logtype'] = event['preinfo']['task']['log']jobname"
}
split {
field => "preinfo[task][log][logmessage]"
target => "output"
}
}

output
{
elasticsearch {
hosts => ["10.201.181.204:9200"]
sniffing => false
manage_template => false
index => "%{logtype}-%{+YYYY.MM.dd}"
}
}

Could some one please help me out with this??

Thanks
Nitin

2

Can you explain this more please, what do you mean, what do you see, can you post logs or any output?

3

Do all the events have the field logtype defined? If this is not the case I suspect the index name could end up being invalid;id, but that should show up in the logs. have you tried outputting them to stdout using a rubydebug codec?

4

Thanks @Christian_Dahlqvist @warkolm i have changed my configuration. Below is my configuration

input
{
    file
    {
        path => ["/var/log/bulk/sample_audit.json"]
        sincedb_path => "/dev/null"
        start_position => "beginning"
    }
}

filter {
json {	source => message }
if ["preinfo[task]jobname"] == "sudoers:linux"{
split {
      field => "preinfo[task][log][logmessage]"
      target => "output"
    }
}
}
output 
{
elasticsearch {
    hosts => ["10.201.181.204:9200"]
    sniffing => false
    manage_template => false
    index => "%{[preinfo[task]jobname]-%{+YYYY.MM.dd}"
  }
}

With this config i can create the desired index as per the jobname but now the problem is I cannot split nested arrays present in the log event and intresting part is i can get my desired result after removing if conditions.

What i am doing wrong?
Is that the right way to use conditions in logstash??

5

can someone please help me on this ??

6 
if ["preinfo[task]jobname"] == "sudoers:linux"{

Do this instead:

if [preinfo][task][jobname] == "sudoers:linux" {

index => "%{[preinfo[task]jobname]-%{+YYYY.MM.dd}"

Do this instead:

 index => "%{[preinfo][task][jobname]}-%{+YYYY.MM.dd}"

Use a stdout { codec => rubydebug } output until you've verified that your events look like you expect them to. Only then is it useful to enable an elasticsearch output.

7

Thanks @magnusbaeck ...i made the changes..but now this time its not showing anything. not even in the logs. I am using logstash 2.4 so i thought that might be a bug in 2.4 but its not wotking in 5.2 also. Anything else i need to do ??

8

It sounds quite unlikely that the changes I suggested would choke everything completely. Did you switch to a stdout output as I suggested? Have you increased the Logstash log level to get more clues in the log output?

9

@magnusbaeck Thanks.. i messed up the stdout part. I made it right and now it works like a charm.
Thanks a lot @magnusbaeck again.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.