How to Seperate docs in Logstash to different ES index

[yasvanth]

Hi All,

I am inserting data to logstash(collector) via nxlog(sender) to single port. The logs are already parsed on nxlog side from .xml to json , these parsed doc contains logs from different application such as apache, tomcat etc. I have field named as Log_type to apache or tomcat.

So I want to set a statement to check if the Log_type = apache send to Index-Apache , If Log_type = tomcat send to Index_tomcat.

My Logstash conf is:

input {
tcp {
codec => json_lines{ charset => CP1252 }
port => 9514
tags =>["tcpjson"]
}
}

output {
if "Log_Type" == [apache_Logs} {
elasticsearch {
hosts => ["localhost:9200"]
action => 'index'
index => 'apache-%{+YYYY.MM.dd}'
}
if "Log_Type" == [tomcat_Logs} {
elasticsearch {
hosts => ["tomcat:9200"]
action => 'index'
index => 'tomcat-%{+YYYY.MM.dd}'
}
}

I am not using filter statement since the logs are parsed with field names. How could I look into data and enable the condition statements.

Thanks,
Yash

[paz]

Hey,

Your syntax is actually malformed. Does logstash start with it?
It should be

output {
    if [Log_Type] == "apache_Logs" {
        elasticsearch {
            hosts => ["localhost:9200"]
            action => 'index'
            index => 'apache-%{+YYYY.MM.dd}'
    }
    if [Log_Type] == "tomcat_Logs" {
        elasticsearch {
            hosts => ["tomcat:9200"]
            action => 'index'
            index => 'tomcat-%{+YYYY.MM.dd}'
    }
}

[yasvanth]

Thanks for your quick reply. I tried this but no success But I will give a try again. Should I need to inspect the doc using filter section or just in output section?

[paz]

If the event is already parsed and structured, there is no need for a filter section.
Based what you said above,

@yasvanth

...I have field named as Log_type to apache or tomcat.

There should be fields inside the event JSON document like so:

{
    #other event fields
    "Log_type": "apache_Logs"
}

Is that the case? If so, it should work as-is.

[yasvanth]

@paz thanks a million. It is small typo in the field in nxlog, fixed and now working fine. Thanks for your help.