Output with condition

Elastic Stack Logstash
1

HI.

I have two businesses that must be separated in two different indexes.

I'm trying to separate them on the output:

output {

if [region] == "au" {
        elasticsearch {
        hosts => ["localhost:9200"]
        # Weekly index (for pruning)
        index => "au-log-index-%{+YYYY.'w'ww}"
        }
    }

else {
        elasticsearch {
        hosts => ["localhost:9200"]
        # Weekly index (for pruning)
        index => "nz-log-index-%{+YYYY.'w'ww}"
        }
    }

    stdout { codec => rubydebug }
}

I'm adding the region field like this; example.conf:

if [type] == "au_uat_apache_access_log" {
        mutate {
            replace => { 'host' => 'uatweb.datacentre.example.com.au' }
            add_field => { 'environment' => 'uat'
                           'service' => 'apache_access'
                           'region' => 'au'
            }
        }
        grok {
            match => {
                "message" => "%{IPORHOST:clientip}%{SPACE}\[%{HTTPDATE:timestamp}\]%{SPACE}%{NUMBER:port}%{SPACE}%{WORD:method}%{SPACE}%{URIPATHPARAM:request_uri}%{SPACE}%{NOTSPACE}%{SPACE}%{NUMBER:status_code}%{SPACE}%{NOTSPACE:bytes_delivered}%{SPACE}%{NUMBER:duration%}%{SPACE}(?:%{URI:referrer}|.*)%{SPACE}%{QS:agent}%{SPACE}%{GREEDYDATA:general_data}"
            }
        }

        date {
            match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
            target => "@timestamp"
        }
    }

However from Kibana I am only able to see the nz* index, which correspond to the else statement.

What am I doing wrong? Thanks in advance.

2

Is this condition ever true? Where is this type set? Can you show a document that had been indexed into the wrong index?

3

Yes, it does exist

file {
        type => "au_uat_apache_access_log"
        start_position => "beginning"
        path => "/mnt/logs/uatweb/access_log"
    }

Can you show a document that had been indexed into the wrong index?

The problem is that only one index is shown to me; I can only see the index below in Kibana:

else {
        elasticsearch {
        hosts => ["localhost:9200"]
        # Weekly index (for pruning)
        index => "nz-log-index-%{+YYYY.'w'ww}"
        }
    }
4

In Kibana, expand a document that is in the wrong index, go to the JSON tab, and post a copy of what is on that tab for a single document. Make sure you include the type and region fields.

5

I can't... I'm getting an error on Kibana so no data is being shown (when creating the index pattern):

"f0aa5110-000d-11ea-98c1-51084078b0aa" is not a configured index pattern ID
Showing the default index pattern: "nz*" (a5034ae0-000e-11ea-98c1-51084078b0aa)

There is no data being displayed.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.