Logstash not sending data to ES

Elastic Stack Logstash
1

I have a logstash config which reads nginx logs and sends data to elasticsearch.
The log file contains two endpoints viz /index and /info. Logstash parses the /index endpoint and sends the data to ES but it doesn't do the same with /info endpoint.

The issue is that when I run logstash in foreground using /opt/logstash/bin/logstash to debug the problem, it is able to parse both the endpoints and send data to ES, whereas as a daemon it does so only with the /index endpoint.

I don't understand what I'm doing wrong here, can anybody please point me in the right direction?

2

Please show your Logstash configuration files and relevant contents from the Logstash log.

3

This is my logstash config file -

input {
    file {
        start_position => "beginning"
        path => '/var/log/nginx/tracker.log'
        codec => plain {
            charset => "ISO-8859-1"
        }
        type => 'tracker'
    }
}
filter {
    if [type] == 'tracker' {
        urldecode {
        }
        grok {
            match => [
                "message", '%{IP:clientip} %{GREEDYDATA} \[%{HTTPDATE:timestamp}\] "%{GREEDYDATA:request} HTTP%{GREEDYDATA}" %{INT:status} %{INT:body_bytes_sent} %{GREEDYDATA} \* "%{GREEDYDATA:http_post_data}"',
                "message", '%{GREEDYDATA:invalid_message}'
            ]
        }
        grok {
            match => [
                "request", '%{WORD:verb} /%{GREEDYDATA:api}\?%{GREEDYDATA:query_string}',
                "request", '%{WORD:verb} /%{GREEDYDATA:api}'
            ]
        }
        mutate {
            gsub => [
              "query_string", "timestamp", "et"
            ]
        }
        kv {
            source => "query_string"
            field_split => "&?"
        }
        if [mcc1] {
            ruby {
                code => "
                    mnnc = []
                    mcc1 = event['mcc1'].gsub(/(^[0]*)/, '')
                    mnc1 = event['mnc1'].gsub(/(^[0]*)/, '')
                    mcc2 = event['mcc2'].gsub(/(^[0]*)/, '')
                    mnc2 = event['mnc2'].gsub(/(^[0]*)/, '')
                    if mcc1 == '' and mnc1 == ''
                        mnnc << '0'
                    else
                        mnnc << mcc1+mnc1
                    end
                    if mcc2 == '' and mnc2 == ''
                        mnnc << '0'
                    else
                        mnnc << mcc2+mnc2
                    end
                    event['mnnc'] = mnnc
                "
            }
        }

        mutate {
            lowercase => [ 'uid', 'keyword', 'code', 'location' ]
        }
        mutate {
            remove_field => [ 'http_post_data', 'blah', 'query_string', 'http_referer', 'http_user_agent', 'ref',
                        'body_bytes_sent', 'http_forward', 'path', 'clientip', 'host', 'status', 'message', 'mcc', 'mnc',
                        'verb', 'lt', 'url' ]
        }

    }
}
output {
    if [type] == 'tracker' {
        elasticsearch {
            user => "uname"
            password => "pass"
            hosts => ["127.0.0.1:9200"]
            index => "my_index"
            document_id => "%{uid}"
            action => "create"
        }
        stdout { codec => rubydebug }
    }
}

And this is an example log file -

202.87.33.164 - - [01/Jun/2017:15:14:20 +0000] "GET /info?uid=d1&keyword=x1&code=y1&location=z1&mcc1=0&mnc1=0&mcc2=4&mnc2=4&timestamp=1496330139314 HTTP/1.1" 204 118 "-" "Java/1.7.0_05" * "-"
202.87.33.164 - - [02/Jun/2017:05:22:00 +0000] "GET /info?uid=d2&keyword=x2&code=y2&location=z2&mcc1=0&mnc1=0&mcc2=4&mnc2=4&timestamp=1496381000548 HTTP/1.1" 204 118 "-" "Java/1.7.0_05" * "-"
1.23.165.186 - - [02/Jun/2017:05:23:45 +0000] "GET /index?uid=d3&keyword=x3&code=y3&location=z3&date=02-06-2017&time=10%3A55%3A02 HTTP/1.1" 200 249 "-" "Dalvik/2.1.0 (Linux; U; Android 7.0; K9 Kavach 4G Build/NRD90M)" * "-"

The logstash.log, logstash.err and logstash.stdout files remain empty when I run logstash as a daemon.

4

How do you know Logstash is even reading from the file? It's probably tailing it, waiting for more lines to be added to it. Also, does the Logstash user have permissions to read the input file (and all directories leading up to it)?

5

I've set the start_position to beginning to ensure that logstash doesn't tail the file. The system is live, therefore new lines are added to the log file automatically, and the logstash parses /index endpoint and send data to Elasticsearch but not the /info endpoint.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.