Problems with logstash codecs (json, rubydebug)

Elastic Stack Logstash
#1

hi,
we're using logstash 2.2.2
I'm trying to feed logstash with kibana logs, kibana log looks like:

{"type":"log","@timestamp":"2016-03-13T10:05:30+00:00","tags":["status","plugin:elasticsearch","info"],"pid":15856,"name":"plugin:elasticsearch","state":"green","message":"Status changed from yellow to green - Kibana index ready","prevState":"yellow","prevMsg":"Waiting for Elasticsearch"}

so its json

when I do like this:

input {
    file {
        type => "ELK_kibana"
        path => "/var/log/kibana/kibana.stdout"
    }
}

and output goes to elasticsearch, it works ok, the only thing its unformatted

ok, so I did:

input {
    file {
        type => "ELK_kibana"
        path => "/var/log/kibana/kibana.stdout"
        codec => "json"
    }
}

And logs never goes to output, and nothing is happening (but other logs are working ok)

the same issue is with rubydebug plugin:

input {
    file {
        type => "ELK_logstash"
        path => "/var/log/logstash/logstash.log"
        codec =>  "rubydebug"
    }
}

And nothing goes to output.

Am I missing something in my configurations?

#2

Can you provide the complete config? It's hard to say what is happening when all we can see is the input.

#3

Unless you delete the sincedb file /var/log/kibana/kibana.stdout won't be reprocessed since Logstash think it's already done with that file. If new data is added to Kibana's logfile that data should show up though.

#4

output {
elasticsearch {
hosts => ["127.0.0.1"]
}
}

#5

Like Magnus said, it's probably your sincedb.

#6

what you mean exactly?

but not just kibana logs is not working - logstash too

how it can be fixed?

#7

Logstash's file input is designed to continuously monitor log files and send newly added data but not resend old data. When you restart Logstash after changing the configuration Logstash won't reprocess the file again unless you delete Logstash's sincedb file which contains the current position in the log files it reads. The file input documentation explains how this works.

#8

yes, but new data goes into file after restart

so , when codecs are used data does not go to output - not to elasticsearch neither to stdout (which I tyde for tests)

#9

yes, but new data goes into file after restart

Yes, new data will always be processed as it arrives.

so , when codecs are used data does not go to output - not to elasticsearch neither to stdout (which I tyde for tests)

This is unrelated to codecs.

#10

what would you suggest to try to find out what is the problem?

#11

From the evidence I've seen Logstash works as expected. Please read the file input documentation and make sure you understand how sincedb works. If you increase Logstash's logging verbosity by starting it with --verbose it'll tell you more about what's going on with the sincedb files.

#12