How do i get elasticsearch logs into elasticsearch?

Elastic Stack Logstash
1

Completely new to ELK, and very confused right now, and in the asking idiotic questions phase, so bear with me...

I've installed ELK on linux and the first thing I wanted to try was getting some logs into elastic and then querying them with Kibana. So I thought, I'll use the elasticsearch logs.

In order to get something to play with, I ran this:

PUT /settings
{
    "index.search.slowlog.threshold.query.warn": "10s",
    "index.search.slowlog.threshold.query.info": "5s",
    "index.search.slowlog.threshold.query.debug": "2s",
    "index.search.slowlog.threshold.query.trace": "0ms",
    "index.search.slowlog.threshold.fetch.warn": "1s",
    "index.search.slowlog.threshold.fetch.info": "800ms",
    "index.search.slowlog.threshold.fetch.debug": "500ms",
    "index.search.slowlog.threshold.fetch.trace": "0ms",
    "index.search.slowlog.level": "trace"
}

That gave me plenty of lines of output in /var/log/elasticsearch/elasticsearch_index_search_slowlog.log to work with.

So, I added a /etc/logstash/conf.d/elasticlogs.conf like this:

#elasticlogs.conf

input {
  file {
    path => "/var/log/elasticsearch/*.log"
    type => "elasticsearch"
    start_position => "beginning"
    codec => multiline {
      pattern => "^\["
      negate => true
      what => "previous"
    }
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "elasticsearch"
  }
}

Then I went to Kibana, and on the front page it says this:

"You only have a single index. You can create an index pattern to match it."

The index that it shows is NOT the index named "elasticsearch" that I want logstash to log to, but rather one I was testing with earlier.

Why doesn't logstash tell me anything is wrong in its logs?

I'm guessing? this must be a pretty common setup/standard procedure, as you'd want to be able to query elasticsearch/logstash's own logs in production for slow queries/warnings/errors, but my googling didn't turn up anything with a full guide as to how to set it up. Anyone got a link to a blog or anything for setting that up?

EDIT: I fixed the problem with nothing appearing in the index - the reason it was not working is because the default apt install does not give read permission to all users for the /var/log/elasticsearch folder, so it couldn't read the log files, so i did a sudo usermod -a -G elasticsearch logstash to resolve.

2

Why doesn't logstash tell me anything is wrong in its logs when it's not working?

How would Logstash know that things aren't working?

3

@magnusbaeck I'm guessing you are saying the file plugin cannot know whether not being able to read an input file is an error? i.e. it's impossible to know whether its a permissions issue or just that the file does not exist yet?

If it can somehow know it's a permissions error to get to the input file, then I would expect an error to be logged. If it cannot know, then fair enough.

4

Now that we know that the problem indeed was permission-related I'd say it's possible for Logstash to know that things weren't okay.

If the loglevel is increased Logstash logs what filename patterns expand to, and if they expand to zero files it might very well be a permission problem or a typo in the pattern.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.