Elasticsearch Input Plugin - Index field string to match system date

Elastic Stack Logstash
1

Good morning/afternoon/evening/hi,

using logstash version 1.5.3 Sorry if this has been bought up before, I have been searching for a while and can't find anything.

Life story, you don't need to read
I've been working on a pretty unique project where I have a logstash server on Centos which pulls logs from another department's elasticsearch box, which then forwards to syslog to a security box where the logs are stored and correlated.

may want to read for background
I have been having a couple of issues, as I want to run logstash all the time in the background. I have tried running the bin/logstash agent -f x.conf but I can't point it to logstash-* because there's so much data, I can't even point it to logstash-2015-08-1* because there's so much data. It times out and throws up an Exception in Thread Unsupported Operation Exception message.

The query that I run only takes a very small amount of data from the server. The only way the above works is if it I point it to today (logstash-2015-08-12 for example). That's fine.

pls read this bit
What I want to do: I want to be able for that script to run in the background all the time against logstash-YYYY-MM-DD is that possible? I get errors whenever I try.

I've seen in older versions of elasticsearch.rb (ex: https://www.omniref.com/github/elasticsearch/logstash/1.0.9/files/lib/logstash/outputs/elasticsearch.rb#line=18) that there's syntax for it. But when I try, I get an error.

  # The index to write events to. This can be dynamic using the %{foo} syntax.
  # The default value will partition your indeces by day so you can more easily
  # delete old data or only search specific date ranges.
  config :index, :validate => :string, :default => **"logstash-%{+YYYY.MM.dd}"**

If I was to use that example above I get the error message:
Error: [404] {"error":"IndexMissingException[[logstash-%{+YYYY.MM.dd}] missing]","status":404} {:level=>:error}

is there a way of doing this, once this is working I can get to work of having this script running all the time. Please could someone help, I've been working on this a while with no previous experience so I just keep poking it until it works.

Thank you for your time,

  • Chris
2

What does your config with the query look like?

3

Hi Mark, thanks for the quick response, here's my input. Note I've commented out the index part. It's also worth noting that I can get the same results without the query unless I set it to today's date.

input {
  elasticsearch {
    hosts => "servername.x.x"
   #index => "logstash-%{+YYYY.MM.dd}"
    query => '{ "query": { "filtered": { "query": { "bool": { "should": [ { "query_string": { "query": "program:sshd AND NOT @Feature:logstash" } }, { "query_string": { "query": "@LogType:eventlog AND Channel:Security" } } ] } } } } }'
    codec => json
  }
}

Ideally, it would be good to use the maths from the Elasticsearch output so the config will always get the logs for today. https://www.elastic.co/guide/en/elasticsearch/reference/master/date-math-index-names.html

4

Not to worry, I've got around it by putting this in the elasticsearch.rb file

  # The index or alias to search.
  d = Time.now
  d = d.strftime("%y.%m.%d")

  config :index, :validate => :string, :default => "logstash-*" + d

hope that helps someone one day.