Logstash not appending date to ES indices

Elastic Stack Logstash
1

Hello,

I'm currently working through upgrading a 2.6 ELK setup to 7.5 on a new server using what I can from the old config. The goal is to send rsyslog data from clients to Logstash. Previously, this worked (mostly) fine using the following Logstash Input, Filter, and Output config:

input {
  tcp {
    port => 1514
    type => syslog
  }
  udp {
    port => 1514
    type => syslog
  }
  udp {
    port => 1515
    type => PFSense
  }
}

filter {
    if [type] == "syslog" {
        grok {
           match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
           add_field => [ "received_at", "%{@timestamp}" ]
           add_field => [ "received_from", "%{host}" ]

        }
        syslog_pri { }
        date
        {
            match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }

    }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => true
    index => "logstash-%{+YYYY.MM.dd}"
    document_type => "syslog"
  }
}

In the updated VM, I'm using an rsyslog template to send the syslogs as JSON:

template(name="json-template"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}

And using an updated Logstash config to accommodate:

input {
  tcp {
    port => 1514
    type => syslog
    codec => json
  }
  tcp {
    port => 514
    type => syslog
    codec => json
  }
  udp {
    port => 1514
    type => syslog
    codec => json
  }
  udp {
    port => 1515
    type => PFSense
  }
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

This mostly works well - we aren't getting grokparsefailures like before, the messages in Kibana are generally readable, and the fields are all relevant. You might notice there's no filter section - with the old filter in place on the new server, I was getting more grokparsefailure errors and it seems to be working fine (perhaps better) without any filter at all. However, every message has the same index, which is simply logstash. Previously, the index would have the date appended to the end. This is crucial, since we then use Curator to rotate old indices after a certain amount of time.

I've tried the following:

  • Match date on "@timestamp" in the filter section
  • Add the old filter back in its current state
  • Set manage_template to false in the ES output section
  • Remove the index line from the ES output section

Please let me know if there's any more info I can provide. I'm admittedly weak at Logstash and grok so this might be super obvious, but most of what I've been able to find online deals with either an old version of ELK or using Filebeats as inputs. Thank you.

2

If you are planning to use curator to rotate indexes then you should disable ILM.

3

Thanks for the reply - I actually didn't know this existed.

I might look into using this instead of Curator, seems like it might be better, but I think my problem still exists; even with ILM, it would just delete the entire logstash index, which could contain many days/weeks worth of logs. Am I understanding that correctly?

4

No, that is not correct. By default it will rollover when the index either reaches 50 GB or is 30 days old. You can change those numbers and do time-only or size-only rolling. So you can certainly do one index per day.

ILM can also do other things that curator was useful for, like merging rolled indexes into a single segment.

5

My concern is that logs created on different days will have the same index name (logstash) rather than an index name that is unique to their day (logstash-2019-12-10, etc.) So if ILM rotates at some amount of time, then the entire logstash index and its associated logs would be deleted, rather than just the index specific to that log.

Does ILM have some role in the name of the index when logs are outputted from Logstash to Elasticsearch? I assumed it happened somewhere between the filter and output phase of Logstash processing. The docs even say that the default value for the index field in the elasticsearch output is "logstash-%{+YYYY.MM.dd}", which is what I'd expect.

6

The index option to the output is ignored if ILM is enabled.

To your first point, if ILM worked like that then it would be useless. It does not work like that. Indexes are renamed as they rollover. Trust me, it can be configured to do what you want.

7

Great, I'll give it a try. Thank you for your help!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.