Logstash output Indexname with old Date

Elastic Stack Logstash
1

How to add in logstash output Index name as "logstash-old date" , not with timestamp. Current Date is working fine for me. how to convert 2016-05-23T10:42:04.370Z to normal date 2016-05-23 and use this date in logstash output index name "logstatsh-2016-05-23"

actual requirement : index name should be "logstash-2016-04-12"

currently I am getting data from mysql date as 2016-05-23T10:42:04.370Z which is not accaptable in as logstash output Index name.

2

I can see why having one index per millisecond of data could be problematic :slight_smile:

What does your elasticsearch output section of your pipeline configuration look like (please be sure to redact passwords and hostnames)? If the value for its index parameter contains references to any fields, what are some example values for those fields, and, how were these fields populated (e.g., were they populated by logstash-filter-grok, created by logstash-filter-date, or created by your codec)?

3

The date formatting in the Elasticsearch output plugin is always based on @timestamp. If you want to base it on some other date/timestamp, you will need to extract that into a separate field and then use this as a index name suffix instead of the timestamp logic.

4

I have already one date field in my mysql. when i import data from mysql to elastic by using logstash , logstatsh converting the date in yyyy-MM-dd'T'HH:mm:ss.SSSZ format like
"2018-03-14T10:21:52.081Z" , but i want his date as simple date like "2018-03-14" and the same date i want to use as index-suffix name. I tried different but its not working. the same date and as well as "EEE MMM d HH:mm:ss yyyy Z" format also

elastic Search Result

{
"lastname" => "Kallis",
"city" => "Cape Town",
"@timestamp" => 2018-03-14T12:38:03.881Z,
"firstname" => "Jaques",
"date" => 2016-05-23T10:42:03.568Z,
"personid" => 4005,
"@version" => "1"
}

I want to convert date field format like '2016-05-23' and 'Sun Feb 25 20:53:34 2018 -0500'using logstash. I tried different scenario's ,but its not working.can you please help me out.

5

I have already one date field in my mysql. when i import data from mysql to elastic by using logstash , logstatsh converting the date in yyyy-MM-dd'T'HH:mm:ss.SSSZ format like
"2018-03-14T10:21:52.081Z" , but i want his date as simple date like "2018-03-14" and the same date i want to use as index-suffix name. I tried different but its not working. the same date and as well as "EEE MMM d HH:mm:ss yyyy Z" format also

6

@sumanbehara

In order to empower others to help in a forum, it's a good idea to always provide the following when asking for help:

  • what you tried (including relevant configurations)
  • what you expected to happen (and why you expected this, if possible)
  • what actually happened

The index property of logstash-output-elasticsearch uses Logstash's sprintf format, meaning it can use context from each event to produce its value; when this format string includes a date-format, Logstash automatically pulls from the @timestamp field, so if we can populate @timestamp with the value of date, or if we can reference a field that already has the right format, we'll be all set.

We have two options:

  • overwrite the @timestamp with the value from our date field; OR
  • pre-generate a formatted string into our @metadata, and reference it when building the index name template

Overwriting @timestamp

It looks like your current @timestamp is being automatically filled in with the current time; is it okay to overwrite this? If so, adding a mutate filter above your output to rename date to @timestamp might be helpful:

# ...
filter {
  mutate {
    rename { "date" => "@timestamp" }
  }
}
# ...

Then we could use the built-in formatters when building the index pattern; for example, if our event's @timestamp was 2016-05-23T10:42:03.568Z, an Elasticsearch output configured like so would put the event in index something-2016-05-23:

# ...
output {
  elasticsearch {
    index => "something-%{+yyyy-MM-dd}"
    # ...
  }
}

Pre-generating a formatted string

If it's not okay to overwrite the @timestamp, then we may need to use a filter like wiibaa's logstash-filter-date_formatter, a well-tested community-contributed filter that can combine any timestamp field with a format specification to place a string representation in the event's metadata:

# ...
filer {
  date_formatter {
    source => "date"
    target => "[@metadata][date]"
    pattern => "YYYY-MM-dd"
  }
}
output {
  elasticsearch {
    index => "something-%{[@metadata][date]}"
    # ...
  }
}

The internal representation of the date field in your rubydebug output is an object that fully defines that exact moment in time down to the millisecond, which is great -- we can format it however we want (the rubydebug output just happens to print it as ISO8601).

In the Elastic Stack, we have a convention where the @timestamp is a little bit special, so it would be convenient if we could populate it with the value from your date field:

output {
  elasticsearch {
    index => "something-%{+yyyy-MM-dd}"
  }
}
1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.