# Sorting based on event\_time in logstash

**URL:** <https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636>\
**Category:** Logstash\
**Created:** [July 3, 2017, 1:30pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636 "2017-07-03T13:30:16Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![blackOcean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blackocean/32/19722_2.png) [@blackOcean](https://discuss.elastic.co/u/blackOcean)\
**Post date:** [July 3, 2017, 1:30pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/1 "2017-07-03T13:30:16Z")

</div>

Hi,

Thanks for helping me in all the stuff. Now, I am facing a weird problem.

These are my logs

```
2017-01-03 05:40:50.522 INFO main ---> org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [class org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$88c2216e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2017-01-03 05:40:50.543 INFO main ---> org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$af188c46] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

```

And my logstash server configuration file is this.

```
filter {
  grok { match => { "message" => "^%{TIMESTAMP_ISO8601:event_time}\s+%{LOGLEVEL:level}\s+%{SYSLOGPROG}\s---\s%{JAVACLASS:class}\s+:\s+%{GREEDYDATA:message}$"}}
}

```

Now I want to sort the data on the basis of event\_time. But when I am trying to do the same. it says field data is not true. Can somebody help me what is the mistake .

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 3, 2017, 1:56pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/2 "2017-07-03T13:56:32Z")

</div>

> Now I want to sort the data on the basis of event\_time.

Exactly how are you doing this?

> But when I am trying to do the same. it says field data is not true.

Please quote the actual error message in full.

---

<div class="post-metadata">

**Author:** ![blackOcean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blackocean/32/19722_2.png) [@blackOcean](https://discuss.elastic.co/u/blackOcean)\
**Post date:** [July 3, 2017, 2:16pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/3 "2017-07-03T14:16:48Z")

</div>

Hi @magnusbaeck,

Query

```
GET logstash-2017.07.03/_search 
{
  "sort": [
    {
      "event_time": {
        "order": "desc"
      }
    }
  ]
} 

```

Error

```
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "logstash-2017.07.03",
        "node": "FDpTDtSMQo2YpsSoSrPKGg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
        }
      }
    ],
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
    }
  },
  "status": 400
}

```

 ![](https://us1.discourse-cdn.com/elastic/original/3X/e/0/e0c4d192f2c9ce8e72e61937b2083071a5d4a397.png)

My question I am unable to map event\_time with date datatype and do sorting.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 3, 2017, 2:40pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/4 "2017-07-03T14:40:36Z")

</div>

The default field for the event's timestamp is `@timestamp`. Unless you really want the field to be named `event_time` you can save yourself some trouble by sticking to the defaults.

It could be that ES doesn't recognize "2017-01-03 05:40:50.522" as a timestamp and therefore mapped the field as text. If you use the date filter you can transform the timestamp into something that ES _will_ recognize as a timestamp and then your query should work just fine. (But note that you'll have to reindex to change the mapping of the `event_time` field.)

---

<div class="post-metadata">

**Author:** ![blackOcean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blackocean/32/19722_2.png) [@blackOcean](https://discuss.elastic.co/u/blackOcean)\
**Post date:** [July 3, 2017, 3:32pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/5 "2017-07-03T15:32:39Z")

</div>

I[quote="magnusbaeck, post:4, topic:91636"]  
If you use the date filter you can transform the timestamp into something that ES will recognize as a timestamp and then your query should work just fine. (But note that you'll have to reindex to change the mapping of the event\_time field.)  
[/quote]

Thanks. I am struggling to visualize this part. Can you please give me a small example or how I can proceed with that.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 3, 2017, 6:05pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/6 "2017-07-03T18:05:25Z")

</div>

You're having issues understanding the date filter? Its documentation contains a couple of examples.

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match)

---

<div class="post-metadata">

**Author:** ![blackOcean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blackocean/32/19722_2.png) [@blackOcean](https://discuss.elastic.co/u/blackOcean)\
**Post date:** [July 4, 2017, 9:51am UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/7 "2017-07-04T09:51:05Z")

</div>

Hi @magnusbaeck, I believe there is some confuse in our communication. I don't want to convert or change the name of @timestamp into event\_time. Please check my logstash filter, I want to break log message into event\_time, log level and other parts.

```
filter {
  grok { match => { "message" => "^%{TIMESTAMP_ISO8601:event_time}\s+%{LOGLEVEL:level}\s+%{SYSLOGPROG}\s---\s%{JAVACLASS:class}\s+:\s+%{GREEDYDATA:message}$"}}
}

```

Now this event\_time is coming as text, but I want it as date. I tried mapping then logstash parser fails. I have no idea how to fix it.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 4, 2017, 11:32am UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/8 "2017-07-04T11:32:12Z")

</div>

> Now this event\_time is coming as text, but I want it as date.

I know.

Use the date filter to parse the `event_time` field. You can save yourself some trouble by saving the parsed result into the `@timestamp` field but you can also use the `target` option to overwrite the existing `event_time` value with the parsed value.

The string parsed by the date filter will be recognized as a timestamp by Elasticsearch, but the mapping of existing indexes will not change. I suggest you delete your current index(es) and have Logstash recreate them.

---

<div class="post-metadata">

**Author:** ![blackOcean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blackocean/32/19722_2.png) [@blackOcean](https://discuss.elastic.co/u/blackOcean)\
**Post date:** [July 4, 2017, 12:16pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/9 "2017-07-04T12:16:24Z")

</div>

Thanks @magnusbaeck.

please find my logstash conf file.

```
input {
  beats {
    port => 5044
  }
}
filter {
  grok { match => { "message" => "^%{TIMESTAMP_ISO8601:event_time}\s+%{LOGLEVEL:level}\s+%{SYSLOGPROG}\s---\s%{JAVACLASS:class}\s+:\s+%{GREEDYDATA:message}$"}}
  date {
            match => ["event_time", "YYYY-MM-dd HH:mm:ss.SSS", "ISO8601"]
            target => "@timestamp"
    }
}
output {
  elasticsearch {
  hosts => ["localhost:9200"]
  index => "logstash-%{+YYYY.MM.dd}"
  manage_template => true
  template_name => "logstash*"
 }
  stdout { codec => rubydebug }
}

```

See this is my logs which also contains multiple lines log stacktrace.

```
2017-01-03 05:40:49.681 INFO main --- org.springframework.context.annotation.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@41d16cc3: startup date [Tue Jan 03 05:40:49 UTC 2017]; root of context hierarchy
2017-01-03 05:40:49.693 INFO main --- com.getsentry.raven.DefaultRavenFactory : Using an HTTP connection to Sentry.
2017-01-03 05:40:49.935 INFO background-preinit --- org.hibernate.validator.internal.util.Version : HV000001: Hibernate Validator 5.2.4.Final
2017-01-03 05:40:50.355 INFO main --- org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2017-01-03 05:40:50.522 INFO main --- org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [class org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$88c2216e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2017-01-03 05:40:50.543 INFO main --- org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$af188c46] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2017-01-03 05:40:51.430 INFO main --- com.getsentry.raven.connection.AsyncConnection : Gracefully shutdown sentry threads.
2017-01-03 05:40:52.430 WARN main --- com.getsentry.raven.connection.AsyncConnection : Graceful shutdown took too much time, forcing the shutdown.
2017-01-03 05:40:52.430 INFO main --- com.getsentry.raven.connection.AsyncConnection : 5 tasks failed to execute before the shutdown.
2017-01-03 05:40:52.430 INFO main --- com.getsentry.raven.connection.AsyncConnection : Shutdown finished.
2017-01-03 05:40:52.445 INFO main --- com.getsentry.raven.DefaultRavenFactory : Using an HTTP connection to Sentry. 

```

My Kibana Query :

```
GET logstash-2017.07.04/_search 
{
"sort": [
    {
      "event_time": {
        "order": "desc"
      }
    }
  ]
}

```

Output : grep parse failure. There is no event\_time in it.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 4, 2017, 12:21pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/10 "2017-07-04T12:21:32Z")

</div>

You've configured the date filter to save the parsed result in the `@timestamp` field, yet it's the `event_time` field you're trying to sort on. That doesn't make sense.

> Output : grep parse failure. There is no event\_time in it.

What do you mean?

---

<div class="post-metadata">

**Author:** ![blackOcean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/blackocean/32/19722_2.png) [@blackOcean](https://discuss.elastic.co/u/blackOcean)\
**Post date:** [July 4, 2017, 12:44pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/11 "2017-07-04T12:44:45Z")

</div>

@magnusbaeck.

Thats my stupidity. Now I check the mapping. I can find event\_time as date field. Thanks to you. 🙂

But for my kibana query. I am getting only this.

```
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "logstash-2017.07.04",
        "node": "suo9gTyRRxWBZiqGOt3nzg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
        }
      }
    ]
  },
  "status": 400
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 1, 2017, 12:54pm UTC](https://discuss.elastic.co/t/sorting-based-on-event-time-in-logstash/91636/12 "2017-08-01T12:54:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.