Sorting based on event_time in logstash

[blackOcean (Sagar Rout)]

Hi,

Thanks for helping me in all the stuff. Now, I am facing a weird problem.

These are my logs

2017-01-03 05:40:50.522 INFO  main ---> org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [class org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$88c2216e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2017-01-03 05:40:50.543 INFO  main ---> org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$af188c46] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)

And my logstash server configuration file is this.

filter {
  grok { match => { "message" => "^%{TIMESTAMP_ISO8601:event_time}\s+%{LOGLEVEL:level}\s+%{SYSLOGPROG}\s---\s%{JAVACLASS:class}\s+:\s+%{GREEDYDATA:message}$"}}
}

Now I want to sort the data on the basis of event_time. But when I am trying to do the same. it says field data is not true. Can somebody help me what is the mistake .

[magnusbaeck (Magnus Bäck)]

Now I want to sort the data on the basis of event_time.

Exactly how are you doing this?

But when I am trying to do the same. it says field data is not true.

Please quote the actual error message in full.

[blackOcean (Sagar Rout)]

Hi @magnusbaeck,

Query

GET logstash-2017.07.03/_search 
{
  "sort": [
    {
      "event_time": {
        "order": "desc"
      }
    }
  ]
} 

Error

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "logstash-2017.07.03",
        "node": "FDpTDtSMQo2YpsSoSrPKGg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
        }
      }
    ],
    "caused_by": {
      "type": "illegal_argument_exception",
      "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory."
    }
  },
  "status": 400
}

My question I am unable to map event_time with date datatype and do sorting.

[magnusbaeck (Magnus Bäck)]

The default field for the event's timestamp is @timestamp. Unless you really want the field to be named event_time you can save yourself some trouble by sticking to the defaults.

It could be that ES doesn't recognize "2017-01-03 05:40:50.522" as a timestamp and therefore mapped the field as text. If you use the date filter you can transform the timestamp into something that ES will recognize as a timestamp and then your query should work just fine. (But note that you'll have to reindex to change the mapping of the event_time field.)

[blackOcean (Sagar Rout)]

I[quote="magnusbaeck, post:4, topic:91636"]
If you use the date filter you can transform the timestamp into something that ES will recognize as a timestamp and then your query should work just fine. (But note that you'll have to reindex to change the mapping of the event_time field.)
[/quote]

Thanks. I am struggling to visualize this part. Can you please give me a small example or how I can proceed with that.

[magnusbaeck (Magnus Bäck)]

You're having issues understanding the date filter? Its documentation contains a couple of examples.

https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match

[blackOcean (Sagar Rout)]

Hi @magnusbaeck, I believe there is some confuse in our communication. I don't want to convert or change the name of @timestamp into event_time. Please check my logstash filter, I want to break log message into event_time, log level and other parts.

filter {
  grok { match => { "message" => "^%{TIMESTAMP_ISO8601:event_time}\s+%{LOGLEVEL:level}\s+%{SYSLOGPROG}\s---\s%{JAVACLASS:class}\s+:\s+%{GREEDYDATA:message}$"}}
}

Now this event_time is coming as text, but I want it as date. I tried mapping then logstash parser fails. I have no idea how to fix it.

[magnusbaeck (Magnus Bäck)]

Now this event_time is coming as text, but I want it as date.

I know.

Use the date filter to parse the event_time field. You can save yourself some trouble by saving the parsed result into the @timestamp field but you can also use the target option to overwrite the existing event_time value with the parsed value.

The string parsed by the date filter will be recognized as a timestamp by Elasticsearch, but the mapping of existing indexes will not change. I suggest you delete your current index(es) and have Logstash recreate them.

[blackOcean (Sagar Rout)]

Thanks @magnusbaeck.

please find my logstash conf file.

input {
  beats {
    port => 5044
  }
}
filter {
  grok { match => { "message" => "^%{TIMESTAMP_ISO8601:event_time}\s+%{LOGLEVEL:level}\s+%{SYSLOGPROG}\s---\s%{JAVACLASS:class}\s+:\s+%{GREEDYDATA:message}$"}}
  date {
            match => ["event_time", "YYYY-MM-dd HH:mm:ss.SSS", "ISO8601"]
            target => "@timestamp"
    }
}
output {
  elasticsearch {
  hosts => ["localhost:9200"]
  index => "logstash-%{+YYYY.MM.dd}"
  manage_template => true
  template_name => "logstash*"
 }
  stdout { codec => rubydebug }
}

See this is my logs which also contains multiple lines log stacktrace.

2017-01-03 05:40:49.681 INFO  main --- org.springframework.context.annotation.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@41d16cc3: startup date [Tue Jan 03 05:40:49 UTC 2017]; root of context hierarchy
2017-01-03 05:40:49.693 INFO  main --- com.getsentry.raven.DefaultRavenFactory : Using an HTTP connection to Sentry.
2017-01-03 05:40:49.935 INFO  background-preinit --- org.hibernate.validator.internal.util.Version : HV000001: Hibernate Validator 5.2.4.Final
2017-01-03 05:40:50.355 INFO  main --- org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor : JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
2017-01-03 05:40:50.522 INFO  main --- org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.retry.annotation.RetryConfiguration' of type [class org.springframework.retry.annotation.RetryConfiguration$$EnhancerBySpringCGLIB$$88c2216e] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2017-01-03 05:40:50.543 INFO  main --- org.springframework.context.support.PostProcessorRegistrationDelegate$BeanPostProcessorChecker : Bean 'configurationPropertiesRebinderAutoConfiguration' of type [class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$af188c46] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
2017-01-03 05:40:51.430 INFO  main --- com.getsentry.raven.connection.AsyncConnection : Gracefully shutdown sentry threads.
2017-01-03 05:40:52.430 WARN  main --- com.getsentry.raven.connection.AsyncConnection : Graceful shutdown took too much time, forcing the shutdown.
2017-01-03 05:40:52.430 INFO  main --- com.getsentry.raven.connection.AsyncConnection : 5 tasks failed to execute before the shutdown.
2017-01-03 05:40:52.430 INFO  main --- com.getsentry.raven.connection.AsyncConnection : Shutdown finished.
2017-01-03 05:40:52.445 INFO  main --- com.getsentry.raven.DefaultRavenFactory : Using an HTTP connection to Sentry. 

My Kibana Query :

GET logstash-2017.07.04/_search 
{
"sort": [
    {
      "event_time": {
        "order": "desc"
      }
    }
  ]
}

Output : grep parse failure. There is no event_time in it.

[magnusbaeck (Magnus Bäck)]

You've configured the date filter to save the parsed result in the @timestamp field, yet it's the event_time field you're trying to sort on. That doesn't make sense.

Output : grep parse failure. There is no event_time in it.

What do you mean?

[blackOcean (Sagar Rout)]

@magnusbaeck.

Thats my stupidity. Now I check the mapping. I can find event_time as date field. Thanks to you.

But for my kibana query. I am getting only this.

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
      }
    ],
    "type": "search_phase_execution_exception",
    "reason": "all shards failed",
    "phase": "query",
    "grouped": true,
    "failed_shards": [
      {
        "shard": 0,
        "index": "logstash-2017.07.04",
        "node": "suo9gTyRRxWBZiqGOt3nzg",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [event_time] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
        }
      }
    ]
  },
  "status": 400
}