Elasticsearch output: LogStash::Error: timestamp field is missing

[ouss]

Hello,

I have an elasticsearch output working without any problem

elasticsearch {
  hosts => [ "${LOGS_ELASTICSEARCH_CLUSTER}:443" ]
  ssl => true
  index => "cwl-%{+YYYY.MM.dd}"
  document_type => "log"
}

I wanted to redirect some type of logs to a new elasticsearch index but I'm getting this error

[2019-09-03T14:39:00,242][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>, :backtrace=>["org/logstash/ext/JrubyEventExtLibrary.java:202:in `sprintf'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.8-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `event_action_params'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.8-java/lib/logstash/outputs/elasticsearch/common.rb:48:in `event_action_tuple'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.8-java/lib/logstash/outputs/elasticsearch/common.rb:42:in `multi_receive'", "org/jruby/RubyArray.java:2414:in `map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-7.3.8-java/lib/logstash/outputs/elasticsearch/common.rb:42:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:13:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:47:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:420:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:419:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:365:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:330:in `start_workers'"]}

No newer events found at the moment. Retry.

The new output is:

  if "http" in [tags] {
    elasticsearch {
      hosts => [ "${LOGS_ELASTICSEARCH_CLUSTER}:443" ]
      ssl => true
      index => "cwl-http-%{+YYYY.MM.dd}"
      document_type => "log"
    }
  }
  else {
    elasticsearch {
      hosts => [ "${LOGS_ELASTICSEARCH_CLUSTER}:443" ]
      ssl => true
      index => "cwl-%{+YYYY.MM.dd}"
      document_type => "log"
    }
  }

I modified my filter like this

 ### HTTP logs
 if [logger_name] == "HttpLoggerService" {
   mutate {
     add_tag => [ "http" ]
   }
 }

any thing wrong in my configuration ?

Thanks!

[Badger]

@ouss

An unexpected error occurred! {:error=>#<LogStash::Error: timestamp field is missing>

You are using %{+YYYY.MM.dd} in your index name, but the event does not have a @timestamp field from which to extract the year, month and day.

If the only difference between the two elasticsearch outputs is the index name it would be slightly more efficient to put the index name into a [@metadata] field and use a sprintf reference to it.

[ouss]

It solved my problem, Thanks @Badger