Not able to index records which contain missing fields ElasticSearch 7.4

Hello all,

Can someone point me in the right direction of any documents or links that may be able to help me.

I have am using kinesis to ingest just 2 simple events

2020-01-09 15:35:51,334 tid:LE7WZpV000NCG4NeSUVwBvW6HuQ DEBUG [org.sourceid.webders.servlet.IntegrationControllerServlet] POST:
2020-01-09 15:35:51,335  DEBUG [com.appy.jgroups.MuxInvocationHandler] invocation of retrieveAndRemoveState on InterReqStateMgmtMapImpl, state map size: 0, attributes map size: 50

The first event does get indexed but the second one does not ..

Now my index_template is fairly simple:

PUT /_template/appy-10.0-serverlog
  "index_patterns": [
  "settings": {
    "number_of_shards": 1, 
    "number_of_replicas": 1
  "mappings": {

      "_source": {"enabled": true}, 
      "properties": {
      "timestamp": {
        "type": "date",
        "format":"yyyy-MM-dd HH:mm:ss,SSS"
      "tid": {"type":"keyword"}, 
      "level": {"type": "keyword"},
      "message": {"type": "text"}


Is there something i am missing in my template that would allot a field if it didnt exist in the log stream not ot be indexed ? I have been reading should i be using the

"enabled" : "false"

or should i be looking at the kinesis side to work out why this is happening. (apologies the debugging in AWS console is a bit limited with kinesis etc)

Thank you again for any help or guidance on this.



What do the events look like? What is the error message? Is there anything in the Elasticsearch logs?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.