Not able to index records which contain missing fields ElasticSearch 7.4

Elastic Stack Elasticsearch
1

Hello all,

Can someone point me in the right direction of any documents or links that may be able to help me.

I have am using kinesis to ingest just 2 simple events

2020-01-09 15:35:51,334 tid:LE7WZpV000NCG4NeSUVwBvW6HuQ DEBUG [org.sourceid.webders.servlet.IntegrationControllerServlet] POST: https://idg.appy.com/idp/mTgCR/resumeSAML20/idp/SSO.ping
2020-01-09 15:35:51,335  DEBUG [com.appy.jgroups.MuxInvocationHandler] invocation of retrieveAndRemoveState on InterReqStateMgmtMapImpl, state map size: 0, attributes map size: 50

The first event does get indexed but the second one does not ..

Now my index_template is fairly simple:

PUT /_template/appy-10.0-serverlog
{
  "index_patterns": [
    "appy-srvlog-test*"
  ], 
  "settings": {
    "number_of_shards": 1, 
    "number_of_replicas": 1
  }, 
  "mappings": {

      "_source": {"enabled": true}, 
      "properties": {
      "timestamp": {
        "type": "date",
        "format":"yyyy-MM-dd HH:mm:ss,SSS"
      }, 
      "tid": {"type":"keyword"}, 
      "level": {"type": "keyword"},
      "message": {"type": "text"}

    }
  }
}

Is there something i am missing in my template that would allot a field if it didnt exist in the log stream not ot be indexed ? I have been reading should i be using the

"enabled" : "false"

or should i be looking at the kinesis side to work out why this is happening. (apologies the debugging in AWS console is a bit limited with kinesis etc)

Thank you again for any help or guidance on this.

cheers

yoyomonkey

2

What do the events look like? What is the error message? Is there anything in the Elasticsearch logs?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.