I need help! elastic doesn't get my mapping!

[sina12 (Sina)]

Hi Guys!

I am in trouble. I need to visualize my data but elasticsearch doesn't accept my mapping. I edited the mapping like that:

PUT /my_example/_mappings
{
 
  "properties" : {
  "id" :{
    "type": "boolean"
  },
  "sha" :{
    "type" : "text"
  },
  "ref" : {
    "type" : "text"
  },
  "status": {
    "type" : "text"
    },
  "created_at": {
    "type" : "date"
    },
    "updated_at": {
       "type" : "date"
    },
    "web_url": {
    "type" : "text"
    }
}}

and I removed "message" and "field". For some reason my mapping doesn't work and elastic puts the data into the "message" field like that:

{
  "took" : 4,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "my_example",
        "_type" : "_doc",
        "_id" : "idx",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-06-19T16:20:54.819Z",
          "field" : "\"created_at\"",
          "message" : """        "created_at": "2020-06-19T16:20:54.819Z","""
        }
      },
      {
        "_index" : "my_example",
        "_type" : "_doc",
        "_id" : "idy",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2020-06-19T16:52:15.314Z",
          "field" : "\"updated_at\"",
          "message" : """        "updated_at": "2020-06-19T16:52:15.314Z",
        "web_url": "http://example1"
    },
    {
        "id": 2,
        "sha": "example",
        "ref": "ref",
        "status": "success","""
        }
      }, 

So actually Elastic created two hits for one and it writes all the inputs into "message".
I am really desperate and it would be a pleasure if someone could help me.
THANKS!!!

[aaron_nimocks]

Can you show how you are ingesting this data?

[mykael (Michael Clarke)]

Try asking with an elasticsearch tag. Kibana is just the visualization/editor level.

Mik

[sina12 (Sina)]

Hi Aaron, thanks for you answer!
My Ingest pipeline looks like this:

{
  "description": "Ingest pipeline created by file structure finder",
  "processors": [
    {
      "grok": {
        "field": "message",
        "patterns": [
          "        %{QUOTEDSTRING:field}: \"%{TIMESTAMP_ISO8601:timestamp}\",.*"
        ]
      }
    },
    {
      "date": {
        "field": "timestamp",
        "formats": [
          "ISO8601"
        ]
      }
    },
    {
      "remove": {
        "field": "timestamp"
      }
    }
  ]
}