# Missing .keyword fields

**URL:** https://discuss.elastic.co/t/missing-keyword-fields/317610
**Category:** Kibana
**Created:** [October 27, 2022, 11:40am UTC](https://discuss.elastic.co/t/missing-keyword-fields/317610 "2022-10-27T11:40:24Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![khaoz](https://avatars.discourse-cdn.com/v4/letter/k/958977/32.png) [@khaoz](https://discuss.elastic.co/u/khaoz)
#### Post date: [October 27, 2022, 11:40am UTC](https://discuss.elastic.co/t/missing-keyword-fields/317610/1 "2022-10-27T11:40:24Z")

</div>

Hello,

I am quite new to ELK and I am facing the following issue.  
For some reason, the .keyword fields (e.g. hostname.keyword) are not being populated in Kibana and I cannot use them in setting up alerts etc.

Let's take for example the following fields from an expanded document:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/a/da0f6fa0ba6bb262d6471d5070486fe122704929.png)  
They have no field.keyword entry.

However, if you look at the json view of the same event:

```auto
{
  "_index": "xxxxx",
  "_id": "xxxxxx",
  "_version": 1,
  "_score": 0,
  "_ignored": [
    "message.keyword",
    "event.original.keyword",
    "xxxxx.keyword"
  ],
  "_source": {
		### shortened...###
    "type": "syslog",
    "object_type": "File",
    "@timestamp": "xxxxx",
    "need_restart": false,
    "object_uri": "https://xxxxx",
    "severity": "Warning",
    "threat_type": "Application",
    "ipv4": "xxxxx",
    "today": "xxxxx",
    "username": "xxxxx",
    "processname": "C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe",
    "circumstances": "xxxxx",
    "hash": "xxxxx"
  },
  "fields": {
    "hash.keyword": [
      "xxxxx"
    ],
###shortened...###
    "type": [
      "syslog"
    ],
    "object_type.keyword": [
      "File"
    ],
    "severity.keyword": [
      "Warning"
    ],
    "username.keyword": [
      "xxxxx"
    ],
    "circumstances": [
      "xxxxx"
    ],
    "event.original": [
      "xxxxx"
    ],
    "action_taken": [
      "xxxxx"
    ],
    "@version.keyword": [
      "1"
    ],
    "threat_type.keyword": [
      "Application"
    ],
    "processname.keyword": [
      "C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe"
    ],
    "circumstances.keyword": [
      "xxxxx"
    ]
  }
}

```

My index mapping is as follows:

```auto
{
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "@version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "action_taken": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "circumstances": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "datasource": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "engine_version": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "esetraw": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "event": {
        "properties": {
          "original": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "event_type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "hash": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "host": {
        "properties": {
          "ip": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      },
      "hostname": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "ipv4": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "message": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "need_restart": {
        "type": "boolean"
      },
      "object_type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "object_uri": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "occured": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "processname": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "scan_id": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "scanner_id": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "severity": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "source_uuid": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "tags": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "threat_handled": {
        "type": "boolean"
      },
      "threat_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "threat_type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "today": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "type": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "username": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      }
    }
  }
}

```

Events are coming from Logstash with the following conf file:

```auto
input {
  tcp {
    port => 5140
    type => syslog
  }
  udp {
    port => 5140
    type => syslog
  }
}

filter {

  grok {
     match => { "message" => "(?<today>\w{3}\s*\d{1,2}\s\d\d:\d\d:\d\d) (?<datasource>TEST TEST)\[\d{1,4}\]\:\s(?<xxxxx>.*)" }
     add_tag => ["TEST"]
  }

        if "TEST" in [tags] {
        json {
        source => "xxxxx"
        }

}

}

output {
  elasticsearch {
    hosts => ["https://xxxxx"]
    index => "xxxxx"
    user => "elastic"
    password => "xxxxx"
    cacert => "xxxxx"
  }
}

```

I have tried refreshing the index, flushing it and even deleting it, but it's still the same.  
My index mapping looks fine, as far as i understand...  
Tried to also create a new index and use KV filter instead of json - same result.

The .keyword fields are also not available for using them in the Security \> Alerts page columns.

I think there was even a moment after the initial setup where I was seeing the .keyword fields, but they suddenly disappeared. I have no idea what has changed...

---

<div class="post-metadata">

### Author: ![khaoz](https://avatars.discourse-cdn.com/v4/letter/k/958977/32.png) [@khaoz](https://discuss.elastic.co/u/khaoz)
#### Post date: [October 27, 2022, 1:58pm UTC](https://discuss.elastic.co/t/missing-keyword-fields/317610/2 "2022-10-27T13:58:32Z")

</div>

I dont know if this is helpful, but I just noticed that the .keyword fields exist in the Data View of my index, but they are empty:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/1/e108534e14df58eb2d8d8ca64d7e04c447a4e46a.png)

---

<div class="post-metadata">

### Author: ![matw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matw/32/13913_2.png) [@matw](https://discuss.elastic.co/u/matw)
#### Post date: [November 2, 2022, 10:08am UTC](https://discuss.elastic.co/t/missing-keyword-fields/317610/3 "2022-11-02T10:08:18Z")

</div>

Hi and welcome to our community!

I think while you have the fields in your mapping, e.g. when I have a look the the JSON of your event there's no "hostname" given. this means this fields has not been set when ingesting the document. So you should have a look into your log stash configuration. I can seen no indicator where hostname would be set as a separate field? Or should hostname be part of the syslog event incoming? then you should have a look if it has been sent to logstash. So I'd recommend debugging your ingest pipeline first.  
Hope this helps!  
Best,  
Matthias

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 29, 2022, 4:26am UTC](https://discuss.elastic.co/t/missing-keyword-fields/317610/4 "2022-11-29T04:26:04Z")

</div>



---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 27, 2022, 4:26am UTC](https://discuss.elastic.co/t/missing-keyword-fields/317610/5 "2022-12-27T04:26:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.