# SIEM does not show data

**URL:** <https://discuss.elastic.co/t/siem-does-not-show-data/229010>\
**Category:** SIEM\
**Created:** [April 21, 2020, 9:47am UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010 "2020-04-21T09:47:32Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Minh\_Ti\_n\_Tr\_n](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/minh_ti_n_tr_n/32/60754_2.png) [@Minh\_Ti\_n\_Tr\_n](https://discuss.elastic.co/u/Minh_Ti_n_Tr_n)\
**Post date:** [April 21, 2020, 9:47am UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/1 "2020-04-21T09:47:32Z")

</div>

I got the problem:  
`Fielddata is disabled on text fields by default. Set fielddata=true on [source.ip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead`  
I tried on dev tool

```auto
PUT filebeat-*/_mapping
{
  "properties": {
    "my_field": { 
      "type": "text",
      "fielddata": true
    }
  }
}

```

and got:

```auto
{
  "acknowledged" : true
}

```

But till not see the data on kibana. Then I tried:

```auto
PUT filebeat-*/_mapping/text
{
   "your_type": {
      "properties": {
        "publisher": {
          "type": "text",
          "fielddata": true
        }
      }
   }
}

```

And got:

```auto
{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
  },
  "status": 400
}

```

Any solution to make data show on kibana please!  
Thanks & Regards

---

<div class="post-metadata">

**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\
**Post date:** [April 21, 2020, 11:59am UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/2 "2020-04-21T11:59:13Z")

</div>

Hi, sorry to see you're having trouble getting your data to be displayed in the SIEM app.

I am assuming from your dev tool screen shots that you are using Filebeat. Is this correct?  
I see from another [post](https://discuss.elastic.co/t/elastic-siem-does-not-show-the-netflow-data-using-filebeat/228771) that you were using NetFlow data from pfsense. Is this the same situation?

Sorry to go back to the start, but can you tell us how you set up Filebeat to send data to your cluster? Did you enable the NetFlow module per the typical process shown below?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/7/37193aae3572ca3f203fc5d9350db99181a05672.png)

If so, you should not have to manually adjust the Elasticsearch index mappings.

Also, is it possible for you to send a screen shot of one of your NetFlow documents taken from Kibana Discover? (Please be sure not to include any confidential information). We'd like to see what fields are included and their datatypes.

---

<div class="post-metadata">

**Author:** ![Frank\_Hassanabad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frank_hassanabad/32/49255_2.png) [@Frank\_Hassanabad](https://discuss.elastic.co/u/Frank_Hassanabad)\
**Post date:** [April 21, 2020, 2:29pm UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/3 "2020-04-21T14:29:05Z")

</div>

@Minh_Ti_n_Tr_n,

It looks like you haven't pushed a set of mappings and that's why it is giving you this message below. Although this advice it is giving you looks "good" on the surface it really isn't and is _not_ what you want to do to fix your mapping issues.

> Fielddata is disabled on text fields by default. Set fielddata=true on [source.ip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead

^ Avoid doing this in favor of using correct mappings. This has serious consequences and not what you want.

What you want to do is to "push" your filebeat template mappings and ILM poloicies if your beats have access to your Elasticsearch cluster like so:

> **[Load the Elasticsearch index template | Filebeat Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html)**

You might have to overwrite your current mappings that you have for filebeat and you might have to reindex your data as well once completed. I can't make assumptions about how sensitive or the volume of your data so I can only recommend you back up anything important first through [snapshots](https://www.elastic.co/guide/en/elasticsearch/reference/current/backup-cluster-data.html) and then perform any and all steps to get a valid mapping.

Here are earlier threads that go into more detail about it that others have encountered when they accidentally do not set their mappings below. In a nutshell when your templates and mappings are not setup what is happening is that all your data is "auto creating mappings" for its data types such as "source.ip" being mapped to a "text field" when in reality it should be a field type of "ip".

> [@SIEM doesn't show any Winlogbeat events, despite ES receiving them](https://discuss.elastic.co/t/siem-doesnt-show-any-winlogbeat-events-despite-es-receiving-them/224008/10):
>
> @Aura, ahhh, I think I might see what's going on. I bet when you first setup winlog beat you accidentally forgot to push your templates which control the mapping? [https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html](https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html) You can check your mapping in your dev tools: [Screen Shot 2020-04-10 at 7.47.05 AM] like so....To see the entire mapping of your winlog beats: GET winlogbeat-7.6.0/\_mapping If you want to just concentrate on…

---

<div class="post-metadata">

**Author:** ![Minh\_Ti\_n\_Tr\_n](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/minh_ti_n_tr_n/32/60754_2.png) [@Minh\_Ti\_n\_Tr\_n](https://discuss.elastic.co/u/Minh_Ti_n_Tr_n)\
**Post date:** [April 22, 2020, 2:40am UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/4 "2020-04-22T02:40:22Z")

</div>

I will show everything here  
My filebeat mapping

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/1/f1e3efc050741b8f9da09ab46f8c1067be9ace13.png)

Filebeat.yml

```auto
# Wazuh - Filebeat configuration file
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

#filebeat.config.inputs:
# enabled: true
# path: inputs.d/*.yml

filebeat.config.modules:
  enabled: true
  path: /etc/filebeat/modules.d/*.yml

setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.template.overwrite: true
setup.ilm.enabled: false

output.elasticsearch.hosts: ['https://192.168.1.17:9200']
output.elasticsearch.username: "elastic"
output.elasticsearch.password: "<password>"
output.elasticsearch.ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"]
output.elasticsearch.ssl.certificate: "/etc/elasticsearch/certs/node-0.crt"
output.elasticsearch.ssl.key: "/etc/elasticsearch/certs/node-0.key"

```

Module enabled

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/7/87543bcad914e24a475258456fd98722076a3d16.png)

netflow.yml

```auto
# Module: netflow
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.5/filebeat-module-netflow.html

- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 2055

```

Then I try the tutorial via link: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually-alternate](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually-alternate)

My Kibana discover but dont have indice

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/c/5c6fc218fca9a5ea74b86ea76ba9a4464cdbb876.png)

Thanks

---

<div class="post-metadata">

**Author:** ![Frank\_Hassanabad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frank_hassanabad/32/49255_2.png) [@Frank\_Hassanabad](https://discuss.elastic.co/u/Frank_Hassanabad)\
**Post date:** [April 22, 2020, 1:00pm UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/5 "2020-04-22T13:00:29Z")

</div>

You didn't show me your mapping above? You just showed me a screen shot of where you tried to modify your mapping to turn on fielddata which is still not really recommended.

I think you wanted to show me from line 23 and below which would be:

```auto
GET filebeat-*/_mapping

```

However, if you have a direct connection from filebeat you can follow the commands from here:

[https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#setup-command](https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#setup-command)

to setup your templates such as:

```auto
filebeat setup

```

Then you should see that your have indexes, ILM policies, etc... setup

---

<div class="post-metadata">

**Author:** ![Minh\_Ti\_n\_Tr\_n](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/minh_ti_n_tr_n/32/60754_2.png) [@Minh\_Ti\_n\_Tr\_n](https://discuss.elastic.co/u/Minh_Ti_n_Tr_n)\
**Post date:** [April 22, 2020, 1:24pm UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/6 "2020-04-22T13:24:19Z")

</div>

My filebeat mapping

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/4/24073553fdca21c941e6aa895b44b65820abe725.png)  
Sorry text input not allow me to put over 13000 characters

---

<div class="post-metadata">

**Author:** ![Frank\_Hassanabad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frank_hassanabad/32/49255_2.png) [@Frank\_Hassanabad](https://discuss.elastic.co/u/Frank_Hassanabad)\
**Post date:** [April 22, 2020, 1:54pm UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/7 "2020-04-22T13:54:49Z")

</div>

Yea it is looking like your setup has not installed the mapping because on a normal system if you do this in dev tools:

```auto
GET filebeat-*/_mapping

```

You get back `_meta` data at the start like so:

```auto
{
  "filebeat-7.6.0-2020.04.07-000006" : {
    "mappings" : {
      "_meta" : {
        "beat" : "filebeat",
        "version" : "7.6.0"
      },
      "dynamic_templates" : [
        {
          "labels" : {
            "path_match" : "labels.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },

```

If you were just look at say `source.ip` mapping like so:

```auto
GET filebeat-*/_mapping/field/source.ip

```

You should get back a datatype of `ip` and _not_ `text` like so:

```auto
  "filebeat-7.6.0-2020.03.27-000004" : {
    "mappings" : {
      "source.ip" : {
        "full_name" : "source.ip",
        "mapping" : {
          "ip" : {
            "type" : "ip"
          }
        }
      }
    }
  },

```

If the mapping does not initially exist and beats are pushing data indirectly what happens is Elastic Search "guesses" at your mapping by using what is referred to as dynamic mapping. The issue is that it is going to guess wrong with fields such as `source.ip` and mark them as `text`

If this is true, you will have to reindex your existing data into another index, delete this index, push the filebeats mapping correctly either directly or manually use dev tools to get your mapping set correctly, and then reindex your data back into this index. There's a lot of blog posts and articles about how to do mapping changes:

[https://www.elastic.co/blog/changing-mapping-with-zero-downtime](https://www.elastic.co/blog/changing-mapping-with-zero-downtime)

If your data is test data and not production data you can alternatively shut down beats, delete your current indexed file beat data and then push your mapping changes in, check them using dev tools to ensure things like `source.ip` are of the type `ip` and then turn filebeats back on.

---

<div class="post-metadata">

**Author:** ![Minh\_Ti\_n\_Tr\_n](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/minh_ti_n_tr_n/32/60754_2.png) [@Minh\_Ti\_n\_Tr\_n](https://discuss.elastic.co/u/Minh_Ti_n_Tr_n)\
**Post date:** [April 23, 2020, 7:50am UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/8 "2020-04-23T07:50:13Z")

</div>

Hi, I will share with you my command to fix that issue

```auto
curl -k --request PUT --user elastic https://192.168.1.17:9200/_template/default-filebeat-fromslack --data @filebeat-template.json --header 'Content-Type: application/json'

curl -k --request PUT --header 'Content-Type: application/json' --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_settings --data '{"settings": {"index.blocks.write": "true"}}'

curl -k --request POST --header 'Content-Type: application/json' --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_clone/filebeat-7.5.2-2020.04.23-1 --data '{"settings": {"index.blocks.write": null}}'

curl -k --request GET --user elastic https://192.168.1.17:9200/cluster/health/filebeat-7.5.2-2020.04.23-1?wait_for_status=green&timeout=90s

curl -k --request DELETE --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23

curl --user elastic -k --request GET https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_mappings

```

filebeat-template.json: [https://pastebin.com/dQ0GiWxC](https://pastebin.com/dQ0GiWxC)  
please change the index name  
Thanks all

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2020, 7:50am UTC](https://discuss.elastic.co/t/siem-does-not-show-data/229010/9 "2020-05-21T07:50:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.