SIEM does not show data

[Minh_Ti_n_Tr_n (Minh Tiến Trần)]

I got the problem:
Fielddata is disabled on text fields by default. Set fielddata=true on [source.ip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead
I tried on dev tool

PUT filebeat-*/_mapping
{
  "properties": {
    "my_field": { 
      "type":     "text",
      "fielddata": true
    }
  }
}

and got:

{
  "acknowledged" : true
}

But till not see the data on kibana. Then I tried:

PUT filebeat-*/_mapping/text
{
   "your_type": {
      "properties": {
        "publisher": {
          "type": "text",
          "fielddata": true
        }
      }
   }
}

And got:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
  },
  "status": 400
}

Any solution to make data show on kibana please!
Thanks & Regards

[Mike_Paquette]

Hi, sorry to see you're having trouble getting your data to be displayed in the SIEM app.

I am assuming from your dev tool screen shots that you are using Filebeat. Is this correct?
I see from another post that you were using NetFlow data from pfsense. Is this the same situation?

Sorry to go back to the start, but can you tell us how you set up Filebeat to send data to your cluster? Did you enable the NetFlow module per the typical process shown below?

If so, you should not have to manually adjust the Elasticsearch index mappings.

Also, is it possible for you to send a screen shot of one of your NetFlow documents taken from Kibana Discover? (Please be sure not to include any confidential information). We'd like to see what fields are included and their datatypes.

[Frank_Hassanabad]

@Minh_Ti_n_Tr_n,

It looks like you haven't pushed a set of mappings and that's why it is giving you this message below. Although this advice it is giving you looks "good" on the surface it really isn't and is not what you want to do to fix your mapping issues.

Fielddata is disabled on text fields by default. Set fielddata=true on [source.ip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead

^ Avoid doing this in favor of using correct mappings. This has serious consequences and not what you want.

What you want to do is to "push" your filebeat template mappings and ILM poloicies if your beats have access to your Elasticsearch cluster like so:

Load the Elasticsearch index template | Filebeat Reference [8.11] | Elastic

You might have to overwrite your current mappings that you have for filebeat and you might have to reindex your data as well once completed. I can't make assumptions about how sensitive or the volume of your data so I can only recommend you back up anything important first through snapshots and then perform any and all steps to get a valid mapping.

Here are earlier threads that go into more detail about it that others have encountered when they accidentally do not set their mappings below. In a nutshell when your templates and mappings are not setup what is happening is that all your data is "auto creating mappings" for its data types such as "source.ip" being mapped to a "text field" when in reality it should be a field type of "ip".

SIEM doesn't show any Winlogbeat events, despite ES receiving them

@Aura, ahhh, I think I might see what's going on. I bet when you first setup winlog beat you accidentally forgot to push your templates which control the mapping? https://www.elastic.co/guide/en/beats/winlogbeat/current/winlogbeat-template.html You can check your mapping in your dev tools: [Screen Shot 2020-04-10 at 7.47.05 AM] like so....To see the entire mapping of your winlog beats: GET winlogbeat-7.6.0/_mapping [Screen Shot 2020-04-10 at 7.48.30 AM] If you want to just concentrate on…

[Minh_Ti_n_Tr_n (Minh Tiến Trần)]

I will show everything here
My filebeat mapping

Filebeat.yml

# Wazuh - Filebeat configuration file
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

#filebeat.config.inputs:
#  enabled: true
#  path: inputs.d/*.yml

filebeat.config.modules:
  enabled: true
  path: /etc/filebeat/modules.d/*.yml

setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.template.overwrite: true
setup.ilm.enabled: false

output.elasticsearch.hosts: ['https://192.168.1.17:9200']
output.elasticsearch.username: "elastic"
output.elasticsearch.password: "<password>"
output.elasticsearch.ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"]
output.elasticsearch.ssl.certificate: "/etc/elasticsearch/certs/node-0.crt"
output.elasticsearch.ssl.key: "/etc/elasticsearch/certs/node-0.key"

Module enabled

netflow.yml

# Module: netflow
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.5/filebeat-module-netflow.html

- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 2055

Then I try the tutorial via link: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually-alternate

My Kibana discover but dont have indice

Thanks

[Frank_Hassanabad]

You didn't show me your mapping above? You just showed me a screen shot of where you tried to modify your mapping to turn on fielddata which is still not really recommended.

I think you wanted to show me from line 23 and below which would be:

GET filebeat-*/_mapping

However, if you have a direct connection from filebeat you can follow the commands from here:

https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#setup-command

to setup your templates such as:

filebeat setup

Then you should see that your have indexes, ILM policies, etc... setup

[Minh_Ti_n_Tr_n (Minh Tiến Trần)]

My filebeat mapping

Sorry text input not allow me to put over 13000 characters

[Frank_Hassanabad]

Yea it is looking like your setup has not installed the mapping because on a normal system if you do this in dev tools:

GET filebeat-*/_mapping

You get back _meta data at the start like so:

{
  "filebeat-7.6.0-2020.04.07-000006" : {
    "mappings" : {
      "_meta" : {
        "beat" : "filebeat",
        "version" : "7.6.0"
      },
      "dynamic_templates" : [
        {
          "labels" : {
            "path_match" : "labels.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },

If you were just look at say source.ip mapping like so:

GET filebeat-*/_mapping/field/source.ip

You should get back a datatype of ip and not text like so:

  "filebeat-7.6.0-2020.03.27-000004" : {
    "mappings" : {
      "source.ip" : {
        "full_name" : "source.ip",
        "mapping" : {
          "ip" : {
            "type" : "ip"
          }
        }
      }
    }
  },

If the mapping does not initially exist and beats are pushing data indirectly what happens is Elastic Search "guesses" at your mapping by using what is referred to as dynamic mapping. The issue is that it is going to guess wrong with fields such as source.ip and mark them as text

If this is true, you will have to reindex your existing data into another index, delete this index, push the filebeats mapping correctly either directly or manually use dev tools to get your mapping set correctly, and then reindex your data back into this index. There's a lot of blog posts and articles about how to do mapping changes:

https://www.elastic.co/blog/changing-mapping-with-zero-downtime

If your data is test data and not production data you can alternatively shut down beats, delete your current indexed file beat data and then push your mapping changes in, check them using dev tools to ensure things like source.ip are of the type ip and then turn filebeats back on.

[Minh_Ti_n_Tr_n (Minh Tiến Trần)]

Hi, I will share with you my command to fix that issue

curl -k --request PUT --user elastic https://192.168.1.17:9200/_template/default-filebeat-fromslack --data @filebeat-template.json --header 'Content-Type: application/json'

curl -k --request PUT --header 'Content-Type: application/json' --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_settings --data '{"settings": {"index.blocks.write": "true"}}'

curl -k --request POST --header 'Content-Type: application/json' --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_clone/filebeat-7.5.2-2020.04.23-1 --data '{"settings": {"index.blocks.write": null}}'

curl -k --request GET --user elastic https://192.168.1.17:9200/cluster/health/filebeat-7.5.2-2020.04.23-1?wait_for_status=green&timeout=90s

curl -k --request DELETE --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23

curl --user elastic -k --request GET https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_mappings

filebeat-template.json: https://pastebin.com/dQ0GiWxC
please change the index name
Thanks all