SIEM does not show data

Elastic Security SIEM
1

I got the problem:
Fielddata is disabled on text fields by default. Set fielddata=true on [source.ip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead
I tried on dev tool

PUT filebeat-*/_mapping
{
  "properties": {
    "my_field": { 
      "type":     "text",
      "fielddata": true
    }
  }
}

and got:

{
  "acknowledged" : true
}

But till not see the data on kibana. Then I tried:

PUT filebeat-*/_mapping/text
{
   "your_type": {
      "properties": {
        "publisher": {
          "type": "text",
          "fielddata": true
        }
      }
   }
}

And got:

{
  "error": {
    "root_cause": [
      {
        "type": "illegal_argument_exception",
        "reason": "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
      }
    ],
    "type": "illegal_argument_exception",
    "reason": "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
  },
  "status": 400
}

Any solution to make data show on kibana please!
Thanks & Regards

2

Hi, sorry to see you're having trouble getting your data to be displayed in the SIEM app.

I am assuming from your dev tool screen shots that you are using Filebeat. Is this correct?
I see from another post that you were using NetFlow data from pfsense. Is this the same situation?

Sorry to go back to the start, but can you tell us how you set up Filebeat to send data to your cluster? Did you enable the NetFlow module per the typical process shown below?

image
image1243×1231 92.1 KB

If so, you should not have to manually adjust the Elasticsearch index mappings.

Also, is it possible for you to send a screen shot of one of your NetFlow documents taken from Kibana Discover? (Please be sure not to include any confidential information). We'd like to see what fields are included and their datatypes.

3

@Minh_Ti_n_Tr_n,

It looks like you haven't pushed a set of mappings and that's why it is giving you this message below. Although this advice it is giving you looks "good" on the surface it really isn't and is not what you want to do to fix your mapping issues.

Fielddata is disabled on text fields by default. Set fielddata=true on [source.ip] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead

^ Avoid doing this in favor of using correct mappings. This has serious consequences and not what you want.

What you want to do is to "push" your filebeat template mappings and ILM poloicies if your beats have access to your Elasticsearch cluster like so:

You might have to overwrite your current mappings that you have for filebeat and you might have to reindex your data as well once completed. I can't make assumptions about how sensitive or the volume of your data so I can only recommend you back up anything important first through snapshots and then perform any and all steps to get a valid mapping.

Here are earlier threads that go into more detail about it that others have encountered when they accidentally do not set their mappings below. In a nutshell when your templates and mappings are not setup what is happening is that all your data is "auto creating mappings" for its data types such as "source.ip" being mapped to a "text field" when in reality it should be a field type of "ip".

4

I will show everything here
My filebeat mapping

image
image1531×398 27.4 KB

Filebeat.yml

# Wazuh - Filebeat configuration file
filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

#filebeat.config.inputs:
#  enabled: true
#  path: inputs.d/*.yml

filebeat.config.modules:
  enabled: true
  path: /etc/filebeat/modules.d/*.yml

setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.template.overwrite: true
setup.ilm.enabled: false

output.elasticsearch.hosts: ['https://192.168.1.17:9200']
output.elasticsearch.username: "elastic"
output.elasticsearch.password: "<password>"
output.elasticsearch.ssl.certificate_authorities: ["/etc/elasticsearch/certs/ca.crt"]
output.elasticsearch.ssl.certificate: "/etc/elasticsearch/certs/node-0.crt"
output.elasticsearch.ssl.key: "/etc/elasticsearch/certs/node-0.key"

Module enabled

image
image1307×119 6.29 KB

netflow.yml

# Module: netflow
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.5/filebeat-module-netflow.html

- module: netflow
  log:
    enabled: true
    var:
      netflow_host: 0.0.0.0
      netflow_port: 2055

Then I try the tutorial via link: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually-alternate

My Kibana discover but dont have indice

image
image1548×390 42 KB

Thanks

5

You didn't show me your mapping above? You just showed me a screen shot of where you tried to modify your mapping to turn on fielddata which is still not really recommended.

I think you wanted to show me from line 23 and below which would be:

GET filebeat-*/_mapping

However, if you have a direct connection from filebeat you can follow the commands from here:

https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html#setup-command

to setup your templates such as:

filebeat setup

Then you should see that your have indexes, ILM policies, etc... setup

6

My filebeat mapping

image
image1551×752 43.6 KB

Sorry text input not allow me to put over 13000 characters

7

Yea it is looking like your setup has not installed the mapping because on a normal system if you do this in dev tools:

GET filebeat-*/_mapping

You get back _meta data at the start like so:

{
  "filebeat-7.6.0-2020.04.07-000006" : {
    "mappings" : {
      "_meta" : {
        "beat" : "filebeat",
        "version" : "7.6.0"
      },
      "dynamic_templates" : [
        {
          "labels" : {
            "path_match" : "labels.*",
            "match_mapping_type" : "string",
            "mapping" : {
              "type" : "keyword"
            }
          }
        },

If you were just look at say source.ip mapping like so:

GET filebeat-*/_mapping/field/source.ip

You should get back a datatype of ip and not text like so:

  "filebeat-7.6.0-2020.03.27-000004" : {
    "mappings" : {
      "source.ip" : {
        "full_name" : "source.ip",
        "mapping" : {
          "ip" : {
            "type" : "ip"
          }
        }
      }
    }
  },

If the mapping does not initially exist and beats are pushing data indirectly what happens is Elastic Search "guesses" at your mapping by using what is referred to as dynamic mapping. The issue is that it is going to guess wrong with fields such as source.ip and mark them as text

If this is true, you will have to reindex your existing data into another index, delete this index, push the filebeats mapping correctly either directly or manually use dev tools to get your mapping set correctly, and then reindex your data back into this index. There's a lot of blog posts and articles about how to do mapping changes:

https://www.elastic.co/blog/changing-mapping-with-zero-downtime

If your data is test data and not production data you can alternatively shut down beats, delete your current indexed file beat data and then push your mapping changes in, check them using dev tools to ensure things like source.ip are of the type ip and then turn filebeats back on.

8

Hi, I will share with you my command to fix that issue

curl -k --request PUT --user elastic https://192.168.1.17:9200/_template/default-filebeat-fromslack --data @filebeat-template.json --header 'Content-Type: application/json'

curl -k --request PUT --header 'Content-Type: application/json' --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_settings --data '{"settings": {"index.blocks.write": "true"}}'

curl -k --request POST --header 'Content-Type: application/json' --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_clone/filebeat-7.5.2-2020.04.23-1 --data '{"settings": {"index.blocks.write": null}}'

curl -k --request GET --user elastic https://192.168.1.17:9200/cluster/health/filebeat-7.5.2-2020.04.23-1?wait_for_status=green&timeout=90s

curl -k --request DELETE --user elastic https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23

curl --user elastic -k --request GET https://192.168.1.17:9200/filebeat-7.5.2-2020.04.23/_mappings

filebeat-template.json: https://pastebin.com/dQ0GiWxC
please change the index name
Thanks all

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.