Kibana - Failed to query events data

[panagiss]

Here are the full errors:

That is the error i see in KIbana when i try to visualize auditbeat data from auditbeat index.

i cannot visualize anything in Kibana now from that index.
The error states:

type": "illegal_argument_exception",
"reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [host.name] in order to load field data by uninverting the inverted index. Note that this can use significant memory."

[matw (Matthias Wilhelm)]

Hi

Which version of Auditbeat are you using and could you share the mapping of one of the indices you are trying to query events data?

Thx!
Matthias

[panagiss]

I use auditbeat-7.13.2.

Should it be automated? The mapping i mean. Because it's an auditbeat index. So if i understood you correctly you want the whole json mapping index management.
Here it is:

{
  "mappings": {
    "_doc": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "hostname": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "auditd": {
          "properties": {
            "data": {
              "properties": {
                "acct": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "audit_enabled": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "cmd": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "entries": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "family": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "hostname": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "old": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "old-ses": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "op": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "table": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "terminal": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "tty": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "unit": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "message_type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },

.
.
.
.
.
.
.

It doesn't let me paste the whole mappings Json so i have pasted a portion and the whole Json is in here

The name of the index is ** auditbeat-7.13.2** and i create it from logstash output config like this:

 elasticsearch { 
        hosts => ["https://localhost:9200"] 
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        user => logstash_internal
        password => logstashpass
        ssl => true
        cacert => "/usr/share/logstash/config/elasticsearch-ca.pem"
        http_compression => true
	 	sniffing => false 
    } 

[matw (Matthias Wilhelm)]

Could you try to visualize by host.name.keyword?
Also I'm not sure you're using the audit beat index template, it looks more like the default ES one, have a look here:

Load the Elasticsearch index template | Auditbeat Reference [7.14] | Elastic

[panagiss]

How I'm going to do that? Whenever i go to kibana i see this

And are the same error i send you before.

How I'm going to see the mapping attached to an index? Here are the mapping templates

and here are the Indexes

Also to point out that i didn't make any changes to the default ones.
Just instead of sending auditbeat straight to ES, it now goes first to Logstash, but there is no change in the fields.

[panagiss]

Anyone could know why these error occur ?

[stephenb (Stephen Brown)]

Looks like you did not run setup.

setup sets up the correct mappings, ingest pipeline etc.

Hopefully your are following the steps closely in the Quick start guide

Setup see here

Auditbeat Quick start guide here

If you do not run setup your mappings will not be correct which is what your issue looks like to me.

Also if you Architecture is

Auditbeat > Logstash > Elasticsearch you need to follow this process.

This example is metricbeat but same process

Metricbeat to Logstash to ElasticSearch - Cannot See Any Hosts Defined, But Data Is Definitely Coming In

Hi @jthart Welcome to the community apologies that you're having some struggles getting this set up Perhaps we can help. Assuming you want to run an architecture like this Metricbeat (1 to Many) -> Logstash -> Elasticsearch Basically using Logstash as a collect and pass through Here is my recommendation try to resist the urge to make this more complex. Do not try to manually load index templates dashboards anything else follow the quick start / basic setup. Clean everything up we're star…

Also I would get

Auditbeat > Elasticsearch working first then introduce Logstash

[panagiss]

The solution wasn't obvious. I had already done the auditbeat setup thing. So i made the output of Auditbeat straight to ES.
And then i noticed the error resource .... exists, but it is not an alias so i solved in on that thread.

So the problem was that i deleted the auditbeat-7.13.2 index and i created another one via logstash when i had the output like this:

output { 

    elasticsearch { 
        hosts => ["https://localhost:9200"] 
        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
        user => logstash_user
        password => apass
        ssl => true
        cacert => "/usr/share/logstash/config/elasticsearch-ca.pem"
        http_compression => true
	 	sniffing => false 
    } 
}

So i get HOW this happened but even after i read that thread on github, i don't fully understand WHY is that behavior happening. I mean i just Deleted the index and logstash created automatically...

Can someone explain it ?