Visualizations has errors default page

bevano · August 17, 2020, 4:22am

Hi All,

Just started to play around with SIEM. When opening the menu, we are constantly presented with this error. The only logs we are ingesting is our building security logs, and I have manually created logstash filters for this. All events are tied to fields founds in the ECS.

Is there a way we resolve these errors?

stephmilovic · August 17, 2020, 2:02pm

Hi @bevano, thanks for posting. It looks like one of your fields is mapped as text and it needs to be mapped as a keyword in order to aggregate in ES. Could you please post the mapping of your index and we can take a look at what field it might be?

Frank_Hassanabad · August 17, 2020, 7:35pm

Yeah this often happens if you haven't pushed your index templates and you can see us answering this here with some pointers on what to do:

I don't know if it's your winlogbeat, audtibeat, some custom one, etc... but usually if you can get your templates and ECS mapping pushed up and then do a reindex, data reset, you will be ok.

bevano · August 18, 2020, 12:38am

Thanks @stephmilovic and @Frank_Hassanabad for the reply.

And yes, i haven't pointed it to a template since I was just playing around. After re-reading this https://www.elastic.co/guide/en/ecs/current/index.html under hosts, i can see the type for each of the fields.

I assume if I follow those types in the guide i should be fine?

Frank_Hassanabad · August 18, 2020, 2:05am

Depends on what you're doing. If you're writing your own downstream agent/beat/script that is going to parse logs and push it as ECS you can use existing tools from the ECS repo here:

and from their tooling guide:

github.com

elastic/ecs/blob/master/USAGE.md

# ECS Tooling Usage

In addition to the published schema and artifacts, the ECS repo also contains tools to generate artifacts based on the current published and custom schemas.

You may be asking if ECS is a specification for storing event data, where does the ECS tooling fit into the picture?  As users implement ECS into their Elastic stack, common questions arise:

* ECS has too many fields. Users don't want to generate mappings for fields they don't plan on using soon.
* Users want to adopt ECS but also want to painlessly maintain their own custom field mappings alongside ECS.

Users can use the ECS tools to tackle both problems. What artifacts are relevant will also vary based on need. Many users will find the Elasticsearch templates most useful, but Beats
contributors will instead find the Beats-formatted YAML field definition files valuable. By maintaining only their customizations and use the tools provided by ECS, they can generate
relevant artifacts for their unique set of data sources.

**NOTE** - These tools and their functionality are considered experimental.

## Table of Contents

- [Terminology](#terminology)
- [Setup and Install](#setup-and-install)
  * [Prerequisites](#prerequisites)

This file has been truncated. show original

If you want to start without using the tooling you can just push in a mapping you customize using the dev tooling of Kibana from the ECS generated template here:

github.com

elastic/ecs/blob/master/generated/elasticsearch/7/template.json

{
  "index_patterns": [
    "ecs-*"
  ],
  "mappings": {
    "_meta": {
      "version": "2.0.0-dev"
    },
    "date_detection": false,
    "dynamic_templates": [
      {
        "strings_as_keyword": {
          "mapping": {
            "ignore_above": 1024,
            "type": "keyword"
          },
          "match_mapping_type": "string"
        }
      }
    ],

This file has been truncated. show original

If you're playing around with beats such as auditbeat, metricbeat, etc... you can do a setup and run one of the beats and then look at the mappings and ILM index lifecycle management (ILM) policies they have as good examples as well.

Those things will probably save you a lot of time. Have fun and let us know how it goes!

bevano · August 18, 2020, 2:09am

Perfect. Majority of it will be done using the existing beats/agent modules but there are some custom stuff that we need ingested.

Thank you very much for the info. Much appreciated

system · September 15, 2020, 2:09am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.