Timeline result of events not showing

Kupauw · May 25, 2021, 12:28pm

Hi everyone.

Today i upgraded my ELK stack to 7.12.1 and everything seems to be working fine except for the SIEM timeline.

I ingest logs from Cisco FTD firewalls, F5 loadbalancers and our infoblox (DHCP/DNS).

When i search for an ip address i can see i get 143 events but the event list stays empty. Can anyone explain why this is happening?

The timeline was working fine in 7.6.

This is the response i get:

Frank_Hassanabad · May 25, 2021, 5:28pm

I would look at the mappings for that index above of permimeter-fw-000411. It probably has text fields for source.ip rather than the data type of ip.

You can see mappings from dev tools using the query of GET permimeter-fw-000411/_mapping

You will want to make sure those fields adhere to the ECS guidelines for data types within the mapping:

In the meantime I will reach out and see if timeline isn't showing partial results when it should be even though it has some errors with some indexes. I don't know if that is a desired behavioral change if that is true.

Kupauw · May 25, 2021, 6:26pm

Hi Frank, thanks for the reply,

I just checked the mappings (im using the default mappings following ECS standard) and the source.ip field is of type IP.
These are the mappings for everything related to source. in this particular index:

        "source" : {
          "properties" : {
            "address" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "as" : {
              "properties" : {
                "number" : {
                  "type" : "long"
                },
                "organization" : {
                  "properties" : {
                    "name" : {
                      "type" : "keyword",
                      "ignore_above" : 1024,
                      "fields" : {
                        "text" : {
                          "type" : "text",
                          "norms" : false
                        }
                      }
                    }
                  }
                }
              }
            },
            "bytes" : {
              "type" : "long"
            },
            "domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "continent_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "country_iso_code" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "country_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "location" : {
                  "type" : "geo_point"
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "region_iso_code" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "region_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "ip" : {
              "type" : "ip"
            },
            "mac" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "nat" : {
              "properties" : {
                "ip" : {
                  "type" : "ip"
                },
                "port" : {
                  "type" : "long"
                }
              }
            },
            "packets" : {
              "type" : "long"
            },
            "port" : {
              "type" : "long"
            },
            "registered_domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "service" : {
              "properties" : {
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "top_level_domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "user" : {
              "properties" : {
                "domain" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "email" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "full_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text",
                      "norms" : false
                    }
                  }
                },
                "group" : {
                  "properties" : {
                    "domain" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "id" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "name" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    }
                  }
                },
                "hash" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text",
                      "norms" : false
                    }
                  }
                }
              }
            }
          }
        }

Another thing i noticed is that the generated mappings of this index are generated by filebeat version 7.8.1, could this be an issue?

"perimeter-fw-000411" : {
    "mappings" : {
      "_meta" : {
        "beat" : "filebeat",
        "version" : "7.8.1"
      }

Frank_Hassanabad · May 25, 2021, 7:04pm

It could be, we don't test that much with 7.8.1 these days.

You could try and see what you have for this setting and maybe play around with subtracting indexes to see if that helps narrow things down maybe?

Kupauw · May 26, 2021, 12:46pm

I have allready been playing around with the data source selection.

I have 3 indexes that are filled by 3 filbeat modules (cisco/F5/infoblox). So they are using the ECS format.

When i disable the perimeter-fw-* as datasource and leave the f5* and infoblox* enabled i get the following error:

"failures": [
      {
        "shard": 0,
        "index": "infoblox-00001",
        "node": "FBzt0gwOTc6kfZa5K3PL0g",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [message] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
        }

So now its giving the same error only for the infoblox index.

When i disable the f5* index and only leave infoblox* enabled i get to see my result in the events list.

Frank_Hassanabad · May 26, 2021, 3:26pm

I think I might have figured out what is going on here and reproduced this. If you could, can you tell me what your mappings are for your f5* message field like so from dev tools:

GET f5*/_mapping/field/message

Paste the results here and we can see if message is a keyword vs a text field. I think the bug is where we have both a text field in 1 index and then a keyword in another index when using columns for timeline.

If that's true, your workaround would be to re-index any indexes that have message into a text field per ECS guidelines.

EDIT: Looking like removing message as a column might not work here if this is the case, so I removed that suggestion.

Here is the Kibana ticket I am getting a peer review on:

github.com/elastic/kibana

[Security Solution][Timeline] Timeline does not show all data when message is indexed as a keyword

opened 04:16PM - 26 May 21 UTC

FrankHassanabad

Feature:Timeline Team:Threat Hunting bug

**Describe the bug:** When we have mixed `keyword` and `text` fields for a lo…g message, timeline will not show inconsistent data results due to elastic search having partial errors and not returning all of the data set in some instances. Timeline doesn't indicate to the user that a partial error has happened through a toaster error. Timeline UI/UX ends up showing inconsistent data results and different numbers as seen by the screen shots below. However, if the user clicks on the inspect button after they notice that the UI is showing inconsistent data they will see the partial errors. As an aside, when doing an audit of `logs-*`, I saw that we mappings that are mixed on test systems and I think we might be shipping non-ECS compliant mappings where in some cases message is `keyword` instead of `text`. For example these index patterns contain `keyword` instead of `text` (even though ECS says the [field](https://www.elastic.co/guide/en/ecs/current/ecs-base.html) should be `text`): Audit all logs-* for data type of message ``` GET logs-*/_mapping/field/message ``` Notice these two are `keyword` and not `text` ``` GET .ds-logs-system.application-default-2021.02.24-000001/_mapping/field/message GET .ds-logs-system.system-default-2021.02.24-000001/_mapping/field/message ``` **Expected behavior:** If possible, users would like timeline to bring back all the data across multiple indexes even if they have different data types such as `keyword` and `text` for the field `message`. We might have mistakes in log mappings today which if so would mean that our logs have mixed data types with regards to message so anything we can do to correct this within timeline (if possible) would be great as well as directly within the logs mappings as well. I don't know if timeline wants to show error toasters for its UI/UX when it gets partial errors (we don't do that on a lot of UI) but this is leading to false and misleading UI/UX where we are not showing all the data when we get partial errors. At the very least, I think users want to see consistent UI/UX data numbers. **Steps to reproduce:** 1. Go to dev tools and add these mappings and data ```json # This has a keyword field message DELETE test-index-delme PUT test-index-delme { "mappings": { "dynamic": "strict", "properties": { "@timestamp": { "type": "date" }, "message": { "type": "keyword" }, "source": { "properties": { "ip": { "type": "ip" } } } } } } # This has a text field for message DELETE test-text-index-delme PUT test-text-index-delme { "mappings": { "dynamic": "strict", "properties": { "@timestamp": { "type": "date" }, "message": { "type": "text" }, "source": { "properties": { "ip": { "type": "ip" } } } } } } PUT test-index-delme/_doc/1 { "@timestamp": "2021-05-26T15:32:04.869Z", "message": "message 1 from keyword field" } PUT test-index-delme/_doc/2 { "@timestamp": "2021-05-26T15:30:50.440Z", "message": "message 2 from keyword field" } PUT test-text-index-delme/_doc/1 { "@timestamp": "2021-05-26T15:28:35.234Z", "message": "message 1 from text field", "source": { "ip": "127.0.0.1" } } PUT test-text-index-delme/_doc/2 { "@timestamp": "2021-05-26T15:26:35.234Z", "message": "message 2 from text field", "source": { "ip": "127.0.0.2" } } ``` Create the kibana index patterns for both of those and then use them within Kibana timeline: <img width="687" alt="Screen Shot 2021-05-26 at 10 05 37 AM" src="https://user-images.githubusercontent.com/1151048/119693878-f3406980-be09-11eb-90ed-781df5ce55fa.png"> Query for `source.ip: *` with that given date time range and then you can see that it is showing the following mixture of data and no data: <img width="1666" alt="Screen Shot 2021-05-26 at 10 06 05 AM" src="https://user-images.githubusercontent.com/1151048/119694478-7bbf0a00-be0a-11eb-91d0-c04959c46017.png"> If you query for `*:*` it shows only 2 of the 4 rows but says there are 4 records. <img width="1667" alt="Screen Shot 2021-05-26 at 10 10 23 AM" src="https://user-images.githubusercontent.com/1151048/119694786-c93b7700-be0a-11eb-9c82-0244c9add33f.png"> Message if you click the inspect button: ```json { "took": 6, "timed_out": false, "_shards": { "total": 2, "successful": 1, "skipped": 0, "failed": 1, "failures": [ { "shard": 0, "index": "test-text-index-delme", "node": "m88rV4HIS7uEbH2ZTe4G1g", "reason": { "type": "illegal_argument_exception", "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [message] in order to load field data by uninverting the inverted index. Note that this can use significant memory." } } ] }, "hits": { "total": 4, "max_score": null, "hits": [ { "_index": "test-index-delme", "_id": "1", "_score": null, "_source": {}, "fields": { "message": [ "message 1 from keyword field" ], "@timestamp": [ "2021-05-26T15:32:04.869Z" ] }, "sort": [ 1622043124869 ] }, { "_index": "test-index-delme", "_id": "2", "_score": null, "_source": {}, "fields": { "message": [ "message 2 from keyword field" ], "@timestamp": [ "2021-05-26T15:30:50.440Z" ] }, "sort": [ 1622043050440 ] } ] } } ``` **Workaround** * Update all the mappings to be text instead of keyword and in any dynamic ILM policies or * Remove the indexes or separate the indexes across Kibana spaces so the keyword ones don't mix with the text field ones. **Kibana version:** At least 7.12.1 and master **Elasticsearch version:** At least 7.12.1 and master

Kupauw · May 27, 2021, 8:27am

Thanks for the feedback. This is the mapping for the message field:

{
  "f5-logging-000003" : {
    "mappings" : {
      "message" : {
        "full_name" : "message",
        "mapping" : {
          "message" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      }
    }
  }
}

So it seems that the mappings are not correct. This is really strange since im using the default mappings and configuration from the latest filebeat client.

I just tried to remove the message collumn but this wont fix the issue. The events are still not shown.

Kupauw · May 27, 2021, 8:39am

Ok and update from my side. It seems that the F5 index is causing the problems. Its the only index that has keyword as type for the message field.

I just corrected this in the index template. Can i just re-index the F5 indexes now?

Frank_Hassanabad · May 27, 2021, 8:54pm

Yes, you should re-index and hopefully be ok now.

system · June 24, 2021, 8:54pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.