Timeline result of events not showing

Hi everyone.

Today i upgraded my ELK stack to 7.12.1 and everything seems to be working fine except for the SIEM timeline.

I ingest logs from Cisco FTD firewalls, F5 loadbalancers and our infoblox (DHCP/DNS).

When i search for an ip address i can see i get 143 events but the event list stays empty. Can anyone explain why this is happening?

The timeline was working fine in 7.6.

This is the response i get:

I would look at the mappings for that index above of permimeter-fw-000411. It probably has text fields for source.ip rather than the data type of ip.

You can see mappings from dev tools using the query of GET permimeter-fw-000411/_mapping

You will want to make sure those fields adhere to the ECS guidelines for data types within the mapping:

In the meantime I will reach out and see if timeline isn't showing partial results when it should be even though it has some errors with some indexes. I don't know if that is a desired behavioral change if that is true.

Hi Frank, thanks for the reply,

I just checked the mappings (im using the default mappings following ECS standard) and the source.ip field is of type IP.
These are the mappings for everything related to source. in this particular index:

        "source" : {
          "properties" : {
            "address" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "as" : {
              "properties" : {
                "number" : {
                  "type" : "long"
                },
                "organization" : {
                  "properties" : {
                    "name" : {
                      "type" : "keyword",
                      "ignore_above" : 1024,
                      "fields" : {
                        "text" : {
                          "type" : "text",
                          "norms" : false
                        }
                      }
                    }
                  }
                }
              }
            },
            "bytes" : {
              "type" : "long"
            },
            "domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "continent_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "country_iso_code" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "country_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "location" : {
                  "type" : "geo_point"
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "region_iso_code" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "region_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "ip" : {
              "type" : "ip"
            },
            "mac" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "nat" : {
              "properties" : {
                "ip" : {
                  "type" : "ip"
                },
                "port" : {
                  "type" : "long"
                }
              }
            },
            "packets" : {
              "type" : "long"
            },
            "port" : {
              "type" : "long"
            },
            "registered_domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "service" : {
              "properties" : {
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                }
              }
            },
            "top_level_domain" : {
              "type" : "keyword",
              "ignore_above" : 1024
            },
            "user" : {
              "properties" : {
                "domain" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "email" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "full_name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text",
                      "norms" : false
                    }
                  }
                },
                "group" : {
                  "properties" : {
                    "domain" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "id" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    },
                    "name" : {
                      "type" : "keyword",
                      "ignore_above" : 1024
                    }
                  }
                },
                "hash" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "id" : {
                  "type" : "keyword",
                  "ignore_above" : 1024
                },
                "name" : {
                  "type" : "keyword",
                  "ignore_above" : 1024,
                  "fields" : {
                    "text" : {
                      "type" : "text",
                      "norms" : false
                    }
                  }
                }
              }
            }
          }
        }

Another thing i noticed is that the generated mappings of this index are generated by filebeat version 7.8.1, could this be an issue?

"perimeter-fw-000411" : {
    "mappings" : {
      "_meta" : {
        "beat" : "filebeat",
        "version" : "7.8.1"
      }

It could be, we don't test that much with 7.8.1 these days.

You could try and see what you have for this setting and maybe play around with subtracting indexes to see if that helps narrow things down maybe?

I have allready been playing around with the data source selection.
image

I have 3 indexes that are filled by 3 filbeat modules (cisco/F5/infoblox). So they are using the ECS format.

When i disable the perimeter-fw-* as datasource and leave the f5* and infoblox* enabled i get the following error:

"failures": [
      {
        "shard": 0,
        "index": "infoblox-00001",
        "node": "FBzt0gwOTc6kfZa5K3PL0g",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [message] in order to load field data by uninverting the inverted index. Note that this can use significant memory."
        }

So now its giving the same error only for the infoblox index.

When i disable the f5* index and only leave infoblox* enabled i get to see my result in the events list.

I think I might have figured out what is going on here and reproduced this. If you could, can you tell me what your mappings are for your f5* message field like so from dev tools:

GET f5*/_mapping/field/message

Paste the results here and we can see if message is a keyword vs a text field. I think the bug is where we have both a text field in 1 index and then a keyword in another index when using columns for timeline.

If that's true, your workaround would be to re-index any indexes that have message into a text field per ECS guidelines.

EDIT: Looking like removing message as a column might not work here if this is the case, so I removed that suggestion.

Here is the Kibana ticket I am getting a peer review on:

Thanks for the feedback. This is the mapping for the message field:

{
  "f5-logging-000003" : {
    "mappings" : {
      "message" : {
        "full_name" : "message",
        "mapping" : {
          "message" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      }
    }
  }
}

So it seems that the mappings are not correct. This is really strange since im using the default mappings and configuration from the latest filebeat client.

I just tried to remove the message collumn but this wont fix the issue. The events are still not shown.

Ok and update from my side. It seems that the F5 index is causing the problems. Its the only index that has keyword as type for the message field.

I just corrected this in the index template. Can i just re-index the F5 indexes now?

Yes, you should re-index and hopefully be ok now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.