Bulk indexing of signals failed in Kibana 7.10.2

Elastic Security SIEM
1

Hello,

Just noticed our custom SIEM rules seem to throw errors, such as:

Bulk Indexing of signals failed: reason: "No mapping found for [@timestamp] in order to sort on" type: "query_shard_exception" name: "Elastic Auditing - Builtin Account - elastic" id: "301a5e03-6f0c-418f-9fd0-74393335e9dc" rule id: "8b8506a6-69c6-4299-8702-ba7ab55229f4" signals index: ".siem-signals-default"

image
image2092×627 142 KB

Is this related to https://github.com/elastic/kibana/issues/79865 ?

Can I solve it in 7.10?

Checking the mapping of @timestamp in .siem-signals-*

GET .siem-signals-default/_mapping/field/@timestamp

{
  ".siem-signals-default-000001" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  ".siem-signals-default-000002" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  ".siem-signals-default-000003" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  ".siem-signals-default-000004" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  }
}

The weird thing is that if I remember right, this was still working in 7.10.1. So this might have started when we updated to 7.10.2.

Grtz

Willem

2

It might be related to this pull request, which is tagged for v7.11.0 release.

3

Ok, so in 7.10.2 detection rules are not working then? Is there a way to fix it or do I have to wait for 7.11?

4

Ok, very weird, today I check these rules again and seems they suddenly work again.. Afaik nothing has changed, no idea what's going on..

image
image2461×467 85.6 KB

5

Yes, the Multiple timestamp fields should help fix things in the future but we don't want you to be broken now either.

If you have multiple filebeat indexes form your query which has filebeat-* I would do something like this to see if one is missing the timestamp:

GET filebeat-*/_mapping/field/@timestamp

By looking at each of those mappings you can see if you have a missing @timestamp or not to eliminate that as a possibility.

Also in the next release we are doing partial failures which should allow detections to proceed if one bad index is added:

Edit: Changed my wording that the above PR should be helpful with the upcoming release

2 Likes
6

@Frank_Hassanabad Well, as I said currently things are working again:

today I check these rules again and seems they suddenly work again.. Afaik nothing has changed, no idea what's going on..

Checked our filebeat-* indices and I didn't notice any indices with different or missing timestamp mappings:

GET filebeat-*/_mapping/field/@timestamp
{
  "filebeat-7.10.1-2020.12.31-000028" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2020.12.30-000026" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2020.12.31-000027" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.2-2021.01.28-000007" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.01-000029" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.2-2021.01.19-000001" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.2-2021.01.21-000002" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.2-2021.01.26-000006" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.2-2021.01.25-000005" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.2-2021.01.22-000003" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.2-2021.01.24-000004" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.07-000036" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.08-000037" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.04-000034" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.05-000035" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.11-000039" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.10-000038" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.01-000030" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.02-000031" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.03-000033" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.03-000032" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.16-000042" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.13-000040" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.14-000041" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.17-000043" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2021.01.19-000044" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  },
  "filebeat-7.10.1-2020.12.30-000025" : {
    "mappings" : {
      "@timestamp" : {
        "full_name" : "@timestamp",
        "mapping" : {
          "@timestamp" : {
            "type" : "date"
          }
        }
      }
    }
  }
}

Anyway, it works now, so let's assume it was a temporary glitch which might be prevented by the PR above.

Grtz

Willem

7

Sometimes miracles do happen. :rofl:

1 Like
8

Thanks for the update, I brought it up with several people and we're double checking a few things from our end and making sure from this post that even with the possibility of hiccups that can happen during say an index rollover or an agent coming online which publishes an odd mixed in mapping without a valid timestamp we do only a partial fail in our next upcoming release which will be more like a warning you could have missed signals from xyz-index rather than a total failure.

The more pointed we can be with our messages and resiliency, the better we're going to be is what we're shooting for. We don't want to fail where you get no signals and didn't see errors, but also we don't want to completely fail (if possible) and not execute against the indexes because something odd happened to one or more of them spontaneously.

Instead we are working towards just pointing out the flaw in the index but still processing any others in the list and only for that one rule run. If you see this happen again please reach out.

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.