Kibana Plugins Degraded

francescouk · July 15, 2022, 2:57pm

Hi there,
I´ve done a few searchs for this kind of solution and I could not find anywhere. Can someone help me?

Jul 15 12:00:36 kbn kibana[3899]: [2022-07-15T12:00:36.438-03:00][ERROR][plugins.securitySolution] Bulk Indexing of signals failed: index: ".ds-logs-checkpoint.firewall-default-2022.07.14-000024" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-elastic_agent.endpoint_security-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-elastic_agent.filebeat-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-elastic_agent.fleet_server-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-elastic_agent.osquerybeat-default-2022.07.14-000004" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-endpoint.events.file-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-endpoint.events.library-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-endpoint.events.network-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-endpoint.events.process-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-endpoint.events.registry-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-endpoint.events.security-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-osquery_manager.result-default-2022.07.14-000004" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-system.application-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-system.security-default-2022.07.15-000032" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-system.system-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-ti_abusech.malware-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-ti_abusech.malwarebazaar-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-ti_abusech.url-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-windows.powershell-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception",index: ".ds-logs-windows.powershell_operational-default-2022.07.15-000030" reason: "failed to create query: end-of-string expected at position 1" type: "query_shard_exception" caused by reason: "end-of-string expected at position 1" caused by type: "illegal_argument_exception" name: "Possible DC Shadow" id: "b9e5a5d0-fe1b-11ec-b0f4-c9dfbc3bebf9" rule id: "32e19d25-4aed-4860-a55a-be99cb0bf7ed" execution id: "566dae0a-dc79-488b-8d70-51898fed8b98" space ID: "default"

Yesterday was everything fine so... have no idea how to sort this out.

Thanks for the attention.

LeeDr · July 15, 2022, 3:24pm

Hi Francesco,

Can you check your Elasticsearch cluster for any issues. One problem that can happen is that disk drives get above a high water mark and that makes indices turn to read-only mode. If you have monitoring enabled and if that's working, that would be a good place to check.

If Kibana monitoring isn't enabled or isn't working because of those degraded plugins, you might have to use curl commands something like this (you may have to add -u username:password);

curl -XGET "https://localhost:9200/_cluster/health" -H "kbn-xsrf: reporting"

and

curl -XGET "https://localhost:9200/_nodes/stats" -H "kbn-xsrf: reporting"

If those don't reveal any problems, it looks like there might be a problem with a rule. It could be a built-in rule or a custom rule. If you have rules you could try disabling them and see if the issue clears up.

Please let us know if you find the problem.

Regards,
Lee

francescouk · July 15, 2022, 3:34pm

Hi @LeeDr ,
I´ve done a few checks on the cluster and looks like is all fine:

{
  "cluster_name": "om-tvt",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 241,
  "active_shards": 482,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_millis": 0,
  "active_shards_percent_as_number": 100
}

LeeDr · July 15, 2022, 4:06pm

Are you using the Security features in Kibana? Creating rules, etc? That error log you posted is all about securitySolution plugin. I'm looking in to how to disable that in case it's something you're not using. But if you're using it and created rules, that would take us down a different path to troubleshoot.

francescouk · July 15, 2022, 4:22pm

Yes, I´m!

I´m using the builtin rules plus a few other rules I had. A part from the status page, everything looks like is working fine. So I dunno if this is something that I really need to look at or not.

Maybe some rules are not working as expected? Or the results that I´m receiving is not fully correct....

I´ll try to disable all of them and then see if makes any difference towards this problem....

system · August 12, 2022, 4:22pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.