Missing Integrations from Installed integrations

matled · January 29, 2023, 6:58pm

Elastic-Stack: 8.6.1

We have encountered an issue where Installed Integrations are missing from the Installed Integration page of Fleet Integrations. As an example, the Windows Integration isn't shown as an installed integration. However when we navigate to an agent policy containing the windows integration we see it. If we click on it we also see that the integration was updatable and we were able to upgrade the integration.

The same issue is for the System integration. We've tried to reinstall by clicking on the reinstall button of the integration however the integration is still not tracked.

It seems like there is a constellation where integrations are installed but not correctly tracked by the elastic integration. We've recently migrated to new fleet server if this could be related.

matled · January 29, 2023, 7:07pm

For example, Osquery Integration is stuck completely. new Version is available but we cannot upgrade

Rios · January 29, 2023, 8:23pm

Why you cannot upgrade? Any trace in logs? Please provide more details.

matled · January 29, 2023, 9:07pm

What I've tried

Restarting both Elastic Agents with the Fleet role
Removing and readding osquery to policy, results in:

image1564×458 51 KB

Currently looking for the fleet-server logs, would the following be useful?
/opt/Elastic/Agent/data/elastic-agent-b8553c/logs/elastic-agent-20230129.ndjson

Wasn't able to spot anything suspicious there...

Currently, it seems as the upgrade process is locked somehow but I don't know why and where to look for to release or debug this

Cristina_Amico · January 30, 2023, 11:34am

Hi @matled,

I'm going to ask you a few questions first to better understand the context.

Did you upgrade kibana from a previous version? If yes, which one?
Do you use a space different than "default"? I'm asking because in previous versions there have been cases when using a space other than Default causes some integrations to disappear from the UI. However this bug should be solved with 8.6.1

First I would try to force the reinstall of the integration. Please go to dev tools on the left sidebar and execute this command from there:

POST kbn:/api/fleet/epm/packages/osquery_manager/1.6.0
{"force":true}

Could you then paste the result of this command here?

Thanks,
Cristina

matled · January 30, 2023, 1:19pm

Hi Cristina

Yes we've upgraded the system from 8.5.3 to 8.6.1. Our configuration is:

7 Elasticsearch nodes (2 Hot, 2 Warm, 3 Cold)
2 Kibana Nodes
2 Elastic Fleet Nodes
Sharding in general 1 Primary 1 Replica

Yes we use different Spaces and I've tried to Upgrade OsQuery on the Space where it was active. I've seen the issue you've described. Currently OsQuery is not active in Kibana and it looks like a unconfigured installation

Input

POST kbn:/api/fleet/epm/packages/osquery_manager/1.6.0
{"force":true}

Output

{
  "statusCode": 409,
  "error": "Conflict",
  "message": "Concurrent installation or upgrade of osquery_manager-1.6.0 detected, aborting. Original error: Saved object [tag/fleet-pkg-osquery_manager-nml-betrieb] conflict"
}

The account I've used had superuser permissions.

I've restarted the elastic agents with the fleet integration and both kibana instances but it didn't resolve this issue. I'm thinking about removing all OsQuery Manager Integrations form the agent policy, however I'm still cautious because we hade issues reinstalling OsQuery Integrations in the past where stale osquery processes blocked a possible reinstall so I have to test this first. Maybe there is another way to unblock this.

LG
Matthias

matled · January 30, 2023, 1:49pm

Well, I should have read the output more carefully

We have about 5 Spaces. OsQuery was used in only one space. Looking into the saved object gives me a headache somehow. I'll try to summarize it a bit.

In all Spaces were some OsQuery objects
Some objects were pretty new (somehow tried to reinstall the integration on its own)
Some objects were really old and might never been managed correctly i guess (?)

some impressions:

As we were not able to use the feature I've removed all the objects related to osquery. Afterwards, I was able to upgrade!

Running the command again seemed to finally upgrade the integration correctly.

input

POST kbn:/api/fleet/epm/packages/osquery_manager/1.6.0
{"force":true}

output

{
  "items": [
    {
      "id": "osquery_manager-69f5ae20-eb02-11e7-8f04-51231daa5b05",
      "type": "dashboard"
    },
    {
      "id": "osquery_manager-c0a7ce90-f4aa-11e7-8647-534bb4c21040",
      "type": "dashboard"
    },
    {
      "id": "osquery_manager-1da1ed30-eb03-11e7-8f04-51231daa5b05",
      "type": "visualization"
    },
    {
      "id": "osquery_manager-240f3630-eb05-11e7-8f04-51231daa5b05",
      "type": "visualization"
    },
    {
      "id": "osquery_manager-2d6e0760-f4ab-11e7-8647-534bb4c21040",
      "type": "visualization"
    },
    {
      "id": "osquery_manager-6ec10290-f4aa-11e7-8647-534bb4c21040",
      "type": "visualization"
    },
    {
      "id": "osquery_manager-a9fd8bb0-eb01-11e7-8f04-51231daa5b05",
      "type": "visualization"
    },
    {
      "id": "osquery_manager-ab587180-f4a9-11e7-8647-534bb4c21040",
      "type": "visualization"
    },
    {
      "id": "osquery_manager-ffdbba50-f4a9-11e7-8647-534bb4c21040",
      "type": "visualization"
    },
    {
      "id": "osquery_manager-0fe5dc00-f49b-11e7-8647-534bb4c21040",
      "type": "search"
    },
    {
      "id": "osquery_manager-3824b080-eb02-11e7-8f04-51231daa5b05",
      "type": "search"
    },
    {
      "id": "osquery_manager-7a9482d0-eb00-11e7-8f04-51231daa5b05",
      "type": "search"
    },
    {
      "id": "osquery_manager-b5d6baa0-eb02-11e7-8f04-51231daa5b05",
      "type": "search"
    },
    {
      "id": "osquery_manager-f59e21e0-eb03-11e7-8f04-51231daa5b05",
      "type": "search"
    },
    {
      "id": "osquery_manager-03e88290-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-07fe8000-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-0c09a800-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-0f652f10-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-135ccf10-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-190860a0-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-1fc03210-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-35f10af0-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-3b28cc10-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-3f96fba0-a6df-11ec-b2f9-c732a3845c54",
      "type": "osquery-pack-asset"
    },
    {
      "id": "osquery_manager-0796f890-b4a9-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-0f61edf0-17e1-11ed-89c6-331eb0db6d01",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-128b90b0-b4a6-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-157d5550-fd27-11ec-8645-83a23bc513b5",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-239dce60-b4a9-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-23af51c0-d75f-11ec-879b-83915b27217e",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-2de24900-b4a9-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-315bfda0-d75f-11ec-879b-83915b27217e",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-363d6a30-b4a9-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-3e553650-17fd-11ed-89c6-331eb0db6d01",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-3e7155d0-0db5-11ed-a49c-6b13b058b135",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-47d96fe0-d75f-11ec-879b-83915b27217e",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-55955db0-0c07-11ed-a49c-6b13b058b135",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-5c144ac0-b4a5-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-63c1fe20-176f-11ed-89c6-331eb0db6d01",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-6fc00190-b4b4-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-7ee71870-b4b4-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-83869f40-0dab-11ed-a49c-6b13b058b135",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-a08d7320-1823-11ed-89c6-331eb0db6d01",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-a8870ff0-b4a5-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-b0683c20-0dbb-11ed-a49c-6b13b058b135",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-ccd3f850-b4a5-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-cebd7b00-b4b4-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-e640e200-b4a8-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-ee586dc0-1801-11ed-89c6-331eb0db6d01",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-f8649710-b4a8-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "osquery_manager-fc4e34b0-b4a5-11ec-8f39-bf9c07530bbb",
      "type": "osquery-saved-query"
    },
    {
      "id": "logs-osquery_manager.result-1.6.0",
      "type": "ingest_pipeline"
    },
    {
      "id": "logs-osquery_manager.result",
      "type": "index_template"
    },
    {
      "id": "logs-osquery_manager.result@package",
      "type": "component_template"
    },
    {
      "id": "logs-osquery_manager.result@custom",
      "type": "component_template"
    }
  ],
  "_meta": {
    "install_source": "registry"
  }
}

The integration is there again

However still not tracked as installed

To summarize it from the user perspective

Some Integrations, in our case Windows, System, some Cisco Integrations, OsQuery Manager are not actively tracked in the Integrations pace under "All Installed". We are able to select the agent integration policies and upgrade them but this is tedious and it would be better having them tracked correctly. They would have been displayed under the "Updates available"
Upgrading OsQuery Integration from the previous version to 1.6.0 wasn't possible due to some saved objects issues. Removing some saved objects related to osquery seems to resolve the issue.
There are many saved objects that look like unmanaged or stale. This somehow creates confusion if the objects are still needed or can be deleted. Not sure if they are not tracked correctly.

Cristina_Amico · January 30, 2023, 2:08pm

Removing some saved objects related to osquery seems to resolve the issue.

That's exactly the workaround I was going to suggest. I'm glad that you found it.

Looking at the error message, I think that this could be an occurrence of [Fleet] Integration installation issues in multi-space Kibana environment · Issue #143388 · elastic/kibana · GitHub.

This issue was fixed with 8.5.1, so I find it strange that it's still happening. Perhaps you have upgraded Kibana from a version < 8.5.3 ? Sometimes when Kibana get upgraded from a version to another, it creates weird edge cases like this one, which are also very difficult to track.

Thanks,
Cristina

matled · January 30, 2023, 2:55pm

I've looked up the Version of Kibana it is on 8.6.1 on both instances. I've just restarted them one by one again but wasn't able to spot anything suspicious .

I'm still not able to upgrade the individual OsQuery Manager Integrations as they are not tracked.

I'm aware about the Issue with the Default Space, I think we had some Issues in Version 7 where it wasn't even possible to customize from Default to any other Space using OsQuery Manager.

I'll post again if I have something new but I'm happy about any debugging support

matled · January 30, 2023, 3:16pm

I was able to remove and readd the OsQuery Manager Integration for a Agent Policy. The upgrade worked but it is still not tracked as Installed in the Fleet Integration page

system · February 27, 2023, 3:17pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
IIS integration dashboards missing Elastic Agent integrations	3	248	November 15, 2023
Elastic Integrations fail to install and lead to broken Dashboards when used with multiple Kibana Spaces Kibana fleet	7	736	September 19, 2023
Elastic agent integration stuck on upgrade \| fleet affected Elastic Agent	5	426	February 20, 2023
Fleet Elastic Agent Upgrade only partially available Kibana fleet	8	235	April 1, 2024
Fleet server stuck upgrading 8.5.3 -> 8.6.0 Elastic Agent fleet	6	1345	February 16, 2023

Missing Integrations from Installed integrations

Related Topics