[Integration] Facing error while adding transform in integration package

Elastic Security
1

I have referred elastic-package spec to ship transform along with integration package. However, I am not able to install integration package when added transform under /root/integrations/packages/my_package/elasticsearch/transform .

It gives me below error for elastic-package install command. Although, it's passing elastic-package check command:

Install the package Error: can't install the package: can't install the package: could not install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"Unexpected token s in JSON at position 0"}

Can someone please help me with this? Does integration package support transforms yet? because I didn't find any integration package with transform.

2

@Alex_R

First a couple of questions for context: What are your goals with your integration? Is it meant to be used with Security?

the installation of transforms conforming to this new spec is new functionality and has only been possible since the 8.5 stack release in a limited way.

The package registry includes the schema for transforms now, but the installation is still in a pre-release state. It's not recommended to try this in production yet as there are transform upgrade scenarios which are not supported yet, but you may have success in simpler use cases.

Which version of the stack are you running?

If you are running 8.5+ and it's not installing, then we may need to dig a bit deeper into the transform itself. Are you OK with sharing the transform and destination index json? It will help debug the problem further.

Alternatives to the above are to add the transforms you need manually through the Transform API. Or you can also add transforms through the UI at "Stack Management > Transforms"

image
image3456×2002 460 KB

3

Thanks for responding @Kevin_Logan!

To answer your question, the primary goal of the integration is to collect threat data and as collected data will be historical, it is required to maintain the latest documents ( or updated documents). Hence, this can be achieved using the latest transform. However, as per your suggestion, we tried to bundle transforms with 8.5.0 stack and we are not getting the previous error but it is throwing the new error:

Error: can't install the package: can't install the package: could not install package; API status code = 500; response body = {"statusCode":500,"error":"Internal Server Error","message":"security_exception: [security_exception] Reason: action [indices:admin/create] is unauthorized for service account [elastic/kibana] on indices [transform-destination-index], this action is granted by the index privileges [create_index,manage,all]"}.

So I believe, this error is due to the index privileges issue and according to this open issue kibana_system must have the required privileges to install any package assets.

How can I provide privileges to my destination index, as far as I know, we can not modify the built-in roles?

4

@Alex_R

So I believe, this error is due to the index privileges issue and according to this open issue kibana_system must have the required privileges to install any package assets.

Yes, this is the case. Apologies, I forgot that we still need to resolve this permissions issue as we continue to build on transform installation through packages. I should have remembered that in my first reply. It is true that you cannot modify the permissions for the built in role. Because of these reasons, you won't be able to install via integration packages yet.

To move forward with your transforms, you will need to install them manually either through the API or the UI. See my reply from above.

Alternatives to the above are to add the transforms you need manually through the Transform API. Or you can also add transforms through the UI at "Stack Management > Transforms"

You will still be able to install a transform using latest. You will also need to install your destination index. You can try either the UI or the API mentioned above. You can also take a look at this tutorial for more guidance on how to manually install transforms.

Let me know if this helps!

5

Yes, @Kevin_Logan, We are able to add transform using APIs and it's working as expected.

In which version could we expect this issue to be resolved so we can bundle transform along with the integration package?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.