Store data into elasticsearch without parsing

Elastic Stack Beats
,
1

Good day,

I'm setting an elasticsearch as a datalake for AI/ML purposes, I want to have all data in text blobs unparsed, so the AI scripts can work with unstructured data for an NLP project.

I have no problems with the custom logs/data I send using custom index/mappings since I put everything in a specific flattened field with type text.

Now my problem arises when I try to send windows logs extracted with winbeat, since it uses its own processors to parse/map all data into multiple fields, and I don't think re-writting all processors is the way to go, specially since I plan to use more out-of-the-box beats to ingest logs from other providers.

I'm trying to look for a solution to put all data of a document into a single field, doesn't have to be in the same index, preferible without using logstash due to HW constrains, any sugestions?

I'm thinking into moving the data from the pre-made winbeat index to a new index with the flattened mapping, but I'm not sure on how to do that either and if it would work or it would be the best way to do so.

Best regards,

2

Winlogbeat assumes you want everything parsed and doesn't give you an option to skip this inbuilt step and assume that you want to do it elsewhere (or not do it in your case).

I can't think of a good way to do this unfortunately :frowning:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.