Streaming Binary files’s decoded data to elastic :)

I am wondering what tools I can use for streaming decoded data from binary files to elastic the fastest way.

Here is what I am trying to do.
files are located at directory .
Those files are binary encoded files to save space. After decoding each file I get human readable structs (in c++). A lot of regular c++ structs.
I want my directory to be continuously checked for new files automatically and then I will decode the new files with my own decoder code into structs . every struct object is a record in elastic.

Please help me , it’s important

I don't know how this would help but may be look at FSCrawler project.
It "monitors" a directory and index its content.

If you have your own decoding process, you need I guess to implement everything by yourself. You could always use FSCrawler source code as a start.


Thank u for answering :slight_smile:

I read about it but unfortunately ,my code is written in C++😓

But I need to upload decoded data to elastic and not the binary files themselves

And I need it to recognize and process new files without delete the old ones

I don’t want to make any changes in destination directory because another programmers use it for another use cases

I have my own decoding process

do u know if I can use Logstash
or another tool for this use case?:pray:t2:

Can't you translate your code to Java? The languages are super closed.
I'm not aware of anything out of the box which would help.

I have exists folder updated with lots of new binary files all the time. I can’t change their format or replace it . And the format is wrriten in a very complicated way in c++ headers and there r some technicians to save space
So I have to read the data with the c++ format (the way it was wrriten)
I can’t translate it with java because it was wrriten in c++ sizes. :disappointed_relieved:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.