Automate ingest through a pipeline (ingest attachment processor)

Elastic Community and Ecosystem
1

hello there.
Does anyone know if there a tool something like beats to ingest human created files such a pdf or word files (dockx) using my attachment pipeline that processes files with ingest attachment pluing

2

You can use the ingest attachment plugin.

There an example here: https://www.elastic.co/guide/en/elasticsearch/plugins/current/using-ingest-attachment.html

PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    }
  ]
}
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
GET my_index/_doc/my_id

The data field is basically the BASE64 representation of your binary file.

But you need to crawl yourself the filesystem.

You can use FSCrawler. There's a tutorial to help you getting started.

3

hi @dadoonet thanks for your response. i know about the plugin and how to used,but so far have only seen this the use of ingest-attachment through a pipeline using API calls. what i want is a tool that do the ingest for me. i tried filebeats and it reads my system logs on its own. that is what im looking for, something to read the my files stored in my device or in a server or even a cloud storage without the need to write code to do the readings of files turned in base64 encoded and then send the files to be ingested.

4

Did you read the last part of my answer?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.