Automate ingest through a pipeline (ingest attachment processor)

hello there.
Does anyone know if there a tool something like beats to ingest human created files such a pdf or word files (dockx) using my attachment pipeline that processes files with ingest attachment pluing

You can use the ingest attachment plugin.

There an example here: https://www.elastic.co/guide/en/elasticsearch/plugins/current/using-ingest-attachment.html

PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information",
  "processors" : [
    {
      "attachment" : {
        "field" : "data"
      }
    }
  ]
}
PUT my_index/_doc/my_id?pipeline=attachment
{
  "data": "e1xydGYxXGFuc2kNCkxvcmVtIGlwc3VtIGRvbG9yIHNpdCBhbWV0DQpccGFyIH0="
}
GET my_index/_doc/my_id

The data field is basically the BASE64 representation of your binary file.

But you need to crawl yourself the filesystem.

You can use FSCrawler. There's a tutorial to help you getting started.

hi @dadoonet thanks for your response. i know about the plugin and how to used,but so far have only seen this the use of ingest-attachment through a pipeline using API calls. what i want is a tool that do the ingest for me. i tried filebeats and it reads my system logs on its own. that is what im looking for, something to read the my files stored in my device or in a server or even a cloud storage without the need to write code to do the readings of files turned in base64 encoded and then send the files to be ingested.

Did you read the last part of my answer?