Ingesting whole files

[Rihards]

Looking for the best way to ingest whole files.
Potentially three outcomes would be of interest:
a) Ingest the whole plaintext file as a single-field document.
b) Ingest the file as k-v pairs (ini-style file).
c) Ingest XML file, parsing the nodes.

In each of these cases, looking for a way to re-read the whole file [if it has changed].

The attachment processor seems like an overkill, as it deals with binary files.

Found similar discussion topics, but didn't grasp how to achieve the desired outcome.

Log file parsing and ingesting into ES

What else is in the log file? I would expect there to be another ERROR message. Your multiline configuration looks wrong to me. It will consume the first 4 lines as one event, then the first two event elements will be flushed as events, then the last two lines. How about pattern => "</trouble_shooter_log>" negate => true what => next

How to index full xml file into ElasticSearch

Hi Magnus, We are planning to use Distributed computing framework like Spark to work on top of ES. If we store whole xml data as text in ES, and parse it, it would be double work. Splunk has a feature to work on top of xmls. If you specify a breaking String or Xpath, it will break the data repeatedly there and provides you events. To above xml, we will get list of <Record> nodes. Does ES have any such solution.? Lets say if i specify as line breaker, can it give all the events of file by br…

Are there "howtos for dummies" on such a setup?