Medical Data and elasticsearch

[Michael5 (Michael K)]

I am trying to find a clever approach to screen / index / search / connect all kinds of data that go throug my company's Enterprise Service Bus (WSO2). The files are mostly HL7, XML and JSON. Sometimes there are Scans (PDF) or Images (BMP) but these files are not the focus. We kept all the files passing our ESB in the past two years.

Now we feel it is time to use this data meaning make it searchable, connect and analyze it.

My main question is: Is the current Version of Elasticsearch able to work as a file dump / file store, analytics engine and presentation layer? Is there a way to deal with formats like HL7 effectively? Would it be smarter to use another product like MongoDB and search it with Elasticsearch? Or use a completely different product?

Another question is: Does anybody know of a published project that did something similar?

I would be highly interested in best practices.

Thanks

[warkolm (Mark Walkom)]

Elasticsearch is not a file store, but you could put the data in it for search and analysis.

If you need to store the data for legal compliance, you might want to look at something else.

[Michael5 (Michael K)]

Hi warkolm

Does elasticsearch only hold the indizes? Or does it hold JSON Fields + Indices?
For future searches: this paper [1] describes the use & search of HL7 files. Seems to be a good solution.

[1]https://ieeexplore.ieee.org/document/7546006/

[warkolm (Mark Walkom)]

Elasticsearch holds both.

[lonangel (Duarte Ferreira)]

Hi @Michael5,
I'm one of the authors of the paper and the idea in the paper is to use the ES as a temporary repository for data analysis and searching.
I don't think is a great idea to use ES as a PDF and image repository but you can store those files on a HDFS system and use the hadoop plugin to index metadata based on the files.