We have to index standards FHIR document which depends on the FHIR resource the same field name could be a string or a nested json. We cannot change the name of the fields neither create an elastic index for each FHIR resource (there is a lots of them). What would be the solution with elastic for this issue??
For instance .... the Appointment resource (elastic Type) has a field "priority" with value "low" and the Encounter resource (elastic Type) has a field "priority" with value { codesystem: "http:/...." , value="4"}
So if we insert a Encounter into the elastic Type Encounter (ok) and later we try to insert a Appointment ... We get the follow error:
[priority] is defined as a field in mapping [Appointment] but this name is already used for an object in other types.
Please, What would be the solution with elastic for this issue??
All the fields in one index share the same underlying mapping. So even if two fields are in two different types, they share the same mapping and must be the same data type. It's just the way things are, no getting around it.
Could you prefix all the fields with their "type"? E.g. instead of "priority", you'd have "appointment_priority" and "encounter_priority"? That would give each field it's own distinct mapping and resolve the conflict.
It’s Is no possible because we have to store the standard fhir document (it’s no a custom format which we could change). See below a couple of examples (there are a lots more). I have attached 2 json documents as example which we are trying to store in the same index.
When you has said “Put that field in a different index” … do you really want to say “Store these json documents in a diffent index” don’t you ??
Is there any restriction about the limit number of index supported or if has impact in terms of performance to have many indexes??
I have CC to my managers. We have evaluating elastic, and we would need check if we can solve this issue to select elastics as our search engine (and get licenses) or continue evaluating other alternatives.
Correct. So for example, you'd have an 'appointment' index and an 'encounter' index. Each will have their own mapping, so there is no conflict. You can still search across both if required, although some functionality (like parent/child/nested) won't work cross-index.
Technically, no, there isn't a limit. There are practical limits. Every index comes with a certain overhead, so too many indexes will create a lot of wasted resources due to overhead. Generally tens or even hundreds of indices is ok. Thousands of indices is starting to become a problem. Tens of thousands+ will definitely be problematic.
So it's a spectrum. But having many indexes is not necessarily a problem if it is a reasonable number. For example, some people have time-based data and do an index per day. Then they retain 100 days (i.e. 100 indices) before deleting old data. Just as an example of a very common setup with many indices.
Just a note: if you're storing your data in another source (database, etc) it isn't uncommon for the "search index" to take liberties with the data format. E.g. while you cannot change the source format, it is often useful to transform the data when mirroring it into the search engine, be it Elasticsearch or any other engine. That's because search engines operate on different principles, so the format that is ideal in a relational database is often not ideal to a search engine. Search engines prefer flatter hierarchies for example.
Just something to keep in mind. It's very common for users to do this transformation to get the best performance/flexibility out of their search engine.
Thanks a lots ! We think hundreds should be enough On the other hand... What do you mean with with "although some functionality (like parent/child/nested) won't work cross-index." Could you show an example please ?
There are a set of relational queries (parent/child and nested) which rely on the documents being queried to coexist in the same index. These queries attempt to add a bit of relational functionality into an otherwise flat search engine, but come with various caveats such as requiring them to be in one index.
So for example, the parent of a child document cannot live in a different index. Both parent and child must be in the same index.
I'd generally try to avoid the relation queries anyway, since they are relatively limited in functionality. If you need heavy relational support, a true RDBM is better
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
On the other hand... What do you mean with with "although some functionality (like parent/child/nested) won't work cross-index." Could you show an example please ?