# Strategies for having changes in graph data being reflected in ElasticSearch index

**URL:** <https://discuss.elastic.co/t/strategies-for-having-changes-in-graph-data-being-reflected-in-elasticsearch-index/20724>\
**Category:** Elasticsearch\
**Created:** [November 13, 2014, 3:15pm UTC](https://discuss.elastic.co/t/strategies-for-having-changes-in-graph-data-being-reflected-in-elasticsearch-index/20724 "2014-11-13T15:15:57Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Eric\_van\_der\_Staaij](https://avatars.discourse-cdn.com/v4/letter/e/258eb7/32.png) [@Eric\_van\_der\_Staaij](https://discuss.elastic.co/u/Eric_van_der_Staaij)\
**Post date:** [November 13, 2014, 3:15pm UTC](https://discuss.elastic.co/t/strategies-for-having-changes-in-graph-data-being-reflected-in-elasticsearch-index/20724/1 "2014-11-13T15:15:57Z")

</div>

Hi all,

I'm investigating possible strategies for the following situation:

I have data in a graph where the nodes and edges represent knowledge about  
certain topics. These topics may occur in unstructured text. The knowledge  
about these topics is used in an analysis process to make sense of  
unstructured text. The analysis results are indexed in ElasticSearch. The  
graph is stored simply in MySQL for now. It's not really large (about 4000  
nodes and 4000 edges/relationships), but the expectation is that this will  
grow substantially.

The most important part of the analysis process involves identifying how  
well topics are represented in the unstructured text. This is done based on  
a number of rules which are represented in the knowledge graph. The  
analysis results of a single piece of unstructured text consists of a list  
of identified topics as well as a number of characteristics per topic. A  
topic is considered to be well-represented when it is found by more rules  
coming from the knowledge graph. I.e. a piece of text can have a topic to  
be represented if it meets a single rule, but if a second piece of text has  
the same topic represented by meeting 10 rules, the seconds document should  
score better in search results.

Searching the analysis results through ElasticSearch is performed using a  
combination of filters and queries. Score is calculated using a function  
score query. The script score part of this uses document fields (the  
characteristics for each topic) as well as a number of parameters in the  
formula.

When I search for the data, the query contains a number of topics I wish to  
search for (let's say 40 topics) and finds documents that match best. I am  
getting the right results when I search the data, which is great.

The only issue I have is the following: The knowledge in the graph is  
updated regularly. Updates to the graph are required to be reflected in the  
scoring of documents in the ElasticSearch index, leading to better search  
results.

There are different strategies to have the changes to the graph reflected  
in the scoring by ElasticSearch:

- _Periodically re-analyse all pieces of unstructured text and index the  
results in ElasticSearch again_ - A lot of precalculations are performed  
and stored in the ElasticSearch index. An index alias could be used to  
switch between a "live" and "rebuilding" index. The benefit here is that it  
is easy to implement and the queries are really fast as like \<50ms as much  
is precalculated. The drawback here is that changes in the graph are only  
reflected in the ElasticSearch search scoring after a period of time (in my  
case about 8 hours) as the analysis process takes long to perform.

- _Move parts of the analysis process to query-execution time_ by  
dynamically building a filter+query using the knowledge graph to identify  
the topics and calculate the characteristics where possible on the fly  
using function score queries with script scores. The benefit is that the  
changes in the graph do not always require periodic updates to the entire  
index. The drawback here is that if a graph section used to build the query  
has lots of related nodes, the resulting query DSL becomes huge and has  
lots of bool clauses. This requires overhead to programmatically construct  
the query, provide it to ElasticSearch and ElasticSearch also takes longer  
to perform the query (800 milliseconds). Going this route I have queries  
which are about 2 megabytes and contain 4000+ boolean clauses.

My wish is that I have changes updated asap in ElasticSearch. Within a  
couple of seconds is fine.

I am wondering if there are other strategies possible. I hope the above  
clarifies my challenges enough for you to answer, but ask away if you have  
questions. I just can't detail too much because of non-disclosure 🙂

I'm open to using other technologies aside ElasticSearch, and ElasticSearch  
plugins.

Kind regards,

Eric

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3d1092a7-a708-4706-bc59-df4523cab47c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3d1092a7-a708-4706-bc59-df4523cab47c%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:50am UTC](https://discuss.elastic.co/t/strategies-for-having-changes-in-graph-data-being-reflected-in-elasticsearch-index/20724/2 "2017-07-06T00:50:15Z")

</div>

Closed on July 6, 2017, 12:50am UTC