Strategies for having changes in graph data being reflected in ElasticSearch index

Hi all,

I'm investigating possible strategies for the following situation:

I have data in a graph where the nodes and edges represent knowledge about
certain topics. These topics may occur in unstructured text. The knowledge
about these topics is used in an analysis process to make sense of
unstructured text. The analysis results are indexed in ElasticSearch. The
graph is stored simply in MySQL for now. It's not really large (about 4000
nodes and 4000 edges/relationships), but the expectation is that this will
grow substantially.

The most important part of the analysis process involves identifying how
well topics are represented in the unstructured text. This is done based on
a number of rules which are represented in the knowledge graph. The
analysis results of a single piece of unstructured text consists of a list
of identified topics as well as a number of characteristics per topic. A
topic is considered to be well-represented when it is found by more rules
coming from the knowledge graph. I.e. a piece of text can have a topic to
be represented if it meets a single rule, but if a second piece of text has
the same topic represented by meeting 10 rules, the seconds document should
score better in search results.

Searching the analysis results through ElasticSearch is performed using a
combination of filters and queries. Score is calculated using a function
score query. The script score part of this uses document fields (the
characteristics for each topic) as well as a number of parameters in the

When I search for the data, the query contains a number of topics I wish to
search for (let's say 40 topics) and finds documents that match best. I am
getting the right results when I search the data, which is great.

The only issue I have is the following: The knowledge in the graph is
updated regularly. Updates to the graph are required to be reflected in the
scoring of documents in the ElasticSearch index, leading to better search

There are different strategies to have the changes to the graph reflected
in the scoring by ElasticSearch:

  • Periodically re-analyse all pieces of unstructured text and index the
    results in ElasticSearch again
    - A lot of precalculations are performed
    and stored in the ElasticSearch index. An index alias could be used to
    switch between a "live" and "rebuilding" index. The benefit here is that it
    is easy to implement and the queries are really fast as like <50ms as much
    is precalculated. The drawback here is that changes in the graph are only
    reflected in the ElasticSearch search scoring after a period of time (in my
    case about 8 hours) as the analysis process takes long to perform.

  • Move parts of the analysis process to query-execution time by
    dynamically building a filter+query using the knowledge graph to identify
    the topics and calculate the characteristics where possible on the fly
    using function score queries with script scores. The benefit is that the
    changes in the graph do not always require periodic updates to the entire
    index. The drawback here is that if a graph section used to build the query
    has lots of related nodes, the resulting query DSL becomes huge and has
    lots of bool clauses. This requires overhead to programmatically construct
    the query, provide it to ElasticSearch and ElasticSearch also takes longer
    to perform the query (800 milliseconds). Going this route I have queries
    which are about 2 megabytes and contain 4000+ boolean clauses.

My wish is that I have changes updated asap in ElasticSearch. Within a
couple of seconds is fine.

I am wondering if there are other strategies possible. I hope the above
clarifies my challenges enough for you to answer, but ask away if you have
questions. I just can't detail too much because of non-disclosure :slight_smile:

I'm open to using other technologies aside ElasticSearch, and ElasticSearch

Kind regards,


You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To view this discussion on the web visit
For more options, visit