How to do python analysis on elastic dashboard after applying filters using KQL

For example, i have created a dashboard with many plots built on index A*, then i applied one filter by clicking on a bar chart.
(1) How can i export the filtered dataset into python and do some NLP analysis there?
(2)Even one step further - how to reflect the NLP result on the dashboard on a timely basis? like a dynamic word list? Currently i used markdown to show my still result, but it will not be refreshed when users applied a new filter on the dashboard.

response = es.search(
    index='A*',
    body={},
    size=total_docs
)

elastic_docs = response["hits"]["hits"]

#extract data from list of nested dictionaries
new_list=[i['_source'] for i in elastic_docs]
new_list_a=pandas.DataFrame(new_list)

df=new_list_a.loc[:,'text']

#generate my word list
word_list=generate_wl(df)

Currently the queries selected by users are not reflected on index A*, so the result is not automatically refreshed/ trigged by any action on the dashboard. Any guidances here will be appreciated!

Hi @Betty_Tian Welcome to the community.

Unfortunately I am not a python expert but perhaps someone else is.

(1) How can i export the filtered dataset into python and do some NLP analysis there?

In General you will use the scroll api to scroll through the documents from a search
See here

Here is another example : How to Paginate/Scroll Elasticsearch Data using Python - Simplernerd

I think you can find some other example if you search for : elastcisearch python scroll

(2)Even one step further - how to reflect the NLP result on the dashboard on a timely basis? like a dynamic word list? Currently i used markdown to show my still result, but it will not be refreshed when users applied a new filter on the dashboard.

I am not clear on what you are actually doing? Can you provide more details? Where is this dashboard displayed in Kibana or some other tool?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.