How to do python analysis on elastic dashboard after applying filters using KQL

Elastic Stack Elasticsearch
1

For example, i have created a dashboard with many plots built on index A*, then i applied one filter by clicking on a bar chart.
(1) How can i export the filtered dataset into python and do some NLP analysis there?
(2)Even one step further - how to reflect the NLP result on the dashboard on a timely basis? like a dynamic word list? Currently i used markdown to show my still result, but it will not be refreshed when users applied a new filter on the dashboard.

response = es.search(
    index='A*',
    body={},
    size=total_docs
)

elastic_docs = response["hits"]["hits"]

#extract data from list of nested dictionaries
new_list=[i['_source'] for i in elastic_docs]
new_list_a=pandas.DataFrame(new_list)

df=new_list_a.loc[:,'text']

#generate my word list
word_list=generate_wl(df)

Currently the queries selected by users are not reflected on index A*, so the result is not automatically refreshed/ trigged by any action on the dashboard. Any guidances here will be appreciated!

2

Hi @Betty_Tian Welcome to the community.

Unfortunately I am not a python expert but perhaps someone else is.

(1) How can i export the filtered dataset into python and do some NLP analysis there?

In General you will use the scroll api to scroll through the documents from a search
See here

Here is another example : How to Paginate/Scroll Elasticsearch Data using Python - Simplernerd

I think you can find some other example if you search for : elastcisearch python scroll

(2)Even one step further - how to reflect the NLP result on the dashboard on a timely basis? like a dynamic word list? Currently i used markdown to show my still result, but it will not be refreshed when users applied a new filter on the dashboard.

I am not clear on what you are actually doing? Can you provide more details? Where is this dashboard displayed in Kibana or some other tool?