We are evaluating ELK stack for log analysis and visualization
ES 2.x,
Logstash2.1
and Kibana 4.
Does anyone happen to know how to track the user search queries made on kibana search bar. I am more interested to know about the non-saved searches from the user.
I understand Kibana stores the search queries , dashboards and visualization in .kibana index.
Is there a way to pull what the user had searched from kibana interface?
Can you help to understand with an example if thats possible to pull out from ES indexes?
While there isn't a specific way to tie into the act of using the search bar in the Kibana UI, you could examine your standard web server access logs to track search requests to elasticsearch. Kibana proxies searches directly to elasticsearch, so you could look at the access logs for your kibana webserver to see the searches to elasticsearch.
Thanks for the reply !! We are not using any web server other than nginx to reverse proxy kibana. However, nginx doesnt help to track the user search on the kibana search bar. All i can see is few GET and POST methods
Every time a person searches in Kibana, it results in a couple of HTTP requests to the Kibana backend which then proxy to elasticsearch. The request you want to look at is the "msearch" request.
On my local kibana setup, when I filter my data based on the extension "jpg", I get the following request:
POST http://localhost:5601/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1453303185156
{"index":["logstash-2016.01.20"],"ignore_unavailable":true}
{"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"":{}},"require_field_match":false,"fragment_size":2147483647},"query":{"filtered":{"query":{"query_string":{"query":"","analyze_wildcard":true}},"filter":{"bool":{"must":[{"query":{"match":{"extension":{"query":"jpg","type":"phrase"}}}},{"range":{"@timestamp":{"gte":1453302324311,"lte":1453303224311,"format":"epoch_millis"}}}],"must_not":[]}}}},"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30s","time_zone":"America/New_York","min_doc_count":0,"extended_bounds":{"min":1453302324308,"max":1453303224309}}}},"fields":["*","_source"],"script_fields":{},"fielddata_fields":["@timestamp","utc_time","relatedContent.article:modified_time","relatedContent.article:published_time"]}
As you can see, that post body includes both the "filters" and the "query" that are used in my search via the search and filter bars in Kibana.
We had achieved this audit feature via a proxy servlet - all the requests were intercepted by the proxy to check if its a search query and accordingly audited.
Thanks Court for the detaield investigation.. I do see POST requests on my nginx access log, however i dont see detailed query parameters logged in the logs.
I had to explicitly add request_body to NGINX log_format to gather POST request parameters. I can see the query string now. Thanks much for the pointer.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.