Visualizing API endpoint hit counts in Kibana

[afcooper]

Hi, I'm new to Kibana and ELS. I'm working on a python app that uses Flask. The api endpoint logs are being dumped into a generic "log" field that is not cleanly parseable. I should be able to search for endpoints, and then attach them to a visualization. Kibana isn't quite getting it right when there is a variable portion of the url, like a user id or name.

For example, I'm able to query for something of the format "/api/users/", but i am not able to query for something of the format "/api/users/123/posts".

Any help would be greatly appreciated. I'm trying to evaluate the efficacy of Kibana as a tool, and without being able to extract simple endpoint hit counts I'm not able to get very far.

[ppisljar (Peter Pisljar)]

i might be wrong, but i think your problem is with indexing. You should index your data in a way that you can easily query it. take a look at logstash grok plugin if this can help https://www.elastic.co/guide/en/logstash/5.4/plugins-filters-grok.html

you can use regex to parse your data .... you could also write a python script that would correctly parse your logs and index them in elasticsearch.

[afcooper]

Thanks for the prompt response. I'm indexing using "logstash-*". I'd like to be able to then drill down further to endpoint hit logs emitted by flask. The format of those logs looks a lot like the following format:

logstash-plugins/logstash-patterns-core/blob/master/patterns/httpd#L5

It's unclear to me how to leverage these patterns in Kibana. The same goes for regex. I'm using plain old regex syntax, and getting parse errors for things as simple as trying to match 1 or more a-z characters. This seems to be more of a tooling confusion for me than anything else.

Per my example in the original post, how would you search using regex (or a predefined logstash pattern, like the following component of httpd common log - "?:%{WORD:verb} %{NOTSPACE:request}" ) to search for endpoint hits with variable alphanumeric ids in them? An example query would be really helpful for me to learn from, since that's where Kibana is behaving unpredictably for me.

For example, using ES syntax, even something as simple as this yields strange results:

{ "match" : { "message" : "GET /api" } }

I also retrieve records for all my "POST /api" logs

Thanks.

[afcooper]

Update: It looks like Kibana is automatically analyzing fields, and there isn't a super obvious way through the UI to create an index with unanalyzed fields. The analysis seems to be the source of the problem. Any advice for how to create an index in Kibana without analysis turned on?