How to search for a complete URL and Create a Visualization for counts

Elastic Stack Kibana
1

I am not able to search for URL since it has slashes. i.e. https://123.123.123/MyService/api/calls

Also,
How can I create a visualization for searched URL counts i.e. how many times a particular URL was invoked ? So the chart should show "X Axis" -> all the diff. URLs , "Y Axis"-> Total Count with color separation of each server?

2

Try using this syntax in the filter bar: <field-name>: "url"

You can create the visualization you're looking for by using a Filter aggregation. It'll look something like this:

Screen Shot 2016-03-11 at 4.53.36 PM.png1984×1364 141 KB

3

Here is my service log structure. In below scenario I should be getting "http://servername:8080/MyService/api/orders/phonesearch" URL count as 2. Will the above solution work here ?
Should I use : "url" or it will be something else for me, such as requestUri : "url".

Also, I need to know how much time my service responded as OK/Failure. i.e. reasonPhrase.

Log 1 -

EventId : 1, Level : Informational, Message : Request, Payload : [sessionID :** 9abab532-5f4e-4050-97e4-416e423cd6f3**] [method : POST] [requestUri : http://servername:8080/MyService/api/orders/phonesearch] [content : {"areaCode":"651","countryCode":"01","number":"2911000"}] , EventName : RequestInfo, Timestamp : 2016-03-11T18:20:42.4351450Z, ProcessId : 7768, ThreadId : 4516

EventId : 2, Level : Informational, Message : Response, Payload : [sessionID : 9abab532-5f4e-4050-97e4-416e423cd6f3] [reasonPhrase : ApplicationException] [content : [{}]

Log 2 -

EventId : 1, Level : Informational, Message : Request, Payload : [sessionID :** 97bab532-5f4e-4050-97e4-416e423cd6f3**] [method : POST] [requestUri : http://servername:8080/MyService/api/orders/phonesearch] [content : {"areaCode":"651","countryCode":"01","number":"2911000"}] , EventName : RequestInfo, Timestamp : 2016-03-11T18:20:42.4351450Z, ProcessId : 7768, ThreadId : 4516

EventId : 2, Level : Informational, Message : Response, Payload : [sessionID : 97bab532-5f4e-4050-97e4-416e423cd6f3] [reasonPhrase : OK] [content : [{"created":"2011-10-19T16:34:57", "isComplete":"false","isSaleVoided":"false"}]

4

You should be able to do:

requestUri: "http://servername:8080/MyService/api/orders/phonesearch"

For your second question, on the visualize tab a terms aggregation or a filter aggregation will probably be what you're looking for, depending on whether you want to see which reasonPhrse values are the "top n" or if you want to see counts for specific values.

5

How can i combine my reasonphrase: OK and requestURI: url in the filter aggregation ?

Is there a sample link which i can refer to ?

6

Just following up on this.

7

You should be able to simply combine the clauses like this:

requestURI: "uri" AND reasonPhrase: "OK"

Also, if you need to do anything really complex, you can always click on the "Advanced" dropdown on the filters aggregation editor and use the full JSON query language: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filters-aggregation.html

8

Now, I am trying to read the IIS Logs for pulling the service hits and success/failure count. Since, it has everything in well organized manner and I was suggested to do the same.

2016-03-13 23:59:37 188.14.34.149 GET /MyService/api/gettimes/350 - 4433 - 111.11.122.192 Java/1.8.0_45 - 200 0 0 0

2016-03-13 23:59:37 188.14.34.149 GET /MyService/api/gettimes/350 - 4433 - 111.11.122.192 Java/1.8.0_45 - 400 0 0 0

Can you please help me here to get this into Visualization?

9

Is there a way that I can use Regex to pull this info. I have created this

.?(GET|POST|PUT|DELETE).?(/MyService/api/gettimes/350).?((1.)|(2.)|(3.)|(4.)|(5.))

10

Any help on this ?

11

Sorry for the incredibly late reply, for some reason I didn't get an email from Discourse when these replies came in and I just noticed them in my unread list.

To answer your question, breaking up a field like that is something you'll need to do at ingestion time. Logstash's grok would be the natural choice, or in 5.0 you can use the new ingest node feature in Elasticsearch.

12

Thanks for the reply Bargs.

I got this working already using GROKs. :slight_smile:

13

Awesome, glad you got it working. Sorry again for ridiculously late reply. Now I know not to rely on email notifications :wink:

14

Its ok. Glad you replied.