How to search for a complete URL and Create a Visualization for counts


(Sameer Panicker) #1

I am not able to search for URL since it has slashes. i.e. https://123.123.123/MyService/api/calls

Also,
How can I create a visualization for searched URL counts i.e. how many times a particular URL was invoked ? So the chart should show "X Axis" -> all the diff. URLs , "Y Axis"-> Total Count with color separation of each server?


(Matt Bargar) #2

Try using this syntax in the filter bar: <field-name>: "url"

You can create the visualization you're looking for by using a Filter aggregation. It'll look something like this:


(Sameer Panicker) #3

Here is my service log structure. In below scenario I should be getting "http://servername:8080/MyService/api/orders/phonesearch" URL count as 2. Will the above solution work here ?
Should I use : "url" or it will be something else for me, such as requestUri : "url".

Also, I need to know how much time my service responded as OK/Failure. i.e. reasonPhrase.

Log 1 -

EventId : 1, Level : Informational, Message : Request, Payload : [sessionID :** 9abab532-5f4e-4050-97e4-416e423cd6f3**] [method : POST] [requestUri : http://servername:8080/MyService/api/orders/phonesearch] [content : {"areaCode":"651","countryCode":"01","number":"2911000"}] , EventName : RequestInfo, Timestamp : 2016-03-11T18:20:42.4351450Z, ProcessId : 7768, ThreadId : 4516

EventId : 2, Level : Informational, Message : Response, Payload : [sessionID : 9abab532-5f4e-4050-97e4-416e423cd6f3] [reasonPhrase : ApplicationException] [content : [{}]

Log 2 -

EventId : 1, Level : Informational, Message : Request, Payload : [sessionID :** 97bab532-5f4e-4050-97e4-416e423cd6f3**] [method : POST] [requestUri : http://servername:8080/MyService/api/orders/phonesearch] [content : {"areaCode":"651","countryCode":"01","number":"2911000"}] , EventName : RequestInfo, Timestamp : 2016-03-11T18:20:42.4351450Z, ProcessId : 7768, ThreadId : 4516

EventId : 2, Level : Informational, Message : Response, Payload : [sessionID : 97bab532-5f4e-4050-97e4-416e423cd6f3] [reasonPhrase : OK] [content : [{"created":"2011-10-19T16:34:57", "isComplete":"false","isSaleVoided":"false"}]


(Matt Bargar) #4

You should be able to do:

requestUri: "http://servername:8080/MyService/api/orders/phonesearch"

For your second question, on the visualize tab a terms aggregation or a filter aggregation will probably be what you're looking for, depending on whether you want to see which reasonPhrse values are the "top n" or if you want to see counts for specific values.


(Sameer Panicker) #5

How can i combine my reasonphrase: OK and requestURI: url in the filter aggregation ?

Is there a sample link which i can refer to ?


(Sameer Panicker) #6

Just following up on this.


(Matt Bargar) #7

You should be able to simply combine the clauses like this:

requestURI: "uri" AND reasonPhrase: "OK"

Also, if you need to do anything really complex, you can always click on the "Advanced" dropdown on the filters aggregation editor and use the full JSON query language: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-filters-aggregation.html


(Sameer Panicker) #8

Now, I am trying to read the IIS Logs for pulling the service hits and success/failure count. Since, it has everything in well organized manner and I was suggested to do the same.

2016-03-13 23:59:37 188.14.34.149 GET /MyService/api/gettimes/350 - 4433 - 111.11.122.192 Java/1.8.0_45 - 200 0 0 0

2016-03-13 23:59:37 188.14.34.149 GET /MyService/api/gettimes/350 - 4433 - 111.11.122.192 Java/1.8.0_45 - 400 0 0 0

Can you please help me here to get this into Visualization?


(Sameer Panicker) #9

Is there a way that I can use Regex to pull this info. I have created this

.?(GET|POST|PUT|DELETE).?(/MyService/api/gettimes/350).?((1.)|(2.)|(3.)|(4.)|(5.))


(Sameer Panicker) #10

Any help on this ?


(Matt Bargar) #11

Sorry for the incredibly late reply, for some reason I didn't get an email from Discourse when these replies came in and I just noticed them in my unread list.

To answer your question, breaking up a field like that is something you'll need to do at ingestion time. Logstash's grok would be the natural choice, or in 5.0 you can use the new ingest node feature in Elasticsearch.


(Sameer Panicker) #12

Thanks for the reply Bargs.

I got this working already using GROKs. :slight_smile:


(Matt Bargar) #13

Awesome, glad you got it working. Sorry again for ridiculously late reply. Now I know not to rely on email notifications :wink:


(Sameer Panicker) #14

Its ok. Glad you replied.


(system) #15