we are using Google App Engine to host our SaaS app. Google offers a nice
log browser but it is way too sloooow. So one of my colleagues suggested we
pipe our logs to logstash and make them accessible via Kibana. So far so
good, we managed to set everything up.
But when Kibana was shown to the other team members they weren't really
excited. It was much faster, yes. It allowed to make better queries, yes.
BUT it broke the pattern they knew from the Google App Engine log browser:
we are using Google App Engine to host our SaaS app. Google offers a
nice log browser but it is way too sloooow. So one of my colleagues
suggested we pipe our logs to logstash and make them accessible via
Kibana. So far so good, we managed to set everything up.
But when Kibana was shown to the other team members they weren't
really excited. It was much faster, yes. It allowed to make better
queries, yes. BUT it broke the pattern they knew from the Google App
Engine log browser:
/some-request
log message 1
log message 2
/another-request
log message 3
/yet-another-request
log message 4
While Kibana works like this:
log message 1 /some-request
log message 2 /some-request
log message 3 /another-request
log message 4 /yet-another-request
So basically App Engine groups log messages by request. To get my
team on board, can we make Kibana do the same?
Not out of the box, no. Kibana doesn't have any such contextual
understanding of messages and currently can't be configured as
such either.
--
Magnus Bäck | Software Engineer, Development Tools magnus.back@sonymobile.com | Sony Mobile Communications
Thanks for the answer!
I think wasn't clear enough: all our log messages already have a requestID.
So if there was a grouping feature we'd apply it to that field.
I'm just wondering, how do you troubleshoot a issue of a user? When we see
a problem we look at all requests of that user in the GAE log viewer. Then
you quickly see requests that have non-200 status codes. Then we drill into
a request and see all logs of that request chronologically. While in
Kibana I can also look at all logs from a user ordered by time, but it's
not always completely clear which request log messages belong to. It's more
like one big stream.
My point is, you should really try out the Google App Engine log viewer -
then you would know what you are missing!
Stephan
On Monday, December 22, 2014 7:38:26 AM UTC+1, Magnus Bäck wrote:
On Tuesday, December 16, 2014 at 10:03 CET,
stephanos <stephan...@gmail.com <javascript:>> wrote:
we are using Google App Engine to host our SaaS app. Google offers a
nice log browser but it is way too sloooow. So one of my colleagues
suggested we pipe our logs to logstash and make them accessible via
Kibana. So far so good, we managed to set everything up.
But when Kibana was shown to the other team members they weren't
really excited. It was much faster, yes. It allowed to make better
queries, yes. BUT it broke the pattern they knew from the Google App
Engine log browser:
/some-request
log message 1
log message 2
/another-request
log message 3
/yet-another-request
log message 4
While Kibana works like this:
log message 1 /some-request
log message 2 /some-request
log message 3 /another-request
log message 4 /yet-another-request
So basically App Engine groups log messages by request. To get my
team on board, can we make Kibana do the same?
Not out of the box, no. Kibana doesn't have any such contextual
understanding of messages and currently can't be configured as
such either.
--
Magnus Bäck | Software Engineer, Development Tools magnu...@sonymobile.com <javascript:> | Sony Mobile Communications
Maybe graylog2 can interst you as a solution to store your data in ES, and
therefore you have better searching of your data.
Especially data coming form a webserver as I understand. You then can stil
search or display data with Kibana.
A.
Op maandag 22 december 2014 09:58:57 UTC+1 schreef stephanos:
Thanks for the answer!
I think wasn't clear enough: all our log messages already have a
requestID. So if there was a grouping feature we'd apply it to that
field.
I'm just wondering, how do you troubleshoot a issue of a user? When we see
a problem we look at all requests of that user in the GAE log viewer. Then
you quickly see requests that have non-200 status codes. Then we drill into
a request and see all logs of that request chronologically. While in
Kibana I can also look at all logs from a user ordered by time, but it's
not always completely clear which request log messages belong to. It's more
like one big stream.
My point is, you should really try out the Google App Engine log viewer -
then you would know what you are missing!
Stephan
On Monday, December 22, 2014 7:38:26 AM UTC+1, Magnus Bäck wrote:
On Tuesday, December 16, 2014 at 10:03 CET,
stephanos stephan...@gmail.com wrote:
we are using Google App Engine to host our SaaS app. Google offers a
nice log browser but it is way too sloooow. So one of my colleagues
suggested we pipe our logs to logstash and make them accessible via
Kibana. So far so good, we managed to set everything up.
But when Kibana was shown to the other team members they weren't
really excited. It was much faster, yes. It allowed to make better
queries, yes. BUT it broke the pattern they knew from the Google App
Engine log browser:
/some-request
log message 1
log message 2
/another-request
log message 3
/yet-another-request
log message 4
While Kibana works like this:
log message 1 /some-request
log message 2 /some-request
log message 3 /another-request
log message 4 /yet-another-request
So basically App Engine groups log messages by request. To get my
team on board, can we make Kibana do the same?
Not out of the box, no. Kibana doesn't have any such contextual
understanding of messages and currently can't be configured as
such either.
--
Magnus Bäck | Software Engineer, Development Tools magnu...@sonymobile.com | Sony Mobile Communications
I'm not sure whether I understand your issue in full depth but you can use
nested aggregations to have hierarchical grouping in Kibana 4. Maybe this
solves your issue?
Am Montag, 22. Dezember 2014 09:58:57 UTC+1 schrieb stephanos:
Thanks for the answer!
I think wasn't clear enough: all our log messages already have a
requestID. So if there was a grouping feature we'd apply it to that
field.
I'm just wondering, how do you troubleshoot a issue of a user? When we see
a problem we look at all requests of that user in the GAE log viewer. Then
you quickly see requests that have non-200 status codes. Then we drill into
a request and see all logs of that request chronologically. While in
Kibana I can also look at all logs from a user ordered by time, but it's
not always completely clear which request log messages belong to. It's more
like one big stream.
My point is, you should really try out the Google App Engine log viewer -
then you would know what you are missing!
Stephan
On Monday, December 22, 2014 7:38:26 AM UTC+1, Magnus Bäck wrote:
On Tuesday, December 16, 2014 at 10:03 CET,
stephanos stephan...@gmail.com wrote:
we are using Google App Engine to host our SaaS app. Google offers a
nice log browser but it is way too sloooow. So one of my colleagues
suggested we pipe our logs to logstash and make them accessible via
Kibana. So far so good, we managed to set everything up.
But when Kibana was shown to the other team members they weren't
really excited. It was much faster, yes. It allowed to make better
queries, yes. BUT it broke the pattern they knew from the Google App
Engine log browser:
/some-request
log message 1
log message 2
/another-request
log message 3
/yet-another-request
log message 4
While Kibana works like this:
log message 1 /some-request
log message 2 /some-request
log message 3 /another-request
log message 4 /yet-another-request
So basically App Engine groups log messages by request. To get my
team on board, can we make Kibana do the same?
Not out of the box, no. Kibana doesn't have any such contextual
understanding of messages and currently can't be configured as
such either.
--
Magnus Bäck | Software Engineer, Development Tools magnu...@sonymobile.com | Sony Mobile Communications
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
