I have implemented a Distributed Transaction Logging library with Tree like
Structure as mention in Google Dapper(
http://research.google.com/pubs/pub36356.html) and eBay CAL Transaction
Logging Framework(
http://devopsdotcom.files.wordpress.com/2012/11/screen-shot-2012-11-11-at-10-06-39-am.png).
The following is log format.
Log Format
TIMESTAMP HOSTNAME DATACENTER ENVIRONMENT EVENT_GUID PARENT_GUID TRACE_GUID APPLICATION_ID TREE_LEVEL TRANSACTION_TYPE TRANSACTION_NAME STATUS_CODE DURATION(in ms) PAYLOAD(key1=value2,key2=value2)
GUID HEX NUMBER FORMAT
MURMER_HASH(HOSTNAME + DATACENTER +
ENVIRONMENT)-JVM_THREAD_ID-(TIME_STAMP+Atomic Counter)
The log format is like Relational Database Tables.
Here is list of search query we will be doing:
Also I would like to know how I can index elastic search payload data so
user specify some expression like in payload (duration > 1000) then,
Elastic Search will bring all the loglines that satisfy condition. Also, I
would like to index Payload as Name=Value pair so user can query
(key3=value2 or key4 = exception) some sort of regular expression. Please
let me know if this can be achieved. Any help pointer would be great..
Please note the Payload can be any string name or value(can be integer, or
decimal or string).
Payload search example:
// (Duration > 1000) and (key1 > 10 or key1 < 100) etc
Finally I would be integrating this with Kibana UI so we can have nice UI
for searching and sorting etc.
I am new to Elastic Search and any pointer will be great. We will be using
this in walmartlabs with around 100 machine elastic search cluster and
~6000 FE servers logging data into Kafka and we will index from the Kafka
into Elastic Search. Any pointer will be great.
Thanks,
Bhavesh
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/79f04627-29ac-42aa-9170-35c22ef76ffa%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.