Statistics using ES on group of data

Elastic Stack Elasticsearch
1

Hello,
we already set up an Elasticsearch node with Kibana on top for log analysis.

Now, we would like to make it more scalable.

I have few questions about ES and I would like to hear some
opinions/hints/suggestions from you.

  • How can we exploit quite good machines such as (24cores, 48GB RAM
    etc...) with ES? With Java 32bit the heap can be set at 2GB MAX...
  • Our architecture is a message oriented distributed platform.
    • Our idea is to make each node of this platform log the properties
      of the transaction
    • We would like to obtain statistics on differend properties by group
      of linked transactions (and perform drill down on those properties)
      • E.g.
        • Data:
          • Service X(time:4;type:abc) -> Service
            Y(time:9;type:abc) -> Service W(time:1;type:zzz)
          • Service X(time:3;type:abc) -> Service
            Y(time:7;type:abc) -> Service W(time:1;type:zzz)
          • Service X(time:3;type:abc) -> Service
            Y(time:7;type:abc) -> Service W(time:1;type:zzz)
          • Service A(time:3;type:abc) -> Service
            B(time:7;type:abc) -> Service X(time:1;type:zzz) -> Service
            Y(time:9;type:abc) -> Service W(time:1;type:zzz)
        • Queries:
          • a) Mean of property(time) for all the transactions started
            by X
          • b) List and Mean of types of property(type) for all the
            transactions started by X
        • Expected reply:
          • a) ((4+9+1), (3+7+1), (3+7+1), (1+9+1)) / 4
          • b) ((type: abc count: 2), (type: zzz count:1)),
            ((type: abc count: 2), (type: zzz count:1)),
            ((type: abc count: 2), (type: zzz count:1)),
            ((type: abc count: 1), (type: zzz count:2)):
            ((type: abc mean: 7/4), (type: zzz count:5/4))
        • In my opinion, I should go with nested documents or find a
          way to be able to group those transactions.

What do you think about it?
Luca

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.