Statistics using ES on group of data

Hello,
we already set up an Elasticsearch node with Kibana on top for log analysis.

Now, we would like to make it more scalable.

I have few questions about ES and I would like to hear some
opinions/hints/suggestions from you.

  • How can we exploit quite good machines such as (24cores, 48GB RAM
    etc...) with ES? With Java 32bit the heap can be set at 2GB MAX...
  • Our architecture is a message oriented distributed platform.
    • Our idea is to make each node of this platform log the properties
      of the transaction
    • We would like to obtain statistics on differend properties by group
      of linked transactions (and perform drill down on those properties)
      • E.g.
        • Data:
          • Service X(time:4;type:abc) -> Service
            Y(time:9;type:abc) -> Service W(time:1;type:zzz)
          • Service X(time:3;type:abc) -> Service
            Y(time:7;type:abc) -> Service W(time:1;type:zzz)
          • Service X(time:3;type:abc) -> Service
            Y(time:7;type:abc) -> Service W(time:1;type:zzz)
          • Service A(time:3;type:abc) -> Service
            B(time:7;type:abc) -> Service X(time:1;type:zzz) -> Service
            Y(time:9;type:abc) -> Service W(time:1;type:zzz)
        • Queries:
          • a) Mean of property(time) for all the transactions started
            by X
          • b) List and Mean of types of property(type) for all the
            transactions started by X
        • Expected reply:
          • a) ((4+9+1), (3+7+1), (3+7+1), (1+9+1)) / 4
          • b) ((type: abc count: 2), (type: zzz count:1)),
            ((type: abc count: 2), (type: zzz count:1)),
            ((type: abc count: 2), (type: zzz count:1)),
            ((type: abc count: 1), (type: zzz count:2)):
            ((type: abc mean: 7/4), (type: zzz count:5/4))
        • In my opinion, I should go with nested documents or find a
          way to be able to group those transactions.

What do you think about it?
Luca

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.