Kibana4 -> Elasticsearch API using Javascript

Elastic Stack Kibana
#1

What I want:

In Kibana, make a custom query in Javascript to produce a JSON response containing information like this:

  "hits": {
    "total": 8888229,
    "max_score": null,
    "hits": [
      {
        "_index": "logstash-2015.07.20",
        "_type": "logs",
        "_id": "AU6s1kvTAluRVDz068io",
        "_score": null,
        "_source": {
          "message": "",
          "@version": "1",
          "@timestamp": "2015-07-20T19:00:01.595Z",
          "type": "logs",
          "file": "/home/karl/sys_info.log",
          "host": "dood",
          "offset": "0",
          "tags": [
            "_grokparsefailure"
          ]
        }, //...etc...//

What I've tried:

  1. Sending POST requests using Javascript to /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1437514278665 that look very much like the POST requests that are sent when you search in Kibana's query bar.
  2. Attempt to use the add() function which in my tacked on Javascript file I seem to have access to.
  3. Attempt to use require('components/courier/data_source/search_source') to get the SearchSource object. In doing this I wasn't able to figure out how to use Private and the other parameters to ***SearchSourceFactory()***.

Relevant links which I've viewed: https://github.com/elastic/kibana/issues/3580

###What should I do?

#2

Ultimately, what are you trying to achieve with this? What's your use case?

#3

I'm attempting to create a button which will fire off multiple queries to Elasticsearch, each query using the previous query result in the next.

More Specific Info: I am working collecting logs from multiple machines. Each machine handles many orders which travel from one machine to the next. Each machine also has a pointer to the previous machine in the chain and the next machine in this chain. I want to trace that order. So to do this I want to get one order_id from one of the machines, and then search the parent machine and child machine for that order. If I find the order and it too has a parent, repeat the process.

Another option on how to implement the queries: I realize that sending multiple requests on the client side and doing calculations there isn't the best way to go about it as round trip time goes way up. Instead I realize I should send one request, which then on the backend fires of a bunch of queries. I know Kibana4 uses Nodejs, is extending the Nodejs backend what I should be looking into doing?

#4

For this type of use case, have you considered using entity-centric indexing?

Mark Harwood, one of our Engineers, has given several talks on the subject. Here are some pointers:

Slides: https://www.elastic.co/elasticon/2015/sf/building-entity-centric-indexes1
Recent video: https://www.elastic.co/videos/entity-centric-indexing-mark-harwood?q=entity
Video from Meetup: https://www.elastic.co/videos/entity-centric-indexing-london-meetup-sep-2014

#5

My gut response is that this may be difficult because to construct each entity I would have to build a long chain of order IDs. Each node in the chain represents one machine, each with a previous and next node pointer. These chains can potentially be added to over a span of days. I believe because of this I would have to store each node in the ES, then when I add a new one, I check to see if there is an entity that contains that order ID. If there is, then I add this new node to that entity.

However, with that solution, there is a problem that if I have 3 nodes. Each part of the chain as such: A -> B -> C. A and C were first added to ES, my preprocessing would look at the data like this: ? -> A -> ?, and ? -> C -> ? which would cause me to store the two nodes in different indices, even if they are actually from the same chain.

Once C was added, I would have to, after the fact, grab C, delete the entity, and then put C into A's entity. This seems overly complex when my data will likely never grow to a size which I would run into a speed bottleneck from doing all these queries at request time, instead of being preprocessed.

After thinking about it: I think that entity-centric indices could be an OK solution. Is the implementation I wrote about above is accurate (with the ABC stuff)? Is it correct that I would have to continually shift around data inside ES to handle the case I specified above? Also, on a more technical note, he is speaking about grouping these entities by Type or by Index. Wouldn't there be too many indices?

While this option could be good, I'm not sure if it is the most time-efficient considering the added development time vs what benefits we would gain. Our data will only scale up to a few hundred GB at most, so I don't think our the queries will be too slow considering each will be made separately and against indexed fields.

#6

##My Solution
I finally realized that I should be extending the backend of Kibana4 written in Nodejs. I then edited:

/kibana/src/routes/index.js

and included a new URL. This one would be:

router.get('/trace-order')

From there I used the Nodejs Elasticsearch API documented here: https://github.com/elastic/elasticsearch-js to perform the queries that I needed. Then when they completed and I did my calculations, I returned all my values in JSON.

#7