Elasticsearch filter plugin - how to query data


(Cristian) #1

Hi All,
I ingest forex data into an index called forex1. The data received looks like this per minute. I do a split on candles:

{
     "@timestamp" => 2017-09-29T12:25:00.434Z,
    "granularity" => "M1",
       "@version" => "1",
     "instrument" => "EUR_USD",
        "candles" => {
          "volume" => 31,
        "closeMid" => 1.180655,
         "highMid" => 1.18076,
         "openMid" => 1.180595,
          "lowMid" => 1.18058,
            "time" => "2017-09-29T12:24:00.000000Z",
        "complete" => false
    }
}
{
     "@timestamp" => 2017-09-29T12:26:00.335Z,
    "granularity" => "M1",
       "@version" => "1",
     "instrument" => "EUR_USD",
        "candles" => {
          "volume" => 26,
        "closeMid" => 1.180735,
         "highMid" => 1.18079,
         "openMid" => 1.18063,
          "lowMid" => 1.1806,
            "time" => "2017-09-29T12:25:00.000000Z",
        "complete" => false
    }
}

There's a "time" on each doc that I want to use to query the last 5 docs. But for having it simple lets say I just want to fetch the previous doc. That is "time" minus 1 minute.
Im using elasticsearch filter plugin for this but I dont know how to create the query:

elasticsearch {
    query => "instrument:%{[instrument]} AND time:[ "TIME-1 MINUTE" to %{[time]}}"
    fields => { "closeMid" => "closeMid_t1" }
}

My questions (sorry for the noob questions bear with me):
1 - How do I refer to data in my current event? I want the query to be made on the same "instrument" that I have in the event. Is this the correct syntax? - instrument:%{[instrument]} (in my case above the query should be done on EUR_USD but I can have different pairs in my data)
2 - How do I subtract 1 one minute from my timestamp called "time"? time:%{[time]} - 1min or something like that should be better than querying a range.
3 - When the query is done I want to take the field 'closeMid' from previous event and save it in a new field 'closeMid_t1' in current event. Is this the correct way?
4 - I want to do this 5 times because a need values from 5 previous docs. Should I create a for loop outside the filter or inside?
5 - How to handle errors? What if I don't have any previous values. How will this be handled by the filter? Should I do an explicit closeMid_t1=0 some how?

Thanks for a suggestions before hand,
Cris


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.