# Need help to indentify what needs to be done in order to seperate and injest in elasticsearch

**URL:** <https://discuss.elastic.co/t/need-help-to-indentify-what-needs-to-be-done-in-order-to-seperate-and-injest-in-elasticsearch/360475>\
**Category:** Logstash\
**Created:** [May 29, 2024, 1:38pm UTC](https://discuss.elastic.co/t/need-help-to-indentify-what-needs-to-be-done-in-order-to-seperate-and-injest-in-elasticsearch/360475 "2024-05-29T13:38:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![kishorkumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kishorkumar/32/132930_2.png) [@kishorkumar](https://discuss.elastic.co/u/kishorkumar)\
**Post date:** [May 29, 2024, 1:38pm UTC](https://discuss.elastic.co/t/need-help-to-indentify-what-needs-to-be-done-in-order-to-seperate-and-injest-in-elasticsearch/360475/1 "2024-05-29T13:38:45Z")

</div>

I am trying to generate the logic in order to track last value injested in the elasticsearch for http\_poller.

till now for testing version what i have done is i am using elastic index to elastic index pipeline.

first i have created 2 pipelines and used the pipeline to pipeline communication.

to get last records timestamp i have used http\_poller and tracked the last timestamp and in output i have sent it to second pipline to use it

```auto
output {
  pipeline { send_to => syslog }
}

```

now second pipeline

```auto
input {
    
    pipeline { address => syslog }
 
}

```

in filter i have used the elasticsearch with query template to get data from particualr index and also listed all the fields that i wanted.

and added the path to the query\_template **matching-requestaw.json**

```auto
{
  "query": {
    "bool": {
      "filter": [
        {
          "range": {
            "date": {
              "gt": "%{[Timestamp]}"
            }
          }
        }
      ]
    }
  },
  "sort": [
    {
      "@timestamp": {
        "order": "asc"
      }
    }
  ]
}

```

```auto
fitler {

 elasticsearch {
    hosts => [""]
    index => "data-v2"
    user => ""
    password => ""
    ssl_verification_mode => "none"
    query_template => "matching-requestaw.json"
    fields => {
      "@timestamp" => "queried_timestamp"
      "day" =>"Thursday"
     
    }
  }

}

```

Now the out i am getting is

```auto
{
      "day" => [
        [0] "Thursday",
        [1] "Thursday",
        [2] "Thursday"
    ],
           "@timestamp" => 2024-05-23T21:05:42.000Z
}
{
      "day" => [
        [0] "Thursday",
        [1] "Thursday",
        [2] "Thursday"
    ],
           "@timestamp" => 2024-05-23T21:05:42.000Z
}
{
      "day" => [
        [0] "Thursday",
        [1] "Thursday",
        [2] "Thursday"
    ],
           "@timestamp" => 2024-05-23T21:05:42.000Z
}

```

what is the way to get the records return by elasticsearch fitler

```auto
{
{
      "day" => "Thursday",
      "@timestamp" => 2024-05-23T21:05:42.000Z
}
{
      "day" => "Thursday",
      "@timestamp" => 2024-05-23T21:05:42.000Z
}
{
      "day" => "Thursday",
      "@timestamp" => 2024-05-23T21:05:42.000Z
}
}

```

or any type of object that i can split and injest all retreived reocrds into elasticsearch like this .

```auto
{
      "day" => "Thursday",
      "@timestamp" => 2024-05-23T21:05:42.000Z
}
{
      "day" => "Thursday",
      "@timestamp" => 2024-05-23T21:05:42.000Z
}
{
      "day" => "Thursday",
      "@timestamp" => 2024-05-23T21:05:42.000Z
}

```

#logstash #elasticsearchfitler #Logstash#logstash