I am trying to generate the logic in order to track last value injested in the elasticsearch for http_poller.
till now for testing version what i have done is i am using elastic index to elastic index pipeline.
first i have created 2 pipelines and used the pipeline to pipeline communication.
to get last records timestamp i have used http_poller and tracked the last timestamp and in output i have sent it to second pipline to use it
output {
pipeline { send_to => syslog }
}
now second pipeline
input {
pipeline { address => syslog }
}
in filter i have used the elasticsearch with query template to get data from particualr index and also listed all the fields that i wanted.
and added the path to the query_template matching-requestaw.json
{
"query": {
"bool": {
"filter": [
{
"range": {
"date": {
"gt": "%{[Timestamp]}"
}
}
}
]
}
},
"sort": [
{
"@timestamp": {
"order": "asc"
}
}
]
}
fitler {
elasticsearch {
hosts => [""]
index => "data-v2"
user => ""
password => ""
ssl_verification_mode => "none"
query_template => "matching-requestaw.json"
fields => {
"@timestamp" => "queried_timestamp"
"day" =>"Thursday"
}
}
}
Now the out i am getting is
{
"day" => [
[0] "Thursday",
[1] "Thursday",
[2] "Thursday"
],
"@timestamp" => 2024-05-23T21:05:42.000Z
}
{
"day" => [
[0] "Thursday",
[1] "Thursday",
[2] "Thursday"
],
"@timestamp" => 2024-05-23T21:05:42.000Z
}
{
"day" => [
[0] "Thursday",
[1] "Thursday",
[2] "Thursday"
],
"@timestamp" => 2024-05-23T21:05:42.000Z
}
what is the way to get the records return by elasticsearch fitler
{
{
"day" => "Thursday",
"@timestamp" => 2024-05-23T21:05:42.000Z
}
{
"day" => "Thursday",
"@timestamp" => 2024-05-23T21:05:42.000Z
}
{
"day" => "Thursday",
"@timestamp" => 2024-05-23T21:05:42.000Z
}
}
or any type of object that i can split and injest all retreived reocrds into elasticsearch like this .
{
"day" => "Thursday",
"@timestamp" => 2024-05-23T21:05:42.000Z
}
{
"day" => "Thursday",
"@timestamp" => 2024-05-23T21:05:42.000Z
}
{
"day" => "Thursday",
"@timestamp" => 2024-05-23T21:05:42.000Z
}