Hello All,
I am trying to send two fields from mis-monitoring-webserver to new index mis-monitoring-webui.
Issue faced: This webserver index gets data every 10 seconds and from this my expectation is- I have written sperate config file which will fetch only two fileds from this index and send to webui index.
Now when the elastic query is run on DEV Tools I get top 1 record- EXPECTED, but when this query is run in logstash Input filter then I get around 10000 records.
Root cause Identified, but don't know resolution:
It always tries to get data from 10000 records, regardless size:1, some how i need to manage only top 1 record to be sent from webserver index to webui index.
Below is config file.
Plz suggest.(How to send only top1 record from one index to other index)...Devtools above query gives correct output/ but config files provide many documents in index : approx 250000
Logstash config:
input {
elasticsearch {
hosts => ["https://abc:443"]
index => ["mis-monitoring-webserver"]
query => '
{
"size": 1,
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"exists": {
"field": "usecaseStatus"
}
},
{
"exists": {
"field": "usecaseCategory"
}
}
]
}
}
}'
schedule => "*/1 * * * *"
api_key => ""
ssl_enabled => true
ssl_certificate_authorities => ""
}
}
filter {
prune {
whitelist_names => ["usecaseCategory","usecaseStatus","@timestamp"] # Corrected field names
}
}
output {
elasticsearch {
hosts => "https://abc:443"
ilm_pattern => "{now/d}-000001"
ilm_rollover_alias => "mis-monitoring-webui"
ilm_policy => "mis-monitoring-policy"
api_key => ""
ssl_enabled => true
ssl_certificate_authorities => ""
http_compression => true
data_stream => false
}
}
From below query run on dev tools I get only one record, this is expected, Similarly when this query is processed by logstash to send this record from webserver index to webui index- I get more than 20000 records. This is the issue. Expected by logstash was to send only top 1 record as in devtools.
