Hi,
Problem Statement: To export the logs from elasticsearch and write into csv in timewise order(either increasing or decreasing)
Solution: I am using logstash , configuration is given below,
input {
elasticsearch {
hosts => "localhost:9200"
index => "mergnginxinfo"
schedule => "*/1 * * * *"
query => '
{
"query": {
"bool": {
"must": [
{
"match_all": {}
},
{
"exists": {
"field": "transaction.id.keyword"
}
} ,
{
"range":{
"@timestamp":
{
"gte": "now-4m",
"lte": "now-3m"
}
}
}
]
}
},
"sort": [
{
"@timestamp": {
"order": "asc"
}
}
]}'
}
}
output {
csv {
# elastic field name
fields => ["@timestamp","requestid","ngnix.cpu.usage","ngnix.responsebytes","ngnix.memory.total.free","ngnix.memory.total.test","ngnix.transaction.duration","ngnix.urlpath","ngnix.Error_ID","ngnix.Exception","ngnix.ipaddress","ngnix.useragent","app1.user.id","app1.firstname","app1.lastname","app1.cpu.usage","app1.memory.total.free","app1.memory.total.test","app1.responsecode","app1.user.email","app1.transaction.duration.us","app1.methodname","app1.http.Port","app1.error.id","app1.error.exception.type","app1.transaction.result","app1.uripath","app2.user.id","app2.firstname","app2.lastname","app2.cpu.usage","app2.memory.total.free","app2.memory.total.test","app2.methodname","app2.responsecode","app2.error.id","app2.error.exception.type","app2.user.email","app2.http.Port","app2.transaction.duration.us","app2.transaction.result","app2.uripath"]
# This is path where we store output.
path => "C:/Users/M1056317/ELK/csv/try2/csv-export-%{+YYYY-MM-dd_hh.mm}.txt"
}
}
Problem: I am not getting the timestamp in particular order, I checked the same with GET API, it is coming fine there but not in csv file. Seems like there is some issue while writing.