I have ELK cluster setup and i delete indices older than 10 days but i want to create a separate index which only has 1 required field so that it is always present .
The main index has huge data and multiple fields, i only want below info in new index
message:
{"log_model": "request", "written_at": "2020-10-23T02:20:55.289Z", "written_ts": 1603419655289617000, "remote_user": "-", "request_url": "/login", "referer": "https://mytestingapp.com/testapp/", "x_forwarded_for": "192.168.1.1", "protocol": "HTTP/1.1", "method": "POST", "remote_ip": "10.234.2.1", "request_size_b": 49, "remote_host": "10.234.2.1", "remote_port": "58070", "request_received_at": "2020-10-23T02:20:55.289Z", "log_type": "user-service", "correlation_id": null}
i only want the below data
"request_url": "/login", "referer": "https://mytestingapp.com/testapp/",```
For this i had created another logstash.conf and added below
input {
beats {
port => 5044
add_field => { "log_level" => "-" }
tags => [ "filebeat-log" ]
type => "beats"
client_inactivity_timeout => 300
}
}
filter {
if [type] == "beats" {
json {
source => "message"
}
prune {
interpolate => true
whitelist_names => ["request_url"]
}
}
}
output {
if [type] == "beats" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "login-index"
action => "index"
}
}
}
but i dont se the index "login-index" in indices so i'm unable to create index pattern.
Also i have stopped receiving any data in other index since i added the above