How to create a separate index which has only required fields

I have ELK cluster setup and i delete indices older than 10 days but i want to create a separate index which only has 1 required field so that it is always present .
The main index has huge data and multiple fields, i only want below info in new index

 message:
      {"log_model": "request", "written_at": "2020-10-23T02:20:55.289Z", "written_ts": 1603419655289617000, "remote_user": "-", "request_url": "/login", "referer": "https://mytestingapp.com/testapp/", "x_forwarded_for": "192.168.1.1", "protocol": "HTTP/1.1", "method": "POST", "remote_ip": "10.234.2.1", "request_size_b": 49, "remote_host": "10.234.2.1", "remote_port": "58070", "request_received_at": "2020-10-23T02:20:55.289Z", "log_type": "user-service", "correlation_id": null}

i only want the below data

"request_url": "/login", "referer": "https://mytestingapp.com/testapp/",```

For this i had created another logstash.conf and added below

input {
  beats {
    port => 5044
    add_field => { "log_level" => "-" }
    tags => [ "filebeat-log" ]
    type => "beats"
    client_inactivity_timeout => 300
  }
}
filter {
  if [type] == "beats" {
   json {
      source => "message"
    }

   prune {
       interpolate => true
       whitelist_names => ["request_url"]
   }
  }
}
output {
  if [type] == "beats" {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "login-index"
      action => "index"
    }
  }
}

but i dont se the index "login-index" in indices so i'm unable to create index pattern.
Also i have stopped receiving any data in other index since i added the above

Hi,
any input will be appreciated.
I have create a separate pipeline but getting below error in logstash.

warning: thread "[login-pipeline]>worker2" terminated with exception (report_on_exception is true):
Nov  3 05:29:06 ip-122-13-4-31 logstash[29473]: java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `[^`

The logstash syntax test does not show any syntax errors though

I suspect that is a problem with interpolation. Why have you set

interpolate => true

Thanks @Badger, I saw it somewhere while checking for prune filter.

After removing that, i dont see any errors but i don't see the new index in index paterns

cat pipeline.yml

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: login-pipeline
  path.config: "/home/ronnie/login_index.conf"

login-index.conf

input {
  beats {
    port => 5044
    client_inactivity_timeout => 300
  }
}
filter {
   json {
      source => "message"
    }

   prune {
       whitelist_names => [ "^_source" ]
   }
}
output {
    elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "login-index"
    }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.