Configure logstash for dev and prod with different index pattern

[sundarm (sundar)]

Hi ,

I am having single logstash server and installed filebeats on dev and prod instances and in my logstash config i am using the following index pattern index => "applogs-%{+YYYY.MM.dd}" so now all my dev and prod logs in kibana are listing under the same index pattern applogs. I would like to separate dev and prod index pattern . Can some one help to separate the environment logs .

Thanks,
sundar

[yaauie (Ry Biesemeyer)]

It's a little hard to give exact advice without knowing the shape of yur events and what differentiates your dev events from prod events, but perhaps this will help:

The Elasticsearch Output's index directive uses the Logstash sprintf syntax, which allows access to individual fields on the event; assuming you had a field called env that contained either "dev" xor "prod", you could supply the field as follows:

output {
  elasticsearch {
    # ...
    index => "applogs-%{[env]}-%{+YYYY.MM.dd}"
  }
}

[sundarm (sundar)]

HI yaauie,

Thanks for your immediate reply. Its really very helpful but could you please tell me where should i mention this env . In my filebeat config and logstash config both ?
Below is my simple filebeat config

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/www/html/var/log/*.log
fields:
      # used in the output section to send each log to its
      # proper index instead of the default 'filebeat-*'
      index_name: applogs
setup.template.enabled: false
output.logstash:
  hosts: ["10.91.96.201:5044"]
  index: "%{[fields.index_name]:logs}-%{+YYYY.MM.dd}"

logging.to_syslog: true
logging.to_files: false

In my logstash config output i have configured as below

index => "applogs-%{+YYYY.MM.dd}"

Thanks

[yaauie (Ry Biesemeyer)]

In Filebeat, the fields directive allows you to add fields; if you find it acceptable to have a different config in dev vs prod, then adding the relevant field in Filebeat may be appropriate.

Otherwise, if you can differentiate dev and prod via the contents of the event messages, then the field can be added in Logstash.

I cannot be any more clear without knowing what differentiates your logs from each other.

[sundarm (sundar)]

HI Yaauie,

Now i am not using filebeat. I directly installed logstash on my webserver and trying to send drupal logs to my kibana. Could you please help me to configure logstash filter for drupal logs .

Thanks

[sundarm (sundar)]

HI,

Now i am not using filebeat. I directly installed logstash on my webserver and trying to send drupal logs to my kibana. Could you please help me to configure logstash filter for drupal logs .

Thanks