Not able to create multiple index through logstash 7.3

Elastic Stack Logstash
1

Hi Team,
I have a setup of dockerized elk setup of version 7.3.1 (Elasticsearch, logstash, Kibana, Filebeat).
ELK is running on docker only. I have installed filebeat package on same machine and provide input to logstash.
I have multiple log file like messages, auth.log, cron.log etc. and all these have configured in filebeat.yml with fields parameter for identification.
fields:
service: auth_logs

I want to all these logs file will go logstash as input beat and output will different index as per file name in elk.
How can I achieve this ? Please help.

Version: 7.3.1

below is the logstash config file
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => hostip
index => "%{[field][service]-%{+YYYY.MM.dd}"
}
}

2

Hi @manjeetsingh,

You can specify the fields in filebeats and then apply the condition in logstash output plugin to create different index as specify the fields.

Below is the example for your reference:

Fields log_type: access specify in the filebeat:

filebeat:
  prospectors:
    - paths:
        - /path/to/logs/access.log
      fields:  {log_type: access}

Apply condition in logstash o/p plugin:

output {
if [fields][log_type] == "access" {
elasticsearch {
hosts => hostip
index => "%{[field][service]-%{+YYYY.MM.dd}"
}
}
if [fields][log_type] == "access_1" {
elasticsearch {
hosts => hostip
index => "index_1"
}
}
}

Regards,
Harsh Bajaj

3

Hi Harsh Bajaj,
I have done changes as per your reply but index is not creating. Below is the config of filebeat

filebeat.inputs:

  • type: log
    • paths:
      • /var/log/auth.log
        fields:
        log_type: auth
    • paths:
      • /var/log/syslog
        fields:
        log_type: system

Logstash:

input {
beats {
port => 5044
}
}
output {
if [fields][log_type] == "auth" {
elasticsearch {
hosts => hostip
index => "%{[field][log_type]-%{+YYYY.MM.dd}"
}
}
if [fields][log_type] == "system" {
elasticsearch {
hosts => hostip
index => "%{[field][log_type]-%{+YYYY.MM.dd}"
}
}
}

After that i have restarted services but no index is creating.

4

Your index name patterns states ‘field’ and not ‘fields’.

1 Like
5

Thanks for correction but only one index is created i.e, system-index not the other (auth-index). I am getting request on other log file as well.
So how will it be achieve ?
I want to create separate index for log files for multiple instances.

6

Can somebody tell me how to create separate index for log files using filebeat and logstash. I have tried above approach but didn't work.

7

Look at your data in the Kibana Discover app. Do records have the correct additional field set from Filebeat? Do you see both types of documents? Is it possible that Filebeat is not correctly configured?

8

I have checked kibana and field.log_type is system which is correct for path /var/log/syslog but other field which has path /var/log/auth.log is not coming in kibana at all. Only one document is present i.e, system.

filebeat.inputs:

  • type: log
    • paths:
      • /var/log/auth.log
        fields:
        log_type: auth
    • paths:
      • /var/log/syslog
        fields:
        log_type: system

output.logstash:
hosts: ["logstship:5044"]

Please let me know what am i doing wrong.

9

How can I create different file per client in logstash for input beat ?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.