Logging information is displayed in the wrong index

Elastic Stack Logstash
1

Hello,
I have two conf files with different indexes in the "/etc/logstash/conf.d/" folder. These are as follows:

root@dsme01:~# cat /etc/logstash/conf.d/02-snmp.conf
input {
  snmp {
    tables => [{
      "name" => "interfaces" 
      "columns" => ["1.3.6.1.2.1.2.2.1.1", "1.3.6.1.2.1.2.2.1.2", "1.3.6.1.2.1.2.2.1.10", "1.3.6.1.2.1.2.2.1.16"]
    }]
    hosts => [{
      host => "udp:192.168.1.10/161" 
      community => "elk"  
      version => "2c"  
      retries => 2  
      timeout => 1000
    }]
    interval => 300
  }
}
filter {
  split {
    field => "interfaces"
  }
  mutate { 
    rename => { "[interfaces][index]" => "index" }
    rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifIndex]" => "ifIndex" } 
    rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifDescr]" => "ifDescr" } 
    rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifInOctets]" => "ifInOctets" } 
    rename => { "[interfaces][iso.org.dod.internet.mgmt.mib-2.interfaces.ifTable.ifEntry.ifOutOctets]" => "ifOutOctets" } 
  }
  mutate {
    convert => {
      "ifInOctets" => "integer"
      "ifOutOctets" => "integer"
    }
  }
  mutate {
    remove_field => [ "interfaces", "@version" ]
  }
}
output {
  elasticsearch {
    hosts => ["https://elasticsearch.intern.example.com:9200"]
    user => "elastic"
    password => "elastic"
    ssl => true
    cacert => "/etc/logstash/config/certs/HarbichCA.cacert.pem"
    manage_template => false
    index => "snmp-8.4.3-%{+YYYY.MM.dd}"
  }
}

and

root@dsme01:~# cat /etc/logstash/conf.d/05-filebeat.conf
input {
  beats {
    host => "logstash.intern.example.com"
    port => 5044
    client_inactivity_timeout => "600"
    ssl => true
    ssl_certificate => "/etc/logstash/config/certs/logstash.intern.example.com.crt"
    ssl_key => "/etc/logstash/config/certs/logstash.intern.example.com.pkcs8.key"
  }
}
output {
  elasticsearch {
    data_stream => false
    hosts => ["https://elasticsearch.intern.example.com:9200"]
    user => "elastic"
    password => "elastic"
    ssl => true
    cacert => "/etc/logstash/config/certs/HarbichCA.cacert.pem"
    manage_template => false
    index => "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"
  }
}

The logging from the "02-snmp.conf" configuration file is also written in the "filebeat-. . . " index.
The logging from the "05-filebeat.conf" configuration file is also written in the "snmp-. . . " index.
I do not understand why? Do you have a tip for me?

2

Hi @sharbich
When you put 2 conf files in the same directory they are concatenated into a single file / single pipeline.

If you want the to run independently use the pipelines.yml file to name them as separate pipelines.

3

I made the following configuration.

root@dsme01:~# cat /etc/logstash/pipelines.yml 
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html

## - pipeline.id: main
##   path.config: "/etc/logstash/conf.d/*.conf"

- pipeline.id: snmp
  path.config: "/etc/logstash/conf.d/02-snmp.conf"
  pipeline.workers: 3
- pipeline.id: filebeat
  path.config: "/etc/logstash/conf.d/05-filebeat.conf"
  queue.type: persisted

However, in both "Data views"
filebeat-* and snmp-* all logging entries are displayed under Discover.
Just why?

4

Look at the documents and see which index they are getting written too?

Is it duplicated documents or a Data View Issue.

I also notice you are missing In the first pipeline if you want to write indices

data_stream => false

Keep Looking... and you made sure you saved the pipeline.yml

Comment 1 out and try... there is something simple going on..

5

I do not get it. I'll try it again.
I want everything that I log in snmp to only be displayed in the DataView / Index snmp-.
Everything that is logged in beats is only displayed in the DataView / Index filbeat-.
Only when I comment out a file (either 02-snmp.conf or 05-filebeat.conf) the log entries no longer appear in the other DataView/Index.
I just can not manage it.

6

You have not showed the details of your data views

I also asked did you look explicitly at the indices that the document are written to.

Run both and open the documents and look at the index.. is it the correct expected index snmp- or filebeat? For the docs.

Show me... If so it is most likely the data views that are the issue... They might need to be cleaned up or deleted and recreated.

Show me the in details..

Without explicit details I can't help.

The most important is to understand are the documents being indexed into the correct index!

Then we can fix the data views

7

i think i solved it.

  1. I commented out the following entry in logstash.yml:
path.config: /etc/logstash/conf.d/*.conf
  1. Moved both conf files from "/etc/logstash/conf.d/" folder to different subdirectories.

Since then, what belongs to snmp is written/displayed in Index / DataView snmp-* and everything that belongs to filebeat-* is displayed in Index / DataView.

8

Interesting. Something's still a little odd but glad you got it to work.... I thought if you named them separately it always would keep them separate only to take a look at that.

I think this is the problem

path.config: /etc/logstash/conf.d/*

If you took that out and then name them individually in the pipelines.yml I think it would work as well.

But glad you got it to work!

9

Indeed. If path.config is set in logstash.yml then logstash will log

[WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

and concatenate the files in the path.config directory.

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.