Logstash output struggle

Hello, iam new to whole ELK problematics and iam trying to redo "logstack" i inherited from scratch. Previous configuration was using logstash - graylog- elasticserach with custom filtering in logstash. Iam trying to use ELK config, but iam not entirely sure i understand how to configure output for logstash -> elasticserach and then elasticserach -> kibana. My logstash config is partly tutorial-made (filebeats) and partly copy-paste of custom filter. Iam receiving beats just fine in kibana (tutorial based).
Versions of apps should be latest/week old.

Iam prety sure of that my output is configured wrong, but iam not able to figure out the proper way how this can work (wrap my head around it), the previous way was using output GELF plugin => directly to graylog via port and localhost.

my cfg:

      input {

     beats {
       port => 5044
    #   ssl => true
    #   ssl_certificate => "/etc/ssl/logstash-forwarder.crt"
    #   ssl_key => "/etc/ssl/logstash-forwarder.key.pem"
       congestion_threshold => "40"
      }

      # SNMP traps from IPTV devices
      snmptrap {
        port => 10162
        yamlmibdir => "/etc/logstash/yaml-mibs"
        type => "snmptrap"
      }

    }

    ########################################################

    filter {
    if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGLINE}" }
        }

        date {
    match => [ "timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      }
    }


    # Custom processing for RFGW
        if "rfgw" in [host] {

            grok {
                match => { "rfgw1TrapDetails" => "^%{DATA}(?<stream>%{IPV4}/%{POSINT})%{DATA}$" }
                keep_empty_captures => true
            }

            translate {
                    field => "rfgw1TrapState"
                    destination => "alarm_status"
                    dictionary => [ "1", "set",
                                    "2", "cleared" ]
                    remove_field => [ "msgStatus" ]
            }

            mutate {
                replace => { "message" => "%{rfgw1TrapLabel} | %{rfgw1TrapDetails} | value: %{rfgw1TrapValue}" }
                gsub => [ "stream", "/", ":"]
                add_field => { "alarm_class" => "%{rfgw1TrapLabel}" }
                remove_field => [ "rfgw1TrapInstance", "rfgw1TrapSeverity", "rfgw1TrapThreshold", "rfgw1TrapSequence", "rfgw1TrapUnits", "SNMPv2-MIB::sysName", "SNMPv2-MIB::sysLocation", "rfgw1TrapDetails", $

            }

            }

            if [alarm_class] ==  "NCS CAM Change" {
                    drop { }
            }

            if "CC Alarm Set" in [message] {
                    mutate {
                            add_tag => [ "cc" ]
                    }
            }

    }
    }

    ###########################################

    output {

     elasticsearch {
     hosts => localhost
     index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
             }

    if "snmptrap" in [type] {

     elasticsearch {
            hosts => localhost
            }
        }

ADD

Alsowhen i restart logstash, in syslog i see several pages of logs filled with following warnings regarding MIB containing different names

Apr  3 07:12:37: Started logstash.
Apr  3 07:12:52 debian logstash[3319]: Sending Logstash's logs to /var/log/logstash which is now configured via log4j2.properties
warning: overwriting old MIB name 'system'
warning: overwriting old MIB name 'sysDescr'
warning: overwriting old MIB name 'sysObjectID'
warning: overwriting old MIB name 'sysUpTime'
.......

logstash-plain.log in case, after restart

[INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[WARN ][logstash.inputs.beats    ] You are using a deprecated config setting "congestion_threshold" set in beats. Deprecated settings will continue to work, but are scheduled for$
[INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"$
[INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"$
][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost"]}
[INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>125}
][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ][logstash.inputs.snmptrap ] checking /etc/logstash/yaml-mibs for MIBs
[INFO ][logstash.inputs.snmptrap ] found MIBs: SAEUROPE-ADMINISTRATION-MIB,RIPv2-MIB,RFC1269-MIB,TOKEN-RING-RMON-MIB,LMP-MIB,IPOA-MIB,SNA-NAU-MIB,SNMP-USER-BASED-SM-MIB,IANA-ITU-$
[INFO ][logstash.pipeline        ] Pipeline main started
[INFO ][logstash.inputs.snmptrap ] It's a Trap! {:Port=>10162, :Community=>["public"], :Host=>"0.0.0.0", :MibDir=>"/etc/logstash/yaml-mibs", :MibModules=>["SAEUROPE-ADMINISTRATIO$
[INFO ][org.logstash.beats.Server] Starting server on port: 5044
[INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.