Start two logstash and one of them create default template and one not


(zeroliuhuang) #1
  • Version: logstash-5.4.1
  • Operating System: CentOS 6.9
  • Config File (if you have sensitive info, please remove it):
input {
    beats {
        port => 5044
    #    codec => multiline {
    #        pattern => "^# User@Host"
    #        negate => true
    #        what => "previous"
    #    }
    }
    #tcp {
    #    port => 1928
    #}
}

filter {

    grok {
        match => [ "message", "(?m)^# User@Host: %{USER:query_user}\[[^\]]+\] @ (?:(?<query_host>\S*) )?\[(?:%{IP:query_ip})?\](?:\s*Id: %{NUMBER:id:int})?\s+# Query_time: %{NUMBER:query_time:float}\s+Lock_time: %{NUMBER:lock_time:float}\s+Rows_sent: %{NUMBER:rows_sent:int}\s+Rows_examined: %{NUMBER:rows_examined:int}\s*(?:use %{DATA:database};\s*)?SET timestamp=%{NUMBER:timestamp};\s*(?<query>(?<action>\w+)\s+.*)" ]
    }

    grok {
        match => { "message" => "# Time: " }
        add_tag => [ "drop" ]
        tag_on_failure => []
    }

    if  "drop" in [tags] {
        drop {}
    }

    date {
        match => ["mysql.slowlog.timestamp", "UNIX", "YYYY-MM-dd HH:mm:ss"]
        target => "@timestamp"
        timezone => "Asia/Chongqing"
    }
    ruby {
        code => "event.set('[@metadata][today]', Time.at(event.get('@timestamp').to_i).localtime.strftime('%Y.%m.%d'))"
    }
    mutate {
        remove_field => [ "message" ]
    }
}

output {
    stdout { codec => rubydebug }
    elasticsearch {
        hosts => ["10.10.10.10:9200"]
        index => "mysql-slow-%{[@metadata][today]}"
        document_type => "mysql-slow"
    }
}
  • Sample Data:
  • Steps to Reproduce:
    I want collect mysql slow log into elasticsearch and start tow logstash . they have the same configuration , but when I restart a logstash it seems create a default template names logstash-* and other one fist start and no create a default template.
    I find the parameter manage_template , it will control the logstash whether or not create default template logstash-* . so I want know it will always create a default template logstash-* when the parameter manage_template set true or not config in configfile(it seems default value is true) ? but I hava start multiple logstash and not config the manage_template , why they hava not created default template logstash-*
    So I want to know which parameter about the generate of default template logstash-* . maybe the manage_tempate or template_overwriter or index_name ,env if the template director ?

(Magnus Bäck) #2

I want collect mysql slow log into elasticsearch and start tow logstash . they have the same configuration , but when I restart a logstash it seems create a default template names logstash-* and other one fist start and no create a default template.

Why do you make that conclusion?


(zeroliuhuang) #3

Sorry , I have not say message clear. We have some template named logstash-nginx-* and logstash-qba-, when I start a new logstash, the all templates is OK . But when I restart the old logstash, it will create a default template logstash- , the log will send to the logstash-* and nothing send to logstash-qba-. but it only the logstash-qba- have no info and the other template like logstash-nginx-* it work OK .


(Magnus Bäck) #4

I don't understand. Instead of describing what happens, give examples that include

  • your configuration,
  • the result you get, and
  • the result you expected.

(zeroliuhuang) #5

I'm sorry, I can describe is not very clear.Specific logstash startup configuration file above the information I have given out.

Specific operation:

I put the old logstash binary package and configuration file copy to a new server, and then create a new logstash.And then restart the old logstash.

The result of:

Because we have to create some templates online, similar to the logstash - qba - *, logsatsh - nginx - *, logstash - remix - * so named template, but after the resumption of the old logstash process has generated a default template logstash - *.Then we had some application log is sent to the above the logstash - qba - * this template, the results of these all the data sent to logstash - * this template, cause logstash qba - * this template, there is no data to send.But when I create new logstash didn't happen this kind of situation, and only logstash qba - * the template.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.