Index created with wrong name


(John Woo Lee) #1

Hi,

I have a server with below setup

#cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

#timedatectl
      Local time: Thu 2017-01-12 07:21:08 GMT
  Universal time: Thu 2017-01-12 07:21:08 UTC
        RTC time: Thu 2017-01-12 07:21:08
       Time zone: Etc/GMT (GMT, +0000)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: n/a

#rpm -qa | grep logstash
logstash-5.1.1-1.noarch

#java -version
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

But for some reason the index being created is not what i specify in the config. Here is trace showing index created as powerdns-1-49002.07.31 even though i specify powerdns-1-%{+YYYY.MM.dd}. Can someone help explain why this is happening please ? To me the year is the last part of the hosts IP and port i.e. 4:9200 !

[2017-01-12T07:05:34,767][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@hosts = ["10.149.62.14:9200"]
[2017-01-12T07:05:34,767][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@index = "powerdns-1-%{+YYYY.MM.dd}"

....

[2017-01-12T07:06:31,501][DEBUG][org.apache.http.wire     ] http-outgoing-0 << "{"took":760,"errors":false,"items":[{"create":{"_index":"powerdns-1-49002.07.30","_type":"logs","_id":"CQcj3m4VRk2GCfW3fhxATQ","_version":1,"status":201}},{"create":{"_index":"powerdns-1-49002.07.31","_type":"logs","_id":"iTtXt6RaSAWz4afOVRsegA","_version":1,"status":201}},{"create":{"_index":"powerdns-1-49002.07.31","_type":"logs","_id":"RQJQfPuuR6i00vgPnNd-Hg","_version":1,"status":201}}]}"

Here is config i'm using

input {
  file {
    path => "/var/log/mariadb/mariadb.log"
    type => "mysql_log"
    start_position => "end"
  }
}

filter {
    if [type] == "mysql_log" {
    grok {
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "message" => "%{MDBLOG}" }
    }

    mutate { rename => [ "message", "raw" ] }
    }
}

output {

  if "_grokparsefailure" in [tags] {
    file { path => "/var/log/logstash/failed_mariadb.log.events-%{+YYYY-MM-dd}.log" }
  }

  file {
    path => '/var/log/logstash/mariadb_log.json'
  }
  
  elasticsearch {
    hosts => ["10.149.62.14:9200"]
    index => "powerdns-1-%{+YYYY.MM.dd}"
  }
  
}

(Mark Walkom) #2

Does your message match?
What does that grok pattern look like? What does the data look like?


(John Woo Lee) #3

Yes the grok filter works fine.. but whats the grok pattern got to do with the index that logstash sends data too.. given that i explicitly tell it what it should be named.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.