Index created with wrong name

(John Woo Lee) #1


I have a server with below setup

#cat /etc/redhat-release
CentOS Linux release 7.3.1611 (Core)

      Local time: Thu 2017-01-12 07:21:08 GMT
  Universal time: Thu 2017-01-12 07:21:08 UTC
        RTC time: Thu 2017-01-12 07:21:08
       Time zone: Etc/GMT (GMT, +0000)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: n/a

#rpm -qa | grep logstash

#java -version
java version "1.8.0_112"
Java(TM) SE Runtime Environment (build 1.8.0_112-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.112-b15, mixed mode)

But for some reason the index being created is not what i specify in the config. Here is trace showing index created as powerdns-1-49002.07.31 even though i specify powerdns-1-%{+YYYY.MM.dd}. Can someone help explain why this is happening please ? To me the year is the last part of the hosts IP and port i.e. 4:9200 !

[2017-01-12T07:05:34,767][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@hosts = [""]
[2017-01-12T07:05:34,767][DEBUG][logstash.outputs.elasticsearch] config LogStash::Outputs::ElasticSearch/@index = "powerdns-1-%{+YYYY.MM.dd}"


[2017-01-12T07:06:31,501][DEBUG][org.apache.http.wire     ] http-outgoing-0 << "{"took":760,"errors":false,"items":[{"create":{"_index":"powerdns-1-49002.07.30","_type":"logs","_id":"CQcj3m4VRk2GCfW3fhxATQ","_version":1,"status":201}},{"create":{"_index":"powerdns-1-49002.07.31","_type":"logs","_id":"iTtXt6RaSAWz4afOVRsegA","_version":1,"status":201}},{"create":{"_index":"powerdns-1-49002.07.31","_type":"logs","_id":"RQJQfPuuR6i00vgPnNd-Hg","_version":1,"status":201}}]}"

Here is config i'm using

input {
  file {
    path => "/var/log/mariadb/mariadb.log"
    type => "mysql_log"
    start_position => "end"

filter {
    if [type] == "mysql_log" {
    grok {
        patterns_dir => ["/etc/logstash/patterns"]
        match => { "message" => "%{MDBLOG}" }

    mutate { rename => [ "message", "raw" ] }

output {

  if "_grokparsefailure" in [tags] {
    file { path => "/var/log/logstash/{+YYYY-MM-dd}.log" }

  file {
    path => '/var/log/logstash/mariadb_log.json'
  elasticsearch {
    hosts => [""]
    index => "powerdns-1-%{+YYYY.MM.dd}"

(Mark Walkom) #2

Does your message match?
What does that grok pattern look like? What does the data look like?

(John Woo Lee) #3

Yes the grok filter works fine.. but whats the grok pattern got to do with the index that logstash sends data too.. given that i explicitly tell it what it should be named.

(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.