Index wont create

Elastic Stack Logstash
1

Hi all, i know its discussed a lot of times but i cant find my solution. A index is not created from one of my logstash config files and i cant find why. this is the config:

input {
  file {
    path => "/var/log/named/resolve_dnsproxy.log"
    exclude => "*.gz"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    ignore_older => 0
    type => "named"
  }
}

filter {
  if [type] == "named" {
    grok {
      patterns_dir => "/etc/logstash/patterns" 
      match => { "message" => ["%{NAMED}"] }
      }
    }
 }

output {
  if [type] == "named" {
      elasticsearch {
      index => "named-%{+YYYY.MM.dd}"
      hosts => [ "192.168.0.12:9200" ]
      user => elastic
      password => elastic
      manage_template => true
      template_overwrite => true
      template => "/etc/logstash/templates/named.json"
      template_name => "named"
    }
  }
}

Same config, different values is working a lot of times....

This is my template:

{
    "template" : "named-*",
    "order" : 1,
    "settings" : {
        "number_of_shards" : 2,
        "index.refresh_interval" : "90s"
    },
    "mappings" : {
        "named" : {
            "properties" : {
                "timestamp" : { "index": "true", "doc_values": true, "type" : "keyword" },
                "srcip" : { "index": "true", "doc_values": true, "type" : "keyword" },
                "port" : { "index": "true", "doc_values": true, "type" : "keyword" },
                "query" : { "index": "true", "doc_values": true, "type" : "keyword" },
                "dnstype" : { "index": "true", "doc_values": true, "type" : "keyword" },
                "resolver" : { "index": "true", "doc_values": true, "type" : "keyword" },
                "@timestamp" : { "format" : "dateOptionalTime", "type" : "date" } 
            }
        }
    }
}

Also this works for different log files successful!
I checked my patterns multiple times, at least with ELK6 and the grok debugger, no problems.

Access rights for the file(s):

-rwxr-xr-x 1 bind bind 2581 Jun 26 12:24 resolve_dnsproxy.log
-rwxr-xr-x 1 bind bind 36239 Jun 26 03:04 resolve_dnsproxy.log.1.gz

The user "logstash" is member of the group bind.

Please, can anyone shed some light. Thanks.

2

Comment out your elasticsearch output and use a stdout { codec => rubydebug } output while debugging.

Increase Logstash's loglevel and check out the log entries containing "resolve_dnsproxy.log". Any clues? I'm specifically interested in whether there is a file permission problem.

3

Hi Magnus,

i bet you asking for this (i saw an old thread with you):

[2018-06-26T23:19:16,250][DEBUG][logstash.inputs.file ] _globbed_files: /var/log/named/resolve_dnsproxy.log: glob is: ["/var/log/named/resolve_dnsproxy.log"]

Folder permissions:

drwxr-xr-x  2 bind     bind        4096 Jun 26 06:33 .
drwxr-xr-x 27 logstash logstash   12288 Jun 26 06:33 ..
-rwxr-xr-x  1 bind     bind      631112 Jun 26 23:24 resolve_dnsproxy.log
4

Yes, that's exactly what I was thinking of. Well, it's hardly a permissions problem then.

What does a stdout output produce? And the Logstash logs?

5

No errors in logstash debug....

Rubydebug produces:

{
      "path" => "/var/log/named/resolve_dnsproxy.log",
      "resolver" => "172.16.100.254",
      "@timestamp" => 2018-06-27T06:57:15.348Z,
      "port" => "64220",
      "query" => "api-global.eu-west-1.prodaa.netflix.com",
      "@version" => "1",
      "host" => "myhost",
      "source" => "172.17.200.136",
      "message" => "27-Jun-2018 08:57:14.955 client 172.17.200.136#64220 (api-global.eu-  west-1.prodaa.netflix.com): query: api-global.eu-west-1.prodaa.netflix.com IN A + (172.16.100.254)",
      "type" => [
       [0] "named",
       [1] "A"
    ],
     "timestamp" => "27-Jun-2018 08:57:14.955"
}

Not sure if it matters, but not all field names are present in the above (compared to my mapping file).

6

What a mess, it was a difference between mapping file and patterns that didnt match:

Patterns:

%{WORD:type}

Mapping:

"dnstype" : { "index": "true", "doc_values": true, "type" : "keyword" }

Changed patterns to ":dnstype" and index is created. Magnus thank you, i received a good learning curve.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.