I cannot see two indices in Kibana

Elastic Stack Logstash
1

I have the following output in my logstash output file. When I look at the loaded indices in Kibana I only see the var_log_common-*

30-elasticsearch-output.conf

output {
  if [source] == "/var/log/nginx/access.log" or [source] == "/var/log/nginx/error.log" {
    elasticsearch {
      hosts => "localhost:9200"
      index => "var_log_nginx-%{+YYYY.MM.dd}"
    }
  } 
  else {
    elasticsearch {
      hosts => "localhost:9200"
      index => "var_log_common-%{+YYYY.MM.dd}"
    }
  }
}

Does this look correct to you?

2

It looks OK. Are you sure that you are receiving messages that should match? If the condition isn't matching you would see these messages in the var_log_common-* index, as that is where they would land if the condition was false.

3

Update:

After running this in elasticsearch I see the index is not listed:

curl http://localhost:9200/_aliases?pretty=true
{
  ".kibana_1" : {
    "aliases" : {
      ".kibana" : { }
    }
  },
  "var_log_common-2020.03.09" : {
    "aliases" : { }
  },
  ".kibana_task_manager_1" : {
    "aliases" : {
      ".kibana_task_manager" : { }
    }
  },
  "var_log_common-2020.03.08" : {
    "aliases" : { }
  },
  ".apm-agent-configuration" : {
    "aliases" : { }
  }
}
4

So if I understand the logic correctly my if statement is say that I should receive messages from both sources if they are available or?

5

What your statement says is... if the value of the source field is /var/log/nginx/access.log or the value of the source field is /var/log/nginx/error.log then output the messages to the index var_log_nginx-%{+YYYY.MM.dd}, otherwise output the messages to var_log_common-%{+YYYY.MM.dd}.

If the source field is not being set as you expect, then the messages won't match and will land in var_log_common-%{+YYYY.MM.dd}.

6

ok then that is correct. Here are the grok filters for my nginx:

 match => { "message" => "%{NGINX_ERROR}" } 
 match => { "message" => "%{NGINX_ACCESS}" }

Patterns

ERRORDATE %{YEAR}/%{MONTHNUM}/%{MONTHDAY} %{TIME}
NGINX_ERROR %{ERRORDATE:timestamp} \[%{LOGLEVEL:loglevel}\] %{INT:process_id}#%{INT:thread_id}: \*(%{INT:connection_id})? %{NOTSPACE:request} %{NOTSPACE:method} %{GREEDYDATA:Error_Message} %{IP:ClientIP}

METHOD (OPTIONS|GET|HEAD|POST|PUT|DELETE|TRACE|CONNECT)
NGINX_ACCESS %{IPORHOST:visitor_ip} - %{USERNAME:remote_user} \[%{HTTPDATE:time_local}\] "%{DATA:server_name}" "%{METHOD:method} %{URIPATHPARAM:path} HTTP/%{NUMBER:http_version}" %{INT:status} %{INT:body_bytes_sent} "%{URI:referer}" %{QS:user_agent}
7

That grok stuff isn't setting the source field. How are you setting source?

8

Good question how to identify what you want to see?

9

If you don't know, you probably aren't setting it. Show your logstash input section.

10 
input {
  beats {
  port => 2561
  ssl => true
  ssl_certificate => "/etc/ssl/certs/mon-1.crt"
  ssl_key => "/etc/ssl/private/mon-1.p8"
  }
}

And here is my filebeats.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/nginx/error.log
    - /var/log/nginx/access.log
    - /var/log/common/admin.log
    
  exclude_files: ['\.gz$']
  multiline.pattern: ^\[
  multiline.negate: true
  multiline.match: after

  #filebeat.config.modules:
  # Glob pattern for configuration loading
  #path: ${path.config}/modules.d/*.yml
  
  logging.level: debug
  logging.to_files: true
  logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644
  
  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

name: "app2-filebeat"
tags: ["app2", "app"]

output.logstash:
  hosts: ["mon-1:2561"]
  ssl.certificate_authorities: #####
  ssl.certificate: ######
  ssl.key: #####
  ssl.verification_mode: none

ignore the ssl stuff I have commented it out for this post.

11

This is coming from Filebeat. In filebeat 6.x the name of the field was source, as of 7.x the field is named log.file.path. Try changing your if statement to use [log][file][path] instead of [source].

1 Like
12

that was it. Thanks man

13

Glad to help. I am going to add this Beats-to-Logstash use-case to my list of future "how-to" videos (https://www.youtube.com/channel/UCivWvTx1DwrWNcDLV58kmOg). I use this method a lot, but have found there are a lot of questions about it.

14

great I just subscribed, but now thats working none of my filters are working :wink:

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.