Logstash failed to send logs to elasticsearch

Elastic Stack Logstash
1

Hello,

I am new to ELK and I just installed ELK Stack 6.4.2. I am trying to send windows Events to ELK.
I installed Elasticsearch and Kibana and then Logstash. I installed winlogbeat in a windows server and configured to send the events to logstash but logstash not sending the events to elasticsearch. Logstash log file showing this error,

[2018-10-04T00:28:02,495][ERROR][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"%{host}-%{[beat.version]}-%{+yyyy.MM.dd}-2018.10.04", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x106e99], :response=>{"index"=>{"_index"=>"%{host}-%{[beat.version]}-%{+yyyy.MM.dd}-2018.10.04", "_type"=>"doc", "_id"=>nil, "status"=>400, "error"=>{"type"=>"invalid_index_name_exception", "reason"=>"Invalid index name [%{host}-%{[beat.version]}-%{+yyyy.MM.dd}-2018.10.04], must be lowercase", "index_uuid"=>"na", "index"=>"%{host}-%{[beat.version]}-%{+yyyy.MM.dd}-2018.10.04"}}}}

My Logstash conf,

input {
beats {
port => 5044
}
}

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

output {
elasticsearch {
hosts => ["172.20.11.19:9200"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

Can somebody tell me what's wrong?

3

That error message does not seem to match your config. Do you have any other config files in the directory that could be causing this?

The problem seems to be that something goes wrong when creating the index name, which leaves upper case characters (not allowed) in the index name.

4

I have only one conf file in conf.d directory.

5

Was that error generated based on a different config? The error message complains that %{host}-%{[beat.version]}-%{+yyyy.MM.dd}-2018.10.04 is not a valid index name, and this does not match the configuration you provided.

6

My winlogbeat.yml configuration.

winlogbeat.event_logs:
  #- name: Application
    #ignore_older: 72h
  #- name: Security
  - name: System

output.logstash:
  # Boolean flag to enable or disable the output module.
  enabled: true

  # The Logstash hosts
  hosts: ["172.20.11.19:5044"]

  index: '172.20.11.16-%{[beat.version]}-%{+yyyy.MM.dd}'

# Set to false to disable template loading.
setup.template.enabled: true

# Template name. By default the template name is "winlogbeat-%{[beat.version]}"
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.name: "172.20.11.16-%{[beat.version]}"

# Template pattern. By default the template pattern is "-%{[beat.version]}-*" to apply to the default index settings.
# The first part is the version of the beat and then -* is used to match all daily indices.
# The template name and pattern has to be set in case the elasticsearch index pattern is modified.
setup.template.pattern: "172.20.11.16-%{[beat.version]}-*"
setup.template.settings:

  # A dictionary of settings to place into the settings.index dictionary
  # of the Elasticsearch template. For more details, please check
  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
  index:
    number_of_shards: 3
    #codec: best_compression
    #number_of_routing_shards: 30


setup.template.settings:

  # A dictionary of settings to place into the settings.index dictionary
  # of the Elasticsearch template. For more details, please check
  # https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
  index:
    number_of_shards: 3
    #codec: best_compression
    #number_of_routing_shards: 30

# files.
logging.to_files: true
logging.files:
  logging.level: debug
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  path: C:/ProgramData/winlogbeat/Logs
7

image
image.png1366×738 13.2 KB

8

image

9

when i was editing winlogbeat.yml I entered %{host}-%{[beat.version]}-%{+yyyy.MM.dd} as index name and then i got error starting winlogbeat service then i changed it to %{host}-%{[beat.version]}-%{+yyyy.MM.dd} and start the service.

10

i re-installed logstash and it's now sending logs to elasticsearch.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.