Can't create Winlogbeat index in Elasticsearch


#1

Hi,

I'm having trouble to send Winlogbeat index in Elasticsearch.

I tried to do it from the config file

In winlogbeat.yml :

output.elasticsearch:
hosts: ["172.25.3.18:9200"]

template.name: "winlogbeat"
#template.path:"winlogbeat.template.json"
dashboards.enabled: true
index: "winlogbeat"

(I had to comment the line "template.path: winlogbeat.template.json" because it gave me an error on this line when I tried to turn on Winlogbeat)

I also tried to upload the index manually :

curl -u user:pass -X PUT 'http://172.25.3.18:9200/_template/winlogbeat?pretty' @/path/to/conf/file but it returns this error :

{
"error" : {
"root_cause" : [
{
"type" : "parse_exception",
"reason" : "Failed to parse content to map"
}
],
"type" : "parse_exception",
"reason" : "Failed to parse content to map",
"caused_by" : {
"type" : "json_parse_exception",
"reason" : "Unexpected end-of-input within/between Object entries\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@714b28c2; line: 1, column: 16141]"
}
},
"status" : 400
}

I'm running Elasticsearch with Docker.

My elasticsearch config :

elasticsearch:
image: 08794661d6b0
container_name: elk5_elasticsearch
hostname: elasticsearch
ports:
- 9200:9200
- 9300:9300
environment:
- cluster.name=docker-elk5
- node.name=elk5
- bootstrap.memory_lock=true
- network.host=172.25.3.18
- network.bind_host=0.0.0.0
- transport.tcp.port=9300-9400
- transport.host=172.25.3.18
- transport.bind_host=0.0.0.0
- "ES_JAVA_OPTS=-Xms4g -Xmx4g"
- discovery.zen.ping.unicast.hosts=172.25.3.18:9300
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 5g
cap_add:
- IPC_LOCK
volumes:
- elastic:/etc/elasticsearch
- /opt/docker/elasticdata/docker-el5:/usr/share/elk5/data
restart: on-failure

My Winlogbeat config :

#======================= Winlogbeat specific options ==========================`

# event_logs specifies a list of event logs to monitor as well as any
# accompanying options. The YAML data type of event_logs is a list of
# dictionaries.
#
# The supported keys are name (required), tags, fields, fields_under_root,
# forwarded, ignore_older, level, event_id, provider, and include_xml. Please
# visit the documentation for the complete details of each option.
# https://go.es.io/WinlogbeatConfig
winlogbeat.event_logs:
  - name: Application
    ignore_older: 72h
  - name: Security
  - name: System

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging

#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat.
# Multiple outputs may be used.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["172.25.3.18:9200"]

  template.name: "winlogbeat"
  #template.path:"winlogbeat.template.json"
  dashboards.enabled: true
  index: "winlogbeat"
  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "user"
  password: "pass"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  #hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
logging.level: info

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

The weirdest thing is that I can see "winlogbeat-*" in the index patterns in Kibana, but I can't see Winlogbeat index when I run :

curl -u user:pass http://172.25.3.18:9200/_cat/indices

yellow open .monitoring-logstash-2-2017.06.07 PeEydERRQ5Gv2Dhz3NPxOA 1 1 936 0 285.2kb 285.2kb
yellow open .triggered_watches I9U9E9dRRjaXbGk3C687AA 1 1 0 0 32.1kb 32.1kb
yellow open .monitoring-alerts-2 hDhH-oYIR9KmGNN_JD0wmQ 1 1 1 0 6.5kb 6.5kb
yellow open .monitoring-es-2-2017.06.07 aBvRJ9nYRE2jlfGxiaDHsw 1 1 14213 108 7.1mb 7.1mb
yellow open .monitoring-kibana-2-2017.06.07 ObcYgENoSpiCNLd5lXNcYg 1 1 937 0 347.5kb 347.5kb
yellow open .watches IhjiyZOZQuCV_fmo-IPUUw 1 1 4 0 11.7kb 11.7kb
yellow open .watcher-history-3-2017.06.07 J-H5-FQsQs6ZF1mw-Lo3hQ 1 1 780 0 829.3kb 829.3kb
yellow open .monitoring-data-2 S77fdDltT_Ond0Bdsqnh8g 1 1 4 0 10.2kb 10.2kb
yellow open .kibana vXNkATasRIyx29Fx5lzQyg 1 1 8 0 19.8kb 19.8kb

Has anyone ever seen that?

Thank you


#2

I am not expert on ELK ( yet ), but reading your post brings up some ideas:

The index pattern is already in Kibana probably because you used the script to import Kibana dashboards. If not created by script, Index patterns are usually created manually.

The index alone will be created when you start winlogbeat and it will first time send real data to ES.

In the config file, you have modified index name to winlogbeat. That name probably won't be seen under index pattern winlogbeat-*, because it's missing the -. Is there any reason why didn't you leave default index name? Default name would be based on current day and look like winlogbeat-2017.06.04.

The error you posted seems like some syntax error. Do you need to use customized template ? If not, winlogbeat will load default template if you don't specify anything in configuration. If you need to use customized template, try to load it through Kibana -> Developer tools. Developer tools console is automatically checking syntax and marking errors.


(Andrew Kroh) #3

This command is missing the -d.

curl -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/winlogbeat?pretty' -d@winlogbeat.template.json

But you shouldn't need to manually install the template if you are using the Elasticsearch output.

Ditto that. I recommend using default settings if you are getting started. Then once you have it working you can begin to customize things. Changing the index name has a lot of implications.


#4

Thanks for your replies,

I modified Winlogbeat config, now I have :

-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:

Array of hosts to connect to.

hosts: ["172.25.3.18:9200"]

dashboards.enabled: true
index: "winlogbeat-%{+yyyy.MM.dd}"

Optional protocol and basic auth credentials.

#protocol: "https"
username: "elastic"
password: "changeme"

But I still can't fetch "winlogbeat-*" when configuring a search pattern in Kibana and I still can't see Winlogbeat in the indices :

curl -u elastic:changeme http://172.25.3.18:9200/_cat/indices/

yellow open .monitoring-es-2-2017.06.07 6P7aBXFiRX6dMAJSW96VdA 1 1 2696 126 1.5mb 1.5mb
yellow open .triggered_watches FTnSE9QeTf-xktFM8zR6ng 1 1 0 0 130b 130b
yellow open .watches 8gMkUP8XTRuswvc0yIb_vg 1 1 4 0 19.6kb 19.6kb
yellow open .watcher-history-3-2017.06.07 pc_u-JrHTaKD2EIojJQu3w 1 1 140 0 271.5kb 271.5kb
yellow open .monitoring-alerts-2 948p6yRuShia2-Xc62i_pQ 1 1 1 0 6.5kb 6.5kb
yellow open .monitoring-logstash-2-2017.06.07 06EaV8qDSw6UQay5PP1QDw 1 1 172 0 173.6kb 173.6kb
yellow open .monitoring-kibana-2-2017.06.07 kMmA6hbtQISCKKq8_LdtGQ 1 1 172 0 176kb 176kb
yellow open .monitoring-data-2 X_7prVMATEm4vQj1p52uEQ 1 1 4 0 10.2kb 10.2kb
yellow open .kibana 7wTj6Rq1RkWPqEySyhuwUA 1 1 1 0 3.8kb 3.8kb

For the modified template, I already had Winlogbeat installed on a Windows machine, and the index in Elasticsearch, but I had an error in Kibana saying that "fielddata" was set to false on log_name, source_name and level, so I modified the template to add "fielddata":true on these fields and I deleted Winlogbeat index, in order to reinstall it properly.

I also tried to add it manually with the command andrewkroh gave me, but still get the same error :

 curl -u elastic:changeme -H 'Content-Type: application/json' -XPUT 'http://localhost:9200/_template/winlogbeat?pretty' -d@/usr/share/elk5/config/index_winlogbeat.json

{
  "error" : {
    "root_cause" : [
      {
        "type" : "parse_exception",
        "reason" : "Failed to parse content to map"
      }
    ],
    "type" : "parse_exception",
    "reason" : "Failed to parse content to map",
    "caused_by" : {
      "type" : "json_parse_exception",
      "reason" : "Unexpected end-of-input within/between Object entries\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@2f94c8a3; line: 1, column: 16141]"
    }
  },
  "status" : 400

Thanks again for your replies !


#5

Sorry for double posting,

I tried to reimport the dashboard, now I can see winlogbeat-* in the search patterns (I selected it as default pattern).

My Windows machine and Winlogbeats are running (I installed Winlogbeat as service), but I can't see any data in Kibana (in Discover and in Winlogbeat dashboard)

And I still can't see Winlogbeat with the command : curl -u user:pass http://172.25.3.18:9200/_cat/indices

Thanks


#6

Winlogbeat logs be your friend here. You can also share them, if there is something interesting.


#7

Didn't thought to look at these logs...

It seems that I was still using the modified template, redownloaded it and replaced it with the original, now everything seems to be OK.

I can see data in Discover, I can see the dashboard and I can see Winlogbeats index when I run the command : curl -u user:pass http://172.25.3.18:9200/_cat/indices

Thanks for your help !! :slight_smile:


(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.