No matching indices found: No indices match pattern "winlogbeat-*"

Elastic Stack Elasticsearch
1

I am getting the errors "No matching indices found: No indices match pattern "winlogbeat-*" when I try to go to the Winlogbeat dashboard or Discover.

There was a winlogbeat-* index pattern created, just no traffic hitting Elastic.

When I TCP dump I can see the logs coming from the windows server to the Elastic server. I have checked the UFW in Ubuntu and it is disabled.

I am betting this is a N00B issue, but I can't seem to get it figured out. None of the other posts in the forum seem to apply to me.

2

Hi @sixstringssteve welcome to the community.

What versions are you running?

Can you navigate to Dev Tools in Kibana and run this command and provide the output?

GET /_cat/indices?v&s=pri.store.size:desc

Also when you say there "was" a winlogbeat-* index pattern created, can you verify it is still there? It will be under Kibana - Stack Management - Index Patterns. You can delete and re-create it if necessary.

3

When I go to Index Patterns it is still listed as an index. i am running 7.8.1

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open metricbeat-7.8.1-2021.01.04-000001 U2415ztsQDC3wlXh2VmnYw 1 1 20406 0 14.1mb 14.1mb
green open .async-search rxZCD5vaRZKw5Nbj33cStQ 1 0 3 0 3.9mb 3.9mb
green open .kibana_1 9PSy9mX2RKe3h54TXBcGWA 1 0 3056 9 2.2mb 2.2mb
green open .kibana_task_manager_1 66P4hSSnQS6QSDPqfajq-A 1 0 5 0 8.4kb 8.4kb
green open .kibana-event-log-7.8.1-000001 xUPWGbZURGOVLRROcFKzbQ 1 0 1 0 5.3kb 5.3kb
yellow open filebeat-7.10.1-2021.01.04-000001 zUIqZELYRZWrktcz9S18ew 1 1 0 0 208b 208b
green open .apm-custom-link 3QCiWwRTSle3wT61Z1Zwew 1 0 0 0 208b 208b
green open .apm-agent-configuration 1AePUx3jRqSf8onkSuruFQ 1 0 0 0 208b 208b

4

Right so there is no winlogbeat-... indices you are collecting metricbeat and filebeat data at this point. An Index Pattern is not an actual index, it contains no actual data, it is just really an alias with datatypes to aid searching and visualizations.

You probably have a filebeat-* index pattern and it is related to filebeat-7.10.1-2021.01.04-000001 actual index... it will be related ot any actual index that starts with filebeat-*

Did you actually install winlogbeat and is it running on at least 1 host? If so I would go to one of the host that you are running winlogbeat on and look at the winlogbeat logs, the data is not getting to this elasticsearch cluster.

5

I see what you mean, my mistake. When I look in Index Management, I see the list of indexes missing winlogbeat.

I am running it on a Windows Server. When I TCPDump the Ununtu box, I can see traffic hitting 9200 so it looks like the Winlogbeat agent on the Windows Server is sending data, just doesn't look like it it being received by elastic.

23:00:13.488593 IP dc1.xxx.61333 > elastic.9200: Flags [SEW], seq 3272084836, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
23:00:13.988022 IP dc1.xxx.61333 > elastic.9200: Flags [S], seq 3272084836, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
23:00:14.488032 IP dc1.xxx.61333 > elastic.9200: Flags [S], seq 3272084836, win 8192, options [mss 1460,nop,nop,sackOK], length 0

What are my options?

6

@sixstringssteve

So you need to show us the logs from winlogbeat.... the tcpdump is nice but not informative enough to debug / tell us why the data is rejected.

Can you stop, then start winlogbeat and provide the logs...

7

I cannot seem to find the logs. Do you have a link to the process for enabling logging? I found the section in the .yml file, but cannot find the log file path.

8

I would suggest to just start winlogbeat from the command line and watch the output.

winlogbeat -e

When started as a service it should log by default, this raises my question what changes have been made to the the configuration.

9

found them...sorry

There are several logs files winlogbeat, winlogbeat.1, and winlogbeat.2 but they are a total of 22,000KB. How do i share these?

When I run the app from command line the logs it produces overload the buffer. I put logging on debug to try to get the details.

10

Could this be a permissions issue? I did not set this instance up with authentication so I did not set up an elastic user/password. So I left those items commented out in the .yml file.

Could this be the problem?

11

When you say "This instance" do you mean winlogbeat or the elastic cluster / node.

If the elastic instance requires authentication then winlogbeat needs to provide username / pw , if the elastic instance does not require authentication the winlogbeat does not.

Turn off debug logging

Start from the command line or clean out the logs and start as a service.

Paste in the first 100 lines of logs or so.

Please format using the </> button above.

12

Look like the server is actively refusing the data flow.

c:\Program Files\winlogbeat>winlogbeat -e
2021-01-04T18:40:22.626-0600 INFO instance/beat.go:645 Home path: [c:\P
rogram Files\winlogbeat] Config path: [c:\Program Files\winlogbeat] Data path: [
c:\Program Files\winlogbeat\data] Logs path: [c:\Program Files\winlogbeat\logs]
2021-01-04T18:40:22.627-0600 INFO instance/beat.go:653 Beat ID: 13d3c4f
1-d20f-474e-8518-1845908494e5
2021-01-04T18:40:22.640-0600 INFO [beat] instance/beat.go:981 Beat inf
o {"system_info": {"beat": {"path": {"config": "c:\Program Files\winlogb
eat", "data": "c:\Program Files\winlogbeat\data", "home": "c:\Program Files
\winlogbeat", "logs": "c:\Program Files\winlogbeat\logs"}, "type": "winlogbea
t", "uuid": "13d3c4f1-d20f-474e-8518-1845908494e5"}}}
2021-01-04T18:40:22.640-0600 INFO [beat] instance/beat.go:990 Build in
fo {"system_info": {"build": {"commit": "1da173a9e716715a7a54bb3ff4db05b5c2
4fc8ce", "libbeat": "7.10.1", "time": "2020-12-04T23:46:31.000Z", "version": "7.
10.1"}}}
2021-01-04T18:40:22.641-0600 INFO [beat] instance/beat.go:993 Go runti
me info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"ver
sion":"go1.14.12"}}}
2021-01-04T18:40:22.648-0600 INFO [beat] instance/beat.go:997 Host inf
o {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-01-0
2T18:56:38.76-06:00","name":"dc1","ip":["172.16.100.11/24","::1/128","127.0.0.1/
8"],"kernel_version":"6.3.9600.19880 (winblue_ltsb.201021-0600)","mac":["00:1a:4
b:cc:4f:66","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"wind
ows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":
0,"patch":0,"build":"9600.19893"},"timezone":"CST","timezone_offset_sec":-21600,
"id":"a0b76e9f-ecf1-4757-898e-528c7383a804"}}}
2021-01-04T18:40:22.649-0600 INFO [beat] instance/beat.go:1026 Process
info {"system_info": {"process": {"cwd": "c:\Program Files\winlogbeat", "ex
e": "C:\Program Files\winlogbeat\winlogbeat.exe", "name": "winlogbeat.exe", "
pid": 4616, "ppid": 776, "start_time": "2021-01-04T18:40:22.506-0600"}}}
2021-01-04T18:40:22.649-0600 INFO instance/beat.go:299 Setup Beat: winl
ogbeat; Version: 7.10.1
2021-01-04T18:40:22.650-0600 INFO [index-management] idxmgmt/std.go:1
84 Set output.elasticsearch.index to 'winlogbeat-7.10.1' as ILM is enabled.

2021-01-04T18:40:22.650-0600 INFO eslegclient/connection.go:99 elastics
earch url: http://172.16.100.22:9200
2021-01-04T18:40:22.652-0600 INFO [publisher] pipeline/module.go:113
Beat name: dc1
2021-01-04T18:40:22.652-0600 INFO beater/winlogbeat.go:69 State will be re
ad from and persisted to c:\Program Files\winlogbeat\data.winlogbeat.yml
2021-01-04T18:40:22.724-0600 WARN [cfgwarn] registered_domain/regist
ered_domain.go:60 BETA: The registered_domain processor is beta.
2021-01-04T18:40:22.804-0600 WARN [cfgwarn] registered_domain/regist
ered_domain.go:60 BETA: The registered_domain processor is beta.
2021-01-04T18:40:22.824-0600 INFO [monitoring] log/log.go:118 Starting
metrics logging every 30s
2021-01-04T18:40:22.824-0600 INFO kibana/client.go:119 Kibana url: http
://172.16.100.22:5601
2021-01-04T18:40:23.203-0600 INFO kibana/client.go:119 Kibana url: http
://172.16.100.22:5601
2021-01-04T18:40:25.647-0600 INFO [add_cloud_metadata] add_cloud_metada
ta/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not de
tected.
2021-01-04T18:40:30.162-0600 INFO instance/beat.go:815 Kibana dashboard
s successfully loaded.
2021-01-04T18:40:30.162-0600 INFO instance/beat.go:455 winlogbeat start
running.
2021-01-04T18:40:30.264-0600 WARN beater/eventlogger.go:124 EventLog
[Microsoft-Windows-Sysmon/Operational] Open() error. No events will be read from
this source. The specified channel could not be found. Check channel configurat
ion.
2021-01-04T18:40:31.625-0600 INFO [publisher_pipeline_output] pipeline
/output.go:143 Connecting to backoff(elasticsearch(http://172.16.100.22:9200))
2021-01-04T18:40:31.629-0600 INFO [publisher] pipeline/retry.go:219
retryer: send unwait signal to consumer
2021-01-04T18:40:31.634-0600 INFO [publisher] pipeline/retry.go:223
done
2021-01-04T18:40:33.975-0600 ERROR [publisher_pipeline_output] pipeline
/output.go:154 Failed to connect to backoff(elasticsearch(http://172.16.100.22:
9200)): Get "http://172.16.100.22:9200": dial tcp 172.16.100.22:9200: connectex:
No connection could be made because the target machine actively refused it.
2021-01-04T18:40:33.980-0600 INFO [publisher_pipeline_output] pipeline
/output.go:145 Attempting to reconnect to backoff(elasticsearch(http://172.16.1
00.22:9200)) with 1 reconnect attempt(s)
2021-01-04T18:40:33.981-0600 INFO [publisher] pipeline/retry.go:219
retryer: send unwait signal to consumer
2021-01-04T18:40:33.989-0600 INFO [publisher] pipeline/retry.go:223
done
2021-01-04T18:40:37.381-0600 ERROR [publisher_pipeline_output] pipeline
/output.go:154 Failed to connect to backoff(elasticsearch(http://172.16.100.22:
9200)): Get "http://172.16.100.22:9200": dial tcp 172.16.100.22:9200: connectex:
No connection could be made because the target machine actively refused it.
2021-01-04T18:40:37.386-0600 INFO [publisher_pipeline_output] pipeline
/output.go:145 Attempting to reconnect to backoff(elasticsearch(http://172.16.1
00.22:9200)) with 2 reconnect attempt(s)
2021-01-04T18:40:37.386-0600 INFO [publisher] pipeline/retry.go:213
retryer: send wait signal to consumer
2021-01-04T18:40:37.395-0600 INFO [publisher] pipeline/retry.go:217
done
2021-01-04T18:40:44.397-0600 ERROR [publisher_pipeline_output] pipeline
/output.go:154 Failed to connect to backoff(elasticsearch(http://172.16.100.22:
9200)): Get "http://172.16.100.22:9200": dial tcp 172.16.100.22:9200: connectex:
No connection could be made because the target machine actively refused it.
2021-01-04T18:40:44.402-0600 INFO [publisher_pipeline_output] pipeline
/output.go:145 Attempting to reconnect to backoff(elasticsearch(http://172.16.1
00.22:9200)) with 3 reconnect attempt(s)
2021-01-04T18:40:45.219-0600 INFO beater/winlogbeat.go:161 Stopping
Winlogbeat
2021-01-04T18:40:45.221-0600 INFO beater/eventlogger.go:129 EventLog
[System] Stop processing.
2021-01-04T18:40:45.225-0600 INFO beater/eventlogger.go:129 EventLog
[Security] Stop processing.
2021-01-04T18:40:45.430-0600 INFO beater/eventlogger.go:129 EventLog
[ForwardedEvents] Stop processing.
2021-01-04T18:40:45.430-0600 INFO beater/eventlogger.go:129 EventLog
[Microsoft-Windows-PowerShell/Operational] Stop processing.
2021-01-04T18:40:45.725-0600 INFO beater/eventlogger.go:129 EventLog
[Windows PowerShell] Stop processing.
2021-01-04T18:40:45.975-0600 INFO beater/eventlogger.go:129 EventLog
[Application] Stop processing.
2021-01-04T18:40:45.979-0600 INFO [monitoring] log/log.go:153 Total no
n-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":609
,"time":{"ms":609}},"total":{"ticks":5827,"time":{"ms":5827},"value":5827},"user
":{"ticks":5218,"time":{"ms":5218}}},"handles":{"open":223},"info":{"ephemeral_i
d":"4516a614-1620-4289-974f-d3613fe31f44","uptime":{"ms":23423}},"memstats":{"gc
_next":76336016,"memory_alloc":39950680,"memory_total":243544000,"rss":101261312
},"runtime":{"goroutines":22}},"libbeat":{"config":{"module":{"running":0}},"out
put":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":4116,"f
ailed":124,"published":4116,"total":4240}}},"msg_file_cache":{"ApplicationHits":
61,"ApplicationMisses":15,"ApplicationSize":15,"SecurityHits":1499,"SecurityMiss
es":1,"SecuritySize":1,"SystemHits":2195,"SystemMisses":5,"SystemSize":5,"Window
s PowerShellHits":464,"Windows PowerShellMisses":1,"Windows PowerShellSize":1},"
system":{"cpu":{"cores":2}}}}}
2021-01-04T18:40:45.993-0600 INFO [monitoring] log/log.go:154 Uptime:
23.4383784s
2021-01-04T18:40:46.000-0600 INFO [monitoring] log/log.go:131 Stopping
metrics logging.
2021-01-04T18:40:46.006-0600 INFO instance/beat.go:461 winlogbeat stopp
ed.

13

Yup you got a FW issue or network issue, that is not an authentication issue.

You can try to simply

wget http://172.16.100.22:9200

If it can connect you will get a response but I suspect it won't

You could also try to telnet but I suspect it won't connect, think you need to figure out your connectivity issue

14

OK, so when I set up the server I configured the Kibana .yml file to bind the network host to 0.0.0.0:5061 and I never did anything with the elasticserach config.

When I do the same with elasticsearch.yml and I try to restart the service I get a failure.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 0.0.0.0

Set a custom port for HTTP:

http.port: 9200

For more information, consult the network module documentation.

root@elastic:/etc/elasticsearch# /etc/init.d/elasticsearch start
[....] Starting elasticsearch (via systemctl): elasticsearch.serviceJob for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.
failed!

it appears as if I cannot bind the elasticsearch network connection in a way that allows any access other than localhost.

15

First @sixstringssteve in the future please format your yml with the formatting button above </> otherwise it is very difficult to help.

2nd there are some important concept that you really need to understand / take a look at with respect to Elasticsearch ...

Take a look at This post that I wrote up a while ago... this is one windows but the concepts are the same

In short, Elastic forces you to make a very conscious decision to attach your cluster to the network... as soon as you do you should be thinking about auth / auth data security etc. With the proper configuration you can dev / test etc , but you need to set some settings and you should read about them first... but in no way do we recommend putting production data in an unsecured cluster.

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.