No indexes created in Elasticsearch


#1

Hello. I'm using Elastic Stack 6.5 on my Windows 7. My workaround is Filebeat -> Logstash -> Elasticsearch -> Kibana. I run them and they look like working just fine as there's no error occurred. But there's no indexes/indices created in the Elasticsearch, thus my Kibana isn't showing anything.
My question is, what could be the problem or what should I check to find the problem?
Thank you.


(Pjanzen) #2

Hi,

Can you post you logstash input / filter / output ?

Please use the formatting tools :slight_smile:


#3

Hi, here's my Logstash config.

input {
	beats {
        port => "5044"
    }
}

filter {
	grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
	elasticsearch {
        hosts => ["localhost:9200"]
    }
    stdout { codec => rubydebug }
}

(Pjanzen) #4

Ok, you need to define an index to write to in your elasticsearch output.

Have a look here plugins-outputs-elasticsearch-index


#5

I've added the index definition but, still no indexes created. Instead, Filebeat seems to stop processing the data (before, it printed out the parsed data to the cmd).
Here's the log from Logstash/Filebeat:

[2018-12-17T03:01:58,118][INFO ][org.logstash.beats.BeatsHandler] [local: 0:0:0:0:0:0:0:1:5044, remote: 0:0:0:0:0:0:0:1:58059] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
[2018-12-17T03:01:58,119][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
	... 10 more

(Pjanzen) #6

I am unsure whats going on here this might help


#7

Thank you for the suggestion, but I don't think it's SSL issue as I don't have it enabled on both sides. (I made sure to check both config files) :smile:


(xeraa) #8

Could you add the Filebeat log as well, please?


#9

Sorry for the very late reply.

I just run it again but the Filebeat didn't create any log files anymore (it used to do tho). But, in the command prompt, it wrote this kind of output:

2019-01-10T15:52:11.893+0700    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":202},"total":{"ticks":857,"value":857},"user":{"ticks":655}},"handles":{"open":159},"info":{"ephemeral_id":"b70ccd0d-a965-4279-95ca-020306ebf7d8","uptime":{"ms":1906959}},"memstats":{"gc_next":4194304,"memory_alloc":2607360,"memory_total":23322816,"rss":8192}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"pipeline":{"clients":6,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}}}}}

Also, actually I want to ask, what's the difference of running Filebeat through command prompt and PowerShell?
When I try it on PowerShell, by running ".\filebeat.exe setup" I got message saying "Exiting: Template loading requested but the Elasticsearch output is not configured/enabled"

(I tried on both and I can't remember which one worked and created Filebeat logs earlier.)


#10

Oh, I just checked again and the log was created.
Here it is. Sorry for the confusion.

2019-01-10T15:51:58.731+0700	INFO	instance/beat.go:592	Home path: [D:\filebeat-6.5.2-windows-x86_64] Config path: [D:\filebeat-6.5.2-windows-x86_64] Data path: [D:\filebeat-6.5.2-windows-x86_64\data] Logs path: [D:\filebeat-6.5.2-windows-x86_64\logs]
2019-01-10T15:51:58.865+0700	INFO	instance/beat.go:599	Beat UUID: 13720ea1-7cf1-42eb-a033-8db29b20cc48
2019-01-10T15:51:58.865+0700	INFO	[beat]	instance/beat.go:825	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\filebeat-6.5.2-windows-x86_64", "data": "D:\\filebeat-6.5.2-windows-x86_64\\data", "home": "D:\\filebeat-6.5.2-windows-x86_64", "logs": "D:\\filebeat-6.5.2-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "13720ea1-7cf1-42eb-a033-8db29b20cc48"}}}
2019-01-10T15:51:58.865+0700	INFO	[beat]	instance/beat.go:834	Build info	{"system_info": {"build": {"commit": "b48d073b84e874a182c122d8ef2bad867f714a11", "libbeat": "6.5.2", "time": "2018-11-29T23:03:04.000Z", "version": "6.5.2"}}}
2019-01-10T15:51:58.866+0700	INFO	[beat]	instance/beat.go:837	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.10.3"}}}
2019-01-10T15:51:58.951+0700	INFO	[beat]	instance/beat.go:841	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-07T16:27:24.19+07:00","name":"Aurelia","ip":["fe80::987d:ecc7:3b6f:d919/64","169.254.217.25/16","fe80::a479:6679:e042:97b8/64","192.168.88.23/24","fe80::e86b:f8bf:bf80:3c84/64","169.254.60.132/16","fe80::e9fe:9397:46a7:4468/64","169.254.68.104/16","::1/128","127.0.0.1/8","fe80::e0:0:0:0/64","fe80::5efe:c0a8:5817/128"],"kernel_version":"6.1.7601.17592 (win7sp1_gdr.110408-1631)","mac":["70:f3:95:b2:67:d7","70:f3:95:b2:67:d7","70:f3:95:d4:83:c4","60:eb:69:6f:89:66","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows 7 Professional","version":"6.1","major":1,"minor":0,"patch":0,"build":"7601.0"},"timezone":"+07","timezone_offset_sec":25200,"id":"df7fe80c-f48d-4f51-acc6-ed55f3eb1897"}}}
2019-01-10T15:51:58.952+0700	INFO	[beat]	instance/beat.go:870	Process info	{"system_info": {"process": {"cwd": "D:\\filebeat-6.5.2-windows-x86_64", "exe": "D:\\filebeat-6.5.2-windows-x86_64\\filebeat.exe", "name": "filebeat.exe", "pid": 6840, "ppid": 1676, "start_time": "2019-01-10T15:51:58.601+0700"}}}
2019-01-10T15:51:58.952+0700	INFO	instance/beat.go:278	Setup Beat: filebeat; Version: 6.5.2
2019-01-10T15:52:01.992+0700	INFO	add_cloud_metadata/add_cloud_metadata.go:319	add_cloud_metadata: hosting provider type not detected.
2019-01-10T15:52:01.992+0700	INFO	[publisher]	pipeline/module.go:110	Beat name: Aurelia
2019-01-10T15:52:01.993+0700	ERROR	instance/beat.go:800	Exiting: Template loading requested but the Elasticsearch output is not configured/enabled

(xeraa) #11

That's the problem. You have configured the Beat to load templates, which is the default. To do that the Beat needs to talk to Elasticsearch directly (and not through Logstash as you have configured it). And because the loading step is failing, Beats just stops and doesn't transfer any data. It is a little hidden in the docs — see the note in https://www.elastic.co/guide/en/beats/filebeat/current/configuration-template.html#configuration-template.

To fix that: Disable automatic template loading with this setting in the filebeat.yml setup.template.enabled: false and then load it manually: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-template.html#load-template-manually.


(andy_zhou) #12

output error.
see the data.


#13

I've followed the instruction. Now the message "Exiting: ..." isn't showing anymore, but there is still no index created in the Elasticsearch. When I run "Start-Service filebeat" nothing showed in the screen.

Here's the latest Filebeat logs.

2019-01-14T13:26:03.906+0700	INFO	instance/beat.go:592	Home path: [D:\filebeat-6.5.2-windows-x86_64] Config path: [D:\filebeat-6.5.2-windows-x86_64] Data path: [D:\filebeat-6.5.2-windows-x86_64\data] Logs path: [D:\filebeat-6.5.2-windows-x86_64\logs]
2019-01-14T13:26:04.245+0700	INFO	instance/beat.go:599	Beat UUID: 3304ca32-b5a1-4858-9967-47569e9b89a4
2019-01-14T13:26:04.245+0700	INFO	[beat]	instance/beat.go:825	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\filebeat-6.5.2-windows-x86_64", "data": "D:\\filebeat-6.5.2-windows-x86_64\\data", "home": "D:\\filebeat-6.5.2-windows-x86_64", "logs": "D:\\filebeat-6.5.2-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "3304ca32-b5a1-4858-9967-47569e9b89a4"}}}
2019-01-14T13:26:04.381+0700	INFO	[beat]	instance/beat.go:834	Build info	{"system_info": {"build": {"commit": "b48d073b84e874a182c122d8ef2bad867f714a11", "libbeat": "6.5.2", "time": "2018-11-29T23:03:04.000Z", "version": "6.5.2"}}}
2019-01-14T13:26:04.422+0700	INFO	[beat]	instance/beat.go:837	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.10.3"}}}
2019-01-14T13:26:04.659+0700	INFO	[beat]	instance/beat.go:841	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-13T21:14:36.03+07:00","name":"Aurelia","ip":["fe80::987d:ecc7:3b6f:d919/64","169.254.217.25/16","fe80::a479:6679:e042:97b8/64","10.5.26.4/22","fe80::e86b:f8bf:bf80:3c84/64","169.254.60.132/16","fe80::e9fe:9397:46a7:4468/64","167.205.34.104/24","::1/128","127.0.0.1/8","fe80::e0:0:0:0/64","fe80::5efe:a05:1a04/128","fe80::200:5efe:a7cd:2268/128","2002:a7cd:2268::a7cd:2268/128"],"kernel_version":"6.1.7601.17592 (win7sp1_gdr.110408-1631)","mac":["70:f3:95:b2:67:d7","70:f3:95:b2:67:d7","70:f3:95:d4:83:c4","60:eb:69:6f:89:66","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows 7 Professional","version":"6.1","major":1,"minor":0,"patch":0,"build":"7601.0"},"timezone":"+07","timezone_offset_sec":25200,"id":"df7fe80c-f48d-4f51-acc6-ed55f3eb1897"}}}
2019-01-14T13:26:04.660+0700	INFO	[beat]	instance/beat.go:870	Process info	{"system_info": {"process": {"cwd": "D:\\filebeat-6.5.2-windows-x86_64", "exe": "D:\\filebeat-6.5.2-windows-x86_64\\filebeat.exe", "name": "filebeat.exe", "pid": 2828, "ppid": 2532, "start_time": "2019-01-14T13:25:42.240+0700"}}}
2019-01-14T13:26:04.660+0700	INFO	instance/beat.go:278	Setup Beat: filebeat; Version: 6.5.2
2019-01-14T13:26:07.825+0700	INFO	add_cloud_metadata/add_cloud_metadata.go:319	add_cloud_metadata: hosting provider type not detected.
2019-01-14T13:26:07.904+0700	INFO	elasticsearch/client.go:163	Elasticsearch url: http://localhost:9200
2019-01-14T13:26:07.942+0700	INFO	[publisher]	pipeline/module.go:110	Beat name: Aurelia
2019-01-14T13:26:07.997+0700	INFO	elasticsearch/client.go:163	Elasticsearch url: http://localhost:9200
2019-01-14T13:26:08.457+0700	INFO	elasticsearch/client.go:712	Connected to Elasticsearch version 6.5.2
2019-01-14T13:26:08.511+0700	INFO	template/load.go:129	Template already exists and will not be overwritten.

Where could I go wrong?


#14

I checked my filebeat.yml again and found mistakes there and fixed it and now it's working fine. Thank you so much.