Hello. I'm using Elastic Stack 6.5 on my Windows 7. My workaround is Filebeat -> Logstash -> Elasticsearch -> Kibana. I run them and they look like working just fine as there's no error occurred. But there's no indexes/indices created in the Elasticsearch, thus my Kibana isn't showing anything.
My question is, what could be the problem or what should I check to find the problem?
Thank you.
Hi,
Can you post you logstash input / filter / output ?
Please use the formatting tools 
Hi, here's my Logstash config.
input {
beats {
port => "5044"
}
}
filter {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
geoip {
source => "clientip"
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}Ok, you need to define an index to write to in your elasticsearch output.
Have a look here plugins-outputs-elasticsearch-index
I've added the index definition but, still no indexes created. Instead, Filebeat seems to stop processing the data (before, it printed out the parsed data to the cmd).
Here's the log from Logstash/Filebeat:
[2018-12-17T03:01:58,118][INFO ][org.logstash.beats.BeatsHandler] [local: 0:0:0:0:0:0:0:1:5044, remote: 0:0:0:0:0:0:0:1:58059] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
[2018-12-17T03:01:58,119][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:405) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:372) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext.access$300(AbstractChannelHandlerContext.java:38) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.AbstractChannelHandlerContext$4.run(AbstractChannelHandlerContext.java:236) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 84
at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.1.8.jar:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
... 10 moreI am unsure whats going on here this might help
Thank you for the suggestion, but I don't think it's SSL issue as I don't have it enabled on both sides. (I made sure to check both config files) 
Could you add the Filebeat log as well, please?
Sorry for the very late reply.
I just run it again but the Filebeat didn't create any log files anymore (it used to do tho). But, in the command prompt, it wrote this kind of output:
2019-01-10T15:52:11.893+0700 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":202},"total":{"ticks":857,"value":857},"user":{"ticks":655}},"handles":{"open":159},"info":{"ephemeral_id":"b70ccd0d-a965-4279-95ca-020306ebf7d8","uptime":{"ms":1906959}},"memstats":{"gc_next":4194304,"memory_alloc":2607360,"memory_total":23322816,"rss":8192}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"pipeline":{"clients":6,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}}}}}
Also, actually I want to ask, what's the difference of running Filebeat through command prompt and PowerShell?
When I try it on PowerShell, by running ".\filebeat.exe setup" I got message saying "Exiting: Template loading requested but the Elasticsearch output is not configured/enabled"
(I tried on both and I can't remember which one worked and created Filebeat logs earlier.)
Oh, I just checked again and the log was created.
Here it is. Sorry for the confusion.
2019-01-10T15:51:58.731+0700 INFO instance/beat.go:592 Home path: [D:\filebeat-6.5.2-windows-x86_64] Config path: [D:\filebeat-6.5.2-windows-x86_64] Data path: [D:\filebeat-6.5.2-windows-x86_64\data] Logs path: [D:\filebeat-6.5.2-windows-x86_64\logs]
2019-01-10T15:51:58.865+0700 INFO instance/beat.go:599 Beat UUID: 13720ea1-7cf1-42eb-a033-8db29b20cc48
2019-01-10T15:51:58.865+0700 INFO [beat] instance/beat.go:825 Beat info {"system_info": {"beat": {"path": {"config": "D:\\filebeat-6.5.2-windows-x86_64", "data": "D:\\filebeat-6.5.2-windows-x86_64\\data", "home": "D:\\filebeat-6.5.2-windows-x86_64", "logs": "D:\\filebeat-6.5.2-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "13720ea1-7cf1-42eb-a033-8db29b20cc48"}}}
2019-01-10T15:51:58.865+0700 INFO [beat] instance/beat.go:834 Build info {"system_info": {"build": {"commit": "b48d073b84e874a182c122d8ef2bad867f714a11", "libbeat": "6.5.2", "time": "2018-11-29T23:03:04.000Z", "version": "6.5.2"}}}
2019-01-10T15:51:58.866+0700 INFO [beat] instance/beat.go:837 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.10.3"}}}
2019-01-10T15:51:58.951+0700 INFO [beat] instance/beat.go:841 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-07T16:27:24.19+07:00","name":"Aurelia","ip":["fe80::987d:ecc7:3b6f:d919/64","169.254.217.25/16","fe80::a479:6679:e042:97b8/64","192.168.88.23/24","fe80::e86b:f8bf:bf80:3c84/64","169.254.60.132/16","fe80::e9fe:9397:46a7:4468/64","169.254.68.104/16","::1/128","127.0.0.1/8","fe80::e0:0:0:0/64","fe80::5efe:c0a8:5817/128"],"kernel_version":"6.1.7601.17592 (win7sp1_gdr.110408-1631)","mac":["70:f3:95:b2:67:d7","70:f3:95:b2:67:d7","70:f3:95:d4:83:c4","60:eb:69:6f:89:66","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows 7 Professional","version":"6.1","major":1,"minor":0,"patch":0,"build":"7601.0"},"timezone":"+07","timezone_offset_sec":25200,"id":"df7fe80c-f48d-4f51-acc6-ed55f3eb1897"}}}
2019-01-10T15:51:58.952+0700 INFO [beat] instance/beat.go:870 Process info {"system_info": {"process": {"cwd": "D:\\filebeat-6.5.2-windows-x86_64", "exe": "D:\\filebeat-6.5.2-windows-x86_64\\filebeat.exe", "name": "filebeat.exe", "pid": 6840, "ppid": 1676, "start_time": "2019-01-10T15:51:58.601+0700"}}}
2019-01-10T15:51:58.952+0700 INFO instance/beat.go:278 Setup Beat: filebeat; Version: 6.5.2
2019-01-10T15:52:01.992+0700 INFO add_cloud_metadata/add_cloud_metadata.go:319 add_cloud_metadata: hosting provider type not detected.
2019-01-10T15:52:01.992+0700 INFO [publisher] pipeline/module.go:110 Beat name: Aurelia
2019-01-10T15:52:01.993+0700 ERROR instance/beat.go:800 Exiting: Template loading requested but the Elasticsearch output is not configured/enabledThat's the problem. You have configured the Beat to load templates, which is the default. To do that the Beat needs to talk to Elasticsearch directly (and not through Logstash as you have configured it). And because the loading step is failing, Beats just stops and doesn't transfer any data. It is a little hidden in the docs — see the note in Configure Elasticsearch index template loading | Filebeat Reference [8.11] | Elastic.
To fix that: Disable automatic template loading with this setting in the filebeat.yml setup.template.enabled: false and then load it manually: Load the Elasticsearch index template | Filebeat Reference [8.11] | Elastic.
1 Like
output error.
see the data.
I've followed the instruction. Now the message "Exiting: ..." isn't showing anymore, but there is still no index created in the Elasticsearch. When I run "Start-Service filebeat" nothing showed in the screen.
Here's the latest Filebeat logs.
2019-01-14T13:26:03.906+0700 INFO instance/beat.go:592 Home path: [D:\filebeat-6.5.2-windows-x86_64] Config path: [D:\filebeat-6.5.2-windows-x86_64] Data path: [D:\filebeat-6.5.2-windows-x86_64\data] Logs path: [D:\filebeat-6.5.2-windows-x86_64\logs]
2019-01-14T13:26:04.245+0700 INFO instance/beat.go:599 Beat UUID: 3304ca32-b5a1-4858-9967-47569e9b89a4
2019-01-14T13:26:04.245+0700 INFO [beat] instance/beat.go:825 Beat info {"system_info": {"beat": {"path": {"config": "D:\\filebeat-6.5.2-windows-x86_64", "data": "D:\\filebeat-6.5.2-windows-x86_64\\data", "home": "D:\\filebeat-6.5.2-windows-x86_64", "logs": "D:\\filebeat-6.5.2-windows-x86_64\\logs"}, "type": "filebeat", "uuid": "3304ca32-b5a1-4858-9967-47569e9b89a4"}}}
2019-01-14T13:26:04.381+0700 INFO [beat] instance/beat.go:834 Build info {"system_info": {"build": {"commit": "b48d073b84e874a182c122d8ef2bad867f714a11", "libbeat": "6.5.2", "time": "2018-11-29T23:03:04.000Z", "version": "6.5.2"}}}
2019-01-14T13:26:04.422+0700 INFO [beat] instance/beat.go:837 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.10.3"}}}
2019-01-14T13:26:04.659+0700 INFO [beat] instance/beat.go:841 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-01-13T21:14:36.03+07:00","name":"Aurelia","ip":["fe80::987d:ecc7:3b6f:d919/64","169.254.217.25/16","fe80::a479:6679:e042:97b8/64","10.5.26.4/22","fe80::e86b:f8bf:bf80:3c84/64","169.254.60.132/16","fe80::e9fe:9397:46a7:4468/64","167.205.34.104/24","::1/128","127.0.0.1/8","fe80::e0:0:0:0/64","fe80::5efe:a05:1a04/128","fe80::200:5efe:a7cd:2268/128","2002:a7cd:2268::a7cd:2268/128"],"kernel_version":"6.1.7601.17592 (win7sp1_gdr.110408-1631)","mac":["70:f3:95:b2:67:d7","70:f3:95:b2:67:d7","70:f3:95:d4:83:c4","60:eb:69:6f:89:66","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows 7 Professional","version":"6.1","major":1,"minor":0,"patch":0,"build":"7601.0"},"timezone":"+07","timezone_offset_sec":25200,"id":"df7fe80c-f48d-4f51-acc6-ed55f3eb1897"}}}
2019-01-14T13:26:04.660+0700 INFO [beat] instance/beat.go:870 Process info {"system_info": {"process": {"cwd": "D:\\filebeat-6.5.2-windows-x86_64", "exe": "D:\\filebeat-6.5.2-windows-x86_64\\filebeat.exe", "name": "filebeat.exe", "pid": 2828, "ppid": 2532, "start_time": "2019-01-14T13:25:42.240+0700"}}}
2019-01-14T13:26:04.660+0700 INFO instance/beat.go:278 Setup Beat: filebeat; Version: 6.5.2
2019-01-14T13:26:07.825+0700 INFO add_cloud_metadata/add_cloud_metadata.go:319 add_cloud_metadata: hosting provider type not detected.
2019-01-14T13:26:07.904+0700 INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200
2019-01-14T13:26:07.942+0700 INFO [publisher] pipeline/module.go:110 Beat name: Aurelia
2019-01-14T13:26:07.997+0700 INFO elasticsearch/client.go:163 Elasticsearch url: http://localhost:9200
2019-01-14T13:26:08.457+0700 INFO elasticsearch/client.go:712 Connected to Elasticsearch version 6.5.2
2019-01-14T13:26:08.511+0700 INFO template/load.go:129 Template already exists and will not be overwritten.
Where could I go wrong?
I checked my filebeat.yml again and found mistakes there and fixed it and now it's working fine. Thank you so much.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.