Problems getting started with Filebeat and Elasticsearch?

Forcep · April 27, 2020, 2:28pm

Hi,

Im running the following on a local computer for test :

Elastcisserach 7.5.1
Kibana 7.5.1
Filebeat 7.5.1

Its using HTTPS connection with ignor cert errors. I have followed the guide on how ti install filebeat with the kibana dashboard. Now I can see that the following indeices have red flag : kibana_1 and filebeat

I can also see that these shards is red :

{"status":"red","number_of_shards":1,"number_of_replicas":0,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"filebeat-7.5.1-2020.04.27-000001":{"status":"red","number_of_shards":1,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}}}}

Elastciserach is throwing this exception :

[2020-04-27T16:11:00,081][WARN ][r.suppressed             ] [My-node-1] path: /.kibana/_count, params: {index=.kibana}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:534) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:305) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:563) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:384) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:219) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:284) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

I have already reinstalled the entire environment but that just results in the same exception? It seems like everything works great until filebeat is insatlled and running.

Regards

warkolm · April 27, 2020, 11:02pm

What's the output from the _cat/indices?v and _cat/node?v endpoints?

Forcep · April 28, 2020, 10:01am

I restarted Elasticsearch and Kibana, then I re-installed filebeat :

Load template, had to remove the -e and the outputelastic to get the command running else it just complained about the format.

filebeat.exe setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

This command could not be run, Invoke-RestMethod is not known in PowerShell :

Invoke-RestMethod -Method Delete "http://localhost:9200/filebeat-*"

This command was successful :

filebeat.exe setup --dashboards

This command did not work :

Start-Service filebeat

So I started the filebeat service with TaskManager > Services

No problem to browse Kibana(with login), the discovery grapth was shown. But when trying to click a couple of filebeat modules I got alot of errors, and when I went back to discovery it was stuck in loading loop, refresh did nothing.

I went back to the CMDs/PS and both Elasticsearch and Kibana was complaining about all shards failed again. Restarting Kibana did not help(could not connect all shared failed), Elasticsearch needs to be restarted again and maybe I need to setup the filebeat again.

Also, even if filebeat have been running as a service it does not seems like it is picking up my logfiles? Not sure where I can see what work the filebeat really do?

When trying to browse _cat/node?v I get :

{"error":"Incorrect HTTP method for uri [/_cat/node?v] and method [GET], allowed: [POST]","status":405}

The /localhost:9200/_cat/indices?v gets this :

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .security-7 xssVpomvQYCjyJtzp6ILMQ 1 0
green open .kibana_task_manager_1 OBJL2YGpRSGu3V6JDQXD3g 1 0
yellow open 5.12test_ecacherowpatient iBIKh0NXSZO0KGIfVbueTQ 1 1 0 0 283b 283b
green open .apm-agent-configuration 4ZptB0agRAq6Nv83UsZ1OA 1 0
red open .kibana_1 dEiI9h1SQauLEiKGFZB4Vg 1 0
red open filebeat-7.5.1-2020.04.28-000001 _tbLDOUqRuysMXuzI5zecA 1 1

Note : Im using a certificate but it is not fully valid so I hade to set allow all on some of these components.

Right now there is many lose parts that do not really work, hope we can solve it.

warkolm · April 28, 2020, 10:18am

It's a little hard to follow what's happening here sorry.

If shards are failing in Elasticsearch and your cluster is red, there is a problem. To resolve that we'd need to see more of your Elasticsearch logs please.

Forcep · April 28, 2020, 12:18pm

I have started Elastic from cmd but all I can see at this point is :

[2020-04-28T14:12:33,072][DEBUG][o.e.a.s.TransportSearchAction] [My-node-1] All shards failed for phase: [query]
[2020-04-28T14:12:37,019][WARN ][r.suppressed             ] [My-node-1] path: /.kibana/_count, params: {index=.kibana}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:534) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:305) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:563) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:384) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:219) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:284) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

I have checked the ElasticSearch\log folder but there is only vary old logfiles.

How do I get what you need?

Christian_Dahlqvist · April 28, 2020, 12:29pm

Have you edited any settings in you Elasticsearch.yml file?

Forcep · April 28, 2020, 2:04pm

The elasticsearch.yml looks like this :

cluster.name: My-cluster
cluster.initial_master_nodes: ["My-node-1"]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 1gb
cluster.routing.allocation.disk.watermark.low: 3gb
cluster.routing.allocation.disk.watermark.high: 2gb

node.name: My-node-1

http.port: 9200
transport.tcp.port: 9300
network.host: 0.0.0.0

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none 
xpack.security.transport.ssl.keystore.path: certs/cert.pfx
xpack.security.transport.ssl.truststore.path: certs/cert.pfx

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: none 
xpack.security.http.ssl.keystore.path: certs/cert.pfx
xpack.security.http.ssl.truststore.path: certs/cert.pfx

It is already in use by another part of my software which works fine. Its however on a diffrent index.

Forcep · April 29, 2020, 12:34pm

Is there anymore information you need?

Forcep · May 4, 2020, 8:13am

So, any thoughts on why this is not working?

system · June 1, 2020, 8:13am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat not sending data to Elasticsearch, index_not_found_exception Beats filebeat	2	771	March 23, 2022
Filebeat PKI certificate authentication not working to es Elasticsearch elastic-stack-security	2	417	August 8, 2022
Filebeat not able to connect to kibana hosted in another server Beats filebeat	8	41	December 3, 2024
Filebeat does not connect to elastcisearch Elastic Agent filebeat	1	151	April 30, 2024
Problem with filebeat Beats	21	828	December 14, 2023

Problems getting started with Filebeat and Elasticsearch?

Related topics