Problems getting started with Filebeat and Elasticsearch?

Hi,

Im running the following on a local computer for test :

  • Elastcisserach 7.5.1
  • Kibana 7.5.1
  • Filebeat 7.5.1

Its using HTTPS connection with ignor cert errors. I have followed the guide on how ti install filebeat with the kibana dashboard. Now I can see that the following indeices have red flag : kibana_1 and filebeat

I can also see that these shards is red :

{"status":"red","number_of_shards":1,"number_of_replicas":0,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":1}}},"filebeat-7.5.1-2020.04.27-000001":{"status":"red","number_of_shards":1,"number_of_replicas":1,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2,"shards":{"0":{"status":"red","primary_active":false,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":2}}}}}

Elastciserach is throwing this exception :

[2020-04-27T16:11:00,081][WARN ][r.suppressed             ] [My-node-1] path: /.kibana/_count, params: {index=.kibana}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:534) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:305) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:563) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:384) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:219) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:284) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

I have already reinstalled the entire environment but that just results in the same exception? It seems like everything works great until filebeat is insatlled and running.

Regards

What's the output from the _cat/indices?v and _cat/node?v endpoints?

I restarted Elasticsearch and Kibana, then I re-installed filebeat :

Load template, had to remove the -e and the outputelastic to get the command running else it just complained about the format.

filebeat.exe setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'

This command could not be run, Invoke-RestMethod is not known in PowerShell :

Invoke-RestMethod -Method Delete "http://localhost:9200/filebeat-*"

This command was successful :

filebeat.exe setup --dashboards

This command did not work :

Start-Service filebeat

So I started the filebeat service with TaskManager > Services

No problem to browse Kibana(with login), the discovery grapth was shown. But when trying to click a couple of filebeat modules I got alot of errors, and when I went back to discovery it was stuck in loading loop, refresh did nothing.

I went back to the CMDs/PS and both Elasticsearch and Kibana was complaining about all shards failed again. Restarting Kibana did not help(could not connect all shared failed), Elasticsearch needs to be restarted again and maybe I need to setup the filebeat again.

Also, even if filebeat have been running as a service it does not seems like it is picking up my logfiles? Not sure where I can see what work the filebeat really do?

When trying to browse _cat/node?v I get :

{"error":"Incorrect HTTP method for uri [/_cat/node?v] and method [GET], allowed: [POST]","status":405}

The /localhost:9200/_cat/indices?v gets this :

health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .security-7 xssVpomvQYCjyJtzp6ILMQ 1 0
green open .kibana_task_manager_1 OBJL2YGpRSGu3V6JDQXD3g 1 0
yellow open 5.12test_ecacherowpatient iBIKh0NXSZO0KGIfVbueTQ 1 1 0 0 283b 283b
green open .apm-agent-configuration 4ZptB0agRAq6Nv83UsZ1OA 1 0
red open .kibana_1 dEiI9h1SQauLEiKGFZB4Vg 1 0
red open filebeat-7.5.1-2020.04.28-000001 _tbLDOUqRuysMXuzI5zecA 1 1

Note : Im using a certificate but it is not fully valid so I hade to set allow all on some of these components.

Right now there is many lose parts that do not really work, hope we can solve it.

It's a little hard to follow what's happening here sorry.

If shards are failing in Elasticsearch and your cluster is red, there is a problem. To resolve that we'd need to see more of your Elasticsearch logs please.

I have started Elastic from cmd but all I can see at this point is :

[2020-04-28T14:12:33,072][DEBUG][o.e.a.s.TransportSearchAction] [My-node-1] All shards failed for phase: [query]
[2020-04-28T14:12:37,019][WARN ][r.suppressed             ] [My-node-1] path: /.kibana/_count, params: {index=.kibana}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:534) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:305) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:563) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.onShardFailure(AbstractSearchAsyncAction.java:384) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction.lambda$performPhaseOnShard$0(AbstractSearchAsyncAction.java:219) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.action.search.AbstractSearchAsyncAction$2.doRun(AbstractSearchAsyncAction.java:284) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.5.1.jar:7.5.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.5.1.jar:7.5.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

I have checked the ElasticSearch\log folder but there is only vary old logfiles.

How do I get what you need?

Have you edited any settings in you Elasticsearch.yml file?

The elasticsearch.yml looks like this :

cluster.name: My-cluster
cluster.initial_master_nodes: ["My-node-1"]
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 1gb
cluster.routing.allocation.disk.watermark.low: 3gb
cluster.routing.allocation.disk.watermark.high: 2gb

node.name: My-node-1

http.port: 9200
transport.tcp.port: 9300
network.host: 0.0.0.0

xpack.security.enabled: true

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none 
xpack.security.transport.ssl.keystore.path: certs/cert.pfx
xpack.security.transport.ssl.truststore.path: certs/cert.pfx

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: none 
xpack.security.http.ssl.keystore.path: certs/cert.pfx
xpack.security.http.ssl.truststore.path: certs/cert.pfx

It is already in use by another part of my software which works fine. Its however on a diffrent index.

Is there anymore information you need?

So, any thoughts on why this is not working?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.