Filebeat Connection to Elastic Search Error

Elastic Stack Kibana
1

HI, I tried installing filebeat on win10.
used ' .\filebeat.exe -e test config' the output says Config OK.
but while testing output of filebeat using '.\filebeat.exe -e test output' it dsiplays the following error. please help in resolvinf this issue.

Windows PowerShell
Copyright (C) 2016 Microsoft Corporation. All rights reserved.

PS C:\WINDOWS\system32> cd 'C:\Program Files (x86)\ELK\filebeat'
PS C:\Program Files (x86)\ELK\filebeat> .\filebeat.exe -e test output
2021-10-02T15:37:11.570+0530    INFO    instance/beat.go:665    Home path: [C:\Program Files (x86)\ELK\filebeat] Config
path: [C:\Program Files (x86)\ELK\filebeat] Data path: [C:\Program Files (x86)\ELK\filebeat\data] Logs path: [C:\Program
 Files (x86)\ELK\filebeat\logs]
2021-10-02T15:37:11.571+0530    INFO    instance/beat.go:673    Beat ID: 96b66a2f-533c-4b43-b85f-1568a98aeb00
2021-10-02T15:37:11.756+0530    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index t
o 'filebeat-7.15.0' as ILM is enabled.
2021-10-02T15:37:11.758+0530    INFO    [esclientleg]   eslegclient/connection.go:100   elasticsearch url: http://localh
ost:9200
elasticsearch: http://localhost:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: ::1, 127.0.0.1
    dial up... OK
  TLS... WARN secure connection disabled
2021-10-02T15:37:11.772+0530    INFO    [esclientleg]   eslegclient/connection.go:273   Attempting to connect to Elastic
search version 5.5.0
  talk to server... ERROR Connection marked as failed because the onConnect callback failed: could not connect to a comp
atible version of Elasticsearch: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":
"No endpoint or operation is available at [_license]"}],"type":"illegal_argument_exception","reason":"No endpoint or ope
ration is available at [_license]"},"status":400}
PS C:\Program Files (x86)\ELK\filebeat>
2

Welcome @siddhesh_jadhav

What is the configuration for the Elasticsearch Output in the Filebeat.yml ?

Are Elasticsearch and Filebeat on the same machine, if not, is Elasticsearch reachable from the filebeat machine.

3

The output in filebeatyml is
hosts: ["localhost:9200"]

4

later I tried to connect Elastic cloud with settings :

cloud.id: "My_deployment:dXMtY2VudHJhbDEuZ2NwLmNsb3VkLmVzLmlvJDFmYmY1Zjc2YThiMzQwOWZiMTZhNWUwM2QwYzJhYzkzJDg5NDZmOWIwNzE2NTRiNTg5Y2FhZThkOTQwY2MxNDQ5"

cloud.auth: "filebeat_setup:$iddheshisFromj@dhavnagar"

I got this error

2021-10-02T16:55:35.306+0530    INFO    [monitoring]    log/log.go:193  Uptime: 1.5140761s
2021-10-02T16:55:35.309+0530    INFO    [monitoring]    log/log.go:160  Stopping metrics logging.
2021-10-02T16:55:35.317+0530    INFO    instance/beat.go:479    filebeat stopped.
2021-10-02T16:55:35.319+0530    ERROR   instance/beat.go:989    Exiting: Failed to start crawler: creating module reloader failed: error checking input configuration: No paths were defined for
 input accessing config
Exiting: Failed to start crawler: creating module reloader failed: error checking input configuration: No paths were defined for input accessing config
5

Can you share your filebeat.yml config ? Have you configured any Inputs or Modules?

Are filebeat and elasticsearch on the same machine ? Also in your elasticsearch.yml, what do you have for the network host.

network.host:

6 

# ============================== Filebeat inputs ===============================

filebeat.input:

- type: log

 
  enabled: true

 
  paths:
- C:\Program Files (x86)\ELK\filebeat\logs\*.log
   
 setup.template.settings:
  index.number_of_shards: 2
 
# =============================== Elastic Cloud ================================

cloud.id: "My_deployment:dXMtY2VudHJhbDEuZ2NwLmNsb3VkLmVzLmlvJDFmYmY1Zjc2YThiMzQwOWZiMTZhNWUwM2QwYzJhYzkzJDg5NDZmOWIwNzE2NTRiNTg5Y2FhZThkOTQwY2MxNDQ5"

cloud.auth: "siddhesh:$iddheshisFromj@dhavnagar"

# ---------------------------- Elasticsearch Output ----------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["localhost:9200"]
# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~
7

No, I don't any idea about Elasticsearch.yml file . where I can find it and What configurations are needed.

8

First, where is your Elasticsearch running?

It is running in the same machine as filebeat or you are using elastic cloud? You can't have both outputs.

The filebeat.yml that you shared has 2 outputs enabled, the one for elastic cloud with cloud.id and cloud.auth and the one with the output.elasticsearch.

The first post you shared shows a bad request, that could mean that you have an Elasticsearch running on 127.0.0.1 on port 9200 but your request was wrong, in this case the error was with the _license endpoint. Are you running the OSS version or other fork in this machine?

You need to provide more information about where your Elasticsearch is running and which version it is.

Second, you shared your cloud.id and cloud.auth, do not share sensitive information on public forums, you should change your elastic cloud password now.

9

I'm running OSS version. .msi file for windows its 5.5.0 version. I have disabled output for local host. could you guide me to send logs from filebeat to elastic cloud in windows.
And Yes I have changed password before sending you file.

10

To send data to the elastic cloud using filebeat you just need to configure the cloud.auth and cloud.id setting, is describes in the documentation.

In your first post you were sending data to a local Elasticsearch using filebeat version 7.15, since version 7.13, I think, filebeat can not output data to OSS versions of Elasticsearch as it will check the _license endpoint, also, Elasticsearch 5 is too old, I do not think filebeat 7.X is even compatible with this version.

If you want to send data to this 5.X local instance you need to use Filebeat version 5.X

11

ok I will follow the documentation and try to send data to cloud. I will get back if I have any problem regarding configuration.
thanks for the support.
and one more thing do we need to install Elasticsearch.exe install on our host to send data to cloud?

12

No, you do not a local Elasticsearch instance if you want to send data to elastic cloud, just filebeat is enough.

13

That is malformed not indented correctly and windows path need a single quote to interpreted correctly

  paths:
    - 'C:\Program Files (x86)\ELK\filebeat\logs\*.log'

There are some tips here

14

ok I will let you know once I do the config.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.