Having an issue with filebeat pushing to Elasticsearch

Hello,

I am receiving the following error when attempting to push log files to Elasticsearch:

Exiting: error loading config file: yaml: line 2: did not find expected key

Here is my .yaml file:

filebeat.inputs:
- type: log
 enabled: true
 paths:
   - "/path/to/log/file/*log"


setup.template.name: "test_application_drop"
setup.template.pattern: "test_*_drop"

output.elasticsearch:
 hosts: ["host.host:9200"]
 index: "test_%{+yyyy.MM.dd}_drop"
 pipeline: "custom_pipeline"
 username: "changeme"
 password: "changeme"
 protocol: "https"

I'm using an existing Elasticsearch ingest pipeline which I'm referencing in the yaml. Any feedback is appreciated.

Looks like your yml is malformed, spaces indents are part of the syntax.
Your enabled: true needs to be indented properly see below.
Also all those field under output.elasticsearch also do not look indented corrected 2 spaces...

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

That worked, but introduced another error. Running the config I'm getting an authentication error despite my username/password being correct. Here's the message I am receiving:

Failed to connect to backoff(elasticsearch(https://localhost:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [user] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [user] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

Any thoughts on how to resolve this authentication issue?

Did you fix the rest of the formatting?
Is your elasticsearch really running on https?
did you enable security authentication etc. setup certs do all that stuff?

Looks like bad creds... that is the error you get with bad creds...

pretty simple try to curl from filebeat host with the same host and creds that are in the elasticsearch.output section filebeat.yml see what you get

First try this is ignores the self signed cert.
curl --insecure -u "username:password" https://elastichost:9200

Hello,

I would have assumed I fixed the rest of the formatting. I made the suggested changes and the authentication error is a new error message from the previous.

I have this cluster up and running already with https enabled. I'm just not using filebeat,. I'm now attempting to ship logs via filebeat and this is where I'm running into issues.

Running your curl command shows the cluster output information with the cluster_name, cluster_uuid, and version information as expected.

That is an auth error. I am not sure if you "sanitized" the log message which is fine so its a little hard to know exactly what the issue is.

If the user [user] or (elasticsearch(https://localhost:9200) don't match the config that will indicate there is a config issue.

You can run this command and it will just try to connect / test the output it will give you a little more detail.

filebeat test output

The only thing I sanitized in the log message is the name of the user. When you say the user doesn't match the config, which config?

the output after running the test is:

filebeat test output
logstash: xxx.xxx.xxx.xxx:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: xxx.xxx.xxx.xxx
    dial up... ERROR dial tcp xxx.xxx.xxx.xxx:5044: connect: connection refused

I think I'm finding something here. In the output for filebeat test output the IP xxx.xxx.xxx.xxx:5044 referenced in the logstash: and dial up.. messages is not the same IP as used in the cluster. The IP belongs to a test instance of Elasticsearch. My filebeat config is pointing to the correct IP space. Where would I make these adjustments?

So your filebeat.yml above has elasticsearch set as an output but the output of the command indicates a logstash output, logstash beats input runs on port 5044.

Logstash is a streaming ETL tools so you need to understand are you trying to send filebeat directly to Elasticsearch or are you trying to send data to logstash and then on to elasticsearch.

i.e. is your architecture
Filebeat -> Elasticsearch
or
Filebeat -> Logstash -> Elasticsearch

If you are not trying to send to logstash, my guess is that you have an uncommented logstash.output section in your filebeat.yml

The easiest what for use to help is to provide your full filebeat.yml exactly how it is sans creds ...