Having an issue with filebeat pushing to Elasticsearch

Elastic Stack Beats
1

Hello,

I am receiving the following error when attempting to push log files to Elasticsearch:

Exiting: error loading config file: yaml: line 2: did not find expected key

Here is my .yaml file:

filebeat.inputs:
- type: log
 enabled: true
 paths:
   - "/path/to/log/file/*log"


setup.template.name: "test_application_drop"
setup.template.pattern: "test_*_drop"

output.elasticsearch:
 hosts: ["host.host:9200"]
 index: "test_%{+yyyy.MM.dd}_drop"
 pipeline: "custom_pipeline"
 username: "changeme"
 password: "changeme"
 protocol: "https"

I'm using an existing Elasticsearch ingest pipeline which I'm referencing in the yaml. Any feedback is appreciated.

2

Looks like your yml is malformed, spaces indents are part of the syntax.
Your enabled: true needs to be indented properly see below.
Also all those field under output.elasticsearch also do not look indented corrected 2 spaces...

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
3

That worked, but introduced another error. Running the config I'm getting an authentication error despite my username/password being correct. Here's the message I am receiving:

Failed to connect to backoff(elasticsearch(https://localhost:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [user] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}}],"type":"security_exception","reason":"unable to authenticate user [user] for REST request [/]","header":{"WWW-Authenticate":["Basic realm=\"security\" charset=\"UTF-8\"","Bearer realm=\"security\"","ApiKey"]}},"status":401}

Any thoughts on how to resolve this authentication issue?

4

Did you fix the rest of the formatting?
Is your elasticsearch really running on https?
did you enable security authentication etc. setup certs do all that stuff?

Looks like bad creds... that is the error you get with bad creds...

pretty simple try to curl from filebeat host with the same host and creds that are in the elasticsearch.output section filebeat.yml see what you get

First try this is ignores the self signed cert.
curl --insecure -u "username:password" https://elastichost:9200

5

Hello,

I would have assumed I fixed the rest of the formatting. I made the suggested changes and the authentication error is a new error message from the previous.

I have this cluster up and running already with https enabled. I'm just not using filebeat,. I'm now attempting to ship logs via filebeat and this is where I'm running into issues.

Running your curl command shows the cluster output information with the cluster_name, cluster_uuid, and version information as expected.

6

That is an auth error. I am not sure if you "sanitized" the log message which is fine so its a little hard to know exactly what the issue is.

If the user [user] or (elasticsearch(https://localhost:9200) don't match the config that will indicate there is a config issue.

You can run this command and it will just try to connect / test the output it will give you a little more detail.

filebeat test output

7

The only thing I sanitized in the log message is the name of the user. When you say the user doesn't match the config, which config?

the output after running the test is:

filebeat test output
logstash: xxx.xxx.xxx.xxx:5044...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: xxx.xxx.xxx.xxx
    dial up... ERROR dial tcp xxx.xxx.xxx.xxx:5044: connect: connection refused

I think I'm finding something here. In the output for filebeat test output the IP xxx.xxx.xxx.xxx:5044 referenced in the logstash: and dial up.. messages is not the same IP as used in the cluster. The IP belongs to a test instance of Elasticsearch. My filebeat config is pointing to the correct IP space. Where would I make these adjustments?

8

So your filebeat.yml above has elasticsearch set as an output but the output of the command indicates a logstash output, logstash beats input runs on port 5044.

Logstash is a streaming ETL tools so you need to understand are you trying to send filebeat directly to Elasticsearch or are you trying to send data to logstash and then on to elasticsearch.

i.e. is your architecture
Filebeat -> Elasticsearch
or
Filebeat -> Logstash -> Elasticsearch

If you are not trying to send to logstash, my guess is that you have an uncommented logstash.output section in your filebeat.yml

The easiest what for use to help is to provide your full filebeat.yml exactly how it is sans creds ...

9

Hello,

This was one of those situations where a "turn it off and turn it back on" again situations actually resolved the problem. Not sure what was going on with ssl but restarting the cluster resolved the issue connecting, but now I'm getting a further issue actually getting data into elasticsearch.

I'm going filebeat --> elasticsearch. I have my ingest pipeline created and index patterns created. My entire workflow is already working sans the filebeat. That is I have a working ingest pipeline, index tempalte, viz and dashboards so its working. The only thing I'm attempting to do is use filebeat as the log shipper. I'm getting a weird object error:

{"type":"mapper_parsing_exception","reason":"object mapping for [entries] tried to parse field [entries] as object, but found a concrete value"}

In my ingest pipeline "entries" is a scripted field. Its an integer field that just adds 1 to 'entries' for a length of stay greater than 5. My index template shows "entries" mapped to integer as well. In the index pattern, entries is set to integer, and in discover its set to integer. Any thoughts on how I would resolve this?

10

I would suggest reading this thread where I explained that error

11

I'm not following this, too be honest. My data type is integer. From the Grok parser, to the index template, index pattern, etc., I only have a single field, "entries" as a numeric type. Entries is not stored as an object anywhere.

12

yeah it is a bit confusing but I can tell you for sure if you are getting that error you are trying to push and concrete type such as an integer into a mapping that is an object. That is the error and that is what it comes from. As you can see we can reproduce the exact error message.

Also is this happening on every log line or just a few... Rogue Log Lines can cause that error..

If you would like some help... Show me all of it... its there...

  • The raw logs
  • The Ingest pipeline
  • The Index Template
  • The Actual Mapping of the Index after its created (Not the template) <!- Probably where the issue is...
    GET /my-index
13

I'm working on getting the data to you, trying to obfuscate information so I don't give all our secret sauce to the public (per bosses). In the meantime, I have a question as to how to test this on a new index.

I created a brand new ingest pipeline, index template with mappings and then tried to push filebeat to the new pipelines/index. Still the same issue with concrete values. Same exact error. If a document has not been ingested yet, how can there be a mapping conflict? I get the error trying to ingest this into this new space. What would cause a mapping conflict when no data exists at all?

14

The error happens at ingest time, AFTER the pipeline... that is often confusing...

The error happens as the document is trying to be written and it is in a format that conflicts with the defined mapping from your template.

What you can try is to try this.

Do everything EXCEPT for using the template (i.e. name the index something else that wont apply the template / mapping) and see how the document gets written, then you can see the default mapping that is created that should give you a clue.

Another Approach

You Can Also use pipeline simulate...

Then capture the resulting document

Then try to PUT / POST that document into the the index using a name that will apply the template / mapping.

15

I've created an index "test_object_ingest" which does not match any of the templates in the my filebeat configuration. I still get the exact same error as before. I'm trying the simulate api but not sure what goes under the _source section:

POST /_ingest/pipeline/my-pipeline-id/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "bar"
      }
    },
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "foo": "rab"
      }
    }
  ]
}

Index and id are clearly the index and document id but I'm not sure what I should put under source foo/bar foo/rab. I thought I could just add in the document id and index info similar to the simulate pipeline when using ingest pipelines via the kibana ui. When testing this in the Kibana UI with the simulate, I receive no error messages but I'm not seeing anywhere in the test pipeline output that tells me the type.

16

I am confused....
From here on out please show the command and the result... we are trying to find a subtle issue... when I only see half the equation I can't help.

The result of above (which I don't know what it was) will just show what the doc looks like right before it is ingested against the mapping... did it execute the pipeline? Did it create the entries field?

You could have taken that result and POST ed it to your index(s) one with your mapping one without and observed the difference.

Can you share the pipeline or just a slimmed down version with the entries script processor? Can you share the mapping for the entries fields (compete if it is a sub object or nested etc..)

OR you can try this all in one step

Use an index name that does not match the template index convention and you need to put in a doc that will trigger the entries logic of the pipeline.

DELETE /my-index

POST /my-index/_doc?pipeline=my-pipeline-id
{
  "foo": "rab" 
}

GET /my-index

Please run the 3 commands and show the results.

Then you can repeat these steps with an index name that matches your template I suspect the first set will work and the seconds will not...

17

Hello,

That is where I am confused. What goes in the foo/bar foo/rab placeholders. In my case since I'm trying to create a numeric value would "foo" be replaced by "entries" then what value do I put in "rab"?

18

Hi I would go back to this... what I asked for originally... otherwise I can not really help...
Please provide a few sample lines of the logs, the ingest pipeline and the index template. Then run the last command on your index and include that.

19

Hello,

I am unable to provide raw logs or the ingest pipeline. You are unable to tell me how to format this portion:

POST /my-index/_doc?pipeline=my-pipeline-id
{
"foo": "rab"
}

foo: rab is a placeholder. Where I believe foo is replaced by the field in question, right? So in my case "entries." If that is true, then what should go in "rab?" Just any arbitrary integer since I'm trying to determine why it says "entries" is an object and not integer?

20

Yes substitute your field name and an example value that you expect to be in the document example

POST /my-index/_doc?pipeline=my-pipeline-id
{
    "entries": "123"
}

Also there's a difference between the _source document and how field is stored.

Could you perhaps just provide the mapping (not the template) pretty much can't help much more without some concrete data. The error you are seeing has been discussed many times on this forum in short the Data Your Logs + Pipeline is Providing and The Mapping do not match / are in conflict.

GET /my-index/

There are several ways this can occur. It could be that many DO match and only a few do not... Or the very first log could be defining the mapping 1 way as an object and then the rest of the logs fail because they are a concrete integer. Do you logs have a Header? Again I have no clue with out some concrete data.

Typically logs can be sanitized however that is your choice but it does limit my ability to understand and answer questions. I'll take another look or perhaps someone else would be suited better to answer this question.