Data loss when sending logs to Kibana through Filebeat

Elastic Stack Beats
1

Hi, I am new to elastic stalk and currently I am facing some difficulties to read all the logs in JSON format.

{"code":"28000","file":"auth.c","length":164,"level":"error","line":"496","message":"no entry for host","name":"error","timestamp":"2022-10-05 08:40:14.308"}

All the lines started with "code":"28000" is not able to send to Kibana. Does anyone know what is the issue?

{"code":"SELF_SIGNED_CERT_IN_CHAIN","level":"error","message":"self signed certificate","timestamp":"2022-10-05 08:40:39.908"}

However, the above line is able to be sent to Kibana.

Thank you for helping

2

Welcome to our community! :smiley:

Are those what you are trying to send to Elasticsearch?
Is there an error when you are sending these you can share?

3

No, there is no error message but when I check the number of hits in data view, it does not vary with the number of lines I have in a log

4

You will need to share more information then please. Things like configs, logs, how are you comparing the number of logs to what is in Elasticsearch.

5

{"code":"SELF_SIGNED_CERT_IN_CHAIN","level":"error","message":"self signed certificate","timestamp":"2022-10-05 08:40:39.908"}

Most likely you have selfsigned certs. Try with this if you are sending directly to Elasticsearch:

output.elasticsearch.verification_mode: "none"

Default mode is full. Documentation is here

If is working then set certificate_authorities on both side and set full

6

This does not seem to be an error from filebeat, but the log message the OP is trying to send.

7

Please share your filebeat.yml file and also check the mapping of your index, it may be a mapping issue.

In this message:

{"code":"28000","file":"auth.c","length":164,"level":"error","line":"496","message":"no entry for host","name":"error","timestamp":"2022-10-05 08:40:14.308"}

You have some extra fields, maybe one of them have a different mapping.

8

Might be. Well until we see filebeat.yml and logs, it's hard to know.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.