Filebeat is sending the empty value to logstash for the paramters configured in filebeat

Elastic Stack Beats
1

Hi,

Further to my discussion in the logstash forum regarding the junk directory getting created by logstash.

It is conveyed that filebeat is sending the data empty, Hence logstash is writing the directory with junk name.

FYI: The Junk log directory created in logstash is invalid

Can you please let us know why the paramters are sent empty from filebeat to logstash.

Thanks In Advance,
Swetha. M

2

Please provide your configuration and a debug log of filebeat so we can diagnose the issue.

Run filebeat with -d '*' and paste the log and configuration here enclosed in triple backticks:

```
like this
```

3

Hi Adrisr,

please find the configuration below:

filebeat.prospectors:

    input_type: log
    paths:
        /var/log/oes.log
        encoding: utf-8
        fields_under_root: true
        document_type: log
        fields:
        service_name: ops
        app_name: oes

======================General=======
registry_file: /var/lib/filebeat/registry
fields_under_root: true
fields:
vnf_id:
instance_id: xyz
component_name: yz

The service_name, app_name, vnf_id, instance_id and component_name is used in logstash configuration for creating the directory.
.................................................................................................................................

Logstash.conf file is as below:

output {
if [service_name] == "ops" and [component_name] == "yz" {
file{
path => "/var/log/store/%{vnf_id}/%{instance_id}/%{service_name}/%{app_name}_%{+yyyy-MM-dd-HH}.log.gz"
codec => line { format => "%{message}, %{vnf_id}, %{instance_id}, %{component_name}, %{app_name}" }
gzip => true
}
}

should i have to run filebeat -d '*' and reproduce the issue to collect logs?

Thanks,
Swetha. M

4

To understand your configuration I would need that you pasted it respecting the original indentation.

Nonetheless, this seems wrong:

fields:
vnf_id:
instance_id: xyz
component_name: yz

vnf_id has no value, thus is not sent to Logstash.

5

Hi Adrisr,

We keep the vnfd_id value empty parameter in filebeat.yml during bootup where filebeat service is stopped. Later, we configure the filebeat.yml with values for vnf_id, instance_id and component_name parameters with appropriate values and restart the filebeat.

In this case we see the folder at logstash is created as %{vnf_id}/ which is wrong

We have also tried keep some default value such as vnf_id: LSDL-1 and we configure filebeat.yml with proper value such as vnf_id:LSDL-1 to vnf_id:SDL then restart filebeat service.

then we see the folder created with default value as LSDL-1/ even this is wrong.

Basically if the parameter vnf_id is empty or filled with any default value, we see the directory is created with the same.

I request you to please help me how to over some this problem.

Thanks,
Swetha. M

6

Can you run filebeat with the -d publish argument and share the logs with us?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.