# The Junk log directory created in logstash is invalid **URL:** \ **Category:** Logstash\ **Created:** [July 26, 2018, 5:02pm UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821 "2018-07-26T17:02:17Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![swethamahesh](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@swethamahesh](https://discuss.elastic.co/u/swethamahesh)\ **Post date:** [July 26, 2018, 5:02pm UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821/1 "2018-07-26T17:02:17Z")
Hi, My filebeat configuration is as below: ```auto filebeat.prospectors: - input_type: log paths: - /var/log/oes.log encoding: utf-8 fields_under_root: true document_type: log fields: service_name: ops app_name: oes ======================General======= registry_file: /var/lib/filebeat/registry fields_under_root: true fields: vnf_id: instance_id: xyz component_name: yz ``` The service\_name, app\_name, vnf\_id, instance\_id and component\_name is used in logstash configuration for creating the directory. ................................................................................................................................. Logstash.conf file is as below: ```auto output { if [service_name] == "ops" and [component_name] == "yz" { file{ path => "/var/log/store/%{vnf_id}/%{instance_id}/%{service_name}/%{app_name}_%{+yyyy-MM-dd-HH}.log.gz" codec => line { format => "%{message}, %{vnf_id}, %{instance_id}, %{component_name}, %{app_name}" } gzip => true } } ``` * * * There are two log directory created at the /var/log/store/ path: #cd /var/log/store/ #ls -l %{vnf\_id}/ ABC/ -- this value i.e.. vnf\_id is filled after some time the filebeat started and filebeat is restarted. Can you please help why the junk %{vnf\_id} is getting created. Thanks In Advance, swetha. M ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [July 26, 2018, 10:44pm UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821/2 "2018-07-26T22:44:23Z")
It is part of your path string specified in the file output plugin: ```auto output { if [service_name] == "ops" and [component_name] == "yz" { file{ path => "/var/log/store/%{vnf_id}/%{instance_id}/%{service_name}/%{app_name}_%{+yyyy-MM-dd-HH}.log.gz" codec => line { format => "%{message}, %{vnf_id}, %{instance_id}, %{component_name}, %{app_name}" } gzip => true } } ``` When performing [string interpolation](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html), if the field does not exist on the event, the reference is not expanded. ---
**Author:** ![swethamahesh](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@swethamahesh](https://discuss.elastic.co/u/swethamahesh)\ **Post date:** [July 27, 2018, 4:22am UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821/3 "2018-07-27T04:22:04Z")
Let me explain what are we doing as part of process. -\> during the bootup of the system the filebeat.yml is copied with empty data of vnf\_id parameter and other dummy parameters in the filebeat.yml file. -\> Disabled and masked the filebeat service at bootup untill configuration is completed. -\> once the system is up we have written script which takes input for vnf\_id, instance\_id, component name and fill the filebeat.yml and restart and unmask the service. -\> during this process logstash is still active and waiting for input from filebeat. Is this parameter is not properly sent from filebeat hence the logstash is writing the junk directory. if yes, how do we handle this issue as some of the logs are going to junk directory and rest to actual. Its being difficult to integrate the logs. Thanks In Advance, Swetha. ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [July 30, 2018, 11:13pm UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821/4 "2018-07-30T23:13:18Z")
When `vnf_id` is not set on an event that Logstash processes, where do you want the logs to go? Similar is true for the other attributes that are part of the path you're specifying (e.g., `instance_id`, `service_name`, `app_name`, etc.)? ---
**Author:** ![swethamahesh](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@swethamahesh](https://discuss.elastic.co/u/swethamahesh)\ **Post date:** [July 31, 2018, 4:49am UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821/5 "2018-07-31T04:49:04Z")
Thanks, So, its the filebeat who is sending the event without vnf\_id filled. Can you please suggest how can we overcome this issue. As i conveyed before the filebeat is up with some default configuration as vnf\_id as empty and logstash server ip is not yet configured. Later point while system boot up we configure the proper logstash ip in filebeat and vnf\_id is filled with value example "ONEDS-1" and restart the filebeat service. Thanks In Advance, Swetha. M ---
**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\ **Post date:** [August 1, 2018, 5:45pm UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821/6 "2018-08-01T17:45:09Z")
If you're looking to figure out how to handle events with _missing_ fields in Logstash (perhaps by specifying a default for the field if it is missing using the [Mutate Filter Plugin's `coerce` directive](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-coerce)), you would add something like this to your filters for each field: ```auto filter { mutate { coerce => { "[vnf_id]" => "UNDEFINED" "[instance_id]" => "UNDEFINED" "[service_name]" => "UNDEFINED" "[app_name]" => "UNDEFINED" } } } ``` * * * If, however, you're looking to _prevent_ Filebeat from sending events until it has been configured, that question may be out of scope for the Logstash forums and would better be addressed in the [Filebeat forum](https://discuss.elastic.co/c/beats/filebeat). In principal, I would suggest looking at your service manager configuration for filebeat (e.g. upstart, initd) and change the conditions on which it starts to _not_ start on system startup, but instead to have your configuration helper start the service after it has been fully configured. ---
**Author:** ![swethamahesh](https://avatars.discourse-cdn.com/v4/letter/s/8c91f0/32.png) [@swethamahesh](https://discuss.elastic.co/u/swethamahesh)\ **Post date:** [August 9, 2018, 5:33am UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821/7 "2018-08-09T05:33:51Z")
Thanks a lot for the information. I would check in filebeat forum why filebeat is sending the empty fields resulting in creating junk directory. Thanks, Swetha. M ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 6, 2018, 5:33am UTC](https://discuss.elastic.co/t/the-junk-log-directory-created-in-logstash-is-invalid/141821/8 "2018-09-06T05:33:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.