Filebeat processors not working

Elastic Stack Beats
1

Hi ,

I am new to Elastic stack. I am using filebeat to send logs to ES and using processors in filebeat. When I saw index from Kibana, I noticed that the convert processor is not being applied. Sometimes they work as expected but sometimes it doesn't...

My Filebeat config:

path.data: /central-data/jfrog/filebeat-8.16.3-linux-x86_64/data
filebeat.inputs:
  - type: filestream
    enabled: true
    paths:
      - /central-data/jfrog/artifactory/var/log/artifactory-request-out.log
    processors:
      - dissect:
          tokenizer: "%{timestamp}|%{req_trace_id}|%{remote_repo_name}|%{req_user}|%{req_type}|%{remote_url}|%{res_status}|%{req_content_length}|%{res_content_length}|%{duration}"
          target_prefix: ""
      - add_fields:
          target: ''
          fields:
            req_user: "anonymous"
          when:
            equals:
              req_user: ""
      - convert:
          fields:
            - {from: "duration", to: "duration", type: "long"}
            - {from: "res_status", to: "res_status", type: "long"}
            - {from: "req_content_length", to: "req_content_length", type: "long"}
            - {from: "res_content_length", to: "res_content_length", type: "long"}
          ignore_missing: true
          fail_on_error: false
    fields:
      record_type: "jfrt_remote_repo"
    fields_under_root: true

  # Artifactory Request
  - type: filestream
    id: artifactory_request
    enabled: true
    paths:
      - /central-data/jfrog/artifactory/var/log/artifactory-request.log
    processors:
      - dissect:
          tokenizer: "%{timestamp}|%{req_trace_id}|%{req_remote_address}|%{req_user}|%{req_type}|%{req_url}|%{res_status}|%{req_content_length}|%{res_content_length}|%{res_duration}|%{req_user_agent}"
          target_prefix: ""
#      - script:
#          lang: javascript
#          id: extract_domain
#          source: >
#            function process(event) {
#              var req_user = event.Get("req_user");
#              if (req_user && typeof req_user === 'string') {
#                if (req_user.includes("@")) {
#                   var parts = req_user.split("@");
#                   var username = parts[0];
#                   var domain = parts[1];
#                   event.Put("req_user", username);
#                   event.Put("domain", domain);
#                }
#              }
#            }
      - convert:
          fields:
#            - {from: "res_duration", to: "res_duration", type: "long"}
#            - {from: "res_status", to: "res_status", type: "long"}
#            - {from: "req_content_length", to: "req_content_length", type: "long"}
#            - {from: "res_content_length", to: "res_content_length", type: "long"}
            - {from: "req_remote_address", to: "req_remote_address", type: "ip"}
          ignore_missing: true
          fail_on_error: false
      - if:
          or:
            - contains.req_remote_address: "127.0.0.1"
            - contains.req_user_agent: "JFrog-Router"
        then:
          - add_fields:
              target: ''
              fields:
                req_call_type: "internal"
        else:
          - add_fields:
              target: ''
              fields:
                req_call_type: "external"
    fields:
      record_type: "jfrt"
    fields_under_root: true

  # Metadata Request
  - type: filestream
    id: metadata_request
    enabled: true
    paths:
      - /central-data/jfrog/artifactory/var/log/metadata-request.log
    processors:
      - dissect:
          tokenizer: "%{timestamp}|%{req_trace_id}|%{req_remote_address}|%{req_user}|%{req_type}|%{req_url}|%{res_status}|%{req_content_length}|%{res_content_length}|%{res_duration}|%{req_user_agent}"
          target_prefix: ""
      - if:
          or:
            - contains.req_remote_address: "127.0.0.1"
            - contains.req_user_agent: "JFrog-Router"
        then:
          - add_fields:
              target: ''
              fields:
                req_call_type: "internal"
        else:
          - add_fields:
              target: ''
              fields:
                req_call_type: "external"
    fields:
      record_type: "jfmd"
    fields_under_root: true

  # Event Request
  - type: filestream
    id: event_request
    enabled: true
    paths:
      - /central-data/jfrog/artifactory/var/log/event-request.log
    processors:
      - dissect:
          tokenizer: "%{timestamp}|%{req_trace_id}|%{req_remote_address}|%{req_user}|%{req_type}|%{req_url}|%{res_status}|%{req_content_length}|%{res_content_length}|%{res_duration}|%{req_user_agent}"
          target_prefix: ""
      - if:
          or:
            - contains.req_remote_address: "127.0.0.1"
            - contains.req_user_agent: "JFrog-Router"
        then:
          - add_fields:
              target: ''
              fields:
                req_call_type: "internal"
        else:
          - add_fields:
              target: ''
              fields:
                req_call_type: "external"
    fields:
      record_type: "jfevt"
    fields_under_root: true

  # Access Request
  - type: filestream
    id: access_request
    enabled: true
    paths:
      - /central-data/jfrog/artifactory/var/log/access-request.log
    processors:
      - dissect:
          tokenizer: "%{timestamp}|%{req_trace_id}|%{req_remote_address}|%{req_user}|%{req_type}|%{req_url}|%{res_status}|%{req_content_length}|%{res_content_length}|%{res_duration}|%{req_user_agent}"
          target_prefix: ""
      - if:
          or:
            - contains.req_remote_address: "127.0.0.1"
            - contains.req_user_agent: "JFrog-Router"
        then:
          - add_fields:
              target: ''
              fields:
                req_call_type: "internal"
        else:
          - add_fields:
              target: ''
              fields:
                req_call_type: "external"
    fields:
      record_type: "jfac"
    fields_under_root: true

  - type: filestream
    id: audit_security_audit_log
    enabled: true
    paths:
      - /central-data/jfrog/artifactory/var/log/access-security-audit.log
    processors:
    - dissect:
        tokenizer: "%{timestamp}|%{trace_id}|%{user_ip}|%{user}|%{logged_principal}|%{entity_name}|%{event_typee}|%{eevent}|%{json_data}"
        field: "message"
        target_prefix: "dissected"

    - decode_json_fields:
        fields: ["dissected.json_data"]
        target: ""
        overwrite_keys: true
    fields:
      record_type: "jf_access_audit"
    fields_under_root: true

  # Artifactory Access Log
  - type: filestream
    id: artifactory_access
    enabled: true
    paths:
      - "/central-data/jfrog/artifactory/var/log/artifactory-access.log"
    processors:
      - dissect:
          tokenizer: "%{timestamp} [%{trace_id}] [%{action_response}] %{repo_path} for client : %{username} %{ip_delimiter} %{ip} [%{type}]"
          target_prefix: ""
      - drop_fields:
          fields: ["ip_delimiter"]
          ignore_missing: true
      - convert:
          fields:
            - {from: "ip", to: "ip", type: "ip"}
          ignore_missing: true
          fail_on_error: false
    fields:
      record_type: "jfrt-ac"
    fields_under_root: true

filebeat.modules:
  - module: nginx
    access:
      enabled: true
      var.paths: ["/var/log/nginx/*access.log*"]
    error:
      enabled: true
      var.paths: ["/var/log/nginx/error.log*"]

#setup.dashboards.enabled: true
setup.kibana:
  host: "Kibana LB"

output.elasticsearch:
  hosts: ["ES LB"]
  username: "admin"
  password: "*****"
  indices:
    - index: "active_jfrt_gc_metrics_data-%{+yyyy.MM.dd}"
      when.or:
        - equals:
            record_type: "jfrt_artifacts_gc"
    - index: "active_jfrt_request_data-%{+yyyy.MM.dd}"
      when.or:
        - equals:
            record_type: "jfmd"
        - equals:
            record_type: "jfac"
        - equals:
            record_type: "jfrt"
        - equals:
            record_type: "jfevt"
        - equals:
            record_type: "jffe"
      pipeline: "convert_to_long"
    - index: "active_jfrt_outbound_data-%{+yyyy.MM.dd}"
      when.or:
        - equals:
            record_type: "jfrt_remote_repo"
    - index: "active_jfac_audit-%{+yyyy.MM.dd}"
      when.or:
        - equals:
            record_type: "jf_access_audit"
    - index: "active_jfac-%{+yyyy.MM.dd}"
      when.or:
        - equals:
            record_type: "jfrt-ac"
    - index: "active_nginx_logs-%{+yyyy.MM.dd}"
      when.or:
        - equals:
            event.module: "nginx"


logging.level: debug
logging.to_files: true
logging.files:
  path: /central-data/jfrog/artifactory/var/log
  name: filebeat.log
  keepfiles: ${FILEBEAT_LOG_KEEPFILES:2}
  permissions: 0644

Index

{
  "active_jfrt_request_data-2025.02.06": {
    "aliases": {},
    "mappings": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "agent": {
          "properties": {
            "ephemeral_id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "id": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "ecs": {
          "properties": {
            "version": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "host": {
          "properties": {
            "name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "input": {
          "properties": {
            "type": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "log": {
          "properties": {
            "file": {
              "properties": {
                "device_id": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "inode": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                },
                "path": {
                  "type": "text",
                  "fields": {
                    "keyword": {
                      "type": "keyword",
                      "ignore_above": 256
                    }
                  }
                }
              }
            },
            "flags": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "offset": {
              "type": "long"
            }
          }
        },
        "message": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "record_type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "req_call_type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "req_content_length": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "req_remote_address": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "req_trace_id": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "req_type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "req_url": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "req_user": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "req_user_agent": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "res_content_length": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "res_duration": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "res_status": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "timestamp": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "routing": {
          "allocation": {
            "include": {
              "_tier_preference": "data_content"
            }
          }
        },
        "number_of_shards": "1",
        "provided_name": "active_jfrt_request_data-2025.02.06",
        "creation_date": "1738793065811",
        "number_of_replicas": "2",
        "uuid": "jQmhmUMRS-GrGPlf6X1_sg",
        "version": {
          "created": "8521000"
        }
      }
    }
  }
}

{AE8986AD-6D5A-4030-9DCF-5256D39CFD6F}

I have faced this 3days before when there was no such error mentioned as below.

And the filebeat error log shows

{"log.level":"error","@timestamp":"2025-02-06T07:17:39.537+0200","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/filestream/internal/input-logfile.(*updateOp).Execute","file.name":"input-logfile/publish.go","file.line":148},"message":"Failed to update state in the registry for 'filestream::access_request::native::21124515-64772': failed in store/get operation on store 'filebeat': write /central-data/jfrog/filebeat-8.16.3-linux-x86_64/data/registry/filebeat/log.json: no space left on device","service.name":"filebeat","input_type":"filestream","ecs.version":"1.6.0"}

Is it because of above error?

Is there any other option to use instead of filebeat processors?

ES and Kibana - 8.17.1
Filebeat - 8.16.3

Thanks!!