Filebeat ignoring json created by ruby

Elastic Stack Beats
1

Thanks in advance for any help - I've been fighting with this thing all day and I don't feel like I've made much progress.
Here's the situation. Using Ansible facts, we are pulling out a list of all installed packages, then using ruby to parse out the values we need and create a (valid) json file. This part works great.
For whatever reason, filebeat will completely ignore this file no matter what you do (short of changing a single character in the file manually). I have tried creating it in a different folder and moving it to the prospector path, I've tried to copy, cat, echo, rename, chown to the local user, chmod to 755, etc, etc, etc, this file and nothing makes filebeat see it besides opening it in vim and changing even a single character and saving it. I guess saving it from vim causes it to be streamed, which causes filebeat to see it.
Further, I can see filebeat actually LOOKING at and actively ignoring the file. I see line after line of the file with "offset 0" and "backoff now" so it's clear filebeat thinks this is an old file. We are not using ignore_older.
Everything seems to be configured correctly, but it's not picking up the file until the offset goes to >0.
Oh, and I almost forgot to mention, I was putting the Ansible dump.json file (pre-parsed) in the same folder and IT got picked up just fine, but the one created by ruby does not.
Any help is greatly appreciated.

2

hey @DevopsNewb, welcome to discuss.

can you please share your FB config?
please make sure indentation is correct when you paste it and use </> button to format

3 
filebeat.prospectors:

  - input_type: log
    paths:
      - "/home/ec2-user/*.json"
    document_type: package-log-json
    fields:
      host_type: "{{ app_name }}-{{ bld_env }}"
      beats: true
output.logstash:
  hosts: ["logstash-int:5044"]
  timeout: 45
  ssl.certificate_authorities: ["/etc/filebeat/logstash-forwarder.crt"]
logging.to_files: true
logging.files:
    path: /var/log/filebeat
    name: filebeat.log
logging.selectors: ["*"]
logging.level: debug
4

I removed the extraneous prospectors. I know the yml is in good working order, or ansible would pitch a fit and crash the deployment. Also, this works just fine, if the file is edited in place.

5

Alright, I never got it to work, but I did figure out a workaround. Since I'm modifying the JSON on write anyway, I just had it leave off the closing brace, then inserted it with Ansible (echo '}' >> file) and Filebeat catches it now. Weird, but we can mark it complete.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.