Resend old logs from filebeat to logstash

[Arthur19 (Arthur)]

Hi,

Thanks in advance for your help. I would like to reload some logs to customize additional fields. I have noticed that registry file in filebeat configuration keeps track of the files already picked. However, if I remove the content in that file, I am not getting the old logs back. I have tried also to change the timestamp of the source in registry file with no sucsess. What changes are needed to sent old logs from filebeat to logstash?

How can I get back specific logs ?

Thanks a lot

[steffens (Steffen Siering)]

If you remove the file from the registry (or set "offset" to 0), then filebeat should pick it up again. Have you set ignore_older ?

[Arthur19 (Arthur)]

Thanks Steffen, I have set the offset to 0 and include ignore_older: 0 in filebeat.yml. I can see the messages I need in filebeat and logstasth, but I am not able to retrieve them in Kibana.

I am getting the event in Filebeat:

2019-03-14T16:18:45.374-0700    DEBUG   [publish]       pipeline/processor.go:308       Publish event: {
  "@timestamp": "2019-03-14T23:18:45.373Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.6.0"
  },
  "message": "2019-03-11 06:11:52 [localhost-startStop-1] DEBUG:   org.springframework.beans.factory.support.DefaultListableBeanFactory Returning cached instance of singleton bean 'swaggerBeanConfig'",
  "host": {
    "name": "tomcat",
    "os": {
      "platform": "centos",
      "version": "7 (Core)",
      "family": "redhat",
      "name": "CentOS Linux",
      "codename": "Core"
    },
    "id": "6aaed308aa5a419f880c5e45eea65414",
    "containerized": true,
    "architecture": "x86_64"
  },
  "meta": {
    "cloud": {
      "instance_id": "6452bcf4-7f5d-4fc3-9f8e-5ea57f00724b",
      "instance_name": "tomcat",
      "machine_type": "Standard_D8s_v3",
      "region": "CanadaCentral",
      "provider": "az"
    }
  },
  "input": {
    "type": "log"
  },
  "prospector": {
    "type": "log"
  },
  "beat": {
    "version": "6.6.0",
    "name": "tomcat",
    "hostname": "tomcat"
  },
  "source": "/apps/logs/WEB/web-rest-api/web-rest-api.log",
  "offset": 6764222,
  "log": {
    "file": {
      "path": "/apps/logs/WEB/web-rest-api/web-rest-api.log"
    }
  }
}

This event is taken by Logstash:

[2019-03-15T10:32:26,055][DEBUG][logstash.outputs.gelf    ] Sending GELF event {:
    event=>{"short_message"=>
	["2019-03-11 06:11:52 [localhost-startStop-1] DEBUG:   org.springframework.beans.factory.support.DefaultListableBeanFactory Returning cached instance of singleton bean 'swaggerBeanConfig'", " Returning cached instance of singleton bean 'swaggerBeanConfig'"], 
	"full_message"=>"2019-03-11 06:11:52 [localhost-startStop-1] DEBUG:   org.springframework.beans.factory.support.DefaultListableBeanFactory Returning cached instance of singleton bean 'swaggerBeanConfig', Returning cached instance of singleton bean 'swaggerBeanConfig'", 
	"host"=>"{\"name\":\"tomcat\",\"os\":{\"name\":\"CentOS Linux\",\"version\":\"7 (Core)\",\"codename\":\"Core\"}}", 
	"_source"=>"/apps/logs/WEB/web-rest-api/web-rest-api.log", 
	"_class"=>"org.springframework.beans.factory.support.DefaultListableBeanFactory, %{JAVACLASS}", 
	"_tags"=>"beats_input_codec_plain_applied", 
	"_beat_hostname"=>"tomcat", 
	"_beat_name"=>"tomcat", 
	"_meta_cloud"=>{}, 
	"_log_file"=>{"path"=>"/apps/logs/WEB/web-rest-api/web-rest-api.log"}, 
	"level"=>6}}

What would be a possible stopper to receive this type of message in Kibana?

Thanks