Filebeat repeat send log

Elastic Stack Beats
1 
2017/09/08 07:34:20.656217 client.go:184: DBG  Publish: {
  "@timestamp": "2017-09-08T07:34:16.242Z",
  "beat": {
    "hostname": "85",
    "name": "10.58.51.85",
    "version": "5.2.2"
  },
  "input_type": "log",
  "message": "\u001b[m\u001b[30m[2017-09-08 15:34:16,204] INFO  - httpAcc - {\"id\":\"14VKQI2nzzi\",\"server\":\"10.58.51.85\",\"time\":28,\"atime\":10,\"status\":200,\"region\":\"11010200\",\"ip\":\"10.58.24.99\",\"cookieId\":\"149239695917150050\",\"h\":\"list.gome.com.cn\",\"frm\":\"category\",\"brw\":\"Chrome\",\"ref\":\"https://list.gome.com.cn/cat10000062-00-0-48-1-0-0-0-1-15L4-0-1-10-0-0-0-0-0.html?intcmp=kt-1000073982-13\",\"osv\":\"\",\"isRtb\":false,\"rbt\":\"No\",\"userId\":\"\",\"url\":\"http://list.gome.com.cn/category/cat10000062-00-0-48-1-0-0-0-1-15L41b6G-0-1-10-0-0-0-0-0.html?intcmp=list-9000000500-6\u0026\"}",
  "offset": 6104112,
  "source": "/app/logs/gateway/gateway8080.log",
  "type": "tomcat_log"
}

2017/09/08 07:34:20.650469 client.go:184: DBG  Publish: {
  "@timestamp": "2017-09-08T07:34:16.237Z",
  "beat": {
    "hostname": "85",
    "name": "10.58.51.85",
    "version": "5.2.2"
  },
  "input_type": "log",
  "message": "\u001b[m\u001b[30m[2017-09-08 15:34:16,204] INFO  - httpAcc - {\"id\":\"14VKQI2nzzi\",\"server\":\"10.58.51.85\",\"time\":28,\"atime\":10,\"status\":200,\"region\":\"11010200\",\"ip\":\"10.58.24.99\",\"cookieId\":\"149239695917150050\",\"h\":\"list.gome.com.cn\",\"frm\":\"category\",\"brw\":\"Chrome\",\"ref\":\"https://list.gome.com.cn/cat10000062-00-0-48-1-0-0-0-1-15L4-0-1-10-0-0-0-0-0.html?intcmp=kt-1000073982-13\",\"osv\":\"\",\"isRtb\":false,\"rbt\":\"No\",\"userId\":\"\",\"url\":\"http://list.gome.com.cn/category/cat10000062-00-0-48-1-0-0-0-1-15L41b6G-0-1-10-0-0-0-0-0.html?intcmp=list-9000000500-6\u0026\"}",
  "offset": 6091964,
  "source": "/app/logs/gateway/gateway8080.log",
  "type": "tomcat_log"
}

The message is exactly the same except for the offset.
filebeat version 5.2.2

filebeat.yml:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

#input_type: log
- document_type: tomcat_log
  paths:
       - /app/logs/gateway/gateway8070.log
       - /app/logs/gateway/gateway8080.log
       - /app/logs/gateway/gateway8090.log
  tail_files: true
  multiline.pattern: '^[[:space:]]+|^Caused by:+|^[a-zA-Z]'
  multiline.negate: false
  multiline.match: after

output.logstash:
  # The Logstash hosts
  hosts: ["10.58.69.47:5045"]
  timeout: 300
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"
2

Could you please try removing tail_files: true from your config.

3

I've changed it, but it still doesn't work, Repetition still occur
We also did not restart filebeat。
If filebeat has bug???
I check the filebeat log, find rebeat send log。one data send many times.
Who can help me?

4

You must restart Filebeat after making the config changes.

5

Sorry, I didn't make myself clear
I change config, and restart filebeat, but after a period of time
Repeated data still happen.
What was it that led to this happening？

6

That is very strange given that the offsets are different. Can you confirm that the log message is not duplicated in the file.

How are the log files rotated? Can you please try Filebeat 5.6.1 (latest stable).

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.