# Filebeat adds a log and Logstash reports it twice to elasticsearch. (Does not include old data, only upload new newly added data repeatedly)
**URL:** https://discuss.elastic.co/t/filebeat-adds-a-log-and-logstash-reports-it-twice-to-elasticsearch-does-not-include-old-data-only-upload-new-newly-added-data-repeatedly/138837
**Category:** Logstash
**Created:** [July 6, 2018, 7:24am UTC](https://discuss.elastic.co/t/filebeat-adds-a-log-and-logstash-reports-it-twice-to-elasticsearch-does-not-include-old-data-only-upload-new-newly-added-data-repeatedly/138837 "2018-07-06T07:24:54Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@sun\_changlong](https://discuss.elastic.co/u/sun_changlong)
#### Post date: [July 6, 2018, 7:24am UTC](https://discuss.elastic.co/t/filebeat-adds-a-log-and-logstash-reports-it-twice-to-elasticsearch-does-not-include-old-data-only-upload-new-newly-added-data-repeatedly/138837/1 "2018-07-06T07:24:54Z")
Add a new data to the filebeat detection log file，Filebeat debugging information output is as follows:
```
2018-07-06T15:12:21.543+0800 DEBUG [publish] pipeline/processor.go:291 Publish event: {
"@timestamp": "2018-07-06T07:12:16.327Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.3.0"
},
"input": {
"type": "log"
},
"prospector": {
"type": "log"
},
"beat": {
"name": "server2",
"hostname": "server2",
"version": "6.3.0"
},
"host": {
"name": "server2"
},
"source": "/home/testlog/taw.log",
"offset": 380,
"message": "recorder=ads vsid=2 sub_type=attacklog dst_addr=123.125.127.183 \nid=tos time=\"2017-08-16 16:58:50\" fw=IPS01.PUB.BEIJING-B pri=6 type=mgmt user=superman src=10.3.42.19 op=\"network interface eth13 show\" result=0 recorder=config msg=\"nuwwwww\"\nfw=IPS01.PUB.BEIJING-B pri=6 type=mgmt user=superman src=10.3.42.1 op=\"network interface eth13 show\" result=0 recorder=config msg=\"nuw\""
}
2018-07-06T15:12:22.970+0800 ERROR logstash/async.go:235 Failed to publish events caused by: write tcp 192.168.33.212:51106->192.168.33.85:5044: write: connection reset by peer
2018-07-06T15:12:23.971+0800 ERROR pipeline/output.go:92 Failed to publish events: write tcp 192.168.33.212:51106->192.168.33.85:5044: write: connection reset by peer
```
Logstash reports to elasticsearch twice in a row：
```
{
"source" => "/home/testlog/taw.log",
"sub_type" => "attacklog",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"host" => "server2",
"dst_addr" => "123.125.127.183",
"offset" => 380,
"show" => "This data is the test data",
"prospector" => {
"type" => "log"
},
"@timestamp" => 2018-07-06T07:12:16.322Z,
"input" => {
"type" => "log"
},
"@version" => "1",
"message" => "recorder=ads vsid=2 sub_type=attacklog dst_addr=123.125.127.183 \nid=tos time=\"2017-08-16 16:58:50\" fw=IPS01.PUB.BEIJING-B pri=6 type=mgmt user=superman src=10.3.42.19 op=\"network interface eth13 show\" result=0 recorder=config msg=\"nuwwwww\"\nfw=IPS01.PUB.BEIJING-B pri=6 type=mgmt user=superman src=10.3.42.1 op=\"network interface eth13 show\" result=0 recorder=config msg=\"nuw\"",
"type" => "replace_test",
"beat" => {
"version" => "6.3.0",
"name" => "server2",
"hostname" => "server2"
}
}
{
"source" => "/home/testlog/taw.log",
"sub_type" => "attacklog",
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"host" => "server2",
"dst_addr" => "123.125.127.183",
"offset" => 380,
"show" => "This data is the test data",
"prospector" => {
"type" => "log"
},
"@timestamp" => 2018-07-06T07:12:16.327Z,
"input" => {
"type" => "log"
},
"@version" => "1",
"message" => "recorder=ads vsid=2 sub_type=attacklog dst_addr=123.125.127.183 \nid=tos time=\"2017-08-16 16:58:50\" fw=IPS01.PUB.BEIJING-B pri=6 type=mgmt user=superman src=10.3.42.19 op=\"network interface eth13 show\" result=0 recorder=config msg=\"nuwwwww\"\nfw=IPS01.PUB.BEIJING-B pri=6 type=mgmt user=superman src=10.3.42.1 op=\"network interface eth13 show\" result=0 recorder=config msg=\"nuw\"",
"type" => "replace_test",
"beat" => {
"version" => "6.3.0",
"name" => "server2",
"hostname" => "server2"
}
}
```
I don't know why, and I didn't set a copy, it's a single node.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 3, 2018, 7:28am UTC](https://discuss.elastic.co/t/filebeat-adds-a-log-and-logstash-reports-it-twice-to-elasticsearch-does-not-include-old-data-only-upload-new-newly-added-data-repeatedly/138837/2 "2018-08-03T07:28:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.