# ERR Fail to publish event to REDIS: write tcp sourceHost:sourcePort-\>redisHost:redisPort: i/o timeout

**URL:** <https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 30, 2016, 11:49am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399 "2016-06-30T11:49:02Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [June 30, 2016, 11:49am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/1 "2016-06-30T11:49:02Z")

</div>

I have recently setup below ELK stack for one of my application:  
FileBeat --\> Redis --\> Logstash --\> Elasticsearch --\> Kibana  
My FileBeat config:

> ```
> filebeat:
> prospectors:
> -
> paths:
> - LogPath
> input_type: log
> tail_files: true
> output:
> redis:
> host: "<host>"
> port: port
> save_topology: true
> index: "filebeat"
> db: 0
> db_topology: 1
> timeout: 5
> reconnect_interval: 1
> shipper:
> logging:
> to_files: true
> files:
> path: /tmp
> name: mybeat.log
> level: error
> 
> ```

My redis config:  
\> bind host  
\> port port  
\> tcp-backlog 511  
\> timeout 0  
\> tcp-keepalive 0  
\> daemonize no  
\> supervised no  
\> pidfile /var/run/redis.pid  
\> loglevel warning  
\> logfile "logpath"  
\> databases 16  
\> stop-writes-on-bgsave-error yes  
\> rdbcompression yes  
\> rdbchecksum yes  
\> dbfilename dump.rdb  
\> dir ./  
\> slave-serve-stale-data yes  
\> slave-read-only yes  
\> repl-diskless-sync no  
\> repl-diskless-sync-delay 5  
\> repl-disable-tcp-nodelay no  
\> slave-priority 100  
\> maxmemory 2GB  
\> maxmemory-policy volatile-lru  
\> appendonly no  
\> appendfilename "appendonly.aof"  
\> appendfsync everysec  
\> no-appendfsync-on-rewrite no  
\> auto-aof-rewrite-percentage 100  
\> auto-aof-rewrite-min-size 64mb  
\> aof-load-truncated yes  
\> lua-time-limit 5000  
\> slowlog-log-slower-than 10000  
\> slowlog-max-len 128  
\> latency-monitor-threshold 0  
\> notify-keyspace-events ""  
\> hash-max-ziplist-entries 512  
\> hash-max-ziplist-value 64  
\> list-max-ziplist-size -2  
\> list-compress-depth 0  
\> set-max-intset-entries 512  
\> zset-max-ziplist-entries 128  
\> zset-max-ziplist-value 64  
\> hll-sparse-max-bytes 3000  
\> activerehashing yes  
\> client-output-buffer-limit normal 0 0 0  
\> client-output-buffer-limit slave 256mb 64mb 60  
\> client-output-buffer-limit pubsub 32mb 8mb 60  
\> hz 10  
\> aof-rewrite-incremental-fsync yes

I am getting below error on FileBeat logs:

> ERR Fail to publish event to REDIS: write tcp sourceHost:sourePort-\>redisHost:redisPort: i/o timeout

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 1, 2016, 9:14am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/2 "2016-07-01T09:14:27Z")

</div>

Which filebeat version are you using. It's not recommended to use filebeat 1.x releases with redis (due to potential data loss). The redis output has been completely rewritten in 5.0alpha3 giving you features like:

- proper error handling and error recovery
- load balancing support
- support for redis over SSL if redis is run behind stunnel for example
- SOCKS5 proxy support

have you checked with netstat on filebeat and redis machine if filebeat is actually connected? Any firewall in between?

Have you tried to increase the timeout? Timeout of 5 seconds you configured is already hughe.

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 1, 2016, 2:13pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/3 "2016-07-01T14:13:15Z")

</div>

Hi Steffens..Thanks for your reply mate..  
I have tried with all 3 versions. i.e 1.2.2 , 1.2.3 and 5.0.alpha4. Below is my observation:  
1.2.2 & 1.2.3 same result i.e they are able to connect to my redis server and send log events but intermittently I see this error in the beats.log:

> ERR Fail to publish event to REDIS: write tcp sourceHost:sourcePort-\>redisHost:redisPort: i/o timeout

When I tried with 5.x version of FileBeat for some reason it is not able to connect to redis server (although all the details remain same)  
Below is my 5.x filebeat.yml

> filebeat.prospectors:
> 
> - input\_type: log  
> paths:
> - logPath  
> multiline.pattern: "\[1\]_:[0-9]_:[0-9]\*,[0-9]?[0-9]?[0-9]?"  
> multiline.negate: true  
> multiline.match: after  
> tail\_files: false

> output.redis:  
> hosts: "hostname"  
> port: 6379  
> index: "filebeat"  
> db: 0  
> db\_topology: 1  
> timeout: 5s  
> max\_retries: 3  
> logging.level: info  
> logging.to\_files: true  
> logging.files:  
> path: /tmp  
> name: mybeat.log

Log output from 5.x version of Filebeat as belows (set at INFO level):

> 2016-07-01T21:59:50+08:00 INFO Home path: [filebeathome] Config path: [filebeathome] Data path: [filebeathome/data] Logs path: [filebeathome/logs]  
> 2016-07-01T21:59:50+08:00 INFO Setup Beat: filebeat; Version: 5.0.0-alpha4  
> 2016-07-01T21:59:50+08:00 INFO Max Retries set to: 3  
> 2016-07-01T21:59:50+08:00 INFO Activated redis as output plugin.  
> 2016-07-01T21:59:50+08:00 INFO Publisher name: publisher\_name  
> 2016-07-01T21:59:50+08:00 INFO Flush Interval set to: 1s  
> 2016-07-01T21:59:50+08:00 INFO Max Bulk Size set to: 2048  
> 2016-07-01T21:59:50+08:00 INFO filebeat start running.  
> 2016-07-01T21:59:50+08:00 INFO Registry file set to: filebeathome/data/registry  
> 2016-07-01T21:59:50+08:00 INFO No registry file found under: filebeathome/data/registry. Creating a new registry file.  
> 2016-07-01T21:59:50+08:00 INFO Loading Prospectors: 1  
> 2016-07-01T21:59:50+08:00 INFO Load previous states from registry into memory  
> 2016-07-01T21:59:50+08:00 INFO Previous states loaded: 0  
> 2016-07-01T21:59:50+08:00 INFO Starting Registrar  
> 2016-07-01T21:59:50+08:00 INFO Loading Prospectors completed. Number of prospectors: 1  
> 2016-07-01T21:59:50+08:00 INFO Start sending events to output  
> 2016-07-01T21:59:50+08:00 INFO All prospectors are initialised and running with 0 states to persist  
> 2016-07-01T21:59:50+08:00 INFO Starting prospector of type: log  
> 2016-07-01T21:59:50+08:00 INFO Starting spooler: spool\_size: 2048; idle\_timeout: 5s  
> 2016-07-01T22:00:00+08:00 INFO Run prospector  
> 2016-07-01T22:00:10+08:00 INFO Run prospector  
> 2016-07-01T22:00:20+08:00 INFO Run prospector  
> .  
> .  
> .  
> .  
> .

In case of 5.x none of the log events reach redis server, but in case of 1.2.2 & 1.2.3 atleast most of them are reaching but intermittently i am getting that error.  
Can you tell me what can i do in this regards  
Your help is much appreciated  
Thanks in Advance  
Regards  
Mrunal

* * *

1. 0-9

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 1, 2016, 6:36pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/4 "2016-07-01T18:36:13Z")

</div>

In 5.0alpha4 the connections are established lazily (after first event is going to be pushed), in order to not block startup (e.g. init scripts) on startup if network is unresponsive. I don't see any events being pushed yet, so no attempt to connect to redis is made.

Due to new redis output supporting load-balancing, the `output.redis.hosts` requires you to use an array:

```auto
output.redis:
  hosts: ["hostname:port"]
  save_topology: true
  db_topology: 1

```

Have you tried to telnet you redis server? e.g.

```auto
$ telnet host port
INFO
...

```

redis being mixed of test/binary protocol you can execute some commands like INFO right via telnet.

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 5, 2016, 7:48am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/5 "2016-07-05T07:48:22Z")

</div>

HI Steffen  
As you suggested I did specify the host port in filebeat's config file and started filebeat..But still same issue nothing being sent over to redis server. I also tried doing telnet host port to redis server host and port from filebeat host and issued INFO command and below is the output:

> INFO
> 
> # Server
> 
> redis\_version:3.2.0  
> redis\_git\_sha1:00000000  
> redis\_git\_dirty:0  
> redis\_build\_id:926e4c00a759e20e  
> redis\_mode:standalone  
> os:Linux   
> arch\_bits:64  
> multiplexing\_api:epoll  
> gcc\_version:4.8.4  
> process\_id:  
> run\_id:  
> tcp\_port:6379  
> uptime\_in\_seconds:593  
> uptime\_in\_days:0  
> hz:10  
> lru\_clock:  
> executable:  
> config\_file:

> # Clients
> 
> connected\_clients:2  
> client\_longest\_output\_list:0  
> client\_biggest\_input\_buf:0  
> blocked\_clients:0

> # Memory
> 
> used\_memory:898080  
> used\_memory\_human:877.03K  
> used\_memory\_rss:2396160  
> used\_memory\_rss\_human:2.29M  
> used\_memory\_peak:898080  
> used\_memory\_peak\_human:877.03K  
> total\_system\_memory:33669894144  
> total\_system\_memory\_human:31.36G  
> used\_memory\_lua:32768  
> used\_memory\_lua\_human:32.00K  
> maxmemory:2147483648  
> maxmemory\_human:2.00G  
> maxmemory\_policy:volatile-lru  
> mem\_fragmentation\_ratio:2.67  
> mem\_allocator:libc

> # Persistence
> 
> loading:0  
> rdb\_changes\_since\_last\_save:0  
> rdb\_bgsave\_in\_progress:0  
> rdb\_last\_save\_time:1467703634  
> rdb\_last\_bgsave\_status:ok  
> rdb\_last\_bgsave\_time\_sec:-1  
> rdb\_current\_bgsave\_time\_sec:-1  
> aof\_enabled:0  
> aof\_rewrite\_in\_progress:0  
> aof\_rewrite\_scheduled:0  
> aof\_last\_rewrite\_time\_sec:-1  
> aof\_current\_rewrite\_time\_sec:-1  
> aof\_last\_bgrewrite\_status:ok  
> aof\_last\_write\_status:ok

> # Stats
> 
> total\_connections\_received:2  
> total\_commands\_processed:6913  
> instantaneous\_ops\_per\_sec:11  
> total\_net\_input\_bytes:216805  
> total\_net\_output\_bytes:9263  
> instantaneous\_input\_kbps:0.34  
> instantaneous\_output\_kbps:0.01  
> rejected\_connections:0  
> sync\_full:0  
> sync\_partial\_ok:0  
> sync\_partial\_err:0  
> expired\_keys:0  
> evicted\_keys:0  
> keyspace\_hits:0  
> keyspace\_misses:2304  
> pubsub\_channels:0  
> pubsub\_patterns:0  
> latest\_fork\_usec:0  
> migrate\_cached\_sockets:0

> # Replication
> 
> role:master  
> connected\_slaves:0  
> master\_repl\_offset:0  
> repl\_backlog\_active:0  
> repl\_backlog\_size:1048576  
> repl\_backlog\_first\_byte\_offset:0  
> repl\_backlog\_histlen:0

> # CPU
> 
> used\_cpu\_sys:0.12  
> used\_cpu\_user:0.18  
> used\_cpu\_sys\_children:0.00  
> used\_cpu\_user\_children:0.00

> # Cluster
> 
> cluster\_enabled:0

> # Keyspace

Also the fact that filebeat 1.2.2 and 1.2.3 version are able to send log events to redis server (with intermittent i/o timeout issues) tells me that there is nothing wrong with connectivity between 1.x versions of filebeat with redis but something is wrong with the 5.0.alpha4 version. Can you tell me what could be done in the 1.x version itself to resolve i/o timeout issue. I see max\_retries option in 5.x version of filebeat which suggests that filebeat will try to send failed log events thrice..is this supported in 1.x version?

Regards  
Mrunal

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 5, 2016, 1:40pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/6 "2016-07-05T13:40:24Z")

</div>

Well, you can always try to increase the timeout, but I can not recommend using filebeat 1.x with redis output. In first place, I wonder why requests do timeout.

Do you have some logs (in debug mode) with 5.x release?

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 8, 2016, 11:41am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/7 "2016-07-08T11:41:38Z")

</div>

Hi Steffen  
In my view instead of timeout if we had a feature of retry N times after every N milliSeconds (pooling) then that should resolve this case..I wonder if such a thing is available or not? Also in my second post / reply I have added logs from 5.0.alpha4 (DEBUG) and as you could see there it is not able to send any log events it just keeps crying Running prospectors..Same details when added 1.2.3 version does the job..but there is only that intermittent issue of i/o timeout. I am not sure why it times out.  
For now I guess I will have to go ahead with this issue as is..until you have any concrete solution to resolve this.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 8, 2016, 5:05pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/8 "2016-07-08T17:05:57Z")

</div>

I can not tell if 1.2.3 did work correctly. 1.2.3 does not check the redis response or for timeout errors and therefore will not retry.

> [@mrunalgosar](#):
>
> In my view instead of timeout if we had a feature of retry N times after every N milliSeconds (pooling)

This is exactly what 5.0alpha4 is supposed todo.

If all you see is messages from prosepector, I wonder if there are any files to be harvested as harvesters and spooler should get you some messages too if debug log is enabled. Did you delete the registry files between runs?

Some network trace between filebeat-5.0alpha4 and redis would be interesting to see what's happening on network level: `$ tcpdump -w trace.pcap tcp port 6379`. Are redis commands send correctly. Is there a response from redis?

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 12, 2016, 5:36pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/9 "2016-07-12T17:36:59Z")

</div>

Hi Steffen  
For testing purpose I have kept filebeat 1.2.3 running now for more than 6 days and I can see messages being sent over to the redis instance and all the data being collected..But with the intermittent issue of i/o timeout. On the other hand, on a parallel environment doing same thing I have setup 5.0.alpha4 filebeat setup and yet nothing has been sent over to redis instance (Note: I have 2 ELK stack setup one running filebeat 1.2.3 and other one running 5.0.alpha4.). I did try the commands you suggested earlier and things look ok for both the setup. Its just that when filebeat 5.0.aplha4 runs with a similar config nothing is being sent over to [redis.So](http://redis.So) my guess is there is something wrong on that version.  
On the other hand can u tell me if I can run 5.0.alpha4 in debug mode with source code to see complete flow and understand where the break is?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 13, 2016, 12:54pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/10 "2016-07-13T12:54:44Z")

</div>

Which commands exactly have you run?

Can you share filebeat 5.0alpha4 config?

Have you delete registry file before starting filebeat?

Any log output from filebeat?

Have you created a network trace as requested?

Debug log for redis can be enabled by adding the logging selector 'redis', either to your config file (logging section) or command line flag `-d 'redis'`. In case filebeat tries to publish any events you will see a 'connect' message from redis + when enabling INFO mode (e.g. `-v` on command line) some state update from filebeat:

```auto
2016/07/13 12:32:17.048831 publish.go:111: INFO Events sent: 2048
2016/07/13 12:32:17.050219 registrar.go:237: INFO Registry file updated. 1 states written.

```

We're having some automated (well, still minimal) integration tests for redis + I just successfully tested redis output with alpha4.

My filebeat config:

```auto
filebeat.prospectors:
- input_type: log
  paths:
    - tmp/nasa/*.log

output.redis:
  hosts: ["localhost"]
  index: "test"
  db: 0
  datatype: "list"

```

No need to configure anything-topology in filebeat redis output, as this is a packetbeat only feature.

I tested with NASA HTTP access log: [http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html](http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html)

My test system has redis 3.0.5 installed. Need to test manually with more recent release, but integration tests use [redis 3.2.0](https://github.com/elastic/beats/blob/master/libbeat/docker-compose.yml#L34).

My corresponding logstash config:

```auto
input {
    redis {
        host => "localhost"
        key => "test"
        data_type => "list"
        db => 0
    }
}

output {
    stdout {
        codec => "dots"
    }
}

```

This config prints one dot per received event. Remove or comment out the `codec` field in stdout to print the `message` field to standard out.

Running filebeat + redis + logstash with these configs happily spams my console until all logs have been processed.

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 18, 2016, 11:14am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/11 "2016-07-18T11:14:33Z")

</div>

Hi Steffen  
I think I now know what is the issue here..5.x filebeat version cannot support symlink..So in our current setup a symlink points to the latest log file and in 1.2.3 i was specifying that symlink path for sourcing logs..so i was repeatedly copying the same path..but it occurred to my mind to run this in debug mode and specify only one log file without symlink and 5.x version started running.. so the issue with 5.x version is it is skipping the path if its a symlink..which in my case it will always be  
Log excerpts:

> 2016-07-18T07:12:40-04:00 DBG File \<symlink\_path\> skipped as it is a symlink.  
> 2016-07-18T07:12:40-04:00 DBG File \<symlink\_path\> skipped as it is a symlink.  
> 2016-07-18T07:12:40-04:00 DBG File \<symlink\_path\> skipped as it is a symlink.  
> 2016-07-18T07:12:40-04:00 DBG File \<symlink\_path\> skipped as it is a symlink.  
> 2016-07-18T07:12:40-04:00 DBG File \<symlink\_path\> skipped as it is a symlink.  
> 2016-07-18T07:12:40-04:00 DBG File \<symlink\_path\> skipped as it is a symlink.  
> 2016-07-18T07:12:40-04:00 DBG File \<symlink\_path\> skipped as it is a symlink.  
> 2016-07-18T07:12:40-04:00 DBG File \<symlink\_path\> skipped as it is a symlink.

..so i have hit a dead end i guess here..any suggestions or workarounds?

Regards  
Mrunal

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 18, 2016, 4:45pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/12 "2016-07-18T16:45:44Z")

</div>

[this is the issue](https://github.com/elastic/beats/issues/1686) for removing symlink support in filebeat. I understand symlinks being somewhat tricky at times, leading to additional edge cases in filebeat. As far as I can tell, there is currently no option to enable symlink support. Workarounds might include hard-links or copying files to be forwarded.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 19, 2016, 7:37am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/13 "2016-07-19T07:37:31Z")

</div>

@mrunalgosar Can you share some more details on how it works with your symlink? It always helps to understand the use case before we implement a potential flag to enable symlinks again.

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 19, 2016, 10:33am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/14 "2016-07-19T10:33:46Z")

</div>

@ruflin We have a background process which rolls over the log files based on either day has ended / file size has exceeded certain limit or a restart of application is triggered..and then all the log events are sent to new log files. so we have a symlink which always points to the latest log file created..which is why i had specified that in "paths" property of filebeat. If symlink feature is removed from filebeat 5.x then we will not be able to upgrade ourselves anytime from 1.x to 5.x. Can this feature be made configurable and leave it to the end user to enable it or disable it?

Regards  
Mrunal

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 19, 2016, 10:35am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/15 "2016-07-19T10:35:31Z")

</div>

@steffens It seems hardlink is not possible for us as we have a background process which switches symlinks to latest log file when ever log files roll over. so i suggest the point mentioned in that issue that we keep symlinks feature as configurable.

Regards  
Mrunal

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [July 19, 2016, 1:59pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/16 "2016-07-19T13:59:35Z")

</div>

why you need symlinks? Why not point filebeat at location files are placed into? filebeat detects files being renamed, continues renamed file until fully processed and continues with new file. Using symlinks here, I'm not sure events are either lost, or ingestion of new-file is deferred due to previous log file still being processed.

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 19, 2016, 6:36pm UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/17 "2016-07-19T18:36:51Z")

</div>

@steffens Symlinks comes handy, in that we don't have to know where latest log file is to check..all the log files are placed inside same directory so its quite difficult to look for latest log file..so a symlink helps us.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 20, 2016, 6:37am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/18 "2016-07-20T06:37:34Z")

</div>

You write "to find the most recent log file": Is that for you manually or for filebeat? Because filebeat would do that automatically.

---

<div class="post-metadata">

**Author:** ![mrunalgosar](https://avatars.discourse-cdn.com/v4/letter/m/eada6e/32.png) [@mrunalgosar](https://discuss.elastic.co/u/mrunalgosar)\
**Post date:** [July 20, 2016, 6:45am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/19 "2016-07-20T06:45:59Z")

</div>

For us manually as well as how would filebeat come to know automatically? there are several log files in the same folder with the same extension and name pattern with slight difference in the time stamp prefixed in their names. I would suggest that we keep symlinks feature as configurable as against completely removing it.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [July 20, 2016, 7:26am UTC](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399/20 "2016-07-20T07:26:33Z")

</div>

Filebeat does the detection automatically as it doesn't track files by name but by `inode` and `device` id. It seems like removing the symlink would only make your manual look harder. At the same time I would hope that you don't have to touch the config files manually anymore as you have all the data in elasticsearch.

For the config option: Please open a feature request for it on Github and link to this issue: [https://github.com/elastic/beats/issues/1686](https://github.com/elastic/beats/issues/1686) Most efficient way to get it in would be a PR 😉

[Next page](https://discuss.elastic.co/t/err-fail-to-publish-event-to-redis-write-tcp-sourcehost-sourceport-redishost-redisport-i-o-timeout/54399.md?page=2)