# ELK state management! **URL:** \ **Category:** Logstash\ **Created:** [August 18, 2015, 2:25am UTC](https://discuss.elastic.co/t/elk-state-management/27545 "2015-08-18T02:25:04Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![sreeram](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sreeram](https://discuss.elastic.co/u/sreeram)\ **Post date:** [August 18, 2015, 2:25am UTC](https://discuss.elastic.co/t/elk-state-management/27545/1 "2015-08-18T02:25:04Z")
Hi, I'm new to ELK, need clarifications on 1. what is the purpose of logstash-forwarder when logstash itself can ship files ? 2. I have a Usecase where my system might get restarted at times(killing all services abruptly), - when logstash-forwarder is restarted while reading a huge log file, how to implement state management? (I assume, .logstash-forwarder is not updated while reading) 3. which has better state management logstash(with .sincedb) as shipper or logstash-forwarder (with .logstash-forwarder) as shipper? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 18, 2015, 2:51am UTC](https://discuss.elastic.co/t/elk-state-management/27545/2 "2015-08-18T02:51:02Z")
1. It's light weight, so if you don't want to install a JVM everywhere you can use it. Plus it ships logs securely. 2. LSF and LS handle that with sincedb. 3. Dunno. ---
**Author:** ![sreeram](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sreeram](https://discuss.elastic.co/u/sreeram)\ **Post date:** [August 18, 2015, 3:18am UTC](https://discuss.elastic.co/t/elk-state-management/27545/3 "2015-08-18T03:18:42Z")
Hi Mark 🙂 , But when I restart logstash-forwarder service while shipping a huge log file, its duplicating logs. Should I do any explicit configurations to handle this ? Below is how my configurations look, **Logstash Config** input { lumberjack { # The port to listen on port =\> 5544 ssl\_certificate =\> "D:/logstash-forwarder/ssl/logstash-forwarder.crt" ssl\_key =\> "D:/logstash-forwarder/ssl/logstash-forwarder.key" type =\> "Logs" codec =\> plain { charset =\> "UTF-16" } } } **Logstash-forwarder config** { "network": { "servers": ["localhost:5544"], "ssl key": "D:/logstash-forwarder/ssl/logstash-forwarder.key", "ssl ca": "D:/logstash-forwarder/ssl/logstash-forwarder.crt", "timeout": 15 }, "files": [ { "paths": ["D:/logpath/\*\*/\*.txt"], "fields": { "type": "log" } } ] } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 18, 2015, 3:40am UTC](https://discuss.elastic.co/t/elk-state-management/27545/4 "2015-08-18T03:40:32Z")
LSF stores the state information in .logstash-forwarder, i.e. a file in the current directory. Is that file ever created for you? Are you starting LSF from the same directory both times? ---
**Author:** ![sreeram](https://avatars.discourse-cdn.com/v4/letter/s/b77776/32.png) [@sreeram](https://discuss.elastic.co/u/sreeram)\ **Post date:** [August 18, 2015, 8:10am UTC](https://discuss.elastic.co/t/elk-state-management/27545/5 "2015-08-18T08:10:44Z")
Yeah .logstash-forwarder file is created in current directory. But when i restart LSF service while reading data is duplicated. I'm still testing ELK, So it will be of great help if you can suggest me ELK setup best for my requirement, 1. Highly available (incase of system failure) 2. Need to schedule log reads (during non-business hours). 3. Better if all components are Freeware Thanks in advance. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [August 18, 2015, 8:32am UTC](https://discuss.elastic.co/t/elk-state-management/27545/6 "2015-08-18T08:32:18Z")
> Yeah .logstash-forwarder file is created in current directory. But when i restart LSF service while reading data is duplicated. That's weird and unexpected. I suggest you increase the logging and post the results. > Highly available (incase of system failure) Please be more specific. "High availability" is a fluffy term that means different things to different people. > Need to schedule log reads (during non-business hours). There's nothing built-in for this, but a cronjob to start and stop log collection daemons is easy to write. > Better if all components are Freeware Elasticsearch, Logstash, and Kibana are all open source. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:31am UTC](https://discuss.elastic.co/t/elk-state-management/27545/7 "2017-07-06T05:31:39Z")