# Restarting logstash cloudwatch plugin **URL:** \ **Category:** Logstash\ **Created:** [April 10, 2023, 6:54pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681 "2023-04-10T18:54:14Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 10, 2023, 6:54pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/1 "2023-04-10T18:54:14Z")
We have an ELK stack app that has been down for over a month due to a credentials issue in the logstash cloudwatch plugin. The plugin is digesting data again now, but what is strange is that it is digesting logs from the beginning of time. So logs from over two years ago. Also, no data is being outputted to Elasticsearch, perhaps because that data has already been transformed and outputted previously? My main question: is this typical behavior? I'm not very familiar with logstash and elasticsearch, but I can't imagine every time you restart logstash it starts digesting every cloudwatch log from the very first logs. Not sure if it will help, but here is the logstash conf fiel for the cloudwatch plugin: ```auto input { cloudwatch_logs { access_key_id => access_here secret_access_key => secret_here log_group => ["xwingui-Prod", "xwingui-Dev", "xwingui-Exp", "xwingui-Staging", "xwingui-Test", "xwingui-Jawn"] region => "us-west-1" sincedb_path => "/var/lib/.sincedb" } } filter { if "Monitoring - " in [message] { if "API" in [message] { grok { match => { "message" => "API Monitoring - %{GREEDYDATA:json}" } } mutate { add_field => { "monitorType" => "API" } } } else if "RUM" in [message] { grok { match => { "message" => "RUM Monitoring - %{GREEDYDATA:json}" } } mutate { add_field => { "monitorType" => "RUM" } } } else if "PikaWorker" in [message] { grok { match => { "message" => "PikaWorker Monitoring - %{GREEDYDATA:json}" } } mutate { add_field => { "monitorType" => "PikaWorker" } } } else if "DataAgent" in [message] { grok { match => { "message" => "DataAgent Monitoring - %{GREEDYDATA:json}" } } mutate { add_field => { "monitorType" => "DataAgent" } } } else if "Database" in [message] { grok { match => { "message" => "Database Monitoring - %{GREEDYDATA:json}" } } mutate { add_field => { "monitorType" => "Database" } } } json { source => "json" remove_field => "message" } mutate { add_field => { "isMonitor" => True } } } } output { elasticsearch { hosts => ["localhost:9200"] user => user_here password => pwd_here } stdout { codec => json } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 10, 2023, 9:40pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/2 "2023-04-10T21:40:24Z")
> [@mphilip9](#): > > My main question: is this typical behavior? No. The input tracks what it has ingested in the sincedb. If "/var/lib/.sincedb" were removed then it would start over at the beginning, as you are seeing. How do you know no data is going to elasticsearch? Could it be that your index rotation is automatically delete indexes containing two year old data? ---
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 10, 2023, 10:36pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/3 "2023-04-10T22:36:50Z")
I just checked the index lifecycle policies and it looks like they have been saving everything. I checked the elasticsearch logs and there is nothing there. ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/7/c71e4490524ce1d916c9d3181c0e0619ba238cda.png) /var/lib/.sincedb is still there, is there a way to check its contents? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 10, 2023, 10:50pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/4 "2023-04-10T22:50:46Z")
> [@mphilip9](#): > > /var/lib/.sincedb is still there, is there a way to check its contents? `more /var/lib/.sincedb` should work. It is [written](https://github.com/lukewaite/logstash-input-cloudwatch-logs/blob/bde0fd896418c5861d56639c2d5ae47abc0cb725/lib/logstash/inputs/cloudwatch_logs.rb#L260) as a text file. The number next to the group identifier is the timestamp of the last message that was read in milliseconds since the epoch (.strftime("%Q")). ---
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 10, 2023, 10:59pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/5 "2023-04-10T22:59:39Z")
Hmm interesting, the time is 1610132873632 which is today. So technically it shouldn't be ingesting all those old logs? ---
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 10, 2023, 11:03pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/6 "2023-04-10T23:03:06Z")
For example, I'm seeing logs like this ```auto "log_stream":"root","ingestion_time":"2021-08-10T21:23:00.999Z" ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 11, 2023, 1:56am UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/7 "2023-04-11T01:56:37Z")
The AWS [API](https://docs.aws.amazon.com/sdk-for-ruby/v2/api/Aws/CloudWatchLogs/Client.html#filter_log_events-instance_method) returns an array of events, each of which has log\_stream\_name, timestamp, message, ingestion\_time, event\_id fields. The [@timestamp] field is set from the timestamp field, not the ingestion\_time field. Is the @timestamp current or in 2021? If the latter it sounds like the issue is on the AWS side. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 11, 2023, 2:08am UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/8 "2023-04-11T02:08:08Z")
> [@mphilip9](#): > > Hmm interesting, the time is 1610132873632 which is today. This date is not today, it is the epoch in milisconds for January 08th, 2021. You can check on bash by removing the 3 last numbers and running the following command: ```auto date -d@1610132873 ``` ---
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 11, 2023, 3:33am UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/9 "2023-04-11T03:33:48Z")
Ah thanks for that! I l had used an epoch converter but got the wrong date. That makes sense. It appears the timestamp is in 2021 and not current ```auto "@timestamp":"2021-12-10T21:25:23.645Z" ``` Shouldn't the .sincedb file be updated as new logs are ingested? ---
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 11, 2023, 3:51am UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/10 "2023-04-11T03:51:43Z")
Would it make sense to simply manually edit the file? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 11, 2023, 3:06pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/11 "2023-04-11T15:06:45Z")
> [@mphilip9](#): > > Would it make sense to simply manually edit the file? That should work, or delete the entry for the group and set `start_position => end`. (You need to stop logstash before changing the file and restart it afterwards.) ---
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 11, 2023, 6:48pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/12 "2023-04-11T18:48:27Z")
What do you mean by delete the entry for the group? the sincedb\_path? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 11, 2023, 6:52pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/13 "2023-04-11T18:52:52Z")
Yes, you should be able to delete the line from the file (make sure to have a backup). ---
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 11, 2023, 7:30pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/14 "2023-04-11T19:30:51Z")
Sorry to ask the same question twice, but to make doubly sure. If I change this line in the cloudwatch conf file: ```auto sincedb_path => "/var/lib/.sincedb" ``` to ```auto start_position => end ``` Logstash will essentially just process log groups that are brand new (sincedb time will be set to now)? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 11, 2023, 8:06pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/15 "2023-04-11T20:06:30Z")
You would add the `start_position => end` to the existing configuration. Do not remove the sincedb\_path. ---
**Author:** ![mphilip9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mphilip9/32/119592_2.png) [@mphilip9](https://discuss.elastic.co/u/mphilip9)\ **Post date:** [April 13, 2023, 1:42pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/16 "2023-04-13T13:42:41Z")
Hey that worked like a charm and everything is running smoothly now. Thanks for all you help Badger! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 11, 2023, 1:43pm UTC](https://discuss.elastic.co/t/restarting-logstash-cloudwatch-plugin/329681/17 "2023-05-11T13:43:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.