# Running Logstash on multiple servers, avoiding double processing

**URL:** <https://discuss.elastic.co/t/running-logstash-on-multiple-servers-avoiding-double-processing/337780>\
**Category:** Logstash\
**Created:** [July 6, 2023, 10:28am UTC](https://discuss.elastic.co/t/running-logstash-on-multiple-servers-avoiding-double-processing/337780 "2023-07-06T10:28:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![BenSeb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benseb/32/8714_2.png) [@BenSeb](https://discuss.elastic.co/u/BenSeb)\
**Post date:** [July 6, 2023, 10:28am UTC](https://discuss.elastic.co/t/running-logstash-on-multiple-servers-avoiding-double-processing/337780/1 "2023-07-06T10:28:24Z")

</div>

Hi

We have logstash running on our worker servers, and they run with an identical config, to ensure if one hosts goes down, we are still processing events.

The source data is Mysql, then logstash pushes the latest records to Elasticsearch

eg

```auto
input {
  jdbc {
    type => event_v4
    clean_run => false
    jdbc_driver_library => "/usr/share/java/mysql-connector-java.jar"
    jdbc_driver_class => "com.mysql.jdbc.Driver"
    jdbc_connection_string => "jdbc:mysql://xx"
    jdbc_user => "xx"
    jdbc_password => "xx"
    jdbc_paging_enabled => true
    jdbc_page_size => 10000
    schedule => "*/5 * * * *"
    statement =>"xx"

```

We then schedule the logstash config every 5 mins.

Ideally I want to avoid doubling up the work, ie both servers picking up the same records and pushing them up - it doesn't cause issues but it's a waste of resources. Because we use Ansible to build the servers It's hard to set the cron template differently on each.

I was thinking that once the logstash job runs, we could update the mysql records with a 'lastindexed' date, then in the input/jdbc params, exclude anthing where that date is within the last few mins

eg:

WHERE lastUpdated \> :sql\_last\_value AND lastIndexed \< :sql\_last\_value

Is that possible, how can we update that column in mysql once the records are indexed (we currently only read from Mysql and write to Elastic)

Or any better way to handle this scenario?

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [July 10, 2023, 3:00pm UTC](https://discuss.elastic.co/t/running-logstash-on-multiple-servers-avoiding-double-processing/337780/2 "2023-07-10T15:00:20Z")

</div>

Hi @BenSeb,

I have a similar use case as you, and while I don't have an exact solution do have some pointers. We use the [user\_column\_value](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#plugins-inputs-jdbc-use_column_value) in our query so if a host goes down it can pick up where it left off. Only have one host pulling from the database, but setup a watcher to let me know if the data flow stops for any reason. If that happens it's usually easy to see what the problem is and again because the jdbc input is keeping track of where it left off is pretty seamless.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 7, 2023, 3:00pm UTC](https://discuss.elastic.co/t/running-logstash-on-multiple-servers-avoiding-double-processing/337780/3 "2023-08-07T15:00:45Z")

</div>

Closed on August 7, 2023, 3:00pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.