# Ingestion pipeline
**URL:** https://discuss.elastic.co/t/ingestion-pipeline/79122
**Category:** Logstash
**Created:** [March 18, 2017, 7:54am UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122 "2017-03-18T07:54:59Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [March 18, 2017, 7:54am UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/1 "2017-03-18T07:54:59Z")
Hi,
I am looking at ingestion pipeline using logstash into elasticsearch.
Source data comes from sql database (MySQL)
I have used the jdbc input plugin for logstash which is fantastic.
I have a question:
I want to schedule logstash to index the data on a recurring basis. (aka Update existing index) I know that the jdbc plugin has this capability already.
Is this a reindex or new index of the data?
My scheduler prob only has to run once a month.
I have used aliases before and found them useful to atomically switch from one index to the next. I understand logstash does not support that. I would have to use the curator.
So...
Do i simply just go with scheduler or look at using an alias!?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 19, 2017, 10:53pm UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/2 "2017-03-19T22:53:54Z")
The scheduler will pull any new data it finds, based on the `sql_last_value`.
You can configure things so that you generate a custom document ID so that it'll update things in ES though.
---
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [March 20, 2017, 12:14am UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/3 "2017-03-20T00:14:15Z")
thanks for the reply mark.
1. If there is new records in the database are they bulk indexed into the existing index?
2. if there is updates to existing records in database, are the existing documents updated in existing index?
3. same for removed records...
thanks,
shane.
---
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [March 20, 2017, 12:20pm UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/4 "2017-03-20T12:20:52Z")
I found some interesting topics by searching for `sql_last_value`.
This post below demonstrates using sql\_last\_value and how to actually update documents.
As you said, it might be best to use mysql primary key as document id to handle any updates.
> [@MySQL to Elasticsearch](https://discuss.elastic.co/t/mysql-to-elasticsearch/41367/19):
>
> Please consult the documentation: [https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html#updating-plugins](https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html#updating-plugins) (But you might as well upgrade to Logstash 2.2.1. There's no reason to run anything older.)
Only issue I have with that is I remember reading if you let elasticsearch handle document id the bulk index is much faster?
The only action left then is delete. If record has been removed, will logstash handle that also?
Regards,
Shane.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 20, 2017, 8:59pm UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/5 "2017-03-20T20:59:41Z")
> [@shane\_lee](#):
>
> Only issue I have with that is I remember reading if you let elasticsearch handle document id the bulk index is much faster?
Yes, but it's a trade off.
> [@shane\_lee](#):
>
> The only action left then is delete. If record has been removed, will logstash handle that also?
No, only updates.
---
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [March 21, 2017, 1:09am UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/6 "2017-03-21T01:09:12Z")
> [@warkolm](#):
>
> No, only updates.
So what do you do in this circumstance?
is this a feature requested by users out of interest?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 21, 2017, 5:41am UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/7 "2017-03-21T05:41:10Z")
Well, LS isn't a state machine so it doesn't track this sort of thing to allow deletes.
You could just reindex everything into a new index every time, then delete the old one.
---
### Author:  [@shane\_lee](https://discuss.elastic.co/u/shane_lee)
#### Post date: [March 22, 2017, 1:16am UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/8 "2017-03-22T01:16:52Z")
> [@shane\_lee](#):
>
> I have used aliases before and found them useful to atomically switch from one index to the next. I understand logstash does not support that. I would have to use the curator.
No worries.
Which brings me back to my initial question...
Should i use logstash scheduler to create a new index everytime? Switch the alias across and delete the old index?
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 22, 2017, 2:09am UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/9 "2017-03-22T02:09:02Z")
Sounds like a good idea given your use case!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 19, 2017, 2:09am UTC](https://discuss.elastic.co/t/ingestion-pipeline/79122/10 "2017-04-19T02:09:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.