# ETL with logstash

**URL:** <https://discuss.elastic.co/t/etl-with-logstash/92979>\
**Category:** Logstash\
**Created:** [July 13, 2017, 9:57am UTC](https://discuss.elastic.co/t/etl-with-logstash/92979 "2017-07-13T09:57:38Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sanketshinde](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@sanketshinde](https://discuss.elastic.co/u/sanketshinde)\
**Post date:** [July 13, 2017, 9:57am UTC](https://discuss.elastic.co/t/etl-with-logstash/92979/1 "2017-07-13T09:57:38Z")

</div>

I am trying to implement an ETL using Logstash. The ETL is between two instances of ES DB, from a remote server to a localhost. I want to get the max of a field "endTime" from the documents in my localhost. I want to then store this value in an environment variable and use it in the ETL pipeline to filter events from the remote server for which the value of the "endTime" field is greater than this stored value of the environment variable.

I have set the pipeline for this ETL and the conf file looks like so:

input {  
elasticsearch {  
hosts =\> ["remotehost"]  
index =\> "source\_index"  
query =\> '{"query": {"bool": {"filter": [{"range": {"end": {"gt": "${my\_var}"}}}]}}}'  
}  
}  
output {  
elasticsearch{  
user =\> user  
password =\> pass  
hosts =\> ["localhost:9200"]  
index =\> "destination\_index"  
}  
}

Here my\_var is the environment variable. I would like to have a cascaded execution where the first pipeline computes the max of the endTime field from the localhost, stores it in the environment variable and then triggers the execution of the main pipeline(described in the conf file above).

Is there a way to achieve this ? Is there a better way to do this than how I thought of? Thanks for the thoughts in advance.

Sanket-

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [July 13, 2017, 2:12pm UTC](https://discuss.elastic.co/t/etl-with-logstash/92979/3 "2017-07-13T14:12:33Z")

</div>

I don't think you need two pipelines. The first pipeline would just make a single ES query to find the max value, and that's something you can do with curl (using jq to filter out the boring parts of the JSON response).

So basically you'd end up with something like this (preferably with better error handling, but you get the idea):

```
export my_var="$(curl ... | jq ...)"
logstash -f conf-file-you-just-posted.conf
```

---

<div class="post-metadata">

**Author:** ![sanketshinde](https://avatars.discourse-cdn.com/v4/letter/s/b5ac83/32.png) [@sanketshinde](https://discuss.elastic.co/u/sanketshinde)\
**Post date:** [July 14, 2017, 12:29pm UTC](https://discuss.elastic.co/t/etl-with-logstash/92979/4 "2017-07-14T12:29:09Z")

</div>

Thanks 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 11, 2017, 12:29pm UTC](https://discuss.elastic.co/t/etl-with-logstash/92979/5 "2017-08-11T12:29:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.