# Logstash creates a new index every 24 hours
**URL:** https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793
**Category:** Logstash
**Created:** [July 19, 2018, 7:00pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793 "2018-07-19T19:00:51Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [July 19, 2018, 7:00pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/1 "2018-07-19T19:00:51Z")
Hi all,
I am using [https://openweathermap.org/](https://openweathermap.org/current) current weather API to display current weather data.
I was able successfully visualize the data on Kibana after it created a new index **weather-2018.07.18**.
Here is my logstash config file.
```
input {
http_poller {
urls => {
weather => {
url => "http://api.openweathermap.org/data/2.5/weather?id=5490223&appid=MY_APP_ID&units=metric"
headers => {
Accept => "application/json"
}
}
}
schedule => { cron => "* */2 * * * *" }
codec => json
}
}
filter {
mutate {
remove_field => ["@version" ,"command" ,"host" ,"cod" ,"id" ,"base" ,"coord" ,"sys" ,"dt"]
}
split { field => "weather" }
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "weather-%{+YYYY.MM.dd}"
}
stdout {
codec => rubydebug
}
}
```
However, after 24 hours a new index **weather-2018.07.19** was created on elasticsearch.
```
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open .kibana ZB7x3FQyQlyRvkRIaEGi2g 1 0 5 0 32.7kb 32.7kb
green open .monitoring-kibana-6-2018.07.12 8iFI69JhQL-bLWsGZzFQYA 1 0 2370 0 697.8kb 697.8kb
green open .monitoring-es-6-2018.07.12 4-XcOMWsRkW7_enzqslTQg 1 0 34226 115 12.7mb 12.7mb
yellow open logstash-2018.07.13 _oiQOnpMQ4KQcHssK8PmxA 5 1 9999 0 7mb 7mb
green open .monitoring-es-6-2018.07.18 h_ylFE80RS6153Sx5A7o6A 1 0 51522 175 23.4mb 23.4mb
green open .monitoring-es-6-2018.07.15 27pTcx8TTDqchfQgo7WQ-w 1 0 234 0 160.2kb 160.2kb
green open .monitoring-kibana-6-2018.07.16 Lmydu8eOS8-p6aKPP7SUqw 1 0 1383 0 465.1kb 465.1kb
green open .monitoring-es-6-2018.07.17 HThfGRdATO-C455Xm_-e9Q 1 0 49107 175 18.1mb 18.1mb
green open .monitoring-kibana-6-2018.07.17 u1EedigZSWeXOaOwdueEdQ 1 0 2450 0 700.7kb 700.7kb
green open .monitoring-kibana-6-2018.07.14 17TgPEalTAOJRe3gb59eZA 1 0 15 0 87.5kb 87.5kb
green open .monitoring-es-6-2018.07.19 YbmtJ2qLSNSAJIEluEz3Zw 1 0 9304 235 9.9mb 9.9mb
green open .monitoring-es-6-2018.07.16 3iZspMNoQUuty5lYM5ylvg 1 0 39028 135 17.3mb 17.3mb
yellow open weather-2018.07.19 M7b9ojh3Qc631m8NqlZOzA 5 1 1364 0 348.7kb 348.7kb
green open .monitoring-kibana-6-2018.07.19 br9oBb0ISgi5x-ZkDb7Hnw 1 0 372 0 346.5kb 346.5kb
green open .monitoring-es-6-2018.07.13 1FBGvkSzR4yf7WcyrwN3LQ 1 0 51214 50 19.2mb 19.2mb
yellow open weather-2018.07.18 EAxPkKAPQceMEH9K2u_EEQ 5 1 4378 0 731.1kb 731.1kb
green open .monitoring-kibana-6-2018.07.18 ZnYApUXKQbiH9CVFRqmzdw 1 0 2463 0 742.6kb 742.6kb
green open .monitoring-kibana-6-2018.07.15 9TZ9csgUQp-LrAbRYT4aGw 1 0 9 0 127.5kb 127.5kb
green open .monitoring-kibana-6-2018.07.13 Cpt83ctnQiqLWiydy1nmUw 1 0 2585 0 830.1kb 830.1kb
green open .monitoring-es-6-2018.07.14 fuXHKIMNS5u15-XR0dvrTw 1 0 380 162 575.3kb 575.3kb
yellow open current_weather-2018.07.19 Xm-WY_9oTQOP1gXk9k3T9Q 5 1 1 0 460b 460b
```
I'm guessing this has something to do with the index configuration option. How do I make sure that there is only one index pattern for logstash that keeps pulling live weather. Also, how do I configure http\_poller plugin to ensure it keeps pulling data every second? Would really appreciate if you could help me. Thanks in advance!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 19, 2018, 7:08pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/2 "2018-07-19T19:08:41Z")
Remove the -%{+YYYY.MM.dd} from the [index](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index) option on your elasticsearch output.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 19, 2018, 7:14pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/3 "2018-07-19T19:14:26Z")
> [@APJ](#):
>
> Also, how do I configure http\_poller plugin to ensure it keeps pulling data every second?
You mean you want it to fetch the current weather once a second? I don't think the OpenWeather folks will be happy if you do that.
---
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [July 19, 2018, 7:17pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/4 "2018-07-19T19:17:06Z")
It is possible to do it. I read the documentation, and if I am not wrong, I can make 60 requests per minute.
---
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [July 19, 2018, 7:20pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/5 "2018-07-19T19:20:52Z")
Thanks! 🙂
Also, if I am not wrong about the API calling, how do I schedule the cron to make a request every second?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [July 19, 2018, 7:32pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/6 "2018-07-19T19:32:10Z")
> [@APJ](#):
>
> how do I schedule the cron to make a request every second?
Remove the /2 from the schedule option in your input.
---
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [August 2, 2018, 4:03am UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/7 "2018-08-02T04:03:26Z")
Thanks a lot! Also, Is there a way to alter the value of live API data polled using http\_poller input plugin? I want to modify the data retrieved from openweathermap every 30 seconds before sending it into elasticsearch. Is there a way to do it?
I would really appreciate if someone could help me out. Thanks!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 2, 2018, 1:28pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/8 "2018-08-02T13:28:55Z")
Yes. logstash has a wide variety of [filter plugins](https://www.elastic.co/guide/en/logstash/current/filter-plugins.html) that can manipulate the data, including a ruby plugin that can do pretty much anything.
What does the data look like (show us the output from output { stdout { codec =\> rubydebug } } or copy and paste a document from the JSON tab in Kibana Discover) and explain exactly what you want to change in it.
---
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [August 2, 2018, 6:35pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/9 "2018-08-02T18:35:20Z")
Hi Badger,
Here is a JSON response being printed onto the console.
```
{
"visibility" => 16093,
"@timestamp" => 2018-08-02T18:08:40.310Z,
"name" => "Santa Clara",
"wind" => {
"deg" => 290,
"speed" => 3.1
},
"clouds" => {
"all" => 1
},
"main" => {
"temp_min" => 30,
"humidity" => 25,
"temp_max" => 30,
"temp" => 30,
"pressure" => 1024
},
"@version" => "1",
"weather" => {
"id" => 800,
"main" => "Clear",
"description" => "clear sky",
"icon" => "01d"
}
}
```
Here is the JSON document from the JSON tab in the Kibana output
```
{
"_index": "weather",
"_type": "doc",
"_id": "wsDX-2QBol0Vk4cfQdI2",
"_version": 1,
"_score": null,
"_source": {
"visibility": 16093,
"@timestamp": "2018-08-02T18:12:25.421Z",
"name": "Santa Clara",
"wind": {
"deg": 290,
"speed": 3.1
},
"clouds": {
"all": 1
},
"main": {
"temp_min": 30,
"humidity": 25,
"temp_max": 30,
"temp": 30,
"pressure": 1024
},
"@version": "1",
"weather": {
"id": 800,
"main": "Clear",
"description": "clear sky",
"icon": "01d"
}
},
"fields": {
"@timestamp": [
"2018-08-02T18:12:25.421Z"
]
},
"sort": [
1533233545421
]
}
```
And this is the logtstash config file I am currently using,
**weather\_data.conf**
```
input {
http_poller {
urls => {
weather => {
url => "http://api.openweathermap.org/data/2.5/weather?id=5490223&appid="MY_APP_ID"&units=metric"
headers => {
Accept => "application/json"
}
}
}
schedule => { cron => "* * * * * *" }
codec => json
}
}
filter {
mutate {
remove_field => ["@version" ,"command" ,"host" ,"cod" ,"id" ,"base" ,"coord" ,"sys" ,"dt"]
}
split { field => "weather" }
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "weather"
}
stdout {
codec => rubydebug
}
}
```
My goal is to perform addition operation on `main.temp` value (main.temp = main.temp + 10) every 30 seconds, while at the same time piping data into logstash every second.
I am clueless about how this can be achieved. I'd be really grateful if you could let me know what needs to be added or changed.
Thanks!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 2, 2018, 6:57pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/10 "2018-08-02T18:57:55Z")
A strange looking ask. But it could be done...
```
ruby { code => ' if Time.now.to_i % 30 == 0 ; event.set("[main][temp]", 10 + event.get("[main][temp]")); end' }
```
---
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [August 2, 2018, 7:18pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/11 "2018-08-02T19:18:16Z")
Thanks a lot Badger! 🙂 It works! The purpose of this is to simulate an environment where in if the temperature exceeds a certain threshold, a email notification is sent to the user. Since, the temperature is so constant over a small period of time, it is hard to capture drastic changes.
With that said, if I wanted it to be changed every 5 mins instead of every 30s, is the below configuration correct?
`ruby { code => ' if Time.now.to_i % 300 == 0 ; event.set("[main][temp]", 10 + event.get("[main][temp]")); end' }`
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 2, 2018, 7:19pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/12 "2018-08-02T19:19:18Z")
Yes, that is correct.
---
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [August 2, 2018, 7:36pm UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/13 "2018-08-02T19:36:37Z")
Thanks! 🙂
---
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [August 3, 2018, 12:45am UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/14 "2018-08-03T00:45:54Z")
Hey Badger,
I created a node.js server to so that I could fetch latest JSON document present in the index and send an email if the weather exceeds 40℉. But unfortunately, I cannot do that because I would need the document id in order to fetch the JSON document. So, I figured I could maybe send email directly from logstash instead.
How do I configure my logstash configuration file to send an email whenever the temperature exceeds 40℉? Could you please help me with this? I really am unable to figure out another way to do this.
Thanks in advance again!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 3, 2018, 1:10am UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/15 "2018-08-03T01:10:41Z")
I am not convinced you need to know the document id in advance to fetch the most recent document, since that is exactly what the Kibana Discover tab does. But that would be a question for the elasticsearch category.
You can conditionalize an output based on the fields. So something like
```
output {
if [main][temp] >= 40 {
email {
[...]
}
}
}
```
I have never used the email output plugin, but there is [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-email.html), and if you have a problem you can post another question and someone else might reply.
---
### Author:  [@APJ](https://discuss.elastic.co/u/APJ)
#### Post date: [August 3, 2018, 4:04am UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/16 "2018-08-03T04:04:17Z")
Thanks Badger,
I have created a new topic [https://discuss.elastic.co/t/logstash-error-while-using-email-plugin/142840](https://discuss.elastic.co/t/logstash-error-while-using-email-plugin/142840). I would appreciate if you could take a look at it and let me know what wrong I am doing.
Thanks!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 31, 2018, 4:04am UTC](https://discuss.elastic.co/t/logstash-creates-a-new-index-every-24-hours/140793/17 "2018-08-31T04:04:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.