Did an upgrade to my single-node cluster yesterday evening, now up on 7.17.8 and everything's basically working, except that it seems that Logstash is no longer creating logstash-<YYYY.MM.DD> indexes... I see there is now a logstash index, which was created around the time of the upgrade. However, I do not see the document count increasing in it... How can I determine what index is currently being used by Logstash, and how can I configure the system to create logstash-<YYYY.MM.DD> indexes as it did before?
I did figure out how to see what index is being used (unsurprisingly, the logstash one) but I still need some guidance on how to revert to the prior behavior of one index per day. Could someone who knows please assist?
(For those "RTFM" fans out there, I am reading thru "Learning Elastic Stack 7.0 - Second Edition" by Pranav Shukla and Sharath Kumar M N. But I'd like to fix this issue faster than I can read such a big book )
I don't know what that means, meaning you're not shipping anything... Or it's not working?
Are you using filebeat?
The more specific you are, the better we can help...
Seems you want daily index which is fine. Of course, now there is ILM which is a more sophisticated way to handle time series data... But daily indices are fine as well..
And we'd really want to see your filebeat configuration as well...
Much has changed since 6.8...
We can get you working I'm sure, but you will need to kind of understand the concepts. And now you're just getting to 7.x and we're already nearly a year into 8.x which is a significant change from 7.x
Not shipping anything. Not using Filebeat on any servers. Sorry I don't quite know the lingo yet.
All I'm doing right now is sending syslog messages to Logstash via the 5514 port listeners.
I think I want daily indexes, but I see they are still all open...
green open logstash pWNKLUU8TGG5jeGqZjnfdw 1 0 1655812 0 619.3mb 619.3mb
green open logstash-2022.01.02 5FOfSP1wSZ-va2oTbcPalw 5 0 1238 0 736.3kb 736.3kb
green open logstash-2022.01.03 HPq9v_aEShaJ_NWNLOC-ZA 5 0 31353 0 9.2mb 9.2mb
green open logstash-2022.01.04 1HuzxNy2RRi0QmSSN56u8Q 5 0 1344 0 768kb 768kb
green open logstash-2022.01.05 abIhWjzDTFupI__oiCscew 5 0 1296 0 655.3kb 655.3kb
green open logstash-2022.12.29 o17hILWoSo2xjTTJDSiTpw 5 0 594214 0 205.5mb 205.5mb
green open logstash-2022.12.30 i_SFdwF1SmqhYc4xJ_U8jw 5 0 593818 0 205.4mb 205.4mb
green open logstash-2022.12.31 I5FXRhrrTqCAU1wOQifdBg 5 0 595241 0 206.2mb 206.2mb
So is it better (more idiomatic) to use a single index, and use ILM to roll it / etc ?
(As you can guess, I'm a lightweight user of Elastic, wanted to learn it but have had limited time to devote to this; as well, it is "non-production", just for my analytics use. I started out with 5.x on this single server, upgraded to 6.x, now upgraded to 7.x; I'm probably going to roll a whole new ELK 8.x server at some point this year, and will try to migrate my data over to it. The official log analytics solution where I work is S*****, although we have a very limited license. Having an ELK box is good when I have a large quantity of log data to analyze, and don't want to worry about exceeding a license )
Daily Indices vs ILM You should read a bit about that, make sure you read the right 7.17.x docs there are some material changes in 8.x ... Data streams are available in 7.17.x and are the norm in 8.x
ILM with write alias and rollover has more flexibility/efficiency and automated capability over time
Daily is simpler and can also use ILM, but the daily part is fixed.
Daily is fine... Why do you want daily? Easy to understand, cleanup, segment etc... although often not as efficient from a resource perspective, but for a test cluster it is fine.
we don't really close indices much anymore... heap has gotten much more efficient... you still can if you want. Not sure why you would expect them to be closed... ILM does not close indices either.
The default format output in logstash 7.17.x should be
lines => [
"The quick brown fox did a lot",
"Live Long and Prosper",
"May the Shwartz Be With You",
"With Great Power Comes Lots of Headaches"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2023.01.07-000001 a0tRZqxtSUGPpdozd8NF1Q 1 1 666376 0 22.8mb 22.8mb