After 6.8 -> 7.17 upgrade, logstash not sending data to logstash-* indexes

willdennis · January 4, 2023, 3:17pm

Hi all,

Did an upgrade to my single-node cluster yesterday evening, now up on 7.17.8 and everything's basically working, except that it seems that Logstash is no longer creating logstash-<YYYY.MM.DD> indexes... I see there is now a logstash index, which was created around the time of the upgrade. However, I do not see the document count increasing in it... How can I determine what index is currently being used by Logstash, and how can I configure the system to create logstash-<YYYY.MM.DD> indexes as it did before?

willdennis · January 6, 2023, 10:57pm

I did figure out how to see what index is being used (unsurprisingly, the logstash one) but I still need some guidance on how to revert to the prior behavior of one index per day. Could someone who knows please assist?

(For those "RTFM" fans out there, I am reading thru "Learning Elastic Stack 7.0 - Second Edition" by Pranav Shukla and Sharath Kumar M N. But I'd like to fix this issue faster than I can read such a big book )

stephenb · January 6, 2023, 11:00pm

Hi @willdennis

Perhaps share your complete logstash config... probably the easiest way for us to help...

Also, are you using beats upstream?

willdennis · January 6, 2023, 11:20pm

Thanks for replying @stephenb

My Logstash config is as follows:

# cat /etc/logstash/conf.d/central.conf | grep -v -e ^# -e ^$
input {
  tcp {
    port => 5514
    type => syslog
  }
  udp {
    port => 5514
    type => syslog
  }
  beats {
    port => 5044
  }
}
filter {
  mutate {
    add_field => { "raw_input" => "%{message}" }
  }
  if  "APC_PDU_LEGAMPS"  in [message] {
    mutate {
      gsub => [
        "message", "[=]", " \0 ",
        "message", "\s+", " "
      ]
    }
    grok {
      match => { "message" => "(?<logTime>%{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}) envctl APC_PDU_LEGAMPS: \[INFO\] PDU = %{NOTSPACE:PDUName} Leg = %{NUMBER:LegNum:int} Amps = %{NUMBER:Amps:float}" }
    }
    date {
      match => [ "logTime", "MMM dd HH:mm:ss" ]
      target => "@timestamp"
    }
    mutate {
      remove_field => ['year']
      remove_field => ['month']
      remove_field => ['day']
      remove_field => ['hour']
      remove_field => ['minute']
      remove_field => ['second']
      remove_field => ['logTime']
    }

  }
  else if  "GET_GPU_UTIL"  in [message] {
    mutate {
      gsub => [
        "message", "[=]", " \0 ",
        "message", "\s+", " "
      ]
    }
    grok {
      match => { "message" => "(?<logTime>%{MONTH:month} %{MONTHDAY:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}) %{NOTSPACE:hostname} %{NOTSPACE:program}: \[INFO\] Hostname = %{NOTSPACE:GpuServer} GPU_Inst = %{NUMBER:GpuInstNum:int} GPU_Util = %{NUMBER:GpuPctUtil:int}" }
    }
    date {
      match => [ "logTime", "MMM dd HH:mm:ss" ]
      target => "@timestamp"
    }
    mutate {
      remove_field => ['year']
      remove_field => ['month']
      remove_field => ['day']
      remove_field => ['hour']
      remove_field => ['minute']
      remove_field => ['second']
      remove_field => ['logTime']
    }

  }
  else if [type] == "syslog" {
    mutate {
        remove_field => ['message']
    }
    grok {
        "match" => { "raw_input" => "%{SYSLOGLINE}" }
    }
    date {
        match => [ "timestamp", "MMM dd HH:mm:ss", "MMM  dd HH:mm:ss" ]
        target => "@timestamp"
    }
    mutate {
      remove_field => ['timestamp']
    }
  }

}
output {
    elasticsearch { }
}

As you can see, Beats input is configured, but nothing is sending using Beats atm...

willdennis · January 6, 2023, 11:22pm

Oh, and if you meant wanting to see my logstash.yml file --

# cat /etc/logstash/logstash.yml | grep -v -e ^# -e ^$
path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
path.logs: /var/log/logstash

stephenb · January 6, 2023, 11:27pm

This is the section I was most interested in....

And yes, beats input has some impact...

I don't know what that means, meaning you're not shipping anything... Or it's not working?

Are you using filebeat?

The more specific you are, the better we can help...

Seems you want daily index which is fine. Of course, now there is ILM which is a more sophisticated way to handle time series data... But daily indices are fine as well..

And we'd really want to see your filebeat configuration as well...

Much has changed since 6.8...

We can get you working I'm sure, but you will need to kind of understand the concepts. And now you're just getting to 7.x and we're already nearly a year into 8.x which is a significant change from 7.x

willdennis · January 7, 2023, 3:32am

Not shipping anything. Not using Filebeat on any servers. Sorry I don't quite know the lingo yet.

All I'm doing right now is sending syslog messages to Logstash via the 5514 port listeners.

I think I want daily indexes, but I see they are still all open...

green open logstash                          pWNKLUU8TGG5jeGqZjnfdw 1 0   1655812       0 619.3mb 619.3mb
green open logstash-2022.01.02               5FOfSP1wSZ-va2oTbcPalw 5 0      1238       0 736.3kb 736.3kb
green open logstash-2022.01.03               HPq9v_aEShaJ_NWNLOC-ZA 5 0     31353       0   9.2mb   9.2mb
green open logstash-2022.01.04               1HuzxNy2RRi0QmSSN56u8Q 5 0      1344       0   768kb   768kb
green open logstash-2022.01.05               abIhWjzDTFupI__oiCscew 5 0      1296       0 655.3kb 655.3kb
[...]
green open logstash-2022.12.29               o17hILWoSo2xjTTJDSiTpw 5 0    594214       0 205.5mb 205.5mb
green open logstash-2022.12.30               i_SFdwF1SmqhYc4xJ_U8jw 5 0    593818       0 205.4mb 205.4mb
green open logstash-2022.12.31               I5FXRhrrTqCAU1wOQifdBg 5 0    595241       0 206.2mb 206.2mb

So is it better (more idiomatic) to use a single index, and use ILM to roll it / etc ?

(As you can guess, I'm a lightweight user of Elastic, wanted to learn it but have had limited time to devote to this; as well, it is "non-production", just for my analytics use. I started out with 5.x on this single server, upgraded to 6.x, now upgraded to 7.x; I'm probably going to roll a whole new ELK 8.x server at some point this year, and will try to migrate my data over to it. The official log analytics solution where I work is S*****, although we have a very limited license. Having an ELK box is good when I have a large quantity of log data to analyze, and don't want to worry about exceeding a license )

stephenb · January 7, 2023, 4:45pm

Daily Indices vs ILM You should read a bit about that, make sure you read the right 7.17.x docs there are some material changes in 8.x ... Data streams are available in 7.17.x and are the norm in 8.x

ILM with write alias and rollover has more flexibility/efficiency and automated capability over time
Daily is simpler and can also use ILM, but the daily part is fixed.

Daily is fine... Why do you want daily? Easy to understand, cleanup, segment etc... although often not as efficient from a resource perspective, but for a test cluster it is fine.

we don't really close indices much anymore... heap has gotten much more efficient... you still can if you want. Not sure why you would expect them to be closed... ILM does not close indices either.

The default format output in logstash 7.17.x should be

input {
  generator {
    lines => [
          "The quick brown fox did a lot",
          "Live Long and Prosper",
          "May the Shwartz Be With You",
          "With Great Power Comes Lots of Headaches"
        ]
  }
}

output {
  elasticsearch {}
}

curl localhost:9200/_cat/indices?v
health status index                                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   logstash-2023.01.07-000001                 a0tRZqxtSUGPpdozd8NF1Q   1   1     666376            0     22.8mb         22.8mb

system · February 4, 2023, 4:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.