Logstash not creating new index

Elastic Stack Logstash
1

I am having an issue with multiple tcp inputs and logstash not pulling in messages from the second tcp input.

The data received on port 6050 is going into the default index logstash-%{+YYYY.MM.dd} but the data sent from rsyslog to port 6052 is not being recorded at all.

I gave the logstash user temp superuser permissions to rule out permissions and still no luck.

Here is what I have for config files (edited to remove sensitive data and not applicable data):
rsyslog.conf

:fromhost-ip, isequal, "X.X.X.X" @@localhost:6052;json-template
& stop
*.* @@localhost:6050;json-template

logstash configs
01input.config

input {
  tcp {
    host => "localhost"
    port => 6050
    codec => "json"
    tags => "rsyslog"
  }
  tcp {
    host => "localhost"
    port => 6052
    codec => "json"
    type => "debug"
    tags => "cubes"
  }
  }
}

#filter {  }
#output {  }

99output.conf

output {
  if [tags] == "cubes" {
    elasticsearch { hosts => [ "10.30.97.248:9200" ]
    user => ["logstash"]
    password => ["password"]
    index => ["cubes-%{+YYYY.MM.dd}"]
  }
    } else {
    elasticsearch { hosts => [ "10.30.97.248:9200" ] 
    user => ["logstash"]
    password => ["password"]
    }
  }
}

When I perform a GET _cat/indices none of the cubes-%{+YYYY.MM.dd} indices are even being created.

2

If ILM is enabled, and it is on by default for recent versions, then the index option is silently ignored. Personally I think that is a bad idea, but I don't get a vote on it.

3

Im fairly new to ELK. How do I disable that?
And if I disable that will it stop the index lifecycle policies?

4

You can use the ilm_enabled option to disable it.

5

I set that setting to FALSE but no change. It is still not creating the new index and putting data anywhere.

6

I added the following code and now it is creating the index, but no data is being loaded into the new index

 ilm_enabled => true
    ilm_pattern => "{now/d}"
    ilm_rollover_alias => "cubes"
7

Try if "cubes" in [tags]

8

I did finally get this working. to document the fix:
Insead of stating a second tcp input and using rsyslog to send those messages to the second input to get the tag, I just created a new filter with a mutate to add the tag. Once that tag was there I was able to use the if "cubes" in [tags] to add it to the new Index.

Thank you for the help!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.