Logstash Index

Elastic Stack Logstash
1

Hi guys,

I have a problem with the logstash output, it work fine when I didn't put index, but when I put my custom index name then it won't store in elasticsearch.

elasticsearch {
        hosts => ["localhost:9200"]
        index => "%{[@metadata][beat]}-%{index_day}}"
}

I tried the default one also not working

elasticsearch {
        hosts => ["localhost:9200"]
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}}"
}

It only work with

elasticsearch {
        hosts => ["localhost:9200"]
}

Logstash status:

[root@sla-bvt-elk-sjc01 conf.d]# systemctl status logstash -l
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-12-12 20:33:35 CST; 5min ago
 Main PID: 14980 (java)
   Memory: 482.7M
   CGroup: /system.slice/logstash.service
           └─14980 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash

Dec 12 20:39:18 sla-bvt-elk-sjc01.sdad.sl.dst.ibm.com logstash[14980]: "index_day" => "2017.12.13",
Dec 12 20:39:18 sla-bvt-elk-sjc01.sdad.sl.dst.ibm.com logstash[14980]: "Timestamp" => "2017-12-13T10:39:13.741+08:00",
Dec 12 20:39:18 sla-bvt-elk-sjc01.sdad.sl.dst.ibm.com logstash[14980]: "Testcase_Name" => "default",
Dec 12 20:39:18 sla-bvt-elk-sjc01.sdad.sl.dst.ibm.com logstash[14980]: "@timestamp" => 2017-12-13T02:39:17.927Z,
Dec 12 20:39:18 sla-bvt-elk-sjc01.sdad.sl.dst.ibm.com logstash[14980]: "Policy_Name" => "policy_linux_entire_dir_perms",
Dec 12 20:39:18 sla-bvt-elk-sjc01.sdad.sl.dst.ibm.com logstash[14980]: "Log_Message" => "-----> Testing <default-REDHAT73--9-30-80-104>\n",
[root@sla-bvt-elk-sjc01 conf.d]# \
2

Have you looked in the Logstash log for clues? Bumping the log level might provide additional clues if needed.

3

Hi, thanks for replying.

I just noticed this thread [ATTENTION] Logstash 6.0.0 - Known Issue Indexing to Elasticsearch 6.0

After I adding the document type, now it can work but the result still not what I am expecting

[2017-12-13T10:17:54,533][INFO ][o.e.c.m.MetaDataMappingService] [6p1K6V1] [%{[@metadata][beat]}-2017.12.14/iMGnWDl4QdCsN-jbdWXezg] create_mapping [doc]
[2017-12-13T10:17:58,651][INFO ][o.e.c.m.MetaDataMappingService] [6p1K6V1] [%{[@metadata][beat]}-2017.12.14/iMGnWDl4QdCsN-jbdWXezg] update_mapping [doc]
[2017-12-13T10:18:05,290][INFO ][o.e.c.m.MetaDataMappingService] [6p1K6V1] [%{[@metadata][beat]}-2017.12.14/iMGnWDl4QdCsN-jbdWXezg] update_mapping [doc]
[2017-12-13T10:19:36,614][INFO ][o.e.c.m.MetaDataMappingService] [6p1K6V1] [%{[@metadata][beat]}-2017.12.14/iMGnWDl4QdCsN-jbdWXezg] update_mapping [doc]

Does the double quote or single quote affect the result?

elasticsearch { 
     hosts => ["localhost:9200"] 
     index => "%{[@metadata][beat]}-%{index_day}"
     document_type => doc
}
4

Does the double quote or single quote affect the result?

No. The evidence suggests that you don't actually have a [@metadata][beat] field in your events.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.