Index not created by Logstash

[Albert (Albert J Kettner)]

Hi all,

I'm following this blog (https://www.elastic.co/blog/monitoring-the-search-queries) to capture search queries of Elastic Search that is installed on a MediaWiki site with a CirrusSearch extension using the following versions:

• logstash 5.6.10
• packetbeat version 5.6.10 (amd64), libbeat 5.6.10
• elasticsearch Version: 5.6.10, Build: b727a60/2018-06-06T15:48:34.860Z, JVM: 1.8.0_141
• kibana 5.6.10

All is running on the same server (linux redhat).

As logstash conf file, I'm using the same as indicated by the blog:
</>
input {
beats {
port => 5044
}
}
filter {
if "search" in [request]{
grok {
match => { "request" => ".\n{(?<query_body>.)"}
}
grok {
match => { "path" => "/(?.*)/_search"}
}
if [index] {
} else {
mutate {
add_field => { "index" => "All" }
}
}
mutate {
update => { "query_body" => "{%{query_body}" }
}
}
}
output {
if "search" in [request] and "ignore_unmapped" not in [query_body]{
elasticsearch {
hosts => ["http://localhost:9200"]
#index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
}
</>

To the elasticsearch.yml file the following line is added:
action.auto_create_index: logstash-*

However, no logstash-* indices are generated when searching on the wiki.
E.g. curl localhost:9200/_cat/indices?v
</>
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open csdms_wiki-mw__general_first S8fUiAoKRpKjh4lA163hJQ 4 0 5202 1 19.5mb 19.5mb
green open csdms_wiki-mw__content_first 81VvhnP2SE2hzYFtgqn0DQ 4 0 11339 1908 125.5mb 125.5mb
green open mw_cirrus_metastore_first EGfxsk15Qp61eLw4ydIr_A 1 0 3 2 6.2kb 6.2kb
yellow open logstash-x mdA01kCNTUilnRYi2PisAg 5 1 1 0 4.6kb 4.6kb
yellow open .kibana D9w9nJ6ORf6y4kwAzdGSwg 1 1 3 0 22.3kb 22.3kb
</>

(The "logstash-x" was created by hand as a testcase). Any idea what I'm doing wrong, why no logstash-* are generated?

Thank you!,
Albert

[magnusbaeck (Magnus Bäck)]

Have you looked in the logs for Logstash and Packetbeat for clues?

[Albert (Albert J Kettner)]

Thank you Magnus,

Yes, logs don't provide any error. What I did figure out is that the provided logstash conf file given in the blog doesn't work in my case. Specifically the "if" statement in the output part:

</>
if "search" in [request] and "ignore_unmapped" not in [query_body]{
</>

but also the filter needs adjustments.

The if statement in the output part makes that no logstash-* index is generated (so no output is generated). Probably because the [request] field is not sufficient. Maybe this should be replaced for me by [http][request][params]. I'm not familiar with logstash conf files so this will be trial and error. But it looks like my problem is solved. Indices are created by Logstash when removing filter and if statement (now generating tons of data, so have to turn it off).

Thanks,
Albert.