LogstashからElasticsearch(Ingest node)の連携について、ご質問させて頂きます。
実現したいこと
・Logstashでtanslateを実施(前段階としてmutateのsplit,add_fieldを使い特定の値をfieldにセット)
・Elasticsearch Ingest nodeにて全体値をGrok
Logstash config
input {
beats {
port => XXXX
}
}
filter {
mutate {
split => ["message", " "]
add_field => { "client_address" => "%{[message][2]}" }
}
}
translate {
field => "client_address"
destination => "blacklist"
exact => true
regex => true
dictionary_path => "/var/tmp/data/translate.yml"
fallback => "2"
}
output {
elasticsearch{
hosts => [ "XX.XXX.XXX.XXX:XXXX" ]
user => "XXXXX"
password => "XXXXX"
pipeline => "squid_translate"
}
}
Elasticsearch Ingest node
PUT /_ingest/pipeline/squid_translate
{
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{NUMBER:timestamp}%{SPACE}%{NUMBER:duration} %{WORD:cache_result}/{POSINT:status_code} %{WORD:request_method} %{NOTSPACE:url} %{NOTSPACE:user} %{WORD:hierarchy_code}/{NOTSPACE:server} %{NOTSPACE:content_type}"
]
}
}
]
}
上記を元に処理を実行すると、Elasticsearchにて以下のエラーが発生します。
spritにてmessageがarraylist型になったが、Ingest nodeのGrokがString型にて受け取ろうとしたため、型違いでエラーになったようです。
[2019-09-25T11:30:35,562][DEBUG][o.e.a.b.TransportBulkAction] failed to execute pipeline [squid_translate] for document [squid_translate/_doc/null]
org.elasticsearch.ElasticsearchException: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [message] of type [java.util.ArrayList] cannot be cast to [java.lang.String]
at org.elasticsearch.ingest.CompoundProcessor.newCompoundProcessorException(CompoundProcessor.java:194) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:133) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.ingest.Pipeline.execute(Pipeline.java:100) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.ingest.IngestService.innerExecute(IngestService.java:427) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.ingest.IngestService.access$100(IngestService.java:70) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.ingest.IngestService$3.doRun(IngestService.java:355) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) [elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.3.2.jar:7.3.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: java.lang.IllegalArgumentException: java.lang.IllegalArgumentException: field [message] of type [java.util.ArrayList] cannot be cast to [java.lang.String]
... 11 more
Caused by: java.lang.IllegalArgumentException: field [message] of type [java.util.ArrayList] cannot be cast to [java.lang.String]
at org.elasticsearch.ingest.IngestDocument.cast(IngestDocument.java:550) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.ingest.IngestDocument.getFieldValue(IngestDocument.java:116) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.ingest.IngestDocument.getFieldValue(IngestDocument.java:131) ~[elasticsearch-7.3.2.jar:7.3.2]
at org.elasticsearch.ingest.common.GrokProcessor.execute(GrokProcessor.java:58) ~[?:?]
at org.elasticsearch.ingest.CompoundProcessor.execute(CompoundProcessor.java:123) ~[elasticsearch-7.3.2.jar:7.3.2]
... 9 more
ご質問
上記エラーを解決する方法が分かりましたら教えてください。