I am trying to use curl to to log something, just something, to an index, any index, on a ELK 8.4.1 cluster.
On my Logstash node I have the following as part of the configuration in the conf.d directory:
input {
beats {
port => 9093
ssl => true
(other SSL stuff)
id => "PM-DefaultBeatsInput-Tcp9093"
}
http {
port => 9094
ssl => true
codec => json
(more SSL stuff)
id => "PM-DefaultHTTPJsonInput-Tcp9094"
tags => ["http_input_9094"]
}
}
The beat plugin works because I can append a little bit of JSON data to the log file at one of the Logstash clients that is running Filebeat and it appears as expected in Kibana's Discover module. The JSON I am using looks like this:
{
"timestamp": "2023-03-17 19:07:27.793 +01:00",
"level": "debug",
"msg": "Testing 123"
}
My next step was to try to use curl to PUT the same same JSON data directly into Logstash using the HTTP plugin. Something like:
$ curl --header 'Content-Type: application/json' --request PUT --data @data.json https://<name-of-logstash-node>:9094/logstash-mmost
Curl reports that all is well so Logstash accepts the JSON data, but then Logstash fails and adds the following to its log-file (logstash-plain.log):
[2023-03-17T19:14:29,889][WARN ][logstash.outputs.elasticsearch][main][8e0517f6b8e059245bd5e087b80dd7c149f513aa2b905007cc1eaf520985273c] Could not index event to Elasticsearch. status: 400, action: ["index", {:_id=>nil, :_index=>"logstash-unknown", :routing=>nil}, {"@timestamp"=>2023-03-17T18:14:29.831634833Z, "headers"=>{"http_host"=>"mgxlostapp01.mgx.ppm.nu:9094", "http_user_agent"=>"curl/7.29.0", "content_length"=>"90", "request_path"=>"/logstash-mmost", "content_type"=>"application/json", "request_method"=>"PUT", "http_version"=>"HTTP/1.1", "http_accept"=>"*/*"}, "level"=>"DEBUG", "timestamp"=>"2023-03-17T18:14:29.691+00:00", "host"=>"10.252.72.15", "msg"=>"Testing 123", "@version"=>"1", "tags"=>["http_input_9094", "Ingested via logstash on host mgxlostapp01.mgx.ppm.nu", "Ruleset unknown"]}], response: {"index"=>{"_index"=>"logstash-unknown-2023.02.27-000029", "_id"=>"ZEHH8IYBP3XCgXYtxj7-", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [host] tried to parse field [host] as object, but found a concrete value"}}}
What am I missing here? Obviously something about a host field, but what?
I have tried adding various forms of
"fields:" {
"host": "a-host-name",
"host.name": "a-host-name",
"host.name.keyword": "a-host-name",
"host": {
"name": "a-host-name",
"ip": "127.0.0.1"
}
},
"timestamp": ...........
"level": .........
"msg": .......
}
but I keep getting the same error. There is something fundamentally that I am not understanding.
Can anyone come up with the magic formula for using curl to PUT a message into an index? Any index would do for a start.
}