I am having a problem writing a correct .conf file for Logstash to ingest into ElasticSearch. Currently I have a "Conversational_history.log" file from a Chatbot application that I have linked up to Filebeat, and from Filebeat I am running Logstash to index these logs into ElasticSearch. I have been able to index these logs into ElasticSearch, but my problem is I only want one specific field called "message". Currently "message" has all the relevant information inside of it ("@timestamp, user_input, response" etc.). I feel as if I need to do this through a filter in my .conf file when running logstash. Currently My output in Kibana is
""hits": {
"total": 22,
"max_score": 1,
"hits": [
{
"_index": "chatbot_new",
"_type": "doc",
"_id": "EuDknGYBtEtWC4p_MH-o",
"_score": 1,
"_source": {
"source": "/Users/drewmahoney/Desktop/Chatbot/DevelopBranchBackend/dexi_app/logs/conversation_history.log",
"prospector": {
"type": "log"
},
"@version": "1",
"host": {
"name": "c02wj1r1htdg.users.bah.com"
},
"input": {
"type": "log"
},
"beat": {
"hostname": "c02wj1r1htdg.users.bah.com",
"name": "c02wj1r1htdg.users.bah.com",
"version": "6.3.2"
},
"offset": 612,
"message": """{"timestamp": "2018-09-17T16:41:23.723028Z", "level": "INFO", "name": "rasa_model", " user_input - response": null, "user_input": "hello", "response": [{"recipient_id": "default", "text": "Hello, my name is DEXi. How can I help you?"}]}""",
"tags": [
"beats_input_codec_plain_applied",
"_grokparsefailure"
],
"@timestamp": "2018-10-22T17:47:39.819Z"
}
}......
"
But I want to only index the "message" field and parse the data out so that after each comma that is it's only field in Kibana.
My .log file is structured like so...
"{"timestamp": "2018-09-18T17:00:29.644572Z", "level": "INFO", "name": "rasa_model", " user_input - response": null, "user_input": "string", "response": [{"recipient_id": "default", "text": "Hello, my name is DEXi. How can I help you?"}]}
{"timestamp": "2018-09-18T17:00:31.541068Z", "level": "INFO", "name": "rasa_model", " user_input - response": null, "user_input": "string", "response": [{"recipient_id": "default", "text": "Hello, my name is DEXi. How can I help you?"}]}
{"timestamp": "2018-09-19T14:25:01.543610Z", "level": "INFO", "name": "rasa_model", " user_input - response": null, "user_input": "undefined", "response": [{"recipient_id": "default", "text": "Hello, my name is DEXi. How can I help you?"}]}"
and my .conf file when I run Logstash is...
"input {
#stdin {}
beats {
port => "5044"
}
}
The filter part of this file is commented out to indicate that it is
optional.
filter {
grok {
patterns_dir => ["./elasticsearch-6.3.2/datasets/Chatbot_logs/patterns/Regexp.txt"]
match => { "message" => "%{TIMESTAMP:@timestamp} %{LEVEL:level} %{NAME:name} %{USER_INPUT:user_input} %{RESPONSE:response}" }
}
}
output {
elasticsearch {
hosts => [ "localhost:9200" ]
action => "index"
index => "chatbot_new"
document_type => "doc"
}
}"
and I have also created a patterns dir because I could not find any Macros for my log....
"TIMESTAMP (?<="timestamp": )"."(?=, "level")
LEVEL (?<="level": )"."(?=, "name")
NAME (?<="name": )"."(?=, " user_input - response")
USER_INPUT (?<="user_input": )"."(?=, "response")
RESPONSE (?<="response": ).*(?=})"
I think my problem is in the "filter" part of my .conf file.