I want Message values as feild using logstash

Elastic Stack Logstash
1

Hello Every One

Following is my message I want to split message value and add as field for filter and visulize
how can I achive that please help any one

message

url=&type=click&date_time=2013-01-01+12:00:00&initiated_by=admin&initiated_from=admin&list=1&campaign[id]=98&contact[id]=42&contact[email]=test@test.com&contact[first_name]=First&contact[last_name]=Last&contact[ip]=127.0.0.1&contact[fields][39]=custom+field+value&link[id]=754&link[url]=http://example.com

This is Logstash Config

input {
http {
host => "0.0.0.0"
port => 0000

}
}

filter {
  urldecode {
    all_fields => true

}
}

output {
Elasticsearch {
hosts => [ 'https://x.x.x.x:9200' ]
user => 'user'
password => 'pass'
ssl => "true"
ssl_certificate_verification => "false"
index => "_test"

}
stdout {
codec => "json"
}
}

2

Please do not post pictures of text, they are hard to read, impossible to search, and cannot be used to reproduce the issue (and find a solution).

3

Thank you for prompt reply @Badger

Actually this picture is related to my message output is there any solution for the same? Are got my point? and edited post with text please suggest if you have a any solution

4

urldecode is used to undo url encoding, like replacing %20 with a space. It sounds like you want a kv filter. You could try

    mutate { gsub => [ "message", "\]", "}", "message", "\[", "{" ] }
    kv { field_split => "&" allow_empty_values => true }

You cannot have a field called "link[url]". That will result in an Invalid FieldReference exception.

5

@Badger
This is current Configuration

input {
http {
host => "0.0.0.0" #
port => 0000

}
}

filter {
   mutate { gsub => [ "message", "\]", "}", "message", "\[", "{" ] }
   kv { field_split => "&" allow_empty_values => true }

}

output {
Elasticsearch {
hosts => [ 'https://es:9200' ]
user => 'user'
password => 'pass'
ssl => "true"
ssl_certificate_verification => "false"
index => "crm_webhook_test"

}
stdout {
codec => "json"
}
}

Getting following error

2022-04-28T17:53:08,904][INFO ][logstash.runner ] Logstash shut down.
[2022-04-28T17:53:23,351][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2022-04-28T17:53:25,077][ERROR][logstash.filters.kv ] Unknown setting 'allow_empty_values' for kv
[2022-04-28T17:53:25,085][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:86:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:126:in initialize'", "org/logstash/plugins/PluginFactoryExt.java:78:in filter_delegator'", "org/logstash/plugins/PluginFactoryExt.java:248:in plugin'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in block in converge_state'"]}
[2022-04-28T17:53:25,457][INFO ][logstash.runner ] Logstash shut down.
[2022-04-28T17:53:40,827][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.1.1"}
[2022-04-28T17:53:42,992][ERROR][logstash.filters.kv ] Unknown setting 'allow_empty_values' for kv
[2022-04-28T17:53:43,003][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Something is wrong with your configuration.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:86:in config_init'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:126:in initialize'", "org/logstash/plugins/PluginFactoryExt.java:78:in filter_delegator'", "org/logstash/plugins/PluginFactoryExt.java:248:in plugin'", "org/logstash/execution/JavaBasePipelineExt.java:50:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:23:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:36:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in block in converge_state'"]}
[2022-04-28T17:53:43,403][INFO ][logstash.runner ] Logstash shut down.

6

allow_empty_values was added in 8.1.0. You can remove it from your configuration but then you will not get a [url] field from url=&.

7

**@Badger **

Thank you Sir for Prompt reply my issue is resolve and really appriciate your time and Kind of support really Thanks Sir

8

@Badger

i have one Other quetsion after getting proper splite message custome feild is showing inproper data can you Help me for the same ?
@timestamp Apr 29, 2022 @ 12:36:38.313
@version 1
_id 3qsjdIAB3FWXhrhZyUAg
_index crm_webhook_test
_score -
_type _doc
campaign{id} 57
contact{email} test@test.com
contact{fields}{39} custom+field+value
contact{first_name} First
contact{id} 42
contact{ip} 127.0.0.1
contact{last_name} Last
date_time 2013-01-01+12:00:00
event.original url=&type=unsubscribe&date_time=2013-01-01+12:00:00&initiated_by=admin&initiated_from=admin&list=1&form[id]=1004&campaign[id]=57&unsubscribe[reason]=My+interests+are+changing&contact[id]=42&contact[email]=test@test.com&contact[first_name]=First&contact[last_name]=Last&contact[ip]=127.0.0.1&contact[fields][39]=custom+field+value
form{id} 1004
host.ip 0.0.0.0
http.method POST
http.request.body.bytes 375
http.request.mime_type application/x-www-form-urlencoded
http.version HTTP/1.1
initiated_by admin
initiated_from admin
list 1
message url=&type=unsubscribe&date_time=2013-01-01+12:00:00&initiated_by=admin&initiated_from=admin&list=1&form{id}=1004&campaign{id}=57&unsubscribe{reason}=My+interests+are+changing&contact{id}=42&contact{email}=test@test.com&contact{first_name}=First&contact{last_name}=Last&contact{ip}=127.0.0.1&contact{fields}{39}=custom+field+value
type unsubscribe
unsubscribe{reason} My+interests+are+changing
url

Contact field value and unsubscribe field value is now showing proper

9

This is issue is solved using split and joint mutate

Now I have another question to @Badger I am using OpenSearch Elasticsearch version

and if I am using Logstash 8.1 for KV filter How can I send Data to My Opensearch You have any idea ? please help for the same.

Thanks in Advance

10

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.