Remove a field that starts with an integer

Gabriel_Rosca · July 27, 2015, 9:43pm

Hi Guys,

I want to remove a field that starts with an integer. How can I do that using logstash.

EX: of the field.

"1. 2015.07.27 17": "25:35 message sent"

Regards,
Gabriel

warkolm · July 27, 2015, 11:08pm

Well if you build a grok pattern for the message and define that field, you can just use mutate and drop to remove it.

Gabriel_Rosca · July 27, 2015, 11:31pm

Hi Mark,

I receive the message in JSON format ... And that field gets created every hour. I want to be able to remove it before gets inserted in ES.

Regards,
Gabriel

warkolm · July 27, 2015, 11:33pm

Ok so take a look at the mutate filter with drop

suyograo · July 28, 2015, 5:48am

@Gabriel_Rosca so I understand, your event has a field foo which some times start with an int and some times does not. And you want to remove it when it starts with an int? is that correct?

Gabriel_Rosca · July 28, 2015, 1:03pm

@suyograo No. That field always starts with 1. follow my the data and hour

EX:

"1. 2015.07.28 08"

I need to remove this field from the message before gets in to ES.

something like remove_field => [ "^1.* ]

Not sure how I can do that. Mutate filter works only if I know the name of the field. And the name of the field it is static. In this case when a new message comes I get a new field per hour.

So now that is 9 AM when I get a new message ... a new field gets created and named "1. 2015.07.28 09"

Regards,
Gabriel

magnusbaeck · July 28, 2015, 5:46pm

It's not terribly clear what's going on here. Can you provide a sample message? What is creating these fields? What's your configuration?

Gabriel_Rosca · July 28, 2015, 6:26pm

Hi Magnus,

here is my config:

filter {
if [type] == "zabbix-alerts" {
json {
source => "message"
remove_field => "message"
}
csv {
source => "trigger_hostgroup_name"
columns => [ "group1", "group2", "group3", "group4", "group5", "group6", "group7", "group8", "group9", "group10", "group11", "group12", "group13", "group14", "group15" ]
}
mutate {
gsub => [
"group2", "^ ", "",
"group3", "^ ", "",
"group4", "^ ", "",
"group5", "^ ", "",
"group6", "^ ", "",
"group7", "^ ", "",
"group8", "^ ", "",
"group9", "^ ", "",
"group10", "^ ", "",
"group11", "^ ", "",
"group12", "^ ", "",
"group13", "^ ", "",
"group14", "^ ", "",
"group15", "^ ", ""
]
}
}
}

Here is one sample message that I received:

I just want to remove the first field "1. 2015.07.28 09" which gets created ever hour as 1. date and hours

2015.07.28 09 36:45 message sent Logstash nysv0654.example.com "Logstash User logstash (Logstash User)"
@timestamp July 28th 2015, 09:41:58.129
@version 1
name zabbix_data
_id AU7U5fj8aHSptZQGBhc8
_index logstash-ptc-zabbix-alerts-2015.07.28
_type zabbix-alerts
action_id 14
action_name Logstash Notification
date 2015.07.28
esc_history Problem started: 2015.07.28 09:36:42 Age: 5m
event_ack_history
event_ack_status No
event_age 5m
event_conn1 10.250.15.111
event_date 2015.07.28
event_dns1 PD-CATS-P-WEB-001.example.com
event_host1 PD-CATS-P-WEB-001.example.com
event_id 4614976
event_ip1 10.250.15.111
event_name1 PD-CATS-P-WEB-001.example.com
event_port1 10050
event_recovery_date 2015.07.28
event_recovery_id 4615767
event_recovery_status OK
event_recovery_time 09:41:42
event_recovery_value 0
event_status PROBLEM
event_time 09:36:42
event_value 1
group1 10.250.15.0/24
group2 10.250.15.0/24 Windows Agent
group3 Discovered hosts
group4 PTC
group5 PTC PROD
group6 PTC PROD Windows
group7 Windows Servers
host 10.250.26.108
inventory_contact1
inventory_hw_arch1 X86-based PC
inventory_model1 VMware Virtual Platform
inventory_name1 nyvm0669
inventory_os1 Microsoft? Windows Server? 2008 Standard
inventory_poc_primary_phone_b1
inventory_poc_primary_screen1
inventory_poc_secondary_cell1 1
inventory_poc_secondary_email1 1
inventory_poc_secondary_name1 1
inventory_poc_secondary_notes1
inventory_poc_secondary_phone_a1 example.com
inventory_poc_secondary_phone_b1 2 GB
inventory_poc_secondary_screen1
inventory_serialno_a1 VMware-42 1f 38 d0 42 14 81 ca-87 6d 54 4b 52 ef d7 43
item_name1 Processor load (1 min average)
item_name_orig1 Processor load (1 min average)
item_orgi1 system.cpu.load[percpu,avg1]
item_value1 3.833333
message_type recovery
node_id1 {NODE.ID1}
node_name1 {NODE.NAME1}
problem ended 2015.07.28 09:41:42
proxy_name1
time 09:41:48
trigger_description
trigger_events_ack 0
trigger_events_problem_ack 0
trigger_events_problem_unack 523
trigger_events_unack 1047
trigger_expression {PD-CATS-P-WEB-001.example.com:system.cpu.load[percpu,avg1].avg(5m)}>5
trigger_hostgroup_name 10.250.15.0/24, 10.250.15.0/24 Windows Agent, Discovered hosts, PTC, PTC PROD, PTC PROD Windows, Windows Servers
trigger_id 16165
trigger_name Processor load is too high on PD-CATS-P-WEB-001.example.com
trigger_name_orig Processor load is too high on {HOST.NAME}
trigger_nseverity 3
trigger_severity Average
trigger_status OK
trigger_template_name Custom Template OS Windows
trigger_value 0
type zabbix-alerts

Gabriel_Rosca · July 28, 2015, 6:45pm

Hi Magnus,

Here is my config

filter {
if [type] == "zabbix-alerts" {
json {
source => "message"
remove_field => "message"
}
csv {
source => "trigger_hostgroup_name"
columns => [ "group1", "group2", "group3", "group4", "group5", "group6", "group7", "group8", "group9", "group10", "group11", "group12", "group13", "group14", "group15" ]
}
mutate {
gsub => [
"group2", "^ ", "",
"group3", "^ ", "",
"group4", "^ ", "",
"group5", "^ ", "",
"group6", "^ ", "",
"group7", "^ ", "",
"group8", "^ ", "",
"group9", "^ ", "",
"group10", "^ ", "",
"group11", "^ ", "",
"group12", "^ ", "",
"group13", "^ ", "",
"group14", "^ ", "",
"group15", "^ ", ""
]
}
}
}

and here is the json message that I received:

What I want is to remove the field highlight in bold which gets created every hour

{
"_index": "logstash-ptc-zabbix-alerts-2015.07.28",
"_type": "zabbix-alerts",
"_id": "AU7V5kheXErN6gNFz_bs",
"_score": null,
"_source": {
"@version": "1",
"@timestamp": "2015-07-28T18:21:55.667Z",
"host": "10.250.26.108",
"type": "zabbix-alerts",
"event_status": "PROBLEM",
"event_ip1": "10.250.15.208",
"event_time": "14:16:37",
"event_value": "1",
"event_age": "5m",
"inventory_url_b1": "",
"trigger_hostgroup_name": [
"10.250.15.0/24, 10.250.15.0/24 Windows Agent, Discovered hosts, PTC, PTC PROD, PTC PROD Windows, Windows Servers"
],
"inventory_os1": "Microsoft(R) Windows(R) Server 2003, Standard Edition",
"event_ack_status": "No",
"item_lastvalue1": "3.85",
"trigger_events_unack": "1489",
"item_id1": "35917",
"event_recovery_status": "OK",
"inventory_poc_secondary_cell1": "1",
"event_dns1": "NYVM0428",
"inventory_poc_secondary_phone_a1": "example.com",
"trigger_template_name": "UN Template OS Windows",
"item_name1": "Processor load (1 min average)",
"trigger_expression": "{NYVM0428:system.cpu.load[percpu,avg1].avg(5m)}>5",
"inventory_model1": "VMware Virtual Platform",
"action_id": "14",
"inventory_poc_secondary_name1": "1",
"problem ended": "2015.07.28 14:21:37",
"item_name_orig1": "Processor load (1 min average)",
"inventory_serialno_a1": "VMware-42 1f cb f0 57 fc d9 44-ef e2 4e 0e cd 76 94 98",
"inventory_poc_secondary_phone_b1": "1023.39 MB",
"name": "zabbix_data",
"event_conn1": "10.250.15.208",
"inventory_vendor1": "Phoenix Technologies LTD",
"event_name1": "NYVM0428",
"trigger_value": "0",
"1. 2015.07.28 14": "16:41 message sent Logstash nysv0654 "Logstash User logstash (Logstash User)"",
"event_port1": "10050",
"trigger_severity": "Average",
"trigger_name_orig": "Processor load is too high on {HOST.NAME}",
"event_host1": "NYVM0428.",
"inventory_name1": "nyvm0428",
"trigger_status": "OK",
"item_key1": "system.cpu.load[percpu,avg1]",
"trigger_events_problem_unack": "744",
"item_value1": "3.85",
"trigger_events_ack": "0",
"event_recovery_time": "14:21:37",
"inventory_poc_primary_cell1": "",
"trigger_events_problem_ack": "0",
"event_recovery_value": "0",
"item_orgi1": "system.cpu.load[percpu,avg1]",
"event_id": "4663681",
"trigger_nseverity": "3",
"event_date": "2015.07.28",
"inventory_hw_arch1": "X86-based PC",
"action_name": "Logstash Notification",
"event_recovery_id": "4664691",
"trigger_id": "16318",
"date": "2015.07.28",
"trigger_name": "Processor load is too high on NYVM0428",
"event_recovery_date": "2015.07.28",
"time": "14:21:39",
"esc_history": "Problem started: 2015.07.28 14:16:37 Age: 5m",
"message_type": "recovery",
"group1": "10.250.15.0/24",
"group2": "10.250.15.0/24 Windows Agent",
"group3": "Discovered hosts",
"group4": "PTC",
"group5": "PTC PROD",
"group6": "PTC PROD Windows",
"group7": "Windows Servers"
},
"fields": {
"@timestamp": [
1438107715667
]
},
"sort": [
1438107715667
]
}

magnusbaeck · July 28, 2015, 7:54pm

I can't find any field name or field value that contains the string "1. 2015.07.28 09" in the example that you posted.

Gabriel_Rosca · July 28, 2015, 8:31pm

I got it to work by using ruby filter

   ruby {
    code => "
    event.to_hash.keys.each { |k|
    if k.start_with?('1.')
      event.remove(k)
    end
    }
   "
   }

Gabriel_Rosca · July 28, 2015, 8:43pm

I mark that in bold

"1. 2015.07.28 14": "16:41 message sent Logstash nysv0654 "Logstash User logstash (Logstash User)"",