Remove a field that starts with an integer


(Gabriel Rosca) #1

Hi Guys,

I want to remove a field that starts with an integer. How can I do that using logstash.

EX: of the field.

"1. 2015.07.27 17": "25:35 message sent"

Regards,
Gabriel


(Mark Walkom) #2

Well if you build a grok pattern for the message and define that field, you can just use mutate and drop to remove it.


(Gabriel Rosca) #3

Hi Mark,

I receive the message in JSON format ... And that field gets created every hour. I want to be able to remove it before gets inserted in ES.

Regards,
Gabriel


(Mark Walkom) #4

Ok so take a look at the mutate filter with drop :slight_smile:


(Suyog Rao) #5

@Gabriel_Rosca so I understand, your event has a field foo which some times start with an int and some times does not. And you want to remove it when it starts with an int? is that correct?


(Gabriel Rosca) #6

@suyograo No. That field always starts with 1. follow my the data and hour

EX:

"1. 2015.07.28 08"

I need to remove this field from the message before gets in to ES.

something like remove_field => [ "^1.* ]

Not sure how I can do that. Mutate filter works only if I know the name of the field. And the name of the field it is static. In this case when a new message comes I get a new field per hour.

So now that is 9 AM when I get a new message ... a new field gets created and named "1. 2015.07.28 09"

Regards,
Gabriel


(Magnus Bäck) #7

It's not terribly clear what's going on here. Can you provide a sample message? What is creating these fields? What's your configuration?


(Gabriel Rosca) #8

Hi Magnus,

here is my config:

filter {
if [type] == "zabbix-alerts" {
json {
source => "message"
remove_field => "message"
}
csv {
source => "trigger_hostgroup_name"
columns => [ "group1", "group2", "group3", "group4", "group5", "group6", "group7", "group8", "group9", "group10", "group11", "group12", "group13", "group14", "group15" ]
}
mutate {
gsub => [
"group2", "^ ", "",
"group3", "^ ", "",
"group4", "^ ", "",
"group5", "^ ", "",
"group6", "^ ", "",
"group7", "^ ", "",
"group8", "^ ", "",
"group9", "^ ", "",
"group10", "^ ", "",
"group11", "^ ", "",
"group12", "^ ", "",
"group13", "^ ", "",
"group14", "^ ", "",
"group15", "^ ", ""
]
}
}
}

Here is one sample message that I received:

I just want to remove the first field "1. 2015.07.28 09" which gets created ever hour as 1. date and hours

  1. 2015.07.28 09 36:45 message sent Logstash nysv0654.example.com "Logstash User logstash (Logstash User)"
    @timestamp July 28th 2015, 09:41:58.129
    @version 1
    name zabbix_data
    _id AU7U5fj8aHSptZQGBhc8
    _index logstash-ptc-zabbix-alerts-2015.07.28
    _type zabbix-alerts
    action_id 14
    action_name Logstash Notification
    date 2015.07.28
    esc_history Problem started: 2015.07.28 09:36:42 Age: 5m
    event_ack_history
    event_ack_status No
    event_age 5m
    event_conn1 10.250.15.111
    event_date 2015.07.28
    event_dns1 PD-CATS-P-WEB-001.example.com
    event_host1 PD-CATS-P-WEB-001.example.com
    event_id 4614976
    event_ip1 10.250.15.111
    event_name1 PD-CATS-P-WEB-001.example.com
    event_port1 10050
    event_recovery_date 2015.07.28
    event_recovery_id 4615767
    event_recovery_status OK
    event_recovery_time 09:41:42
    event_recovery_value 0
    event_status PROBLEM
    event_time 09:36:42
    event_value 1
    group1 10.250.15.0/24
    group2 10.250.15.0/24 Windows Agent
    group3 Discovered hosts
    group4 PTC
    group5 PTC PROD
    group6 PTC PROD Windows
    group7 Windows Servers
    host 10.250.26.108
    inventory_contact1
    inventory_hw_arch1 X86-based PC
    inventory_model1 VMware Virtual Platform
    inventory_name1 nyvm0669
    inventory_os1 Microsoft? Windows Server? 2008 Standard
    inventory_poc_primary_phone_b1
    inventory_poc_primary_screen1
    inventory_poc_secondary_cell1 1
    inventory_poc_secondary_email1 1
    inventory_poc_secondary_name1 1
    inventory_poc_secondary_notes1
    inventory_poc_secondary_phone_a1 example.com
    inventory_poc_secondary_phone_b1 2 GB
    inventory_poc_secondary_screen1
    inventory_serialno_a1 VMware-42 1f 38 d0 42 14 81 ca-87 6d 54 4b 52 ef d7 43
    item_name1 Processor load (1 min average)
    item_name_orig1 Processor load (1 min average)
    item_orgi1 system.cpu.load[percpu,avg1]
    item_value1 3.833333
    message_type recovery
    node_id1 {NODE.ID1}
    node_name1 {NODE.NAME1}
    problem ended 2015.07.28 09:41:42
    proxy_name1
    time 09:41:48
    trigger_description
    trigger_events_ack 0
    trigger_events_problem_ack 0
    trigger_events_problem_unack 523
    trigger_events_unack 1047
    trigger_expression {PD-CATS-P-WEB-001.example.com:system.cpu.load[percpu,avg1].avg(5m)}>5
    trigger_hostgroup_name 10.250.15.0/24, 10.250.15.0/24 Windows Agent, Discovered hosts, PTC, PTC PROD, PTC PROD Windows, Windows Servers
    trigger_id 16165
    trigger_name Processor load is too high on PD-CATS-P-WEB-001.example.com
    trigger_name_orig Processor load is too high on {HOST.NAME}
    trigger_nseverity 3
    trigger_severity Average
    trigger_status OK
    trigger_template_name Custom Template OS Windows
    trigger_value 0
    type zabbix-alerts

(Gabriel Rosca) #9

Hi Magnus,

Here is my config

filter {
if [type] == "zabbix-alerts" {
json {
source => "message"
remove_field => "message"
}
csv {
source => "trigger_hostgroup_name"
columns => [ "group1", "group2", "group3", "group4", "group5", "group6", "group7", "group8", "group9", "group10", "group11", "group12", "group13", "group14", "group15" ]
}
mutate {
gsub => [
"group2", "^ ", "",
"group3", "^ ", "",
"group4", "^ ", "",
"group5", "^ ", "",
"group6", "^ ", "",
"group7", "^ ", "",
"group8", "^ ", "",
"group9", "^ ", "",
"group10", "^ ", "",
"group11", "^ ", "",
"group12", "^ ", "",
"group13", "^ ", "",
"group14", "^ ", "",
"group15", "^ ", ""
]
}
}
}

and here is the json message that I received:

What I want is to remove the field highlight in bold which gets created every hour

{
"_index": "logstash-ptc-zabbix-alerts-2015.07.28",
"_type": "zabbix-alerts",
"_id": "AU7V5kheXErN6gNFz_bs",
"_score": null,
"_source": {
"@version": "1",
"@timestamp": "2015-07-28T18:21:55.667Z",
"host": "10.250.26.108",
"type": "zabbix-alerts",
"event_status": "PROBLEM",
"event_ip1": "10.250.15.208",
"event_time": "14:16:37",
"event_value": "1",
"event_age": "5m",
"inventory_url_b1": "",
"trigger_hostgroup_name": [
"10.250.15.0/24, 10.250.15.0/24 Windows Agent, Discovered hosts, PTC, PTC PROD, PTC PROD Windows, Windows Servers"
],
"inventory_os1": "Microsoft(R) Windows(R) Server 2003, Standard Edition",
"event_ack_status": "No",
"item_lastvalue1": "3.85",
"trigger_events_unack": "1489",
"item_id1": "35917",
"event_recovery_status": "OK",
"inventory_poc_secondary_cell1": "1",
"event_dns1": "NYVM0428",
"inventory_poc_secondary_phone_a1": "example.com",
"trigger_template_name": "UN Template OS Windows",
"item_name1": "Processor load (1 min average)",
"trigger_expression": "{NYVM0428:system.cpu.load[percpu,avg1].avg(5m)}>5",
"inventory_model1": "VMware Virtual Platform",
"action_id": "14",
"inventory_poc_secondary_name1": "1",
"problem ended": "2015.07.28 14:21:37",
"item_name_orig1": "Processor load (1 min average)",
"inventory_serialno_a1": "VMware-42 1f cb f0 57 fc d9 44-ef e2 4e 0e cd 76 94 98",
"inventory_poc_secondary_phone_b1": "1023.39 MB",
"name": "zabbix_data",
"event_conn1": "10.250.15.208",
"inventory_vendor1": "Phoenix Technologies LTD",
"event_name1": "NYVM0428",
"trigger_value": "0",
"1. 2015.07.28 14": "16:41 message sent Logstash nysv0654 "Logstash User logstash (Logstash User)"",
"event_port1": "10050",
"trigger_severity": "Average",
"trigger_name_orig": "Processor load is too high on {HOST.NAME}",
"event_host1": "NYVM0428.",
"inventory_name1": "nyvm0428",
"trigger_status": "OK",
"item_key1": "system.cpu.load[percpu,avg1]",
"trigger_events_problem_unack": "744",
"item_value1": "3.85",
"trigger_events_ack": "0",
"event_recovery_time": "14:21:37",
"inventory_poc_primary_cell1": "",
"trigger_events_problem_ack": "0",
"event_recovery_value": "0",
"item_orgi1": "system.cpu.load[percpu,avg1]",
"event_id": "4663681",
"trigger_nseverity": "3",
"event_date": "2015.07.28",
"inventory_hw_arch1": "X86-based PC",
"action_name": "Logstash Notification",
"event_recovery_id": "4664691",
"trigger_id": "16318",
"date": "2015.07.28",
"trigger_name": "Processor load is too high on NYVM0428",
"event_recovery_date": "2015.07.28",
"time": "14:21:39",
"esc_history": "Problem started: 2015.07.28 14:16:37 Age: 5m",
"message_type": "recovery",
"group1": "10.250.15.0/24",
"group2": "10.250.15.0/24 Windows Agent",
"group3": "Discovered hosts",
"group4": "PTC",
"group5": "PTC PROD",
"group6": "PTC PROD Windows",
"group7": "Windows Servers"
},
"fields": {
"@timestamp": [
1438107715667
]
},
"sort": [
1438107715667
]
}


(Magnus Bäck) #10

I can't find any field name or field value that contains the string "1. 2015.07.28 09" in the example that you posted.


(Gabriel Rosca) #11

I got it to work by using ruby filter

   ruby {
    code => "
    event.to_hash.keys.each { |k|
    if k.start_with?('1.')
      event.remove(k)
    end
    }
   "
   }

Remove field based on name regex
(Gabriel Rosca) #12

I mark that in bold

"1. 2015.07.28 14": "16:41 message sent Logstash nysv0654 "Logstash User logstash (Logstash User)"",


(system) #13