Logstash does not create fields using grok

Elastic Stack Logstash
1

Hi! This is another "help me with grok" topic, so please help... :blush:

Source Log (simple f2b):

2019-04-18 16:39:03,454 fail2ban.actions        [25673]: NOTICE  [postfix-sasl] Ban 185.234.218.237
2019-04-18 19:19:20,432 fail2ban.filter         [25673]: INFO    [postfix-sasl] Found 82.140.81.38
2019-04-18 19:19:21,270 fail2ban.actions        [25673]: NOTICE  [postfix-sasl] Ban 82.140.81.38
2019-04-19 04:09:51,432 fail2ban.actions        [25673]: NOTICE  [postfix-sasl] Unban 170.80.134.250
2019-04-19 04:50:22,040 fail2ban.filter         [25673]: INFO    [postfix-sasl] Found 141.98.80.32
2019-04-19 04:50:31,258 fail2ban.filter         [25673]: INFO    [postfix-sasl] Found 141.98.80.32
2019-04-19 04:50:31,582 fail2ban.actions        [25673]: NOTICE  [postfix-sasl] Ban 141.98.80.32
2019-04-19 14:10:08,320 fail2ban.filter         [25673]: INFO    [postfix-sasl] Found 23.254.231.87

This log captured by filebeat and go to logstash:

- type: log
  enabled: true
  paths:
    - /var/log/fail2ban.log
  tags: ["fail2ban"]
output.logstash:
  # The Logstash hosts
  hosts: ["hostname:5044"]

Logstash makes its work:

input {
beats {
    port => 5044
}
}

filter {
  if [tags] == "fail2ban" {
grok {
  match => { "message" => "%{TIMESTAMP_ISO8601:date}\s%{WORD:log_src}.%{WORD:src_action}\s*\[%{INT:fail2ban_pid}\]:\s%{LOGLEVEL:level}\s*\[%{NOTSPACE:service}\]\s%{WORD:action}\s%{IP:ip}"}      
}
geoip {
  source => "ip"
}
  }
}
output {
if "fail2ban" in [tags] {
    elasticsearch {
        hosts     => "10.1.1.11:9200"
        index    => "fail2ban-%{+YYYY.MM}"
    }
    file {
        path => "/var/log/logstash/debug.log"
        codec=>rubydebug
    }
}
}

I add index fail2ban-%{+YYYY.MM} to kibana, but no additional fields appear:

@timestamp	Apr 19, 2019 @ 14:35:53.878
       	t  @version	1
      	t  _id	pQpiNWoBiVr8gQ-rCo3X
      	t  _index	fail2ban-2019.04
    	#  _score	 - 
      	t  _type	_doc
       	t  agent.ephemeral_id	c6bbe7df-cb34-43be-a81e-b17eb52ad2eb
       	t  agent.hostname	ihostname
       	t  agent.id	86e5338e-c3f6-414b-9b0e-63359cbe016c
       	t  agent.type	filebeat
       	t  agent.version	7.0.0
       	t  ecs.version	1.0.0
       	t  host.architecture	x86_64
       	 host.containerized	true
       	t  host.hostname	ihostname
       	t  host.id	ddac9954b8134d2999ae8149db6365ce
       	t  host.name	ihostname
       	t  host.os.codename	Core
       	t  host.os.family	redhat
       	t  host.os.kernel	3.10.0-957.5.1.el7.x86_64
       	t  host.os.name	CentOS Linux
       	t  host.os.platform	centos
       	t  host.os.version	7 (Core)
       	t  input.type	log
       	t  log.file.path	/var/log/fail2ban.log
       	#  log.offset	10,645
       	t  message	2019-04-18 16:39:03,454 fail2ban.actions        [25673]: NOTICE  [postfix-sasl] Ban 185.234.218.237
       	t  tags	fail2ban, beats_input_codec_plain_applied

Grok debugger show me parsed fields:

{
  "date": "2019-04-18 16:39:03,454",
  "level": "NOTICE",
  "service": "postfix-sasl",
  "ip": "185.234.218.237",
  "action": "Ban",
  "log_src": "fail2ban",
  "src_action": "actions",
  "fail2ban_pid": "25673"
}

So what is wrong?

2

I suspect this should instead be:

3

Thanks for response , but no effect.

4

What does the output to your debug log file look like?

5 
  "input" => {
    "type" => "log"
},
       "log" => {
    "offset" => 11337,
      "file" => {
        "path" => "/var/log/fail2ban.log"
    }
},
  "@version" => "1",
       "ecs" => {
    "version" => "1.0.0"
},
"@timestamp" => 2019-04-19T12:44:53.571Z,
      "tags" => [
    [0] "fail2ban",
    [1] "beats_input_codec_plain_applied"
],
      "host" => {
               "os" => {
          "kernel" => "3.10.0-957.5.1.el7.x86_64",
        "codename" => "Core",
            "name" => "CentOS Linux",
        "platform" => "centos",
         "version" => "7 (Core)",
          "family" => "redhat"
    },
    "containerized" => true,
             "name" => "ihostname",
               "id" => "ddac9954b8134d2999ae8149db6365ce",
     "architecture" => "x86_64",
         "hostname" => "ihostname"
},
     "agent" => {
              "id" => "86e5338e-c3f6-414b-9b0e-63359cbe016c",
            "type" => "filebeat",
         "version" => "7.0.0",
    "ephemeral_id" => "2901a7c6-38a2-4772-bd98-39d9cdb8fc25",
        "hostname" => "ihostname"
},
   "message" => "2019-04-19 14:10:08,320 fail2ban.filter         [25673]: INFO    [postfix-sasl] Found 23.254.231.87"
6

After second filebeat regestry clean and restart logstash+filebeat needed
data have apeared
Thanks for help!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.