Logstash Patterns help needed

Elastic Stack Logstash
1

Hi grok-coders,

I need a little bit help.

I want to get some grok patterns for the following lines. They are coming via Filebeats from a CentOS 7 Server (Odoo Access Logs). How can I start with getting this message proper into fields. I don't need that (+0200) this can be removed. I want to learn so maybe someone can just give some start until the GET/POST argument after that I want to try on my own.

192.168.0.68 - - [25/Apr/2019:16:32:42 +0200] "GET /shop HTTP/2.0" 200 3993 "https://test.test.ch/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
192.168.0.68 - - [25/Apr/2019:16:32:43 +0200] "POST /website/translations HTTP/2.0" 200 9382 "https://test.test.ch/shop" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
192.168.0.68 - - [25/Apr/2019:16:32:43 +0200] "GET /web/webclient/locale/de_CH HTTP/2.0" 200 1127 "https://test.test.ch/shop" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
192.168.0.68 - - [25/Apr/2019:16:32:45 +0200] "GET /shop/product/klemme-25mm-3 HTTP/2.0" 200 4117 "https://test.test.ch/shop" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
192.168.0.68 - - [25/Apr/2019:16:32:46 +0200] "POST /website/translations HTTP/2.0" 200 9379 "https://test.test.ch/shop/product/klemme-25mm-3" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
192.168.0.68 - - [25/Apr/2019:16:32:46 +0200] "POST /product_configurator/get_combination_info_website HTTP/2.0" 200 464 "https://test.test.ch/shop/product/klemme-25mm-3" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36"
2

This looks like a standard HTTP log file so the HTTPD_COMMONLOG pattern should do. See an example (that uses the similar COMBINEDAPACHELOG pattern) at https://www.elastic.co/guide/en/logstash/current/config-examples.html.

3

Hi Magnus,

I'm getting closer but it's not perfectly.

{
  "_index": "filebeat-7.0.0-rc2-2019.04.26",
  "_type": "_doc",
  "_id": "hBrzV2oBNMwXgYCyEAQb",
  "_version": 1,
  "_score": null,
  "_source": {
    "host": {
      "os": {
        "version": "7 (Core)",
        "kernel": "3.10.0-957.10.1.el7.x86_64",
        "codename": "Core",
        "name": "CentOS Linux",
        "family": "redhat",
        "platform": "centos"
      },
      "id": "86349b978d384fd3876cc916f212d71c",
      "hostname": "SVGWMA-ODOO-XX.XXXXXX.ch",
      "name": "SVGWMA-ODOO-XX.XXXXXX.ch",
      "containerized": true,
      "architecture": "x86_64"
    },
    "fileset": {
      "name": "access"
    },
    "agent": {
      "version": "7.0.0-rc2",
      "hostname": "SVGWMA-ODOO-XX.XXXXXX.ch",
      "id": "278463b0-c5d8-43ab-be12-9ec280d1acbc",
      "type": "filebeat",
      "ephemeral_id": "8b130d50-73d0-4e09-9f79-adb71036f2be"
    },
    "service": {
      "type": "nginx"
    },
    "timestamp": "26/Apr/2019:06:41:43 +0200",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "type": "odoo",
    "@timestamp": "2019-04-26T04:41:45.395Z",
    "event": {
      "module": "nginx",
      "dataset": "nginx.access"
    },
    "clientip": "192.168.0.68",
    "message": "192.168.0.68 - - [26/Apr/2019:06:41:43 +0200] \"POST /product_configurator/get_combination_info_website HTTP/2.0\" 200 473 \"https://XXX.XXXXXX.ch/shop/product/stern-schlusselring-2\" \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36\"",
    "@version": "1",
    "ident": "-",
    "request": "/product_configurator/get_combination_info_website",
    "httpversion": "2.0",
    "auth": "-",
    "ecs": {
      "version": "1.0.0"
    },
    "verb": "POST",
    "bytes": "473",
    "log": {
      "offset": 5128,
      "file": {
        "path": "/var/log/nginx/odoo.access.log"
      }
    },
    "input": {
      "type": "log"
    },
    "response": "200"
  },
  "fields": {
    "suricata.eve.timestamp": [
      "2019-04-26T04:41:45.395Z"
    ],
    "@timestamp": [
      "2019-04-26T04:41:45.395Z"
    ]
  },
  "sort": [
    1556253705395
  ]
}

My inputs config:

input {
  beats {
    port => 5044
    type => "odoo"
  }
}
filter {
  if [type] == "odoo" {
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    grok {
      match => { "message" => "%{HTTPD_COMMONLOG}" }
    }

  }
}

Now what would you do next for parsing: https://XXX.XXXXXX.ch/shop/product/stern-schlusselring-2\ and then the rest of it?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.