Filebeat dissect

Elastic Stack Beats
1

Hi, I'm trying to parse that type of line via dissect. I know that I can do pipeline/logstash grok but I want to find a way to do it with dissect directly on filebeat side :slight_smile:

filebeat.yml

################### filebeat Configuration #########################

############################# filebeat ######################################
filebeat:
  inputs:
  - enabled: true
    paths:
    - /opt/IBM/TWA/TWS/stdlist/logs/*.log
    tags: ['TWS','Qualite','tws_merge']

  - enabled: true
    paths:
    - /opt/IBM/TWA/TWS/event.log
    include_lines: ['^1']
    tags: ['TWS','Qualite','tws_event','tws_event_master']

###########################  Modules configuration ###########################

filebeat.modules:

############################ System Module ###################################

- module: system
  syslog:
    enabled: true
  auth:
    enabled: true

######################## Index Lifecycle Management (ILM) #####################

setup:
  ilm:
    check_exists: false

#######################  Elasticsearch template setting ######################

setup.template.settings:
  index.number_of_shards: 3
  index.codec: best_compression

###############################################################################
############################# Libbeat Config ##################################
# Base config file used by all other beats for using libbeat features

############################# Output ##########################################

output:
  elasticsearch:
    hosts:
    - elastic.domain:9200
    password: password
    username: filebeat_writer

############################# Processors ######################################
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - dissect:
      tokenizer: "%{TWS_event_number} %{TWS_schedule_cpu} %{TWS_schedule_id} %{TWS_job_name} %{TWS_job_cpu} %{TWS_job_number} %{TWS_job_status} %{TWS_real_name} %{TWS_job_user} %{TWS_jcl_name}"
      field: "message"
      target_prefix: ""
      trim_values: all
      trim_chars: " \t"
      when:
        contains:
          tags: tws_event


############################# Logging #########################################

logging:
  level: info
  to_files: true
  files:
    rotateeverybytes: 10485760
    path: /var/log/filebeat
    name: filebeat
    keepfiles: 7
    permissions: 0644

filebeat dissect debug

2021-01-13T17:25:17.926-0500    DEBUG   [processors]    processing/processors.go:203    Publish event: {
  "@timestamp": "2021-01-13T22:25:17.926Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.10.1"
  },
  "host": {
    "os": {
      "version": "6.9 (Santiago)",
      "family": "redhat",
      "name": "Red",
      "kernel": "2.6.32-696.23.1.el6.x86_64",
      "codename": "Santiago",
      "platform": "redhat"
    },
    "id": "f01906b4e97f59f14f308ffe00000017",
    "containerized": false,
    "name": "servername",
    "ip": [
      "172.16.1.50",
      "fe80::250:56ff:fe8c:308b"
    ],
    "mac": [
      "00:50:56:8c:30:8b"
    ],
    "hostname": "servername",
    "architecture": "x86_64"
  },
  "TWS_job_name": "",
  "TWS_job_number": "",
  "log": {
    "file": {
      "path": "/opt/IBM/TWA/TWS/event.log"
    },
    "offset": 764
  },
  "agent": {
    "name": "servername",
    "type": "filebeat",
    "version": "7.10.1",
    "hostname": "servername",
    "ephemeral_id": "00466804-0db9-4f22-a34d-bd9b3b642ea7",
    "id": "ce2ce694-4ef1-4926-b8ad-0432bca36800"
  },
  "TWS_job_user": "",
  "TWS_schedule_cpu": "",
  "TWS_job_cpu": "",
  "tags": [
    "TWS",
    "Qualite",
    "tws_event",
    "tws_event_master"
  ],
  "ecs": {
    "version": "1.6.0"
  },
  "TWS_jcl_name": "JOBS                                   TWS003       ABCDEF0033 16777502 5                                   TWS003 jde920 /jde/jdedwardsppack/maestro/PY/SCHED_RUNUBE\\040R5942119_TWS003\\040VE-012 0 5 2021011114370000 0      +++      3791 0 1610375819 0 0 0",
  "TWS_job_status": "",
  "TWS_event_number": "101",
  "message": "101       ABCDEF0033             JOBS                                   TWS003       ABCDEF0033 16777502 5                                   TWS003 jde920 /jde/jdedwardsppack/maestro/PY/SCHED_RUNUBE\\040R5942119_TWS003\\040VE-012 0 5 2021011114370000 0      +++      3791 0 1610375819 0 0 0",
  "TWS_schedule_id": "",
  "TWS_real_name": "ABCDEF0033"
}

Expected result

######################
# SOURCE LINE OF LOG #
######################
101       SERVERNAME        PP-001-Q7                                 PPP1002J       SERVERNAME 7603 5                                 PPP1002J ppqlf001 /path/app.sh 0 1 2021011303060000 0  +++      221 0 1610507167 0 7 0 250        PP-001-Q7 221        PP-001-Q7 2021011303000000 v.4 NONE NONE 4 NONE NONE NONE 0 0 0 0 0 NONE


###################
# EXPECTED RESULT #
###################
TWS_event_number: 101
TWS_schedule_cpu: SERVERNAME
TWS_schedule_id: PP-001-Q7
TWS_job_name: PPP1002J
TWS_job_cpu: SERVERNAME
TWS_job_number: 7603
TWS_job_status: 5
TWS_real_name: PPP1002J
TWS_job_user: ppqlf001
TWS_jcl_name: /path/app.sh

Thx in advance!!!

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.