Remove timestamp and host frpm logfile

Elastic Stack Logstash
1

Hello
I have a logstash that receive log from other logstash using tcp:
input {
tcp {
port => 65009
mode => server
id => "wm-id"
tags => "wm"
codec => line
}
}

In output, I put this syslog into file:
filter {
ruby {
code => '
if File.readlines("/proc/mounts").any?{ |line| line.split(" ")[0] == "/dev/drbd0" }
#logger.info("drbd is mounted.");
else
logger.info("drbd is not mounted. I drop every event");
event.cancel;
end
'
}

    ruby {
            code => "event.set('index_day', event.get('[@timestamp]').time.localtime.strftime('%Y%m%d%H'))"
    }

    grok {
           match => ["message", "%{SYSLOGTIMESTAMP:date} %{IPORHOST:hostname} %{GREEDYDATA:message}"]
           overwrite => [ "message"]
    }

}

output {
if "wm" in [tags] {
file {
#questo server per avere nel file solo log vero e proprio
codec => line {
format => "%{message}"
}
path => "/drbd/web-logs/wm-%{index_day}.log"
}
}
}

I tried with different filter to remove, before to write into file, timestamp and source ip.
The genuine log is:
xxx.xxx.62.231 1571064805.792 4766 xxx.xxx.35.86 64375 0 TUNNELED 5593 2406 unknown ssl ssl://kkkkk-t.kkkkk.tv:443/ kkkk k-t.kkkkk.tv - - OBSERVED Web%20Ads/Analytics -

write into log file is:
2019-10-14T14:53:26.127Z vlan22.tt-hhhhh.foo.com xxx.xxx.62.231 1571064805.792 4766 xxx.xxx.35.86 64375 0 TUNNELED 5593 24 06 unknown ssl ssl://kkkkk-t.kkkkk.tv:443/ kkkkk-t.kkkkk.tv - - OBSERVED Web%20Ads/Analytics -

I want to remove "2019-10-14T14:53:26.127Z" and "vlan22.tt-hhhhh.foo.com".
Please can I help me?

Thank you

2

I would expect a line codec to add a timestamp and hostname if you do not specify the format option. With format => "%{message}" it should just write out the contents of the [message] field.

3

Hi Badger your tip help me to resolve the problem.
Below the filter:
filter {
ruby {
code => '
if File.readlines("/proc/mounts").any?{ |line| line.split(" ")[0] == "/dev/drbd0" }
#logger.info("drbd is mounted.");
else
logger.info("drbd is not mounted. I drop every event");
event.cancel;
end
'
}

    ruby {
            code => "event.set('index_day', event.get('[@timestamp]').time.localtime.strftime('%Y%m%d%H'))"
    }

    grok {
           match => ["message", "%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:hostname} %{GREEDYDATA:message}"]
           overwrite => [ "message"]
    }

}

The problem is that I haven't found documentation about the mean of TIMESTAMP_ISO8601, IPORHOST and GREENDYDATA.

4

Those are all defined in the basic grok patterns.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.