Hi all,
After I ve been searching on the web, cannot find solution to my current issue.
I am goint to explain the full context: I am completey a newbie with Kibana, Elastic search and Logstash. I am completely discovering it.
- In my test-client A machine, I have logs (csv format) with that header:
type,timeLogged,timeQueued,orig,rcpt,orcpt,dsnAction,dsnStatus,dsnDiag,dsnMta,bounceCat,srcType,srcMta,dlvType,dlvSourceIp,dlvDestinationIp,dlvEsmtpAvailable,dlvSize,vmta,jobId,envId,queue,vmtaPool,header_X-EMarSys-Identify,header_X-EMarSys-Environment,header_x-mtnltcampaignid,header_x-mtnltserviceid,header_x-mtnltm2pid,header_x-mtnltid,header_x-quelle-identify,header_x-mailing-info,header_mpxcustomerid,header_x-member-id,header_x-wld-type-id,header_message-id,header_X-MM-Id,header_x-wld-site-id,header_x-wld-network-id,header_From,dsnReportingMta
(Indeed that comes from a Pmta server).
- On my ELK server, I have this current configuration
[root@emaelktestinstallation conf.d]# ls -alh
total 36K
drwxrwxr-x. 2 root root 4.0K Nov 5 11:19 .
drwxr-xr-x. 3 root root 4.0K Sep 21 19:31 ..
-rw-r--r-- 1 root root 193 Sep 22 21:26 01-lumberjack-input.conf
-rw-r--r-- 1 root root 854 Nov 5 11:14 02-pmta.conf
-rw-r--r-- 1 root root 2.0K Nov 5 10:53 02-pmta.conf.bkp
-rw-r--r-- 1 root root 12K Nov 5 11:19 .02-pmta.conf.swp
-rw-r--r-- 1 root root 81 Sep 22 18:44 30-lumberjack-output.conf
[root@emaelktestinstallation conf.d]#
And the configuration file which is important for me : 02-pmta.conf
The import is working fine.
However, the issue is that I want to use the timeLogged from my CSV as a @timestamp to be able to build up some graphs.
Indeed, I would like to use mostly these 2 variables timeLogged and timeQueued to proceed to some performance graphs.
My logstash configuration is currently like that:
input {
lumberjack {
port => 5000
type => "acct"
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
host => "10.40.0.48"
}
}
filter {
date {
locale => "en"
match => ["timeQueued", "YYYY-MM-dd;HH:mm:ss.SSS"]
timezone => "Europe/Vienna"
target => "@timestamp"
add_field => { "debug" => "timestampMatched"}
}
}
output {
elasticsearch {
action => "index"
host => "localhost"
index => "pmta_acct"
workers => 1
}
stdout {}
}
I can see all the entries in Kibana, But the timeQueued is still not used as @timestamp.
Any idea, how to proceed ?
As you can see in the picture bellow:
