New user confused about odd @Timestamp

[nfored (Fred)]

I have 1 logs stash server that host 8 end points, outputs to elasticsearch 6 of the index correctly log @timestamp in UTC and life is lovely two index @timestamp is in the local timezone. This is the time stamp shown in json form in kibana.

I am greatly confused as all 8 index work as expected on normal vm deployment with the same longstash config, but moving to k8 I did have to combine input and output into the same file and use multiple pipelines in the k8 vs multiple inputs and one output with one pipeline on normal vm.

please let me know what data to show that would help in seeking help.

[leandrojmp (Leandro Pereira)]

You need to provide more context and information as it is not clear what is your issue.

How are you running Logstash? Are you using pipelines.yml? Share your pipelines.yml file.

What does your configuration looks like? Do you have a date filter in the configuration?

Which is the timezone in your logs? What does your messages looks like? Share a sample of your messages.

[nfored (Fred)]

cat pipelines.yml

• pipeline.id: alpr
  path.config: "/opt/bitnami/logstash/config/alpr.conf"
• pipeline.id: f5afm
  path.config: "/opt/bitnami/logstash/config/f5afm.conf"
• pipeline.id: f5asm
  path.config: "/opt/bitnami/logstash/config/f5asm.conf"
• pipeline.id: f5bot
  path.config: "/opt/bitnami/logstash/config/f5bot.conf"
• pipeline.id: ingress
  path.config: "/opt/bitnami/logstash/config/ingress.conf"
• pipeline.id: logstashrsyslog
  path.config: "/opt/bitnami/logstash/config/logstashrsyslog.conf"
• pipeline.id: fortifw
  path.config: "/opt/bitnami/logstash/config/fortifw.conf"

Wrong TIMESTAMP
input {
   udp {
     port => 9904
     type => syslog
     id => fortigate_9904
     tags => ["fortigate","9904"]
     }
}

filter {
if "fortigate" in [tags] {
                mutate {
                        add_field => { "[@metadata][index]" => "fortigate-%{+YYYY.MM.dd}" }
                        add_tag => ["fortigate"]
                        add_field => [ "forti_host", "fw.mcfeetershq.local" ]
                }
                grok {
                        match => ["message", "%{SYSLOG5424PRI:syslog_index}%{GREEDYDATA:message}"]
                        overwrite => [ "message" ]
                        tag_on_failure => [ "failure_grok_fortigate" ]
                }

                kv { }

                if [msg] {
                        mutate {
                                replace => [ "message", "%{msg}" ]
                        }
                }

               if [totalsession] {
                        mutate {
                                convert => { "totalsession" => "integer" }
                        }
                }


               if [cpu] {
                        mutate {
                                convert => { "cpu" => "integer" }
                        }
                }


               if [mem] {
                        mutate {
                                convert => { "mem" => "integer" }
                        }
                }


               if [setuprate] {
                        mutate {
                                convert => { "setuprate" => "integer" }
                        }
                }
                mutate {
                        add_field => ["logTimestamp", "%{date} %{time}"]
                        add_field => ["loglevel", "%{level}"]
                        replace => [ "fortigate_type", "%{type}"]
                        replace => [ "fortigate_subtype", "%{subtype}"]
                        remove_field => [ "msg", "type", "level", "date", "time" ]
                }

                date {
                        locale => "en"
                        match => ["logTimestamp", "YYYY-MM-dd HH:mm:ss"]
                        remove_field => ["logTimestamp", "year", "month", "day", "time", "date"]
                        add_field => ["type", "syslog"]
                }
}
}

output {
  elasticsearch {
    hosts => ["https://mhqelastic-es-http:9200"]
    index => "%{[@metadata][index]}"
    user => "elastic"
    password => "${ELASTICSEARCH_PASSWORD}"
    ssl => true
    cacert => "/var/data/tls.crt"
 }
}

Correct Timestamp

input {
syslog {
   port => 5344
   tags => ["bot","5344"]
   type => bot
   id => "bot_syslog_listener_5344"

 }
}
filter {
if [type] == 'bot' {
 mutate { add_field => { "[@metadata][index]" => "bot-%{+YYYY.MM.dd}" } }
# grok {
#   match => ["message", 'hostname=%{QS},bigip_mgmt_ip=%{QS},(bigip_mgmt_ip2=%{QS})?,client_ip=%{QS},client_ip_geo_location="N%{URIPATHPARAM}",client_port=%{QS},client_request_uri=%{QS},configuration_date_time="%{CISCOTIMESTAMP}",context_name=%{QS},context_type=%{QS},dest_ip=%{QS},dest_port=%{QS},device_product=%{QS},device_vendor=%{QS},device_version=%{QS},errdefs_msgno=%{QS},http_method=%{QS},http_protocol_indication=%{QS},http_protocol_info=%{QS},route_domain=%{QS},timestamp=%{QS},virtual_server_name=%{QS},device_id=%{QS},host=%{QS},request_date_time="%{CISCOTIMESTAMP}",profile_name=%{QS},support_id=%{QS},request_status=%{QS},action=%{QS},reason=%{QS},previous_action=%{QS},previous_support_id=%{QS},previous_request_date_time=%{QS},bot_signature=%{QS},bot_signature_category=%{QS},bot_name=%{QS},session_id=%{QS},class=%{QS},anomaly_categories=%{QS},anomalies=%{QS},additional_bot_signatures=%{QS},micro_service_name=%{QS},micro_service_type=%{QS},micro_service_matched_wildcard_url=%{QS},micro_service_hostname=%{QS},configured_mitigation_action=%{QS},configured_mitigation_action_reason=%{QS},actual_mitigation_action=%{QS},actual_mitigation_action_reason=%{QS},browser_configured_verification_action=%{QS},browser_actual_verification_action=%{QS},browser_actual_verification_action_reason=%{QS},captcha_status=%{QS},browser_verification_status=%{QS},device_id_status=%{QS},device_id_action=%{QS},previous_initiated_action=%{QS},previous_initiated_action_status=%{QS},new_request_status=%{QS},enforced_by=%{QS},mobile_is_app=%{QS},challenge_failure_reason=%{QS},classification_reason=%{QS},client_type=%{QS},application_display_name=%{QS},application_version=%{QS},mobile_in_emulation_mode=%{QS},os_name=%{QS},jailbroken_or_rooted_device=%{QS},mobile_debugger_enabled_device=%{QS},imei=%{QS},human_behaviour=%{QS},http_request=%{QS}']
#
#  }
      kv {
field_split => ","
}

              }
}
output {
  elasticsearch {
    hosts => ["https://mhqelastic-es-http:9200"]
    index => "%{[@metadata][index]}"
    user => "elastic"
    password => "${ELASTICSEARCH_PASSWORD}"
    ssl => true
    cacert => "/var/data/tls.crt"
 }
}

[nfored (Fred)]

Sorry looks like the the post put both working and not working in the same code block but if you look there is a line break that says correct or wrong

Sorry one last thing here is the other broken config

input {
 syslog {
   port => 5244
   tags => ["asm","5244"]
   type => asm
   id => "asm_syslog_listener_5244"
 }

 }
filter {
if [type] == 'asm' {
 mutate { add_field => { "[@metadata][index]" => "asm-%{+YYYY.MM.dd}" } }
 grok {
   match => {
     "message" => [
       "attack_type=\"%{DATA:attack_type}\"",
       ",blocking_exception_reason=\"%{DATA:blocking_exception_reason}\"",
       ",date_time=\"%{DATE:date_time}\"",
       ",dest_port=\"%{DATA:dest_port}\"",
       ",ip_client=\"%{DATA:ip_client}\"",
       ",is_truncated=\"%{DATA:is_truncated}\"",
       ",method=\"%{WORD:method}\"",
       ",policy_name=\"%{DATA:policy_name}\"",
       ",protocol=\"%{DATA:protocol}\"",
       ",request_status=\"%{DATA:request_status}\"",
       ",response_code=\"%{DATA:response_code}\"",
       ",severity=\"%{DATA:severity}\"",
       ",sig_cves=\"%{DATA:sig_cves}\"",
       ",sig_ids=\"%{DATA:sig_ids}\"",
       ",sig_names=\"%{DATA:sig_names}\"",
       ",sig_set_names=\"%{DATA:sig_set_names}\"",
       ",src_port=\"%{DATA:src_port}\"",
       ",sub_violations=\"%{DATA:sub_violations}\"",
       ",support_id=\"%{DATA:support_id}\"",
       "unit_hostname=\"%{DATA:unit_hostname}\"",
       ",uri=\"%{DATA:uri}\"",
       ",violation_rating=\"%{DATA:violation_rating}\"",
       ",vs_name=\"%{DATA:vs_name}\"",
       ",x_forwarded_for_header_value=\"%{DATA:x_forwarded_for_header_value}\"",
       ",outcome=\"%{DATA:outcome}\"",
       ",outcome_reason=\"%{DATA:outcome_reason}\"",
       ",violations=\"%{DATA:violations}\"",
       ",violation_details=\"%{DATA:violation_details}\"",
       ",request=\"%{DATA:request}\""
     ]
   }
   break_on_match => false
 }
#date {
#  match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss +0500" ]
#  target => "@timestamp"
#}
 mutate {
   split => { "attack_type" => "," }
   split => { "sig_ids" => "," }
   split => { "sig_names" => "," }
   split => { "sig_cves" => "," }
   split => { "staged_sig_ids" => "," }
   split => { "staged_sig_names" => "," }
   split => { "staged_sig_cves" => "," }
   split => { "sig_set_names" => "," }
   split => { "threat_campaign_names" => "," }
   split => { "staged_threat_campaign_names" => "," }
   split => { "violations" => "," }
   split => { "sub_violations" => "," }
 }
 if [x_forwarded_for_header_value] != "N/A" {
   mutate { add_field => { "source_host" => "%{x_forwarded_for_header_value}"}}
 } else {
   mutate { add_field => { "source_host" => "%{ip_client}"}}
 }
}
}
output {
  elasticsearch {
    hosts => ["https://mhqelastic-es-http:9200"]
    index => "%{[@metadata][index]}"
    user => "elastic"
    password => "${ELASTICSEARCH_PASSWORD}"
    ssl => true
    cacert => "/var/data/tls.crt"
 }
}

[nfored (Fred)]

ASM Message
 "hostname=\"awf01.mcfeetershq.local\",bigip_mgmt_ip=\"192.168.100.14\",bigip_mgmt_ip2=\"::\",client_ip=\"172.16.1.4\",client_ip_geo_location=\"N/A\",client_port=\"34000\",client_request_uri=\"/\",configuration_date_time=\"Sep 21 2022 19:39:58\",context_name=\"/Common/tk_wp_vs\",context_type=\"Virtual Server\",dest_ip=\"172.16.1.180\",dest_port=\"443\",device_product=\"Application Security Module\",device_vendor=\"F5\",device_version=\"pgo_use x86_64 vadc TMM Version 17.0.0.1.0.0.4 \",errdefs_msgno=\"23003147\",http_method=\"GET\",http_protocol_indication=\"HTTPS\",http_protocol_info=\"HTTP/1.1\",route_domain=\"0\",timestamp=\"Sep 25 2022 17:21:54\",virtual_server_name=\"/Common/tk_wp_vs\",device_id=\"N/A\",host=\"75.81.29.97\",request_date_time=\"Sep 25 2022 12:21:54\",profile_name=\"/Common/TK_WP_Bot\",support_id=\"4145068333164682490\",request_status=\"legal\",action=\"alarm\",reason=\"\",previous_action=\"None\",previous_support_id=\"N/A\",previous_request_date_time=\"N/A\",bot_signature=\"N/A\",bot_signature_category=\"N/A\",bot_name=\"OpenNMS\",session_id=\"0\",class=\"Unknown\",anomaly_categories=\"N/A\",anomalies=\"N/A\",additional_bot_signatures=\"N/A\",micro_service_name=\"N/A\",micro_service_type=\"N/A\",micro_service_matched_wildcard_url=\"N/A\",micro_service_hostname=\"N/A\",configured_mitigation_action=\"Alarm\",configured_mitigation_action_reason=\"\",actual_mitigation_action=\"Alarm\",actual_mitigation_action_reason=\"None\",browser_configured_verification_action=\"None\",browser_actual_verification_action=\"None\",browser_actual_verification_action_reason=\"None\",captcha_status=\"None\",browser_verification_status=\"None\",device_id_status=\"None\",device_id_action=\"None\",previous_initiated_action=\"None\",previous_initiated_action_status=\"None\",new_request_status=\"Alarmed\",enforced_by=\"Profile Mitigation and Verification Settings\",mobile_is_app=\"false\",challenge_failure_reason=\"\",classification_reason=\"\",client_type=\"Uncategorized\",application_display_name=\"N/A\",application_version=\"N/A\",mobile_in_emulation_mode=\"N/A\",os_name=\"N/A\",jailbroken_or_rooted_device=\"N/A\",mobile_debugger_enabled_device=\"N/A\",imei=\"N/A\",human_behaviour=\"N/A\",http_request=\"GET / HTTP/1.1\\r\\nConnection: CLOSE \\r\\nHost: 75.81.29.97\\r\\nUser-Agent: OpenNMS HttpMonitor\\r\\n\\r\\n\"\n",



Forti message

	
firewall,info drop-all-fwd forward: in:WAN out:RT, src-mac 00:19:07:80:ac:00, proto TCP (SYN), 185.21.216.183:50358->172.16.108.250:6944, NAT 185.21.216.183:50358->(208.94.244.51:6944->172.16.108.250:6944), len 52