What does your Logstash config look like?
input {
tcp {
port => "5141"
codec => json
type => "syslog"
}
}
filter {
grok {
match => { "message" => "%{SYSLOG5424PRI:syslog_index}-\s*%{SYSLOGHOST:syslog_hostname} %{GREEDYDATA:syslog_message}" }
}
json
{
source => "syslog_message"
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
hosts => ["https://ELASTIC1:9200", "https://ELASTIC2:9200"]
user => "elastic"
password => "xxxxxxx"
cacert => "/etc/logstash/certs/ca.crt"
index => "jstest-%{+YYYY.MM.dd}"
action => "index"
}
}
What version are you on?
7.8
What do your Elasticsearch logs look like?
<01>-.hostname.{"name":"DefaultProfile","version":"1.0","isoTimeFormat":"yyyy-MM-dd'T'HH:mm:ss.SSSZ","type":"Event","category":"deny","protocolID":"6","sev":"4","src":"10.66.7.32","dst":"192.168.2.111","srcPort":"63298","dstPort":"445","relevance":"5","credibility":"5","startTimeEpoch":"1609264894432","startTimeISO":"2020-12-29T19:01:34.432+01:00","storageTimeEpoch":"1609264894432","storageTimeISO":"2020-12-29T19:01:34.432+01:00","deploymentID":"5c15c102-a647-11ea-8226-00505601062b","devTimeEpoch":"1609264893000","devTimeISO":"2020-12-29T19:01:33.000+01:00","srcPreNATPort":"0","dstPreNATPort":"0","srcPostNATPort":"0","dstPostNATPort":"0","hasIdentity":"false","payload":"<189>timestamp=1609264893.devname="DCL0001FW".devid="FG100FTK20004077".vd="VPN-PARTNER".date=2020-12-29.time=19:01:33.logid="000000001".type="traffic".subtype="forward".level="notice".eventtime=1609264893808550963.tz="+0100".srcip=10.66.7.32.srcport=63298.srcintf="To-GCP".srcintfrole="undefined".dstip=192.168.2.111.dstport=445.dstintf="To-DATALOG_PPD".dstintfrole="undefined".srccountry="Reserved".dstcountry="Reserved".sessionid=2062428193.proto=6.action="deny".policyid=0.policytype="policy".service="SMB".trandisp="noop".duration=0.sentbyte=0.rcvdbyte=0.sentpkt=0.vpn="To-GCP".vpntype="ipsec-static".appcat="unscanned".crscore=30.craction=131072.crlevel="high"\n","eventCnt":"1","hasOffense":"false","domainID":"4","domainName":"Decathlon","eventName":"Firewall.Deny","lowLevelCategory":"Firewall.Deny","highLevelCategory":"Access","eventDescription":"Firewall.Deny","protocolName":"tcp","logSource":"FortiGate.@.192.168.0.3","srcNetName":"Net-10-172-192.Net_10_0_0_0","dstNetName":"Net-10-172-192.Net_192_168_0_0","logSourceType":"Fortinet.FortiGate.Security.Gateway","logSourceGroup":"Other","logSourceIdentifier":"192.168.0.3"}
What is the output from the _cluster/stats?pretty&human API?
which command should I use?
He can no longer load the kibana page to see the logs.
Below the capture
I have the output below, when I do the following command:
tail -f /var/log/logstash/logstash-plain.log
[2021-01-07T18:07:31,699][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@ELASTIC1:9200/"}
[2021-01-07T18:07:31,712][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] ES Output version determined {:es_version=>7}
[2021-01-07T18:07:31,714][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2021-01-07T18:07:31,757][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"https://logstash_system:xxxxxx@ELASTIC2:9200/"}
[2021-01-07T18:07:31,782][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["https://ELASTIC1:9200", "https://ELASTIC2:9200"]}
[2021-01-07T18:07:31,785][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2021-01-07T18:07:31,790][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x45b9a85 run>"}
[2021-01-07T18:07:31,818][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2021-01-07T18:07:31,824][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:main, :".monitoring-logstash"], :non_running_pipelines=>}
[2021-01-07T18:07:31,934][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}