Hello,
i have an issue with my logstash when i am running my configuration file then some logs are coming then automatically its giving some error like
[2017-11-29T11:20:51,023][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::Elastic
Search::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-11-29T11:20:51,023][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::Elastic
Search::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-11-29T11:20:51,023][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but
no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message
=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError",
:will_retry_in_seconds=>2}
[2017-11-29T11:20:51,023][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but
no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message
=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError",
:will_retry_in_seconds=>8}
[2017-11-29T11:20:51,132][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connect
ion is working {:healthcheck_url=>http://elastic:xxxxxx@172.24.33.133:9200/, :path=>"/"}
[2017-11-29T11:20:51,291][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::Elastic
Search::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-11-29T11:20:51,291][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but
no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message
=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError",
:will_retry_in_seconds=>4}
[2017-11-29T11:20:51,382][WARN ][logstash.outputs.elasticsearch] UNEXPECTED POOL ERROR {:e=>#<LogStash::Outputs::Elastic
Search::HttpClient::Pool::NoConnectionAvailableError: No Available connections>}
[2017-11-29T11:20:51,382][ERROR][logstash.outputs.elasticsearch] Attempted to send a bulk request to elasticsearch, but
no there are no living connections in the connection pool. Perhaps Elasticsearch is unreachable or down? {:error_message
=>"No Available connections", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError",
:will_retry_in_seconds=>8}
you can see in this photo:
this is my configuration file:
input
{
udp
{
port => "5000"
type => "syslog"
}
}
filter
{
if [type] == "syslog"
{
if ([message] =~ /TRAFFIC/)
{
csv
{
separator => ","
columns => ["FUTURE_USE_1", "Receive Time", "Serial Number", "Type", "Threat/Content Type", "FUTURE_USE_2", "Generated Time", "Source IP", "Destination IP", "NAT Source IP", "NAT Destination IP", "Rule Name", "Source User", "Destination User", "Application", "Virtual System", "Source Zone", "Destination Zone", "Inbound Interface", "Outbound Interface", "Log Forwarding Profile", "FUTURE_USE_3", "Session ID", "Repeat Count", "Source Port", "Destination Port", "NAT Source Port", "NAT Destination Port", "Flags", "Protocol", "Action", "Bytes", "Bytes Sent", "Bytes Received", "Packets", "Start Time", "Elapsed Time", "Category", "FUTURE_USE_4", "Sequence Number", "Action Flags", "Source Location", "Destination Location", "FUTURE_USE_5", "Packets Sent", "Packets Received", "Session End Reason", "Device Group Hierarchy Level 1", "Device Group Hierarchy Level 2", "Device Group Hierarchy Level 3", "Device Group Hierarchy Level 4", "Virtual System Name", "Device Name", "Action Source", "Source VM UUID", "Destination VM UUID", "Tunnel ID/IMSI", "Monitor Tag/IMEI", "Parent Session ID", "Parent Start Time", "Tunnel Type"]
skip_empty_columns => false
}
}
else if ([message] =~ /THREAT/)
{
csv
{
separator => ","
columns => [
"FUTURE_USE_1", "ReceiveTime", "SerialNumber", "Type", "Threat_ContentType", "FUTURE_USE_2",
"GeneratedTime", "SourceIP", "DestinationIP", "NATSourceIP", "NATDestinationIP", "RuleName",
"SourceUser", "DestinationUser", "Application", "VirtualSystem", "SourceZone", "DestinationZone",
"InboundInterface", "OutboundInterface", "LogForwardingProfile", "FUTURE_USE_3", "SessionID",
"RepeatCount", "SourcePort", "DestinationPort", "NATSourcePort", "NATDestinationPort", "Flags",
"Protocol", "Action", "Miscellaneous", "ThreatID", "URLCategory", "Severity", "Direction",
"SequenceNumber", "ActionFlags", "SourceLocation", "DestinationLocation", "FUTURE_USE_4",
"ContentType", "PCAP_ID", "FileDigest", "Cloud", "URLIndex", "UserAgent", "FileType",
"X-Forwarded-For", "Referer", "Sender", "Subject", "Recipient", "ReportID",
"DeviceGroupHierarchyLevel1", "DeviceGroupHierarchyLevel2", "DeviceGroupHierarchyLevel3",
"DeviceGroupHierarchyLevel4", "VirtualSystemName", "DeviceName", "FUTURE_USE_5", "SourceVMUUID",
"DestinationVMUUID", "HTTPMethod", "TunnelID_IMSI", "MonitorTag_IMEI", "ParentSessionID",
"ParentStartTime", "TunnelType", "ThreatCategory", "ContentVersion", "FUTURE_USE_6"]
skip_empty_columns => false
}
}
}
}
output
{
stdout { codec => rubydebug }
#if [type] == "syslog"
if ([message] =~ /TRAFFIC/)
{
elasticsearch
{
index => "traffic-%{+YYYY.MM.dd}"
hosts => ["x.x.x.x:9200"]
user => "elastic"
password => "elastic"
}
}
else if ([message] =~ /THREAT/)
{
elasticsearch
{
index => "threat-%{+YYYY.MM.dd}"
hosts => ["x.x.x.x:9200"]
user => "elastic"
password => "elastic"
}
}
else
{
elasticsearch
{
index => "system-%{+YYYY.MM.dd}"
hosts => ["x.x.x.x:9200"]
user => "elastic"
password => "elastic"
}
}
}
please resolve this issue
Thanks,
Krunal.
