Hello, I am new to Logstash and the whole Elasticsearch stack. I am trying to make a Logstash config file to ingest Cisco Telemetry data.
This is the structure of the data I am trying to ingest:
[host 10.10.20.30 session-id 35] Delivering to <ncclient.operations.subscribe.EstablishSubscription object at 0x7fb82b2e5780>
Subscription Result : notif-bis:ok
Subscription Id : 2147483650
-->>
(Default Callback)
Event time : 2019-09-12 13:13:30.290000+00:00
Subscription Id : 2147483650
Type : 1
Data :
<datastore-contents-xml xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push">
<memory-statistics xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-memory-oper">
<memory-statistic>
<name>Processor</name>
<total-memory>2238677360</total-memory>
<used-memory>340449924</used-memory>
<free-memory>1898227436</free-memory>
<lowest-usage>1897220640</lowest-usage>
<highest-usage>1264110388</highest-usage>
</memory-statistic>
<memory-statistic>
<name>lsmpi_io</name>
<total-memory>3149400</total-memory>
<used-memory>3148576</used-memory>
<free-memory>824</free-memory>
<lowest-usage>824</lowest-usage>
<highest-usage>412</highest-usage>
</memory-statistic>
</memory-statistics>
</datastore-contents-xml>
<<--
-->>
(Default Callback)
Event time : 2019-09-12 13:13:40.290000+00:00
Subscription Id : 2147483650
Type : 1
Data :
<datastore-contents-xml xmlns="urn:ietf:params:xml:ns:yang:ietf-yang-push">
<memory-statistics xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-memory-oper">
<memory-statistic>
<name>Processor</name>
<total-memory>2238677360</total-memory>
<used-memory>340449924</used-memory>
<free-memory>1898227436</free-memory>
<lowest-usage>1897220640</lowest-usage>
<highest-usage>1264110388</highest-usage>
</memory-statistic>
...........
I have commented all the problems that I am having with the config, I am open to any suggestions that can improve the config.
This is my current configuration:
input {
file {
path => "/home/elastic-stack/logstash-7.3.2/event-data/telemetry.log"
start_position => "beginning"
type => "sandbox-out"
codec => multiline {
pattern => "^<\?datastore-contents-xml .*\>"
negate => "true"
what => "previous"
}
}
http {
host => "127.0.0.1"
port => 8080
type => "sandbox-out"
}
}
filter {
grok {
match => { "message" => "\[%{USER:host_name} %{IP:ip_address} %{USER:session-id} %{NUMBER:session-id-num}\]"}
}
grok {
match => { "message" => "\Subscription Id \: %{BASE16NUM:subcription-id:int}"}
}
grok {
match => { "message" => "\Event time \: %{TIMESTAMP_ISO8601:event-time}"}
}
#grok {
# match => {"message" => "\<%{USERNAME:Statistic} \xmlns="%{QS:yang-model}\"\>"
# I want to accesss the http link with this config but it is not responding
# <memory-statistics xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-memory-oper">
#}
grok {
match => {"message" => "\<%{USERNAME:Statistic}\>"}
}
mutate {
remove_field => ["headers", "host_name", "session-id","message"]
}
date {
match => ["timestamp","dd/MMM/yyyy:HH:mm:ss Z"]
}
xml {
namespaces => {
"xmlns" => "http://cisco.com/ns/yang/Cisco-IOS-XE-memory-oper"
}
#not even the namspace option is working to access the http link
source => "message"
target => "xml_content"
xpath => [
"/*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='name']/text()" , "name" ,
"/*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='total-memory']/text()" , "total-memory",
"/*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='used-memory']/text()" , "used-memory",
"/*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='free-memory']/text()" , "free-memory" ,
"/*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='lowest-memory']/text()" , "lowest-memory" ,
"/*[name()='datastore-contents-xml']/*[name()='memory-statistics']/*[name()='memory-statistic'][1]/*[name()='highest-memory']/text()" , "highest-memory"
]
#logstash is not dectecting any of these xpaths in the config
}
mutate {
convert => {
"total-memory" => "integer"
"used-memory" => "integer"
"free-memory" => "integer"
"lowest-memory" => "integer"
"highest-memory" => "integer"
}
}
}
output {
stdout {
codec => rubydebug
}
file {
path => "%{type}_%{+dd_MM_yyyy}.log"
}
}
OUTPUT I am getting:
{
"ip_address" => "10.10.20.30",
"subcription-id" => 2147483650,
"event-time" => "2019-09-12 13:13:30.290000+00:00",
"host" => "127.0.0.1",
"Statistic" => "memory-statistic",
"type" => "sandbox-out",
"@version" => "1",
"@timestamp" => 2019-09-26T10:03:00.620Z,
"session-id-num" => "35"
}
Desired OUTPUT:
{
"ip_address" => "10.10.20.30",
"subcription-id" => 2147483650,
"event-time" => "2019-09-12 13:13:30.290000+00:00",
"host" => "127.0.0.1",
"Statistic" => "memory-statistic",
"type" => "sandbox-out",
"@version" => "1",
"@timestamp" => 2019-09-26T10:03:00.620Z,
"session-id-num" => "35"
"yang-model" => "http://cisco.com/ns/yang/Cisco-IOS-XE-memory-oper"
"name" => "Processor"
"total-memory" => 2238677360
"used-memory" => 340449924
"free-memory" => 1898227436
"lowest-usage" => 1897220640
"highest-usage" => 1264110388
}
I would appreciate any help really much, as I am a complete noob to Logstash.