Links status based on sysloghost and message

mushfiq · January 18, 2020, 5:19am

Hi,

I am new ELK stack & started testing it from last week. I must say, I just love it.

What I Have
Basic SOHO dual WAN routers, first link leased line and second link broadband (no static IP)
These routers support syslog.
Example logs that appears on Kibana are below
NSD FAIL WAN[2] (WAN1 is Leased line and WAN2 is Broadband)
NSD SUCCESS WAN[2]

What I did
Installed and configured ELK stack and Rsyslog on single Ubuntu server.
Redirected routers syslogs to this server.
Configured filter to drop unwanted logs.

What I want
Kibana dashboard where I can monitor both the links, based on "sysloghost" unique entries and it's log message (logs example above)

I tried to search but no luck, need help to accomplish this.

mushfiq · January 20, 2020, 9:05am

Elaborating

Is it possible to visualize based on last log message? Like, two types of logs appears on Kibana from routers as below.

Log: NSD SUCCESS WAN[1]
Log: NSD FAIL WAN[1]

So, when the last log from a router is "NSD SUCCESS WAN[1]" then it should appear in "up" list and if the last log is "NSD FAIL WAN[1]" then it should appear in "down" list

Guys I can provide more details if required.

azasypkin · January 20, 2020, 10:36am

Hi @mushfiq,

I think in general you need to consider entity-centric index approach. But if you just want to get the last record for every unique status message you could:

term split on the router so that you get one bucket for every unique router
top hits on timestamp field (you get the last N entries for each bucket and you can limit it just to one).

Something like this maybe (assuming I understood your use case correctly), but experiment and see what works for you and what doesn't:

Best,
Oleg

mushfiq · January 21, 2020, 10:29am

Hi @azasypkin

Thanks for reply

There is one issue, while creating visualization it does not show the "message", not aggregatable.

Screenshot from 2020-01-20 16-32-53

Also when search "NSD FAIL WAN[2]" on discover it does not show the result, I believe "[" "]" are causing it.

azasypkin · January 21, 2020, 11:03am

Your index mapping must be wrong, can you share it: https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-get-mapping.html

mushfiq · January 22, 2020, 7:43am

Hi @azasypkin

Sorry for bothering you, as I am not developer, I am having little difficulty rectifying mapping.

Output of " curl -XGET http://localhost:9200/logstash-2020.01.18?pretty "

{
"logstash-2020.01.18" : {
"aliases" : { },
"mappings" : {
"doc" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"norms" : false,
"type" : "text"
}
}
},
{
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
},
"norms" : false,
"type" : "text"
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword"
},
"facility" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"geoip" : {
"dynamic" : "true",
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
}
},
"host" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"message" : {
"type" : "text",
"norms" : false
},
"procid" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"programname" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"severity" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"sysloghost" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"tags" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"type" : {
"type" : "text",
"norms" : false,
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
},
"default" : {
"dynamic_templates" : [
{
"message_field" : {
"path_match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"norms" : false,
"type" : "text"
}
}
},
{
"string_fields" : {
"match" : "",
"match_mapping_type" : "string",
"mapping" : {
"fields" : {
"keyword" : {
"ignore_above" : 256,
"type" : "keyword"
}
},
"norms" : false,
"type" : "text"
}
}
}
],
"properties" : {
"@timestamp" : {
"type" : "date"
},
"@version" : {
"type" : "keyword"
},
"geoip" : {
"dynamic" : "true",
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "half_float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "half_float"
}
}
}
}
}
},
"settings" : {
"index" : {
"refresh_interval" : "5s",
"number_of_shards" : "5",
"provided_name" : "logstash-2020.01.18",
"creation_date" : "1579306701290",
"number_of_replicas" : "1",
"uuid" : "YN35vCHXR2CxIGONMygV0Q",
"version" : {
"created" : "6080699"
}
}
}
}
}

Indices being created automatically

Screenshot from 2020-01-22 13-02-32

I have created index in Kibana as below.

Screenshot from 2020-01-22 13-05-45

I am not able understand where should I make changes to get the mapping right.

mushfiq · January 22, 2020, 8:40am

{
  "logstash-2020.01.18" : {
"aliases" : { },
"mappings" : {
  "doc" : {
    "dynamic_templates" : [
      {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "norms" : false,
            "type" : "text"
          }
        }
      },
      {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            },
            "norms" : false,
            "type" : "text"
          }
        }
      }
    ],
    "properties" : {
      "@timestamp" : {
        "type" : "date"
      },
      "@version" : {
        "type" : "keyword"
      },
      "facility" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "geoip" : {
        "dynamic" : "true",
        "properties" : {
          "ip" : {
            "type" : "ip"
          },
          "latitude" : {
            "type" : "half_float"
          },
          "location" : {
            "type" : "geo_point"
          },
          "longitude" : {
            "type" : "half_float"
          }
        }
      },
      "host" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "message" : {
        "type" : "text",
        "norms" : false
      },
      "procid" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "programname" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "severity" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "sysloghost" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "tags" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      },
      "type" : {
        "type" : "text",
        "norms" : false,
        "fields" : {
          "keyword" : {
            "type" : "keyword",
            "ignore_above" : 256
          }
        }
      }
    }
  },
  "_default_" : {
    "dynamic_templates" : [
      {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "norms" : false,
            "type" : "text"
          }
        }
      },
      {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "fields" : {
              "keyword" : {
                "ignore_above" : 256,
                "type" : "keyword"
              }
            },
            "norms" : false,
            "type" : "text"
          }
        }
      }
    ],
    "properties" : {
      "@timestamp" : {
        "type" : "date"
      },
      "@version" : {
        "type" : "keyword"
      },
      "geoip" : {
        "dynamic" : "true",
        "properties" : {
          "ip" : {
            "type" : "ip"
          },
          "latitude" : {
            "type" : "half_float"
          },
          "location" : {
            "type" : "geo_point"
          },
          "longitude" : {
            "type" : "half_float"
          }
        }
      }
    }
  }
},
"settings" : {
  "index" : {
    "refresh_interval" : "5s",
    "number_of_shards" : "5",
    "provided_name" : "logstash-2020.01.18",
    "creation_date" : "1579306701290",
    "number_of_replicas" : "1",
    "uuid" : "YN35vCHXR2CxIGONMygV0Q",
    "version" : {
      "created" : "6080699"
    }
  }
}
  }
}

in better view.

mushfiq · January 22, 2020, 10:17am

Hi @azasypkin

I believe below template is being used.

{
  "logstash" : {
    "order" : 0,
    "version" : 60001,
    "index_patterns" : [
      "logstash-*"
    ],
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [
          {
            "message_field" : {
              "path_match" : "message",
              "mapping" : {
                "norms" : false,
                "type" : "text"
              },
              "match_mapping_type" : "string"
            }
          },
          {
            "string_fields" : {
              "mapping" : {
                "norms" : false,
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "ignore_above" : 256,
                    "type" : "keyword"
                  }
                }
              },
              "match_mapping_type" : "string",
              "match" : "*"
            }
          }
        ],
        "properties" : {
          "@timestamp" : {
            "type" : "date"
          },
          "geoip" : {
            "dynamic" : true,
            "properties" : {
              "ip" : {
                "type" : "ip"
              },
              "latitude" : {
                "type" : "half_float"
              },
              "location" : {
                "type" : "geo_point"
              },
              "longitude" : {
                "type" : "half_float"
              }
            }
          },
          "@version" : {
            "type" : "keyword"
          }
        }
      }
    },
    "aliases" : { }
  }
}

mushfiq · January 22, 2020, 10:44am

Made bellow changes

    "message_field" : {
      "path_match" : "message",
      "mapping" : {
        "norms" : false,
        "type" : "text",
        "fields" : {
          "keyword" : {
            "ignore_above" : 256,
            "type" : "keyword"
          }
        }   
      },

got below out put

{
  "acknowledged" : true
}

Deleted old index and created new one but still "message is not aggregatable.

azasypkin · January 23, 2020, 8:30am

So you updated message_field, and should have message_field.keyword. Don't you see it in Kibana? If not, try to refresh index pattern mapping cache like described here

Best,
Oleg

mushfiq · January 23, 2020, 10:36am

Thanks @azasypkin

It helped, yesterday I tried it but no luck, to day it worked.

Trying different visuals now.

Thanks once again

system · February 20, 2020, 10:36am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.