Jmeter parsing by date (ELK)

Elastic Stack Logstash
1

Hi All,

I have been struggling with getting Jmeter logs (csv) into an ELK stack and parsing them by the timestamp in the provided file. I can get the logs in fine, but they're indexed by the time logstash has processed and pushed them to elastic, not by the time in my data set.

This is a sample of the data:

data.JPG1273×146 51.3 KB 

input {
        file {
                path => "/home/qassim/results.csv"
                start_position => "beginning"
            }
}
filter {
  csv {
      columns => ["@timestamp", "elapsed", "label", "responseCode", "threadName",
                    "success", "bytes", "grpThreads", "allThreads", "Latency",
                    "SampleCount", "ErrorCount", "Hostname"]
      separator => ","
     }
  date {
        locale => "en"
        match => [ "@timestamp", "yyyy-MM-dd'T'HH:mm:ss.SSS" ]
        timezone => "Europe/London"
      }

}
output {
    elasticsearch {
        action => "index"
        hosts => "localhost:9200"
        index => "logstash-jmeter-results-%{+YYYY.MM.dd}"
        workers => 1
    }
    stdout {}
}

I've tried using a mapping template for elasticsearch, with no luck. Mapping template:

{
    "template": "logstash-jmeter-results-*",
    "settings": {
        "number_of_shards": 1,
        "number_of_replicas": 0,
        "index.refresh_interval": "5s"
    },
    "mappings": {
        "logs": {
            "properties": {
                "@timestamp": {
                    "type": "date",
                    "format": "yyyy-MM-dd'T'HH:mm:ss.SSS"
                },
                "elapsed": {
                    "type": "long"
                },
                "label": {
                    "type": "string"
                },
                "responseCode": {
                    "type": "integer"
                },
                "threadName": {
                    "type": "string"
                },
                "success": {
                    "type": "boolean"
                },
                "bytes": {
                    "type": "long"
                },
                "grpThreads": {
                    "type": "long"
                },
                "allThreads": {
                    "type": "long"
                },
                "Latency": {
                    "type": "long"
                },
                "SampleCount": {
                    "type": "long"
                },
                "ErrorCount": {
                    "type": "long"
                },
                "Hostname": {
                    "type": "string"
                }
            }

I'm probably missing something stupid, but could anyone shed any light on where i'm perhaps going wrong?

Thanks,
Qassim

2

Unrelated error, the config above was not what I was going for - returned back to my original (and what I had assumed to be correct config at the time) and it worked as I expected it to. My issues were caused by something outside of the ELK stack - my own fault :slight_smile: