# Logstash shipping results

**URL:** <https://discuss.elastic.co/t/logstash-shipping-results/58168>\
**Category:** Logstash\
**Created:** [August 16, 2016, 4:54pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168 "2016-08-16T16:54:47Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![joshuaclark](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@joshuaclark](https://discuss.elastic.co/u/joshuaclark)\
**Post date:** [August 16, 2016, 4:54pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/1 "2016-08-16T16:54:47Z")

</div>

Hi,  
I'm trying to ship LocustIO results with logstash which look like the following:

10:09:31 Name # reqs # fails Avg Min Max | Median req/s  
10:09:31 --------------------------------------------------------------------------------------------------------------------------------------------  
10:09:31 GET / 683 0(0.00%) 222 212 392 | 220 9.10  
10:09:31 POST /modules 5 0(0.00%) 555 539 567 | 550 0.00  
10:09:31 GET /ports 62 0(0.00%) 548 536 572 | 550 0.10  
10:09:31 GET /ports/ 250 0(0.00%) 215 212 283 | 210 4.60  
10:09:31 --------------------------------------------------------------------------------------------------------------------------------------------  
10:09:31 Total 1000 0(0.00%) 13.80

10:09:33 Name # reqs 50% 66% 75% 80% 90% 95% 98% 99% 100%  
10:09:33 --------------------------------------------------------------------------------------------------------------------------------------------  
10:09:33 GET / 683 220 220 220 220 220 330 330 340 392  
10:09:33 POST /modules 5 550 570 570 570 570 570 570 570 567  
10:09:33 GET /ports 62 550 550 560 560 560 560 570 570 572  
10:09:33 GET /ports/ 250 210 220 220 220 220 220 220 230 283

I'm trying to find the best way to do this. I have the following configs setup for logstash.  
input {  
file {  
path =\> ["C:\Program Files (x86)\Jenkins\jobs\Locust2\builds\*\log"]  
start\_position =\> "beginning"  
}  
}

filter {  
csv { columns =\> ["Name", "# reqs", "# fails", "Avg", "Min", "Max", "Median", "req/s"] }  
}

output {  
stdout { }  
elasticsearch {  
template =\> "C:\Program Files\Logstash\logstash-2.3.4\bin\locust-mapping.json"  
template\_name =\> "locust"  
template\_overwrite =\> "false"  
hosts =\> ["localhost:9200"]  
user =\> "admin"  
password =\> "password"  
index =\> "locust-%{+YYYY.MM.dd}"  
}  
}

And the following mapping:

{  
"mappings": {  
"properties": {  
"Name": {  
"type": "long"  
},  
"# reqs": {  
"type": "integer"  
},  
"# fails": {  
"type": "integer"  
},  
"Avg": {  
"type": "integer"  
},  
"Min": {  
"type": "integer"  
},  
"Max": {  
"type": "integer"  
},  
"Median": {  
"type": "integer"  
},  
"req/s": {  
"type": "integer"  
}  
}  
}  
},  
"settings": {  
"index.refresh\_interval": "5s"  
},  
"template": "locust-\*"

Please let me know any suggestions you may have.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 5:04pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/2 "2016-08-16T17:04:38Z")

</div>

Is this working as it is and you just want improvement suggestions, or is something _not_ working?

---

<div class="post-metadata">

**Author:** ![joshuaclark](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@joshuaclark](https://discuss.elastic.co/u/joshuaclark)\
**Post date:** [August 16, 2016, 5:06pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/3 "2016-08-16T17:06:06Z")

</div>

Its not working as it is. No metrics are being shipped.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 5:10pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/4 "2016-08-16T17:10:41Z")

</div>

Logstash is probably tailing the input file. While testing you might want to set `sincedb_path => "nul"` to make sure Logstash starts from the top of the input file each time. See the file input's documentation. You might also need to adjust its `ignore_older` option.

---

<div class="post-metadata">

**Author:** ![joshuaclark](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@joshuaclark](https://discuss.elastic.co/u/joshuaclark)\
**Post date:** [August 16, 2016, 5:14pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/5 "2016-08-16T17:14:31Z")

</div>

Thanks for the suggestion! I'll give it a try.

---

<div class="post-metadata">

**Author:** ![joshuaclark](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@joshuaclark](https://discuss.elastic.co/u/joshuaclark)\
**Post date:** [August 16, 2016, 6:03pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/6 "2016-08-16T18:03:50Z")

</div>

It's still not shipping to ES. This is the results of the debug.

{:timestamp=\>"2016-08-16T12:55:59.076000-0500", :message=\>"Pushing flush onto pipeline", :level=\>:debug, :file=\>"/Program Files/Logstash/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb", :line=\>"458", :method=\>"flush"}  
{:timestamp=\>"2016-08-16T12:55:59.338000-0500", :message=\>"Flushing buffer at interval", :instance=\>"#\<LogStash::Outputs::ElasticSearch::Buffer:0x3000e682 @operations\_mutex=#Mutex:0x3ec70ef2, @max\_size=500, @operations\_lock=#Java::JavaUtilConcurrentLocks::ReentrantLock:0x3af88df2, @submit\_proc=#\<Proc:0x4879c942@c:/Program Files/Logstash/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:57\>, @logger=#\<Cabin::Channel:0x4ecffe21 @metrics=#\<Cabin::Metrics:0x30a66d17 @metrics\_lock=#Mutex:0x1fd5fb86, @metrics={}, @channel=#\<Cabin::Channel:0x4ecffe21 ...\>\>, @subscriber\_lock=#Mutex:0x19e002e7, @level=:debug, @subscribers={12644=\>#\<Cabin::Subscriber:0x2ba2c7c3 @output=#\<Cabin::Outputs::IO:0x1b9b00fa @io=#\<File:locust.log\>, @lock=#Mutex:0x2f5ce54\>, @options={}\>}, @data={}\>, @last\_flush=2016-08-16 12:55:58 -0500, @flush\_interval=1, @stopping=#Concurrent::AtomicBoolean:0x5404c0bf, @buffer=[], @flush\_thread=#\<Thread:0x69a59923 run\>\>", :interval=\>1, :level=\>:debug, :file=\>"/Program Files/Logstash/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/buffer.rb", :line=\>"90", :method=\>"interval\_flush"}  
{:timestamp=\>"2016-08-16T12:56:00.344000-0500", :message=\>"Flushing buffer at interval", :instance=\>"#\<LogStash::Outputs::ElasticSearch::Buffer:0x3000e682 @operations\_mutex=#Mutex:0x3ec70ef2, @max\_size=500, @operations\_lock=#Java::JavaUtilConcurrentLocks::ReentrantLock:0x3af88df2, @submit\_proc=#\<Proc:0x4879c942@c:/Program Files/Logstash/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:57\>, @logger=#\<Cabin::Channel:0x4ecffe21 @metrics=#\<Cabin::Metrics:0x30a66d17 @metrics\_lock=#Mutex:0x1fd5fb86, @metrics={}, @channel=#\<Cabin::Channel:0x4ecffe21 ...\>\>, @subscriber\_lock=#Mutex:0x19e002e7, @level=:debug, @subscribers={12644=\>#\<Cabin::Subscriber:0x2ba2c7c3 @output=#\<Cabin::Outputs::IO:0x1b9b00fa @io=#\<File:locust.log\>, @lock=#Mutex:0x2f5ce54\>, @options={}\>}, @data={}\>, @last\_flush=2016-08-16 12:55:59 -0500, @flush\_interval=1, @stopping=#Concurrent::AtomicBoolean:0x5404c0bf, @buffer=[], @flush\_thread=#\<Thread:0x69a59923 run\>\>", :interval=\>1, :level=\>:debug, :file=\>"/Program Files/Logstash/logstash-2.3.4/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/buffer.rb", :line=\>"90", :method=\>"interval\_flush"}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 16, 2016, 6:08pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/7 "2016-08-16T18:08:15Z")

</div>

"Pushing flush onto pipeline" messages are not interesting.

A few additional things to look into:

- Use forward slashes instead of backslashes in the filename pattern.
- Look for log messages about "discover" in the log. They'll tell you whether the filename pattern expands to the correct file(s).

---

<div class="post-metadata">

**Author:** ![joshuaclark](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@joshuaclark](https://discuss.elastic.co/u/joshuaclark)\
**Post date:** [August 16, 2016, 7:11pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/8 "2016-08-16T19:11:09Z")

</div>

Changing the backslashes to forward slashes made an error show itself. It didn't like me putting in the template location. After commenting that out it built the index. However, the metrics are being shipped because they aren't comma separated in the logs.

---

<div class="post-metadata">

**Author:** ![joshuaclark](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@joshuaclark](https://discuss.elastic.co/u/joshuaclark)\
**Post date:** [August 16, 2016, 9:56pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/9 "2016-08-16T21:56:09Z")

</div>

I'm trying this instead  
input {  
file {  
path =\> ["C:/Program Files (x86)/Jenkins/jobs/Locust2/builds/\*/log"]  
start\_position =\> "beginning"  
sincedb\_path =\> "nul"  
}  
}

filter {  
grok {  
match =\> ["message", "(?.{68})(?\<# reqs\>.{7})(?\<# fails\>.{13})(?.{8})(?.{9})(?.{7})(?\<req/s\>.{7})"]  
}  
}  
mutate {  
strip =\> {  
"Name",  
"# reqs",  
"# fails",  
"Avg"  
"Min",  
"Max",  
"Median"  
"req/s"  
}  
}  
However,  
I get this error  
reason=\>"Expected one of #, input, filter, output at line 14, column 5 (byte 323) after ",

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 17, 2016, 5:31am UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/10 "2016-08-17T05:31:44Z")

</div>

Your braces are off. You're closing the filter block after the grok filter instead of after the mutate filter.

---

<div class="post-metadata">

**Author:** ![joshuaclark](https://avatars.discourse-cdn.com/v4/letter/j/5daacb/32.png) [@joshuaclark](https://discuss.elastic.co/u/joshuaclark)\
**Post date:** [August 17, 2016, 5:45pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/11 "2016-08-17T17:45:47Z")

</div>

The following almost works however they aren't coming thru as numbers. Is there a way to configure that with these settings?

filter {  
grok {  
match =\> ["message", "(?.{64})(?.{10})(?.{13})(?.{8})(?.{8})(?.{8})(?.{9})(?\<req\s\>.{5})"]  
}

```
mutate {
	strip => ["Name", "reqs", "fails", "Avg", "Min", "Max", "Median", "req\s"]
    }
}	

```

I tried this but it doesn't seem to read the spaces.

filter {  
grok {  
match =\> ["message", "(?.{64})(?[0-9].{10})(?.{13})(?[0-9].{8})(?[0-9].{8})(?[0-9].{8})(?[0-9].{9})(?\<req\s\>[0-9].{5})"]  
}

```
mutate {
	strip => ["Name", "reqs", "fails", "Avg", "Min", "Max", "Median", "req\s"]
    }
}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 17, 2016, 7:49pm UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/12 "2016-08-17T19:49:09Z")

</div>

If you format your configuration as code with the `</>` button you won't find that important details in your regular expressions are stripped off.

Anyway, you can use the mutate filter's convert option to turn strings into numbers.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:42am UTC](https://discuss.elastic.co/t/logstash-shipping-results/58168/13 "2017-07-06T04:42:58Z")

</div>