# Extract field from url?

**URL:** <https://discuss.elastic.co/t/extract-field-from-url/181782>\
**Category:** Logstash\
**Created:** [May 20, 2019, 9:58am UTC](https://discuss.elastic.co/t/extract-field-from-url/181782 "2019-05-20T09:58:13Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Orest\_Gulman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orest_gulman/32/46463_2.png) [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\
**Post date:** [May 20, 2019, 9:58am UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/1 "2019-05-20T09:58:13Z")

</div>

I want to analyze AWS ELB logs, and I need to grep "instance" word from URL into a separate field, could you please provide an example, how to do it in filter section in logstash. Thanks

logs:  
`"GET https://cloude.company.com:443/instance HTTP/2.0"`  
`"GET https://cloude.company.com:443/instance/Web/Services/Service.html HTTP/2.0"`

---

<div class="post-metadata">

**Author:** ![sjabiulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjabiulla/32/48429_2.png) [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\
**Post date:** [May 20, 2019, 11:25am UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/2 "2019-05-20T11:25:40Z")

</div>

Try the below once

```
filter{
	grok{
		match => { "message" => "%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}"}
	}
	mutate {
     split => { "request" => "/" }
     add_field => { "instance_name" => "%{request[3]}" }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Orest\_Gulman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orest_gulman/32/46463_2.png) [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\
**Post date:** [May 20, 2019, 12:23pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/3 "2019-05-20T12:23:52Z")

</div>

In case of `match => [" message" "%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}"]`

request = `https://cloude.company.com:443/instance/App_Themes/Default/PageSpecific/images/warning.png`

split =\> { "request" =\> "/" }

`https:, , cloude.company.com:443, instance, App_Themes, Default, PageSpecific, images, warning.png`

when i tryed:  
`add_field => { "path_shortx" => "%{request[4]}" }`

```
[2019-05-20T12:18:40,455][FATAL][logstash.runner] An unexpected error occurred! {:error=>java.lang.IllegalStateException: org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: `request[4]`, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:235)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:295)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:270)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2019-05-20T12:18:40,471][ERROR][org.logstash.execution.WorkerLoop] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash.org.logstash.FieldReference$IllegalSyntaxException: Invalid FieldReference: request[4]
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 20, 2019, 1:21pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/4 "2019-05-20T13:21:46Z")

</div>

> [@Orest\_Gulman](#):
>
> "%{request[4]}"

Make that %{[request][4]} (although I think you want [3], not [4]).

---

<div class="post-metadata">

**Author:** ![Orest\_Gulman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orest_gulman/32/46463_2.png) [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\
**Post date:** [May 20, 2019, 1:34pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/5 "2019-05-20T13:34:32Z")

</div>

It separate like this:  
`"request" => [`  
`[0] "https:",`  
`[1] "",`  
`[2] "cloude-company.com:443",`  
`[3] "Instance",`  
`[4] "Web",`  
`[5] "HomePage",`  
`[6] "Widget",`  
`[7] "Data"`  
`]`

---

<div class="post-metadata">

**Author:** ![Orest\_Gulman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orest_gulman/32/46463_2.png) [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\
**Post date:** [May 20, 2019, 1:37pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/6 "2019-05-20T13:37:58Z")

</div>

Notice, that if I copied one field to another, and than split and add new field - it it does't work:  
`mutate {`  
`copy => {"request" => "request_split"}`  
`split => {"request_split" => "/"}`  
`add_field => { "instance" => "%{request_split[3]}" }`  
`}`

but if I split and add new field without copiying, it works!  
`mutate {`  
`split => {"request" => "/"}`  
`add_field => { "instance" => "%{request[3]}" }`  
`}`

And also works:  
`mutate {`  
`copy => {"request" => "request_split"}`  
`}`  
`mutate {`  
`split => {"request_split" => "/"}`  
`add_field => { "instance" => "%{request_split[3]}" }`  
`}`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 20, 2019, 2:11pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/7 "2019-05-20T14:11:33Z")

</div>

A mutate filter performs mutations in a [fixed](https://github.com/logstash-plugins/logstash-filter-mutate/blob/091207d5184af1f38626c9e24cce070674a410f5/lib/logstash/filters/mutate.rb#L247) order, which is unlikely to be the order you want.

---

<div class="post-metadata">

**Author:** ![sjabiulla](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sjabiulla/32/48429_2.png) [@sjabiulla](https://discuss.elastic.co/u/sjabiulla)\
**Post date:** [May 20, 2019, 3:48pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/8 "2019-05-20T15:48:46Z")

</div>

> [@Orest\_Gulman](#):
>
> "instance" word from URL into a separate field

I thought you want to take any value that's available in the place of instance.

What do you exactly want to achieve and place all the scenarios of your log for more clarity

---

<div class="post-metadata">

**Author:** ![Orest\_Gulman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orest_gulman/32/46463_2.png) [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\
**Post date:** [May 21, 2019, 2:37pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/9 "2019-05-21T14:37:46Z")

</div>

The idea is to take value from URL `https://cloude.company.com:443/ **instance** /Web/Services/Service.html`  
Filter that value, and add it into a field, and build all visualization based on that field. Will named that field as **instance** as an example.

This is my ELB LOG:  
`h2 2019-05-09T23:55:51.100101Z elb_name 100.100.100.10:59907 192.168.1.10:443 0.007 0.088 0.000 200 200 1474 36394 "POST https://cloude-company.com:443/instance/Web/HomePage/Widget/Data HTTP/2.0" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-west-1:100101001010:targetgroup/cloude/aa1001010010100a "Root=1-5cd4be07-1001010010100101001010010" "cloude.company.com" "session-reused" 8 2019-05-09T23:55:51.160000Z "forward" "-" "-"`

Create a filter:

```
filter {
if [type] == "elb" {
grok {
match => ["message", "%{WORD:connection} %{TIMESTAMP_ISO8601:timestamp} %{NOTSPACE:elb} %{IP:clientip}:%{INT:clientport:int} (?:(%{IP:backendip}:?:%{INT:backendport:int})|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:-|%{INT:elb_status_code:int}) (?:-|%{INT:backend_status_code:int}) %{INT:received_bytes:int} %{INT:sent_bytes:int} \"%{WORD:verb} %{NOTSPACE:request} HTTP/%{NUMBER:httpversion}\" \"(?:-|%{DATA:user_agent})\" (?:-|%{NOTSPACE:ssl_cipher}) (?:-|%{NOTSPACE:ssl_protocol})"]
}
if [path] and !([path] == "/") { 
mutate {
copy => {"path" => "path_split"}
}
mutate {
split => {"path_split" => "/"}
add_field => { "url" => "%{path_split[1]}" }
}
}
if [url] in ["instance1", "instance2", "instance3", "instance4", "instance5"] {
mutate {      
add_field => { "instance" => "%{url}" }
}
}
mutate {
remove_field => ["path_split"]
}
date {
match => ["timestamp", "ISO8601"]
}
geoip {
source => "clientip"
}
}
}

```

And it works for my needs. But then I discovered the simplest way, for the same log, I used:

```
grok {
match => ["message", "%{ELB_ACCESS_LOG} \"%{DATA:userAgent}\"( %{NOTSPACE:ssl_cipher} %{NOTSPACE:ssl_protocol})?"]
}
grok {
match => ["path", "/%{DATA:url}/%{GREEDYDATA:action_path}"]
}
if [url] in ["instance1", "instance2", "instance3", "instance4", "instance5"] {
mutate {      
add_field => { "instance" => "%{url}" }
}
}

```

but it has one cons, in case when URL comes ending by **instance**  
[https://cloude.company.com:443/](https://cloude.company.com:443/) **instance**  
it does't write a value into field **url**.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [May 21, 2019, 3:04pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/10 "2019-05-21T15:04:40Z")

</div>

You could use something like

```
grok { match => ["path", "/(?<url>[^/]+)($|/%{GREEDYDATA:action_path})"] }
```

---

<div class="post-metadata">

**Author:** ![Orest\_Gulman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/orest_gulman/32/46463_2.png) [@Orest\_Gulman](https://discuss.elastic.co/u/Orest_Gulman)\
**Post date:** [May 21, 2019, 3:46pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/11 "2019-05-21T15:46:29Z")

</div>

It works! thanks a lot.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 18, 2019, 3:55pm UTC](https://discuss.elastic.co/t/extract-field-from-url/181782/12 "2019-06-18T15:55:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.