# Unable to establish connection between Logstash and Elasticsearch to parse iis log files

**URL:** <https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587>\
**Category:** Logstash\
**Created:** [February 6, 2018, 9:10am UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587 "2018-02-06T09:10:58Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![SRIpandu1729](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sripandu1729/32/27368_2.png) [@SRIpandu1729](https://discuss.elastic.co/u/SRIpandu1729)\
**Post date:** [February 6, 2018, 9:10am UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587/1 "2018-02-06T09:10:58Z")

</div>

OS : Ubuntu 16.04  
Installed : Elasticsearch, Kibana, Logstash (All 3 are of version 6.1)  
Log type : iis

This is my config file for Logstash

```
input {
    file {
        type => "iis-w3c"
        path => "/home/harsha/test.log"
        start_position => "beginning"
        sincedb_path => "/dev/null"
    }
}

filter {
    if [message] =~ "^#" {
        drop {}
}

grok {
    match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{WORD:serviceName} %{WORD:serverName} %{IP:serverIP} %{WORD:method} %{URIPATH:uriStem} %{NOTSPACE:uriQuery} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clientIP} %{NOTSPACE:protocolVersion} %{NOTSPACE:userAgent} %{NOTSPACE:cookie} %{NOTSPACE:referer} %{NOTSPACE:requestHost} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:win32response} %{NUMBER:bytesSent} %{NUMBER:bytesReceived} %{NUMBER:timetaken} %{IPORHOST:OriginalIP}"]
}

date {
    match => ["log_timestamp", "YYYY-MM-dd HH:mm:ss"]
    timezone => "Etc/UTC"
}

mutate {
    convert => ["bytesSent", "integer"]
    convert => ["bytesReceived", "integer"]
    convert => ["timetaken", "integer"]

    add_field => { "clientHostname" => "%{clientIP}" }

    remove_field => ["log_timestamp"]
}
}

output {
elasticsearch {
    embedded => false
    host => "localhost"
    port => 9200
    protocol => "http"
    index => "my_index"
}

stdout {codec => rubydebug}
}

```

This is in the correct location etc/logstash/conf.d

I have ES installed and localhost:9200 is giving me what it is supposed to.  
I have kibana installed as well and it is correctly configured to ES.

(Do I need to create the index beforehand in ES?, because I didn't.)

When I start Logstash using this "sudo systemctl start logstash.service" nothing is happening.

The test.log file is in the correct location and an example line of it is

```
2018-01-24 23:59:51 W3SVC11 happyweb 10.2.0.5 GET /blog/wp-content/uploads/2017/08/Mobile-accessories-min.jpg - 443 - 172.68.58.11 HTTP/1.1 Mozilla/5.0+(compatible;+MSIE+9.0;+Windows+NT+6.0;+Trident/5.0;++Trident/5.0) __cfduid=d422c9e6ed63a5957e526427f6341b8891516838382 - 200 0 0 50988 617 485 131.253.25.73

```

I have tested this with my format using grok debugger and the filters are fine according to it.  
So I don't know what is missing. Can someone please help me figure it out?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 6, 2018, 9:33am UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587/2 "2018-02-06T09:33:55Z")

</div>

Logstash is tailing test.log and only picks up new lines. Read the file input documentation and pay special attention to the `sincedb_path` and `start_position` options.

---

<div class="post-metadata">

**Author:** ![SRIpandu1729](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sripandu1729/32/27368_2.png) [@SRIpandu1729](https://discuss.elastic.co/u/SRIpandu1729)\
**Post date:** [February 6, 2018, 9:35am UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587/3 "2018-02-06T09:35:22Z")

</div>

Thanks @magnusbaeck , but what do I have to do to index already existing logs into ES. I even added a few new lines to the test.log file and even they are not being inserted into ES and I checked the status of Logstash and it is up and running.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [February 6, 2018, 5:53pm UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587/5 "2018-02-06T17:53:06Z")

</div>

Your elasticsearch output section looks like it was copied from a very old example, in fact `localhost` and `9200` are the defaults so they don't need to be specified. It should look something like this:

```auto
  elasticsearch {
    index => "my_index"
  }

```

IMO, the Logstash service is being continually restarted because the service is exiting with a configuration error message that is being logged to the journalctl logs.

You can download another copy of Logstash to a different folder and run it manually. What do you see in the console?

---

<div class="post-metadata">

**Author:** ![SRIpandu1729](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sripandu1729/32/27368_2.png) [@SRIpandu1729](https://discuss.elastic.co/u/SRIpandu1729)\
**Post date:** [February 7, 2018, 6:32am UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587/6 "2018-02-07T06:32:06Z")

</div>

Thanks for the answer, it helped but the reason why my logs weren't indexed is because of the data directory used by Logstash. So I changed the path and gave a new directory and it worked fine.

Anyways is there any hack around this? I am experimenting with the ElasticStack and each time I change my configuration I need to give Logstash a new directory for it's data. Thank you 🙂

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [February 8, 2018, 3:18pm UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587/7 "2018-02-08T15:18:59Z")

</div>

Read this [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#\_tracking\_of\_current\_position\_in\_watched\_files](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files)

---

<div class="post-metadata">

**Author:** ![SRIpandu1729](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sripandu1729/32/27368_2.png) [@SRIpandu1729](https://discuss.elastic.co/u/SRIpandu1729)\
**Post date:** [February 9, 2018, 7:24am UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587/8 "2018-02-09T07:24:47Z")

</div>

Thank you very much @guyboertje

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 9, 2018, 7:25am UTC](https://discuss.elastic.co/t/unable-to-establish-connection-between-logstash-and-elasticsearch-to-parse-iis-log-files/118587/9 "2018-03-09T07:25:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.