# Is Logstash working? **URL:** \ **Category:** Logstash\ **Created:** [April 22, 2018, 6:23pm UTC](https://discuss.elastic.co/t/is-logstash-working/129017 "2018-04-22T18:23:49Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![lowdmt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lowdmt/32/30300_2.png) [@lowdmt](https://discuss.elastic.co/u/lowdmt)\ **Post date:** [April 22, 2018, 6:23pm UTC](https://discuss.elastic.co/t/is-logstash-working/129017/1 "2018-04-22T18:23:49Z")
Hi, I'm new to ELK and Docker (learning curve). I'm struggling to get data into Elasticsearch. Using a .csv as a test. My docker-compose file: version: '2' services: elasticsearch: build: context: elasticsearch/ volumes: - ./elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro ports: - "9200:9200" - "9300:9300" environment: ES\_JAVA\_OPTS: "-Xmx256m -Xms256m" networks: - elk logstash: build: context: logstash/ volumes: - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro - ./logstash/pipeline:/usr/share/logstash/pipeline:ro ports: - "6000:6000" environment: LS\_JAVA\_OPTS: "-Xmx256m -Xms256m" networks: - elk depends\_on: - elasticsearch kibana: build: context: kibana/ volumes: - ./kibana/config/:/usr/share/kibana/config:ro ports: - "5601:5601" networks: - elk depends\_on: - elasticsearch networks: elk: driver: bridge The logs from logstash: | date | stream | content | | --- | --- | --- | | 2018-04-22 18:13:13 | stdout | [2018-04-22T18:13:13,459][INFO][logstash.agent] Pipelines running {:count=\>1, :pipelines=\>["main"]} | | 2018-04-22 18:13:12 | stdout | [2018-04-22T18:13:12,906][INFO][logstash.pipeline] Pipeline started succesfully {:pipeline\_id=\>"main", :thread=\>"#\"} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,705][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[http://192.168.1.231:9200](http://192.168.1.231:9200)"]} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,596][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"template"=\>"logstash-_", "version"=\>60001, "settings"=\>{"index.refresh\_interval"=\>"5s"}, "mappings"=\>{"default"=\>{"dynamic\_templates"=\>[{"message\_field"=\>{"path\_match"=\>"message", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false}}}, {"string\_fields"=\>{"match"=\>"_", "match\_mapping\_type"=\>"string", "mapping"=\>{"type"=\>"text", "norms"=\>false, "fields"=\>{"keyword"=\>{"type"=\>"keyword", "ignore\_above"=\>256}}}}}], "properties"=\>{"@timestamp"=\>{"type"=\>"date"}, "@version"=\>{"type"=\>"keyword"}, "geoip"=\>{"dynamic"=\>true, "properties"=\>{"ip"=\>{"type"=\>"ip"}, "location"=\>{"type"=\>"geo\_point"}, "latitude"=\>{"type"=\>"half\_float"}, "longitude"=\>{"type"=\>"half\_float"}}}}}}}} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,548][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>nil} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,540][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,539][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>nil} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,521][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://192.168.1.231:9200/](http://192.168.1.231:9200/)"} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,504][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://192.168.1.231:9200/](http://192.168.1.231:9200/), :path=\>"/"} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,502][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://192.168.1.231:9200/](http://192.168.1.231:9200/)]}} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,431][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//192.168.1.231:9200](https://192.168.1.231:9200)"]} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,359][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,343][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>nil} | | 2018-04-22 18:13:11 | stdout | [2018-04-22T18:13:11,141][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://192.168.1.231:9200/](http://192.168.1.231:9200/)"} | | 2018-04-22 18:13:10 | stdout | [2018-04-22T18:13:10,563][INFO][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck\_url=\>[http://192.168.1.231:9200/](http://192.168.1.231:9200/), :path=\>"/"} | | 2018-04-22 18:13:10 | stdout | [2018-04-22T18:13:10,535][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>[], :added=\>[[http://192.168.1.231:9200/](http://192.168.1.231:9200/)]}} | | 2018-04-22 18:13:09 | stdout | [2018-04-22T18:13:09,151][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>4, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50} | | 2018-04-22 18:12:54 | stdout | [2018-04-22T18:12:54,633][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600} | | 2018-04-22 18:12:53 | stdout | [2018-04-22T18:12:53,548][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.2.2"} | | 2018-04-22 18:12:51 | stdout | [2018-04-22T18:12:51,857][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified | | 2018-04-22 18:12:50 | stdout | [2018-04-22T18:12:50,230][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"} | | 2018-04-22 18:12:50 | stdout | [2018-04-22T18:12:50,177][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"} | | 2018-04-22 18:12:49 | stdout | Sending Logstash's logs to /usr/share/logstash/logs which is now configured via log4j2.properties | I presume thats working as expected? 192.168.1.231:9200/\_cluster/health?pretty=true { "cluster\_name" : "docker-cluster", "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 0, "active\_shards" : 0, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 100.0 [http://192.168.1.231:9200/\_cat/indices?v](http://192.168.1.231:9200/_cat/indices?v) health status index uuid pri rep docs.count docs.deleted store.size pri.store.size (yep - blank) Any help or guidance apprevaited ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 22, 2018, 8:08pm UTC](https://discuss.elastic.co/t/is-logstash-working/129017/2 "2018-04-22T20:08:34Z")
And what's the contents of ./logstash/pipeline? ---
**Author:** ![lowdmt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lowdmt/32/30300_2.png) [@lowdmt](https://discuss.elastic.co/u/lowdmt)\ **Post date:** [April 22, 2018, 8:40pm UTC](https://discuss.elastic.co/t/is-logstash-working/129017/3 "2018-04-22T20:40:11Z")
logstash.conf input { file { path =\> "/usr/share/logstash/Zur.csv" start\_position =\> beginning # to read from the beginning of file sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," columns =\> ["snhID", "SnhAlertedAt", "LastChangedAt", "Severity", "Account", "ElementName", "ClassDisplayName", "ClassName", "EventName", "EventText", "Owner", "Ticket", "RootAction"] } } ## Add your filters / logstash plugins configuration here output { elasticsearch { hosts =\> "192.168.1.231:9200" manage\_template =\> false index =\> "csv\_index" ``` } ``` stdout { codec =\> rubydebug } } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [April 23, 2018, 7:17am UTC](https://discuss.elastic.co/t/is-logstash-working/129017/4 "2018-04-23T07:17:47Z")
Is /usr/share/logstash/Zur.csv really available **inside** the Docker container? ---
**Author:** ![lowdmt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lowdmt/32/30300_2.png) [@lowdmt](https://discuss.elastic.co/u/lowdmt)\ **Post date:** [April 23, 2018, 7:37pm UTC](https://discuss.elastic.co/t/is-logstash-working/129017/5 "2018-04-23T19:37:11Z")
Hi Magnus. That was exactly my problem. It wasn't mounted so the file wasn't inside the docker container. Copied the file into the container and it worked. Kicking myself. Thanks Magnus ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 21, 2018, 7:37pm UTC](https://discuss.elastic.co/t/is-logstash-working/129017/6 "2018-05-21T19:37:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.