# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1078 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 1079 --- ## [Extract the instagram public\_content of a person](https://discuss.elastic.co/t/extract-the-instagram-public-content-of-a-person/66161)
**Author:** [@Bhavana](https://discuss.elastic.co/u/Bhavana)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 7:53pm UTC](https://discuss.elastic.co/t/extract-the-instagram-public-content-of-a-person/66161 "2016-11-15T19:53:45Z")
can we extract the instagram public\_content of a person through logstash ? --- ## [Logstash container in Docker starts and then stops with no obvious error](https://discuss.elastic.co/t/logstash-container-in-docker-starts-and-then-stops-with-no-obvious-error/66132)
**Author:** [@Schadenfred](https://discuss.elastic.co/u/Schadenfred)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 4:02pm UTC](https://discuss.elastic.co/t/logstash-container-in-docker-starts-and-then-stops-with-no-obvious-error/66132 "2016-11-15T16:02:40Z")
I'm a newbie to Elkstack, and I'm having trouble setting up an Elkstack in Docker. The problem I'm having is that my container runs for about 20 seconds, then exits. I don't see any obvious errors in the log (see below… --- ## [Logstash : Failed parsing date from field", :field=\>"timestamp"](https://discuss.elastic.co/t/logstash-failed-parsing-date-from-field-field-timestamp/66123)
**Author:** [@zoram](https://discuss.elastic.co/u/zoram)\ **Replies:** 1\ **Last updated:** [November 15, 2016, 3:10pm UTC](https://discuss.elastic.co/t/logstash-failed-parsing-date-from-field-field-timestamp/66123 "2016-11-15T15:10:33Z")
Dear All, I have now again another logstash error in my configuration. I could add my index pattern ("filebeat-\*") in KIbana, but I could not see any field and any data are showing. elk\_1 | ==\> /var/log/logstash/lo… --- ## [Pubsub input plugin problem](https://discuss.elastic.co/t/pubsub-input-plugin-problem/66126)
**Author:** [@shushu](https://discuss.elastic.co/u/shushu)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 2:06pm UTC](https://discuss.elastic.co/t/pubsub-input-plugin-problem/66126 "2016-11-15T14:06:50Z")
Hi, Trying to use https://github.com/Wondermall/logstash-input-google-cloud-pubsub I think I was able to configure all needed, but I am getting error saying: \[2016-11-15T15:01:36,802\]\[ERROR\]\[logstash.agent \] f… --- ## [LS\_HEAP\_SIZE broken with logstash 5.0.0?](https://discuss.elastic.co/t/ls-heap-size-broken-with-logstash-5-0-0/65980)
**Author:** [@cg-max](https://discuss.elastic.co/u/cg-max)\ **Replies:** 4\ **Last updated:** [November 15, 2016, 1:33pm UTC](https://discuss.elastic.co/t/ls-heap-size-broken-with-logstash-5-0-0/65980 "2016-11-15T13:33:20Z")
Hi, With logstash 5.0.0 on centos 7.2 using the elastic rpm, our logstash runs like this: usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -… --- ## [When will csv autodetect\_column\_names functionality be released?](https://discuss.elastic.co/t/when-will-csv-autodetect-column-names-functionality-be-released/66095)
**Author:** [@Peter\_Andersson](https://discuss.elastic.co/u/Peter_Andersson)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 11:08am UTC](https://discuss.elastic.co/t/when-will-csv-autodetect-column-names-functionality-be-released/66095 "2016-11-15T11:08:04Z")
Hi there we just started using logstash we are using the 5.0 release, but when will autodetect\_column\_names for the csv filter be added? it was commited 4 months ago logstash-plugins/logstash-filter-csv@f0cd29f but its s… --- ## [Logstash Output Gelf TCP?](https://discuss.elastic.co/t/logstash-output-gelf-tcp/66091)
**Author:** [@Navarro](https://discuss.elastic.co/u/Navarro)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 10:44am UTC](https://discuss.elastic.co/t/logstash-output-gelf-tcp/66091 "2016-11-15T10:44:30Z")
Is there any way to send the events with logstash using the output gelf with TCP? Because due to the huge amount of messages I send i get a high rate of packet drops --- ## [Logstash locks the log file](https://discuss.elastic.co/t/logstash-locks-the-log-file/66077)
**Author:** [@VijayKarthikeyan](https://discuss.elastic.co/u/VijayKarthikeyan)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 9:04am UTC](https://discuss.elastic.co/t/logstash-locks-the-log-file/66077 "2016-11-15T09:04:40Z")
Hi, I am stashing the log files of database to elasticsearch using Logstash. When I try to stash the log files, logstash reads the file to process the lines. While reading, logstash locks the file and it does not all… --- ## [Cannot write to elasticsearch](https://discuss.elastic.co/t/cannot-write-to-elasticsearch/66073)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 8:54am UTC](https://discuss.elastic.co/t/cannot-write-to-elasticsearch/66073 "2016-11-15T08:54:33Z")
I have a logstash script that reads log and output to elasticsearch Logstash vesion: 2.2.2 Elasticsearch: 2.3.4 It runs and display fine when stdout { codec =\> rubydebug } is enabled. when I only enable elasticsea… --- ## [Logstash 2.4 output won't execute until stop](https://discuss.elastic.co/t/logstash-2-4-output-wont-execute-until-stop/65769)
**Author:** [@yiju](https://discuss.elastic.co/u/yiju)\ **Replies:** 6\ **Last updated:** [November 15, 2016, 8:40am UTC](https://discuss.elastic.co/t/logstash-2-4-output-wont-execute-until-stop/65769 "2016-11-15T08:40:28Z")
Hi, I am running logstash 2.4 on ubuntu 14.0.4. My config file is: input { file { path =\> "/home/spark/\*.log" start\_position =\> beginning codec =\> multiline { pattern =\> "(^\\d+… --- ## [Logstash kv{} issue with field without any value](https://discuss.elastic.co/t/logstash-kv-issue-with-field-without-any-value/66035)
**Author:** [@rolfberkenbosch](https://discuss.elastic.co/u/rolfberkenbosch)\ **Replies:** 3\ **Last updated:** [November 15, 2016, 8:27am UTC](https://discuss.elastic.co/t/logstash-kv-issue-with-field-without-any-value/66035 "2016-11-15T08:27:50Z")
Hi there, I have the following input: \<134\> device="SFW" date=2016-11-15 time=00:00:35 timezone="CET" device\_name="SG430" device\_id=S50049065BB50F7 log\_id=010302602002 log\_type="Firewall" log\_component="Appliance Acces… --- ## [Export from elasticsearch to csv problem. Where I am wrong?](https://discuss.elastic.co/t/export-from-elasticsearch-to-csv-problem-where-i-am-wrong/65558)
**Author:** [@Swina\_Allen](https://discuss.elastic.co/u/Swina_Allen)\ **Replies:** 3\ **Last updated:** [November 15, 2016, 7:18am UTC](https://discuss.elastic.co/t/export-from-elasticsearch-to-csv-problem-where-i-am-wrong/65558 "2016-11-15T07:18:44Z")
Hi have the following logstash config file: input { elasticsearch { hosts =\> \["localhost"\] index =\> "myindex" type =\> "mytype" query =\> '{"query" : { "match\_all" : {} }}' } } output { csv … --- ## [Check if sub-field exists](https://discuss.elastic.co/t/check-if-sub-field-exists/66046)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 2\ **Last updated:** [November 15, 2016, 7:08am UTC](https://discuss.elastic.co/t/check-if-sub-field-exists/66046 "2016-11-15T07:08:20Z")
I need an if else statement to check if a sub field exists. I want ed to know if a sub-field "city\_name" exists in main field dst\_geoip. Below is what I'm trying to do if !\[dst\_geoip\]\[city\_name\] {mutate{add\_field … --- ## [Udp listener died warning in logstash udp input plugin](https://discuss.elastic.co/t/udp-listener-died-warning-in-logstash-udp-input-plugin/65709)
**Author:** [@sha198989](https://discuss.elastic.co/u/sha198989)\ **Replies:** 1\ **Last updated:** [November 15, 2016, 6:43am UTC](https://discuss.elastic.co/t/udp-listener-died-warning-in-logstash-udp-input-plugin/65709 "2016-11-15T06:43:52Z")
HI all, i am getting logs from the syslog server. i have used logstash udp input plugin to get logs via the network. my logstash configuration is this input{ udp{ port =\> 5000 type=\> syslog } } i am using logstash2.2… --- ## [Logstash RabbitMq input plugin unable to find config file](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-unable-to-find-config-file/63733)
**Author:** [@Ajit\_Pawar](https://discuss.elastic.co/u/Ajit_Pawar)\ **Replies:** 4\ **Last updated:** [November 15, 2016, 6:40am UTC](https://discuss.elastic.co/t/logstash-rabbitmq-input-plugin-unable-to-find-config-file/63733 "2016-11-15T06:40:51Z")
I am using RabbitMq plugin for input in logstash and sending rabbitmq message on elasticsearch in outoput. I have created config file as below input { rabbitmq { host =\> "ip address of rabbitmq server" user… --- ## [How can I monitor Logstash?](https://discuss.elastic.co/t/how-can-i-monitor-logstash/66050)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 4\ **Last updated:** [November 15, 2016, 6:08am UTC](https://discuss.elastic.co/t/how-can-i-monitor-logstash/66050 "2016-11-15T06:08:40Z")
Hi, I am using Elastic GA 5 stack . The architecture is FB -\> Kafka -\> LS -\> ES . At the same time, LS is using other inputs using jdbc too. I have X-Pack installed. I would like to monitor the activity of Logstash in … --- ## [How to get data using Logstash Monitoring APIs and send data to Elasticsearch index?](https://discuss.elastic.co/t/how-to-get-data-using-logstash-monitoring-apis-and-send-data-to-elasticsearch-index/66053)
**Author:** [@elasticheart](https://discuss.elastic.co/u/elasticheart)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 5:23am UTC](https://discuss.elastic.co/t/how-to-get-data-using-logstash-monitoring-apis-and-send-data-to-elasticsearch-index/66053 "2016-11-15T05:23:30Z")
Hi, I got to know that there are experimental Logstash APIs to get Logstash data. I decided to experiment with it. I opened Kibana Dev Tools Console and entered a sample query GET /\_node/pipeline which returned; { … --- ## [Logstash (ELK) not streaming data real-time from dynamodb, duplicates upon restart & fetching in some random order](https://discuss.elastic.co/t/logstash-elk-not-streaming-data-real-time-from-dynamodb-duplicates-upon-restart-fetching-in-some-random-order/66022)
**Author:** [@ameya.aloni](https://discuss.elastic.co/u/ameya.aloni)\ **Replies:** 1\ **Last updated:** [November 15, 2016, 3:46am UTC](https://discuss.elastic.co/t/logstash-elk-not-streaming-data-real-time-from-dynamodb-duplicates-upon-restart-fetching-in-some-random-order/66022 "2016-11-15T03:46:39Z")
We're facing multiple issues while using ELK stack. We suspect they're Logstash Configuration issues. Issues are as follows: Logstash connected to Dynamodb streams isn't showing real-time changes. We even have an expl… --- ## [De\_dot error](https://discuss.elastic.co/t/de-dot-error/65635)
**Author:** [@gringo](https://discuss.elastic.co/u/gringo)\ **Replies:** 2\ **Last updated:** [November 15, 2016, 3:32am UTC](https://discuss.elastic.co/t/de-dot-error/65635 "2016-11-15T03:32:21Z")
When I tried to pass index of elasticsearch 1.7.5 to logstash 2.4 to elasticsearch 2.3.4 i keep getting the following exception \[my\_index\_name\]\[\[my\_index\_name\]\[3\]\] IllegalIndexShardStateException\[CurrentState\[CLOSED\] o… --- ## [Logstash: Extracting Information from differenty log lines into one event](https://discuss.elastic.co/t/logstash-extracting-information-from-differenty-log-lines-into-one-event/66044)
**Author:** [@Ramesh\_Janagam](https://discuss.elastic.co/u/Ramesh_Janagam)\ **Replies:** 0\ **Last updated:** [November 15, 2016, 3:17am UTC](https://discuss.elastic.co/t/logstash-extracting-information-from-differenty-log-lines-into-one-event/66044 "2016-11-15T03:17:10Z")
Hi All, I have case where information is being displayed in different lines. These lines are not even consecutive lines. Is there anyway to read these two and combine into single event? Example log line: where I wan… --- ## [\[SOLVED\] Logstash 5.0, downgraded kafka to 4.10. Could not find plugin name kafka](https://discuss.elastic.co/t/solved-logstash-5-0-downgraded-kafka-to-4-10-could-not-find-plugin-name-kafka/65807)
**Author:** [@carrot](https://discuss.elastic.co/u/carrot)\ **Replies:** 1\ **Last updated:** [November 15, 2016, 2:24am UTC](https://discuss.elastic.co/t/solved-logstash-5-0-downgraded-kafka-to-4-10-could-not-find-plugin-name-kafka/65807 "2016-11-15T02:24:06Z")
I believe I followed all instructions to downgrade kafka to support my Kafka 0.9.x cluster. Although seems like logstash won't find the downgraded plugin. bin/logstash-plugin remove logstash-input-kafka bin/logstash-p… --- ## [Can't parse the destination address in an iis log](https://discuss.elastic.co/t/cant-parse-the-destination-address-in-an-iis-log/65919)
**Author:** [@relentless](https://discuss.elastic.co/u/relentless)\ **Replies:** 4\ **Last updated:** [November 14, 2016, 7:51pm UTC](https://discuss.elastic.co/t/cant-parse-the-destination-address-in-an-iis-log/65919 "2016-11-14T19:51:07Z")
I'm trying to start a what feels like simple parse of an iis log file. Initially I just stole an example grok filter from google and after puzzling over the parse errors for the a while decided to start fresh. Its a def… --- ## [CSV filter last field add all information also with delimiter](https://discuss.elastic.co/t/csv-filter-last-field-add-all-information-also-with-delimiter/65877)
**Author:** [@Hans](https://discuss.elastic.co/u/Hans)\ **Replies:** 0\ **Last updated:** [November 12, 2016, 1:46pm UTC](https://discuss.elastic.co/t/csv-filter-last-field-add-all-information-also-with-delimiter/65877 "2016-11-12T13:46:52Z")
Hi, I am using the CSV filter however the file I have should only have 6 fields where the last field is a string. in this string field however the deliminator can occur, now this field is split again. How can I place all… --- ## [Logstash crash, ERROR Java heap space](https://discuss.elastic.co/t/logstash-crash-error-java-heap-space/65975)
**Author:** [@agonzalez](https://discuss.elastic.co/u/agonzalez)\ **Replies:** 1\ **Last updated:** [November 14, 2016, 3:43pm UTC](https://discuss.elastic.co/t/logstash-crash-error-java-heap-space/65975 "2016-11-14T15:43:32Z")
Logstash 5.0 crashed. Could this be insuficient memory? machine has 8GB RAM. free -m total used free shared buff/cache available Mem: 7823 2388 1975 3… --- ## [Grok pattern all match but still grokparsefailure](https://discuss.elastic.co/t/grok-pattern-all-match-but-still-grokparsefailure/65979)
**Author:** [@txisme](https://discuss.elastic.co/u/txisme)\ **Replies:** 1\ **Last updated:** [November 14, 2016, 2:47pm UTC](https://discuss.elastic.co/t/grok-pattern-all-match-but-still-grokparsefailure/65979 "2016-11-14T14:47:25Z")
Hey, so im parsing multiline logs using filebeat and logstash. The multline pattern is made in filebeat like this: multiline: pattern: '^Offset' negate: true match: after one complete log look… --- ## [Logstash input plugin for kafka](https://discuss.elastic.co/t/logstash-input-plugin-for-kafka/65629)
**Author:** [@sha198989](https://discuss.elastic.co/u/sha198989)\ **Replies:** 3\ **Last updated:** [November 14, 2016, 12:18pm UTC](https://discuss.elastic.co/t/logstash-input-plugin-for-kafka/65629 "2016-11-14T12:18:20Z")
i am having following error- kafka client threw exception, restarting {:exception=\>#\, :level this is the my config file in… --- ## [Is Logstash going to die?](https://discuss.elastic.co/t/is-logstash-going-to-die/65505)
**Author:** [@Patrick\_Salomon](https://discuss.elastic.co/u/Patrick_Salomon)\ **Replies:** 4\ **Last updated:** [November 14, 2016, 10:32am UTC](https://discuss.elastic.co/t/is-logstash-going-to-die/65505 "2016-11-14T10:32:52Z")
Hi there Since Elasticsearch is now able to replace Logstash (with the ingest nodes), and with the fact that Logstash is not running as the same kind of service that all other products in the stack, and finally that Lo… --- ## [Elasticsearch template in logstash does not load?](https://discuss.elastic.co/t/elasticsearch-template-in-logstash-does-not-load/65556)
**Author:** [@irom77](https://discuss.elastic.co/u/irom77)\ **Replies:** 1\ **Last updated:** [November 14, 2016, 10:31am UTC](https://discuss.elastic.co/t/elasticsearch-template-in-logstash-does-not-load/65556 "2016-11-14T10:31:20Z")
I have template which is in fact copy&paste of my current auto mappings with two field types change to geo\_point but when I try to load it, actual mappings do not change ? output { if \[type\] =="pa-traffic" { elastics… --- ## [Date Parsing ignored in Logstash--\>@timestamp not updated](https://discuss.elastic.co/t/date-parsing-ignored-in-logstash-timestamp-not-updated/65826)
**Author:** [@momoXD](https://discuss.elastic.co/u/momoXD)\ **Replies:** 2\ **Last updated:** [November 14, 2016, 10:03am UTC](https://discuss.elastic.co/t/date-parsing-ignored-in-logstash-timestamp-not-updated/65826 "2016-11-14T10:03:38Z")
I've got logs includiong timesamps like: "04/Nov/2016:00:00:16 +0100" I parse them via grok in a variable I call "apacheTime" and I want to use the date filter to parse them and overwrite "@timestamp". So I included: d… --- ## [Multiline not processing event until pipeline stops](https://discuss.elastic.co/t/multiline-not-processing-event-until-pipeline-stops/65561)
**Author:** [@espogian](https://discuss.elastic.co/u/espogian)\ **Replies:** 2\ **Last updated:** [November 14, 2016, 8:08am UTC](https://discuss.elastic.co/t/multiline-not-processing-event-until-pipeline-stops/65561 "2016-11-14T08:08:03Z")
Hi, with XML files with the following structure: \ \ ... \ And input config: codec =\> multiline { pattern =\> "^\<\\?xml .\*\\?\>" negate =\> true what =\> "previou… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1077) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=1079)