# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=723 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 724 --- ## [Logstash OutofDirectMemoryError](https://discuss.elastic.co/t/logstash-outofdirectmemoryerror/139649)
**Author:** [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\ **Replies:** 2\ **Last updated:** [July 12, 2018, 4:48pm UTC](https://discuss.elastic.co/t/logstash-outofdirectmemoryerror/139649 "2018-07-12T16:48:12Z")
I'm using Logstash to receive logs from Filebeat and push to ElasticSearch. Currently there are some errors in the log. But it seems that ElasticSearch is able to receive the data. However, as there is lots of data flowi… --- ## [Service paused error when I try to start the service](https://discuss.elastic.co/t/service-paused-error-when-i-try-to-start-the-service/139795)
**Author:** [@Ike143](https://discuss.elastic.co/u/Ike143)\ **Replies:** 0\ **Last updated:** [July 12, 2018, 4:41pm UTC](https://discuss.elastic.co/t/service-paused-error-when-i-try-to-start-the-service/139795 "2018-07-12T16:41:30Z")
Greetings Elastic Community, I installed Logstash using the NSSM using the same process I did when I installed kibana and elasticsearch. I get this error anytime I try to start the service via "Services" in windows. … --- ## [Why I am seeing type 1 in my logs from logstash](https://discuss.elastic.co/t/why-i-am-seeing-type-1-in-my-logs-from-logstash/139462)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 5\ **Last updated:** [July 12, 2018, 4:28pm UTC](https://discuss.elastic.co/t/why-i-am-seeing-type-1-in-my-logs-from-logstash/139462 "2018-07-12T16:28:39Z")
logstash was configured to receive logs on port 517 and type was defined as "websense". in elasticsearch I am seeing two type values. Logs with value "1" and the other logs with value "websense". Why I am see type ="1"? … --- ## [Remote file beat data](https://discuss.elastic.co/t/remote-file-beat-data/139707)
**Author:** [@Chiranjeevi\_Medichar](https://discuss.elastic.co/u/Chiranjeevi_Medichar)\ **Replies:** 1\ **Last updated:** [July 12, 2018, 3:20pm UTC](https://discuss.elastic.co/t/remote-file-beat-data/139707 "2018-07-12T15:20:20Z")
Pipeline\_id:main Plugin: \5044, host=\>"51.15.13.78", id=\>"0e8bb450290c27073038b98105c55144722f02f7064cb4fc49a8d0e1f7cfc3c4", enable\_metric=\>true, codec=\>\"plain\_… --- ## [Logstash cannot accept the events](https://discuss.elastic.co/t/logstash-cannot-accept-the-events/139646)
**Author:** [@a\_zentavr](https://discuss.elastic.co/u/a_zentavr)\ **Replies:** 3\ **Last updated:** [July 12, 2018, 2:53pm UTC](https://discuss.elastic.co/t/logstash-cannot-accept-the-events/139646 "2018-07-12T14:53:21Z")
Since some time logstash cannot accept the events from other hosts. It just fills the disk with logs of: \[2018-07-12T01:14:05,241\]\[WARN \]\[logstash.outputs.elasticsearch\] Could not index event to Elasticsearch. {:status=\>… --- ## [Missing records from mysql](https://discuss.elastic.co/t/missing-records-from-mysql/139738)
**Author:** [@Sunil\_Khullar](https://discuss.elastic.co/u/Sunil_Khullar)\ **Replies:** 1\ **Last updated:** [July 12, 2018, 1:58pm UTC](https://discuss.elastic.co/t/missing-records-from-mysql/139738 "2018-07-12T13:58:59Z")
While fetching records from mysql randomly about 40 - 50% records are getting missed. in the case below I have fetched 10 records only 5 were imported.... schedule =\> "\* \* \* \* \*" statement =\> "select \* from tbl\_activi… --- ## [Aggregate logs when fields are common in events](https://discuss.elastic.co/t/aggregate-logs-when-fields-are-common-in-events/139768)
**Author:** [@saad](https://discuss.elastic.co/u/saad)\ **Replies:** 0\ **Last updated:** [July 12, 2018, 1:41pm UTC](https://discuss.elastic.co/t/aggregate-logs-when-fields-are-common-in-events/139768 "2018-07-12T13:41:56Z")
How can I aggregate events in logstash whenever let say the sourceAddress, username and sitename fields are equal in events and be aggregated in one event for example the events below: 10.40.1.10 - James - facebook.com … --- ## [Merge different source (sql) to one index](https://discuss.elastic.co/t/merge-different-source-sql-to-one-index/139519)
**Author:** [@christian\_ha](https://discuss.elastic.co/u/christian_ha)\ **Replies:** 2\ **Last updated:** [July 12, 2018, 1:14pm UTC](https://discuss.elastic.co/t/merge-different-source-sql-to-one-index/139519 "2018-07-12T13:14:49Z")
I have two different sources. One is the oracle database of our cmdb and the other is the mysql database of ocs inventory. In the first source my document\_id is an ciid like this 'CI00001' in the second source my id is a… --- ## [Grok regex help](https://discuss.elastic.co/t/grok-regex-help/139597)
**Author:** [@DanV](https://discuss.elastic.co/u/DanV)\ **Replies:** 1\ **Last updated:** [July 12, 2018, 9:19am UTC](https://discuss.elastic.co/t/grok-regex-help/139597 "2018-07-12T09:19:27Z")
Hello, i'm parsing a firewall log that has it's fields separated by " ; ", using this to my advantage i decided to apply the csv filter to the log using ";" as my separator. The problem appears when one of the columns … --- ## [Getting \_grokparsefailure in identical docs](https://discuss.elastic.co/t/getting-grokparsefailure-in-identical-docs/139507)
**Author:** [@PolterFox](https://discuss.elastic.co/u/PolterFox)\ **Replies:** 3\ **Last updated:** [July 12, 2018, 9:17am UTC](https://discuss.elastic.co/t/getting-grokparsefailure-in-identical-docs/139507 "2018-07-12T09:17:56Z")
Hey guys, I have a problem. I use grok to parsing the document for special data, my problem with this is that I am facing a \_grokparsefailure in some documents. When I compare them with documents where I get the result … --- ## [Logstash not receiving beats input](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652)
**Author:** [@bobbytan](https://discuss.elastic.co/u/bobbytan)\ **Replies:** 6\ **Last updated:** [July 12, 2018, 9:15am UTC](https://discuss.elastic.co/t/logstash-not-receiving-beats-input/138652 "2018-07-12T09:15:44Z")
Hello, i have no issues importing data from beats to elasticsearch, however after changing output to logstash in the beats config, logstash do not import data to elasticsearch. No stdout is printed in the shell running … --- ## [Last\_run\_metadata\_path not updating](https://discuss.elastic.co/t/last-run-metadata-path-not-updating/139596)
**Author:** [@Sunil\_Khullar](https://discuss.elastic.co/u/Sunil_Khullar)\ **Replies:** 4\ **Last updated:** [July 12, 2018, 8:57am UTC](https://discuss.elastic.co/t/last-run-metadata-path-not-updating/139596 "2018-07-12T08:57:18Z")
First day with logstash (so excuse stupid posts) (using version 6.2.3 on Windows server) Importing from Mysql - facing 2 issues last\_run\_metadata\_path not updating , remains 0 after every query (though the file time… --- ## [Getting two values in one key in kv filter](https://discuss.elastic.co/t/getting-two-values-in-one-key-in-kv-filter/139555)
**Author:** [@Sakshi\_Aggarwal](https://discuss.elastic.co/u/Sakshi_Aggarwal)\ **Replies:** 2\ **Last updated:** [July 12, 2018, 6:44am UTC](https://discuss.elastic.co/t/getting-two-values-in-one-key-in-kv-filter/139555 "2018-07-12T06:44:00Z")
Greetings Everyone! :slight\_smile: I have attached my output. I have got two Timestamps in my Log file , both of different format . Please help me with the code so that I can achieve either of the following : To … --- ## [Unable to load csv into elastic through logstash](https://discuss.elastic.co/t/unable-to-load-csv-into-elastic-through-logstash/139595)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 3\ **Last updated:** [July 12, 2018, 4:51am UTC](https://discuss.elastic.co/t/unable-to-load-csv-into-elastic-through-logstash/139595 "2018-07-12T04:51:43Z")
input{ file{ start\_position =\> "beginning" path=\>"C:\\Users\\Gokul\\Desktop\\tes.csv" } } output { elasticsearch{ hosts=\>localhost index=\>"test\_prod" } } i have deleted the .null file in logstash think ie(since\_db\_… --- ## [Multiple Logstash Nodes](https://discuss.elastic.co/t/multiple-logstash-nodes/139242)
**Author:** [@Naruhodou](https://discuss.elastic.co/u/Naruhodou)\ **Replies:** 6\ **Last updated:** [July 11, 2018, 10:37pm UTC](https://discuss.elastic.co/t/multiple-logstash-nodes/139242 "2018-07-11T22:37:17Z")
Is is possible to create multiple logstash nodes running the same pipeline to provide high availability? Like if one of the logstash node is down, the beat will push data into another node to make it work. I look at thi… --- ## [Throttle the alerts](https://discuss.elastic.co/t/throttle-the-alerts/139280)
**Author:** [@soumya1](https://discuss.elastic.co/u/soumya1)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 9:45pm UTC](https://discuss.elastic.co/t/throttle-the-alerts/139280 "2018-07-11T21:45:23Z")
Log : 2018-07-10 06:02:22,989 \[myid:1\] - ERROR Processing ruok command from /127.0.0.1:33542 When the same error comes with a different timestamp i am unable to throttle the alerts In the filter i have if \[log\] =~ "… --- ## [Logstash output not showing any data](https://discuss.elastic.co/t/logstash-output-not-showing-any-data/139473)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 2\ **Last updated:** [July 11, 2018, 9:04pm UTC](https://discuss.elastic.co/t/logstash-output-not-showing-any-data/139473 "2018-07-11T21:04:50Z")
i have an input file (ie csv file), ,i have used stdout in output but data are not displaying in logstash cmd.. input{ file{ path=\>"/home/elkuser/Hospital General Information.csv" } } output { stdout{} } want to… --- ## [Kafka plugin for Logstash not working](https://discuss.elastic.co/t/kafka-plugin-for-logstash-not-working/139630)
**Author:** [@Krishna2](https://discuss.elastic.co/u/Krishna2)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 9:01pm UTC](https://discuss.elastic.co/t/kafka-plugin-for-logstash-not-working/139630 "2018-07-11T21:01:44Z")
I am getting theses errors my Kafka version 2.12-1 Logstash version 6.3 Kafka input plugin version. 8.1.1 Kafka output plugin 7.1.1 my requirement read from Kafka. and write to Elasticsearch. Can somebody give corr… --- ## [Link Logstash Input plugins](https://discuss.elastic.co/t/link-logstash-input-plugins/139637)
**Author:** [@pfseth](https://discuss.elastic.co/u/pfseth)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 8:15pm UTC](https://discuss.elastic.co/t/link-logstash-input-plugins/139637 "2018-07-11T20:15:36Z")
Is there any way to link input plugins ? For example The result of exec -- is an input to --\> file input { exec { command =\> "cd /logs; tshark -r working.pcap -T ek \> working.json" interval =\> … --- ## [I cannot create document Ids for my imported data](https://discuss.elastic.co/t/i-cannot-create-document-ids-for-my-imported-data/138861)
**Author:** [@wrahmouni](https://discuss.elastic.co/u/wrahmouni)\ **Replies:** 3\ **Last updated:** [July 11, 2018, 4:54pm UTC](https://discuss.elastic.co/t/i-cannot-create-document-ids-for-my-imported-data/138861 "2018-07-11T16:54:55Z")
Hello, I'm new in using ELK and I'm looking for your help to solve a problem that I'm facing lately. To sum up I'm trying to import data from a database table that doesn't contain a primary key. The jdbc plugin was use… --- ## [Logstash pipeline error by filebeat](https://discuss.elastic.co/t/logstash-pipeline-error-by-filebeat/139605)
**Author:** [@amritesh\_mishra](https://discuss.elastic.co/u/amritesh_mishra)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 4:48pm UTC](https://discuss.elastic.co/t/logstash-pipeline-error-by-filebeat/139605 "2018-07-11T16:48:40Z")
When i am hitting the pipeline of logstash through file beat we are getting this error. \[INFO \] 2018-07-11 16:04:10.935 \[defaultEventExecutorGroup-5-2\] BeatsHandler - \[local: 172...:5044, remote: 172...:45904\] Handling … --- ## [Logstash install error](https://discuss.elastic.co/t/logstash-install-error/138455)
**Author:** [@zekeriya\_tavlasoglu](https://discuss.elastic.co/u/zekeriya_tavlasoglu)\ **Replies:** 3\ **Last updated:** [July 11, 2018, 3:17pm UTC](https://discuss.elastic.co/t/logstash-install-error/138455 "2018-07-11T15:17:22Z")
Hi, when i try to install logstash on ubuntu, i am getting the error : what is the problem? can you help me. thanks. Zekeriya. sudo apt-get update && sudo apt-get install logstash Get:1 http://security.ubuntu.com… --- ## [Logstash loops on ingestion](https://discuss.elastic.co/t/logstash-loops-on-ingestion/138424)
**Author:** [@band](https://discuss.elastic.co/u/band)\ **Replies:** 5\ **Last updated:** [July 11, 2018, 3:14pm UTC](https://discuss.elastic.co/t/logstash-loops-on-ingestion/138424 "2018-07-11T15:14:23Z")
I am facing a problem with Logstash on a single node. I need to ingest a ~30MB xml file with the following format \ \...\ \ \...\ \...\ \ **Author:** [@yodog](https://discuss.elastic.co/u/yodog)\ **Replies:** 4\ **Last updated:** [July 11, 2018, 2:34pm UTC](https://discuss.elastic.co/t/increment-counter-if-document-already-exists/138604 "2018-07-11T14:34:59Z") lets say i have an index iptables-%{+YYYY.MM}, which holds iptables kernel generated logs. i would like to: 1- create the doc if the log line doesn't exists or 2- increment a field called counter if it already exists … --- ## [Issue while running logstash](https://discuss.elastic.co/t/issue-while-running-logstash/139487)
**Author:** [@Gokul\_15](https://discuss.elastic.co/u/Gokul_15)\ **Replies:** 2\ **Last updated:** [July 11, 2018, 2:11pm UTC](https://discuss.elastic.co/t/issue-while-running-logstash/139487 "2018-07-11T14:11:30Z")
bin/logstash -f testing.config Sending Logstash's logs to /home/elkuser/elk/logstash-6.3.1/logs which is now co … --- ## [How can I send logs from file to Elasticsearch using Logstash between two separated servers?](https://discuss.elastic.co/t/how-can-i-send-logs-from-file-to-elasticsearch-using-logstash-between-two-separated-servers/139521)
**Author:** [@white\_rabbit](https://discuss.elastic.co/u/white_rabbit)\ **Replies:** 0\ **Last updated:** [July 11, 2018, 9:07am UTC](https://discuss.elastic.co/t/how-can-i-send-logs-from-file-to-elasticsearch-using-logstash-between-two-separated-servers/139521 "2018-07-11T09:07:51Z")
I try to send logs from logs.csv file to elasticsearch using Logstash. In Elasticsearch I have index logs with type log. At the moment my logstash.conf looks in the way shown below. Logstash seems to be configured correc… --- ## [Can i get the file\_path according to sinceDb file](https://discuss.elastic.co/t/can-i-get-the-file-path-according-to-sincedb-file/139495)
**Author:** [@springCkWyy](https://discuss.elastic.co/u/springCkWyy)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 12:23pm UTC](https://discuss.elastic.co/t/can-i-get-the-file-path-according-to-sincedb-file/139495 "2018-07-11T12:23:03Z")
Like FileBeat， there is a Register File to record the file path. But Now I use logstash. All I can get just .SinceDB File， according to inode, can i get the filepath for logstash harvest? Thanks a lot! --- ## [Python code](https://discuss.elastic.co/t/python-code/139467)
**Author:** [@Chiranjeevi\_Medichar](https://discuss.elastic.co/u/Chiranjeevi_Medichar)\ **Replies:** 1\ **Last updated:** [July 11, 2018, 12:06pm UTC](https://discuss.elastic.co/t/python-code/139467 "2018-07-11T12:06:28Z")
can i use python code in filter --- ## [Need help to parse XML log in logstash](https://discuss.elastic.co/t/need-help-to-parse-xml-log-in-logstash/138973)
**Author:** [@iamlearner123](https://discuss.elastic.co/u/iamlearner123)\ **Replies:** 5\ **Last updated:** [July 11, 2018, 12:05pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-log-in-logstash/138973 "2018-07-11T12:05:32Z")
I recently started learning ELK but having hard time in understanding on how to parse XML data. Below is the sample log data. Any help would be appreciated in helping to write logstash configuration. \ **Author:** [@burnersk](https://discuss.elastic.co/u/burnersk)\ **Replies:** 0\ **Last updated:** [July 11, 2018, 11:43am UTC](https://discuss.elastic.co/t/how-to-convert-on-off-text-into-boolean-with-logstash/139549 "2018-07-11T11:43:13Z") I have a field, that contains either "" (empty string) or "on". I want to convert that field into an boolean, which treats "" (empty string) as false and "on" as true. As mentioned with the documentation, the mutate co… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=722) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=724)