# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=934 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 935 --- ## [Logstash doesn't start when config is bigger than threshold](https://discuss.elastic.co/t/logstash-doesnt-start-when-config-is-bigger-than-threshold/95560)
**Author:** [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\ **Replies:** 1\ **Last updated:** [August 3, 2017, 9:26am UTC](https://discuss.elastic.co/t/logstash-doesnt-start-when-config-is-bigger-than-threshold/95560 "2017-08-03T09:26:40Z")
Running Logstash 5.4.1 on Ubuntu 14.04 (JVM 1.8.0\_131, heap 1G, 10 workers on EC2 with 4 CPUs/16GB RAM). Till now we had about 60 config files under conf.d, with about 2.8K lines, 125KB. In this configuration Logstash s… --- ## [JMX CompositeDataSupport and Map values](https://discuss.elastic.co/t/jmx-compositedatasupport-and-map-values/95680)
**Author:** [@karunakarsingh](https://discuss.elastic.co/u/karunakarsingh)\ **Replies:** 0\ **Last updated:** [August 3, 2017, 9:26am UTC](https://discuss.elastic.co/t/jmx-compositedatasupport-and-map-values/95680 "2017-08-03T09:26:32Z")
Hi, Using the logstash jmx plugin, I’m retrieving a CompositeDataSupport Object which is then pushed to Elasticsearch. The problem is that the value pushed is not an array converted as a String but more something which … --- ## [Logstash - filter the logevents based on time.send only specified date logs to the output plugin](https://discuss.elastic.co/t/logstash-filter-the-logevents-based-on-time-send-only-specified-date-logs-to-the-output-plugin/95675)
**Author:** [@soumya1](https://discuss.elastic.co/u/soumya1)\ **Replies:** 1\ **Last updated:** [August 3, 2017, 9:16am UTC](https://discuss.elastic.co/t/logstash-filter-the-logevents-based-on-time-send-only-specified-date-logs-to-the-output-plugin/95675 "2017-08-03T09:16:47Z")
hi my need is that i have to send logs from date x to date y from a log file containing logs of many days. i am using the log timestamp to compare with the time.now and then cancel the events matching a criteria,. but… --- ## [Logstash memory requirements](https://discuss.elastic.co/t/logstash-memory-requirements/94735)
**Author:** [@soumya1](https://discuss.elastic.co/u/soumya1)\ **Replies:** 2\ **Last updated:** [August 3, 2017, 9:03am UTC](https://discuss.elastic.co/t/logstash-memory-requirements/94735 "2017-08-03T09:03:08Z")
Hi i am configuring logstash so as to send data from 3 kafka clusters which generates 1lac events per second per each cluster. can you tell me the java heap memory allocation and pipeline workers and batch size which … --- ## [Read log file from RHEL 7 through filebeat](https://discuss.elastic.co/t/read-log-file-from-rhel-7-through-filebeat/95615)
**Author:** [@Leela\_Kodali](https://discuss.elastic.co/u/Leela_Kodali)\ **Replies:** 1\ **Last updated:** [August 3, 2017, 8:09am UTC](https://discuss.elastic.co/t/read-log-file-from-rhel-7-through-filebeat/95615 "2017-08-03T08:09:10Z")
Hi, My websphere application server is on RHEL 7 and i am planning to read systemOut log (we did folder share) from this server to another Linux server where i have ELK and Filebeat installed (I already configured my pa… --- ## [Can't get high granularity for a timestamp](https://discuss.elastic.co/t/cant-get-high-granularity-for-a-timestamp/95564)
**Author:** [@dboss101](https://discuss.elastic.co/u/dboss101)\ **Replies:** 3\ **Last updated:** [August 3, 2017, 8:06am UTC](https://discuss.elastic.co/t/cant-get-high-granularity-for-a-timestamp/95564 "2017-08-03T08:06:18Z")
Hey everyone, in one of our use cases with elastic, we're sending logs to it... as most people do. and it seems that the timestamp is limited only to 3 digits in the miliseconds (and lower) space, but we want to have h… --- ## [Unable to start Logstash](https://discuss.elastic.co/t/unable-to-start-logstash/95527)
**Author:** [@Hoffin\_Ag](https://discuss.elastic.co/u/Hoffin_Ag)\ **Replies:** 8\ **Last updated:** [August 3, 2017, 8:00am UTC](https://discuss.elastic.co/t/unable-to-start-logstash/95527 "2017-08-03T08:00:50Z")
Hi, logstash version - 2.3.4 logstash agent -f runconfig.conf shows the message prepend logstash prepend -j when using JRUBY. how to resolve this issue --- ## [Gracefully stopping the logstash](https://discuss.elastic.co/t/gracefully-stopping-the-logstash/95653)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 1\ **Last updated:** [August 3, 2017, 7:24am UTC](https://discuss.elastic.co/t/gracefully-stopping-the-logstash/95653 "2017-08-03T07:24:59Z")
When I do the CTRL+C I do the following message and logstash is not exiting. I have stopped the filebeat as well which send the data to the logstash pipeline. D:\\elastic-5.4.3\\logstash-5.4.3\\bin\>\[2017-08-03T12:44:57,316… --- ## [Starting logstash error: No log4j2 configuration file found. Using default configuration: logging only errors to the console](https://discuss.elastic.co/t/starting-logstash-error-no-log4j2-configuration-file-found-using-default-configuration-logging-only-errors-to-the-console/95655)
**Author:** [@Joe\_Yang](https://discuss.elastic.co/u/Joe_Yang)\ **Replies:** 0\ **Last updated:** [August 3, 2017, 7:23am UTC](https://discuss.elastic.co/t/starting-logstash-error-no-log4j2-configuration-file-found-using-default-configuration-logging-only-errors-to-the-console/95655 "2017-08-03T07:23:57Z")
$ ./logstash -f …/first-pipeline.conf --config.reload.automatic ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Sending Logstash’s logs to /home/m… --- ## [Logstash 5.5 with Scribe plugin](https://discuss.elastic.co/t/logstash-5-5-with-scribe-plugin/95652)
**Author:** [@narendra\_reddy](https://discuss.elastic.co/u/narendra_reddy)\ **Replies:** 0\ **Last updated:** [August 3, 2017, 7:09am UTC](https://discuss.elastic.co/t/logstash-5-5-with-scribe-plugin/95652 "2017-08-03T07:09:26Z")
Hi Team, Need help in using scribe plugin with logstash 5.5 version so that I can get logs from my scribe server and push them to Elastic Search, while trying to compile the plugin its giving errors, did any one tried u… --- ## [Logstash not reading more than 2 files](https://discuss.elastic.co/t/logstash-not-reading-more-than-2-files/95269)
**Author:** [@P\_Kumar](https://discuss.elastic.co/u/P_Kumar)\ **Replies:** 6\ **Last updated:** [August 3, 2017, 2:42am UTC](https://discuss.elastic.co/t/logstash-not-reading-more-than-2-files/95269 "2017-08-03T02:42:13Z")
Hi , I am new to ELK , i am trying to read the logs for IBM dmgr ,node and Application logs . but it is reading top 2 file input and creating index but 3rd file is not creating index and not reading could you please hel… --- ## [Logstash to PostgreSQL as output](https://discuss.elastic.co/t/logstash-to-postgresql-as-output/93500)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 9\ **Last updated:** [August 2, 2017, 10:25pm UTC](https://discuss.elastic.co/t/logstash-to-postgresql-as-output/93500 "2017-08-02T22:25:34Z")
Does anyone tried to output data logs to PostgreSQL? Right now my output is Elasticsearch and I want that every time a data will be output to elasticsearch, it will also output to PostreSQL --- ## [Does File input work recursively](https://discuss.elastic.co/t/does-file-input-work-recursively/95612)
**Author:** [@niraj\_kumar](https://discuss.elastic.co/u/niraj_kumar)\ **Replies:** 3\ **Last updated:** [August 2, 2017, 9:04pm UTC](https://discuss.elastic.co/t/does-file-input-work-recursively/95612 "2017-08-02T21:04:32Z")
Hi Everyone, I am building up my logstash config and i am using the following file input plugin to ingest gzip files. input { file { path =\> "/home/niraj/sync/bigdata-dev" tags =\> "bigdata-dev1" codec =\> "gzip\_lines… --- ## [Extracting fields from GREEDY data in GROK](https://discuss.elastic.co/t/extracting-fields-from-greedy-data-in-grok/95586)
**Author:** [@shroh](https://discuss.elastic.co/u/shroh)\ **Replies:** 1\ **Last updated:** [August 2, 2017, 9:01pm UTC](https://discuss.elastic.co/t/extracting-fields-from-greedy-data-in-grok/95586 "2017-08-02T21:01:27Z")
I have a web access log from which i need to create the fields, below is the log line: 10.202.103.252 - ECG358 \[02/Aug/2017:16:48:53 +0000\] “GET /app-web/tutor/applications/14/notes?category=All&offset=0&limit=30&count=… --- ## [Offline logstash plugin install fails](https://discuss.elastic.co/t/offline-logstash-plugin-install-fails/95592)
**Author:** [@mr\_chang7](https://discuss.elastic.co/u/mr_chang7)\ **Replies:** 1\ **Last updated:** [August 2, 2017, 8:43pm UTC](https://discuss.elastic.co/t/offline-logstash-plugin-install-fails/95592 "2017-08-02T20:43:35Z")
Hi, In the RH 7.3 system with installed logstash 5.4.3.rpm, I have followed instruction in the documentation: Offline Plugin Management | Logstash Reference \[5.4\] | Elastic I've got an error as below. /usr/share/logst… --- ## [Filebeat statup error](https://discuss.elastic.co/t/filebeat-statup-error/95476)
**Author:** [@Joe\_Yang](https://discuss.elastic.co/u/Joe_Yang)\ **Replies:** 1\ **Last updated:** [August 2, 2017, 8:37pm UTC](https://discuss.elastic.co/t/filebeat-statup-error/95476 "2017-08-02T20:37:58Z")
Following this offical doc(https://www.elastic.co/guide/en/logstash/5.4/advanced-pipeline.html) to run filebeat hits the folllowing error: ./filebeat -e -c filebeat.yml -d “publish” 2017/08/02 08:15:24.372074 output.go… --- ## [Logstash configuration file to reference variables found in another file](https://discuss.elastic.co/t/logstash-configuration-file-to-reference-variables-found-in-another-file/95575)
**Author:** [@CDR](https://discuss.elastic.co/u/CDR)\ **Replies:** 3\ **Last updated:** [August 2, 2017, 8:07pm UTC](https://discuss.elastic.co/t/logstash-configuration-file-to-reference-variables-found-in-another-file/95575 "2017-08-02T20:07:45Z")
Hello, I am working on a project that will have logstash ingest data from many different servers. All of the servers will have a unique IP address and will be present on each log message. All of the servers have special… --- ## [Warn : expected UTF-8](https://discuss.elastic.co/t/warn-expected-utf-8/94616)
**Author:** [@marwa](https://discuss.elastic.co/u/marwa)\ **Replies:** 3\ **Last updated:** [August 2, 2017, 6:44pm UTC](https://discuss.elastic.co/t/warn-expected-utf-8/94616 "2017-08-02T18:44:36Z")
i wanted to load data from logsatsh into elasticsearch , this is a sample of my data product-name,price,number\_of\_customers,general\_feddback,promotion\_period,percentage\_promotion,number\_of\_customers\_after\_promotion,cate… --- ## [Please please Please help me in logstash](https://discuss.elastic.co/t/please-please-please-help-me-in-logstash/95419)
**Author:** [@P\_Kumar](https://discuss.elastic.co/u/P_Kumar)\ **Replies:** 1\ **Last updated:** [August 2, 2017, 6:37pm UTC](https://discuss.elastic.co/t/please-please-please-help-me-in-logstash/95419 "2017-08-02T18:37:45Z")
Hi , I am new to ELK , i am trying to read the logs for IBM dmgr ,node and Application logs . but it is reading top 2 file input and creating index but 3rd file is not creating index and not reading could you please hel… --- ## [How can I alias column names in a sql query while loading data through logstash?](https://discuss.elastic.co/t/how-can-i-alias-column-names-in-a-sql-query-while-loading-data-through-logstash/95291)
**Author:** [@patil-akshay](https://discuss.elastic.co/u/patil-akshay)\ **Replies:** 2\ **Last updated:** [August 2, 2017, 6:33pm UTC](https://discuss.elastic.co/t/how-can-i-alias-column-names-in-a-sql-query-while-loading-data-through-logstash/95291 "2017-08-02T18:33:16Z")
I want to change some column name while loading the data through logstash --- ## [Fetch file name from path and append to the index name of ES](https://discuss.elastic.co/t/fetch-file-name-from-path-and-append-to-the-index-name-of-es/95163)
**Author:** [@KML](https://discuss.elastic.co/u/KML)\ **Replies:** 1\ **Last updated:** [August 2, 2017, 5:49pm UTC](https://discuss.elastic.co/t/fetch-file-name-from-path-and-append-to-the-index-name-of-es/95163 "2017-08-02T17:49:49Z")
Hello All, Consider that I have path field having value file { path =\> \[“D:\\ELK\\csvs\\D20150105\_R20160608.csv”\] start\_position =\> “beginning” } So I would like to take the value 20160608 which is date YYYYMMDD var… --- ## [No index created - Question from a total noob](https://discuss.elastic.co/t/no-index-created-question-from-a-total-noob/95487)
**Author:** [@Cristian](https://discuss.elastic.co/u/Cristian)\ **Replies:** 11\ **Last updated:** [August 2, 2017, 4:18pm UTC](https://discuss.elastic.co/t/no-index-created-question-from-a-total-noob/95487 "2017-08-02T16:18:00Z")
Hi all! Im trying to create my index from forex-data that I get through an API. Logstash seems to pick up the data but no index is created. Since I never use ES I don’t really know what to look for to understand why the… --- ## [Index not being created in elasticsearch](https://discuss.elastic.co/t/index-not-being-created-in-elasticsearch/95423)
**Author:** [@pravinnair](https://discuss.elastic.co/u/pravinnair)\ **Replies:** 1\ **Last updated:** [August 2, 2017, 4:00pm UTC](https://discuss.elastic.co/t/index-not-being-created-in-elasticsearch/95423 "2017-08-02T16:00:29Z")
Simple ELK setup I have created the GROK expression for IIB server logs and it works fine. Only problem is the output filter is not creating an index on elasticsearch its always defaults to logstash-\* index My plugin … --- ## [Logstash Oracle setup - Cannot create pipeline](https://discuss.elastic.co/t/logstash-oracle-setup-cannot-create-pipeline/92546)
**Author:** [@bhrtjoshi](https://discuss.elastic.co/u/bhrtjoshi)\ **Replies:** 3\ **Last updated:** [August 2, 2017, 3:36pm UTC](https://discuss.elastic.co/t/logstash-oracle-setup-cannot-create-pipeline/92546 "2017-08-02T15:36:03Z")
I there, I am getting below error in the logstash logs. Can someone please help? \[2017-07-10T19:06:10,405\]\[ERROR\]\[logstash.agent \] Cannot create pipeline {:reason=\>"Expected one of #, =\> at line 15, column 11… --- ## [MongoDB input plugin](https://discuss.elastic.co/t/mongodb-input-plugin/95561)
**Author:** [@SYV](https://discuss.elastic.co/u/SYV)\ **Replies:** 0\ **Last updated:** [August 2, 2017, 3:00pm UTC](https://discuss.elastic.co/t/mongodb-input-plugin/95561 "2017-08-02T15:00:41Z")
Since the last discussion thread about using MongoDB dates end of 2016, I would like to know whether there is any news about a “native” MongoDB input plugin for Logstash. Or is the plugin to use still the one to be foun… --- ## [Best practice with Logstash output and ES](https://discuss.elastic.co/t/best-practice-with-logstash-output-and-es/95189)
**Author:** [@lendoly](https://discuss.elastic.co/u/lendoly)\ **Replies:** 3\ **Last updated:** [August 2, 2017, 1:39pm UTC](https://discuss.elastic.co/t/best-practice-with-logstash-output-and-es/95189 "2017-08-02T13:39:39Z")
Hello, My question is related to the correct way to configure the output, for Elasticsearch. Right now my Elasticsearch cluster is one master node, 2 ingest nodes and 1 data node. My Logstash output configuration is l… --- ## [Action \[indices:data/write/bulk\] is unauthorized for user \[logstash\_system\]\\](https://discuss.elastic.co/t/action-indices-data-write-bulk-is-unauthorized-for-user-logstash-system/95532)
**Author:** [@dencowboy](https://discuss.elastic.co/u/dencowboy)\ **Replies:** 0\ **Last updated:** [August 2, 2017, 1:12pm UTC](https://discuss.elastic.co/t/action-indices-data-write-bulk-is-unauthorized-for-user-logstash-system/95532 "2017-08-02T13:12:39Z")
I’ve created the ELK stack in docker. I started ES. I’ve changed the password of the existing logstash\_system user: curl -u elastic -XPUT 'localhost:9200/\_xpack/security/user/logstash\_system/\_password?pretty' -H 'Conte… --- ## [An unknown error occurred sending a bulk request to Elasticsearch](https://discuss.elastic.co/t/an-unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/95537)
**Author:** [@sadiesh](https://discuss.elastic.co/u/sadiesh)\ **Replies:** 0\ **Last updated:** [August 2, 2017, 1:29pm UTC](https://discuss.elastic.co/t/an-unknown-error-occurred-sending-a-bulk-request-to-elasticsearch/95537 "2017-08-02T13:29:41Z")
Am getting following error in logstash. while running a JDBC Plugin An unknown error occurred sending a bulk request to Elasticsearch. We will retry indefinitely {:error\_message=\>“No serializer found for class java.io.B… --- ## [While parsing XML file getting chunks of message not whole message](https://discuss.elastic.co/t/while-parsing-xml-file-getting-chunks-of-message-not-whole-message/95535)
**Author:** [@aashishchauhan06](https://discuss.elastic.co/u/aashishchauhan06)\ **Replies:** 0\ **Last updated:** [August 2, 2017, 1:21pm UTC](https://discuss.elastic.co/t/while-parsing-xml-file-getting-chunks-of-message-not-whole-message/95535 "2017-08-02T13:21:14Z")
Hi, While parsing XML file i’m getting " Only String and Array types are splittable. field:\[drug\] is of type = NilClass" error. As far as i understand this error is coming because xml parser returning message in small c… --- ## [File input plugin is not working](https://discuss.elastic.co/t/file-input-plugin-is-not-working/95084)
**Author:** [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\ **Replies:** 2\ **Last updated:** [August 2, 2017, 1:12pm UTC](https://discuss.elastic.co/t/file-input-plugin-is-not-working/95084 "2017-08-02T13:12:09Z")
Hello everyone, I am trying to use file input plugin instead of tcp, but it doesn’t work (there are no injected files in elasticsearch), here’s a minimal configuration for logstash: input { file { path =\> \["… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=933) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=935)