# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=464 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 465 --- ## [Using aggregate to add data to previous event](https://discuss.elastic.co/t/using-aggregate-to-add-data-to-previous-event/202904)
**Author:** [@Factor3](https://discuss.elastic.co/u/Factor3)\ **Replies:** 2\ **Last updated:** [October 10, 2019, 5:37pm UTC](https://discuss.elastic.co/t/using-aggregate-to-add-data-to-previous-event/202904 "2019-10-10T17:37:56Z")
Hi, I have a logfile that looks like this: 61f3483e-e7f4-4202-b34a-d85108f0fea9 - some log line 61f3483e-e7f4-4202-b34a-d85108f0fea9 - some other log line ID=123456 61f3483e-e7f4-4202-b34a-d85108f0fea9 - another log lin… --- ## [Parse JSON stringified value of a field of an object in the array of objects](https://discuss.elastic.co/t/parse-json-stringified-value-of-a-field-of-an-object-in-the-array-of-objects/202791)
**Author:** [@xibal](https://discuss.elastic.co/u/xibal)\ **Replies:** 4\ **Last updated:** [October 10, 2019, 4:58pm UTC](https://discuss.elastic.co/t/parse-json-stringified-value-of-a-field-of-an-object-in-the-array-of-objects/202791 "2019-10-10T16:58:41Z")
Hi all, I've been migrating order & order items one-to-many relation from MySQL into ES using Logstash. With the help of filter aggregation for my joined SQL, the items become a nested property under each order and the… --- ## [Logstash sends logs with wrong @timestamp](https://discuss.elastic.co/t/logstash-sends-logs-with-wrong-timestamp/202677)
**Author:** [@kausar](https://discuss.elastic.co/u/kausar)\ **Replies:** 10\ **Last updated:** [October 10, 2019, 4:52pm UTC](https://discuss.elastic.co/t/logstash-sends-logs-with-wrong-timestamp/202677 "2019-10-10T16:52:51Z")
Hi, I have two logstash servers that writes to Elasticsearch. Filebeat reads logs from file and sends them to these logstash servers. I am conducting a test for failover/outage and also what will happen if both the logs… --- ## [Populating pipeline env variable with array of values](https://discuss.elastic.co/t/populating-pipeline-env-variable-with-array-of-values/203052)
**Author:** [@Merlin\_Nunez](https://discuss.elastic.co/u/Merlin_Nunez)\ **Replies:** 1\ **Last updated:** [October 10, 2019, 4:05pm UTC](https://discuss.elastic.co/t/populating-pipeline-env-variable-with-array-of-values/203052 "2019-10-10T16:05:53Z")
Hi All, I´m trying to move from a single note cluster to the following configuration, esentially adding two ES notes for redundancy. es1, es2, es3, Kibana Logstash Multiple filebeats I have successfully configured … --- ## [Advice on parsing a JSON log](https://discuss.elastic.co/t/advice-on-parsing-a-json-log/202968)
**Author:** [@Callum.wyres](https://discuss.elastic.co/u/Callum.wyres)\ **Replies:** 4\ **Last updated:** [October 10, 2019, 3:21pm UTC](https://discuss.elastic.co/t/advice-on-parsing-a-json-log/202968 "2019-10-10T15:21:36Z")
Hi, We are currently experience some issues parsing our JSON logs from filebeat to elasticsearch. An example of a log being sent is: "{"log":"{\\"level\\":30,\\"time\\":1570692192768,\\"msg\\":\\"APP\_STARTED\\",\\"pid\\":1,\\"ho… --- ## [How to create secure connection with elasticsearch in ruby client? help](https://discuss.elastic.co/t/how-to-create-secure-connection-with-elasticsearch-in-ruby-client-help/203041)
**Author:** [@talbehat](https://discuss.elastic.co/u/talbehat)\ **Replies:** 0\ **Last updated:** [October 10, 2019, 1:57pm UTC](https://discuss.elastic.co/t/how-to-create-secure-connection-with-elasticsearch-in-ruby-client-help/203041 "2019-10-10T13:57:47Z")
How to create secure connection with elasticsearch in ruby client? --- ## [Logstash and multiple configurations](https://discuss.elastic.co/t/logstash-and-multiple-configurations/203029)
**Author:** [@beren](https://discuss.elastic.co/u/beren)\ **Replies:** 0\ **Last updated:** [October 10, 2019, 1:02pm UTC](https://discuss.elastic.co/t/logstash-and-multiple-configurations/203029 "2019-10-10T13:02:57Z")
Hi Theoretical question How does logstash understand which logs to handle ? That is, we have configurations in the postfix folder.conf (for Postfix processing), Nginx (for nginx) they look like this . input { beats { … --- ## [Parse inner JSON and split by it](https://discuss.elastic.co/t/parse-inner-json-and-split-by-it/202973)
**Author:** [@Prolog](https://discuss.elastic.co/u/Prolog)\ **Replies:** 2\ **Last updated:** [October 10, 2019, 12:55pm UTC](https://discuss.elastic.co/t/parse-inner-json-and-split-by-it/202973 "2019-10-10T12:55:24Z")
Hi, I have a JSON coming from RabbitMQ structured as shown below: { "MessageItemsCount": 8106, "Message": "\[{\\"Id\\":\\"Foo\_123\_456\\",\\"NetworkShopId\\":\\"333\\",\\"Name\\":\\"Some product\\",\\"Descr… --- ## [Logstash - aggregate messages - start with new line](https://discuss.elastic.co/t/logstash-aggregate-messages-start-with-new-line/202920)
**Author:** [@Vincent\_Chen](https://discuss.elastic.co/u/Vincent_Chen)\ **Replies:** 3\ **Last updated:** [October 10, 2019, 12:49pm UTC](https://discuss.elastic.co/t/logstash-aggregate-messages-start-with-new-line/202920 "2019-10-10T12:49:46Z")
Hi everyone, I am trying to aggregate events related to the same transaction. The aggregate part works fine, I am able to merge all related events into the same field. But I want each event to be placed at a newline. S… --- ## [Logstash doesn't jump to the next line](https://discuss.elastic.co/t/logstash-doesnt-jump-to-the-next-line/203008)
**Author:** [@Alekz](https://discuss.elastic.co/u/Alekz)\ **Replies:** 0\ **Last updated:** [October 10, 2019, 11:05am UTC](https://discuss.elastic.co/t/logstash-doesnt-jump-to-the-next-line/203008 "2019-10-10T11:05:19Z")
Hi everyone, i been a lurker for a while from this community and i've to say thanks for helping me time to time, now i've a problem that can't be solved, ive a jboss with java app running inside of it , the thing is the … --- ## [Migrate mysql nth level data to elasticsearch via logstash (config file)](https://discuss.elastic.co/t/migrate-mysql-nth-level-data-to-elasticsearch-via-logstash-config-file/202833)
**Author:** [@saifrehman](https://discuss.elastic.co/u/saifrehman)\ **Replies:** 1\ **Last updated:** [October 10, 2019, 10:35am UTC](https://discuss.elastic.co/t/migrate-mysql-nth-level-data-to-elasticsearch-via-logstash-config-file/202833 "2019-10-10T10:35:15Z")
I am migrating mysql data to elasticsearch via logstash. The database is too big and have lots of relations. I already imported the simple tables and table with nested data(single level parent/child) as well. Problem : … --- ## [Logstash not listening filebeat](https://discuss.elastic.co/t/logstash-not-listening-filebeat/202999)
**Author:** [@Ghali](https://discuss.elastic.co/u/Ghali)\ **Replies:** 0\ **Last updated:** [October 10, 2019, 10:07am UTC](https://discuss.elastic.co/t/logstash-not-listening-filebeat/202999 "2019-10-10T10:07:46Z")
! 71903847\_575966983186355\_2757365151446335488\_n|666x500 ! 73018616\_2514748882139881\_1200478560783958016\_n|666x500 Logstash is not listening on port 5044 i tried without the host but i got the same result. this… --- ## [How to parse json inside text line](https://discuss.elastic.co/t/how-to-parse-json-inside-text-line/202800)
**Author:** [@Deny7](https://discuss.elastic.co/u/Deny7)\ **Replies:** 17\ **Last updated:** [October 10, 2019, 10:05am UTC](https://discuss.elastic.co/t/how-to-parse-json-inside-text-line/202800 "2019-10-10T10:05:24Z")
Hi, I have following log that looks like this: 1570519737247 I access {"date":"2019-10-08T09:28:57.247","rootTitle":"title1","rootModel":"model1","dcTitle":"\[1a\]"} 1570519737247 I access {"date":"2019-10-08T09:28:57.2… --- ## [Logstash hangs (CLOSE\_WAIT)](https://discuss.elastic.co/t/logstash-hangs-close-wait/201545)
**Author:** [@manunc](https://discuss.elastic.co/u/manunc)\ **Replies:** 2\ **Last updated:** [October 10, 2019, 9:58am UTC](https://discuss.elastic.co/t/logstash-hangs-close-wait/201545 "2019-10-10T09:58:06Z")
Hello, Logstash version 7.3.2-1 on CentOS 7.6.1810 Since several days I noticed that Logstash is frozen every 24h. When I check the server I found a lot of CLOSE\_WAIT (131). Nothing in the logstash-plain.log relevant… --- ## [I have a problem inputing file on logstash](https://discuss.elastic.co/t/i-have-a-problem-inputing-file-on-logstash/202568)
**Author:** [@ByeonggeunSon](https://discuss.elastic.co/u/ByeonggeunSon)\ **Replies:** 2\ **Last updated:** [October 10, 2019, 9:43am UTC](https://discuss.elastic.co/t/i-have-a-problem-inputing-file-on-logstash/202568 "2019-10-10T09:43:40Z")
I'm trying to input the file which name is same for everytime It reads well but I'd like to output only the last line appended in that time my pipeline configuration is here input { file{ path =\> "/home/lisa/elastic… --- ## [Reading a specific file in S3 bucket](https://discuss.elastic.co/t/reading-a-specific-file-in-s3-bucket/202985)
**Author:** [@mruthyu](https://discuss.elastic.co/u/mruthyu)\ **Replies:** 0\ **Last updated:** [October 10, 2019, 9:06am UTC](https://discuss.elastic.co/t/reading-a-specific-file-in-s3-bucket/202985 "2019-10-10T09:06:11Z")
I tried reading a single file from the S3 bucket using prefix field of S3 llogstash input plugin, but is not working. So would like to know if is possible to read a specific file from s3 bucket, --- ## [Field is not displaying proper field output - UDP](https://discuss.elastic.co/t/field-is-not-displaying-proper-field-output-udp/202966)
**Author:** [@akhilsharma.in](https://discuss.elastic.co/u/akhilsharma.in)\ **Replies:** 0\ **Last updated:** [October 10, 2019, 7:47am UTC](https://discuss.elastic.co/t/field-is-not-displaying-proper-field-output-udp/202966 "2019-10-10T07:47:43Z")
Hi, I'm using input as UDP and storing the output in file. Events have a field "message" which is not having the complete value but it's truncated. Example: "message":" ...fkhfhftxt..." Sample: { "@version": "1", … --- ## [Can a Logstash pipeline wait to start until another pipeline completes?](https://discuss.elastic.co/t/can-a-logstash-pipeline-wait-to-start-until-another-pipeline-completes/202943)
**Author:** [@snowboardcto](https://discuss.elastic.co/u/snowboardcto)\ **Replies:** 1\ **Last updated:** [October 10, 2019, 5:27am UTC](https://discuss.elastic.co/t/can-a-logstash-pipeline-wait-to-start-until-another-pipeline-completes/202943 "2019-10-10T05:27:07Z")
Using Logstash multiple pipelines can I configure a pipeline to wait to start until another pipeline has completed? --- ## [Logstash - aggregate filter - for each transaction/thread, only look for the first match message](https://discuss.elastic.co/t/logstash-aggregate-filter-for-each-transaction-thread-only-look-for-the-first-match-message/202936)
**Author:** [@Vincent\_Chen](https://discuss.elastic.co/u/Vincent_Chen)\ **Replies:** 0\ **Last updated:** [October 10, 2019, 3:23am UTC](https://discuss.elastic.co/t/logstash-aggregate-filter-for-each-transaction-thread-only-look-for-the-first-match-message/202936 "2019-10-10T03:23:29Z")
Hi everyone, We have a log that do not provide a unique ID for each transaction. We want to use the thread-id and the message body to define the transaction boundary. Here is the mock-up log: ##transaction 1 \[2019-… --- ## [Logstash/ Synchronize Postgres to EL](https://discuss.elastic.co/t/logstash-synchronize-postgres-to-el/202927)
**Author:** [@sbyamauchi](https://discuss.elastic.co/u/sbyamauchi)\ **Replies:** 0\ **Last updated:** [October 10, 2019, 1:11am UTC](https://discuss.elastic.co/t/logstash-synchronize-postgres-to-el/202927 "2019-10-10T01:11:12Z")
Hi,I try to synchronize postgres to elastic. But I receive the error below and not update index data. Is it necessary to make multiple pipelines or something? Logstash could not be started because there is already ano… --- ## [Logstash Service starts but it could not work without manually starting with --path.settings and -f parameters](https://discuss.elastic.co/t/logstash-service-starts-but-it-could-not-work-without-manually-starting-with-path-settings-and-f-parameters/200981)
**Author:** [@alperensoydan](https://discuss.elastic.co/u/alperensoydan)\ **Replies:** 1\ **Last updated:** [October 10, 2019, 2:15am UTC](https://discuss.elastic.co/t/logstash-service-starts-but-it-could-not-work-without-manually-starting-with-path-settings-and-f-parameters/200981 "2019-10-10T02:15:10Z")
Hello, I have a problem with Logstash, I want to start it on startup. It seems it works as a service but it can not work essentially. When I run /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d --path.settings /… --- ## [Running multiple logstash config](https://discuss.elastic.co/t/running-multiple-logstash-config/202745)
**Author:** [@ashokchinna](https://discuss.elastic.co/u/ashokchinna)\ **Replies:** 2\ **Last updated:** [October 9, 2019, 11:45pm UTC](https://discuss.elastic.co/t/running-multiple-logstash-config/202745 "2019-10-09T23:45:21Z")
I want to run multiple logstash conf file in the same instance or multiple instance of logstash. What is the best approach to follow? --- ## [Lotstash is not picking Apache Logs config](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747)
**Author:** [@sud0](https://discuss.elastic.co/u/sud0)\ **Replies:** 10\ **Last updated:** [October 9, 2019, 11:21pm UTC](https://discuss.elastic.co/t/lotstash-is-not-picking-apache-logs-config/202747 "2019-10-09T23:21:17Z")
I have two Apache webservers (1x uat and 1x prod). I am sending the logs to my ELK setup for both of them. For the UAT one, it is working fine. For the PROD one, the logs are now shown in Kibana. I need some help to d… --- ## [Facing issues when using java plugin with logstash 7.3.2](https://discuss.elastic.co/t/facing-issues-when-using-java-plugin-with-logstash-7-3-2/202609)
**Author:** [@buch11](https://discuss.elastic.co/u/buch11)\ **Replies:** 1\ **Last updated:** [October 9, 2019, 10:42pm UTC](https://discuss.elastic.co/t/facing-issues-when-using-java-plugin-with-logstash-7-3-2/202609 "2019-10-09T22:42:32Z")
Hi All, I am trying to write a java filter plugin - following the instructions at - https://www.elastic.co/guide/en/logstash/current/java-filter-plugin.html I am able to compile and install it locally and able to creat… --- ## [Logstash keeps retrying infinitely when failed to upload to ES due to bulk upload error](https://discuss.elastic.co/t/logstash-keeps-retrying-infinitely-when-failed-to-upload-to-es-due-to-bulk-upload-error/202918)
**Author:** [@user2416](https://discuss.elastic.co/u/user2416)\ **Replies:** 0\ **Last updated:** [October 9, 2019, 10:26pm UTC](https://discuss.elastic.co/t/logstash-keeps-retrying-infinitely-when-failed-to-upload-to-es-due-to-bulk-upload-error/202918 "2019-10-09T22:26:55Z")
logstash keeps retrying infinitely when failed to upload to Elasticsearch cluster due to bulk upload error. It stuck at this error and not processing other logs. Can someone please help me how to resolve this issue? \[20… --- ## [Create Kibana Index Pattern Dynamically](https://discuss.elastic.co/t/create-kibana-index-pattern-dynamically/202611)
**Author:** [@GokcerBelgusen](https://discuss.elastic.co/u/GokcerBelgusen)\ **Replies:** 1\ **Last updated:** [October 9, 2019, 8:18pm UTC](https://discuss.elastic.co/t/create-kibana-index-pattern-dynamically/202611 "2019-10-09T20:18:14Z")
Hi, is there a way of creating Kibana index pattern inside of Logstash ? For instance, you can create an index inside of Logstash while collecting data. After creating index, I also want to create Kibana index pattern au… --- ## [Input simple txt file](https://discuss.elastic.co/t/input-simple-txt-file/202892)
**Author:** [@happycampo](https://discuss.elastic.co/u/happycampo)\ **Replies:** 3\ **Last updated:** [October 9, 2019, 7:22pm UTC](https://discuss.elastic.co/t/input-simple-txt-file/202892 "2019-10-09T19:22:11Z")
Hello, Not sure what I'm missing: I have setup a docker elk stack and now I'm trying to see if I can ingest a log file locally. I was having trouble with logstash shutting down when introducing a new input. See thi… --- ## [Logstash External Component Retry Attempts](https://discuss.elastic.co/t/logstash-external-component-retry-attempts/202738)
**Author:** [@shimpi](https://discuss.elastic.co/u/shimpi)\ **Replies:** 2\ **Last updated:** [October 9, 2019, 7:04pm UTC](https://discuss.elastic.co/t/logstash-external-component-retry-attempts/202738 "2019-10-09T19:04:11Z")
Hello Members, During logstash startup if, for any reason, logstash couldn't connect to the input or output plugin, it keeps on trying indefinitely. My question is, is there any settings which tells it to stop after sp… --- ## [How do you remove $ and % signs that precede integer values on Elasticsearch?](https://discuss.elastic.co/t/how-do-you-remove-and-signs-that-precede-integer-values-on-elasticsearch/202748)
**Author:** [@saomer](https://discuss.elastic.co/u/saomer)\ **Replies:** 4\ **Last updated:** [October 9, 2019, 6:00pm UTC](https://discuss.elastic.co/t/how-do-you-remove-and-signs-that-precede-integer-values-on-elasticsearch/202748 "2019-10-09T18:00:33Z")
"price" =\> "$2,190" thats how it shows on ES! I would like to remove the $ on logtash before sending it to ES. --- ## [Pipeline terminated {"pipeline.id"=\>".monitoring-logstash"} Logstash shutdown](https://discuss.elastic.co/t/pipeline-terminated-pipeline-id-monitoring-logstash-logstash-shutdown/202709)
**Author:** [@happycampo](https://discuss.elastic.co/u/happycampo)\ **Replies:** 4\ **Last updated:** [October 9, 2019, 5:44pm UTC](https://discuss.elastic.co/t/pipeline-terminated-pipeline-id-monitoring-logstash-logstash-shutdown/202709 "2019-10-09T17:44:53Z")
I finally got my elk stack running and working with mysql-connector and then I decided to resize my vm. Now logstash will not stay up. This was working before I'm pretty sure. Logstash.yml --- Default Logstash conf… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=463) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=465)