# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=320 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 321 --- ## [Logstash filebeat Performance](https://discuss.elastic.co/t/logstash-filebeat-performance/243707)
**Author:** [@eth](https://discuss.elastic.co/u/eth)\ **Replies:** 0\ **Last updated:** [August 4, 2020, 12:54pm UTC](https://discuss.elastic.co/t/logstash-filebeat-performance/243707 "2020-08-04T12:54:54Z")
I am sending kubernetes pod logs from filebeat to logstash via kafka plugin. Filebeat, Logstash and kafka running as pods in the single host. filebeat ==\> Kafka ==\> Logstash when i checked the network bandw… --- ## [Multiline Logstah file txt](https://discuss.elastic.co/t/multiline-logstah-file-txt/243613)
**Author:** [@Daniel\_Bergmann](https://discuss.elastic.co/u/Daniel_Bergmann)\ **Replies:** 6\ **Last updated:** [August 4, 2020, 11:53am UTC](https://discuss.elastic.co/t/multiline-logstah-file-txt/243613 "2020-08-04T11:53:30Z")
I have a TXT file that I want to import into Elasticsearch via LogStash, it is a fixed type of columns, in which I used grok and if to separate the columns of this file, that's fine, but I want to use the multiline to g… --- ## [Logstash if conditions connected with and](https://discuss.elastic.co/t/logstash-if-conditions-connected-with-and/243199)
**Author:** [@Souji\_T](https://discuss.elastic.co/u/Souji_T)\ **Replies:** 8\ **Last updated:** [August 4, 2020, 6:08am UTC](https://discuss.elastic.co/t/logstash-if-conditions-connected-with-and/243199 "2020-08-04T06:08:02Z")
Hello, I am trying to add a field with logstash. For example, I have following line in the filter section of my config file from logstash: if \[dest\_ip\] == "194.175.245.9" and \[alert\]\[signature\_id\] == "2009702" { … --- ## [Aggregation Filter 2 logs same index](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624)
**Author:** [@rcavezzale](https://discuss.elastic.co/u/rcavezzale)\ **Replies:** 11\ **Last updated:** [August 4, 2020, 1:38am UTC](https://discuss.elastic.co/t/aggregation-filter-2-logs-same-index/243624 "2020-08-04T01:38:18Z")
Dear Comminuty I need your help to fix my issue. i have 2 logs in same index that should be aggregate to view all information on the Data Table visualize input { udp { port =\> 1514 type =\> "alcatel" } } filter { if \[… --- ## [How to search for code in this GitHub?](https://discuss.elastic.co/t/how-to-search-for-code-in-this-github/242790)
**Author:** [@thanhlam](https://discuss.elastic.co/u/thanhlam)\ **Replies:** 1\ **Last updated:** [August 3, 2020, 9:12pm UTC](https://discuss.elastic.co/t/how-to-search-for-code-in-this-github/242790 "2020-08-03T21:12:09Z")
Hello, I'm new to this GitHub but very interested in searching for code. So that I can find and learn how something works. Can someone point me to instructions on how to search for a text string for example in the code? … --- ## [Ruby code to iterate through list of pipelines received by Logstash API](https://discuss.elastic.co/t/ruby-code-to-iterate-through-list-of-pipelines-received-by-logstash-api/243616)
**Author:** [@aidanoc15](https://discuss.elastic.co/u/aidanoc15)\ **Replies:** 2\ **Last updated:** [August 3, 2020, 7:52pm UTC](https://discuss.elastic.co/t/ruby-code-to-iterate-through-list-of-pipelines-received-by-logstash-api/243616 "2020-08-03T19:52:48Z")
Hi, I am trying to implement monitoring and alerting for some of our pipelines that we have set up in logstash. My thought is to run a cron job on the Logstash API (specifically \_nodes/stats/pipelines) to retrieve both t… --- ## [Looking for a Logstash Kubernetes Deployment/Stateful Configuration](https://discuss.elastic.co/t/looking-for-a-logstash-kubernetes-deployment-stateful-configuration/243558)
**Author:** [@tamilarasanbravo](https://discuss.elastic.co/u/tamilarasanbravo)\ **Replies:** 1\ **Last updated:** [August 3, 2020, 7:07pm UTC](https://discuss.elastic.co/t/looking-for-a-logstash-kubernetes-deployment-stateful-configuration/243558 "2020-08-03T19:07:22Z")
Hi Team, I am trying to setup logstash in K8's cluster and after trying various ways, I ended up nowhere. I would like to know if there is any document which shows a working Logstash K8's Deployment/Statefulset files. --- ## [Delete and Insert documents in elasticsearch from logstash](https://discuss.elastic.co/t/delete-and-insert-documents-in-elasticsearch-from-logstash/243614)
**Author:** [@Gabriel\_Mare](https://discuss.elastic.co/u/Gabriel_Mare)\ **Replies:** 1\ **Last updated:** [August 3, 2020, 6:53pm UTC](https://discuss.elastic.co/t/delete-and-insert-documents-in-elasticsearch-from-logstash/243614 "2020-08-03T18:53:49Z")
Hi, I need to delete all documents and then add documents from logstash, in the same query. Is this possible? How can I do? --- ## [AzureBlob Old/Existing logs not Ingesting](https://discuss.elastic.co/t/azureblob-old-existing-logs-not-ingesting/243603)
**Author:** [@Kevin\_f](https://discuss.elastic.co/u/Kevin_f)\ **Replies:** 0\ **Last updated:** [August 3, 2020, 4:20pm UTC](https://discuss.elastic.co/t/azureblob-old-existing-logs-not-ingesting/243603 "2020-08-03T16:20:15Z")
Hello, Hope someone could help. I have logstash ingesting logs from Azureblob storage. What happened is that logs stopped ingesting 2 days ago. Today I got it working again however, the logs from the last 2 days are no… --- ## [Unrecognized @timestamp value type=class org.jruby.RubyFloat](https://discuss.elastic.co/t/unrecognized-timestamp-value-type-class-org-jruby-rubyfloat/243560)
**Author:** [@111266](https://discuss.elastic.co/u/111266)\ **Replies:** 10\ **Last updated:** [August 3, 2020, 4:08pm UTC](https://discuss.elastic.co/t/unrecognized-timestamp-value-type-class-org-jruby-rubyfloat/243560 "2020-08-03T16:08:22Z")
Use: "logstash.version"=\>"7.8.0" kafka\_2.13-2.5.0 Config: input { kafka { bootstrap\_servers =\> "ip:port" auto\_offset\_reset =\> "earliest" enable\_auto\_commit =\> true topics =\> \["logs… --- ## [Add two value from two record](https://discuss.elastic.co/t/add-two-value-from-two-record/243126)
**Author:** [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Replies:** 24\ **Last updated:** [August 3, 2020, 4:02pm UTC](https://discuss.elastic.co/t/add-two-value-from-two-record/243126 "2020-08-03T16:02:47Z")
I am gone a receive two line of record every few min. example name : total code1 : 100 code2: 101 I would like to create a record { name: code1 total: 100 combine\_total: 201 } { name:code2 total: … --- ## [Logstash Configuration Error in K8](https://discuss.elastic.co/t/logstash-configuration-error-in-k8/243422)
**Author:** [@tamilarasanbravo](https://discuss.elastic.co/u/tamilarasanbravo)\ **Replies:** 2\ **Last updated:** [August 3, 2020, 3:44pm UTC](https://discuss.elastic.co/t/logstash-configuration-error-in-k8/243422 "2020-08-03T15:44:10Z")
Hi Team, Created a Pod for logstash and I am trying to solve the below error. Requesting your help for the same. \[ERROR\] 2020-07-31 04:48:57.595 \[Converge PipelineAction::Create\\] agent - Failed to execute action … --- ## [Docker:LogStash Graylog](https://discuss.elastic.co/t/docker-logstash-graylog/243599)
**Author:** [@DonPerinon](https://discuss.elastic.co/u/DonPerinon)\ **Replies:** 0\ **Last updated:** [August 3, 2020, 3:26pm UTC](https://discuss.elastic.co/t/docker-logstash-graylog/243599 "2020-08-03T15:26:02Z")
Hi, can anyone help me with my initial logstash+graylog configuration? I have 4 docker container in total: Graylog+elasticsearch+mongo Logstash So my idea was that I will send all my logs (nxlog inputs) to the Logst… --- ## [Need to ingest maillog generated by sendmail to Elasticsearch](https://discuss.elastic.co/t/need-to-ingest-maillog-generated-by-sendmail-to-elasticsearch/243598)
**Author:** [@xzelkworld](https://discuss.elastic.co/u/xzelkworld)\ **Replies:** 0\ **Last updated:** [August 3, 2020, 3:24pm UTC](https://discuss.elastic.co/t/need-to-ingest-maillog-generated-by-sendmail-to-elasticsearch/243598 "2020-08-03T15:24:37Z")
We need to ingest sendmail process generated maillog into Elasticsearch cluster. I have created latest installations of filebeat/logstash/elasticsearch stack on my test environment. I did Google search and found this is … --- ## [File path in connection string](https://discuss.elastic.co/t/file-path-in-connection-string/243578)
**Author:** [@Bob\_Schmidt](https://discuss.elastic.co/u/Bob_Schmidt)\ **Replies:** 0\ **Last updated:** [August 3, 2020, 1:26pm UTC](https://discuss.elastic.co/t/file-path-in-connection-string/243578 "2020-08-03T13:26:22Z")
We are trying to connect to a derby database that has a file path in the conncetion string jdbc:derby://host:port/c:path\_to\_database. We have a successfull connection when using datagrip but with logstash we are getting … --- ## [How to reference filebeat module fields in logstash filter?](https://discuss.elastic.co/t/how-to-reference-filebeat-module-fields-in-logstash-filter/242001)
**Author:** [@inhinyera16](https://discuss.elastic.co/u/inhinyera16)\ **Replies:** 5\ **Last updated:** [August 3, 2020, 1:18pm UTC](https://discuss.elastic.co/t/how-to-reference-filebeat-module-fields-in-logstash-filter/242001 "2020-08-03T13:18:53Z")
We have filebeat apache module enabled which sends to data to logstash for further enrichment (geoip, add more fields for our visualizations etc). I tried this one but im not sure if its possible. this is what we're … --- ## [Import csv files to two different index](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456)
**Author:** [@peretz\_shlomi](https://discuss.elastic.co/u/peretz_shlomi)\ **Replies:** 5\ **Last updated:** [August 3, 2020, 1:09pm UTC](https://discuss.elastic.co/t/import-csv-files-to-two-different-index/242456 "2020-08-03T13:09:25Z")
hello I want to import two different CSV file types into two different index. What is the right way to do it ? --- ## [How to add multiple logstash filter](https://discuss.elastic.co/t/how-to-add-multiple-logstash-filter/243554)
**Author:** [@Puneeth\_S\_B\_Gowda1](https://discuss.elastic.co/u/Puneeth_S_B_Gowda1)\ **Replies:** 0\ **Last updated:** [August 3, 2020, 12:06pm UTC](https://discuss.elastic.co/t/how-to-add-multiple-logstash-filter/243554 "2020-08-03T12:06:11Z")
how to add multiple logstash filter example my source file will come with JSON & XML i have 2 different logstash.conf for xml and JSON log so how do i add two logstash,conf? 5:16 JSON logs will parse from JSON filter a… --- ## [Messages that contain more than 32766 bytes in a field](https://discuss.elastic.co/t/messages-that-contain-more-than-32766-bytes-in-a-field/243545)
**Author:** [@sarabande](https://discuss.elastic.co/u/sarabande)\ **Replies:** 0\ **Last updated:** [August 3, 2020, 11:36am UTC](https://discuss.elastic.co/t/messages-that-contain-more-than-32766-bytes-in-a-field/243545 "2020-08-03T11:36:20Z")
Hi, I have a general question. What happens with messages that contain fields longer than 32766 bytes? They will cause the error saying something like "Could not index event to Elasticsearch" and "caused\_by"=\>{"type"=\>"… --- ## [Kv split into next row](https://discuss.elastic.co/t/kv-split-into-next-row/241024)
**Author:** [@Puneeth\_S\_B\_Gowda1](https://discuss.elastic.co/u/Puneeth_S_B_Gowda1)\ **Replies:** 1\ **Last updated:** [August 3, 2020, 11:08am UTC](https://discuss.elastic.co/t/kv-split-into-next-row/241024 "2020-08-03T11:08:18Z")
input { beats { port =\> 5044 } } filter { kv {field\_split =\> "message" } kv { source =\> "n id" field\_split =\> "," }} output { elasticsearch { hosts =\> \["http://localhost:9200"\] index =\> "atpidev1" } } 11:… --- ## [Logstash on Docker using JDBC input plugin does not fetch all rows from SQL Server](https://discuss.elastic.co/t/logstash-on-docker-using-jdbc-input-plugin-does-not-fetch-all-rows-from-sql-server/243483)
**Author:** [@webstruck](https://discuss.elastic.co/u/webstruck)\ **Replies:** 2\ **Last updated:** [August 3, 2020, 9:44am UTC](https://discuss.elastic.co/t/logstash-on-docker-using-jdbc-input-plugin-does-not-fetch-all-rows-from-sql-server/243483 "2020-08-03T09:44:59Z")
I'm trying to index data into Elasticsearch 7.8.0 from SQL Server through Logstash 7.8.0. All works well when I use non docker version of Logstash. I have more than 6 million rows I need to fetch and the count matches e… --- ## [Edit logstash.yml but it's not updating](https://discuss.elastic.co/t/edit-logstash-yml-but-its-not-updating/243493)
**Author:** [@evacch](https://discuss.elastic.co/u/evacch)\ **Replies:** 0\ **Last updated:** [August 3, 2020, 3:55am UTC](https://discuss.elastic.co/t/edit-logstash-yml-but-its-not-updating/243493 "2020-08-03T03:55:06Z")
I edited my logstash.yml to make some changes on the pipeline.batch.size to 1000, pipeline.batch.delay to 30 and enable xpack monitoring. After that I restart the Logstash service with the following command: systemctl … --- ## [Unable to get Date Time from logs into Timestamp](https://discuss.elastic.co/t/unable-to-get-date-time-from-logs-into-timestamp/243467)
**Author:** [@JonnB](https://discuss.elastic.co/u/JonnB)\ **Replies:** 2\ **Last updated:** [August 2, 2020, 6:00pm UTC](https://discuss.elastic.co/t/unable-to-get-date-time-from-logs-into-timestamp/243467 "2020-08-02T18:00:16Z")
Hi, I have gone through previous posts which do not seem to help with my specific issue and its taking a long time to figure out without progress, if you can assist in the group? I want to get the datetime from my logf… --- ## [Java heap space issue with logstash](https://discuss.elastic.co/t/java-heap-space-issue-with-logstash/243459)
**Author:** [@Gauti](https://discuss.elastic.co/u/Gauti)\ **Replies:** 1\ **Last updated:** [August 2, 2020, 4:12pm UTC](https://discuss.elastic.co/t/java-heap-space-issue-with-logstash/243459 "2020-08-02T16:12:27Z")
Hi All, I have been trying to ingest data which is of 6000 documents, its around 73 MB, but i'm repeatedly getting java heap space issue. below is the error. java.lang.OutOfMemoryError: Java heap space Dumping heap to… --- ## [Import Apache Log File from Win10 to Kibana/Logstash on Debian](https://discuss.elastic.co/t/import-apache-log-file-from-win10-to-kibana-logstash-on-debian/242948)
**Author:** [@imransarkar](https://discuss.elastic.co/u/imransarkar)\ **Replies:** 22\ **Last updated:** [August 2, 2020, 10:15am UTC](https://discuss.elastic.co/t/import-apache-log-file-from-win10-to-kibana-logstash-on-debian/242948 "2020-08-02T10:15:38Z")
Hi Everyone, I am new to Kibana, I have recently uploaded Apache Log File through Filebeat (Installed on Win10) but when I check on Kibana Discovery it is displaying only single entry of entire logfile. My Setup: File… --- ## [Unsafe Object Creation Vulnerability in JSON](https://discuss.elastic.co/t/unsafe-object-creation-vulnerability-in-json/243434)
**Author:** [@Sunil\_Chadha](https://discuss.elastic.co/u/Sunil_Chadha)\ **Replies:** 1\ **Last updated:** [August 1, 2020, 8:28pm UTC](https://discuss.elastic.co/t/unsafe-object-creation-vulnerability-in-json/243434 "2020-08-01T20:28:02Z")
Hi, There is vulnerability reported on json version used in ruby https://www.ruby-lang.org/en/news/2020/03/19/json-dos-cve-2020-10663/ The ruby version used in latest Logstash contains this json version, which contains… --- ## [Connect to mongodb using logstash Jdbc\_streaming filter plugin](https://discuss.elastic.co/t/connect-to-mongodb-using-logstash-jdbc-streaming-filter-plugin/242082)
**Author:** [@\_Rasitha\_Ariyarathna](https://discuss.elastic.co/u/_Rasitha_Ariyarathna)\ **Replies:** 2\ **Last updated:** [August 1, 2020, 1:47pm UTC](https://discuss.elastic.co/t/connect-to-mongodb-using-logstash-jdbc-streaming-filter-plugin/242082 "2020-08-01T13:47:03Z")
I'm trying to fetch data from mongodb using Jdbc\_streaming filter plugin in logstash in windows. I'm using mongo-java-driver-3.4.2.jar to connect to the database but, getting a error like this JavaSql::SQLException: N… --- ## [Can I filter logs with if else condition?](https://discuss.elastic.co/t/can-i-filter-logs-with-if-else-condition/243421)
**Author:** [@Bhavin\_Varsur](https://discuss.elastic.co/u/Bhavin_Varsur)\ **Replies:** 1\ **Last updated:** [August 1, 2020, 10:54am UTC](https://discuss.elastic.co/t/can-i-filter-logs-with-if-else-condition/243421 "2020-08-01T10:54:12Z")
my logs chain is Filebeat =\> Logstash =\> Elasticsearch Now i want to filterize logs with my own field. like match or if else condition key value. So Can I do that or what? need to know how can i do? any blog suggest… --- ## [Add Sub fields](https://discuss.elastic.co/t/add-sub-fields/243414)
**Author:** [@nurhambali](https://discuss.elastic.co/u/nurhambali)\ **Replies:** 4\ **Last updated:** [August 1, 2020, 4:38am UTC](https://discuss.elastic.co/t/add-sub-fields/243414 "2020-08-01T04:38:30Z")
hi, how to add sub fields on logstash ex: server, time, message i'm add sub field modsec.server modsec.time modsec.message ? any ide ? --- ## [Block in start\_input](https://discuss.elastic.co/t/block-in-start-input/243410)
**Author:** [@Inder](https://discuss.elastic.co/u/Inder)\ **Replies:** 1\ **Last updated:** [August 1, 2020, 1:58am UTC](https://discuss.elastic.co/t/block-in-start-input/243410 "2020-08-01T01:58:07Z")
Hi, This is my conf file: input { file { path =\> "/Users/Parm/cars.csv" start\_position =\> "beginning" sincedb\_path =\> "/dev/null" } } filter { csv { separator =\> "," columns =\> \[ "maker", "model", "mileage", "… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=319) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=321)