# Logstash **URL:** https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=944 [Latest](https://discuss.elastic.co/latest.md) · [Categories](https://discuss.elastic.co/categories.md) · [Tags](https://discuss.elastic.co/tags.md) **Page:** 945 --- ## [Unable to index JSONB column from Postgresql View](https://discuss.elastic.co/t/unable-to-index-jsonb-column-from-postgresql-view/93310)
**Author:** [@sean\_lindo](https://discuss.elastic.co/u/sean_lindo)\ **Replies:** 1\ **Last updated:** [July 16, 2017, 5:14pm UTC](https://discuss.elastic.co/t/unable-to-index-jsonb-column-from-postgresql-view/93310 "2017-07-16T17:14:09Z")
Hi all, I have a view in Postgres with a single column of type jsonb. To get around some weirdness, I decided to update the view and cast this column as text (it appears I'm not able to access a PGObject, hence the upd… --- ## [Multi-thread aggregate or other strategies?](https://discuss.elastic.co/t/multi-thread-aggregate-or-other-strategies/93307)
**Author:** [@Mike\_Nielsen](https://discuss.elastic.co/u/Mike_Nielsen)\ **Replies:** 0\ **Last updated:** [July 16, 2017, 4:30pm UTC](https://discuss.elastic.co/t/multi-thread-aggregate-or-other-strategies/93307 "2017-07-16T16:30:03Z")
Hi, I'm new to Logstash\[5.5\] and I'd love some advice from the community on this problem: I have a number of Postgres servers on which a monitoring tool is run. This monitoring tool runs a query that generates a multi… --- ## [Extracting the members of the Winlogbeat event\_data\_MemberName array](https://discuss.elastic.co/t/extracting-the-members-of-the-winlogbeat-event-data-membername-array/93305)
**Author:** [@Callahan](https://discuss.elastic.co/u/Callahan)\ **Replies:** 0\ **Last updated:** [July 16, 2017, 2:12pm UTC](https://discuss.elastic.co/t/extracting-the-members-of-the-winlogbeat-event-data-membername-array/93305 "2017-07-16T14:12:26Z")
Hi, So as the title says, I'd like to use individual members of the event\_data\_MemberName array returned from Winlogbeat running on one of our DCs. The end goal is to create a filter that will allow me to create a tabl… --- ## [How do I get the raw message of the netflow/ipfix codec?](https://discuss.elastic.co/t/how-do-i-get-the-raw-message-of-the-netflow-ipfix-codec/93185)
**Author:** [@red888](https://discuss.elastic.co/u/red888)\ **Replies:** 4\ **Last updated:** [July 16, 2017, 10:09am UTC](https://discuss.elastic.co/t/how-do-i-get-the-raw-message-of-the-netflow-ipfix-codec/93185 "2017-07-16T10:09:54Z")
Im using the netflow codec and I seems it doesn't include a raw message field when it outputs to es. for input I just do this: udp { type =\> "ipfix" port =\> "123" codec =\> netflow { … --- ## [Geoip filter plugin missing geohash field](https://discuss.elastic.co/t/geoip-filter-plugin-missing-geohash-field/93250)
**Author:** [@fpr](https://discuss.elastic.co/u/fpr)\ **Replies:** 1\ **Last updated:** [July 15, 2017, 7:25pm UTC](https://discuss.elastic.co/t/geoip-filter-plugin-missing-geohash-field/93250 "2017-07-15T19:25:54Z")
Hi, after upgrading to logstash-filter-geoip (4.2.1) there isn't the geo\_point -\> \[geoip\]\[location\] field anymore. Only the two fields \[geoip\]\[location\]\[lat\] and \[geoip\]\[location\]\[lon\] are created. I am using the defa… --- ## [How to use grok pattern to separate nginx access log fields !?](https://discuss.elastic.co/t/how-to-use-grok-pattern-to-separate-nginx-access-log-fields/92645)
**Author:** [@omid](https://discuss.elastic.co/u/omid)\ **Replies:** 3\ **Last updated:** [July 15, 2017, 6:51am UTC](https://discuss.elastic.co/t/how-to-use-grok-pattern-to-separate-nginx-access-log-fields/92645 "2017-07-15T06:51:24Z")
i want from access log nginx from elk\_stack, Of all fields ((timestamp, version, host, clientip, ident, auth, verb, request, httpversion, response, bytes, referrer, agent)) Separately from each field Visualize and make a… --- ## [Logstash consumes same messages from Kafka](https://discuss.elastic.co/t/logstash-consumes-same-messages-from-kafka/92904)
**Author:** [@John\_06](https://discuss.elastic.co/u/John_06)\ **Replies:** 1\ **Last updated:** [July 15, 2017, 6:26am UTC](https://discuss.elastic.co/t/logstash-consumes-same-messages-from-kafka/92904 "2017-07-15T06:26:01Z")
I have a problem when Logstash consumes data from Kafka - it consumes the same data over and over again. I have sent several hundred messages to Kafka but Logstash has been consuming them for about 3 hours and the tot… --- ## [Logstash grok filter for apache customized logs](https://discuss.elastic.co/t/logstash-grok-filter-for-apache-customized-logs/92915)
**Author:** [@marcelo](https://discuss.elastic.co/u/marcelo)\ **Replies:** 11\ **Last updated:** [July 15, 2017, 3:17am UTC](https://discuss.elastic.co/t/logstash-grok-filter-for-apache-customized-logs/92915 "2017-07-15T03:17:20Z")
Hi guys, I'm trying to use the elk to generate dashboards of apache access logs. However I can't split up in various fields the message field. It turns out that my apache access logs are a little customized because I … --- ## [Filebeat or logstash is better to use](https://discuss.elastic.co/t/filebeat-or-logstash-is-better-to-use/93103)
**Author:** [@P\_Kumar](https://discuss.elastic.co/u/P_Kumar)\ **Replies:** 2\ **Last updated:** [July 15, 2017, 2:11am UTC](https://discuss.elastic.co/t/filebeat-or-logstash-is-better-to-use/93103 "2017-07-15T02:11:25Z")
Hi , Logstash can transfer logs to elasticsearch and fillebeat can transfer to logstash then elasticsearch . so if we will use logstash and ignore the filebeat in client machine . Is it ok or any special reason to add … --- ## [Sqlite plugin encountering error](https://discuss.elastic.co/t/sqlite-plugin-encountering-error/93245)
**Author:** [@rsaeks](https://discuss.elastic.co/u/rsaeks)\ **Replies:** 0\ **Last updated:** [July 14, 2017, 9:44pm UTC](https://discuss.elastic.co/t/sqlite-plugin-encountering-error/93245 "2017-07-14T21:44:27Z")
I'm using the sqlite logstash input plugin (logstash-input-sqlite (3.0.1)) and have built the following configuration: input { sqlite { path =\> "/tmp/db2.sqlite" type =\> weblogs } } output { file { path =\>… --- ## [Handling multiple date formats in logstash](https://discuss.elastic.co/t/handling-multiple-date-formats-in-logstash/93238)
**Author:** [@lleatherby](https://discuss.elastic.co/u/lleatherby)\ **Replies:** 2\ **Last updated:** [July 14, 2017, 9:17pm UTC](https://discuss.elastic.co/t/handling-multiple-date-formats-in-logstash/93238 "2017-07-14T21:17:00Z")
I have a dataset with multiple date formats. if I use this filter: grok { match =\> \[ "message", "(?%{YEAR:year}-%{MONTHNUM:month}-%{MONTHDAY:day}T%{TIME:time})%{SPACE}%{GREEDYDATA:text}" … --- ## [Altering the content of a field](https://discuss.elastic.co/t/altering-the-content-of-a-field/93082)
**Author:** [@mahsa\_b](https://discuss.elastic.co/u/mahsa_b)\ **Replies:** 2\ **Last updated:** [July 14, 2017, 8:24pm UTC](https://discuss.elastic.co/t/altering-the-content-of-a-field/93082 "2017-07-14T20:24:09Z")
Hi , I have a grok filter like this grok { patterns\_dir =\> "./patterns/custom-patterns" match =\> {"message" =\> \["%{IP:remoteIp}%{SPACE}%{MAC:remoteMac}%{SPACE}%{NUMBER}%{SPACE}%{WORD}%{SPACE}%{PORT:… --- ## [Tibco ems logstash input plugin working](https://discuss.elastic.co/t/tibco-ems-logstash-input-plugin-working/93226)
**Author:** [@chetan\_vartak](https://discuss.elastic.co/u/chetan_vartak)\ **Replies:** 0\ **Last updated:** [July 14, 2017, 5:54pm UTC](https://discuss.elastic.co/t/tibco-ems-logstash-input-plugin-working/93226 "2017-07-14T17:54:27Z")
I saw few topics on tibco ems. But none of them had a solution. I got it working. Here is the setting I used to consume messages from a queue. jms.yml --\> # Tibco EMS ems: :factory: com.tibco.tibjms.TibjmsConnecti… --- ## [CloudWatch input plugin config to receive all data](https://discuss.elastic.co/t/cloudwatch-input-plugin-config-to-receive-all-data/93232)
**Author:** [@rpuserh](https://discuss.elastic.co/u/rpuserh)\ **Replies:** 0\ **Last updated:** [July 14, 2017, 6:20pm UTC](https://discuss.elastic.co/t/cloudwatch-input-plugin-config-to-receive-all-data/93232 "2017-07-14T18:20:03Z")
Hi, I seted up API Gateway and enabled logging to cloud watch Now I want to import access log to ES In cloud-watch I get new object called API-Gateway-Execution-Logs\_xv87vewh1w/stage and a lot of stream under it I tr… --- ## [Logstash TCP connection leak](https://discuss.elastic.co/t/logstash-tcp-connection-leak/92119)
**Author:** [@red888](https://discuss.elastic.co/u/red888)\ **Replies:** 3\ **Last updated:** [July 14, 2017, 5:55pm UTC](https://discuss.elastic.co/t/logstash-tcp-connection-leak/92119 "2017-07-14T17:55:01Z")
Logstash: 5.3.0 Logstash is keeping TCP connections open even for servers that are shutdown. It also seems to be preventing the OS from closing the connections via TCP keep alive- I just have the default settings: … --- ## [Logstash 5.5.0 ERROR StatusLogger No log4j2 configuration file found](https://discuss.elastic.co/t/logstash-5-5-0-error-statuslogger-no-log4j2-configuration-file-found/92921)
**Author:** [@dawncold](https://discuss.elastic.co/u/dawncold)\ **Replies:** 1\ **Last updated:** [July 14, 2017, 5:48pm UTC](https://discuss.elastic.co/t/logstash-5-5-0-error-statuslogger-no-log4j2-configuration-file-found/92921 "2017-07-14T17:48:43Z")
After I upgraded logstash from 5.4.1 to 5.5.0, I got this error: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Sending Logstash's logs to /var/… --- ## [Autodetect columns from header row - csv ingest](https://discuss.elastic.co/t/autodetect-columns-from-header-row-csv-ingest/92861)
**Author:** [@jay224](https://discuss.elastic.co/u/jay224)\ **Replies:** 0\ **Last updated:** [July 12, 2017, 6:23pm UTC](https://discuss.elastic.co/t/autodetect-columns-from-header-row-csv-ingest/92861 "2017-07-12T18:23:14Z")
Is it possible to autodetect columns from header(first) row in csv file? I tried to use autodetect\_column\_names =\> true for logstash 5.5 that seems to be skipping actual header row and take second row as header row. Th… --- ## [NameError with DLQ input plugin](https://discuss.elastic.co/t/nameerror-with-dlq-input-plugin/92688)
**Author:** [@scarletdenizen](https://discuss.elastic.co/u/scarletdenizen)\ **Replies:** 2\ **Last updated:** [July 14, 2017, 3:01pm UTC](https://discuss.elastic.co/t/nameerror-with-dlq-input-plugin/92688 "2017-07-14T15:01:58Z")
I'm trying to use the dead\_letter\_queue input plugin to reintroduce Elasticsearch events that get dropped back into the Logstash pipe and tag them. It appears as the Logstash 5.5 distribution fails to find the dead\_lette… --- ## [Add Gem to Logstash production](https://discuss.elastic.co/t/add-gem-to-logstash-production/92761)
**Author:** [@kitz99](https://discuss.elastic.co/u/kitz99)\ **Replies:** 1\ **Last updated:** [July 14, 2017, 2:58pm UTC](https://discuss.elastic.co/t/add-gem-to-logstash-production/92761 "2017-07-14T14:58:35Z")
Hello. I have installed a logstash 5.4 in a production env (RHEL 7) and now I need to add a new config, a new gem and some ruby classes to this logstash. Where I need to put my ruby code, and how can I install new gems? … --- ## [No entries in kibana for a specific host](https://discuss.elastic.co/t/no-entries-in-kibana-for-a-specific-host/93204)
**Author:** [@andsk8](https://discuss.elastic.co/u/andsk8)\ **Replies:** 0\ **Last updated:** [July 14, 2017, 2:33pm UTC](https://discuss.elastic.co/t/no-entries-in-kibana-for-a-specific-host/93204 "2017-07-14T14:33:26Z")
Hi, I have a ELK stack working with filebeat in every server I want to retrieve logs from. I have nginx, apache and monolog logs that work perfectly. Every kind of of log has his own .conf file in the conf.d folder in /e… --- ## [Logstash-tcp TLS not terminating correctly](https://discuss.elastic.co/t/logstash-tcp-tls-not-terminating-correctly/93201)
**Author:** [@runtman](https://discuss.elastic.co/u/runtman)\ **Replies:** 1\ **Last updated:** [July 14, 2017, 2:10pm UTC](https://discuss.elastic.co/t/logstash-tcp-tls-not-terminating-correctly/93201 "2017-07-14T14:10:14Z")
Hello, so I use two listeners on my logstash setup. Beats and TCP (using the plugin). Beats works fine over TLS \* ALPN, server did not agree to a protocol \> GET / HTTP/1.1 \> Host: logstash01.xx:5044 \> User-Agent: c… --- ## [DayOfWeek from log timestamp using logstash](https://discuss.elastic.co/t/dayofweek-from-log-timestamp-using-logstash/93198)
**Author:** [@anisen](https://discuss.elastic.co/u/anisen)\ **Replies:** 0\ **Last updated:** [July 14, 2017, 1:32pm UTC](https://discuss.elastic.co/t/dayofweek-from-log-timestamp-using-logstash/93198 "2017-07-14T13:32:57Z")
Hi, My log looks like this: 66.221.154.0 - - \[05/Feb/2014:17:11:55 +0530\] "GET /mySite/HomePage.asmx HTTP/1.1" 200 110 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.5 … --- ## [Logstash Custom Filters and Patterns](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031)
**Author:** [@thecrimsoncoder](https://discuss.elastic.co/u/thecrimsoncoder)\ **Replies:** 10\ **Last updated:** [July 14, 2017, 12:38pm UTC](https://discuss.elastic.co/t/logstash-custom-filters-and-patterns/93031 "2017-07-14T12:38:13Z")
Hello All, I am very new to the ELK stack and I have been tasked with parsing out some custom logs. I feel like im on the right track but it keeps breaking and Im not sure what Im doing wrong. Basically what I am trying… --- ## [ETL with logstash](https://discuss.elastic.co/t/etl-with-logstash/92979)
**Author:** [@sanketshinde](https://discuss.elastic.co/u/sanketshinde)\ **Replies:** 2\ **Last updated:** [July 14, 2017, 12:29pm UTC](https://discuss.elastic.co/t/etl-with-logstash/92979 "2017-07-14T12:29:09Z")
I am trying to implement an ETL using Logstash. The ETL is between two instances of ES DB, from a remote server to a localhost. I want to get the max of a field "endTime" from the documents in my localhost. I want to the… --- ## [Logstash cpu utilization is 100%](https://discuss.elastic.co/t/logstash-cpu-utilization-is-100/93179)
**Author:** [@deeps123](https://discuss.elastic.co/u/deeps123)\ **Replies:** 0\ **Last updated:** [July 14, 2017, 11:42am UTC](https://discuss.elastic.co/t/logstash-cpu-utilization-is-100/93179 "2017-07-14T11:42:31Z")
Hi, I am using 5.1.3 logstash. My configuration contains tcp as input with multiline codec , followed by filter and output to file. I have 4core CPU with 16G RAM on my redhat machine. openjsk1.8 i am using. I trie… --- ## [Cannot start bin/logstash](https://discuss.elastic.co/t/cannot-start-bin-logstash/93143)
**Author:** [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\ **Replies:** 7\ **Last updated:** [July 14, 2017, 9:42am UTC](https://discuss.elastic.co/t/cannot-start-bin-logstash/93143 "2017-07-14T09:42:19Z")
I installed logstash through yum repo. After installation I make config file in /etc/logstash/conf.d when i started bin/logstash -f it outputs "No such file or directory" --- ## [Email plugin not working](https://discuss.elastic.co/t/email-plugin-not-working/92776)
**Author:** [@desai.amogh](https://discuss.elastic.co/u/desai.amogh)\ **Replies:** 3\ **Last updated:** [July 14, 2017, 7:48am UTC](https://discuss.elastic.co/t/email-plugin-not-working/92776 "2017-07-14T07:48:36Z")
I tried this without success. output { elasticsearch { hosts =\> \["localhost:9200"\] index =\> "logstash-%{+YYYY.MM.dd}" } file { … --- ## [Mutate WinlogBeat](https://discuss.elastic.co/t/mutate-winlogbeat/93098)
**Author:** [@Nobby](https://discuss.elastic.co/u/Nobby)\ **Replies:** 3\ **Last updated:** [July 14, 2017, 7:46am UTC](https://discuss.elastic.co/t/mutate-winlogbeat/93098 "2017-07-14T07:46:51Z")
Hey guys, I'm trying to use mutate to define the data type of various fields generated in some Windows logs. For example, Remote Desktop Gateway has number fields for user\_data.BytesReceived, user\_data.BytesTransfered … --- ## [Json is storing as string in es](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062)
**Author:** [@Narayanan\_Sukumar](https://discuss.elastic.co/u/Narayanan_Sukumar)\ **Replies:** 4\ **Last updated:** [July 14, 2017, 7:07am UTC](https://discuss.elastic.co/t/json-is-storing-as-string-in-es/93062 "2017-07-14T07:07:36Z")
ELK I created is filebeat - \> KAFKA -\> logstash -\> ES &Kibana (AWS elastic search) the logs are generated in json format and it is pushed to kafka with below config: ========================= filebeat.prospectors: i… --- ## [Grok add field and add tag not working](https://discuss.elastic.co/t/grok-add-field-and-add-tag-not-working/93095)
**Author:** [@arunvijayam](https://discuss.elastic.co/u/arunvijayam)\ **Replies:** 1\ **Last updated:** [July 14, 2017, 6:55am UTC](https://discuss.elastic.co/t/grok-add-field-and-add-tag-not-working/93095 "2017-07-14T06:55:49Z")
filter { mutate { add\_field =\> { "message" =\> "%{global\_stats},%{num\_rows}" } remove\_field =\> \["oldfield1", "oldfield"\] } grok { add\_tag =\> \[ "valid" \] match =\> \[ "message" , "^YES,\[123456789\]+$"\] } grok… [Previous page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=943) [Next page](https://discuss.elastic.co/c/elastic-stack/logstash/14.md?page=945)