# Logstash filter for FTP logs **URL:** \ **Category:** Logstash\ **Created:** [July 4, 2019, 11:04am UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896 "2019-07-04T11:04:58Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![akshay\_singh2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_singh2/32/47046_2.png) [@akshay\_singh2](https://discuss.elastic.co/u/akshay_singh2)\ **Post date:** [July 4, 2019, 11:04am UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896/1 "2019-07-04T11:04:58Z")
Hello Team, I need to put filters for logstash for querying data coming from FTP servers. I havent worked on filters much, just have rough idea, so i created one for some below test logs: ``` Thu Jul 4 06:01:45 2019 [pid 43249] [xyz] OK DOWNLOAD: Client \"x.x.x.x\", \"/commonupdater/sitestat.xml\", 118 bytes, 2.64Kbyte/sec","popId":"1","hostIpAddress":"x.x.x.x","host":"ftp-1-2","data_field":"raw","type":"ftp-log Thu Jul 4 06:20:14 2019 [pid 55668] [xyz] OK DOWNLOAD: Client \"x.x.x.x\", \"/commonupdater/sitestat.xml\", 118 bytes, 2.58Kbyte/sec","popId":"2","hostIpAddress":"x.x.x.x","host":"ftp-2-2","data_field":"raw","type":"ftp-log" Thu Jul 4 06:20:13 2019 [pid 55666] [xyz] OK LOGIN: Client \"x.x.x.x\", anon password \"NcFTP@\"","popId":"3","hostIpAddress":"x.x.x.x","host":"ftp-2-3","data_field":"raw","type":"ftp-log" Thu Jul 4 06:20:13 2019 [pid 55667] CONNECT: Client \"x.x.x.x\"","popId":"4","hostIpAddress":"x.x.x.x","host":"ftp-1-2","data_field":"raw","type":"ftp-log" Thu Jul 4 06:20:11 2019 [pid 43201] CONNECT: Client \"x.x.x.x\"","popId":"5","hostIpAddress":"x.x.x.x","host":"ftp-2-4","data_field":"raw","type":"ftp-log" ``` In these logs line I need following filters, rest can be ignored: 1. Status: OK DOWNLOAD, FAIL DOWNLOAD, CONNECT 2. Client IP 3. File Name: /commonupdater/sitestat.xml, etc. 4. Size of file: In bytes 5. Download rate: In bytes/sec For this I created fiter pattern: ``` filter { grok { match => {"message" => "%{MONTH} +%{MONTHDAY} %{TIME} %{YEAR} (\[%{GREEDYDATA:pidno}\] )?(\[%{WORD:comp}\] )?(%{WORD:status} )?(%{WORD:download}:)?(%{WORD:client} )?(\"%{IPV4:ipaddr}\", )?(\"%{GREEDYDATA:filename}\", )?(%{GREEDYDATA:size} )?(%{GREEDYDATA:speed} )?"} } mutate { remove_field => ["pidno", "comp", "download", "client"] } } ``` Thanks in Advance ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 4, 2019, 2:22pm UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896/2 "2019-07-04T14:22:21Z")
Do not bother naming fields if you are just going to drop them. Instead of %{GREEDYDATA:pidno} you can just use %{GREEDYDATA} Personally I would use dissect to take off the prefix to the line that is consistently formatted. ``` dissect { mapping => { "message" => "%{[@metadata][ts]} %{+[@metadata][ts]->} %{+[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} [pid %{}] %{[@metadata][restOfLine]}" } } grok { match => { "[@metadata][restOfLine]" => '(\[%{WORD}\] )?%{DATA:operation}: Client "%{IPV4:ipaddr}",' } } if "DOWNLOAD" in [@metadata][restOfLine] { grok { match => { "[@metadata][restOfLine]" => 'Client "%{IPV4}", "(?[^"]+)", %{INT:filesize:int} bytes, %{NUMBER:rate:float}Kbyte/sec' } } } date { match => ["[@metadata][ts]", "EEE MMM dd HH:mm:ss YYYY" ] } ``` ---
**Author:** ![akshay\_singh2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_singh2/32/47046_2.png) [@akshay\_singh2](https://discuss.elastic.co/u/akshay_singh2)\ **Post date:** [July 5, 2019, 5:11am UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896/3 "2019-07-05T05:11:14Z")
> [@Badger](#): > > Personally I would use dissect to take off the prefix to the line that is consistently formatted Thanks Badger, Is there any docs or some site to refer with and study more about this. ---
**Author:** ![Miguel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/miguel1/32/82094_2.png) [@Miguel1](https://discuss.elastic.co/u/Miguel1)\ **Post date:** [July 5, 2019, 7:44am UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896/4 "2019-07-05T07:44:51Z")
There's and old post about it, but you can take it as a reference ([here](https://www.elastic.co/es/blog/logstash-dude-wheres-my-chainsaw-i-need-to-dissect-my-logs)) ---
**Author:** ![akshay\_singh2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_singh2/32/47046_2.png) [@akshay\_singh2](https://discuss.elastic.co/u/akshay_singh2)\ **Post date:** [July 5, 2019, 9:03am UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896/5 "2019-07-05T09:03:20Z")
> [@Badger](#): > > Do not bother naming fields if you are just going to drop them. Instead of %{GREEDYDATA:pidno} you can just use %{GREEDYDATA} I tried that one, but still same error. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 5, 2019, 2:54pm UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896/6 "2019-07-05T14:54:46Z")
> [@akshay\_singh2](#): > > I tried that one, but still same error. What error? The set of filters I posted works against the set of data you posted. ---
**Author:** ![akshay\_singh2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/akshay_singh2/32/47046_2.png) [@akshay\_singh2](https://discuss.elastic.co/u/akshay_singh2)\ **Post date:** [July 8, 2019, 5:08am UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896/8 "2019-07-08T05:08:21Z")
> [@Badger](#): > > dissect { mapping =\> { "message" =\> "%{[@metadata][ts]} %{+[@metadata][ts]-\>} %{+[@metadata][ts]} %{+[@metadata][ts]} %{+[@metadata][ts]} [pid %{}] %{[@metadata][restOfLine]}" } } After setting filter given by you, I cant see any logs coming to elasticsearch even though there are no error messages in logstash logs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 5, 2019, 5:08am UTC](https://discuss.elastic.co/t/logstash-filter-for-ftp-logs/188896/9 "2019-08-05T05:08:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.