# Memcached in logstash **URL:** \ **Category:** Logstash\ **Created:** [March 8, 2024, 5:41pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047 "2024-03-08T17:41:19Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [March 8, 2024, 5:41pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/1 "2024-03-08T17:41:19Z")
I have huge file that I am using in logstash with translate filter. But I think it is causing some issue as it misses some match for some event. By looking around I discover that memcached may be answer to that. but I can't work out that filter. ```auto memcached { hosts=> ["mem_host1"] get => { "%{project}" => "[site]" } add_tag => ["from_cache"] } ``` I have memcached running on mem\_host1. and I can see key exist but logstash is not producing field called site. project field is also exist on output of logstash what am I doing wrong? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 8, 2024, 6:59pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/2 "2024-03-08T18:59:29Z")
Can you share a sample of your data and the rest of your configuration pipeline? Also share the output that logstash is generating. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [March 8, 2024, 8:07pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/3 "2024-03-08T20:07:23Z")
By the way I am following your blog post for this [[logstash: using the memcached filter | @leandrojmp](https://web.leandrojmp.com/posts/en/2021/04/logstash-memcached)] ([logstash: using the memcached filter | @leandrojmp](https://web.leandrojmp.com/posts/en/2021/04/logstash-memcached)) here is input, I am trying to replace translate filter. ```auto # Add sitefrom translate filter. -- this is working translate { source => "[project]" target => "[site_translate]" dictionary_path => "/s1/logstash/csv_files/project_center.csv" fallback => "na" refresh_interval => 36000 } mutate { add_field => { "site" => "sachin........................." } } # use memcached filter -- this is not working memcached { hosts=> ["10.29.249.111"] namespace => "convert_mm" get => { "%{project}" => "[site1]" } add_tag => ["from_cache"] } ``` here is output ```auto { "site_translate" => "crawley", "job" => 13495137, "project" => "3dsymra" "site" => "sachin........................." } ``` here is memcached result from memcached server. I preloaded key:value using python. ```auto # cat memcached_get.py #!/usr/bin/python3 import memcache mc_client = memcache.Client(['10.29.249.111:11211']) # Retrieve the value for the key value = mc_client.get('3dsymra') # Print the value print(value) ``` Execute code gives me value ```auto # ./memcached_get.py crawley ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 8, 2024, 8:15pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/4 "2024-03-08T20:15:12Z")
> [@elasticforme](#): > > `namespace => "convert_mm"` Oh, you are using a _namespace_ in the `memcached` filter, but in your python example your keys do not have a namespace. When you use a namespace, the filter will look for a key named `namespace:key` as explained in the [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-filters-memcached.html#plugins-filters-memcached-namespace). So your memcached filter is looking for a key named `convert_mm:3dsymra`, but it seems that the key name in your memcached is just `3dsymra`. Can you remove the `namespace` option from your filter and test again? ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [March 8, 2024, 8:50pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/5 "2024-03-08T20:50:17Z")
yes it works. question I use namespace if I am loading more then few other key:value pair like project - site project - user user - address right? then I can use three different namespace on same memcached server to pull info correct? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 8, 2024, 8:53pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/6 "2024-03-08T20:53:50Z")
> [@elasticforme](#): > > then I can use three different namespace on same memcached server to pull info correct? Yeah, you can use the same memcached server and different datasets if you use namespace. Basically you need to prefix all your keys with some name for your namespace, then you can use this same name in your memcached. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [March 8, 2024, 9:19pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/7 "2024-03-08T21:19:12Z")
excellent this is final version. I will do large scale testing now. python3 to load in to memcache ```auto with open ('/home/sachin/project_center.csv','rb') as f: for line in f: # use decod as it is string, # then split to key-value pair and insert in to memcached line = line.decode() fields = line.split(',') (project,site) = fields project = "proj_site_"+project my_dict[project] = site mc_client.set_multi(my_dict) ``` this is to retrive in logstash ```auto memcached { hosts=> ["10.29.249.111"] get => { "proj_site_%{project}" => "[site]" } } ``` problem with this is that if memcached server is down then logstash fails. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [March 8, 2024, 9:27pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/8 "2024-03-08T21:27:31Z")
> [@elasticforme](#): > > problem with this is that if memcached server is down then logstash fails. If I'm not wrong, Logstash won't start if any configuration has a memcached filter and the memcached server is down. But if the memcached server goes down after logstash already started, then Logstash will keep running, but the memcached will stop working. ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 8, 2024, 10:24pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/9 "2024-03-08T22:24:09Z")
> [@leandrojmp](#): > > If I'm not wrong, Logstash won't start if any configuration has a memcached filter and the memcached server is down. You are not wrong 😄 If it cannot get a connection to the memcached server then the plugin's register function will [raise a RuntimeError](https://github.com/logstash-plugins/logstash-filter-memcached/blob/cb86678ac7ab5f9c0b472c6f923ed8c8b759c841/lib/logstash/filters/memcached.rb#L87), and that prevents the pipeline starting. If the pipeline loses the connection it will keep trying to reconnect and just tag the failure on the event if it cannot. I would expect the reconnect attempts to significantly reduce throughput. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [March 8, 2024, 10:50pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/10 "2024-03-08T22:50:00Z")
what about if I have multiple memcached server and place that in hosts. what if one fails. will it go to second one? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [March 8, 2024, 11:11pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/11 "2024-03-08T23:11:08Z")
My reading of the [documentation](https://www.rubydoc.info/github/mperham/dalli/Dalli%2FClient:initialize) is that it will. > :failover - if a server is down, look for and store values on another server in the ring. Default: true It is on by default and the plugin [does not change](https://github.com/logstash-plugins/logstash-filter-memcached/blob/cb86678ac7ab5f9c0b472c6f923ed8c8b759c841/lib/logstash/filters/memcached.rb#L206) the default. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 5, 2024, 11:12pm UTC](https://discuss.elastic.co/t/memcached-in-logstash/355047/12 "2024-04-05T23:12:07Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.