# Problems with GrayLog2 + ES setup \[long\] **URL:** \ **Category:** Elasticsearch\ **Created:** [March 26, 2012, 8:40pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134 "2012-03-26T20:40:51Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![MichaelGlad](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MichaelGlad](https://discuss.elastic.co/u/MichaelGlad)\ **Post date:** [March 26, 2012, 8:40pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/1 "2012-03-26T20:40:51Z")
I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It seems I've hit a wall. Java is CPU bound and there's next to no disk activity. As I believe that the cure could be tuning ES or adding hardware resources, I'm posting to this list. The indexing capacity I would like is: - Documents: long syslog mesages (160 chars avg) from a busy mail filter - Message load exceeding 200 messages/second - Capacity for storing 60 days worth of log messages, that's about 1 billion . - I currently have some 800 million mesages on disk, using about 5x140 gigs of disk. My setup is ( I'm using 64 bit Redhat Linux) - One VM running GrayLog2 server + a ES server with no local shards. - Two VMs with a total of 5 shards distributed with 3 / 2 on each. The two VMs with shards have 32 gigs of memory and 4 cores in a VMware environment. I've applied the following changes to the bin/elasticsearch script: ulimit -n 60000 # fs.file-max = 131000 ulimit -l unlimited export ES\_HEAP\_SIZE=16g export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 export JAVA\_OPTS="-Xloggc:/tmp/gc" and disabled the swap area to prevent swapping. Java on the two VMs containing shards, use about 12 gigs out memory and all CPU resources. The first VM, is not significantly loaded. What could a solution be -- "Kiwi" (kill it with Iron) or should I rather change the ES configuration? Regards, Michael ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 27, 2012, 9:52am UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/2 "2012-03-27T09:52:00Z")
Hi Michael, So what exactly happened? Inserts work slowly, or queries? Or both? I'm also using Elasticsearch for logging on VMs (although I'm not using Graylog) and my "wall" was on inserts, and it was heavily influenced by the storage speed. You kind of need that, since I can bet your index size is much bigger than the total size of RAM. Some of the stuff you can do is to increase the number of shards if you have no replicas configured (although this implies recreating the index AFAIK). And we've also did here the following: - increasted the refresh interval to 3 - compressed \_source - disabled \_all (you need to check first if Graylog uses \_all for searching :D) On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > seems I've hit a wall. Java is CPU bound and there's next to no disk > activity. > As I believe that the cure could be tuning ES or adding hardware > resources, I'm posting to this list. > > The indexing capacity I would like is: > > - Documents: long syslog mesages (160 chars avg) from a busy mail > filter > - Message load exceeding 200 messages/second > - Capacity for storing 60 days worth of log messages, that's about 1 > billion . > - I currently have some 800 million mesages on disk, using about 5x140 > gigs of disk. > > My setup is ( I'm using 64 bit Redhat Linux) > > - One VM running GrayLog2 server + a ES server with no local shards. > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > environment. > > I've applied the following changes to the bin/elasticsearch script: > > ulimit -n 60000 # fs.file-max = 131000 > ulimit -l unlimited > export ES\_HEAP\_SIZE=16g > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > and disabled the swap area to prevent swapping. > > Java on the two VMs containing shards, use about 12 gigs out memory > and all CPU resources. > The first VM, is not significantly loaded. > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > rather change the ES configuration? > > Regards, Michael ---
**Author:** ![Michael\_Sick](https://avatars.discourse-cdn.com/v4/letter/m/22d042/32.png) [@Michael\_Sick](https://discuss.elastic.co/u/Michael_Sick)\ **Post date:** [March 27, 2012, 12:35pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/3 "2012-03-27T12:35:47Z")
Also, if your inserts are slow are you using bulk inserts? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Tue, Mar 27, 2012 at 5:52 AM, Radu Gheorghe [radu0gheorghe@gmail.com](mailto:radu0gheorghe@gmail.com)wrote: > Hi Michael, > > So what exactly happened? Inserts work slowly, or queries? Or both? > > I'm also using Elasticsearch for logging on VMs (although I'm not > using Graylog) and my "wall" was on inserts, and it was heavily > influenced by the storage speed. You kind of need that, since I can > bet your index size is much bigger than the total size of RAM. > > Some of the stuff you can do is to increase the number of shards if > you have no replicas configured (although this implies recreating the > index AFAIK). And we've also did here the following: > > - increasted the refresh interval to 3 > - compressed \_source > - disabled \_all (you need to check first if Graylog uses \_all for > searching :D) > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > activity. > > As I believe that the cure could be tuning ES or adding hardware > > resources, I'm posting to this list. > > > > The indexing capacity I would like is: > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > filter > > - Message load exceeding 200 messages/second > > - Capacity for storing 60 days worth of log messages, that's about 1 > > billion . > > - I currently have some 800 million mesages on disk, using about 5x140 > > gigs of disk. > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > - One VM running GrayLog2 server + a ES server with no local shards. > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > environment. > > > > I've applied the following changes to the bin/elasticsearch script: > > > > ulimit -n 60000 # fs.file-max = 131000 > > ulimit -l unlimited > > export ES\_HEAP\_SIZE=16g > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > and disabled the swap area to prevent swapping. > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > and all CPU resources. > > The first VM, is not significantly loaded. > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > rather change the ES configuration? > > > > Regards, Michael ---
**Author:** ![MichaelGlad](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MichaelGlad](https://discuss.elastic.co/u/MichaelGlad)\ **Post date:** [March 27, 2012, 6:42pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/4 "2012-03-27T18:42:57Z")
Hi Raud, inserts are fast enough to keep up with the ~200 incoming messages / sec without piling up in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 cores ) and very slow query response (~40-50 seconds). The storage is almost idle. I've now per your suggestions increased refresh interval to 10 secs and enabled compression. I'll have to consult the GL2 sources to see if it uses \_all. I've also increased the max Java heap size to 24 out of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU time starting up, search performance is now acceptable (10 secs) and the two ES data nodes only use some 10-20% CPU each. So it seems I'm back to a useful state of the world. Faster query time would be nice though. - Michael On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > Hi Michael, > > So what exactly happened? Inserts work slowly, or queries? Or both? > > I'm also using Elasticsearch for logging on VMs (although I'm not > using Graylog) and my "wall" was on inserts, and it was heavily > influenced by the storage speed. You kind of need that, since I can > bet your index size is much bigger than the total size of RAM. > > Some of the stuff you can do is to increase the number of shards if > you have no replicas configured (although this implies recreating the > index AFAIK). And we've also did here the following: > > - increasted the refresh interval to 3 > - compressed \_source > - disabled \_all (you need to check first if Graylog uses \_all for > searching :D) > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > activity. > > As I believe that the cure could be tuning ES or adding hardware > > resources, I'm posting to this list. > > > The indexing capacity I would like is: > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > filter > > - Message load exceeding 200 messages/second > > - Capacity for storing 60 days worth of log messages, that's about 1 > > billion . > > - I currently have some 800 million mesages on disk, using about 5x140 > > gigs of disk. > > > My setup is ( I'm using 64 bit Redhat Linux) > > > - One VM running GrayLog2 server + a ES server with no local shards. > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > environment. > > > I've applied the following changes to the bin/elasticsearch script: > > > ulimit -n 60000 # fs.file-max = 131000 > > ulimit -l unlimited > > export ES\_HEAP\_SIZE=16g > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > and disabled the swap area to prevent swapping. > > > Java on the two VMs containing shards, use about 12 gigs out memory > > and all CPU resources. > > The first VM, is not significantly loaded. > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > rather change the ES configuration? > > > Regards, Michael ---
**Author:** ![MichaelGlad](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MichaelGlad](https://discuss.elastic.co/u/MichaelGlad)\ **Post date:** [March 27, 2012, 6:51pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/5 "2012-03-27T18:51:03Z")
I've looked at the GL2 sources at GitHub, and it seems that the bulk API is used. - Michael On 27 Mar., 14:35, Michael Sick [michael.s...@serenesoftware.com](mailto:michael.s...@serenesoftware.com) wrote: > Also, if your inserts are slow are you using bulk inserts?[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/bulk.html) > > On Tue, Mar 27, 2012 at 5:52 AM, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com)wrote: > > > Hi Michael, > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > using Graylog) and my "wall" was on inserts, and it was heavily > > influenced by the storage speed. You kind of need that, since I can > > bet your index size is much bigger than the total size of RAM. > > > Some of the stuff you can do is to increase the number of shards if > > you have no replicas configured (although this implies recreating the > > index AFAIK). And we've also did here the following: > > > > - increasted the refresh interval to 3 > > - compressed \_source > > - disabled \_all (you need to check first if Graylog uses \_all for > > searching :D) > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > > activity. > > > As I believe that the cure could be tuning ES or adding hardware > > > resources, I'm posting to this list. > > > > The indexing capacity I would like is: > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > filter > > > - Message load exceeding 200 messages/second > > > - Capacity for storing 60 days worth of log messages, that's about 1 > > > billion . > > > - I currently have some 800 million mesages on disk, using about 5x140 > > > gigs of disk. > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > - One VM running GrayLog2 server + a ES server with no local shards. > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > > environment. > > > > I've applied the following changes to the bin/elasticsearch script: > > > > ulimit -n 60000 # fs.file-max = 131000 > > > ulimit -l unlimited > > > export ES\_HEAP\_SIZE=16g > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > and disabled the swap area to prevent swapping. > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > and all CPU resources. > > > The first VM, is not significantly loaded. > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > rather change the ES configuration? > > > > Regards, Michael ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 28, 2012, 5:46am UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/6 "2012-03-28T05:46:36Z")
Hi Michael, Nice to hear things got better 🙂 Here are some other stuff you might want to try: - increase the number of replicas (searches are done on replicas as well, so it might help) - at off-peak times, optimize your index. This should really help: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-optimize.html) - try to store shards on the server that has Graylog as well. On Mar 27, 9:42 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > messages / sec without piling up > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > cores ) and very > slow query response (~40-50 seconds). The storage is almost idle. > > I've now per your suggestions increased refresh interval to 10 secs > and enabled compression. > I'll have to consult the GL2 sources to see if it uses \_all. I've also > increased the max Java heap size to 24 out > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU > time starting up, search performance > is now acceptable (10 secs) and the two ES data nodes only use some > 10-20% CPU each. > > So it seems I'm back to a useful state of the world. Faster query time > would be nice though. > > - Michael > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > Hi Michael, > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > using Graylog) and my "wall" was on inserts, and it was heavily > > influenced by the storage speed. You kind of need that, since I can > > bet your index size is much bigger than the total size of RAM. > > > Some of the stuff you can do is to increase the number of shards if > > you have no replicas configured (although this implies recreating the > > index AFAIK). And we've also did here the following: > > > > - increasted the refresh interval to 3 > > - compressed \_source > > - disabled \_all (you need to check first if Graylog uses \_all for > > searching :D) > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > > activity. > > > As I believe that the cure could be tuning ES or adding hardware > > > resources, I'm posting to this list. > > > > The indexing capacity I would like is: > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > filter > > > - Message load exceeding 200 messages/second > > > - Capacity for storing 60 days worth of log messages, that's about 1 > > > billion . > > > - I currently have some 800 million mesages on disk, using about 5x140 > > > gigs of disk. > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > - One VM running GrayLog2 server + a ES server with no local shards. > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > > environment. > > > > I've applied the following changes to the bin/elasticsearch script: > > > > ulimit -n 60000 # fs.file-max = 131000 > > > ulimit -l unlimited > > > export ES\_HEAP\_SIZE=16g > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > and disabled the swap area to prevent swapping. > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > and all CPU resources. > > > The first VM, is not significantly loaded. > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > rather change the ES configuration? > > > > Regards, Michael ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 28, 2012, 10:55am UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/7 "2012-03-28T10:55:41Z")
What type of searches were being executed? It certainly might be that the ES process needed more memory to accommodate those (especially if sorting / facets are being used). On Tue, Mar 27, 2012 at 8:42 PM, MichaelGlad [glad@aysabtu.dk](mailto:glad@aysabtu.dk) wrote: > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > messages / sec without piling up > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > cores ) and very > slow query response (~40-50 seconds). The storage is almost idle. > > I've now per your suggestions increased refresh interval to 10 secs > and enabled compression. > I'll have to consult the GL2 sources to see if it uses \_all. I've also > increased the max Java heap size to 24 out > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU > time starting up, search performance > is now acceptable (10 secs) and the two ES data nodes only use some > 10-20% CPU each. > > So it seems I'm back to a useful state of the world. Faster query time > would be nice though. > > - Michael > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > Hi Michael, > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > using Graylog) and my "wall" was on inserts, and it was heavily > > influenced by the storage speed. You kind of need that, since I can > > bet your index size is much bigger than the total size of RAM. > > > > Some of the stuff you can do is to increase the number of shards if > > you have no replicas configured (although this implies recreating the > > index AFAIK). And we've also did here the following: > > > > - increasted the refresh interval to 3 > > - compressed \_source > > - disabled \_all (you need to check first if Graylog uses \_all for > > searching :D) > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > > activity. > > > As I believe that the cure could be tuning ES or adding hardware > > > resources, I'm posting to this list. > > > > > The indexing capacity I would like is: > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > filter > > > - Message load exceeding 200 messages/second > > > - Capacity for storing 60 days worth of log messages, that's about 1 > > > billion . > > > - I currently have some 800 million mesages on disk, using about 5x140 > > > gigs of disk. > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > - One VM running GrayLog2 server + a ES server with no local shards. > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > > environment. > > > > > I've applied the following changes to the bin/elasticsearch script: > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > ulimit -l unlimited > > > export ES\_HEAP\_SIZE=16g > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > and disabled the swap area to prevent swapping. > > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > and all CPU resources. > > > The first VM, is not significantly loaded. > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > rather change the ES configuration? > > > > > Regards, Michael ---
**Author:** ![MichaelGlad](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MichaelGlad](https://discuss.elastic.co/u/MichaelGlad)\ **Post date:** [March 28, 2012, 6:28pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/8 "2012-03-28T18:28:09Z")
Hi thank you for the suggestions, could it be the case the the extreme CPU usage I'm seeing a couple of hours following each start-up is ES doing optimization by its own initative? My shards are 140gigs each and as I understand that each shard is a lucene index this might be resource intensive. - Michael On 28 Mar., 07:46, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > Hi Michael, > > Nice to hear things got better 🙂 > > Here are some other stuff you might want to try: > > - increase the number of replicas (searches are done on replicas as > well, so it might help) > - at off-peak times, optimize your index. This should really help:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-optimi)... > - try to store shards on the server that has Graylog as well. > > On Mar 27, 9:42 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > > messages / sec without piling up > > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > > cores ) and very > > slow query response (~40-50 seconds). The storage is almost idle. > > > I've now per your suggestions increased refresh interval to 10 secs > > and enabled compression. > > I'll have to consult the GL2 sources to see if it uses \_all. I've also > > increased the max Java heap size to 24 out > > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU > > time starting up, search performance > > is now acceptable (10 secs) and the two ES data nodes only use some > > 10-20% CPU each. > > > So it seems I'm back to a useful state of the world. Faster query time > > would be nice though. > > > - Michael > > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > Hi Michael, > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > > using Graylog) and my "wall" was on inserts, and it was heavily > > > influenced by the storage speed. You kind of need that, since I can > > > bet your index size is much bigger than the total size of RAM. > > > > Some of the stuff you can do is to increase the number of shards if > > > you have no replicas configured (although this implies recreating the > > > index AFAIK). And we've also did here the following: > > > > > > - increasted the refresh interval to 3 > > > - compressed \_source > > > - disabled \_all (you need to check first if Graylog uses \_all for > > > searching :D) > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > > > activity. > > > > As I believe that the cure could be tuning ES or adding hardware > > > > resources, I'm posting to this list. > > > > > The indexing capacity I would like is: > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > > filter > > > > - Message load exceeding 200 messages/second > > > > - Capacity for storing 60 days worth of log messages, that's about 1 > > > > billion . > > > > - I currently have some 800 million mesages on disk, using about 5x140 > > > > gigs of disk. > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > - One VM running GrayLog2 server + a ES server with no local shards. > > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > > > environment. > > > > > I've applied the following changes to the bin/elasticsearch script: > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > > ulimit -l unlimited > > > > export ES\_HEAP\_SIZE=16g > > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > and disabled the swap area to prevent swapping. > > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > > and all CPU resources. > > > > The first VM, is not significantly loaded. > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > > rather change the ES configuration? > > > > > Regards, Michael ---
**Author:** ![MichaelGlad](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MichaelGlad](https://discuss.elastic.co/u/MichaelGlad)\ **Post date:** [March 28, 2012, 6:29pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/9 "2012-03-28T18:29:50Z")
It is relatively simple queries like gotit AND glad AND viagra to see if the filter has caught any viagra spammails bound for me. On 28 Mar., 12:55, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > What type of searches were being executed? It certainly might be that the > ES process needed more memory to accommodate those (especially if sorting / > facets are being used). > > On Tue, Mar 27, 2012 at 8:42 PM, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > > messages / sec without piling up > > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > > cores ) and very > > slow query response (~40-50 seconds). The storage is almost idle. > > > I've now per your suggestions increased refresh interval to 10 secs > > and enabled compression. > > I'll have to consult the GL2 sources to see if it uses \_all. I've also > > increased the max Java heap size to 24 out > > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU > > time starting up, search performance > > is now acceptable (10 secs) and the two ES data nodes only use some > > 10-20% CPU each. > > > So it seems I'm back to a useful state of the world. Faster query time > > would be nice though. > > > - Michael > > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > Hi Michael, > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > > using Graylog) and my "wall" was on inserts, and it was heavily > > > influenced by the storage speed. You kind of need that, since I can > > > bet your index size is much bigger than the total size of RAM. > > > > Some of the stuff you can do is to increase the number of shards if > > > you have no replicas configured (although this implies recreating the > > > index AFAIK). And we've also did here the following: > > > > > > - increasted the refresh interval to 3 > > > - compressed \_source > > > - disabled \_all (you need to check first if Graylog uses \_all for > > > searching :D) > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > > > activity. > > > > As I believe that the cure could be tuning ES or adding hardware > > > > resources, I'm posting to this list. > > > > > The indexing capacity I would like is: > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > > filter > > > > - Message load exceeding 200 messages/second > > > > - Capacity for storing 60 days worth of log messages, that's about 1 > > > > billion . > > > > - I currently have some 800 million mesages on disk, using about 5x140 > > > > gigs of disk. > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > - One VM running GrayLog2 server + a ES server with no local shards. > > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > > > environment. > > > > > I've applied the following changes to the bin/elasticsearch script: > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > > ulimit -l unlimited > > > > export ES\_HEAP\_SIZE=16g > > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > and disabled the swap area to prevent swapping. > > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > > and all CPU resources. > > > > The first VM, is not significantly loaded. > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > > rather change the ES configuration? > > > > > Regards, Michael ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 29, 2012, 6:25am UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/10 "2012-03-29T06:25:16Z")
AFAIK Graylog does a "match\_all" and sorts by date each time you open the interface, and then at a certain interval. I guess that's expected for any "logging" solution. How would that impact the memory requirements? I mean, I would expect to need more memory, but by how much? On Mar 28, 1:55 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > What type of searches were being executed? It certainly might be that the > ES process needed more memory to accommodate those (especially if sorting / > facets are being used). > > On Tue, Mar 27, 2012 at 8:42 PM, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > > messages / sec without piling up > > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > > cores ) and very > > slow query response (~40-50 seconds). The storage is almost idle. > > > I've now per your suggestions increased refresh interval to 10 secs > > and enabled compression. > > I'll have to consult the GL2 sources to see if it uses \_all. I've also > > increased the max Java heap size to 24 out > > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU > > time starting up, search performance > > is now acceptable (10 secs) and the two ES data nodes only use some > > 10-20% CPU each. > > > So it seems I'm back to a useful state of the world. Faster query time > > would be nice though. > > > - Michael > > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > Hi Michael, > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > > using Graylog) and my "wall" was on inserts, and it was heavily > > > influenced by the storage speed. You kind of need that, since I can > > > bet your index size is much bigger than the total size of RAM. > > > > Some of the stuff you can do is to increase the number of shards if > > > you have no replicas configured (although this implies recreating the > > > index AFAIK). And we've also did here the following: > > > > > > - increasted the refresh interval to 3 > > > - compressed \_source > > > - disabled \_all (you need to check first if Graylog uses \_all for > > > searching :D) > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > > > activity. > > > > As I believe that the cure could be tuning ES or adding hardware > > > > resources, I'm posting to this list. > > > > > The indexing capacity I would like is: > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > > filter > > > > - Message load exceeding 200 messages/second > > > > - Capacity for storing 60 days worth of log messages, that's about 1 > > > > billion . > > > > - I currently have some 800 million mesages on disk, using about 5x140 > > > > gigs of disk. > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > - One VM running GrayLog2 server + a ES server with no local shards. > > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > > > environment. > > > > > I've applied the following changes to the bin/elasticsearch script: > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > > ulimit -l unlimited > > > > export ES\_HEAP\_SIZE=16g > > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > and disabled the swap area to prevent swapping. > > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > > and all CPU resources. > > > > The first VM, is not significantly loaded. > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > > rather change the ES configuration? > > > > > Regards, Michael ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 29, 2012, 2:15pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/11 "2012-03-29T14:15:32Z")
Hard to say "how much", basically, when sorting, teh value for the field are loaded to memory to do it. You can see the current usage of it under the node stats API, under field cache. On Thu, Mar 29, 2012 at 8:25 AM, Radu Gheorghe [radu0gheorghe@gmail.com](mailto:radu0gheorghe@gmail.com)wrote: > AFAIK Graylog does a "match\_all" and sorts by date each time you open > the interface, and then at a certain interval. I guess that's expected > for any "logging" solution. > > How would that impact the memory requirements? I mean, I would expect > to need more memory, but by how much? > > On Mar 28, 1:55 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > What type of searches were being executed? It certainly might be that the > > ES process needed more memory to accommodate those (especially if > > sorting / > > facets are being used). > > > > On Tue, Mar 27, 2012 at 8:42 PM, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > > > messages / sec without piling up > > > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > > > cores ) and very > > > slow query response (~40-50 seconds). The storage is almost idle. > > > > > I've now per your suggestions increased refresh interval to 10 secs > > > and enabled compression. > > > I'll have to consult the GL2 sources to see if it uses \_all. I've also > > > increased the max Java heap size to 24 out > > > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU > > > time starting up, search performance > > > is now acceptable (10 secs) and the two ES data nodes only use some > > > 10-20% CPU each. > > > > > So it seems I'm back to a useful state of the world. Faster query time > > > would be nice though. > > > > > - Michael > > > > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > > > Hi Michael, > > > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > > > using Graylog) and my "wall" was on inserts, and it was heavily > > > > influenced by the storage speed. You kind of need that, since I can > > > > bet your index size is much bigger than the total size of RAM. > > > > > > Some of the stuff you can do is to increase the number of shards if > > > > you have no replicas configured (although this implies recreating the > > > > index AFAIK). And we've also did here the following: > > > > > > > > - increasted the refresh interval to 3 > > > > - compressed \_source > > > > - disabled \_all (you need to check first if Graylog uses \_all for > > > > searching :D) > > > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > > > seems I've hit a wall. Java is CPU bound and there's next to no > > > > > disk > > > > > activity. > > > > > As I believe that the cure could be tuning ES or adding hardware > > > > > resources, I'm posting to this list. > > > > > > > The indexing capacity I would like is: > > > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > > > filter > > > > > - Message load exceeding 200 messages/second > > > > > - Capacity for storing 60 days worth of log messages, that's about > > > > > 1 > > > > > billion . > > > > > - I currently have some 800 million mesages on disk, using about > > > > > 5x140 > > > > > gigs of disk. > > > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > > > - One VM running GrayLog2 server + a ES server with no local > > > > > shards. > > > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a > > > > > VMware > > > > > environment. > > > > > > > I've applied the following changes to the bin/elasticsearch script: > > > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > > > ulimit -l unlimited > > > > > export ES\_HEAP\_SIZE=16g > > > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > > > and disabled the swap area to prevent swapping. > > > > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > > > and all CPU resources. > > > > > The first VM, is not significantly loaded. > > > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > > > rather change the ES configuration? > > > > > > > Regards, Michael ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 30, 2012, 4:05pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/12 "2012-03-30T16:05:08Z")
I see. Thanks a lot! On Mar 29, 5:15 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Hard to say "how much", basically, when sorting, teh value for the field > are loaded to memory to do it. You can see the current usage of it under > the node stats API, under field cache. > > On Thu, Mar 29, 2012 at 8:25 AM, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com)wrote: > > > AFAIK Graylog does a "match\_all" and sorts by date each time you open > > the interface, and then at a certain interval. I guess that's expected > > for any "logging" solution. > > > How would that impact the memory requirements? I mean, I would expect > > to need more memory, but by how much? > > > On Mar 28, 1:55 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > What type of searches were being executed? It certainly might be that the > > > ES process needed more memory to accommodate those (especially if > > > sorting / > > > facets are being used). > > > > On Tue, Mar 27, 2012 at 8:42 PM, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > > > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > > > > messages / sec without piling up > > > > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > > > > cores ) and very > > > > slow query response (~40-50 seconds). The storage is almost idle. > > > > > I've now per your suggestions increased refresh interval to 10 secs > > > > and enabled compression. > > > > I'll have to consult the GL2 sources to see if it uses \_all. I've also > > > > increased the max Java heap size to 24 out > > > > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU > > > > time starting up, search performance > > > > is now acceptable (10 secs) and the two ES data nodes only use some > > > > 10-20% CPU each. > > > > > So it seems I'm back to a useful state of the world. Faster query time > > > > would be nice though. > > > > > - Michael > > > > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > > > > > Hi Michael, > > > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > > > > using Graylog) and my "wall" was on inserts, and it was heavily > > > > > influenced by the storage speed. You kind of need that, since I can > > > > > bet your index size is much bigger than the total size of RAM. > > > > > > Some of the stuff you can do is to increase the number of shards if > > > > > you have no replicas configured (although this implies recreating the > > > > > index AFAIK). And we've also did here the following: > > > > > > > > > > - increasted the refresh interval to 3 > > > > > - compressed \_source > > > > > - disabled \_all (you need to check first if Graylog uses \_all for > > > > > searching :D) > > > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > > > > seems I've hit a wall. Java is CPU bound and there's next to no > > > > > > disk > > > > > > activity. > > > > > > As I believe that the cure could be tuning ES or adding hardware > > > > > > resources, I'm posting to this list. > > > > > > > The indexing capacity I would like is: > > > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > > > > filter > > > > > > - Message load exceeding 200 messages/second > > > > > > - Capacity for storing 60 days worth of log messages, that's about > > > > > > 1 > > > > > > billion . > > > > > > - I currently have some 800 million mesages on disk, using about > > > > > > 5x140 > > > > > > gigs of disk. > > > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > > > - One VM running GrayLog2 server + a ES server with no local > > > > > > shards. > > > > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a > > > > > > VMware > > > > > > environment. > > > > > > > I've applied the following changes to the bin/elasticsearch script: > > > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > > > > ulimit -l unlimited > > > > > > export ES\_HEAP\_SIZE=16g > > > > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > > > and disabled the swap area to prevent swapping. > > > > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > > > > and all CPU resources. > > > > > > The first VM, is not significantly loaded. > > > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > > > > rather change the ES configuration? > > > > > > > Regards, Michael ---
**Author:** ![MichaelGlad](https://avatars.discourse-cdn.com/v4/letter/m/ec9cab/32.png) [@MichaelGlad](https://discuss.elastic.co/u/MichaelGlad)\ **Post date:** [March 30, 2012, 9:03pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/13 "2012-03-30T21:03:00Z")
Thank your for all your kind help. I've learned a lot about ES. I've located a reason why my searches are slow -- GL2 makes the following Ruby call each time it returns to the main screen - eg. after doing a search: search("\*", :size =\> 0).total With close to a billion messages, that take some time 🙂 I've now hacked the source, disabling the check, and now searches are much faster. I'll contact the GL2 author and suggest adding a proper option to disable counting the number of stored log entries all the time. - Michael On 28 Mar., 20:29, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > It is relatively simple queries like > > gotit AND glad AND viagra > > to see if the filter has caught any viagra spammails bound for me. > > On 28 Mar., 12:55, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > What type of searches were being executed? It certainly might be that the > > ES process needed more memory to accommodate those (especially if sorting / > > facets are being used). > > > On Tue, Mar 27, 2012 at 8:42 PM, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > > > messages / sec without piling up > > > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > > > cores ) and very > > > slow query response (~40-50 seconds). The storage is almost idle. > > > > I've now per your suggestions increased refresh interval to 10 secs > > > and enabled compression. > > > I'll have to consult the GL2 sources to see if it uses \_all. I've also > > > increased the max Java heap size to 24 out > > > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of CPU > > > time starting up, search performance > > > is now acceptable (10 secs) and the two ES data nodes only use some > > > 10-20% CPU each. > > > > So it seems I'm back to a useful state of the world. Faster query time > > > would be nice though. > > > > - Michael > > > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > > > Hi Michael, > > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > > > using Graylog) and my "wall" was on inserts, and it was heavily > > > > influenced by the storage speed. You kind of need that, since I can > > > > bet your index size is much bigger than the total size of RAM. > > > > > Some of the stuff you can do is to increase the number of shards if > > > > you have no replicas configured (although this implies recreating the > > > > index AFAIK). And we've also did here the following: > > > > > > > > - increasted the refresh interval to 3 > > > > - compressed \_source > > > > - disabled \_all (you need to check first if Graylog uses \_all for > > > > searching :D) > > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. It > > > > > seems I've hit a wall. Java is CPU bound and there's next to no disk > > > > > activity. > > > > > As I believe that the cure could be tuning ES or adding hardware > > > > > resources, I'm posting to this list. > > > > > > The indexing capacity I would like is: > > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > > > filter > > > > > - Message load exceeding 200 messages/second > > > > > - Capacity for storing 60 days worth of log messages, that's about 1 > > > > > billion . > > > > > - I currently have some 800 million mesages on disk, using about 5x140 > > > > > gigs of disk. > > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > > - One VM running GrayLog2 server + a ES server with no local shards. > > > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on each. > > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a VMware > > > > > environment. > > > > > > I've applied the following changes to the bin/elasticsearch script: > > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > > > ulimit -l unlimited > > > > > export ES\_HEAP\_SIZE=16g > > > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > > and disabled the swap area to prevent swapping. > > > > > > Java on the two VMs containing shards, use about 12 gigs out memory > > > > > and all CPU resources. > > > > > The first VM, is not significantly loaded. > > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should I > > > > > rather change the ES configuration? > > > > > > Regards, Michael ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 31, 2012, 9:09pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/14 "2012-03-31T21:09:10Z")
A faster way to get the total document is using the count API (or search with search\_type set to count) with match\_all query, yes an even faster way is possibly to use index stats API with docs: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. use the num docs on primary stats (thats assuming no filtering based on type is done). On Sat, Mar 31, 2012 at 12:03 AM, MichaelGlad [glad@aysabtu.dk](mailto:glad@aysabtu.dk) wrote: > Thank your for all your kind help. I've learned a lot about ES. > I've located a reason why my searches are slow -- GL2 makes the > following Ruby call each time it returns > to the main screen - eg. after doing a search: > > search("\*", :size =\> 0).total > > With close to a billion messages, that take some time 🙂 > > I've now hacked the source, disabling the check, and now searches are > much faster. I'll contact the GL2 > author and suggest adding a proper option to disable counting the > number of stored log entries > all the time. > > - Michael > > On 28 Mar., 20:29, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > It is relatively simple queries like > > > > gotit AND glad AND viagra > > > > to see if the filter has caught any viagra spammails bound for me. > > > > On 28 Mar., 12:55, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > What type of searches were being executed? It certainly might be that > > > the > > > ES process needed more memory to accommodate those (especially if > > > sorting / > > > facets are being used). > > > > > On Tue, Mar 27, 2012 at 8:42 PM, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > > > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > > > > messages / sec without piling up > > > > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > > > > cores ) and very > > > > slow query response (~40-50 seconds). The storage is almost idle. > > > > > > I've now per your suggestions increased refresh interval to 10 secs > > > > and enabled compression. > > > > I'll have to consult the GL2 sources to see if it uses \_all. I've > > > > also > > > > increased the max Java heap size to 24 out > > > > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of > > > > CPU > > > > time starting up, search performance > > > > is now acceptable (10 secs) and the two ES data nodes only use some > > > > 10-20% CPU each. > > > > > > So it seems I'm back to a useful state of the world. Faster query > > > > time > > > > would be nice though. > > > > > > - Michael > > > > > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > > > > > Hi Michael, > > > > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > > > > using Graylog) and my "wall" was on inserts, and it was heavily > > > > > influenced by the storage speed. You kind of need that, since I can > > > > > bet your index size is much bigger than the total size of RAM. > > > > > > > Some of the stuff you can do is to increase the number of shards if > > > > > you have no replicas configured (although this implies recreating > > > > > the > > > > > index AFAIK). And we've also did here the following: > > > > > > > > > > - increasted the refresh interval to 3 > > > > > - compressed \_source > > > > > - disabled \_all (you need to check first if Graylog uses \_all for > > > > > searching :D) > > > > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. > > > > > > It > > > > > > seems I've hit a wall. Java is CPU bound and there's next to no > > > > > > disk > > > > > > activity. > > > > > > As I believe that the cure could be tuning ES or adding hardware > > > > > > resources, I'm posting to this list. > > > > > > > > The indexing capacity I would like is: > > > > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > > > > filter > > > > > > - Message load exceeding 200 messages/second > > > > > > - Capacity for storing 60 days worth of log messages, that's > > > > > > about 1 > > > > > > billion . > > > > > > - I currently have some 800 million mesages on disk, using about > > > > > > 5x140 > > > > > > gigs of disk. > > > > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > > > > - One VM running GrayLog2 server + a ES server with no local > > > > > > shards. > > > > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on > > > > > > each. > > > > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a > > > > > > VMware > > > > > > environment. > > > > > > > > I've applied the following changes to the bin/elasticsearch > > > > > > script: > > > > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > > > > ulimit -l unlimited > > > > > > export ES\_HEAP\_SIZE=16g > > > > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > > > > and disabled the swap area to prevent swapping. > > > > > > > > Java on the two VMs containing shards, use about 12 gigs out > > > > > > memory > > > > > > and all CPU resources. > > > > > > The first VM, is not significantly loaded. > > > > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should > > > > > > I > > > > > > rather change the ES configuration? > > > > > > > > Regards, Michael ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [April 1, 2012, 9:12pm UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/15 "2012-04-01T21:12:27Z")
We have a slightly different approach. Please let me know if you see something wrong with it. Currently, we are using the "hits" return value when we are doing any search. So when you go to the main page, we have to do a "match\_all" query sorted by date to give the last logs. The "hits" to that is the total number of logs. Then, when we're doing a specific search, we still get a number of hits and we show that number. Even though the interface itself will only show a limited number of results. On 1 apr., 00:09, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > A faster way to get the total document is using the count API (or search > with search\_type set to count) with match\_all query, yes an even faster way > is possibly to use index stats API with docs:[Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-stats).... > use the num docs on primary stats (thats assuming no filtering based > on > type is done). > > On Sat, Mar 31, 2012 at 12:03 AM, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > Thank your for all your kind help. I've learned a lot about ES. > > I've located a reason why my searches are slow -- GL2 makes the > > following Ruby call each time it returns > > to the main screen - eg. after doing a search: > > > search("\*", :size =\> 0).total > > > With close to a billion messages, that take some time 🙂 > > > I've now hacked the source, disabling the check, and now searches are > > much faster. I'll contact the GL2 > > author and suggest adding a proper option to disable counting the > > number of stored log entries > > all the time. > > > - Michael > > > On 28 Mar., 20:29, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > It is relatively simple queries like > > > > gotit AND glad AND viagra > > > > to see if the filter has caught any viagra spammails bound for me. > > > > On 28 Mar., 12:55, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > What type of searches were being executed? It certainly might be that > > > > the > > > > ES process needed more memory to accommodate those (especially if > > > > sorting / > > > > facets are being used). > > > > > On Tue, Mar 27, 2012 at 8:42 PM, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > > > > > Hi Raud, inserts are fast enough to keep up with the ~200 incoming > > > > > messages / sec without piling up > > > > > in the Graylog2 queue. The problem are the serious CPU usage ( 2 x 3 > > > > > cores ) and very > > > > > slow query response (~40-50 seconds). The storage is almost idle. > > > > > > I've now per your suggestions increased refresh interval to 10 secs > > > > > and enabled compression. > > > > > I'll have to consult the GL2 sources to see if it uses \_all. I've > > > > > also > > > > > increased the max Java heap size to 24 out > > > > > of the 32G RAM and restarted ES. After having spent 2 x 7 hours of > > > > > CPU > > > > > time starting up, search performance > > > > > is now acceptable (10 secs) and the two ES data nodes only use some > > > > > 10-20% CPU each. > > > > > > So it seems I'm back to a useful state of the world. Faster query > > > > > time > > > > > would be nice though. > > > > > > - Michael > > > > > > On 27 Mar., 11:52, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > > > > > > > Hi Michael, > > > > > > > So what exactly happened? Inserts work slowly, or queries? Or both? > > > > > > > I'm also using Elasticsearch for logging on VMs (although I'm not > > > > > > using Graylog) and my "wall" was on inserts, and it was heavily > > > > > > influenced by the storage speed. You kind of need that, since I can > > > > > > bet your index size is much bigger than the total size of RAM. > > > > > > > Some of the stuff you can do is to increase the number of shards if > > > > > > you have no replicas configured (although this implies recreating > > > > > > the > > > > > > index AFAIK). And we've also did here the following: > > > > > > > > > > > > - increasted the refresh interval to 3 > > > > > > - compressed \_source > > > > > > - disabled \_all (you need to check first if Graylog uses \_all for > > > > > > searching :D) > > > > > > > On Mar 26, 11:40 pm, MichaelGlad [g...@aysabtu.dk](mailto:g...@aysabtu.dk) wrote: > > > > > > > > I'm having troubles with a my GrayLog2 + ES 0.19.0 installation. > > > > > > > It > > > > > > > seems I've hit a wall. Java is CPU bound and there's next to no > > > > > > > disk > > > > > > > activity. > > > > > > > As I believe that the cure could be tuning ES or adding hardware > > > > > > > resources, I'm posting to this list. > > > > > > > > The indexing capacity I would like is: > > > > > > > > - Documents: long syslog mesages (160 chars avg) from a busy mail > > > > > > > filter > > > > > > > - Message load exceeding 200 messages/second > > > > > > > - Capacity for storing 60 days worth of log messages, that's > > > > > > > about 1 > > > > > > > billion . > > > > > > > - I currently have some 800 million mesages on disk, using about > > > > > > > 5x140 > > > > > > > gigs of disk. > > > > > > > > My setup is ( I'm using 64 bit Redhat Linux) > > > > > > > > - One VM running GrayLog2 server + a ES server with no local > > > > > > > shards. > > > > > > > - Two VMs with a total of 5 shards distributed with 3 / 2 on > > > > > > > each. > > > > > > > > The two VMs with shards have 32 gigs of memory and 4 cores in a > > > > > > > VMware > > > > > > > environment. > > > > > > > > I've applied the following changes to the bin/elasticsearch > > > > > > > script: > > > > > > > > ulimit -n 60000 # fs.file-max = 131000 > > > > > > > ulimit -l unlimited > > > > > > > export ES\_HEAP\_SIZE=16g > > > > > > > export JAVA\_HOME=/usr/java/latest # SUN JRE 1.6.31 > > > > > > > export JAVA\_OPTS="-Xloggc:/tmp/gc" > > > > > > > > and disabled the swap area to prevent swapping. > > > > > > > > Java on the two VMs containing shards, use about 12 gigs out > > > > > > > memory > > > > > > > and all CPU resources. > > > > > > > The first VM, is not significantly loaded. > > > > > > > > What could a solution be -- "Kiwi" (kill it with Iron) or should > > > > > > > I > > > > > > > rather change the ES configuration? > > > > > > > > Regards, Michael ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:34am UTC](https://discuss.elastic.co/t/problems-with-graylog2-es-setup-long/7134/16 "2017-07-06T03:34:02Z")