Here is the log of Elasticsearch booting up :
[2013-03-06 14:29:46,484][INFO ][node ] [Bench, Morris]
{0.20.4}[7148]: initializing ...
[2013-03-06 14:29:46,511][INFO ][plugins ] [Bench, Morris]
loaded , sites
[2013-03-06 14:29:50,383][INFO ][node ] [Bench, Morris]
{0.20.4}[7148]: initialized
[2013-03-06 14:29:50,383][INFO ][node ] [Bench, Morris]
{0.20.4}[7148]: starting ...
[2013-03-06 14:29:50,545][INFO ][transport ] [Bench, Morris]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/xxx.xxx.xxx.150:9300]}
[2013-03-06 14:29:53,629][INFO ][cluster.service ] [Bench, Morris]
new_master [Bench,
Morris][-ugh0MlRdymGhCE8gYl-w][inet[/xxx.xxx.xxx.150:9300]], reason:
zen-disco-join (elected_as_master)
[2013-03-06 14:29:53,636][INFO ][discovery ] [Bench, Morris]
graylog2/-ugh0MlRdymGhCE8gYl-w
[2013-03-06 14:29:53,707][INFO ][http ] [Bench, Morris]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/xxx.xxx.xxx.150:9200]}
[2013-03-06 14:29:53,707][INFO ][node ] [Bench, Morris]
{0.20.4}[7148]: started
[2013-03-06 14:30:00,244][INFO ][gateway ] [Bench, Morris]
recovered [2] indices into cluster_state
[2013-03-06 14:30:00,267][INFO ][cluster.service ] [Bench, Morris]
added
{[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true,
data=false, master=false},}, reason: zen-disco-receive(join from
node[[graylog2-server][HVCBPKuDSx-DelCoMrC2iw][inet[/xxx.xxx.xxx.150:9350]]{client=true,
data=false, master=false}])
[2013-03-06 14:30:06,454][WARN ][index.shard.service ] [Bench, Morris]
[graylog2_0][1] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][1] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/1/index/_nwa.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,459][WARN ][index.merge.scheduler ] [Bench, Morris]
[graylog2_0][1] failed to merge
java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/1/index/_nwa.nrm
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.SegmentReader.openNorms(SegmentReader.java:639)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:121)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4238)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
[2013-03-06 14:30:06,499][WARN ][index.shard.service ] [Bench, Morris]
[graylog2_0][2] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][2] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/2/index/_o6g.fdx
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:133)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,792][WARN ][index.shard.service ] [Bench, Morris]
[graylog2_recent][3] Failed to perform scheduled engine refresh
org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_recent][3] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.shard.service.InternalIndexShard.refresh(InternalIndexShard.java:403)
at
org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher$1.run(InternalIndexShard.java:733)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/3/index/_m9b.frq
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at
org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)
at
org.apache.lucene.index.FormatPostingsDocsWriter.(FormatPostingsDocsWriter.java:47)
at
org.apache.lucene.index.FormatPostingsTermsWriter.(FormatPostingsTermsWriter.java:33)
at
org.apache.lucene.index.FormatPostingsFieldsWriter.(FormatPostingsFieldsWriter.java:51)
at
org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:85)
at org.apache.lucene.index.TermsHash.flush(TermsHash.java:113)
at org.apache.lucene.index.DocInverter.flush(DocInverter.java:70)
at
org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:60)
at org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:581)
at org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3587)
at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3552)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:450)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 5 more
[2013-03-06 14:30:06,917][WARN ][indices.cluster ] [Bench, Morris]
[graylog2_0][3] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[graylog2_0][3] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:228)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.elasticsearch.index.engine.FlushFailedEngineException:
[graylog2_0][3] Flush failed
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:909)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryFinalization(InternalIndexShard.java:564)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
... 3 more
Caused by: org.elasticsearch.index.engine.RefreshFailedEngineException:
[graylog2_0][3] Refresh failed
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:788)
at
org.elasticsearch.index.engine.robin.RobinEngine.refreshVersioningTable(RobinEngine.java:967)
at
org.elasticsearch.index.engine.robin.RobinEngine.flush(RobinEngine.java:892)
... 6 more
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/nyr.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:71)
at
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:98)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:92)
at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:79)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:537)
at org.apache.lucene.index.FieldsReader.(FieldsReader.java:131)
at
org.apache.lucene.index.SegmentCoreReaders.openDocStores(SegmentCoreReaders.java:234)
at org.apache.lucene.index.SegmentReader.get(SegmentReader.java:118)
at org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:696)
at
org.apache.lucene.index.IndexWriter$ReaderPool.getReadOnlyClone(IndexWriter.java:654)
at org.apache.lucene.index.DirectoryReader.(DirectoryReader.java:142)
at
org.apache.lucene.index.ReadOnlyDirectoryReader.(ReadOnlyDirectoryReader.java:36)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:451)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:399)
at
org.apache.lucene.index.DirectoryReader.doOpenFromWriter(DirectoryReader.java:413)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:432)
at
org.apache.lucene.index.DirectoryReader.doOpenIfChanged(DirectoryReader.java:375)
at org.apache.lucene.index.IndexReader.openIfChanged(IndexReader.java:508)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:109)
at
org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:57)
at
org.apache.lucene.search.ReferenceManager.maybeRefresh(ReferenceManager.java:137)
at
org.elasticsearch.index.engine.robin.RobinEngine.refresh(RobinEngine.java:769)
... 8 more
[2013-03-06 14:30:07,003][WARN ][cluster.action.shard ] [Bench, Morris]
sending failed shard for [graylog2_0][3], node[-ugh0MlRdymGhCE8gYl-w],
[P], s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2_0][3] failed recovery];
nested: FlushFailedEngineException[[graylog2_0][3] Flush failed]; nested:
RefreshFailedEngineException[[graylog2_0][3] Refresh failed]; nested:
FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/nyr.fdt
(Too many open files)]; ]]
[2013-03-06 14:30:07,003][WARN ][cluster.action.shard ] [Bench, Morris]
received shard failed for [graylog2_0][3], node[-ugh0MlRdymGhCE8gYl-w],
[P], s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[graylog2_0][3] failed recovery];
nested: FlushFailedEngineException[[graylog2_0][3] Flush failed]; nested:
RefreshFailedEngineException[[graylog2_0][3] Refresh failed]; nested:
FileNotFoundException[/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_0/3/index/_nyr.fdt
(Too many open files)]; ]]
On Tuesday, March 5, 2013 7:45:46 PM UTC+1, Jörg Prante wrote:
Your index is corrupted because the max number of open files has been
exceeded.
Check ulimit -Sn and increase the setting, e.g. for user 'elasticsearch'
File /etc/security/limits.conf
elasticsearch - nofile 65535
elasticsearch - memlock unlimited
Jörg
Am 05.03.13 18:27, schrieb Kranan:
I am using Graylog2 w/ Elasticsearch for a year now and it worked
perfectly... since last month.
After we changed the servers architecture (but not the server itself),
I am experiencing repetitive
crash of the Elasticsearch service.
So I upgraded the Graylog2 to its last version (0.10.0, and of course
Elasticsearch from the 0.19.10 to the 0.20.4 as required), but it
still crashes every 5-6 hours, sometimes less.
And after each crash, Elasticsearch index is gone and starting all
over again.
Here is an extract of the /var/log/elasticsearch/graylog2.log after
the crash :
org.elasticsearch.index.engine.IndexFailedEngineException:
[graylog2_recent][0] Index failed for
[message#3b25a760-851f-11e2-ab8c-0024e82dd6ba]
at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:494)
at
org.elasticsearch.index.shard.service.InternalIndexShard.index(InternalIndexShard.java:330)
at
org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnPrimary(TransportShardBulkAction.java:158)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction.performOnPrimary(TransportShardReplicationOperationAction.java:531)
at
org.elasticsearch.action.support.replication.TransportShardReplicationOperationAction$AsyncShardOperationAction$1.run(TransportShardReplicationOperationAction.java:429)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
Caused by: java.io.FileNotFoundException:
/scdata.local/syslog/elasticsearch/graylog2/nodes/0/indices/graylog2_recent/0/index/_16bi.fdt
(Too many open files)
at java.io.RandomAccessFile.open(Native Method)
at
java.io.RandomAccessFile.(RandomAccessFile.java:233)
at
org.apache.lucene.store.FSDirectory$FSIndexOutput.(FSDirectory.java:441)
at
org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:306)
at
org.apache.lucene.store.XNIOFSDirectory.createOutput(XNIOFSDirectory.java:48)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:487)
at
org.elasticsearch.index.store.Store$StoreDirectory.createOutput(Store.java:459)
at
org.apache.lucene.index.FieldsWriter.<init>(FieldsWriter.java:83)
at
org.apache.lucene.index.StoredFieldsWriter.initFieldsWriter(StoredFieldsWriter.java:64)
at
org.apache.lucene.index.StoredFieldsWriter.finishDocument(StoredFieldsWriter.java:107)
at
org.apache.lucene.index.StoredFieldsWriter$PerDoc.finish(StoredFieldsWriter.java:151)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.writeDocument(DocumentsWriter.java:1404)
at
org.apache.lucene.index.DocumentsWriter$WaitQueue.add(DocumentsWriter.java:1424)
at
org.apache.lucene.index.DocumentsWriter.finishDocument(DocumentsWriter.java:1043)
at
org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:772)
at
org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:2060)
at
org.elasticsearch.index.engine.robin.RobinEngine.innerIndex(RobinEngine.java:577)
at
org.elasticsearch.index.engine.robin.RobinEngine.index(RobinEngine.java:489)
... 7 more
/etc/init.d/elasticsearch memory conf :
ES_MIN_MEM=256m
ES_MAX_MEM=4g
Graylog2 and elasticsearch is hosted on the same server, no shard, no
replica.
Hope someone can help 
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
