Hey guys,
I have an IT test for my MR job that does the following:
- Precreates index with mapping
- Creates test parquet files
- Executes MR job that reads parquet files and using EsOutputFromat stores them into ES.
- checks if data is in the index
In the test I was creating a node with local set to false and http set to true. And this test was working ok until elasticsearch version 1.7.3. es-hadoop version is 2.1.1
But now Im trying to migrate it to 2.0.0 and I'm having trouble. My MR job is not able to find any nodes and I can see from the console that published address is always 127.0.0.1 where in previous ES version it was someting else. Not sure where the problem lies and I would appreciate any help.
So below the code how I'm creating the node and returning a client in my test:
private static Client buildHttpClient(String dataDirectory, String clusterName) {
Settings.Builder settings = Settings.settingsBuilder()
.put("http.enabled", "true")
.put("path.home", "target/es")
.put("path.data", dataDirectory);
node = nodeBuilder().local(false).clusterName(clusterName).settings(settings).node();
return node.client();
}
This is my method to get the es.node (host / port) to provide in MR job
private String getEsNode() {
final NodesInfoResponse nodeInfos =
client.admin().cluster().prepareNodesInfo().setSettings(true).setHttp(true).get();
NodeInfo nodeInfo = nodeInfos.getNodes()[0];
TransportAddress publishAddress = nodeInfo.getHttp().getAddress().publishAddress();
InetSocketAddress address = ((InetSocketTransportAddress) publishAddress).address();
return address.getHostName() + ":" + address.getPort();
}
Below console output on 1.7.3 version when test was passing:
15/10/30 13:23:13 INFO elasticsearch.node: [Mechamage] version[1.7.3], pid[29294], build[05d4530/2015-10-15T09:14:17Z]
15/10/30 13:23:13 INFO elasticsearch.node: [Mechamage] initializing ...
15/10/30 13:23:13 INFO elasticsearch.plugins: [Mechamage] loaded [], sites []
15/10/30 13:23:13 INFO elasticsearch.env: [Mechamage] using [1] data paths, mounts [[/ (/dev/disk1)]], net usable_space [420.6gb], net total_space [464.7gb], types [hfs]
15/10/30 13:23:14 WARN elasticsearch.bootstrap: JNA not found. native methods will be disabled.
15/10/30 13:23:15 INFO elasticsearch.node: [Mechamage] initialized
15/10/30 13:23:15 INFO elasticsearch.node: [Mechamage] starting ...
15/10/30 13:23:15 INFO elasticsearch.transport: [Mechamage] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.0.143:9300]}
15/10/30 13:23:15 INFO elasticsearch.discovery: [Mechamage] EsTest/7-HKR-WURp6bB2SCmEPn0A
15/10/30 13:23:19 INFO cluster.service: [Mechamage] new_master [Mechamage][7-HKR-WURp6bB2SCmEPn0A][MPLKD186017-976][inet[/192.168.0.143:9300]]{local=false}, reason: zen-disco-join (elected_as_master)
15/10/30 13:23:19 INFO elasticsearch.http: [Mechamage] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.0.143:9200]}
15/10/30 13:23:19 INFO elasticsearch.node: [Mechamage] started
15/10/30 13:23:19 INFO elasticsearch.gateway: [Mechamage] recovered [0] indices into cluster_state
15/10/30 13:23:19 INFO cluster.metadata: [Mechamage] [user-2-1] creating index, cause [api], templates [], shards [5]/[1], mappings []
15/10/30 13:23:19 INFO cluster.metadata: [Mechamage] [user-2-1] create_mapping [attributes]
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
And here console output on version 2.0.0 when test is failing:
15/10/30 13:15:00 INFO elasticsearch.node: [Solara] initialized
15/10/30 13:15:00 INFO elasticsearch.node: [Solara] starting ...
15/10/30 13:15:00 INFO elasticsearch.transport: [Solara] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}, {[fe80::1]:9300}, {[::1]:9300}
15/10/30 13:15:00 INFO elasticsearch.discovery: [Solara] EsTest/cekAX7ZSQHeETODF3xkmJg
15/10/30 13:15:03 INFO cluster.service: [Solara] new_master {Solara}{cekAX7ZSQHeETODF3xkmJg}{127.0.0.1}{127.0.0.1:9300}{local=false}, reason: zen-disco-join(elected_as_master, [0] joins received)
15/10/30 13:15:03 INFO elasticsearch.http: [Solara] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}, {[fe80::1]:9200}, {[::1]:9200}
15/10/30 13:15:03 INFO elasticsearch.node: [Solara] started
15/10/30 13:15:03 INFO elasticsearch.gateway: [Solara] recovered [0] indices into cluster_state
15/10/30 13:15:04 INFO cluster.metadata: [Solara] [user-2-1] creating index, cause [api], templates [], shards [5]/[1], mappings []
15/10/30 13:15:04 INFO cluster.metadata: [Solara] [user-2-1] create_mapping [attributes]