Enabling Multicast and TCP/IP node discovery methods froze my instances

[juan-palacios]

I started two different instances. Configuration was loaded programmatically:

hazelcast.network.multicast=true
hazelcast.network.tcpip=true
hazelcast.network.tcpip.members=localhost:5701,localhost:5702
hazelcast.port=5701

According to the generated thread dumps Hazelcast was trying to complete the Node.join operation, but it kept recursively going from a MulticastJoiner to a TcpIpJoiner and never actually completing the operation. As time went by the stack kept growing. My instances were frozen for over 30minutes until I killed one of them, and the other one was able to complete the join.

[juan-palacios]

Here's a simple project I built to recreate the issue: http://cl.ly/0f1a31122O40

Just unzip and run the deadlock.sh script. It should create two Java processes which never complete.

[juan-palacios]

This is a gist with thread dumps collected from our application:
https://gist.github.com/juan-palacios/60eb7de627cb6d24c4dd

[pveentjer]

It isn't good to have a frozen instance, but you can only have one join configuration active. So either multicast or tcp/ip. You can't have both of them.

I'll have a look at the example and see if I can reproduce it.

[pveentjer]

@mdogan @ali strange problem:

When I run with pure multicast -> no problem. Cluster formed
When I run with pure tcpip -> no problem. Cluster formed.
When I run with both enabled (we don't want this anyway), I see:

Multicast couldn't find cluster. Defaulting to TCP/IP cluster discovery

I improved the log message. But it is strange that the when both are enabled, multicast doesn't work and therefore defaults to TCP. And in TCP it gets stuck, so that also doesn't work.

So there are a few things we need to think about:

• we have no validation on valid join options -> we need to add this. Only one should allowed to be active.
• I find the defaulting mechanism of Multicast to TCP very unexpected.
• It is strange that when both are enabled enabled, that both don't work. When only one of them is enabled (doesn't matter which one) it does work.

[qerub]

[…] but you can only have one join configuration active. So either multicast or tcp/ip. You can't have both of them.

You might want to document that clearly on http://docs.hazelcast.org/docs/latest/manual/html-single/hazelcast-documentation.html#hazelcast-clusters too.