RvR Services should be provided on VIP instead of VR IP

[richardlawley]

ISSUE TYPE

• Bug Report

COMPONENT NAME

Redundant VRs

CLOUDSTACK VERSION

4.11.2

CONFIGURATION

Advanced Networking, possibly also VPCs

OS / ENVIRONMENT

N/A

SUMMARY

Redundant VRs provide their services on their own internal IP rather than the Virtual IP. This leads to a number of problems, mostly triggered around VRs failing over after a VM has booted:

• DHCP server is run on VR IP, which means VM cannot renew its IP. Once the renew + expire times run out, the VM attempts to discover a new DHCP server (this causes additional problems with secondary IPs being removed).
• Userdata services only listen on VR IP, so accessing this data requires checking the DHCP lease info to find the VR's IP. If the VR has failed over, userdata cannot be accessed.
• Hairpin static NAT connections appear to come from the VR's IP, and since that can change it may confuse end clients

STEPS TO REPRODUCE

Build a redundant advanced network with guest range 10.1.1.0/24, deploy a Linux VM.

• Observe DHCP server
• Check userdata (curl http://VRIP/latest/instance-id)
• Add a static NAT rule

EXPECTED RESULTS

• DHCP server should be 10.1.1.1
• Userdata should be accessible on http://10.1.1.1
• POSTROUTING rules on nat table should use 10.1.1.1 as the source

ACTUAL RESULTS

• DHCP server is the unique VR IP from the internal range
• Userdata is only accessible at VR's IP, not 10.1.1.1
• POSTROUTING rules on nat table (hairpin nat) use the VR IP

[richardlawley]

To elaborate on the DHCP problem, this is related to #3351. The VMs we've found most affected by this are Linux machines with a secondary internal IP. Their primary IP comes from DHCP, but the secondaries are all statically configured.

If the DHCP lease is renewed, no problem occurs. However, if a renew fails and a re-discover takes place, the statically configured secondary IPs are all removed along with the DHCP primary, and not restored when the new lease is acquired.

There are two causes of this:

1. DHCP server not being available on the IP it was (e.g. VR failover or rebuild)
2. DHCP server not being aware of the previous lease.

Problem 2 was addressed in #3351, but without running the DHCP server on the VIP, problem 1 still exists.

We are working around this until this issue is resolved by going back to infinite leases in our custom ISO.

[yadvr]

@ustcweizhou can you share the diff/patch or send a PR in case you've solve this issue in your tree?

[ustcweizhou]

@rhtyd
I have checked all my code. it seems very difficult to port our changes to 4.11 or 4.13 as there are big difference between our branch and 4.11/4.13.
Anyone who is interested in the fixes please feel free to pick it up.

[yadvr]

Okay, is your tree public/accessible somewhere @ustcweizhou ?

[ustcweizhou]

@rhtyd
I spent spare time on these issues, they seem to be fixed now.
I will test it and create PR next week.

[yadvr]

Thanks @ustcweizhou looking forward to your PR

[yadvr]

Ping @ustcweizhou any update? Thanks.
Related: #3179

[yadvr]

Thanks @ustcweizhou I'll help review and test

[yadvr]

This is a known issue and it's likely that we are unable to fix issue and the PR seems to be adding regressions. I've moved it to 4.13.1.0 milestone for now, if the regression/issues can be fixed we can consider including the PR towards 4.13.0.0.