Using two Karaf instances in one VM

[amarsico]

Dear all,
I have a question that never comes from you, but when I was testing the Core with ONOS it happens.

If I have one virtual machine and two Karaf instances I always get errors like this one when I start the second instance:

java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)[:1.8.0_74]
    at sun.nio.ch.Net.bind(Net.java:433)[:1.8.0_74]
    at sun.nio.ch.Net.bind(Net.java:425)[:1.8.0_74]
    at sun.nio.ch.AsynchronousServerSocketChannelImpl.bind(AsynchronousServerSocketChannelImpl.java:162)[:1.8.0_74]
    at org.apache.sshd.common.io.nio2.Nio2Acceptor.bind(Nio2Acceptor.java:66)
    at org.apache.sshd.common.io.nio2.Nio2Acceptor.bind(Nio2Acceptor.java:74)
    at org.apache.sshd.SshServer.start(SshServer.java:287)
    at org.apache.karaf.shell.ssh.SshServerFactory.start(SshServerFactory.java:71)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)[:1.8.0_74]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)[:1.8.0_74]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)[:1.8.0_74]
    at java.lang.reflect.Method.invoke(Method.java:498)[:1.8.0_74]
    at org.apache.aries.blueprint.utils.ReflectionUtils.invoke(ReflectionUtils.java:299)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BeanRecipe.invoke(BeanRecipe.java:956)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BeanRecipe.runBeanProcInit(BeanRecipe.java:712)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BeanRecipe.internalCreate2(BeanRecipe.java:824)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BeanRecipe.internalCreate(BeanRecipe.java:787)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.di.AbstractRecipe$1.call(AbstractRecipe.java:79)[15:org.apache.aries.blueprint.core:1.4.4]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)[:1.8.0_74]
    at org.apache.aries.blueprint.di.AbstractRecipe.create(AbstractRecipe.java:88)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BlueprintRepository.createInstances(BlueprintRepository.java:247)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BlueprintRepository.createAll(BlueprintRepository.java:183)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BlueprintContainerImpl.instantiateEagerComponents(BlueprintContainerImpl.java:682)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:377)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:269)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:294)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BlueprintExtender.createContainer(BlueprintExtender.java:263)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:253)[15:org.apache.aries.blueprint.core:1.4.4]
    at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)[9:org.apache.aries.util:1.1.1]
    at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)[9:org.apache.aries.util:1.1.1]
    at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)[9:org.apache.aries.util:1.1.1]
    at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)[9:org.apache.aries.util:1.1.1]
    at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)[9:org.apache.aries.util:1.1.1]
    at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)[org.apache.felix.framework-4.2.1.jar:]
    at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)[org.apache.felix.framework-4.2.1.jar:]
    at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)[org.apache.felix.framework-4.2.1.jar:]
    at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4403)[org.apache.felix.framework-4.2.1.jar:]
    at org.apache.felix.framework.Felix.startBundle(Felix.java:2092)[org.apache.felix.framework-4.2.1.jar:]
    at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1291)[org.apache.felix.framework-4.2.1.jar:]
    at org.apache.felix.framework.FrameworkStartLevelImpl.run(FrameworkStartLevelImpl.java:304)[org.apache.felix.framework-4.2.1.jar:]
    at java.lang.Thread.run(Thread.java:745)[:1.8.0_74]

This is because Karaf uses several services, such as Java RMI and SSH daemon, that bind host ports. I never get issue from you, but I tried to change the default ports of the Karaf instance of NetIDE Core, it does not raise errors but it becomes unstable. (e.g. the loadcomposition command does not work properly).

Has anyone resolved or encountered the issue?

[sergiotamu]

Hello @amarsico,

When I tested the NetIDE Engine before Brussels Meeting I obtained a similar error when I run the ODL shim and the Java Core.

Now, when I run the Java Core and the Onos shim (in this order and in two different karaf instances) I also obtain an exception in the onos terminal:

onos> Exception in thread "JMX Connector Thread [service:jmx:rmi://0.0.0.0:44444/jndi/rmi://0.0.0.0:1099/karaf-root]" java.lang.RuntimeException:
Port already in use: 44444;
You may have started two containers. If you need to start a second container or the default ports are already in use update the config file etc/org.apache.karaf.management.cfg and change the Registry Port and Server Port to unused ports
at org.apache.karaf.management.ConnectorServerFactory$1.run(ConnectorServerFactory.java:254)

[pablomuri]

Hello @sergiotamu,

That issue is solved in the readme odl-shim.

Note: To run another Karaf instance (apart from ODL's), we need to modify the following file: odl-shim/karaf/target/assembly/etc/org.apache.karaf and manually change ports 1099 and 44444 to something different.

[amarsico]

Ok, but this solution is for ODL and its Karaf instance. For ONOS is different, since there are BASH scripts that interacts with Karaf and you have to change them.

I think we might change the Karaf Core configuration and add the following lines to the Java Core README:

vim KARAF_FOLDER/etc/org.apache.karaf.management.cfg and change the following variables to these values:
rmiRegistryPort = 1101
rmiServerPort = 44447

vim KARAF_FOLDER/etc/org.apache.karaf.shell.cfg and change the following variable to this value:
sshPort = 8103

However, changing these values creates an issue in loading the composition file (loadcomposition command), because the URL for JMXServiceURL is static. (see the file eu.netide.core.management.ManagementHandler, line 149, 175). This is why I got issue last time when loading the composition file. (I don't know if there is a way to get it dynamically from blueprints or other stuff)

We have to agree to a common modification, since we are going to deploy also automatically the Engine with ODL or ONOS as servers.

Let me know what you think.

Best

[schwabe]

@amarsico The url for loadcomposition is static because I and Tim did not know how to set karaf config attributes in another way. There is probably a better way that does not depend on the port. I will have to check that.

[muhammad-arslan-591]

Along with the @amarsico changes, you also need to change HTTP port in ./etc/org.ops4j.pax.web.cfg, after that try to restart the karaf, issue will be resolved.

org.osgi.service.http.port=8183