IPNS resolution in the browser

[jsonsivar]

Hello - thanks again for your work on js-ipfs. Excited to be using and testing it, and hopefully contribute. Currently I'm trying to use the in-browser node to resolve an IPNS address and it is failing. I've looked at the existing issues similar to this that are now closed but still unable to get it working. Details below:

js-ipfs version: 0.41.2
go-ipfs version: 0.4.23
browser version: Chrome 80.0.3987.132

In-browser node setup:

const node = await window.Ipfs.create({
  config: {
    dht: {
      enabled: true,
    },
  }
});
  
await node.swarm.connect("/dns4/node1.mydomain.com/tcp/8001/wss/ipfs/QmZRgRJUxUgakETE76vvLGGCdzusdPtsTWD7THv8aJy1iJ");
await node.swarm.connect("/dns4/node2.mydomain.com/tcp/8001/wss/ipfs/Qmcbo9iTcFoqXMfL7vGapt4TnSB6QPvk16aBE87cDB4LFZ");

The 2 nodes I connect to above are go-ipfs nodes on version 0.4.23 and ran with the --enable-namesys-pubsub flag. They are able to publish and resolve IPNS addresses among themselves fine, it's just the browser node that is unable to do so.

So basically when I try

await node.name.resolve('/ipns/QmZRgRJUxUgakETE76vvLGGCdzusdPtsTWD7THv8aJy1iJ');

I get the following error:

I will keep digging into this and post more info as I find it. For now, it's failing here and when I print routingKey.toBuffer().toString() I get "/ipns/� ��d��z?�ꃑ�Y��U���U��,}�87�x1��". It jumped out since part of it is string encoded and the rest is in binary but haven't looked elsewhere yet to see if this is expected.

[lidel]

Your js-ipfs node running in browser is unable to resolve /ipns/{libp2p-key} to an IPNS record due to a gap in peer routing.

I believe the key problem is that experimental DHT in JS is not ready for production use yet (iirc js-ipfs postponed DHT work until DHT changes land around go-ipfs 0.6)

For now, use delegated routing modules which enable js-ipfs nodes to query DHT using remote go-ipfs node. Try setting it up as noted in https://github.com/ipfs/js-ipfs/tree/master/packages/ipfs#configuring-delegate-routers. It should solve the peer routing problem, just make sure you also have preload nodes set up as well, because your browser node won't be able to dial TCP multiaddrs to fetch the content.

(cc #2093)

[lidel]

Hm.. I take that back, seems that right now IPNS resolution is somehow hardcoded to use DHT directly:

• js-ipfs/packages/ipfs-core/src/ipns/routing/config.js

  Lines 30 to 36 in d13d15f

  	// DHT should not be added as routing if we are offline or it is disabled
  	if (get(options, 'offline') || !get(options, 'libp2p.config.dht.enabled', false)) {
  	const offlineDatastore = new OfflineDatastore(repo)
  	ipnsStores.push(offlineDatastore)
  	} else {
  	ipnsStores.push(libp2p._dht)
  	}

We probably could fix this up so delegated routing is used instead, enabling use in the browser.

cc @vasco-santos to think about this in Q2

[kumavis]

@lidel one issue here is that the libp2p api changed here libp2p/js-libp2p#480
libp2p.dht is now libp2p._dht

having some typedoc validation would be good to catch these sort of things
or perhaps having ipnsStores.push validate the interface its being passed in

[vasco-santos]

Bear in mind that the goal for js-libp2p users is to use libp2p.contentRouting for this, instead of using the .dht directly. That was the reason to go with the private dht property, since the DHT should be used for content routing if libp2p is configured that way, otherwise other contentRouting modules should be used.

[kumavis]

@vasco-santos makes sense
this is the quick fix #2996

[kumavis]

@vasco-santos seems like the current state of content-routing is
the specified read/write interface (provide, findProviders) https://github.com/libp2p/js-libp2p-interfaces/tree/afc2aa6891c2f71b0c62aabccdca25666a62da71/src/content-routing#api
but also a wrapper around the dht (put, get, getMany) (is there a name for this interface? datastore?)
is there an issue tracking ipns/dht/content-routing work?
I'm interested in helping but unsure where we are and dont want to conflict

[kumavis]

@vasco-santos I guess I'm confused because the content-routing interface spec seems to imply that it only provides providers. But libp2p.contentRouting exposes a (datastore?) put/get/getMany interface

also confused because ipfs doesn't seem to use contentRouting at all

[vasco-santos]

You make a valid point here, which I also hit sometime ago when refactoring the DHT. With that in mind, we cannot use delegated routing for the time being in the browsers, since they do not implement {put, get}, until you figure out what you are pointing here.

Ideally, I think that {put, get} should be part of the contentRouting interface as well. We do not have a naming for it, in the dht code I named it content-fetching but I am not happy with that name. What do you folks think?

[kumavis]

the name content-fetching makes sense to me next to content-routing, and its function is intuitive except maybe for the put case

a question:
is the content-fetching implementation unique to the content-routing implementation, or can it just be a generic pipeline of getProviders -> request data from providers

starting with get

yes, but no
dht's get uses bestRecord selection with dht.selectors
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-fetching/index.js#L151
but by default its just a generic peer selector https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/index.js#L143

yesish because of desired side effects
after bestRecord selection we send our correction records
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-fetching/index.js#L166

now for getMany

yes, as an optimization
because we make a special query for getting values
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-fetching/index.js#L234
and exit the query path based on how many values we've seen
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-fetching/index.js#L254-L257
which is ultimately backed by a different network call
https://github.com/libp2p/js-libp2p-kad-dht/blob/f0fb2122caa10cf38cb7a5c06a0de1cc56d5830b/src/index.js#L521
than when just finding providers
https://github.com/libp2p/js-libp2p-kad-dht/blob/a94f4d43ce7ab638b8d05e3af3d94b98d23c2fdf/src/content-routing/index.js#L25

you could seemingly use the generic pipeline just fine but would result in either

1. extra network requests to found providers to actually get the data
2. or extra network io if you asked for the value but didnt actually need it

ok I answered my own question thank you 😁

sorry for the thread spam

[vasco-santos]

@kumavis thanks for exposing your thoughts here, this is valuable!

I would like to see us decoupling the dht in the future into the contentRouting and peerRouting, so that we can integrate it in the libp2p, like the delegated modules, as well as other future implementations. However, we need to fill the gap of the put, get, getMany! Since contentRouting modules might not need to implement the latest methods, it will probably make sense to create this new interface called content-fecthing, or anything better named. This would be what would be used for the IPNS here.

But we get to the same issue, since we intended to unblock IPNS resolution in the browser, which is not possible now with the delegated-content-routing, unless it provides the put and get. The other options, is to have a new delegated module for put and get if it makes sense.

cc @jacobheun since we will probably spend some time on the content routing side of things next quarter, this is something that I feel that we need to solve.

[autonome]

@vasco-santos Is this picture different now that we've shipped new content routing to the world and also shipped delegate nodes in js?

[vasco-santos]

However, we need to fill the gap of the put, get, getMany!

Delegate nodes don't do this. They allow findPeer, provide and findProviders.

The conversation here was going on a way to create the same concept for put and get, which are needed for IPNS. However, with the latest discussions and progress made, the way will be to use the DHT as client mode.

So, in theory this should work now if a browser node enables DHT client mode as explained in https://github.com/ipfs/blog/blob/master/content/post/104-js-ipfs-0.48.md#-dht-configuration

However, this still uses the incomplete and not stable DHT implementation for the time being. I would say that it is ready for experimenting with.

[rysiekpl]

I am trying to get inbrowser IPNS to work, and can't seem to. Tried with:

ipfs = await self.Ipfs.create({
            libp2p: {
                config: {
                    dht: {
                        enabled: true,
                        clientMode: true
                    }
                }
            }
        });

...and...

ipfs = await self.Ipfs.create({
            config: {
                dht: {
                    enabled: true,
                }
            }
        });

...and using both. Each time IPNS resolution (ipfs.name.resolve()) just fails with:

Error: "record requested for <CID> was not found in the network"

Would love some pointers what to try.

[vasco-santos]

@rysiekpl
I recommend that you use the ipns over pubsub for the time being, while we work on improving the content routing and discoverability in the browser context.
This example should help you out: https://github.com/ipfs-examples/js-ipfs-examples/blob/master/examples/browser-ipns-publish/
Let me know if you hit any issues