IPNS is not working consistently

[imthe-1]

@BigLep @2color @lidel @tinytb

What we need

hey IPNS, publish a record with lifetime 100years using k1 keys. Published. Oh wait! there is new data, please update with same lifetime. Reslolves from a different node. Data relects. BAMMM!

In detail, what we need is:

1. Resolution of IPNS names to the latest value when records are published/updated within short time span such as 10 seconds.
2. Achieve IPNS Permanence - persist IPNS names for a long period of time(theoretically, forever. lifetime: 10... * 365 * 24 * 60 * 60 * 1000).

Current long standing issues with js-ipfs that we have not been able to fix or find root cause to:

1. The 1st ever resolution attempt with a new secp256k1 keypair always fails. we resolve 2 times as a work around.
   • We learned from the IPFS devs that since the subscription to topic happens in the first resolution hence it fails - Please correct/update us if this is not the case.
2. Resolution then starts working and gives us the 1st published record(the data that was published the first time). great! but subsequent new records(stringified json objects) that we publish does not get resolved to at all instead each resolution returns the 1st published record only. Inconsistent resolution as if the nodes dont honour the new record or the increased sequence number until the lifetime expires. This happens when longer lifetimes such as 24hr or 3day are used.
3. When the same scenario is tried with a lower lifetime value such as 2min, IPNS works consistently and resolves to the latest value each time. This probably happens because the DHT, maybe deletes the record after 2 mins and hence, by the time a new publish reaches, DHT adds a new entry and is able to resolve to the latest value but we have seen it resolving to the latest record even if we publish within the 2mins lifetime. strange. Now the problem with lower lifetime value is - we cannot achieve IPNS permanence and publishing every 2mins has is a major scalibilty issue for thousands of users.

Link to a sample we created with js-ipfs to reproduce the issue: https://github.com/imthe-1/keychain-ipns-sample/tree/debug/lifetime-persistence-verbose

We have the same sample use-case implemented in Helia as well althoug as an inital PoC only. Helia PoC findings:

We followed a similar approach as browser-ipns-publish in Helia for proper IPNS name propagation.

1. We like Helia but it seems to suffer from the same issue as js-ipfs when it comes to IPNS.
2. We need not resolve 2 times for the resolution to work.
3. After publishing via helia in the browser, IPNS resolution on our remote node and public gateways does not always work. Some IPNS names resolve but other IPNS names fail.
4. Of the IPNS names that do resolve from the our remote kubo node via API /api/v0/name/resolve or public IPFS gateways, they still returns the 1st published value even after new reocrds are published. There is still inconsistent resolution, latest record are not returned.
5. IPNS publish and resolution to latest value via helia works fine in the browser but also sometimes fails intermittently.
6. Using lower lifetimes(in ms here) such as 2 * 60 * 1000 OR 10 * 60 * 1000 did not result in any new outcomes and behaved the same as 24hrs or longer lifetimes.

Link to the sample code with Helia: https://github.com/imthe1self/helia-connectivity-demo-forked-/tree/debug/topic
PS: we used code from codesandbox.io that was probably created by @SgtPooki and added our sample use-case to it. thank you @SgtPooki!

Our use-case

Publish data to IPNS and update it over time from the client device(mobile app or browser). Update can happen as quickly as within a few seconds from the last publish in certain scenarios. For this we run an IPFS node in the browser, create an secp256k1 IPNS keypair, obtain the topic name and make this browser node and one remote kubo node subscribe to this topic. We then publish to IPNS.

We have followed this example for browser IPNS publish(link: https://github.com/ipfs-examples/js-ipfs-examples/tree/master/examples/browser-ipns-publish)

Note: we use secp256k1 IPNS keypairs and have modified "createKey" in js-ipfs to accept an optional private key "optPrivateKey" for creating the same IPNS keypair when required. There is a specific reason for this and we can't use RSA or ED25519 keypairs.

[SgtPooki]

We followed a similar approach as browser-ipns-publish in Helia for proper IPNS name propagation.

1. We like Helia but it seems to suffer from the same issue as js-ipfs when it comes to IPNS.

I believe the implementation js-ipfs used is the same as what helia uses: https://github.com/ipfs/js-ipns

2. We need not resolve 2 times for the resolution to work.

That's good news, right?

3. After publishing via helia in the browser, IPNS resolution on our remote node and public gateways does not always work. Some IPNS names resolve but other IPNS names fail.

This is most likely due to problems called out in #182. follow that issue for more.

4. Of the IPNS names that do resolve from the our remote kubo node via API /api/v0/name/resolve or public IPFS gateways, they still returns the 1st published value even after new reocrds are published. There is still inconsistent resolution, latest record are not returned.

While #182 is not resolved, I imagine this will be an issue for browser-authored records.

5. IPNS publish and resolution to latest value via helia works fine in the browser but also sometimes fails intermittently.

Another item I expect resolving #182 to help resolve

6. Using lower lifetimes(in ms here) such as 2 * 60 * 1000 OR 10 * 60 * 1000 did not result in any new outcomes and behaved the same as 24hrs or longer lifetimes.

the lifetime value is set here. I get this is probably a naive question, but can you confirm you are passing the value properly?

The code where you're calling the actual js-ipns implementation with lifetime (1, 2) seems to be hidden behind the private @mdip/client library.

I was able to find the code for the function you're using on npm:

async linkToIPNS(ipfsAddr, keyName, keyId, opts = {}) {
    const namespace = '/record/';
    const b58 = base58btc.decode(`z${keyId}`);
    // console.log('b58', b58);
    const ipnsKeys = ipns.getIdKeys(b58);
    // console.log('ipnsKeys', ipnsKeys);
    const topic = `${namespace}${uint8ArrayToString(
      ipnsKeys.routingKey.uint8Array(),
      'base64url',
    )}`;
    // console.log('topic', topic);

    await this.coreClient.pubsub.subscribe(topic, (msg) => {
      // console.log('[coreClient]', msg);
    });
    await this.ipfsHttpClient.pubsub.subscribe(topic, (msg) => {
      // console.log('[ipfsHttpClient]', msg);
    });

    const httpClientSubs = await this.ipfsHttpClient.pubsub.ls();
    // console.log('[httpClientSubs]', httpClientSubs);
    if (
      !httpClientSubs ||
      !httpClientSubs.length ||
      !httpClientSubs.includes(topic)
    ) {
      console.warn('[httpClient could not subscribe]');
    }
    const coreClientSubs = await this.coreClient.pubsub.peers(topic);
    // console.log('[coreClientSubs]', coreClientSubs);
    const { id: clientPeerId } = await this.ipfsHttpClient.id();
    if (
      !coreClientSubs ||
      !coreClientSubs.length ||
      !coreClientSubs.includes(clientPeerId)
    ) {
      console.warn('[coreClient could not find peer]');
    }

    const multihash = uint8ArrayFromString(keyId, 'base58btc');
    const digest = Digest.decode(multihash);
    const libp2pKey = CID.createV1(0x72, digest);
    const ipnsName = `/ipns/${libp2pKey.toString(base36)}`;

    const httpClientNamesysSubs = await this.ipfsHttpClient.name.pubsub.subs(); // API
    if (!httpClientNamesysSubs.includes(ipnsName)) {
      console.warn('IPNS path not found');
    }
    const resp = await this.coreClient.name.publish(ipfsAddr, {
      key: keyName,
      resolve: false,
      lifetime: opts.lifetime || '3day',
    });
    return resp;
  }

3day isn't a valid option for lifetime in the js-ipns library. Have you tried using a valid nanosecond identifier?

---

Is there a modification to tests at https://github.com/ipfs/js-ipns/tree/3d3807f9a0ed5b1d5b06990c916a426d7c9200d0/test that could reproduce your problem more specifically?

[github-actions]

Oops, seems like we needed more information for this issue, please comment with more details or this issue will be closed in 7 days.

[github-actions]

This issue was closed because it is missing author input.

[imthe-1]

hi @SgtPooki

apologies for the late reply.

That's good news, right?

Yes

This is most likely due to problems called out in #182. follow that issue for more.
While #182 is not resolved, I imagine this will be an issue for browser-authored records.
Another item I expect resolving #182 to help resolve

@SgtPooki after going through the details in #182 it feels like until this issue is addressed and the underlying 2 other issues Helia will continue to have this problem. We can also see that there is some progress on the issue(Enable WebRTC Transport to help solve #256). We do however, have the following concerns:

1. The issues #182 #255 #256 are still in "No Status" tab in the Helia Working Group since this is lower priority and not a pure Helia issue. Concern here is is there an estimated timeline by when this becomes available?
2. We specifically have issues with IPNS. will #182 solve all IPNS issues as well?

Meanwhile this above is worked upon can you please suggest some alternatives or guide us in a direction using which we can achieve:

1. Resolution of the mutable link to the latest value when records are published/updated within short time span such as 10 seconds.
2. Achieve IPNS/Mutable name Permanence - persist IPNS/Mutable name for a long period of time(theoretically, forever. lifetime: 10... * 365 * 24 * 60 * 60 * 1000).

[pcfreak30]

My project, Lume, which uses helia also has some very bad hit/miss on IPNS when trying to even load things like ens.eth. I use a mix of transports including a custom one, and preload/delegate nodes, so a root cause is really not known, but from my pov... IPNS is highly unstable.

IPFS works a lot better, though I still have failed to load JS files from ens.eth, but getting WebRTC would be the best thing long term.

[SgtPooki]

My project, Lume, which uses helia also has some very bad hit/miss on IPNS when trying to even load things like ens.eth

I think ipfs/helia-ipns#55 may help out for users coming to this issue with problems resolving DNSLink records: ipfs/helia-ipns#55

That is separate from this issue, as IPNS records being discussed here involve the records stored in the Amino DHT