Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPFS not announcing CIDs on indirect peers #10266

Open
3 tasks done
iumarabbas opened this issue Dec 19, 2023 · 8 comments
Open
3 tasks done

IPFS not announcing CIDs on indirect peers #10266

iumarabbas opened this issue Dec 19, 2023 · 8 comments
Labels
kind/bug A bug in existing code (including security flaws) P3 Low: Not priority right now

Comments

@iumarabbas
Copy link

iumarabbas commented Dec 19, 2023

Checklist

Installation method

built from source

Version

ipfs version 0.25.0

Config

{
  "API": {
    "HTTPHeaders": {}
  },
  "Addresses": {
    "API": "/ip4/0.0.0.0/tcp/5001",
    "Announce": [],
    "AppendAnnounce": [],
    "Gateway": "/ip4/0.0.0.0/tcp/8080",
    "NoAnnounce": [],
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001",
      "/ip6/::/tcp/4001",
      "/ip4/0.0.0.0/udp/4001/quic",
      "/ip4/0.0.0.0/udp/4001/quic-v1",
      "/ip4/0.0.0.0/udp/4001/quic-v1/webtransport",
      "/ip6/::/udp/4001/quic",
      "/ip6/::/udp/4001/quic-v1",
      "/ip6/::/udp/4001/quic-v1/webtransport"
    ]
  },
  "AutoNAT": {},
  "Bootstrap": [
    "/ip4/__.___.100.178/tcp/4001/p2p/12D3KooJLq6rP69XfYWJkkKN29fsvhX5z5hZe5SYUai6guhMhLxa"
  ],
  "DHT": {
    "Server": true
  },
  "DNS": {
    "Resolvers": {}
  },
  "Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
          "child": {
            "path": "blocks",
            "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2",
            "sync": true,
            "type": "flatfs"
          },
          "mountpoint": "/blocks",
          "prefix": "flatfs.datastore",
          "type": "measure"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "StorageGCWatermark": 90,
    "StorageMax": "10GB"
  },
  "Discovery": {
    "MDNS": {
      "Enabled": true
    }
  },
  "Experimental": {
    "AcceleratedDHTClient": true,
    "FilestoreEnabled": false,
    "GraphsyncEnabled": false,
    "Libp2pStreamMounting": false,
    "OptimisticProvide": false,
    "OptimisticProvideJobsPoolSize": 0,
    "P2pHttpProxy": false,
    "StrategicProviding": false,
    "UrlstoreEnabled": false
  },
  "Gateway": {
    "APICommands": [],
    "HTTPHeaders": {
      "Access-Control-Allow-Headers": [
        "X-Requested-With",
        "Range",
        "User-Agent"
      ],
      "Access-Control-Allow-Methods": [
        "GET"
      ],
      "Access-Control-Allow-Origin": [
        "*"
      ]
    },
    "NoDNSLink": false,
    "NoFetch": false,
    "PathPrefixes": [],
    "PublicGateways": null,
    "RootRedirect": ""
  },
  "Identity": {
    "PeerID": "12D3KooWAAUpb6sUWUkdGngrMTb4QgjokRVxWghnG5FeG7QqJjMz"
  },
  "Internal": {},
  "Ipns": {
    "RecordLifetime": "",
    "RepublishPeriod": "",
    "ResolveCacheSize": 128
  },
  "Migration": {
    "DownloadSources": [],
    "Keep": ""
  },
  "Mounts": {
    "FuseAllowOther": false,
    "IPFS": "/ipfs",
    "IPNS": "/ipns"
  },
  "Peering": {
    "Peers": null
  },
  "Pinning": {
    "RemoteServices": {}
  },
  "Plugins": {
    "Plugins": null
  },
  "Provider": {
    "Strategy": ""
  },
  "Pubsub": {
    "DisableSigning": false,
    "Router": ""
  },
  "Reprovider": {},
  "Routing": {
    "Methods": null,
    "Routers": null,
    "Type": "dhtserver"
  },
  "Swarm": {
    "AddrFilters": null,
    "ConnMgr": {},
    "DisableBandwidthMetrics": false,
    "DisableNatPortMap": false,
    "RelayClient": {},
    "RelayService": {},
    "ResourceMgr": {
      "Limits": {}
    },
    "Transports": {
      "Multiplexers": {},
      "Network": {},
      "Security": {}
    }
  }
}

Description

I have four nodes NodeB, NodeA, Node0, and Node3
NodeB is peer of NodeA,
NodeA is peer of NodeB and Node0
Node0 is a peer of NodeA and Node3
Node3 is a peer of Node3

When I upload file on NodeB it is accessible on NodeA but not on Node0 and Node3 as these nodes are not directly added as peer but have same swarm key. what might be the reason?
I have configured all nodes as dhtserver and turned the dht server check to “active” but its not working.

When we get content from direct peer node, content is retrieved but for indirect peer the content is not accessible directly.

@iumarabbas iumarabbas added kind/bug A bug in existing code (including security flaws) need/triage Needs initial labeling and prioritization labels Dec 19, 2023
@Jorropo
Copy link
Contributor

Jorropo commented Dec 19, 2023

@iumarabbas can you show please ipfs stats dht on all nodes of the network ?

@iumarabbas
Copy link
Author

@Jorropo - Please see the response for both ipfs stats dht and ipfs swarm peers
As we have done the following

  1. Updated config with Routing.Type: dhtserver and dhtclient
  2. Updated config with DHT.Server: true
  3. Updated config with Tested with AccelaratedDHTClient: true
  4. manually announced the CID using ipfs dht provide then tried to fetch it using ipfs dht findprovs but no peer was returned.
  5. Tried annoucing using ipfs dht provide -r
  6. Tried with ipfs bitswap reprovide (command did not work)

All the nodes are connected through public IP and accessible publicly and the swarm key is same.

[ NodeB <--> NodeA ] <==>[ Node0 <--> Node3 ]

The nodes are hosted on azure cloud services:

NodeB and NodeA are in virtual network 1.
Node0 and Node3 are in the virual network 2.

NodeB:

ipfs stats dht

DHT wan (0 peers):
Bucket 0 (0 peers) - refreshed never:
Peer last useful last queried Agent Version
Error: no LAN client found

ipfs swarm peers
/ip4/..99.165/tcp/4001/p2p/12D3KooWSsdB2mJyQkq*****

NodeA

root@nodeA:/home/ssiadmin# ipfs stats dht
DHT wan (0 peers):
Bucket 0 (0 peers) - refreshed never:
Peer last useful last queried Agent Version

Error: no LAN client found
root@nodeA:/home/ssiadmin# ipfs swarm peers
/ip4/..99.199/tcp/4001/p2p/12D3KooWDpY9uHAEDxd*****
root@nodeA:/home/ssiadmin#

Node0

root@node0:/home/ssiadmin# ipfs stats dht
DHT wan (0 peers):
Bucket 0 (0 peers) - refreshed never:
Peer last useful last queried Agent Version

Error: no LAN client found
root@node0:/home/ssiadmin# ipfs swarm peers
/ip4/..98.222/tcp/4001/p2p/12D3KooWLPM8shRFvonSnn******
/ip4/..53.99/tcp/4001/p2p/12D3KooWNy35zuZFaqu3QXa******
root@node0:/home/ssiadmin#

Node3

root@node3:/home/ssiadmin# ipfs stats dht
DHT wan (0 peers):
Bucket 0 (0 peers) - refreshed never:
Peer last useful last queried Agent Version

Error: no LAN client found
root@node3:/home/ssiadmin# ipfs swarm peers
/ip4/..98.178/tcp/4001/p2p/12D3KooWETehntUtx6i63HBvrme*****
root@node3:/home/ssiadmin#

@Jorropo
Copy link
Contributor

Jorropo commented Dec 19, 2023

@iumarabbas the issue here is that the accelerated dht client is constantly failing to crawl your network, so it has no peer to issue queries to. It could also be that it hasn't finished yet however I would find that weird given your network is so slow it should crawl it in no times.
Can you also show ipfs stats dht wanserver on all nodes please, trying to see if this is a dht server or accelereated dht client issue ?

@iumarabbas
Copy link
Author

iumarabbas commented Dec 19, 2023

@Jorropo - As we are using Azure Cloud network so the network bandwidth/speed should not have issue here appearantly.

We are getting the following response to the above query: ipfs stats dht wanserver on all nodes,

root@node3:/home/ssiadmin# ipfs stats dht wanserver
DHT wanserver (0 peers):

@Jorropo
Copy link
Contributor

Jorropo commented Dec 19, 2023

Next steps would be to capture jaeger traces of the DHT to see what it is doing.
I don't have anything in mind and I think the thing I want to see (the server) isn't traced, would need to write some more code.

@mzuhaq7
Copy link

mzuhaq7 commented Dec 19, 2023

@Jorropo Judging from the documentation https://github.com/ipfs/kubo/blob/master/docs/experimental-features.md#private-networks . IPFS private network is an experimental feature. I suppose it's still not stable enough to be used for heavy load production environments

P.S: Same project as @iumarabbas

@Jorropo
Copy link
Contributor

Jorropo commented Dec 19, 2023

@mzuhaq7 it is the inverse problem, we struggle on small networks, if your network is bigger (and is mainly routed over WAN) it should work like the amino network.

@mzuhaq7
Copy link

mzuhaq7 commented Dec 21, 2023

@Jorropo I see. I tested it out on a network of 6 nodes but files were not being fetched from indirect nodes. I doubt if i increase network size more that'll help.

Is this a known issue in private IPFS? This seems to be a very basic functionality that should be working fine. We followed the simple steps to setup a private IPFS network here https://medium.com/@s_van_laar/deploy-a-private-ipfs-network-on-ubuntu-in-5-steps-5aad95f7261b

I see similar issues reported with no solution
https://discuss.ipfs.tech/t/cant-fetch-files-from-other-nodes/10893

@hacdias hacdias added P3 Low: Not priority right now and removed need/triage Needs initial labeling and prioritization labels Jan 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug A bug in existing code (including security flaws) P3 Low: Not priority right now
Projects
None yet
Development

No branches or pull requests

4 participants