Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: bulk CIDs import #2173

Open
asigalov61 opened this issue Oct 18, 2023 · 12 comments
Open

feat: bulk CIDs import #2173

asigalov61 opened this issue Oct 18, 2023 · 12 comments
Labels
effort/days Estimated to take multiple days, but less than a week exp/beginner Can be confidently tackled by newcomers good first issue Good issue for new contributors help wanted Seeking public contribution on this issue kind/enhancement A net-new feature or improvement to an existing feature P3 Low: Not priority right now status/deferred Conscious decision to pause or backlog

Comments

@asigalov61
Copy link

Hey guys,

I was wondering if it would be possible to add a bulk files/CIDs import from a txt or csv file?

I have a list of CIDs but there is no option in IPFS-Desktop to import them in bulk/automatically.

Can you implement this option? Or, if it is possible to do via Kubo or some other script method, it would be nice to see such method in the docs.

Please let me know.

Alex
@asigalov61 asigalov61 added the need/triage Needs initial labeling and prioritization label Oct 18, 2023
@welcome

This comment was marked as outdated.

Sign in to view
@SgtPooki
Copy link
Member

@asigalov61 Great question. You should be able to do this very easily by dragging and dropping files onto the IPFS-Desktop "Files" screen. Please let us know if you experience any issues with this.

@SgtPooki SgtPooki added need/author-input Needs input from the original author and removed need/triage Needs initial labeling and prioritization labels Oct 19, 2023
@asigalov61
Copy link
Author

@SgtPooki Thank you for your response. But I was asking about download, not upload.

I have a csv file with IPFS CIDs and I want to import all CIDs so that I can download them.

There seems to be only option to do it one by one by going to Files -> Import from IPFS, but there is no bulk "Import from IPFS" option.

Am I missing something?

Please let me know.

Thank you.

@github-actions

This comment was marked as off-topic.

Sign in to view
@SgtPooki
Copy link
Member

SgtPooki commented Oct 26, 2023 •
edited

@asigalov61 I see! sorry for my confusion.. I think an option to bulk import CIDs would be a great addition to IPFS-Desktop. Just FYI: that change would likely live in the ipfs-webui repo.

It might be a while until we get to this feature since a lot of our focus has been on Helia, but to unblock yourself for now, I threw together https://github.com/SgtPooki/npm-bulk-import-cid that you can use via npm (its at https://www.npmjs.com/package/@sgtpooki/bulk-import-cid) with

$ npx @sgtpooki/bulk-import-cid <filename> <optionalTimeoutMs>

Running it from that repo, with the example.txt file, here is the output:

╰─ ✘ 1 ❯ npx @sgtpooki/bulk-import-cid@latest example.txt
Starting up Kubo daemon...
Reading CIDs from file at path /Users/sgtpooki/code/gist-scripts/bulk-import-cid/example.txt
Read 3 lines from file
daemon stdout: Initializing daemon...
daemon stdout: Kubo version: 0.23.0
daemon stdout: Repo version: 15
daemon stdout: System version: arm64/darwin
daemon stdout: Golang version: go1.21.1
daemon stdout:
daemon stdout: Swarm listening on /ip4/127.0.0.1/tcp/4001
daemon stdout: Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip4/192.168.1.227/tcp/4001
daemon stdout: Swarm listening on /ip4/192.168.1.227/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip4/192.168.1.227/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip6/::1/tcp/4001
daemon stdout: Swarm listening on /ip6/::1/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/tcp/4001
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/udp/4001/quic-v1
daemon stdout: Swarm listening on /ip6/fd52:f4cb:d580:44c1:c5d:34ba:60ea:4c67/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm listening on /p2p-circuit
daemon stdout:
daemon stdout: Swarm announcing /ip4/127.0.0.1/tcp/4001
daemon stdout: Swarm announcing /ip4/127.0.0.1/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip4/127.0.0.1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip4/192.168.1.227/tcp/4001
daemon stdout: Swarm announcing /ip4/192.168.1.227/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip4/192.168.1.227/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip4/97.126.55.89/tcp/19055
daemon stdout: Swarm announcing /ip4/97.126.55.89/udp/19055/quic-v1
daemon stdout: Swarm announcing /ip4/97.126.55.89/udp/19055/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: Swarm announcing /ip6/::1/tcp/4001
daemon stdout: Swarm announcing /ip6/::1/udp/4001/quic-v1
daemon stdout: Swarm announcing /ip6/::1/udp/4001/quic-v1/webtransport/certhash/uEiDjX9szfgnfDL3oFt4XBHhGVJVB2LsKIhinHZgJ9GThjw/certhash/uEiBLNms_w1pYZ__xDTbNeVngAFvVyUY7EJifUuwv96Cplw
daemon stdout: RPC API server listening on /ip4/0.0.0.0/tcp/5001
daemon stdout: WebUI: http://0.0.0.0:5001/webui
daemon stdout:
daemon stdout: Gateway server listening on /ip4/127.0.0.1/tcp/8081
daemon stdout:
Kubo daemon is ready
daemon stdout: Daemon is ready
daemon stdout:
Creating import folder to host imported CIDs on MFS at path "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Successfully created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Importing bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq with name "bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" on MFS
Importing QmawceGscqN4o8Y8Fv26UUmB454kn2bnkXV5tEQYc4jBd6 with name "barrel.png" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/barrel.png" on MFS
Importing QmbvrHYWXAU1BuxMPNRtfeF4DS2oPmo5hat7ocqAkNPr74 with name "pi equals.png" to "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/pi equals.png" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/pi equals.png" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/bafkreibm6jg3ux5qumhcn2b3flc3tyu6dmlb4xa7u5bf44yegnrjhc4yeq" on MFS
Created "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z/barrel.png" on MFS
Imported all CIDs from "example.txt" to MFS folder "/@sgtpooki_bulk-import-cid:2023-10-26T20:15:02.451Z"
Shutting down...

and we can see them in IPFS desktop after.

image

NOTE: Since this is adding to the same IPFS blockstore IPFS-Desktop uses, you must shutdown IPFS-Desktop in order to use the above tool, but it should tell you that.

@SgtPooki SgtPooki added P3 Low: Not priority right now kind/enhancement A net-new feature or improvement to an existing feature exp/beginner Can be confidently tackled by newcomers effort/days Estimated to take multiple days, but less than a week status/deferred Conscious decision to pause or backlog and removed need/author-input Needs input from the original author kind/stale labels Oct 26, 2023
@asigalov61
Copy link
Author

@SgtPooki Ok. Thank you. I will try it and I will let you know if there are any problems. Thank you again for extra support on this :)

@SgtPooki SgtPooki changed the title Bulk files import (bulk CIDs import) feat: bulk CIDs import Nov 1, 2023
@SgtPooki SgtPooki reopened this Nov 1, 2023
@SgtPooki
Copy link
Member

SgtPooki commented Nov 1, 2023

keeping this open as we still would like to support this in ipfs-desktop/ipfs-webui. Moving to ipfs-webui though

@welcome Welcome

This comment was marked as outdated.

Sign in to view
@SgtPooki SgtPooki transferred this issue from ipfs/ipfs-desktop Nov 1, 2023
@asigalov61
Copy link
Author

@SgtPooki Yes, sorry about closing it. I would love to see the native support for bulk CID import in IPSF desktop :) Thank you, guys, for your hard work. IPFS is an important and needed thing :)

@asigalov61
Copy link
Author

@SgtPooki Dear Sgt Pooki,

Any update on this feature? Any ETA on when it will be implemented?

Thank you so much in advance for your response.

Alex

@SgtPooki
Copy link
Member

@asigalov61 unfortunately, we've got a lot of items on our plate now, and a smaller team. I would be happy to review a PR that added a "bulk import" option to the Import button on the files screen:

image

but won't be able to work on this for the forsee-able future.

@SgtPooki SgtPooki added help wanted Seeking public contribution on this issue good first issue Good issue for new contributors labels Feb 13, 2024
@asigalov61
Copy link
Author

@SgtPooki Thank you for your response and update :)

Unfortunately, IPFS code is too complex for me to contribute to. Sorry. I am mostly Python ML/DL dev :(

But please add this feature when you are able. I think it would be a useful addition to IPFS.

I will use your workaround for now, for which I am grateful nevertherless :)

Sincerely,

Alex.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
effort/days Estimated to take multiple days, but less than a week exp/beginner Can be confidently tackled by newcomers good first issue Good issue for new contributors help wanted Seeking public contribution on this issue kind/enhancement A net-new feature or improvement to an existing feature P3 Low: Not priority right now status/deferred Conscious decision to pause or backlog
Projects
No open projects
IPFS Shipyard Team
Status: No status
IPFS-GUI (PL EngRes)
Status: UX Design Needed
Milestone
No milestone
Development

No branches or pull requests
2 participants
@SgtPooki @asigalov61