Exception when using partial / third party CDN sources

[esteluk]

• I've read and understood the CONTRIBUTING guidelines and have done my best effort to follow.

Report

What did you do?

If you configure a Podfile to use two CDN sources – one of which is the trunk, and the second is a private spec CDN that does not contain a complete mirror of the trunk, pod install fails to complete when I try to use dependencies from both.

source 'https://cdn.mycompany.com/api/pods/pods/'
source 'https://cdn.cocoapods.org/'

pod 'PrivateDependency'
pod 'Swiftlint'

What did you expect to happen?

Installs all pod dependencies successfully.

What happened instead?

### Error

Errno::ENOENT - No such file or directory @ rb_sysopen - /Users/nathan/.cocoapods/repos/pods-private/all_pods_versions_4_0_1.txt
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:327:in `initialize'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:327:in `open'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:327:in `local_file'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:286:in `ensure_versions_file_loaded'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/cdn_source.rb:208:in `search'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/source/aggregate.rb:83:in `block in search'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/source/aggregate.rb:83:in `select'
/Users/nathan/Developer/Projects/open/cocoapods/Core/lib/cocoapods-core/source/aggregate.rb:83:in `search'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:416:in `create_set_from_sources'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:385:in `find_cached_set'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:360:in `specifications_for_dependency'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:165:in `search_for'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:274:in `block in sort_dependencies'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:267:in `each'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:267:in `sort_by'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:267:in `sort_by!'
/Users/nathan/.rbenv/versions/3.2.3/lib/ruby/gems/3.2.0/gems/cocoapods-1.15.2/lib/cocoapods/resolver.rb:267:in `sort_dependencies'

Swiftlint uses CDN shard 4_0_1 and Cocoapods queries each CDN repo for the contents of this shard.
If this CDN shard doesn't exist on a repo (which is very likely for any private CDN that doesn't copy the trunk, but very unlikely for the real trunk), this file returns a 404 and isn't downloaded successfully. This leads to the above exception ☝🏽

If I remove any trunk dependencies from my Podfile, pod install completes as expected.

CocoaPods Environment

Stack

   CocoaPods : 1.15.2
        Ruby : ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [arm64-darwin23]
    RubyGems : 3.5.5
        Host : macOS 14.3.1 (23D60)
       Xcode : 15.3 (15E204a)
         Git : git version 2.39.3 (Apple Git-146)
Ruby lib dir : /Users/nathan/.rbenv/versions/3.2.3/lib
Repositories : 
               pods-private - CDN - https://cdn.mycompany.com/api/pods/pods/
               trunk - CDN - https://cdn.cocoapods.org/

Installation Source

Executable Path: /Users/nathan/.rbenv/versions/3.2/bin/pod

Plugins

cocoapods-deintegrate : 1.0.5
cocoapods-plugins     : 1.0.0
cocoapods-search      : 1.0.1
cocoapods-trunk       : 1.6.0
cocoapods-try         : 1.2.0
slather               : 2.7.4

Project that demonstrates the issue

Will try to identify this later.

[santam85]

@amorde We use JFrog as a CDN repo and we are running into the same. Any clue if they incompletely implemented the protocol, or is it a bug in the client?

[igor-makarov]

@santam85 @esteluk
CDN support isn't limited to the Trunk.

It's entirely possible that JFrog's Artifactory has done something wrong - they've had limited contact with CocoaPods when they developed their products. It could also be a misconfiguration of some sort.

Try contacting their support for assistance.

[santam85]

@igor-makarov do you have any other CDN source different from the trunk we can use to test a configuration similar to the above example? To pinpoint if the issue is with JFrog or not.

[igor-makarov]

@santam85 sadly, I do not. There was one I personally had for experiments, but I've shut it down due to lack of use.

You could build your own CDN if you follow the format as specified at https://github.com/CocoaPods/cdn.cocoapods.org

[santam85]

@igor-makarov how would you explain the error we are getting then? From what i gather, the Cocoapods client tries to download the sharded podspec (4/0/1) from our private repo for Swiftlint (available on trunk), fails to do so as we have no Pod matching that particular shard, and then crashing.

[igor-makarov]

@santam85 It's okay if you don't have the pod in the shard, but you need to have an empty shard that corresponds to 4/0/1.

[santam85]

@igor-makarov the code at https://github.com/CocoaPods/cdn.cocoapods.org/blob/main/Scripts/create_pods_and_versions_index.rb does not seem to behave that way. There might be gaps in the hashes if there is no pod matching a particular prefix, so this needs to be addressed either in the client (my preference, more economical solution) or in the pod version index generation.

[igor-makarov]

The CDN support was intended to deal with the size the trunk repo, and its unique performance needs. Trunk does not have missing shards, due to the sheer number of pods.

There's very limited badwidth to address changes to either the client or server components - my realistic recommendation is to switch to using git for your non-trunk spec repo.

Also, I would like to reiterate that JFrog's implementation of "CDN-like" spec repos is not affiliated with CocoaPods and I do not know how it works.

[santam85]

Unfortunately that recommendation does not work for our enterprise, so we are stuck with what JFrog offers us. We'll work with them trying to address potential bugs in their implementation, but would you be open to us to keep working on #771 to allow the client to keep looking for pods on other sources if it can't find the required shard?

[igor-makarov]

CocoaPods is a project that's entirely dependent on the mainteneace work of volunteers.

Personally, I cannot justify devoting my time to a PR trying to fix a problem whose root cause is a non-compliant implementation by a third party (JFrog) of a feature that's being used in an unintended manner.

@eyalbe4 @yahavi - there's a long-standing compatibility problem in Artifactory that I think you need to address.

[santam85]

I fail to see how their implementation is non-compliant: if they were running the code you posted in this issue to generate the json shards, they would end up with exactly the problem at hand, and gaps in the generated shards. I couldn't find any reference documentation on the CocoaPods docs that also spells out this requirement for empty shards in private CDN repos, can you point me to it?

We are willing to contribute any needed effort to get it fixed, as we are heavy users of CocoaPods and can provide insights on requirements for Enterprise users.

[igor-makarov]

@santam85 there was never an intention for the CDN format to be used for third party repos. The documentation in https://github.com/CocoaPods/cdn.cocoapods.org is not instructions on how to set up your own CDN repo. This is out of scope for CocoaPods. It's a description of how the code works.

Please understand that there's absolutely no bandwidth in CocoaPods to deal with any of this. Talk to your vendor.

[santam85]

You said yourself earlier:

CDN support isn't limited to the Trunk.

but now you seem to have backtracked on that:

there was never an intention for the CDN format to be used for third party repos

We are definitely talking to our vendor about it, but they will also likely require some form of support from Cocoapods to comply with these undocumented standards and expectations.

I don't understand why this much resistance to improve Cocoapods, in the spirit of open-source we're happy to contribute with the resources needed to address this, and also sponsor and help maintain the project.

[maikelvdh]

Great to see this ticket (and PR), as I have raised similar one with comparing patch here: CocoaPods/CocoaPods#12338! I was awaiting some feedback on the issue which unfortunately didn't reached any further attention yet!