Concurrent building occasionally cleans the Pods cache directory

[fabcz]

Report

What did you do?

• Execute pod update --no-repo-update --verbose concurrently on build machines.

What did you expect to happen?

• I hope the command can be executed normally without accidentally deleting the cocoapods cache.

What happened instead?

• Sometimes there may be build failures with the error message Failed to download 'MangoFix': Directory not empty @ dir_s_rmdir - /Users/admin/Library/Caches/CocoaPods/Pods.
• At the same time, the entire cocoapods cache directory /Users/chengcong/Library/Caches/CocoaPods/Pods may be deleted. However, all our projects use the same cocoapods version.

CocoaPods Environment

Stack

   CocoaPods : 1.11.3
        Ruby : ruby 2.6.8p205 (2021-07-07 revision 67951) [universal.arm64e-darwin21]
    RubyGems : 3.0.3.1
        Host : macOS 12.5 (21G72)
       Xcode : 14.2 (14C18)
         Git : git version 2.37.1 (Apple Git-137.1)
Ruby lib dir : /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/lib
Repositories : cocoapods - git - https://github.com/CocoaPods/Specs.git @ 8be767caa237ea7fa8efa91b89fac09ff029e08e
               huya-ci_team-specs - git - https://git.huya.com/ci_team/Specs.git @ 353d57ed49266b83557eb4a77d520c8246409fe9
               huya-huya-rn-public-specs - git - https://git.huya.com/huya-rn-public/Specs.git @ b3e8f60d9e38f76f73bf4816fd5c11663fcacb74

Installation Source

Executable Path: /usr/local/bin/pod

Plugins

cocoapods-deintegrate                 : 1.0.5
cocoapods-disable-podfile-validations : 0.1.1
cocoapods-generate                    : 2.2.2
cocoapods-plugins                     : 1.0.0
cocoapods-search                      : 1.0.1
cocoapods-trunk                       : 1.6.0
cocoapods-try                         : 1.2.0

Project that demonstrates the issue

Crash stack trace as follows

[08:01:03] -> Pre-downloading: `MangoFix` from `https://git.huya.com/huya-app-public/3rd-library-ios/Mango.git`, tag `1.4.6-kiwi-build-in-ext`
[08:01:03] [!] Failed to download 'MangoFix': Directory not empty @ dir_s_rmdir - /Users/admin/Library/Caches/CocoaPods/Pods
[08:01:03]
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/external_sources/abstract_external_source.rb:120:in `rescue in block in pre_download'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/external_sources/abstract_external_source.rb:115:in `block in pre_download'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/user_interface.rb:86:in `titled_section'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/external_sources/abstract_external_source.rb:113:in `pre_download'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/external_sources/downloader_source.rb:13:in `fetch'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:993:in `fetch_external_source'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:972:in `block (2 levels) in fetch_external_sources'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:971:in `each'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:971:in `block in fetch_external_sources'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/user_interface.rb:64:in `section'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:970:in `fetch_external_sources'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer/analyzer.rb:117:in `analyze'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer.rb:416:in `analyze'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer.rb:241:in `block in resolve_dependencies'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/user_interface.rb:64:in `section'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer.rb:240:in `resolve_dependencies'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/installer.rb:161:in `install!'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/command/update.rb:63:in `run'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/claide-1.1.0/lib/claide/command.rb:334:in `run'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/lib/cocoapods/command.rb:52:in `run'
[08:01:03] /Library/Ruby/Gems/2.6.0/gems/cocoapods-1.11.3/bin/pod:55:in `'
[08:01:03] /usr/local/bin/pod:23:in `load'
[08:01:03] /usr/local/bin/pod:23:in `
'
[08:01:03] error: pod update failed

Problem Analysis

• From the stack trace, it can be seen that the code that threw the final exception is located at Failed to download. However, this code is a begin, rescue block, so the actual code that caused the exception to occur should be located at download_result = Downloader.download(download_request, target, :can_cache => can_cache).
• The sentence Failed to download 'MangoFix': Directory not empty @ dir_s_rmdir - /Users/admin/Library/Caches/CocoaPods/Pods indicates that when executing rmdir to delete a directory, the directory is not empty, and this directory is our cache directory.
• Further debugging revealed a suspicious method ensure_matching_version.

# Ensures the cache on disk was created with the same CocoaPods version as
# is currently running.
#
# @return [Void]
#
def ensure_matching_version
  version_file = root + 'VERSION'
  version = version_file.read.strip if version_file.file?

  root.rmtree if version != Pod::VERSION && root.exist?
  root.mkpath

  version_file.open('w') { |f| f << Pod::VERSION }
end

• Here is an operation to delete the cache directory root.rmtree, followed immediately by another line root.mkpath. Therefore, it is suspected that the issue is caused by concurrent operations, because when executing rmtree, another thread may have already created this directory, resulting in the Directory not empty error.
• However, my cocoapods version is consistent, and there should not be a situation where rmtree clears cocoapods cache, so it is suspected that the problem is caused by multiple threads reading and writing the VERSION folder.

Problem Verification

• To verify my hypothesis, I wrote a demonstration code. By calling the ensure_matching_version method continuously in multiple threads for 10 times, I found that there are indeed cases where the version cannot be read.

require "thread"
require "pathname"
require "fileutils"

# Pod::VERSION "https://github.com/CocoaPods/CocoaPods/blob/df6cd9e0c51c54490be617a655fca80817c8b9b1/lib/cocoapods/gem_version.rb#L1-L5"
module Pod
    VERSION = "1.11.3".freeze unless defined?(Pod::VERSION)
end

# cocoapods cache dir "/Users/chengcong/Library/Caches/CocoaPods/Pods"
def root
    Pathname.new("/Users/chengcong/Documents/hy_code/scripts/test_temp/cocoapods")
end

# https://github.com/CocoaPods/CocoaPods/blob/df6cd9e0c51c54490be617a655fca80817c8b9b1/lib/cocoapods/downloader/cache.rb#L164-L172
def ensure_matching_version
    version_file = root + "VERSION" # 1.11.3

    version = version_file.read.strip if version_file.file?

    if version != Pod::VERSION && root.exist?
        puts "version #{version} != Pod::VERSION #{Pod::VERSION}"
        # root.rmtree
    end
    root.mkpath

    version_file.open("w") { |f| f << Pod::VERSION }
end

puts "start"
threads = []
10.times { threads << Thread.new { ensure_matching_version } }
threads.each(&:join)
puts "end"

• As shown in the log output, the version could not be read three times.

start
version  != Pod::VERSION 1.11.3
version  != Pod::VERSION 1.11.3
version  != Pod::VERSION 1.11.3
end

• I modified the source code of cocoapods and added logs to try to reproduce the scenario on the build machine. As a result, it was found that there is indeed a situation where the version cannot be read, which causes the version comparison to fail and the cache directory to be deleted.

Problem solved

• Refer to the locking method in [Cache] Ensure cache integrity on concurrent installations. #10018, and solve the problem by locking the reading and writing of the VERSION file.

def ensure_matching_version
    version = nil
    version_file = root + "VERSION"
    Cache.read_lock(version_file) { version = version_file.read.strip } if version_file.file?

    root.rmtree if version != Pod::VERSION && root.exist?
    root.mkpath

    Cache.write_lock(version_file) { version_file.open("w") { |f| f << Pod::VERSION } }
end

• Locking may have an impact on performance. In executing 1000 operations, it took 0.26 seconds without locking, while it took 2.01 seconds with locking.

[fabcz]

Strange empty cache VERSION file issue #11149

[fabcz]

temporary solution: cocoapods-fix-cache

[wkornewald]

We had such issues also with partially unfinished downloads being left in the cache and never getting auto-repaired. Maybe at some point this was caused by OOM issues.

I think just using a lockfile won't make this fully robust. Probably some journaling solution is needed which detects on the next run that the cache is left in a broken state and then it can delete/repair it.

[Hiuson]

It works very well, thank you so much. When will the CocoaPods author merge this fix? Damn.
很好用，非常感谢。cocoapod作者什么时候能把这个fix合了？册那
它用起來很好，非常感謝。CocoaPods 的作者何時能合併這個修復？該死。
Funciona muy bien, muchas gracias. ¿Cuándo el autor de CocoaPods fusionará esta corrección? Maldición.
Cela fonctionne très bien, merci beaucoup. Quand l'auteur de CocoaPods va-t-il fusionner ce correctif ? Zut.
Es funktioniert sehr gut, vielen Dank. Wann wird der CocoaPods-Autor diesen Fix einfügen? Verdammt.
非常に使いやすい、ありがとうございます。CocoaPodsの作者は、この修正をいつマージしますか？くそ。