Upgrade macOS version and Xcode version for Mac machines #1431

meteorcloudy · 2022-09-12T09:38:05Z

macOS version should be upgraded to 12.5.1
- need to fix Bazel test suites broken due to python no longer available.
Xcode version should be upgraded to 13.4.1, we can provide multiple Xcode versions.

The text was updated successfully, but these errors were encountered:

meteorcloudy · 2022-09-12T15:13:17Z

Failing test on macOS 12.5.1: https://buildkite.com/bazel-testing/bazel-bazel/builds/8481#018330f9-71a1-4bc5-a8be-942f5c5ac3f9

BalestraPatrick · 2023-01-30T14:14:14Z

@meteorcloudy Is there any way we can install Xcode 14? It would unblock some tests in rules_apple. #1348 (comment)

fweikert · 2023-01-30T14:15:37Z

I'm looking into it

fweikert · 2023-01-30T20:17:59Z

I'm in the process of installing both 13.4.1 and 14.2. However, some of the machines require an OS update, too. I hope that the process will be finished by tomorrow EOD.

BalestraPatrick · 2023-02-06T10:18:54Z

Hey @fweikert! Thanks for doing this. I assume the rollout has been completed but the default version hasn't been changed, so we need to add an override?

I also noticed some failures starting Tuesday last week in our scheduled builds on our master branch: https://buildkite.com/bazel/rules-apple-darwin/builds?branch=master

It looks like the main issue is the that the tvOS SDK isn't installed anymore. We should likely make sure that every machine has all SDKs installed (with something like xcodebuild -downloadAllPlatforms for example). The error I'm seeing is: The operation couldn’t be completed. Failed to locate any simulator runtime matching options: { "com.apple.platform.appletvsimulator"

BalestraPatrick · 2023-02-06T14:32:27Z

We also realized that the we have a Activating Xcode 13.0... step which didn't fail, but subsequently we see DEBUG: /private/var/tmp/_bazel_buildkite/e6a8ceab815440f7ffea5ed855e0655d/external/bazel_tools/tools/osx/xcode_configure.bzl:243:14: No default Xcode version is set with 'xcode-select'; picking ':version14_2_0_14C18' in the log. This is likely because Xcode 13 can't run on macOS Ventura.

fweikert · 2023-02-06T15:32:08Z

Hey,
yeah, unfortunately the infra update broke some of our tests (https://github.com/bazelbuild/bazel/labels/macos-infra-update).

Technically our CI should now default to Xcode 14.2 on Ventura, but apparently there is a bug that I still need to find. Can you explicitly specify 14.2 in your test config?

BalestraPatrick · 2023-02-06T15:45:10Z

I guess we could (provided the tvOS simulator runtimes are installed for that specific version), but not sure what other stuff will break in our repos.

brentleyjones · 2023-02-06T15:47:39Z

Is there no way to specify that we want to run on Monterey? Or is the whole fleet assumed to be Xcode 14.2/Ventura now?

fweikert · 2023-02-06T16:14:11Z

It's a bit tricky right now - all the iMacs run Ventura (platform macos), while all the MacStudios still run Monterey (macos_arm64). Our Monterey machines have Xcode versions 13.0, 13.4.1 and 14.2 installed.

fweikert · 2023-02-06T16:18:46Z

To elaborate a bit on the background: Right now CI fleet management is quite painful since we have to manually maintain 40 Macs of different generations (trash can Mac Pros, iMac Pros, MacStudios).

We're working on virtualizing the fleet, which would allow us to offer multiple different OS versions at the same time.

bazelbuild#1542 relied on platform.mac_ver() to detect the MacOS version, which did not work as expected. Consequently, the previous PR led the CI to activate an incorrect version of Xcode. This behavior has been fixed by this change. Related to bazelbuild#1431.

#1542 relied on platform.mac_ver() to detect the MacOS version, which did not work as expected. Consequently, the previous PR led the CI to activate an incorrect version of Xcode. This behavior has been fixed by this change. Related to #1431. Users can still request a specific Xcode version in their CI config.

BalestraPatrick · 2023-02-07T16:01:14Z

We definitely have tests failing on arm64, so switching directly to Mac Studios might be tricky in the short-term. I'm seeing messages related to the simulator runtimes being a bit off on those machines as well:

Failed to find a suitable device for the type IBSimDeviceTypeiPad2x (com.apple.dt.Xcode.IBSimDeviceType.iPad-2x) with runtime iOS 15.0 (15.0 - 19A339) - com.apple.CoreSimulator.SimRuntime.iOS-15-0 (Failure reason: Failed to create SimDeviceSet at path /Users/buildkite/Library/Developer/Xcode/UserData/IB Support/Simulator Devices. You'll want to check the logs in ~/Library/Logs/CoreSimulator to see why creating the SimDeviceSet failed.): Failed to initialize simulator device set. (Failure reason: Failed to subscribe to notifications from CoreSimulatorService.): Unable to determine SimDeviceSet, set_path=(null): Failed to initialize simulator device set. (Failure reason: Allocation or initialization failed.)

I think the best path forward would making sure the simulator runtimes and SDKs are installed properly on all machines, and then we can proceed to see which option gets us back to green asap (I'm thinking Xcode 14.0 running on Ventura at the moment).

meteorcloudy · 2023-02-08T08:12:48Z

making sure the simulator runtimes and SDKs are installed properly on all machines

I guess neither me or Florian is very familiar with this. Can you give some instructions on how to do this?

@fweikert Can you try to run xcrun simctl list and check what's the output?

BalestraPatrick · 2023-02-08T12:45:31Z

I think following this guide should be a good start: https://developer.apple.com/documentation/xcode/installing-additional-simulator-runtimes

Running xcodebuild -downloadAllPlatforms should at least provide all the SDKs that we need (watchOS, tvOS) that are optional in Xcode 14 and not bundled with Xcode anymore by default.

fweikert · 2023-02-08T13:48:20Z

I ran xcodebuild -downloadAllPlatforms on all iMacs. I can see that we got tvOS 16.1 and watchOS 9.1:

% xcodebuild -showsdks               
DriverKit SDKs:
	DriverKit 22.2                	-sdk driverkit22.2

iOS SDKs:
	iOS 16.2                      	-sdk iphoneos16.2

iOS Simulator SDKs:
	Simulator - iOS 16.2          	-sdk iphonesimulator16.2

macOS SDKs:
	macOS 13.1                    	-sdk macosx13.1
	macOS 13.1                    	-sdk macosx13.1

tvOS SDKs:
	tvOS 16.1                     	-sdk appletvos16.1

tvOS Simulator SDKs:
	Simulator - tvOS 16.1         	-sdk appletvsimulator16.1

watchOS SDKs:
	watchOS 9.1                   	-sdk watchos9.1

watchOS Simulator SDKs:
	Simulator - watchOS 9.1       	-sdk watchsimulator9.1

However, xcrun simctl list still shows many "Unavailable" entries such as the aforementioned com.apple.CoreSimulator.SimRuntime.iOS-15-0.

BalestraPatrick · 2023-02-08T15:23:09Z

I think that's good enough, we might not need other runtimes that are not bundled with Xcode by default (I don't think we run tests on lower iOS versions in our repos). Looks like I was able to get a rules_apple PR that switches us to Xcode 14.2 on the iMacs green now:

I only noticed one issue on bk-imacpro-6 (which I didn't see on other iMacs running the same build):

    Failure Reason: Failed to spawn AssetCatalogSimulatorAgent on Apple Watch Series 7 (45mm) (9BAC5919-45D6-4DDD-AB6A-AAAF73F5FB43, (null), Shutdown)

It looks like the "Apple Watch Series 7 (45mm)" simulator is not correctly created. If you run xcrun simctl list and you see that device on that machine, maybe it was just a temporary error. Otherwise something might be different on that machine vs the other iMacs (for example this build on bk-imacpro-19 was successful).

fweikert · 2023-02-08T15:40:50Z

I can see Apple Watch Series 7 (45mm) (com.apple.CoreSimulator.SimDeviceType.Apple-Watch-Series-7-45mm) under "Devices" on bk-imacpro-6. However, the watchOS 9.1 runtime contains an entry with a different value: Apple Watch Series 7 (45mm) (7FF9613B-A581-4703-B2D9-DF589ACAB733) (Shutdown)

Looks like CI ~is~ was broken because the [CI fleet was upgraded to Xcode 14.2](bazelbuild/continuous-integration#1431 (comment)) and some machines were missing the watchOS and tvOS SDKs. Now that the iMacs are properly set up, we should be able to simply use Xcode 14.2. We can't use Xcode 13.0 anymore because it doesn't run on macOS Ventura (which all the x86_64 machines are on after the upgrade). Future tasks might include adding support for arm64 in our tooling so that we can switch to run on Mac Studios instead of iMacs.

BalestraPatrick · 2023-02-13T10:21:53Z

@fweikert I'm still seeing issues with the Apple Watch simulators. If I run xcrun simctl list devices available on a build agent, I only get iOS devices:

== Devices ==
-- iOS 16.2 --
    iPhone 14 (438498C7-2E9A-4A4A-9CC7-5DED49FAC8C4) (Shutdown)
    iPhone 14 Plus (BB68AFF0-037F-4B23-9A0B-2C5DB99D743A) (Shutdown)
    iPhone 14 Pro Max (94EE6ADE-C7B7-4E76-9DFD-78B8B9229449) (Shutdown)
    iPad Air (5th generation) (710C70AD-5DE4-4A57-846F-132519AB808D) (Shutdown)
    iPad Pro (11-inch) (4th generation) (9F4F019B-BBDA-4C8D-84B0-73649107A41F) (Shutdown)
-- tvOS 16.1 --
-- watchOS 9.1 --

Maybe since the watchOS and tvOS simulators were installed post-Xcode install, we need to create some defaults manually?

fweikert · 2023-02-13T10:37:32Z

Which machine was that?

BalestraPatrick · 2023-02-13T12:13:14Z

My latest build is here. Reproduced it on imacpro-19, imacpro-5 and imacpro-16.

fweikert · 2023-04-21T17:48:35Z

After some more debugging I have even more questions:

https://buildkite.com/bazel/rules-apple-darwin/builds/7112#01878136-760a-4e5f-893b-6bb3acaf55c2 -> Why does it need to spawn something on a device during the ´build´(not ´test´) step?
In the Xcode UI I went to "Devices & Simulators" -> "Simulators" and only see iPhone/iPad/Apple TV entries, but no watches. Is this WAI? I do see the watchOS 9.1 runtime and multiple watch devices when running ´xcrun simctl list´
I ran some targets thousands of times, and only observed the error on the very first run. Are there some additional setup / tear down steps we should execute before and after each run? We do not reboot the iMacs between test runs and only try to reconstruct a clean directory tree.

brentleyjones · 2023-04-21T17:51:41Z

Why does it need to spawn something on a device during the ´build´(not ´test´) step?

A simulator is used internally by Xcode for asset compilation, because it needs to run the tool "on device".

BalestraPatrick · 2023-04-21T18:07:21Z

Regarding 3, it sounds like the result is cached on following runs, so the action doesn’t even run. You should try a bazel clean or bazel clean —expunge to make sure the action isn’t picked up from the cache on the following run.

fweikert · 2023-04-21T21:17:16Z

Yeah, I ran bazel clean --expunge between runs, but subsequent runs still succeeded.

BalestraPatrick · 2023-04-22T08:27:44Z

Regarding 2, I checked locally and I also don't see watchOS devices as separate, but they should be listed in the iOS devices and show up as "Paired Watches".

BalestraPatrick · 2023-04-29T07:38:57Z

We're continuing similar errors and this is issue is blocking several PRs that we would like to merge. A new log that is very similar to the previous one (permissions issues) can be found here: https://buildkite.com/bazel/rules-apple-darwin/builds/7156#0187c3d3-c92c-4f01-93c6-086ca39ebe7d

  bazel-out/darwin_x86_64-opt-exec-ST-9bbe478fa10f/bin/external/build_bazel_rules_apple/tools/xctoolrunner/xctoolrunner actool --compile '[ABSOLUTE]bazel-out/applebin_ios-ios_i386-opt-ST-b807712c233f/bin/app/imessage_ext-intermediates/xcassets' --platform iphonesimulator --minimum-deployment-target 12.0 --compress-pngs --product-type com.apple.product-type.app-extension.messages --sticker-pack-identifier-prefix my.bundle.id.extension.sticker-pack. --app-icon app_icon --target-device iphone --output-partial-info-plist '[ABSOLUTE]bazel-out/applebin_ios-ios_i386-opt-ST-b807712c233f/bin/app/imessage_ext-intermediates/xcassets-info.plist' '[ABSOLUTE]external/build_bazel_rules_apple/test/testdata/resources/message_ext_app_icon_ios.xcassets')
# Configuration: 8829be6d4d326c875ebae514ad6e8aa8d5e0de9b185c0c8f9f8253b8bbb6dec3
# Execution platform: @local_config_platform//:host
        Failure Reason: You donâ€™t have permission.
        Recovery Suggestion: To view or change permissions, select the item in the Finder and choose File > Get Info.
    Failure Reason: Failed to spawn AssetCatalogSimulatorAgent on IBSimDeviceTypeiPad3x (1E1EA317-FF08-44A5-8695-819BDC05023E, iOS 16.2, Shutdown)
        Underlying Errors:
            Failure Reason: Operation not permitted
    Recovery Suggestion: To view or change permissions, select the item in the Finder and choose File > Get Info.
        Description: You donâ€™t have permission to save the file â€œ1E1EA317-FF08-44A5-8695-819BDC05023Eâ€� in the folder â€œCoreSimulatorâ€�.

If the Bazel invocations outside buildkite worked fine, I suspect this is an issue in the way buildkite and it's agent's permissions are set up. Would it be possible to tweak those so we can try to get to the bottom of this?

fweikert · 2023-05-02T14:54:10Z

The permissions error looks fixable.

Re other error: Any objection to upgrading to Xcode 14.3?

BalestraPatrick · 2023-05-03T10:49:59Z

I don't have any objections from my side, I'm not aware of anything that would break in 14.3. Is this upgrade to try and see if the errors are fixed in newer versions of Xcode or for some other reason?

meteorcloudy · 2023-05-03T12:45:38Z

Both, looks like we need Xcode 14.3 for TensorFlow: #1617

fweikert · 2023-05-17T11:56:56Z

Unfortunately upgrading to 14.3 led to other problems: bazelbuild/bazel#18278
We've reverted back to 14.2 for now and will look into alternatives.

keith · 2023-05-17T18:32:51Z

looks like the back and forth might have affected the platform setup https://buildkite.com/bazel/bcr-presubmit/builds/1309#01882af9-ad72-4000-90e9-2c93f350e1d1

keith · 2023-05-17T18:44:24Z

maybe a new issue too in that log:

xcodebuild: error: The authorization was denied since no user interaction was possible.

fweikert · 2023-05-23T16:33:30Z

With bazelbuild/bazel#17451 and bazelbuild/bazel#18460 we can hopefully switch to Xcode 14.3 again soon.

BalestraPatrick · 2023-06-11T18:10:35Z

Any news on this? We're unable to merge certain PRs in rules_apple due to this CI issue for the last 2 months. bazelbuild/rules_apple#1942

BalestraPatrick · 2023-08-12T16:16:51Z

@fweikert @meteorcloudy Is there any way we can install Xcode 14.3 on all machines? Using it would fix the failures in the PR above and decrease flakiness. As you can see in this build, the Xcode 14.3 builds passed but not on 14.2.

meteorcloudy · 2023-08-17T11:09:03Z

@BalestraPatrick I'm installing Xcode 14.3 on all Mac machines now.

meteorcloudy · 2023-08-17T12:03:28Z

Xcode 14.3 should be available on all Mac machines now!

BalestraPatrick · 2023-08-17T12:05:59Z

@meteorcloudy Thank you! It looks like we also need all the new SDKs for that version: https://buildkite.com/bazel/rules-apple-darwin/builds/7734#018a035a-690b-4dde-9a93-8e392b924528

We need watchOS 9.4 and tvOS 16.4. I think running xcodebuild -downloadAllPlatforms should do that.

meteorcloudy · 2023-08-17T12:06:31Z

Let me try that

meteorcloudy · 2023-08-17T12:22:21Z

OK, I'm running sudo xcode-select --switch /Applications/Xcode14.3.app && xcodebuild -downloadAllPlatforms on all imac pro machines.

Some succeeded with:

some failed with:

meteorcloudy · 2023-08-17T12:24:44Z

Rerunning the command ended up with All requested platforms are downloaded Can you try the presubmit again?

brentleyjones · 2023-08-17T13:44:43Z

bazelbuild/rules_apple#2078 (comment)

acecilia · 2023-08-17T18:16:01Z

I got also the duplicated of .... It has been a while, but I believe it refers to something with simulators. I recall I solved it by one of below (dont recall exactly which one):

xcrun simctl delete all
xcrun simctl runtime delete <id>

And then run xcodebuild -downloadAllPlatforms again, this time should work

meteorcloudy · 2023-08-18T08:33:20Z

@acecilia Thanks! In the end, I got All requested platforms are downloaded on all machines. I'll try your suggestions if there is actually something wrong with the SDKs installation.

fweikert · 2023-08-22T12:04:52Z

Did the update to 14.3 fix the problem with not being to spawn Watch 7 (45mm) devices?

brentleyjones · 2023-08-22T13:29:17Z

No: https://bazelbuild.slack.com/archives/C04DFUBQGSU/p1692285015357949?thread_ts=1692283649.850709&cid=C04DFUBQGSU

bazelbuild#1542 relied on platform.mac_ver() to detect the MacOS version, which did not work as expected. Consequently, the previous PR led the CI to activate an incorrect version of Xcode. This behavior has been fixed by this change. Related to bazelbuild#1431. Users can still request a specific Xcode version in their CI config.

This is a temporary change whose sole purpose is to allow me to target individual Macs in order to debug problems such as those outlined in bazelbuild#1431.

meteorcloudy self-assigned this Sep 12, 2022

meteorcloudy mentioned this issue Sep 13, 2022

Install Android NDK r25b #1434

Merged

meteorcloudy mentioned this issue Jan 19, 2023

jpsim/SourceKitten@0.34.0 bazelbuild/bazel-central-registry#374

Merged

fweikert self-assigned this Jan 30, 2023

BalestraPatrick mentioned this issue Feb 6, 2023

[CI] Use Xcode 14.2 bazelbuild/rules_apple#1820

Merged

fweikert mentioned this issue Feb 6, 2023

Fix MacOS version detection #1546

Merged

fweikert mentioned this issue Feb 13, 2023

Provide Xcode 13.1 and above on macOS machines #1348

Closed

fweikert mentioned this issue Jun 27, 2023

rules_swift is failing with Bazel@Head bazelbuild/rules_swift#1063

Closed

BalestraPatrick mentioned this issue Aug 17, 2023

Upgrade to Xcode 14.3 bazelbuild/rules_apple#2078

Merged

Upgrade macOS version and Xcode version for Mac machines #1431

Upgrade macOS version and Xcode version for Mac machines #1431

Comments

meteorcloudy commented Sep 12, 2022

meteorcloudy commented Sep 12, 2022

BalestraPatrick commented Jan 30, 2023

fweikert commented Jan 30, 2023

fweikert commented Jan 30, 2023

BalestraPatrick commented Feb 6, 2023

BalestraPatrick commented Feb 6, 2023

fweikert commented Feb 6, 2023 • edited

BalestraPatrick commented Feb 6, 2023

brentleyjones commented Feb 6, 2023

fweikert commented Feb 6, 2023 • edited

fweikert commented Feb 6, 2023

BalestraPatrick commented Feb 7, 2023

meteorcloudy commented Feb 8, 2023

BalestraPatrick commented Feb 8, 2023

fweikert commented Feb 8, 2023

BalestraPatrick commented Feb 8, 2023

fweikert commented Feb 8, 2023

BalestraPatrick commented Feb 13, 2023

fweikert commented Feb 13, 2023

BalestraPatrick commented Feb 13, 2023

fweikert commented Apr 21, 2023

brentleyjones commented Apr 21, 2023

BalestraPatrick commented Apr 21, 2023

fweikert commented Apr 21, 2023 • edited

BalestraPatrick commented Apr 22, 2023

BalestraPatrick commented Apr 29, 2023

fweikert commented May 2, 2023

BalestraPatrick commented May 3, 2023 • edited

meteorcloudy commented May 3, 2023

fweikert commented May 17, 2023

keith commented May 17, 2023

keith commented May 17, 2023

fweikert commented May 23, 2023

BalestraPatrick commented Jun 11, 2023

BalestraPatrick commented Aug 12, 2023

meteorcloudy commented Aug 17, 2023

meteorcloudy commented Aug 17, 2023

BalestraPatrick commented Aug 17, 2023

meteorcloudy commented Aug 17, 2023

meteorcloudy commented Aug 17, 2023

meteorcloudy commented Aug 17, 2023 • edited

brentleyjones commented Aug 17, 2023

acecilia commented Aug 17, 2023 • edited

meteorcloudy commented Aug 18, 2023

fweikert commented Aug 22, 2023

brentleyjones commented Aug 22, 2023 • edited

fweikert commented Feb 6, 2023 •

edited

fweikert commented Feb 6, 2023 •

edited

fweikert commented Apr 21, 2023 •

edited

BalestraPatrick commented May 3, 2023 •

edited

meteorcloudy commented Aug 17, 2023 •

edited

acecilia commented Aug 17, 2023 •

edited

brentleyjones commented Aug 22, 2023 •

edited