Intermittent failure of the remove-disk Ceph test

[ChrisMacNaughton]

2022-09-28 17:30:15 DEBUG unit.ceph-osd/2.juju-log server.go:316 Reweighting OSD
2022-09-28 17:30:16 WARNING unit.ceph-osd/2.remove-disk logger.go:60 reweighted item id 5 name 'osd.5' to 0 in crush map
2022-09-28 17:30:16 DEBUG unit.ceph-osd/2.remove-disk logger.go:60 {"ok_to_stop":true,"osds":[5],"num_ok_pgs":147,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f","1.20","1.21","1.22","1.23","1.24","1.25","1.26","1.27","1.28","1.29","1.2a","1.2b","1.2c","1.2d","1.2e","1.2f","1.30","1.31","1.32","1.33","1.34","1.35","1.36","1.37","1.38","1.39","1.3a","1.3b","1.3c","1.3d","1.3e","1.3f","1.40","1.41","1.42","1.43","1.44","1.45","1.46","1.47","1.48","1.49","1.4a","1.4b","1.4c","1.4d","1.4e","1.4f","1.50","1.51","1.52","1.53","1.54","1.55","1.56","1.57","1.58","1.59","1.5a","1.5b","1.5c","1.5d","1.5e","1.5f","1.60","1.61","1.62","1.63","1.64","1.65","1.66","1.67","1.68","1.69","1.6a","1.6b","1.6c","1.6d","2.0","2.1","2.2","2.3","3.0","3.1","3.2","3.3","3.4","3.5","3.6","3.7","3.8","3.9","3.a","3.b","3.c","3.d","3.e","3.f","3.10","3.11","3.12","3.13","3.14","3.15","3.16","3.17","3.18","3.19","3.1a","3.1b","3.1c","3.1d","3.1e","3.1f","4.0"]}
2022-09-28 17:30:17 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 147 pgs currently mapped to them. 
2022-09-28 17:30:18 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 147 pgs currently mapped to them. 
2022-09-28 17:30:20 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 147 pgs currently mapped to them. 
2022-09-28 17:30:21 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 147 pgs currently mapped to them. 
2022-09-28 17:30:22 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 147 pgs currently mapped to them. 
2022-09-28 17:30:24 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 146 pgs currently mapped to them. 
2022-09-28 17:30:25 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 146 pgs currently mapped to them. 
2022-09-28 17:30:27 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 145 pgs currently mapped to them. 
2022-09-28 17:30:28 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 145 pgs currently mapped to them. 
2022-09-28 17:30:30 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 144 pgs currently mapped to them. 
2022-09-28 17:30:31 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 144 pgs currently mapped to them. 
2022-09-28 17:30:33 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 144 pgs currently mapped to them. 
2022-09-28 17:30:34 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 143 pgs currently mapped to them. 
2022-09-28 17:30:36 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 143 pgs currently mapped to them. 
2022-09-28 17:30:37 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 143 pgs currently mapped to them. 
2022-09-28 17:30:39 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 142 pgs currently mapped to them. 
2022-09-28 17:30:40 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 142 pgs currently mapped to them. 
2022-09-28 17:30:42 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 142 pgs currently mapped to them. 
2022-09-28 17:30:43 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 141 pgs currently mapped to them. 
2022-09-28 17:30:45 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 141 pgs currently mapped to them. 
... SNIP ...
2022-09-28 17:31:08 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 141 pgs currently mapped to them. 
2022-09-28 17:31:09 DEBUG juju.worker.uniter.remotestate watcher.go:666 update status timer triggered for ceph-osd/2
2022-09-28 17:31:10 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 141 pgs currently mapped to them. 
2022-09-28 17:31:20 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 141 pgs currently mapped to them. 
... SNIP ...
2022-09-28 17:35:15 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 141 pgs currently mapped to them. 
2022-09-28 17:35:16 WARNING unit.ceph-osd/2.remove-disk logger.go:60 Error EBUSY: OSD(s) 5 have 141 pgs currently mapped to them. 
2022-09-28 17:35:16 INFO unit.ceph-osd/2.juju-log server.go:316 DEPRECATION WARNING: Function function_fail is being removed : moved to action_fail()
2022-09-28 17:35:17 DEBUG juju.worker.uniter.operation executor.go:132 committing operation "run action 106" for ceph-osd/2

[sabaini]

Another data point fwiw
https://review.opendev.org/c/openstack/charm-ceph-osd/+/878253/11#message-d5ae8b7b66d2d1b814256d59434327c5da1b1d91
https://openstack-ci-reports.ubuntu.com/artifacts/f85/878253/11/check/jammy-zed/f85f698/