puzzling ingestion of item "6bb30310-baa2-47f9-8ebc-5246370db71f" #311
Comments
|
there are some other items exhibiting the same patterns of behaviours. |
|
I think there are five situations we've observed that were recorded with
The first one is the most common and the one you've described. I can take a look at that one in more detail. |
|
To assist your troubleshooting, I dumped the current disk usage on chard:
***@***.*** pushmi_pullyu]# df -lh
Filesystem Size Used Avail Use% Mounted on
devtmpfs 4.8G 0 4.8G 0% /dev
tmpfs 4.9G 0 4.9G 0% /dev/shm
tmpfs 4.9G 932K 4.9G 1% /run
tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
/dev/mapper/system-root 37G 6.8G 30G 19% /
/dev/sda1 509M 316M 194M 63% /boot
tmpfs 984M 0 984M 0% /run/user/0
The line in red is where /var/pushmi_pullyu/tmp is located. It was
allocated 37GB, 6.8GB was used, and currently, 30GB is available.
…On Tue, Apr 11, 2023 at 4:15 PM Tricia Jenkins ***@***.***> wrote:
I think there are five situations we've observed that were recorded with
MaxDepositAttemptsReached
1. undefined method `strip' for false:FalseClass
<https://app.rollbar.com/a/ualbertalib/fix/item/pushmi_pullyu/101>
2. No space left on device @ io_write - /var/pushmi_pullyu/tmp/work/
<https://rollbar.com/ualbertalib/pushmi_pullyu/items/102>
3. No space left on device - sendfile
<https://rollbar.com/ualbertalib/pushmi_pullyu/items/103>
4. No such file or directory @ rb_sysopen - tmp/work/
<https://rollbar.com/ualbertalib/pushmi_pullyu/items/104>
5. Authentication failed with response code 403
OpenStack::Exception::ExpiredAuthToken
<https://rollbar.com/ualbertalib/pushmi_pullyu/items/105>
The first one is the most common and the one you've described. I can take
a look at that one in more detail.
—
Reply to this email directly, view it on GitHub
<#311 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYUCLC2JPYFK7USUHID3BTXAXJXHANCNFSM6AAAAAAW22GD2U>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
Thanks | Merci
Henry Zhang | Henry Zhang
Senior Digital Preservation Specialist | spécialiste principal de la
préservation numérique
|
|
Saving this to come back to later. Somehow gets to this |
|
|
Is there a difference between writing and creating? Are these all instances of one or the other? |
|
@pgwillia if it is desired to simulate OpenStack 403, is it possible to use a fake token in the development stack so that the request for a resource will be forbidden? Then we might be able to trace what is the http initheader's key, and why its value is a boolean false instead of a string. To troubleshoot, I added a line to log the key: value pair in the code of the test server (which is not a good idea to mess up with third party code) as shown below (the line is bold and italicised), and let it run for an extended period of time, unfortunately, the error never happened during that period of time. After that, I removed that line from the third party code. def initialize_http_header(initheader) Similarly, we can simulate a failed access to Redis queue, and any other factors that might cause problems of PMPY to download and write |
The first attempt to save that item started on April 9, 2023, then it throw some errors, as shown by the log file below:
Many re-tries were attempted without success, till April 11, 2023, it finally got deposited into Swift
The logs file for the above *** pushmi_pullyu.log-20230410.gz ***, and pushmi_pullyu.log (on April 11). Tomorrow, it will probably be rotated into pushmi_pullyu.log-20230412.gz.
Proof of that the item is only preserved into Swift once:
proof of that the item was attempted many times, (180 log entries for this item)
The text was updated successfully, but these errors were encountered: