Switch to 64 bit file and time APIs on GNU libc for 32bit systems #3175

snogge · 2023-03-29T12:17:40Z

Use the 64 bit types and APIs included in GNU libc also for 32-bit systems. These are the types and APIs used when you compile your C code against GNU libc headers with the preprocessor options -D_FILE_OFFSET_BITS=64 -D_TIME_BITS=64.

This is required for 2038 compatible software built for 32-bit systems.

This PR is not done, I need some guidance with finishing it.

Does it need some sort of configuration options to turn this on and off?
There is a conflict for some of the stat functions where two signatures link against the same symbol. For C this is not a problem, but we probably need to make struct stat and struct stat64 be different names for the same struct. I don't know how to do that.
It's possible that some types should be moved/split/joined in the file hierarchy. I'd appreciate some feedback on that, but I'm waiting on the answer to question 1 before moving forward.

I've done one commit for each modified type so it is easier to review the changes in isolation.

rustbot · 2023-03-29T12:17:45Z

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @JohnTitor (or someone else) soon.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

@rustbot author: the review is finished, PR author should check the comments and take action accordingly
@rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

snogge · 2023-03-29T13:24:08Z

Oops, left some debug/test commits in there. Removed now.

snogge · 2023-04-11T10:25:46Z

Hi @JohnTitor , do you have any feedback on this change?
Is there anything I can do to facilitate the review?
I'd appreciate any hints on solving question 2:

There is a conflict for some of the stat functions where two signatures link against the same symbol. For C this is not a problem, but we probably need to make struct stat and struct stat64 be different names for the same struct. I don't know how to do that.

I'm not a Rust developer and not sure how I can create that type alias.

bors · 2023-04-23T02:57:43Z

☔ The latest upstream changes (presumably #3218) made this pull request unmergeable. Please resolve the merge conflicts.

JohnTitor · 2023-04-24T17:40:21Z

Thanks for the PR! But I think this is a huge breaking change and we have to discuss it before reviewing the changes (or, even submitting a PR). Could you open an issue instead?

snogge · 2023-04-25T07:39:29Z

Sure, I'll do that.

snogge · 2023-04-25T08:03:44Z

I'm working on updating this PR to handle problem 2 mentioned above and to pass tests on more platforms. Expect an update later this week.

snogge · 2023-04-25T12:04:04Z

Sorry for the many pushes, that was meant only for my fork. Anyway, now the PR should handle the duplicate functions mentioned in problem 2, and also pass tests on more platforms.

bors · 2023-05-06T06:29:22Z

☔ The latest upstream changes (presumably #3237) made this pull request unmergeable. Please resolve the merge conflicts.

snogge · 2023-09-07T13:05:38Z

Just rebased on main. Passes all tests in main.yml and bors.yml workflows.
Still not very clean though, feedback on code style would be welcome.

I considered adding a cargo feature to control this, but as that would be braking the ABI I guess I shouldn't?

bors · 2023-11-12T11:53:17Z

☔ The latest upstream changes (presumably #3437) made this pull request unmergeable. Please resolve the merge conflicts.

plugwash · 2024-04-21T14:46:28Z

src/unix/mod.rs

    }

    // linux x32 compatibility
    // See https://sourceware.org/bugzilla/show_bug.cgi?id=16437
    pub struct timespec {
        pub tv_sec: time_t,
+        #[cfg(all(gnu_time64_abi, any(target_arch = "powerpc", target_arch = "mips")))]
+        __pad: i32,
        #[cfg(all(target_arch = "x86_64", target_pointer_width = "32"))]


Would it make more sense to use target_endian here, rather than hardcoding specific architectures?

snogge · 2024-04-22T11:36:08Z

I just removed the CI-changes I pushed by mistake....again.

plugwash · 2024-04-23T00:32:15Z

Hi, I applied a patch series based on this PR to the rust-libc package in Debian. Following Debian's descisions on which architectures to apply the time64 transition to I added a condition to disable the "gnu_time64_abi" setting on i386.

However, it seems that this patch series causes API breaks, even if the "gnu_time64_abi" config is not set. For example https://ci.debian.net/packages/r/rust-fs-at/testing/i386/45768200/#S6

snogge · 2024-04-23T08:16:55Z

I think the missing f_flags is actually a long-standing bug, the statfs struct in gnu/b32/x86/mod.rs does not have the f_flags member, but rather an extra item in the f_spare array. This was exposed by this patch when the statfs64 struct became an alias to the statfs struct. I'll just check the other archs as well and then update the PR.

plugwash · 2024-04-23T17:23:03Z

Yeah, it looks like the missing f_flags issue affects at least arm too.

snogge · 2024-04-24T08:57:11Z

I've filed a separate PR #3663 to fix the missing f_flags.
There are more problems with the statfs64 struct that has to be fixed in this PR, I'm working on fixing those.

The gnu_time64_abi option indicates that a 32-bit arch should use 64-bit time_t and 64-bit off_t and simliar file related types. 32-bit platforms are identified by CARGO_CFG_TARGET_POINTER_WIDTH, but x86_64-unknown-linux-gnux32 is a 64-bit platform and should not use gnu_time64_abi even if it has 32-bit pointers. riscv32 is a 32-bit platform but has always used 64-bit types for time and file operations. It should not use the gnu_time64_abi The default - all relevant platforms should use 64-bit time - can be overridden by setting the RUST_LIBC_TIME_BITS environment variable to 32. It can also be set to 64 to force gnu_time64_abi, or default which is the same as leaving it unset. The last two options are mainly useful for CI flows.

Set _TIME_BITS=64 and _FILE_OFFSET_BITS=64 preprocessor symbols for the C test code as for building the code.

Set the basic types correctly for gnu_time64_abi (_TIME_BITS=64 and _FILE_OFFSET_BITS=64).

Set the link names of relevant symbols to use be the same as when a C program is built against GNU libc with -D_TIME_BITS=64 and -D_FILE_OFFSET_BITS=64.

The actual values may be different on 32bit archs and glibc with 64-bit time.

Also add the F_[SG]ET*64 constants.

For 64 bit time on 32 bit linux glibc timeval.tv_usec is actually __suseconds64_t (64 bits) while suseconds_t is still 32 bits.

snogge · 2024-05-08T08:10:55Z

I've reconstructed the PR to support an environment variable RUST_LIBC_TIME_BITS which can be set to 32 to force 32-bit time on 32-bit platforms. The default, based on the cargo environment variables is 64-bit time for those platforms.
I've also modified the CI workflows to test these variants.

powerpc and mips wants 32 bits of padding before tv_nsec. arm and x86 wants 32bits of padding after tv_nsec.

Do not use gnu/b32/time*.rs for riscv32 and sparc. Add timex to gnu/b32/(riscv32|sparc)/mod.rs instead.

Move the stat struct from gnu/b32/mod.rs to gnu/b32/time*.rs For arm, mips, powerpc, riscv32, and x86, move stat64 to time32.rs and add an stat64 = stat alias to time64.rs. For sparc, do the same for statfs64 and statvfs64.

In Linux Tier1, run the tests for i686-unknown-linux-gnu with RUST_LIBC_TIME_BITS set to 32, 64, and default. In Linux Tier2, run the tests for arm-unknown-linux-gnueabihf and powerpc-unknown-linux-gnu with RUST_LIBC_TIME_BITS set to 32, 64, and default. Use RUST_LIBC_TIME_BITS=defaults for the other platforms. In Build Channels Linux, build the relevant platforms with RUST_LIBC_TIME_BITS unset and set to 32 and 64.

rustbot assigned JohnTitor Mar 29, 2023

rustbot added the S-waiting-on-review label Mar 29, 2023

snogge force-pushed the time_bits=64 branch from 78cec5c to c22d06d Compare March 29, 2023 13:23

JohnTitor added breakage-candidate S-blocked I-needs-decision and removed S-waiting-on-review labels Apr 1, 2023

snogge force-pushed the time_bits=64 branch 7 times, most recently from 5dfc8ed to 7d2c859 Compare April 24, 2023 13:36

snogge force-pushed the time_bits=64 branch from 7d2c859 to 6e00e1b Compare April 25, 2023 06:41

snogge mentioned this pull request Apr 25, 2023

Use the glibc 64bit time APIs to mitigate the Y2038 problem on 32bit platforms #3223

Open

snogge force-pushed the time_bits=64 branch 2 times, most recently from 5314484 to eb997cb Compare April 25, 2023 12:02

snogge force-pushed the time_bits=64 branch from eb997cb to 4d16795 Compare September 7, 2023 13:03

plugwash reviewed Apr 21, 2024

View reviewed changes

snogge force-pushed the time_bits=64 branch from 5dbc49d to 8db77a0 Compare April 22, 2024 11:35

snogge added 15 commits May 8, 2024 10:00

libc-test: Set the gnu time64 preprocessor symbols for all ptr32

5ceae08

Set _TIME_BITS=64 and _FILE_OFFSET_BITS=64 preprocessor symbols for the C test code as for building the code.

gnu: Handle basic file and time types for 32bit systems with time64

1105854

Set the basic types correctly for gnu_time64_abi (_TIME_BITS=64 and _FILE_OFFSET_BITS=64).

gnu: Use _TIME_BITS and _FILE_OFFSET_BITS=64 versions of glibc symbols

abf88c2

Set the link names of relevant symbols to use be the same as when a C program is built against GNU libc with -D_TIME_BITS=64 and -D_FILE_OFFSET_BITS=64.

linux: Set SO_TIMESTAMP* and SO_RCVTIMEO and SO_SNDTIMEO

ce68774

The actual values may be different on 32bit archs and glibc with 64-bit time.

linux: Set RLIM_INFINITY for mips GNU libc with _FILE_OFFSET_BITS=64

004a194

gnu: Update F_GETLK and F_GETLK64 for 64-bit time

f7f436b

gnu: Update F_SETLK and F_SETLKW for 64-bit time

9177430

Also add the F_[SG]ET*64 constants.

linux: Update struct input_event for glibc _TIME_BITS=64

bd73363

gnu: Handle timeval.tv_usec for glibc 64-bit time_t

22b82c3

For 64 bit time on 32 bit linux glibc timeval.tv_usec is actually __suseconds64_t (64 bits) while suseconds_t is still 32 bits.

gnu 32 powerpc: Rename powerpc.rs -> powerpc/mod.rs

1be7cdc

gnu: Update struct shmid_ds for 64-bit time

f3b3fe7

gnu: Update struct msqid_ds for 64-bit time

d641026

gnu: Update struct semid_ds for 64-bit time

b256f77

gnu: Update struct aiocb for _TIME_BITS=64

ce6efcd

snogge force-pushed the time_bits=64 branch 2 times, most recently from f139cbd to f63dfe2 Compare May 8, 2024 08:06

snogge force-pushed the time_bits=64 branch from f63dfe2 to 2df2142 Compare May 8, 2024 08:59

snogge added 4 commits May 8, 2024 10:59

gnu: Update struct timespec for _TIME_BITS=64

46dddab

powerpc and mips wants 32 bits of padding before tv_nsec. arm and x86 wants 32bits of padding after tv_nsec.

gnu: Move struct timex to gnu/b64 and gnu/b32/time*.rs

5ee74bb

Do not use gnu/b32/time*.rs for riscv32 and sparc. Add timex to gnu/b32/(riscv32|sparc)/mod.rs instead.

gnu: Update the stat* structs for 64-bit time

2389846

Move the stat struct from gnu/b32/mod.rs to gnu/b32/time*.rs For arm, mips, powerpc, riscv32, and x86, move stat64 to time32.rs and add an stat64 = stat alias to time64.rs. For sparc, do the same for statfs64 and statvfs64.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch to 64 bit file and time APIs on GNU libc for 32bit systems #3175

Switch to 64 bit file and time APIs on GNU libc for 32bit systems #3175

snogge commented Mar 29, 2023

rustbot commented Mar 29, 2023

snogge commented Mar 29, 2023

snogge commented Apr 11, 2023

bors commented Apr 23, 2023

JohnTitor commented Apr 24, 2023 •

edited

snogge commented Apr 25, 2023

snogge commented Apr 25, 2023

snogge commented Apr 25, 2023

bors commented May 6, 2023

snogge commented Sep 7, 2023

bors commented Nov 12, 2023

plugwash Apr 21, 2024

snogge May 8, 2024

snogge commented Apr 22, 2024

plugwash commented Apr 23, 2024

snogge commented Apr 23, 2024 •

edited

plugwash commented Apr 23, 2024

snogge commented Apr 24, 2024

snogge commented May 8, 2024

Switch to 64 bit file and time APIs on GNU libc for 32bit systems #3175

Are you sure you want to change the base?

Switch to 64 bit file and time APIs on GNU libc for 32bit systems #3175

Conversation

snogge commented Mar 29, 2023

rustbot commented Mar 29, 2023

snogge commented Mar 29, 2023

snogge commented Apr 11, 2023

bors commented Apr 23, 2023

JohnTitor commented Apr 24, 2023 • edited

snogge commented Apr 25, 2023

snogge commented Apr 25, 2023

snogge commented Apr 25, 2023

bors commented May 6, 2023

snogge commented Sep 7, 2023

bors commented Nov 12, 2023

plugwash Apr 21, 2024

Choose a reason for hiding this comment

snogge May 8, 2024

Choose a reason for hiding this comment

snogge commented Apr 22, 2024

plugwash commented Apr 23, 2024

snogge commented Apr 23, 2024 • edited

plugwash commented Apr 23, 2024

snogge commented Apr 24, 2024

snogge commented May 8, 2024

JohnTitor commented Apr 24, 2023 •

edited

snogge commented Apr 23, 2024 •

edited