Support XDG env variables on Android or use default XDG directories on Android/Termux

[Blocked]

Previous issue #62 with fix #63 messes up all config directories.

Related:

mborsetti/webchanges#28
pypa/pip#10887

The fixed config directory on Termux:

-- platformdirs 2.4.2.dev4+gbd70a11 --
-- app dirs (with optional 'version')
user_data_dir: /data/data/com.termux/files/MyApp/1.0
user_config_dir: /data/data/com.termux/shared_prefs/MyApp/1.0
user_cache_dir: /data/data/com.termux/cache/MyApp/1.0
user_state_dir: /data/data/com.termux/files/MyApp/1.0
user_log_dir: /data/data/com.termux/cache/MyApp/1.0/log
user_documents_dir: /storage/emulated/0/Documents
user_runtime_dir: /data/data/com.termux/cache/MyApp/1.0/tmp
site_data_dir: /data/data/com.termux/files/MyApp/1.0
site_config_dir: /data/data/com.termux/shared_prefs/MyApp/1.0

No app should ever store anything outside $HOME. The correct config directory should reference $HOME first and then the config directory:

$HOME: /data/data/com.termux/files/home
$PREFIX: /data/data/com.termux/files/usr
Config: /data/data/com.termux/files/home/.config

Data structure official reference:

https://github.com/termux/termux-packages/wiki/Termux-file-system-layout

So, Termux dirs should be:

-- platformdirs 2.4.2.dev4+gbd70a11 --
-- app dirs (with optional 'version')
user_data_dir: /data/data/com.termux/files/home/.local/share/MyApp/1.0
user_config_dir: /data/data/com.termux/files/home/.config/MyApp/1.0
user_cache_dir: /data/data/com.termux/files/home/.cache/MyApp/1.0
user_state_dir: /data/data/com.termux/files/home/.local/state/MyApp/1.0
user_log_dir: /data/data/com.termux/files/home/.cache/MyApp/1.0/log
user_documents_dir: /data/data/com.termux/files/home/storage/Documents
user_runtime_dir: /data/data/com.termux/files/usr/var/run/MyApp/1.0/tmp
site_data_dir: /data/data/com.termux/files/usr/share/MyApp/1.0
site_config_dir: /data/data/com.termux/files/usr/etc/MyApp/1.0

[YariKartoshe4ka]

First of all Termux is Andorid app and all Android apps has directories specificated by Android SDK. We just took the values of such methods as getCacheDir as a standard and turn them back without referring to the sdk.

The use of the XDG specification in applications other than Termux is unacceptable. Also I'm not sure how this approach can break current applications and why it cannot be used

[Blocked]

For all intents and purposes, termux is a full unix environment. Moving app specific directories outside their default directories has other issues like inability to access the data through Android storage access framework as termux only exports the $HOME directory to other apps/file manager. All other unix packages other than python/pip packages store their application data in this way. This has always been the default even with python/pip packages that don't use platformdirs.

Termux Packaging policy: 7 says:

7. Patches: hardcoded path references

Software often relies on paths defined by Filesystem Hierarchy Standard:

• /bin
• /etc
• /home
• /run
• /sbin
• /tmp
• /usr
• /var

These paths do not exist in Termux and have been replaced by prefixed
equivalents. Termux installation prefix is

/data/data/com.termux/files/usr

and can be considered as virtual rootfs.

Home directory is stored outside of prefix:

/data/data/com.termux/files/home

Don't hardcode home and prefix, use shortcuts @TERMUX_HOME@ and
@TERMUX_PREFIX@ respectively. Patch files are preprocessed before
being applied.

Directories /run and /sbin should be replaced by
@TERMUX_PREFIX@/var/run and @TERMUX_PREFIX@/bin respectively.

It would be great, if you follow established norms rather than creating new ones.

[YariKartoshe4ka]

I don't create new norms 👍

In addition to termux, there are still a large number of applications using python

• Apps were built with p4a or beeware
• Python IDEs (PyDroid, QPython etc)

They all need to define directories, but they don't have the same file hierarchy as termux, so summarizing all cases, it seems to me more rational to use Android (Google) standards rather than add exceptions for each application.

if SAF is your only problem, you can always make a link from home to the directories you need:

$ pwd
/data/data/com.termux/files/home
$ ln -s ../../shared_prefs shared_prefs

P.S. Android support was added in 2.1.0, not in #63

[Blocked]

There are two scenarios:

1. Python pkg is used to build android apps(p4a)
2. Python pkg is used inside a android app(development environment like Termux/Pydroid)

In scenario 1, The pkg can be used on any platform like unix or mac to create apks for use on android. They have their own mechanism to determine the needed android storage paths. Platformdirs in this case, has nothing to do with android, but with the actual development platform(say unix).

In scenario 2, a android app is used to simulate a full linux distro. I would guess more than 95% python packages used in android are only used in termux. Other use cases(like scenario 1 are extremely rare). Even developmental environmental packages you mentioned like pydroid, actually used termux shell as base. All these apps have accessible os.environ to get exported shell variables. It's only logical to consider these environments full unix environments and use XDG specs. At the very least, if a XDG variable is available, it should be checked and retrieved and used.

Other than SAF:

• The current platformdirs tries to treat android not as a unix distro but as a single app
• pip and other packages, which used appdirs or otherwise used XDG directories in termux.
• New packages will lose existing config
• Backups based on old configs won't get restored
• The config directory and cache directory(/data/data/com.termux/shared_prefs and /data/data/com.termux/cache/) currently in use are for the app (termux) cache and config itself. For eg, configurations related to height of the shell or whether to use full screen or what extra keyboard keys should be shown etc(List here). This shouldn't be used by internal packages.
• Shared preferences use expensive java methods to write. That directory shouldn't be polluted by sub-packages(though they may not use the same expensive methods).

This is creating a new norm by treating all packages inside termux use the same config directories as the parent app and ignores a parent-child/master-slave relationship. At the very least, XDG shell variables should be respected.

[YariKartoshe4ka]

In scenario 1, The pkg can be used on any platform like unix or mac to create apks for use on android ... Platformdirs in this case, has nothing to do with android, but with the actual development platform(say unix).

I think you are confusing something. Platformdirs, of course, directly executed on android, becuse it was packed to APK via p4a or beeware so APKs buit via these packages should be taken into account.

They [p4a] have their own mechanism to determine the needed android storage paths.

Yes, it has, but methods only determine the path to the internal/external storage, not to specific dirs, e.g. a data/cache and other dirs. Also, android module (with pyjnius) can be replaced by more fast, lightweight platformdirs, not requiring compilation

In scenario 2, a android app is used to simulate a full linux distro. I would guess more than 95% python packages used in android are only used in termux. Other use cases(like scenario 1 are extremely rare). Even developmental environmental packages you mentioned like pydroid, actually used termux shell as base. All these apps have accessible os.environ to get exported shell variables. It's only logical to consider these environments full unix environments and use XDG specs. At the very least, if a XDG variable is available, it should be checked and retrieved and used.

Yes, they may be less popular than Termux, but 1+ million downloads on each (Pydroid, QPython) should say something.
Pydroid didn't use termux shell as base and similar IDEs not is used to simulate a full linux distro and they are not required to have XDG env vars

• The current platformdirs tries to treat android not as a unix distro but as a single app
• pip and other packages, which used appdirs or otherwise used XDG directories in termux.
• New packages will lose existing config
• Backups based on old configs won't get restored

You are talking about backward compatibility, but what kind of compatibility can we talk about if Android was not considered in appdirs. All this is on the shoulders of developers moving from appdirs to platformdirs

The config directory and cache directory(/data/data/com.termux/shared_prefs and /data/data/com.termux/cache/) currently in use are for the app (termux) cache and config itself. For eg, configurations related to height of the shell or whether to use full screen or what extra keyboard keys should be shown etc(List here). This shouldn't be used by internal packages.

This is creating a new norm by treating all packages inside termux use the same config directories as the parent app and ignores a parent-child/master-slave relationship.

Why packages cannot use this directory? All packages data contains in separated folders, and won't conflict with termux xml preferences. Sub-foldering each package, i think, follow parent-child/master-slave relationship

Shared preferences use expensive java methods to write. That directory shouldn't be polluted by sub-packages(though they may not use the same expensive methods).

Of course, it can be used, and even polluted, and this won't affect performance in any way. Again, this won't conflict with the Termux settings

Summing up, we can say that this approach is a problem for backward compatibility with appdirs, but it is a working for every method of porting python to Android

[Blocked]

The problem I see here is platformdirs gives undue preference to scenario 1(python built apks) than scenario 2(shell apk using python and other unix tools), but scenario 2 is disproportionately more common.

Even if you want to support scenario 1, the use of shared_preferences for config and cache for cache is completely arbitrary. Hard coded paths are never considered wise in any android development. While the code uses Context().getFilesDir(), it fails to call getCacheDir() or getSharedPreferences(), but adds known strings to the path retrieved from getFilesDir(). What if the cache and config directory changes relative to the files directory in any current vendor modified OS or other custom android OS or even a future android update?

More importantly, if it fails to find paths using jnius, it resorts to hacky way of using regular expressions to build what it thinks should be the correct path. I think this is the point, where it should look for XDG configs instead of trying to create hard coded arbitrary paths.

Pydroid didn't use termux shell as base and similar IDEs not is used to simulate a full linux distro and they are not required to have XDG env vars

They're still shells. They have env variables. If it's not required by them to have XDG env vars, it's also not required by them to have ANDROID_DATA or ANDROID_ROOT env vars. If these are not present, platformdirs automatically falls back to using unix package directories with XDG configs.

If others come across this issue, as a hacky temporary fix for termux, change the android data variable, before pip install:

export ANDROID_DATA=''

Expect some packages like am to break.

---

Since only Python packages are a issue, instead of changing default shell envs, we can also change python os.environ using sitecustomize.py, which will be invoked on each python execution:

The file should be placed in

$PREFIX/lib/pythonX.Y/site-packages/sitecustomize.py

import os

# Get $PREFIX
prefix=os.environ.get('PREFIX')

# Fail Android check 
# https://github.com/platformdirs/platformdirs/blob/73e00ef12bb50bb886fe8a135b7a54085efd131a/src/platformdirs/__init__.py#L27
os.environ['ANDROID_DATA']=''

# Fix XDG DIRS
# https://github.com/termux/termux-packages/wiki/Termux-file-system-layout
os.environ['XDG_DATA_DIRS']=prefix+'/share'
os.environ['XDG_CONFIG_DIRS']=prefix+'/etc'
os.environ['XDG_RUNTIME_DIR']=prefix+'/var/run'

[YariKartoshe4ka]

Even if you want to support scenario 1, the use of shared_preferences for config and cache for cache is completely arbitrary. Hard coded paths are never considered wise in any android development. While the code uses Context().getFilesDir(), it fails to call getCacheDir() or getSharedPreferences(), but adds known strings to the path retrieved from getFilesDir(). What if the cache and config directory changes relative to the files directory in any current vendor modified OS or other custom android OS or even a future android update?

I know, there are a lot of GSIs and ROMs, but I have never seen such a vendor with modified getCacheDir and other, and I would like to consider cases practically, not theoretically.
If AOSP has changed paths, I will definitely find out about it as quickly as i will find out about Termux will abandon xdg

More importantly, if it fails to find paths using jnius, it resorts to hacky way of using regular expressions to build what it thinks should be the correct path. I think this is the point, where it should look for XDG configs instead of trying to create hard coded arbitrary paths.

Yes, it is a hacky way, but it works today correctly. Moreover, android.os.Environment.get* uses the same method of getting directories

They're still shells. They have env variables. If it's not required by them to have XDG env vars, it's also not required by them to have ANDROID_DATA or ANDROID_ROOT env vars. If these are not present, platformdirs automatically falls back to using unix package directories with XDG configs.

ANDROID_DATA and ANDROID_ROOT defined in AOSP in init.environ.rc, so all apps has these vars

---

It seems to me it would be more correct if you put everything directly on the code, platformdirs always open for PR 👍

[Blocked]

ANDROID_DATA and ANDROID_ROOT defined in AOSP in init.environ.rc, so all apps has these vars

Not exactly. Termux, Pydroid and other IDEs doesn't have to, just like XDG env variables doesn't need to be present. Termux does have this present, because they're explicitly added to the shell environment for android related unix packages, but it isn't needed and it can be modified easily.

Are you willing to break up android support to these two scenarios? Say, check for prefixed environments using $PREFIX shell env and fallback to unix/XDG spec directories in these dev environments?

[YariKartoshe4ka]

Not exactly. Termux, Pydroid and other IDEs doesn't have to, just like XDG env variables doesn't need to be present. Termux does have this present, because they're explicitly added to the shell environment for android related unix packages, but it isn't needed and it can be modified easily.

Doesn't have to? These variables adds to environment of each unix (android) process. Termux creates its own environment, so copies these variables, and as you have already said yourself, removal or modification is fraught with

some packages like am to break.

---

Are you willing to break up android support to these two scenarios? Say, check for prefixed environments using $PREFIX shell env and fallback to unix/XDG spec directories in these dev environments?

Yep 🤝

[GalaxySnail]

Today I found that pip can't read my configuration in ~/.config/pip/pip.conf on termux, it is definitely a breaking change for termux users. And every application on termux stores its config files in ~/.cache except those apps depend on platformdirs. I think #72 is a good idea, hopefully it can be accepted.