Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tablet is not syncing (Notifications Socket is Not OK?) #235

Open
QJoly opened this issue Apr 18, 2023 · 12 comments
Open

Tablet is not syncing (Notifications Socket is Not OK?) #235

QJoly opened this issue Apr 18, 2023 · 12 comments

Comments

@QJoly
Copy link
Contributor

QJoly commented Apr 18, 2023

Hi,

Since a few months, my tablet does not synchronize with RMFakecloud. I use the most recent Docker image (tag: always) and my remarkable is up to date (3.2.3.1595).

In log.txt, I have this message :

Apr 18 07:46:51.449 Debug: UserToken: setting a new userToken ("eyJhbGciOiJIUzI1NiIs"...) (/usr/src/debug/xochitl/override+gitAUTOINC+d825cceee7-r0/git/src/network/src/usertoken.cpp:73, setUserToken)
Apr 18 07:46:51.511 Debug: Input locale setting has changed, updating the key map. (:0, )
Apr 18 07:46:51.512 Debug: Read a langCode of  "" (:0, )
Apr 18 07:46:51.512 Warning: No keymap set by QT settings or firmware, defaulting to US. (:0, )
Apr 18 07:46:51.513 Debug: setting US keymap 293 147 (:0, )
Apr 18 07:46:51.513 Debug: numlock=0 , capslock=0, scrolllock=0 (:0, )
Apr 18 07:46:51.515 Debug: Input locale setting has changed, updating the key map. (:0, )
Apr 18 07:46:51.516 Debug: Read a langCode of  "" (:0, )
Apr 18 07:46:51.516 Warning: No keymap set by QT settings or firmware, defaulting to US. (:0, )
Apr 18 07:46:51.517 Debug: setting US keymap 293 147 (:0, )
Apr 18 07:46:51.517 Debug: numlock=0 , capslock=0, scrolllock=0 (:0, )
Apr 18 07:46:51.536 Warning: Notifications socket is not OK: UnconnectedState (/usr/src/debug/xochitl/override+gitAUTOINC+d825cceee7-r0/git/src/notifications/src/notifications.cpp:187, checkIfShouldConnect)
Apr 18 07:47:42.198 Info: Scanning: true (:0, )
Apr 18 07:47:46.061 Info: Scanning: false (:0, )
Apr 18 07:49:22.725 Warning: Already have this address: QHostAddress("2a01:cb14:e12:8901:2250:e7ff:fefa:7e1e") (:0, )

When I install rmfakecloud with 'magic script', I have a sed error:

root@reMarkable:~# ./installer.sh install "https://remarkable.redacted
Extracting embedded binary...
~/rmfakecloud ~
CA exists
Private key exists
Pub key exists
crt exists
The cert has been already installed, it will be removed and reinstalled!!!
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Clearing symlinks in /etc/ssl/certs...
done.
Updating certificates in /etc/ssl/certs...
rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
Setting cloud sync to: https://remarkable.redacted
Patching /etc/hosts
# rmfake_start
Stoping xochitl..
Fixing sync status...
sed: -i requires an argument

but the reverse proxy is runned and fonctionnal:

[[0;1;32m●[[0m proxy.service - reverse proxy
     Loaded: loaded (/etc/systemd/system/proxy.service; enabled; vendor preset: disabled)
     Active: [[0;1;32mactive (running)[[0m since Tue 2023-04-18 07:58:24 UTC; 40s ago
   Main PID: 11203 (rmfake-proxy)
     CGroup: /system.slice/proxy.service
             └─11203 /home/root/rmfakecloud/rmfake-proxy -cert /home/root/rmfakecloud/proxy.bundle.crt -key /home/root/rmfakecloud/proxy.key https://remarkable.redacted

Apr 18 07:58:24 reMarkable systemd[1]: Started reverse proxy.
Apr 18 07:58:24 reMarkable rmfake-proxy[11203]: 2023/04/18 07:58:24 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted
---
Apr 18 07:43:32 reMarkable systemd[1]: Started reverse proxy.
Apr 18 07:43:32 reMarkable rmfake-proxy[1990]: 2023/04/18 07:43:32 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted
Apr 18 07:44:10 reMarkable rmfake-proxy[1990]: terminated
Apr 18 07:44:10 reMarkable systemd[1]: Stopping reverse proxy...
Apr 18 07:44:10 reMarkable systemd[1]: proxy.service: Succeeded.
Apr 18 07:44:10 reMarkable systemd[1]: Stopped reverse proxy.
Apr 18 07:58:02 reMarkable systemd[1]: Started reverse proxy.
Apr 18 07:58:02 reMarkable rmfake-proxy[9586]: 2023/04/18 07:58:02 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted
Apr 18 07:58:16 reMarkable rmfake-proxy[9586]: terminated
Apr 18 07:58:16 reMarkable systemd[1]: Stopping reverse proxy...
Apr 18 07:58:16 reMarkable systemd[1]: proxy.service: Succeeded.
Apr 18 07:58:16 reMarkable systemd[1]: Stopped reverse proxy.
Apr 18 07:58:24 reMarkable systemd[1]: Started reverse proxy.
Apr 18 07:58:24 reMarkable rmfake-proxy[11203]: 2023/04/18 07:58:24 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted
Apr 18 08:00:45 reMarkable rmfake-proxy[11203]: 2023/04/18 08:00:45 http: TLS handshake error from 192.168.1.84:42804: remote error: tls: unknown certificate authority
Apr 18 08:00:45 reMarkable rmfake-proxy[11203]: 2023/04/18 08:00:45 http: TLS handshake error from 192.168.1.84:42810: remote error: tls: unknown certificate authority
Apr 18 08:02:15 reMarkable rmfake-proxy[11203]: terminated
Apr 18 08:02:15 reMarkable systemd[1]: Stopping reverse proxy...
Apr 18 08:02:15 reMarkable systemd[1]: proxy.service: Succeeded.
Apr 18 08:02:15 reMarkable systemd[1]: Stopped reverse proxy.
Apr 18 08:02:15 reMarkable systemd[1]: Started reverse proxy.
Apr 18 08:02:15 reMarkable rmfake-proxy[11273]: 2023/04/18 08:02:15 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted
Apr 18 08:03:34 reMarkable rmfake-proxy[11273]: 2023/04/18 08:03:34 http: TLS handshake error from 192.168.1.84:40556: remote error: tls: unknown certificate authority

(192.168.1.84 is my laptop. I tested the reverse proxy with the IP of the tablet)

root@reMarkable:~# echo Q | openssl s_client -connect localhost:443  -verify_hostname local.appspot.com -CAfile /etc/ssl/certs/ca-certificates.crt 2>&1 | grep Verify
Verify return code: 0 (ok)
    Verify return code: 0 (ok)

Do you have any solution ? Thanks in advance

@ddvk
Copy link
Owner

ddvk commented Apr 24, 2023

the tablet doesn't trust: https://remarkable.redacted, if you are not using an official CA (e.g let's encrypt) and using some self signed CA you need to add it to the trusted CAs on the tablet.

@QJoly
Copy link
Contributor Author

QJoly commented Apr 25, 2023

I'm using let's encrypt, the tablet trust https://remarkable.redacted since I can curl without adding -k to accept untrusted cert :(

@Eeems
Copy link
Contributor

Eeems commented Apr 25, 2023

Apr 18 08:02:15 reMarkable rmfake-proxy[11273]: 2023/04/18 08:02:15 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted
Apr 18 08:03:34 reMarkable rmfake-proxy[11273]: 2023/04/18 08:03:34 http: TLS handshake error from 192.168.1.84:40556: remote error: tls: unknown certificate authority

These lines lead me to believe that it doesn't trust it though. Did you test the curl call from the device, or your computer?

@QJoly
Copy link
Contributor Author

QJoly commented Apr 27, 2023

Apr 18 08:02:15 reMarkable rmfake-proxy[11273]: 2023/04/18 08:02:15 cert-file=/home/root/rmfakecloud/proxy.bundle.crt key-file=/home/root/rmfakecloud/proxy.key listen-addr=:443 upstream-url=https://remarkable.redacted
Apr 18 08:03:34 reMarkable rmfake-proxy[11273]: 2023/04/18 08:03:34 http: TLS handshake error from 192.168.1.84:40556: remote error: tls: unknown certificate authority

These lines lead me to believe that it doesn't trust it though. Did you test the curl call from the device, or your computer?

Sorry for late answer,

I can curl from the device without add untrusted cert:

root@reMarkable:~# ./installer.sh gencert
CA exists
Private key exists
Pub key exists
crt exists
root@reMarkable:~# curl https://remarkable.redacted
<!doctype html><html lang="en"><head><meta charset="utf-8"/><meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/><meta name="description" content="RM FakeApi"/><title>rmfakecloud</title><script defer="defer" src="/static/js/main.9c2de5b1.js"></script><link href="/static/css/main.d94d89ba.css" rel="stylesheet"></head><body><noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>root@reMarkable:~# 

@zeigerpuppy
Copy link

zeigerpuppy commented May 1, 2023

Continuing error reporting of sync here instead of #237

I tried:

  1. remove /usr/local/share/ca-certificates/ca.crt and re-ran installer on tablet
  2. added RM_TRUST_PROXY=true to server (docker-compose) config
  3. unpair and repair with server (pairing successful)

log.txt shows successful pairing but then the following errors:

May 01 10:57:31.861 Debug: UserToken: setting a new userToken ("XXXXXXXXXX"...) (/usr/src/debug/xochitl/override+gitAUTOINC+6a003d604f-r0/git/src/network/src/usertoken.cpp:73, setUserToken)
May 01 10:57:31.954 Warning: Notifications socket is not OK: UnconnectedState (/usr/src/debug/xochitl/override+gitAUTOINC+6a003d604f-r0/git/src/notifications/src/notifications.cpp:187, checkIfShouldConnect)
May 01 10:57:32.626 Warning: Could not find hostname for service "notifications" (/usr/src/debug/xochitl/override+gitAUTOINC+6a003d604f-r0/git/src/network/src/servicehostname.cpp:44, parseNetworkReply)

I checked the setting in etc/hosts and they are directing to the proxy properly

# rmfake_start
127.0.0.1 hwr-production-dot-remarkable-production.appspot.com
127.0.0.1 service-manager-production-dot-remarkable-production.appspot.com
127.0.0.1 local.appspot.com
127.0.0.1 my.remarkable.com
127.0.0.1 internal.cloud.remarkable.com
127.0.0.1 ping.remarkable.com
# rmfake_end
  • I can query the cert of the server (LetsEncrypt cert) and it returns OK
echo Q | openssl s_client -connect localhost:443  -verify_hostname local.appspot.com -CAfile /etc/ssl/certs/ca-certificates.crt 2>&1 | grep Verify
p Verify
Verify return code: 0 (ok)
    Verify return code: 0 (ok)
  • The unit file for the proxy.service looks sane (has my server address) and systemctrl shows the service running with the correct address.

Bit stumped as to what may be the cause of failed tablet-> server sync

Looks like the main error is Could not find hostname for service "notifications" (also showing in the xochitl log)

Any ideas?

@ddvk
Copy link
Owner

ddvk commented May 10, 2023

is there something in the rmfakecloud's logs with "notifications" ?

@develop-Greenant
Copy link

I think this may be caused by a mismatch between the SSL ciphers on the remarkable and the nginx reverse proxy:

When testing wget from client: wget -qO- https://myrmfakecloud.server.net:

wget: note: TLS certificate validation not implemented
wget: TLS error from peer (alert code 80): 80
wget: error getting response: Connection reset by peer

On the nginx server, it reports:

2023/07/15 15:04:03 [error] 21487#21487: *7 connect() failed (111: Connection refused) while connecting to upstream, client: 10.0.0.113, server: myrmfakecloud.server.net, request: "GET /notifications/ws/json/1 HTTP/1.1", upstream: "http://[::1]:3000/notifications/ws/json/1", host: "myrmfakecloud.server.net:443"
2023/07/15 15:04:15 [crit] 21487#21487: *11 SSL_do_handshake() failed (SSL: error:14201044:SSL routines:tls_choose_sigalg:internal error) while SSL handshaking, client: 10.0.0.113, server: 0.0.0.0:443

A connection from other clients is fine.

So, I think the remarkable is trying to use an old cipher/SSL version and the nginx server rejects it.

Tried to loosen the default ciphers with the following in nginx reverse-proxy, but still getting the error:

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers ALL;

Any ideas which cipher may be needed?

I guess this also explains why it stopped working (nginx server update woud have restricted old, insecure ciphers)

@Eeems
Copy link
Contributor

Eeems commented Jul 15, 2023

wget on the rM doesn't support SSL at all out of box, but that isn't related to the proxy. You can grab a version of it that does work to use for testing here: http://toltec-dev.org/thirdparty/bin/wget-v1.21.1

@develop-Greenant
Copy link

develop-Greenant commented Jul 15, 2023

Thanks for tip regarding wget

@pgnhdcrt
Copy link

pgnhdcrt commented Nov 15, 2023

Related issue?
Let's start with the known good, working configuration:
Server: Clean installation from source of rmfakecloud 0.0.15 on AlmaLinux 9.3. Have also tried the Docker image (I'm using a VM, so have snapped and reverted multiple times).
STORAGE_URL=https://<server>
PORT=443
Tablet: Automagic installation of rmfakecloud-proxy 0.0.3 on OS 3.7.0.1930.
ExecStart is calling https://<server>
In this configuration, sync works beautifully and there are no errors in any of the logs. I took packet captures in this config as a baseline.

Change to the problematic configuration. Note that these are the only changes to the above working config:
Server (rmfakecloud restarted post-change):
STORAGE_URL=https://<server>:3000
PORT=300
Tablet (changed via 'installer.sh setcloud', then restarted xochitl):
ExecStart is calling https://<server>:3000
In this configuration, sync is not working.
Tablet: xochitl log shows rm.network.notifications Notifications socket is not OK: UnconnectedState (checkIfShouldConnect /__w/xochitl/xochitl/src/notifications/src/notifications.cpp:190)
Server: Nothing in the logs, however, packet capture (at the server) shows port 443 traffic from the tablet which coincides with the xochitl errors. All other traffic is on port 3000, as expected.

Rebooted the tablet. Sync is still not working, xochitl log still shows the "notifications" error. The weird part? Packet capture no longer shows the port 443 traffic. Changed capture location to the router/AP (OpenWRT), and there is zero traffic (on any port) coming from the tablet when the xochitl errors occur.

My initial thought was a hardcoded sync call to 443, but post-reboot I'm stumped. I'm not familiar enough with the various bits to perform deeper inspection / logging on the tablet (are there debug settings for -proxy?). Happy to dig deeper, just point me in the right direction.

Update: Correction: #271 resolved the issue for me as well. #271 did not resolve the issue of being unable to change to a different (non 443) port. Apologies for the confusion.

@mfussenegger
Copy link

mfussenegger commented Nov 18, 2023

I'm also seeing rm.network.notifications Notifications socket is not OK errors.

  • The tablet is already on 3.8.2.1965 (is this supported?)

  • Built rmfakecloud from source, to include f1827cd

  • rmfakecloud is running with plain HTTP on another host in a local network.

  • Pairing worked

  • rmapi cli seems to work

  • Connectivity is okay (ping, wget -qO-, etc. are working)

  • echo Q | openssl s_client -connect localhost:443 -verify_hostname local.appspot.com -CAfile /etc/ssl/certs/ca-certificates.crt 2>&1 | grep Verify shows ok

  • I see various GET calls to /sync/v3/... and v1/reports endpoints in the logs

  • I'm not seeing any connections to /notifications/ws/json/1 in the server logs. (Should this be the case?)

  • Trying to connect manually with websocat to that endpoint works if using the STORAGE_URL

  • Trying websocat -v wss://local.appspot.com/notifications/ws/json/1 --header "Authorization. Bearer <token>" also works and I see connecting websocket from: ... on the server when doing this manually.

  • Tried running strace xochitl, but for some reason I'm not seeing any connect/socket related information

  • To isolate if it's the local proxy, or the server component I then tried a different variant by disabling the local proxy on the tablet, and instead editing /etc/hosts to point to a public server. Then I put rmfakecloud behind an nginx proxy with something like the following, and ssh reverse tunneling:

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name hwr-production-dot-remarkable-production.appspot.com;
  server_name service-manager-production-dot-remarkable-production.appspot.com;
  server_name local.appspot.com;
  server_name my.remarkable.com;
  server_name internal.cloud.remarkable.com;
  server_name ping.remarkable.com;

  ssl_certificate_key /etc/nginx/proxy.key;
  ssl_certificate     /etc/nginx/proxy.bundle.crt;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  location / {
    proxy_pass http://tunnel;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_redirect off;
  }
}

(proxy.key, and proxy.bundle.crt are generated with the rmfakecloud-proxy install script)

The result is pretty much the same: Pairing works, sync doesn't. Same Notifications socket is not OK in the log.

I'm not sure what else I could try. Unfortunately I haven't been able to get termshark working on the tablet. Does anyone know of a static build?


Update: #271 fixed the issue for me

@nemunaire
Copy link
Collaborator

nemunaire commented Nov 18, 2023

The error Tablet is not syncing (Notifications Socket is Not OK?) is due to nginx closing inactives connections.

The notifications service creates a websocket, and after 1 minute without any exchange between rmfakecloud and the tablet, nginx takes the initiative to close the connection. This error occurs, and on the server side, we can see a warning: msg="Can't read from ws websocket: close 1006 (abnormal closure): unexpected EOF", for the same reason: the tablet doesn't send a proper close. This is true, as it's nginx that closes the connection.

This is not a problem as the tablet reconnects a few seconds later.

It can be delayed with nginx by adding proxy_read_timeout 10800;. With this option the socket will have a maximal duration of 3h. Eg.:

location /notifications/ws/json/1 {
    proxy_pass http://_YOURPROXYADDRESS_;
    proxy_http_version 1.1;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header Host $host;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "Upgrade";
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto https;
    proxy_redirect off;
    proxy_read_timeout 10800;
}

@mfussenegger I just create a pull request for the 3.8 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants