Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

False positive http-crawl-non_statics with Nextcloud Deck mobile app #991

Open
Libsystem-coop opened this issue Mar 6, 2024 · 6 comments
Open

False positive http-crawl-non_statics with Nextcloud Deck mobile app #991

Libsystem-coop opened this issue Mar 6, 2024 · 6 comments

Comments

@Libsystem-coop
Copy link

What happened?

When setup Nextcloud Deck andoid App connection or refreshing multiple Deck entries triggers crowdsecurity/http-crawl-non_statics and blocks the client.

What did you expect to happen?

Allow the app to refresh all data without triggering the crowdsecurity/http-crawl-non_statics

How can we reproduce it (as minimally and precisely as possible)?

Install the Nextcloud Deck app on an existing Nextcloud server instance, create multiple Decks entries and install Deck android app (From F-Droid : no fees).

Anything else we need to know?

Nextcloud instance : ver 27.1.7
PHP 8.1.27
Webserver version: Apache/2.4.38 (Debian)
Android Deck app : ver 1.23.4

The parsers/s02-enrich/crowdsecurity/nextcloud-whitelist.yaml don't seem to handle this at the moment.

Crowdsec version

For LAPI Server:
version: v1.6.0-freebsd-4b8e6cd7
Codename: alphaga
BuildDate: 2024-02-20_01:09:28
GoVersion: 1.21.7
Platform: freebsd
libre2: C++
Constraint_parser: >= 1.0, <= 3.0
Constraint_scenario: >= 1.0, <= 3.0
Constraint_api: v1
Constraint_acquis: >= 1.0, < 2.0

OS version

freedsd 13.2-RELEASE-p10

Enabled collections and parsers

COLLECTIONS (LAPI side)

crowdsecurity/freebsd ✔️ enabled 0.1 /usr/local/etc/crowdsec/collections/freebsd.yaml
crowdsecurity/opnsense ✔️ enabled 0.4 /usr/local/etc/crowdsec/collections/opnsense.yaml
crowdsecurity/opnsense-gui ✔️ enabled 0.1 /usr/local/etc/crowdsec/collections/opnsense-gui.yaml
crowdsecurity/sshd ✔️ enabled 0.3 /usr/local/etc/crowdsec/collections/sshd.yaml
firewallservices/pf ✔️ enabled 0.2 /usr/local/etc/crowdsec/collections/pf.yaml

COLLECTIONS (Nextcloud side)
crowdsecurity/apache2 ✔ enabled 0.1 /etc/crowdsec/collections/apache2.yaml
crowdsecurity/base-http-scenarios ✔ enabled 0.8 /etc/crowdsec/collections/base-http-scenarios.yaml
crowdsecurity/http-cve ✔ enabled 2.6 /etc/crowdsec/collections/http-cve.yaml
crowdsecurity/linux ✔ enabled 0.2 /etc/crowdsec/collections/linux.yaml
crowdsecurity/mysql ✔ enabled 0.1 /etc/crowdsec/collections/mysql.yaml
crowdsecurity/nextcloud ✔ enabled 0.3 /etc/crowdsec/collections/nextcloud.yaml
crowdsecurity/sshd ✔ enabled 0.3 /etc/crowdsec/collections/sshd.yaml

PARSERS (LAPI side)

crowdsecurity/dateparse-enrich ✔️ enabled 0.2 /usr/local/etc/crowdsec/parsers/s02-enrich/dateparse-enrich-
.yaml
crowdsecurity/geoip-enrich ✔️ enabled 0.2 /usr/local/etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
crowdsecurity/opnsense-gui-logs ✔️ enabled 0.1 /usr/local/etc/crowdsec/parsers/s01-parse/opnsense-gui-logs-
.yaml
crowdsecurity/sshd-logs ✔️ enabled 2.3 /usr/local/etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
crowdsecurity/syslog-logs ✔️ enabled 0.8 /usr/local/etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
crowdsecurity/whitelists ✔️ enabled 0.2 /usr/local/etc/crowdsec/parsers/s02-enrich/whitelists.yaml
crowdsecurity/whitelists-local 🏠 enabled,local /usr/local/etc/crowdsec/parsers/s02-enrich/mywhitelist.yaml
firewallservices/pf-logs ✔️ enabled 0.5 /usr/local/etc/crowdsec/parsers/s01-parse/pf-logs.yaml

PARSERS (Nextcloud side)

crowdsecurity/apache2-logs ✔ enabled 1.4 /etc/crowdsec/parsers/s01-parse/apache2-logs.yaml
crowdsecurity/dateparse-enrich ✔ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/dateparse-enrich.yaml
crowdsecurity/geoip-enrich ✔ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/geoip-enrich.yaml
crowdsecurity/http-logs ✔ enabled 1.2 /etc/crowdsec/parsers/s02-enrich/http-logs.yaml
crowdsecurity/mysql-logs ✔ enabled 0.4 /etc/crowdsec/parsers/s01-parse/mysql-logs.yaml
crowdsecurity/nextcloud-logs ✔ enabled 0.3 /etc/crowdsec/parsers/s01-parse/nextcloud-logs.yaml
crowdsecurity/nextcloud-whitelist ✔ enabled 0.7 /etc/crowdsec/parsers/s02-enrich/nextcloud-whitelist.yaml
crowdsecurity/sshd-logs ✔ enabled 2.3 /etc/crowdsec/parsers/s01-parse/sshd-logs.yaml
crowdsecurity/syslog-logs ✔ enabled 0.8 /etc/crowdsec/parsers/s00-raw/syslog-logs.yaml
crowdsecurity/whitelists ✔ enabled 0.2 /etc/crowdsec/parsers/s02-enrich/whitelists.yaml

Acquisition config

LAPI Side 
filenames:
  - /var/log/nginx/*.log
  - ./tests/nginx/nginx.log
#this is not a syslog log, indicate which kind of logs it is
labels:
  type: nginx
---
filenames:
 - /var/log/auth.log
 - /var/log/syslog
labels:
  type: syslog
---
filenames:
 - /var/log/httpd-access.log
 - /var/log/httpd-error.log
labels:
  type: apache2

Nextcloud Side

filenames:
  - /var/log/apache2/nextcloud_errors.log
  - /var/log/apache2/error.log
  - /var/log/apache2/nextcloud_access.log
labels:
  type: apache2
---
#Generated acquisition file - wizard.sh (service: ssh) / files : /var/log/auth.log
filenames:
  - /var/log/auth.log
labels:
  type: syslog
---
#Generated acquisition file - wizard.sh (service: mysql) / files : /var/log/mysql/error.log
filenames:
  - /var/log/mysql/error.log
labels:
  type: mysql
---
#Generated acquisition file - wizard.sh (service: linux) / files : /var/log/syslog /var/log/kern.log /var/log/messages
filenames:
  - /var/log/syslog
  - /var/log/kern.log
  - /var/log/messages
labels:
  type: syslog
---
source: journalctl
journalctl_filter:
  - "SYSLOG_IDENTIFIER=Nextcloud"
labels:
  type: syslog

Config show

Global:
   - Configuration Folder   : /usr/local/etc/crowdsec
   - Data Folder            : /var/db/crowdsec/data
   - Hub Folder             : /usr/local/etc/crowdsec/hub
   - Simulation File        : /usr/local/etc/crowdsec/simulation.yaml
   - Log Folder             : /var/log/crowdsec
   - Log level              : info
   - Log Media              : file
Crowdsec:
  - Acquisition File        : /usr/local/etc/crowdsec/acquis.yaml
  - Parsers routines        : 1
  - Acquisition Folder      : /usr/local/etc/crowdsec/acquis.d/
cscli:
  - Output                  : human
  - Hub Branch              : 
API Client:
  - URL                     : http://xxx.xxx.xxx.xxx:8080/
  - Login                   : localhost
  - Credentials File        : /usr/local/etc/crowdsec/local_api_credentials.yaml
Local API Server:
  - Listen URL              : xxx.xxx.xxx.xxx:8080
  - Profile File            : /usr/local/etc/crowdsec/profiles.yaml

  - Trusted IPs:
      - 127.0.0.1
      - ::1
  - Database:
      - Type                : sqlite
      - Path                : /var/db/crowdsec/data/crowdsec.db
      - Flush age           : 7d
      - Flush size          : 5000

Alert inspection

  • ID : 508

  • Date : 2024-03-06T09:45:05Z

  • Machine : b881

  • Simulation : false

  • Reason : crowdsecurity/http-crawl-non_statics

  • Events Count : 54

  • Scope:Value : Ip:

  • Country :

  • AS :

  • Begin : 2024-03-06 09:44:30.370898407 +0000 UTC

  • End : 2024-03-06 09:45:05.027992578 +0000 UTC

  • UUID : 6c9e49da

  • Context :
    │ Key │ Value │
    │ method │ GET │
    │ status │ 200 │
    │ target_uri │ /index.php/apps/deck/api/v1.1/boards/13/stacks/34/cards/239? │
    │ target_uri │ /index.php/apps/deck/api/v1.1/boards/13/stacks/34/cards/275? │
    │ target_uri │ /index.php/apps/deck/api/v1.1/boards/13/stacks/34/cards/283? │
    │ target_uri │ /index.php/apps/deck/api/v1.1/boards/13/stacks/36/cards/360? │
    │ target_uri │ /index.php/apps/deck/api/v1.1/boards/13/stacks/36/cards/282? │
    │ target_uri │ /index.php/apps/deck/api/v1.1/boards/13/stacks/37/cards/175? │
    │ user_agent │ Mozilla/5.0 (Android) Nextcloud-android/3.28.0 │

  • Events :

  • Date: 2024-03-06 10:45:04 +0100 +0100
    │ Key │ Value │
    │ datasource_path │ /var/log/apache2/nextcloud_access.log │
    │ datasource_type │ file │
    │ http_args_len │ 0 │
    │ http_path │ /index.php/apps/deck/api/v1.1/boards/13/stacks/34/cards/239? │
    │ http_status │ 200 │
    │ http_user_agent │ Mozilla/5.0 (Android) Nextcloud-android/3.28.0 │
    │ http_verb │ GET │
    │ log_type │ http_access-log │
    │ service │ http │
    │ source_ip │ │
    │ timestamp │ 2024-03-06T10:45:04+01:00 │

  • Date: 2024-03-06 10:45:04 +0100 +0100

│ Key │ Value │
│ datasource_path │ /var/log/apache2/nextcloud_access.log │
│ datasource_type │ file │
│ http_args_len │ 0 │
│ http_path │ /index.php/apps/deck/api/v1.1/boards/13/stacks/34/cards/275? │
│ http_status │ 200 │
│ http_user_agent │ Mozilla/5.0 (Android) Nextcloud-android/3.28.0 │
│ http_verb │ GET │
│ log_type │ http_access-log │
│ service │ http │
│ source_ip │ │
│ timestamp │ 2024-03-06T10:45:04+01:00 │

  • Date: 2024-03-06 10:45:04 +0100 +0100
    │ Key │ Value │
    │ datasource_path │ /var/log/apache2/nextcloud_access.log │
    │ datasource_type │ file │
    │ http_args_len │ 0 │
    │ http_path │ /index.php/apps/deck/api/v1.1/boards/13/stacks/34/cards/283? │
    │ http_status │ 200 │
    │ http_user_agent │ Mozilla/5.0 (Android) Nextcloud-android/3.28.0 │
    │ http_verb │ GET │
    │ log_type │ http_access-log │
    │ service │ http │
    │ source_ip │ │
    │ timestamp │ 2024-03-06T10:45:04+01:00 │

  • Date: 2024-03-06 10:45:04 +0100 +0100
    │ Key │ Value │
    │ datasource_path │ /var/log/apache2/nextcloud_access.log │
    │ datasource_type │ file │
    │ http_args_len │ 0 │
    │ http_path │ /index.php/apps/deck/api/v1.1/boards/13/stacks/36/cards/360? │
    │ http_status │ 200 │
    │ http_user_agent │ Mozilla/5.0 (Android) Nextcloud-android/3.28.0 │
    │ http_verb │ GET │
    │ log_type │ http_access-log │
    │ service │ http │
    │ source_ip │ │
    │ timestamp │ 2024-03-06T10:45:04+01:00 │

  • Date: 2024-03-06 10:45:04 +0100 +0100
    │ Key │ Value │
    │ datasource_path │ /var/log/apache2/nextcloud_access.log │
    │ datasource_type │ file │
    │ http_args_len │ 0 │
    │ http_path │ /index.php/apps/deck/api/v1.1/boards/13/stacks/36/cards/282? │
    │ http_status │ 200 │
    │ http_user_agent │ Mozilla/5.0 (Android) Nextcloud-android/3.28.0 │
    │ http_verb │ GET │
    │ log_type │ http_access-log │
    │ service │ http │
    │ source_ip │ │
    │ timestamp │ 2024-03-06T10:45:04+01:00 │

  • Date: 2024-03-06 10:45:04 +0100 +0100
    │ Key │ Value │
    │ datasource_path │ /var/log/apache2/nextcloud_access.log │
    │ datasource_type │ file │
    │ http_args_len │ 0 │
    │ http_path │ /index.php/apps/deck/api/v1.1/boards/13/stacks/37/cards/175? │
    │ http_status │ 200 │
    │ http_user_agent │ Mozilla/5.0 (Android) Nextcloud-android/3.28.0 │
    │ http_verb │ GET │
    │ log_type │ http_access-log │
    │ service │ http │
    │ source_ip │ │
    │ timestamp │ 2024-03-06T10:45:04+01:00 │
@Masgalor
Copy link

Masgalor commented May 7, 2024

This is a rather big problem fore me, but the only solutions i could find are way too broad.
I don't want to whitelist the subdomain for nextcloud so everything is disabled and I also don't want to remove http-crawl-non_statics because I need it for other services.
So, how can I disable a specific scenario for a specific url e.g. https://example.com/nextcloud/remote.php/dav/files should never trigger http-crawl-non_statics?

@LaurenceJJones
Copy link
Contributor

This is a rather big problem fore me, but the only solutions i could find are way too broad.
I don't want to whitelist the subdomain for nextcloud so everything is disabled and I also don't want to remove http-crawl-non_statics because I need it for other services.
So, how can I disable a specific scenario for a specific url e.g. https://example.com/nextcloud/remote.php/dav/files should never trigger http-crawl-non_statics?

The nextcloud whitelist has this

   - evt.Meta.http_status == '404' && evt.Meta.http_verb in ['PROPFIND', 'GET'] && evt.Meta.http_path matches '^/remote.php/(web)?dav/'

I guess you are routing to nextcloud via a url so the path doesn't match?

@Masgalor
Copy link

Masgalor commented May 7, 2024

The url is not the problem but the status-code.
If you sync a lot of files with the desktop-client you would see logs like this one:

Nextcloud log 
[07/May/2024:19:41:34 +0200] "PROPFIND /remote.php/dav/files/b73de1b8-4b2a-39e0-8cd4-2de16c55230f/Projektdaten HTTP/1.1" 207 281 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/ HTTP/1.1" 207 274 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/ HTTP/1.1" 207 806 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/b73de1b8-4b2a-39e0-8cd4-2de16c55230f/Sync HTTP/1.1" 207 277 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera HTTP/1.1" 207 1413 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/Camera HTTP/1.1" 207 762 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/Camera/IMG_20240313_190412488.jpg HTTP/1.1" 200 1949313 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/Camera/IMG_20240330_213446912.jpg HTTP/1.1" 200 2389540 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240207_172556.jpg HTTP/1.1" 200 1433726 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240311_173940.jpg HTTP/1.1" 200 1159471 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240313_185123.jpg HTTP/1.1" 200 1389437 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240313_184955.jpg HTTP/1.1" 200 1580571 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240313_185208.jpg HTTP/1.1" 200 1507254 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240317_211004.jpg HTTP/1.1" 200 1666356 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/Camera/IMG_20240410_191006902.jpg HTTP/1.1" 200 2169135 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240317_211758.jpg HTTP/1.1" 200 1628284 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240317_211837.jpg HTTP/1.1" 200 1343372 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_212532.jpg HTTP/1.1" 200 1825909 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_212556.jpg HTTP/1.1" 200 2058648 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213020.jpg HTTP/1.1" 200 2075997 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213025.jpg HTTP/1.1" 200 2080572 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213029.jpg HTTP/1.1" 200 1843992 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:38 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213031.jpg HTTP/1.1" 200 2509354 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:38 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213035.jpg HTTP/1.1" 200 2153364 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:38 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/ HTTP/1.1" 207 274 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"

So there might get a lot of messages with status-code 200 in a very short time in the log, this can trigger http-crawl-non_statics.
The rule you mentioned only excludes status-code 404, so should I add an identical rule for status-code 200 and 207?

@LaurenceJJones
Copy link
Contributor

The url is not the problem but the status-code.
If you sync a lot of files with the desktop-client you would see logs like this one:

Nextcloud log 
[07/May/2024:19:41:34 +0200] "PROPFIND /remote.php/dav/files/b73de1b8-4b2a-39e0-8cd4-2de16c55230f/Projektdaten HTTP/1.1" 207 281 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/ HTTP/1.1" 207 274 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/ HTTP/1.1" 207 806 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/b73de1b8-4b2a-39e0-8cd4-2de16c55230f/Sync HTTP/1.1" 207 277 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera HTTP/1.1" 207 1413 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/Camera HTTP/1.1" 207 762 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/Camera/IMG_20240313_190412488.jpg HTTP/1.1" 200 1949313 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/Camera/IMG_20240330_213446912.jpg HTTP/1.1" 200 2389540 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:35 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240207_172556.jpg HTTP/1.1" 200 1433726 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240311_173940.jpg HTTP/1.1" 200 1159471 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240313_185123.jpg HTTP/1.1" 200 1389437 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240313_184955.jpg HTTP/1.1" 200 1580571 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240313_185208.jpg HTTP/1.1" 200 1507254 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240317_211004.jpg HTTP/1.1" 200 1666356 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/Camera/IMG_20240410_191006902.jpg HTTP/1.1" 200 2169135 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:36 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240317_211758.jpg HTTP/1.1" 200 1628284 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240317_211837.jpg HTTP/1.1" 200 1343372 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_212532.jpg HTTP/1.1" 200 1825909 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_212556.jpg HTTP/1.1" 200 2058648 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213020.jpg HTTP/1.1" 200 2075997 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213025.jpg HTTP/1.1" 200 2080572 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:37 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213029.jpg HTTP/1.1" 200 1843992 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:38 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213031.jpg HTTP/1.1" 200 2509354 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:38 +0200] "GET /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/OpenCamera/IMG_20240321_213035.jpg HTTP/1.1" 200 2153364 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"
[07/May/2024:19:41:38 +0200] "PROPFIND /remote.php/dav/files/71ab7335-fb23-3b3c-b832-780d0ca4552b/ HTTP/1.1" 207 274 "-" "Mozilla/5.0 (Linux) mirall/3.13.0git (Nextcloud, opensuse-tumbleweed-6.8.8-1-default ClientArchitecture: x86_64 OsArchitecture: x86_64)"

So there might get a lot of messages with status-code 200 in a very short time in the log, this can trigger http-crawl-non_statics.
The rule you mentioned only excludes status-code 404, so should I add an identical rule for status-code 200 and 207?

Yes exactly you can just update the line to be

evt.Meta.http_status in ['200','207','404']

@fracklaus
Copy link

fracklaus commented May 10, 2024 •
edited

I ran into the same issue today with crowdsec: access to the android deck app, as well as access to deck in a browser, triggers http-crawl-non_statics.
Here is the alert from cscli alerts inspect:

╭─────────────────┬────────────────────────────────────────────────────────────╮
│       Key       │                           Value                            │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ datasource_path │ /var/log/apache2/access.log                                │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ datasource_type │ file                                                       │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ http_args_len   │ 0                                                          │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ http_path       │ /index.php/apps/deck/api/v1.1/boards/2/stacks/4/cards/209? │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ http_status     │ 200                                                        │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ http_user_agent │ Mozilla/5.0 (Android) Nextcloud-android/3.29.0             │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ http_verb       │ GET                                                        │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ log_type        │ http_access-log                                            │
├─────────────────┼────────────────────────────────────────────────────────────┤
│ service         │ http                                                       │

Currently in nextcloud-whitelist.yaml, the URL /index.php/apps/deck/api is missing, and the status 200 is also not included.

Please include this in nextcloud-whitelist.yaml to fix the issue. I added a local file /etc/crowdsec/parsers/s02-enrich/nextcloud-whitelist-deck.yaml with content

name: crowdsecurity/nextcloud-whitelist-deck
description: "Whitelist events from nextcloud - deck android app"
filter: "evt.Meta.service == 'http' && evt.Meta.log_type in ['http_access-log', 'http_error-log']"
whitelist:
  reason: "Nextcloud Deck Whitelist"
  expression:
   - evt.Meta.http_status == '200' && evt.Meta.http_verb == 'GET' && evt.Meta.http_path startsWith '/index.php/apps/deck/api/' # browsing deck entries
   - evt.Meta.http_status == '200' && evt.Meta.http_verb == 'GET' && evt.Meta.http_path startsWith '/ocs/v2.php/collaboration/resources/deck-card/' # browsing deck entries

which is fixing the issue for me.

@Libsystem-coop
Copy link
Author

Thank you @fracklaus and @LaurenceJJones, it seems to work also for me on local white-list.

I have also added a copy of this line with code 200 that triggered http-crawl-non_statics when syncing multiple small files in short period of time:

   - evt.Meta.http_status == '200' && evt.Meta.http_verb in ['PROPFIND', 'GET'] && evt.Meta.http_path matches '^/remote.php/(web)?dav/'

Hope all this fixes could be added to nextcloud-whitelist.yaml

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@fracklaus @LaurenceJJones @Masgalor @Libsystem-coop