DTT1 - Validate that DTT allows using the live and pre-release repository

[fcaffieri]

Description

The objective of this issue is to verify that the installation of the agents and central components can be done with both the packages.wazuh.com and packages-dev.wazuh.com repository

Task

• Validate that it allows installing agents from both repositories
• Validate that it allows installing central components from both repositories
• Make necessary modifications if the requirements of the previous points are not met.

Related

• Issue: Deployability testing tier 1 #4495

[fcaffieri]

Update report

• Analyzing provision module:

The provision module allows installation from both repositories, for all types of installations:

Installation by wizard:

- name: Download the Wazuh installation assistant
   shell: "curl -sO https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }}/{{ version.split('.')[0] }}. {{ version.split('.')[1] }}/wazuh-install.sh"

Installation per package:

{% if ansible_os_family.lower() in ['debian'] %}

- name: Update packages cache
   command: apt-get update -y

- name: Install packages
   command: apt-get -y install curl gnupg apt-transport-https

- name: Install GPG key
   shell: "curl -s https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }}/key/GPG-KEY-WAZUH | gpg --no-default-keyring --keyring gnupg-ring:/usr/share/keyrings/wazuh.gpg --import && chmod 644 /usr/share/keyrings/wazuh.gpg"

- name: Add Wazuh repository
   shell: "echo \"deb [signed-by=/usr/share/keyrings/wazuh.gpg] https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }} /{{ '4.x' if live else 'pre-release' }}/apt/ {{ 'stable' if live else 'unstable' }} main\" | tee -a /etc/apt/sources.list. d/wazuh.list"

- name: Update package information
   command: "apt-get update"
{% endif %}

{% if ansible_os_family.lower() in ['redhat', 'oraclelinux', 'amazon', 'centos'] %}
- name: Update packages cache
   command: yum makecache

- name: Import the GPG key
   command: "rpm --import https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }}/key/GPG-KEY-WAZUH"

- name: Add Wazuh repository
   shell: "echo -e '[wazuh]\ngpgcheck=1\ngpgkey=https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }}/key/GPG-KEY -WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }}/{{ '4 .x' if live else 'pre-release' }}/yum/\nprotect=1' | tee /etc/yum.repos.d/wazuh.repo"
{% endif %}

{% if ansible_os_family.lower() in ['suse']%}
- name: Import the GPG key
   command: "rpm --import https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }}/key/GPG-KEY-WAZUH"
- name: Add Wazuh repository
   shell: "echo -e '[wazuh]\ngpgcheck=1\ngpgkey=https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }}/key/GPG-KEY -WAZUH\nenabled=1\nname=EL-$releasever - Wazuh\nbaseurl=https://{{ 'packages.wazuh.com' if live else 'packages-dev.wazuh.com' }}/{{ '4 .x' if live else 'pre-release' }}/yum/\nprotect=1' | tee /etc/zypp/repos.d/wazuh.repo"

{% endif %}

For uninstall the same procedure is used.

By default, it installs from packages-dev.wazuh.com.
The branch https://github.com/wazuh/wazuh-qa/tree/enhancement/5366-change-install-from-live-repository was generated to modify this.

---

• Analyzing test module :

Install agent:

Installs agent from both repositories depending on the live variable passed by parameter (by default it is from the live repo)

def install_agent(inventory_path, agent_name, wazuh_version, wazuh_revision, live) -> None:

         if live:
             s3_url = 'packages'
             release = wazuh_version[:1] + ".x"
         else:
             s3_url = 'packages-dev'
             release = 'pre-release'

Install central components:

In this case it installs directly from live and does not allow modification

if HostInformation.has_curl(inventory_path):
             commands = [
                 f"curl -sO https://packages.wazuh.com/{wazuh_version}/wazuh-install.sh && sudo bash ./wazuh-install.sh -a --ignore-check"
             ]
         else:
             commands = [
                 f"wget https://packages.wazuh.com/{wazuh_version}/wazuh-install.sh && sudo bash ./wazuh-install.sh -a --ignore-check"
             ]

Install manager:

In this case it installs directly from live and does not allow modification

  if os_name == 'debian':
             commands = [
                     f"wget https://packages.wazuh.com/{wazuh_version}/wazuh-install.sh",
                     f"bash wazuh-install.sh --wazuh-server {node_name} --ignore-check"
             ]
         else:
             commands = [
                     f"curl -sO https://packages.wazuh.com/{wazuh_version}/wazuh-install.sh",
                     f"bash wazuh-install.sh --wazuh-server {node_name} --ignore-check"
             ]

---

Default installation when using the live parameter is from packages.wazuh.com

---

Making modifications to allow installing from both repositories.

[fcaffieri]

Update report

The necessary modifications were made and the following tests were executed:

Test manager with param live True 🟢

• Input yaml:

version: 0.1
description: This workflow is used to test manager deployment for DDT1 PoC
variables:
  manager-os:
    - linux-centos-7-amd64
    - linux-centos-8-amd64
  infra-provider: vagrant
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique manager allocate task
  - task: "allocate-manager-{manager}"
    description: "Allocate resources for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{manager}"
          - instance-name: "{manager}"
          - inventory-output: "{working-dir}/manager-{manager}/inventory.yaml"
          - track-output: "{working-dir}/manager-{manager}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: manager-os
        as: manager

  # Generic manager test task
  - task: "run-manager-tests"
    description: "Run tests install for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/manager-linux-centos-7-amd64/inventory.yaml"
            - wazuh-2: "{working-dir}/manager-linux-centos-8-amd64/inventory.yaml"
          - tests: "install"
          - component: "manager"
          - wazuh-version: "4.7.4"
          - wazuh-revision: "40717"
          - live: True
    depends-on:
      - "allocate-manager-linux-centos-7-amd64"
      - "allocate-manager-linux-centos-8-amd64"

• Log:

test-manager-live.log

Test manager without param live should use True 🟢

• Input yaml:

version: 0.1
description: This workflow is used to test manager deployment for DDT1 PoC
variables:
  manager-os:
    - linux-centos-7-amd64
    - linux-centos-8-amd64
  infra-provider: vagrant
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique manager allocate task
  - task: "allocate-manager-{manager}"
    description: "Allocate resources for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{manager}"
          - instance-name: "{manager}"
          - inventory-output: "{working-dir}/manager-{manager}/inventory.yaml"
          - track-output: "{working-dir}/manager-{manager}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: manager-os
        as: manager

  # Generic manager test task
  - task: "run-manager-tests"
    description: "Run tests install for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/manager-linux-centos-7-amd64/inventory.yaml"
            - wazuh-2: "{working-dir}/manager-linux-centos-8-amd64/inventory.yaml"
          - tests: "install"
          - component: "manager"
          - wazuh-version: "4.7.4"
          - wazuh-revision: "40717"
    depends-on:
      - "allocate-manager-linux-centos-7-amd64"
      - "allocate-manager-linux-centos-8-amd64"

• Log:

test-manager-without-live.log

Test central-components with param live True 🟢

• Input yaml:

version: 0.1
description: This workflow is used to test the Wazuh manager deployment for DDT1 PoC
variables:
  central_components-os:
    - linux-ubuntu-22.04-amd64
  infra-provider: vagrant
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique central components allocate task
  - task: "allocate-central_components-{central_components}"
    description: "Allocate resources for the central_components."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{central_components}"
          - instance-name: "{central_components}"
          - inventory-output: "{working-dir}/central_components-{central_components}/inventory.yaml"
          - track-output: "{working-dir}/central_components-{central_components}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: central_components-os
        as: central_components
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/central_components-{central_components-os}/track.yaml"

  # Generic manager test task
  - task: "run-central_components-{central_components}-tests"
    description: "Run tests install for the central_components."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/central_components-{central_components}/inventory.yaml"
          - tests: "install,restart,stop,uninstall"
          - component: "central_components"
          - wazuh-version: "4.7.4"
          - wazuh-revision: "40717"
          - live: True
    on-error: "abort-all"
    foreach:
      - variable: central_components-os
        as: central_components
    depends-on:
      - "allocate-central_components-{central_components}"

• Log:

test-central-componentes-live.log

Test central-components withouth param live made use True 🟢

• Input yaml:

version: 0.1
description: This workflow is used to test the Wazuh manager deployment for DDT1 PoC
variables:
  central_components-os:
    - linux-ubuntu-22.04-amd64
  infra-provider: vagrant
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique central components allocate task
  - task: "allocate-central_components-{central_components}"
    description: "Allocate resources for the central_components."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{central_components}"
          - instance-name: "{central_components}"
          - inventory-output: "{working-dir}/central_components-{central_components}/inventory.yaml"
          - track-output: "{working-dir}/central_components-{central_components}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: central_components-os
        as: central_components
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/central_components-{central_components-os}/track.yaml"

  # Generic manager test task
  - task: "run-central_components-{central_components}-tests"
    description: "Run tests install for the central_components."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/central_components-{central_components}/inventory.yaml"
          - tests: "install,restart,stop,uninstall"
          - component: "central_components"
          - wazuh-version: "4.7.4"
          - wazuh-revision: "40717"
    on-error: "abort-all"
    foreach:
      - variable: central_components-os
        as: central_components
    depends-on:
      - "allocate-central_components-{central_components}"

• Log:

test-central-componentes-without-live.log

[fcaffieri]

Update report

Test manager with param live False 🔴

• Input yaml:

version: 0.1
description: This workflow is used to test manager deployment for DDT1 PoC
variables:
  manager-os:
    - linux-centos-7-amd64
    - linux-centos-8-amd64
  infra-provider: vagrant
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique manager allocate task
  - task: "allocate-manager-{manager}"
    description: "Allocate resources for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{manager}"
          - instance-name: "{manager}"
          - inventory-output: "{working-dir}/manager-{manager}/inventory.yaml"
          - track-output: "{working-dir}/manager-{manager}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: manager-os
        as: manager

  # Generic manager test task
  - task: "run-manager-tests"
    description: "Run tests install for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/manager-linux-centos-7-amd64/inventory.yaml"
            - wazuh-2: "{working-dir}/manager-linux-centos-8-amd64/inventory.yaml"
          - tests: "install"
          - component: "manager"
          - wazuh-version: "4.7.4"
          - wazuh-revision: "40717"
          - live: False
    depends-on:
      - "allocate-manager-linux-centos-7-amd64"
      - "allocate-manager-linux-centos-8-amd64"

• Log:

test-manager-pre-release-3.log

Test central-components with param live False 🔴

• Input yaml:

version: 0.1
description: This workflow is used to test the Wazuh manager deployment for DDT1 PoC
variables:
  central_components-os:
    - linux-ubuntu-22.04-amd64
  infra-provider: vagrant
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique central components allocate task
  - task: "allocate-central_components-{central_components}"
    description: "Allocate resources for the central_components."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{central_components}"
          - instance-name: "{central_components}"
          - inventory-output: "{working-dir}/central_components-{central_components}/inventory.yaml"
          - track-output: "{working-dir}/central_components-{central_components}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: central_components-os
        as: central_components
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/central_components-{central_components-os}/track.yaml"

  # Generic manager test task
  - task: "run-central_components-{central_components}-tests"
    description: "Run tests install for the central_components."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/central_components-{central_components}/inventory.yaml"
          - tests: "install,restart,stop,uninstall"
          - component: "central_components"
          - wazuh-version: "4.7.4"
          - wazuh-revision: "40717"
          - live: False
    on-error: "abort-all"
    foreach:
      - variable: central_components-os
        as: central_components
    depends-on:
      - "allocate-central_components-{central_components}"

• Log:

test-central-componentes-pre-release.log

---

All tests using the pre-release repository failed, because the branch used by the script does not exist since it is procedurally deleted and is not maintained.

Detail:

The installation script download is correct:

[0;31m [32m[2024-05-15 19:13:27] [INFO] TESTER: Installing Manager with https://packages-dev.wazuh.com/4.7/wazuh-install.sh[0m [ 0m
[0;31m [32m[2024-05-15 19:13:27] [INFO] TESTER: Installing Manager in centos-8[0m [0m

But when installing it it fails with the following error:

05/15/2024 16:00:13 INFO: Wazuh indexer cluster initialized.
05/15/2024 16:00:13 INFO: --- Wazuh server ---
05/15/2024 16:00:13 INFO: Starting the Wazuh manager installation.
05/15/2024 16:00:54 INFO: Wazuh manager installation finished.
05/15/2024 16:00:54 INFO: Starting service wazuh-manager.
05/15/2024 16:01:12 INFO: wazuh-manager service started.
05/15/2024 16:01:12 INFO: Starting Filebeat installation.
05/15/2024 16:01:16 INFO: Filebeat installation finished.
05/15/2024 16:01:17 ERROR: Error downloading wazuh-template.json file.

The generated url is correct, the problem is that the script, after installing the filebeat, has to search for the wazuh-template.json from the following url:

readonly wazuh_install_vesion="0.1"
readonly source_branch="${wazuh_version}"
readonly filebeat_wazuh_template="https://raw.githubusercontent.com/wazuh/wazuh/${source_branch}/extensions/elasticsearch/7.x/wazuh-template.json"
eval "common_curl -so /etc/filebeat/wazuh-template.json ${filebeat_wazuh_template} --max-time 300 --retry 5 --retry-delay 5 --fail ${debug}"

When trying to download the template it fails because the 4.7.4 branch does not exist:

curl -o /etc/filebeat/wazuh-template.json https://raw.githubusercontent.com/wazuh/wazuh/4.7.4/extensions/elasticsearch/7.x/wazuh-template.json --max-time 300 --retry 5 --retry-delay 5 --fail
   % Total % Received % Xferd Average Speed Time Time Time Current
                                  Dload Upload Total Spent Left Speed
   0 14 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (22) The requested URL returned error: 404

Therefore the installation fails and does not allow you to continue with the tests.

In order to finish the tests, an edition of the wizard will be made to use the v4.7.4 tag instead of the 4.7.4 branch (this change will not be incorporated into the code after the tests), since when using the tests The mentioned branch should exist since it is deleted after publishing the packages.

---

New test

Test manager with param live False 🟢

• Input yaml:

version: 0.1
description: This workflow is used to test manager deployment for DDT1 PoC
variables:
  manager-os:
    - linux-centos-7-amd64
    - linux-centos-8-amd64
  infra-provider: vagrant
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique manager allocate task
  - task: "allocate-manager-{manager}"
    description: "Allocate resources for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{manager}"
          - instance-name: "{manager}"
          - inventory-output: "{working-dir}/manager-{manager}/inventory.yaml"
          - track-output: "{working-dir}/manager-{manager}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: manager-os
        as: manager

  # Generic manager test task
  - task: "run-manager-tests"
    description: "Run tests install for the manager."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/manager-linux-centos-7-amd64/inventory.yaml"
            - wazuh-2: "{working-dir}/manager-linux-centos-8-amd64/inventory.yaml"
          - tests: "install"
          - component: "manager"
          - wazuh-version: "4.7.4"
          - wazuh-revision: "40717"
          - live: False
    depends-on:
      - "allocate-manager-linux-centos-7-amd64"
      - "allocate-manager-linux-centos-8-amd64"

• Log:

test-manager-live-false.log

Test central-components with param live False 🟢

• Input yaml:

version: 0.1
description: This workflow is used to test the Wazuh manager deployment for DDT1 PoC
variables:
  central_components-os:
    - linux-ubuntu-22.04-amd64
  infra-provider: vagrant
  working-dir: /tmp/dtt1-poc

tasks:
  # Unique central components allocate task
  - task: "allocate-central_components-{central_components}"
    description: "Allocate resources for the central_components."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: create
          - provider: "{infra-provider}"
          - size: large
          - composite-name: "{central_components}"
          - instance-name: "{central_components}"
          - inventory-output: "{working-dir}/central_components-{central_components}/inventory.yaml"
          - track-output: "{working-dir}/central_components-{central_components}/track.yaml"
          - label-termination-date: "1d"
          - label-team: "qa"
    on-error: "abort-all"
    foreach:
      - variable: central_components-os
        as: central_components
    cleanup:
      this: process
      with:
        path: python3
        args:
          - modules/allocation/main.py
          - action: delete
          - track-output: "{working-dir}/central_components-{central_components-os}/track.yaml"

  # Generic manager test task
  - task: "run-central_components-{central_components}-tests"
    description: "Run tests install for the central_components."
    do:
      this: process
      with:
        path: python3
        args:
          - modules/testing/main.py
          - targets:
            - wazuh-1: "{working-dir}/central_components-{central_components}/inventory.yaml"
          - tests: "install,restart,stop,uninstall"
          - component: "central_components"
          - wazuh-version: "4.7.4"
          - wazuh-revision: "40717"
          - live: False
    on-error: "abort-all"
    foreach:
      - variable: central_components-os
        as: central_components
    depends-on:
      - "allocate-central_components-{central_components}"

• Log:

test-central-componentes-live-false.log

[rauldpm]

LGTM