Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replication issues #105

Closed
asow25 opened this issue Apr 16, 2023 · 110 comments
Closed

Replication issues #105

asow25 opened this issue Apr 16, 2023 · 110 comments

Comments

@asow25
Copy link

asow25 commented Apr 16, 2023

I've got the issues with replication
When I create a zone on the master, the zones do not replicate on the slave. With php admin, I see my zones in powerdns, powernds-admin but in powerdns-slave > Domains, there is nothing. Is this normal? I would like it to replicate?

@pschiffe
Copy link
Owner

Try to see if there's an error in the master or slave logs. Do you have set supermaster correctly? https://github.com/pschiffe/docker-pdns/blob/master/docker-compose.yml#L88

@asow25
Copy link
Author

asow25 commented Apr 19, 2023

I respected what you did! I use swarm cluster :

version: '3.8'

services:

  pdns-recursor:
    image: pschiffe/pdns-recursor:${RECURSOR_TAG:-latest}
    networks:
      - pdns
    volumes:
      - /etc/localtime:/etc/localtime:ro

  mariadb:
    image: mariadb:10.10
    networks:
      pdns:
        aliases:
          - db
          - mysql
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/dns/mariadb:/var/lib/mysql:z
    environment:
      - MYSQL_ROOT_PASSWORD=my-secret-pw
    healthcheck:
      test: ['CMD', 'mysqladmin', 'ping', '-h', 'localhost', '-pmy-secret-pw']
      timeout: 10s
      retries: 5

  phpmyadmin:
    image: phpmyadmin:5
    networks:
      - pdns
    ports:
      - '8988:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.185
        aliases:
          - pdns
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_allow_axfr_ips=192.168.100.186,192.168.100.187
      - PDNS_only_notify=192.168.100.186,192.168.100.187
    depends_on:
      - mariadb

  pdns-slave1:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns2.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.186
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave1
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master

  pdns-slave2:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns3.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.187
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave2
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master

  pdns-admin-uwsgi:
    image: pschiffe/pdns-admin-uwsgi
    networks:
      pdns:
        aliases:
          - pdns-admin-uwsgi
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_ADMIN_SQLA_DB_PASSWORD='my-secret-pw'
      - PDNS_VERSION=4.7
      - PDNS_API_KEY=secret
    depends_on:
      - mariadb
      - pdns-master

  pdns-admin-static:
    image: pschiffe/pdns-admin-static
    networks:
      - pdns
    ports:
      - '80:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - pdns-admin-uwsgi

networks:
  pdns:
    ipam:
      config:
        - subnet: 192.168.100.0/24

@pschiffe
Copy link
Owner

Thank you; could you also share logs from master and slave containers, from the start of the containers to after you create the zone? If there's an error in communication, it should be visible there.

@asow25
Copy link
Author

asow25 commented Apr 19, 2023

Thank you for your help :

# pdns-master from the start
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Created local state directory '/var/run/pdns/'
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Listening on controlsocket in '/var/run/pdns/pdns.controlsocket'
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Guardian is launching an instance
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Loading '/usr/lib64/pdns/libgmysqlbackend.so'
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 This is a guarded instance of pdns
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 UDP server bound to 0.0.0.0:53
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 UDP server bound to [::]:53
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 TCP server bound to 0.0.0.0:53
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 TCP server bound to [::]:53
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 PowerDNS Authoritative Server 4.7.2 (C) 2001-2022 PowerDNS.COM BV
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Using 64-bits mode. Built using gcc 12.2.1 20220819 (Red Hat 12.2.1-2) on Nov  1 2022 00:00:00 by mockbuild@localhost.
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 [webserver] Listening for HTTP requests on 0.0.0.0:8081

dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Polled security status of version 4.7.2 at startup, no known issues reported: OK
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Creating backend connection for TCP
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Primary/secondary communicator launching
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 About to create 3 backend threads for UDP

dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:20:30 Done launching threads, ready to distribute questions


# pdns-slave1 from the starting
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Created local state directory '/var/run/pdns/'
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Listening on controlsocket in '/var/run/pdns/pdns.controlsocket'
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Guardian is launching an instance
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Loading '/usr/lib64/pdns/libgmysqlbackend.so'
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 This is a guarded instance of pdns
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 UDP server bound to 0.0.0.0:53
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 UDP server bound to [::]:53
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 TCP server bound to 0.0.0.0:53
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 TCP server bound to [::]:53
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 PowerDNS Authoritative Server 4.7.2 (C) 2001-2022 PowerDNS.COM BV
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Using 64-bits mode. Built using gcc 12.2.1 20220819 (Red Hat 12.2.1-2) on Nov  1 2022 00:00:00 by mockbuild@localhost.
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 PowerDNS comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it according to the terms of the GPL version 2.
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 [webserver] Listening for HTTP requests on 0.0.0.0:8081
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Polled security status of version 4.7.2 at startup, no known issues reported: OK
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Creating backend connection for TCP
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Primary/secondary communicator launching
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 About to create 3 backend threads for UDP
dns-server_pdns-slave1.1.8drwkuex4cwc@srv-manager-pdns-1    | Apr 19 13:20:30 Done launching threads, ready to distribute questions

@asow25
Copy link
Author

asow25 commented Apr 19, 2023

I think this is the mistake

# pdns-master from creating zone
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 13:30:30 Request to queue notification for domain 'new-domain.com' was processed, but no valid nameservers or ALSO-NOTIFYs found. Not notifying!


# pdns-slave1 from zone zone : nothing

@pschiffe
Copy link
Owner

After you create a new zone, you also need to add NS records with all ns servers. Also the domain type must be set as PRIMARY (not native):
Screenshot 2023-04-19 at 15-58-54 Zone Records - domain com - PowerDNS-Admin

Once you add the NS records, wait 1 or 2 minutes and check the master and slave logs, if it's still not working, post them here.

@asow25
Copy link
Author

asow25 commented Apr 19, 2023

It's all the same, I have done as you said but my zone has not spread to the slaves and there are no new logs yet.

# pdns-master
dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 15:09:31 Request to queue notification for domain 'new-domain.com' was processed, but no valid nameservers or ALSO-NOTIFYs found. Not notifying!

Screenshot from 2023-04-19 15-29-37

@asow25
Copy link
Author

asow25 commented Apr 19, 2023

Screenshot from 2023-04-19 15-58-35

@pschiffe
Copy link
Owner

That's weird, I'm not sure here. Try to create a new random record in the new-domain.com. The master should try to notify the change (there may be 2 minutes delay), and some log should appear in the master container.

@asow25
Copy link
Author

asow25 commented Apr 19, 2023

dns-server_pdns-master.1.o817wv96kny1@srv-manager-pdns-2    | Apr 19 16:35:14 Request to queue notification for domain 'new-domain.com' was processed, but no valid nameservers or ALSO-NOTIFYs found. Not notifying!

@pschiffe
Copy link
Owner

Can you verify connectivity between the containers? ie exec to ns1, and try curl -v ns2.devops.com:53 (this is stupid, but curl is installed and it will connect to that port if it's possible).

Also you can try to provide the master env var PDNS_also_notify=192.168.100.186,192.168.100.187 to force the notification to these IPs (if the NS records don't work).

@asow25
Copy link
Author

asow25 commented Apr 20, 2023

I added this line PDNS_also_notify=192.168.100.186,192.168.100.187 The database restarts several times. Then it recovers on another node of my cluster.

# pdns-master
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 07:59:35 Notification for devops.ci to 192.168.100.186:53 failed after retries
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 07:59:35 Notification for devops.ci to 192.168.100.187:53 failed after retries
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 08:00:28 gmysql Connection failed: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 08:00:28 Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 08:00:28 Cleaning up
dns-server_pdns-master.1.rtvrzplmy1jl@srv-manager-pdns-3    | Apr 20 08:00:28 PDNSException while updating zone cache: Unable to launch gmysql connection: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
# pdns-slave
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:49 gmysql Connection failed: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:49 Caught an exception instantiating a backend: Unable to launch gmysql connection: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:49 Cleaning up
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:49 PDNSException while filling the zone cache: Unable to launch gmysql connection: Unable to connect to database: ERROR 2005 (HY000): Unknown server host 'mysql' (-2)
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:50 Our pdns instance exited with code 1, respawning
dns-server_pdns-slave1.1.so7bvttrag7g@srv-manager-pdns-2    | Apr 20 08:04:51 Guardian is launching an instance

@asow25
Copy link
Author

asow25 commented Apr 20, 2023

The stdout output of the docker service ls command.

ID             NAME                           MODE         REPLICAS   IMAGE                               PORTS
pwceb33wior4   dns-server_mariadb             replicated   0/1        mariadb:10.10                       
9nclkfa32tk0   dns-server_pdns-admin-static   replicated   1/1        pschiffe/pdns-admin-static:latest   *:80->80/tcp
qb0nyyx72jje   dns-server_pdns-admin-uwsgi    replicated   1/1        pschiffe/pdns-admin-uwsgi:latest    
fxjrpso0j3v3   dns-server_pdns-master         replicated   1/1        pschiffe/pdns-mysql:latest          
l2lw4mx1ogqe   dns-server_pdns-recursor       replicated   1/1        pschiffe/pdns-recursor:latest       
u3ktxk6wxew3   dns-server_pdns-slave1         replicated   1/1        pschiffe/pdns-mysql:latest          
xjukho1ofrp7   dns-server_pdns-slave2         replicated   1/1        pschiffe/pdns-mysql:latest          
tyhffm1ei653   dns-server_phpmyadmin          replicated   1/1        phpmyadmin:5                        *:8988->80/tcp
5y6kqp0en4n0   portainer                      replicated   1/1        portainer/portainer-ce:latest       *:9000->9000/tcp

@pschiffe
Copy link
Owner

Notification for devops.ci to 192.168.100.186:53 failed after retries seems like the containers cannot reach each other. I'm sorry, I'm not that familiar with docker swarm. You can try that curl from the master container (or install dig - dnf install bind-utils) and try to reach the slaves. If that doesn't work, there's something either with the docker swarm network configuration, or the docker-compose file must be adjusted for docker swarm, I'm not sure.

@asow25
Copy link
Author

asow25 commented Apr 20, 2023

You were right, it is the network that is the problem.

# pdns-master
curl -v ns2.devops.com:53
* Trying 192.168.100.186:53...
* connect to 192.168.100.186 port 53 failed : No route to host
* Failed to connect to ns2.devops.com port 53 after 3068 ms : No route to host
* Closing of the connection 0
curl: (7) Connection to ns2.devops.com port 53 failed after 3068 ms: No route to host

# pdns-slave
curl -v ns1.devops.com:53
* Trying 192.168.100.185:53...
* Connection to 192.168.100.185 port 53 failed: No route to host
* Failed to connect to ns1.devops.com port 53 after 3064 ms: No route to host
* Connection closure 0
curl: (7) Connection to ns1.devops.com port 53 failed after 3064 ms: No route to host

@asow25
Copy link
Author

asow25 commented Apr 20, 2023

# pdns-master
cat /etc/hosts 
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.100.185 ns1.devops.com
192.168.100.186 ns2.devops.com
192.168.100.187 ns3.devops.com
192.168.100.24 ns2.devops.com ns2


# pdn-slave
cat /etc/hosts 
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
192.168.100.185 ns1.devops.com
192.168.100.186 ns2.devops.com
192.168.100.187 ns3.devops.com
192.168.100.21 ns1.devops.com ns1

I actually notice that my services are not getting the ip I assigned in the YAML, there is 192.168.100.24 ns2.devops.com ns2 and 192.168.100.21 ns1.devops.com ns1
And when I redo the curl but with these ip, it works

# pdns-master
curl -v 192.168.100.24:53
* Trying 192.168.100.24:53...
* Connected to 192.168.100.24 (192.168.100.24) port 53 (#0)
> GET / HTTP/1.1
> Host: 192.168.100.24:53
> User-Agent: curl/7.85.0
> Accept: */*
> 
* Empty server response
* Closing the connection 0
curl : (52) Empty server response

# pdns-slave
curl -v 192.168.100.21:53
* Trying 192.168.100.21:53...
* Connected to 192.168.100.21 (192.168.100.21) port 53 (#0)
> GET / HTTP/1.1
> Host: 192.168.100.21:53
> User-Agent: curl/7.85.0
> Accept: */*
> 
* Empty server response
* Closing the connection 0
curl: (52) Empty server response

@pschiffe
Copy link
Owner

Now I see that it's not possible to use static IPs for containers in docker swarm:
moby/moby#24170

@pschiffe
Copy link
Owner

At least supermaster feature requires static IP, and axfr transfers probably as well; I don't see right now how this could be work-arounded for swarm :/

@asow25
Copy link
Author

asow25 commented Apr 21, 2023

How are you? Thanks for your help, I've been looking for how to define static ip's to services with swarm, but I couldn't find anything.

I tried to modify extra_hosts by removing the ip's and I kept the hostnames:

    extra_hosts:
      - 'ns1.devops.com'
      - 'ns2.devops.com'
      - 'ns3.devops.com'

I can do curl inside the containers. My problem is that I have to fill in the fields PDNS_also_notify PDNS_allow_axfr_ips SUPERMASTER_IPS.

Is it possible to proceed in this way:

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.185
        aliases:
          - pdns
    extra_hosts:
      - 'ns1.devops.com'
      - 'ns2.devops.com'
      - 'ns3.devops.com'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_also_notify=ns2.devops.com,ns3.devops.com
      - PDNS_allow_axfr_ips=ns2.devops.com,ns3.devops.com
      - PDNS_only_notify=ns2.devops.com,ns3.devops.com
    depends_on:
      - mariadb

  pdns-slave1:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns2.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.186
    extra_hosts:
      - 'ns1.devops.com'
      - 'ns2.devops.com'
      - 'ns3.devops.com'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave1
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=ns1.devops.com
      - SUPERMASTER_IPS=ns1.devops.com
    depends_on:
      - mariadb
      - pdns-master

  pdns-slave2:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns3.devops.com
    networks:
      pdns:
        ipv4_address: 192.168.100.187
    extra_hosts:
      - 'ns1.devops.com'
      - 'ns2.devops.com'
      - 'ns3.devops.com'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave2
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=ns1.devops.com
      - SUPERMASTER_IPS=ns1.devops.com
    depends_on:
      - mariadb
      - pdns-master

All services start except pdns-admin-static

dns_pdns-admin-static.1.goqydtm2elwt@srv-manager-pdns-1 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48
dns_pdns-admin-static.1.obr031430xkr@srv-manager-pdns-1 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48
dns_pdns-admin-static.1.jl85lkhrbdo2@srv-manager-pdns-1 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48
dns_pdns-admin-static.1.v0j7dggdmyw4@srv-manager-pdns-3 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48
dns_pdns-admin-static.1.yk8rrmofmfr4@srv-manager-pdns-1 | nginx : [emerg] host not found in upstream "pdns-admin-uwsgi" in /etc/nginx/nginx.conf:48

@pschiffe
Copy link
Owner

hi @asow25, I'm good, thank you. The error from static container means that pdns-admin-uwsgi doesn't resolve to the IP of uwsgi container. In the original docker-compose, this works thanks to the alias: https://github.com/pschiffe/docker-pdns/blob/master/docker-compose.yml#L98

Reg. the options, the PDNS_also_notify shouldn't be needed, that was a try to force the updates to slaves, but the original issue was the connection problems.
PDNS_allow_axfr_ips must be IPs or subnets, but this can be set to the whole subnet. It's a security feature to not broadcast transfers to the internet. So PDNS_allow_axfr_ips=192.168.100.0/24 should work in your case.
PDNS_only_notify is also a security feature, and can be set the same way as allow axfr ips: PDNS_only_notify=192.168.100.0/24
The same applies to PDNS_allow_notify_from as well: PDNS_allow_notify_from=192.168.100.0/24
Documentation to these options is here: https://doc.powerdns.com/authoritative/settings.html

Problem is with the SUPERMASTER IPS, there must be an IP: https://doc.powerdns.com/authoritative/modes-of-operation.html#autoprimary-automatic-provisioning-of-secondaries

The slave replication can work without SUPERMASTER feature, but you have to provision the domain on slaves manually (with NS records). pdnsutil cli command is available in the containers for this:
https://doc.powerdns.com/authoritative/manpages/pdnsutil.1.html#zone-manipulation-commands
https://makarainen.net/PowerDNS-pdnsutil-cheat-sheet
You could also create the mysql records directly in DB too. This could work if you don't have too many zones (domains) I guess.

The third option can be to don't use pdns replication, but use mysql replication instead. That's a supported mode of operation: https://doc.powerdns.com/authoritative/modes-of-operation.html#native-replication
Also, if you are using only the one mysql server, all pdns masters and slaves can share the single mysql database, and if the zone type of your zones will be Native, I think you are good to go as well.

Otherwise I'm not sure, as the static IPs are not supported for docker swarm: moby/moby#24170 and we are dealing with DNS here 😅

@asow25
Copy link
Author

asow25 commented Apr 22, 2023

I managed to set static ip to pdns-master and pdns-slave1 and pdns-slave2.

# create mcvlan network on each node pdns-master, pdns-slave1 and pdns-slave2

docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.185/32 pdns-config

docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.186/32 pdns-config

docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.187/32 pdns-config

docker network create -d macvlan --scope swarm --config-from pdns-config --attachable pdns

and a second network for the rest of the services

docker network create -d overlay --subnet 192.168.100.0/24 --gateway 192.168.100.2 --ip-range 192.168.100.0/24 --attachable pdns_net

At first, I had errors that it was the same gateway so I changed it to 192.168.100.2

When I run the stack, it's Ok but when I connect to php-admin the databases of pdns-master, pdns-slave1 and pdns-slave2 were not created, so I looked at the logs of the master and :

dns-server_pdns-master.1.ym0l0yov8kkw@ans1 | MySQL is unavailable - sleeping
dns-server_pdns-master.1.ym0l0yov8kkw@ans1 | ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (-3)
dns-server_pdns-master.1.ym0l0yov8kkw@ans1 | MySQL is unavailable - sleeping
dns-server_pdns-master.1.ym0l0yov8kkw@ans1 | ERROR 2005 (HY000): Unknown MySQL server host 'mysql' (-3)

this means my two networks are not talking to each other but overlay is the swarm network that allows multi-data communication. Anyway, I added an alias

  pdns-master:
  ...
    networks:
      pdns:
        aliases:
          - pdns
          

  pdns-slave1:
  ...
    networks:
      pdns:
        aliases:
          - pdns
          

  pdns-slave2:
  ...
    networks:
      pdns:
        aliases:
          - pdns

networks:
  pdns:
    external: true
  pdns_net:
    external: true      

I still have the same error, my two networks do not communicate so the databases of these three services were not created when I deploy the stack. Can you help me please ?

@pschiffe
Copy link
Owner

I think 2 networks should be fine, but use different subnets, as this will most likely clash. You can attach container to multiple networks. Also alias must be different for every container.

  pdns-master:
  ...
    networks:
      pdns:
        aliases:
          - pdns-master
      pdns_net:
          

  pdns-slave1:
  ...
    networks:
      pdns:
        aliases:
          - pdns-slave1
      pdns_net:

  pdns-slave2:
  ...
    networks:
      pdns:
        aliases:
          - pdns-slave2
      pdns_net:

networks:
  pdns:
    external: true
  pdns_net:
    external: true    

@asow25
Copy link
Author

asow25 commented Apr 22, 2023

It's all the same, Please take a look!

version: '3.8'

services:

  pdns-recursor:
    image: pschiffe/pdns-recursor:${RECURSOR_TAG:-latest}
    networks:
      - pdns_net
    volumes:
      - /etc/localtime:/etc/localtime:ro

  mariadb:
    image: mariadb:10.10
    networks:
      pdns_net:
        aliases:
          - db
          - mysql
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - mariadb:/var/lib/mysql:z
    environment:
      - MYSQL_ROOT_PASSWORD=my-secret-pw
    healthcheck:
      test: ['CMD', 'mysqladmin', 'ping', '-h', 'localhost', '-pmy-secret-pw']
      timeout: 10s
      retries: 5

  phpmyadmin:
    image: phpmyadmin:5
    networks:
      - pdns_net
    ports:
      - '8988:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns:
        aliases:
          - pdns-master
      pdns_net:
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_also_notify=192.168.100.186,192.168.100.187
      - PDNS_allow_axfr_ips=192.168.100.186,192.168.100.187
      - PDNS_only_notify=192.168.100.186,192.168.100.187
    depends_on:
      - mariadb
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == 94ymd6l9cg9njxwqqq71i4rvn
      restart_policy:
        condition: on-failure

  pdns-slave1:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns2.devops.com
    networks:
      pdns:
        aliases:
          - pdns-slave1
      pdns_net:
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave1
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == re89fo9grvr9zx6pi3prxdypp
      restart_policy:
        condition: on-failure

  pdns-slave2:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns3.devops.com
    networks:
      pdns:
        aliases:
          - pdns-slave2
      pdns_net:
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave2
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == 49v6r3dk26i1oteub2vf9buy8
      restart_policy:
        condition: on-failure

  pdns-admin-uwsgi:
    image: pschiffe/pdns-admin-uwsgi
    networks:
      pdns_net:
        aliases:
          - pdns-admin-uwsgi
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_ADMIN_SQLA_DB_PASSWORD='my-secret-pw'
      - PDNS_VERSION=4.7
      - PDNS_API_KEY=secret
    depends_on:
      - mariadb
      - pdns-master

  pdns-admin-static:
    image: pschiffe/pdns-admin-static
    networks:
      - pdns_net
    ports:
      - '80:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - pdns-admin-uwsgi

networks:
  pdns:
    external: true
  pdns_net:
    external: true


volumes:
  mariadb:

@pschiffe
Copy link
Owner

hi @asow25, I'm sorry I cannot tell what could be the problem here. You can try to exec to the containers and try some network debugging to check if the networks are routing correctly.
Alternatively I see that you have node.id constraints for all pdns master and slaves... If they are running on different nodes, maybe you could just expose ports 53 tcp and udp to host and use connection between the nodes directly? Like outside of the docker virtual network, and use IPs of the hosts..

@asow25
Copy link
Author

asow25 commented Apr 24, 2023

Hi @pschiffe all my services are up and running, my 3 powerdns have static ip's and a second ip that allows them to communicate with other services. I checked the container logs, everything seems to be working fine except pdns-admin-uwsgi.

|pdns-server_pdns_uwsgi`` has been updated with the addition of a new module for managing access to the Internet and a new module for managing access to the Internet and a new module for managing access to the Internet and a new module for managing access to the Internet.
dns-server_pdns-admin-uwsgi.1 .o7m9lzjqqb43@srv-manager-pdns-2 | [pid: 133|app: 0|req: 43/43] 10.0.0.2 () {46 vars in 1127 bytes} [Mon Apr 24 07:13:05 2023] GET /user/image?username=pdns => generated 0 bytes in 15 msec (HTTP/1.1 304) 4 headers in 284 bytes (0 switches on core 0)
dns-server_pdns-admin-uwsgi.1 .o7m9lzjqqb43@srv-manager-pdns-2 | [pid: 133|app: 0|req: 44/44] 10.0.0.2 () {46 vars in 1081 bytes} [Mon Apr 24 07:14:30 2023] GET /ping?_=1682320384994 => generated 2 bytes in 48 msec (HTTP/1.1 200) 3 headers in 193 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1.o7m9lzjqqb43@srv-manager-pdns-2 | [pid: 133|app: 0|req: 45/45] 10.0.0.2 () {46 vars in 1140 bytes} [Mon Apr 24 07:14:32 2023] GET /domain/add => generated 31581 bytes in 98 msec (HTTP/1.1 200) 5 headers in 378 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1 .o7m9lzjqqb43@srv-manager-pdns-2 | [pid: 133|app: 0|req: 46/46] 10.0.0.2 () {44 vars in 1072 bytes} [Mon Apr 24 07:14:32 2023] GET /user/image?username=pdns => generated 2718 bytes in 14 msec via sendfile() (HTTP/1.1 200) 7 headers in 367 bytes (0 switches on core 0)
dns-server_pdns-admin-uwsgi.1.o7m9lzjqqb43@srv-manager-pdns-2 | [2023-04-24 07:14:49,498] [domain.py:281] ERROR - Cannot add zone asow.ci. HTTPConnectionPool(host='pdns', port=8081) : Max retries exceeded with url : /api/v1/servers/localhost/zones (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcf14be8f90> : Failed to establish a new connection : [Errno -2] Name or service not known')

When I try to create a zone, I get an error : HTTP 404 Error

@pschiffe
Copy link
Owner

If the pdns-master is not available under pdns alias anymore, you can configure pdns-admin-uwsgi with PDNS_API_URL="http://pdns-master:8081/" or PDNS_API_URL="http://ns1.devops.com:8081/" env var (whichever will work). Also IP should work there too.

@asow25
Copy link
Author

asow25 commented Apr 24, 2023

dns-server_pdns-admin-uwsgi.1.0l5mggn5odp5@srv-manager-pdns-2    | [2023-04-24 16:37:50,832] [server.py:39] ERROR - Can not get server configuration. DETAIL: Invalid URL '/api/v1/servers/localhost/config': No scheme supplied. Perhaps you meant https:///api/v1/servers/localhost/config?
dns-server_pdns-admin-uwsgi.1.0l5mggn5odp5@srv-manager-pdns-2    | [pid: 55|app: 0|req: 16/16] 10.0.0.2 () {46 vars in 1164 bytes} [Mon Apr 24 16:37:50 2023] GET /admin/server/configuration => generated 19278 bytes in 109 msecs (HTTP/1.1 200) 3 headers in 197 bytes (1 switches on core 0)
dns-server_pdns-admin-uwsgi.1.0l5mggn5odp5@srv-manager-pdns-2    | [pid: 55|app: 0|req: 17/17] 10.0.0.2 () {46 vars in 1135 bytes} [Mon Apr 24 16:37:51 2023] GET /user/image?username=pdns => generated 0 bytes in 58 msecs (HTTP/1.1 304) 4 headers in 284 bytes (0 switches on core 0)

@pschiffe
Copy link
Owner

The value must include full URL, with http://, it's an address for pdns API: PDNS_API_URL="http://ns1.devops.com:8081/"
You can check this value also in pdns admin web interface, in settings, to see if it's set correctly.

@asow25
Copy link
Author

asow25 commented Apr 24, 2023

It's the same, honestly, I don't understand, when I fix a problem, another one is created. Before everything worked, but the ip were not fixed so impossible that a zone is propagated to other servers powernds since swarm manages itself the allocation of ip addresses. I found this method:

# node master
docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.185/32 pdns-config

# node slave1
docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.186/32 pdns-config

# node slave2
docker network create --config-only --subnet 192.168.100.0/24 --gateway 192.168.100.1 --ip-range 192.168.100.187/32 pdns-config

# node master
docker network create -d macvlan --scope swarm --config-from pdns-config --attachable pdns

Now the ip's are fixed and I have added another common network for all the services to communicate.

docker network create -d overlay --attachable pdns_net

Then, database problem, it's fixed, now it's the turn of pdns-admin-uwsgi
please take a last look at my YAML and give me your opinion.

version: '3.8'

services:

  pdns-recursor:
    image: pschiffe/pdns-recursor:${RECURSOR_TAG:-latest}
    networks:
      - pdns_net
    volumes:
      - /etc/localtime:/etc/localtime:ro

  mariadb:
    image: mariadb:10.10
    networks:
      pdns_net:
        aliases:
          - db
          - mysql
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /mnt/gfs/mariadb:/var/lib/mysql:z
    environment:
      - MYSQL_ROOT_PASSWORD=my-secret-pw
    healthcheck:
      test: ['CMD', 'mysqladmin', 'ping', '-h', 'localhost', '-pmy-secret-pw']
      timeout: 10s
      retries: 5

  phpmyadmin:
    image: phpmyadmin:5
    networks:
      - pdns_net
    ports:
      - '8988:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns_net:
      pdns:
        aliases:
          - pdns-master
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_master=yes
      - PDNS_api=yes
      - PDNS_api_key=secret
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_default_ttl=1500
      - PDNS_also_notify=192.168.100.186,192.168.100.187
      - PDNS_allow_axfr_ips=192.168.100.186,192.168.100.187
      - PDNS_only_notify=192.168.100.186,192.168.100.187
    depends_on:
      - mariadb
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == 2x07fshor56uthrnj20ew7y3m
      restart_policy:
        condition: on-failure
    ports:
      - '53:53'
      - '53:53/udp'
      - '8081:8081'
      
  pdns-slave1:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns2.devops.com
    networks:
      pdns_net:
      pdns:
        aliases:
          - pdns-slave1
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave1
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == sp4oa4ev5kuj0a6efqx1by0w9
      restart_policy:
        condition: on-failure

  pdns-slave2:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns3.devops.com
    networks:
      pdns_net:
      pdns:
        aliases:
          - pdns-slave2
    extra_hosts:
      - 'ns1.devops.com:192.168.100.185'
      - 'ns2.devops.com:192.168.100.186'
      - 'ns3.devops.com:192.168.100.187'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_gmysql_dbname=powerdnsslave2
      - PDNS_gmysql_password=my-secret-pw
      - PDNS_slave=yes
      - PDNS_superslave=yes
      - PDNS_webserver=yes
      - PDNS_webserver_address=0.0.0.0
      - PDNS_webserver_allow_from=192.168.100.0/24
      - PDNS_version_string=anonymous
      - PDNS_disable_axfr=yes
      - PDNS_allow_notify_from=192.168.100.185
      - SUPERMASTER_IPS=192.168.100.185
    depends_on:
      - mariadb
      - pdns-master
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.id == lvjwq59t0gd9hs13b4ytol7uw
      restart_policy:
        condition: on-failure

  pdns-admin-uwsgi:
    image: pschiffe/pdns-admin-uwsgi
    networks:
      pdns_net:
        aliases:
          - pdns-admin-uwsgi
    volumes:
      - /etc/localtime:/etc/localtime:ro
    environment:
      - PDNS_ADMIN_SQLA_DB_PASSWORD='my-secret-pw'
      - PDNS_API_URL="http://pdns-master:8081/"
      - PDNS_VERSION=4.7
      - PDNS_API_KEY=secret
    depends_on:
      - mariadb
      - pdns-master

  pdns-admin-static:
    image: pschiffe/pdns-admin-static
    networks:
      - pdns_net
    ports:
      - '80:80'
    volumes:
      - /etc/localtime:/etc/localtime:ro
    depends_on:
      - pdns-admin-uwsgi

networks:
  pdns:
    external: true
  pdns_net:
    external: true

@pschiffe
Copy link
Owner

I'm so sorry to hear that :(
I do see one issue in the yaml - pdns-admin-uwsgi container is connected to the pdns_net, but the pdns-master container has alias on the pdns network. Try to move the alias to pdns_net or add it to both (not sure if that will work):

  pdns-master:
    image: pschiffe/pdns-mysql:${PDNS_MYSQL_TAG:-latest}
    hostname: ns1.devops.com
    networks:
      pdns_net:
        aliases:
          - pdns-master
      pdns:
        aliases:
          - pdns-master

It's true that docker swarm doesn't have very good reputation, maybe you could take a look at some simple kubernetes distro, such as https://k3s.io/

@pschiffe
Copy link
Owner

pschiffe commented May 6, 2023

hi @asow25, updated images have been pushed to the registry, please update and try again. the issues with quotes should be hopefully resolved now

@asow25
Copy link
Author

asow25 commented May 7, 2023

Many thanks! One question, which syntax should I use, '"3306"' or "'3306'"

@asow25
Copy link
Author

asow25 commented May 9, 2023

Thanks a lot, it works, I have two zones on pdns-admin. The zones are on my galera nodes
Screenshot from 2023-05-09 08-05-50

@asow25
Copy link
Author

asow25 commented May 9, 2023

I have a question. I have a pdns-master service with 3 replicas. Initially, I used ns1 , ns2 , and ns3 as the hostnames for these services, and I could provide them when creating a zone.

Now that I have only one pdns-master service, I started using NATIVE . How do I handle ns1 , ns2 , and ns3 in this case?

@pschiffe
Copy link
Owner

pschiffe commented May 9, 2023

Awesome, that's great to hear @asow25!

Many thanks! One question, which syntax should I use, '"3306"' or "'3306'"

It doesn't matter anymore, single quotes are fine now. Numbers can also be without quotes.

Now that I have only one pdns-master service, I started using NATIVE . How do I handle ns1 , ns2 , and ns3 in this case?

With Native, replication is handled with your galera cluster, so powerdns doesn't care anymore. Now, the NS records will be used just by DNS clients when resolving the domain. So it depends on how is your single pdns-master reachable. Is it only one domain? If so, that's fine and just use that single domain in one NS record of the hosted domains.

@asow25
Copy link
Author

asow25 commented May 9, 2023

Yes I am talking about client side, I had 02 theories:

  • if I decide to keep ns1.devops.com; ns2.devops.com, ns3.devops.com
  • if I define only one domain name

What is the procedure to follow ?

@pschiffe
Copy link
Owner

pschiffe commented May 9, 2023

If you have only one instance of pdns-master running, I think 1 is fine.

@asow25
Copy link
Author

asow25 commented May 9, 2023

I noticed that pdns-admin after several minutes, I have this error:
Screenshot from 2023-05-09 16-08-09

@pschiffe
Copy link
Owner

pschiffe commented May 9, 2023

Pls post the logs of uwsgi and nginx containers after you trigger the error.

@asow25
Copy link
Author

asow25 commented May 9, 2023

dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | [2023-05-09 18:31:59,389] [app.py:1449] ERROR - Exception on /logout [GET]
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | Traceback (most recent call last):
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     self.dialect.do_execute(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     cursor.execute(statement, parameters)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 206, in execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     res = self._query(query)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |           ^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 319, in _query
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     db.query(q)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/connections.py", line 254, in query
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     _mysql.connection.query(self, query)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | MySQLdb.OperationalError: (2013, 'Lost connection to server during query')
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | The above exception was the direct cause of the following exception:
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | Traceback (most recent call last):
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 2073, in wsgi_app
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     response = self.full_dispatch_request()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1519, in full_dispatch_request
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     rv = self.handle_user_exception(e)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1515, in full_dispatch_request
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     rv = self.preprocess_request()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |          ^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1857, in preprocess_request
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     rv = self.ensure_sync(before_func)()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/opt/powerdns-admin/powerdnsadmin/routes/index.py", line 70, in before_request
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     maintenance = Setting().get('maintenance')
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/opt/powerdns-admin/powerdnsadmin/models/setting.py", line 563, in get
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     result = self.query.filter(Setting.name == setting).first()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3429, in first
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     ret = list(self[0:1])
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                ~~~~^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return list(res)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return self._execute_and_instances(context)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     result = conn.execute(querycontext.statement, self._params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return meth(self, multiparams, params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return connection._execute_clauseelement(self, multiparams, params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     ret = self._execute_context(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |           ^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     self._handle_dbapi_exception(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     util.raise_(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     raise exception
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     self.dialect.do_execute(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     cursor.execute(statement, parameters)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 206, in execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     res = self._query(query)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |           ^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 319, in _query
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     db.query(q)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/connections.py", line 254, in query
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     _mysql.connection.query(self, query)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2013, 'Lost connection to server during query')
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | [SQL: SELECT setting.id AS setting_id, setting.name AS setting_name, setting.value AS setting_value 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | FROM setting 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | WHERE setting.name = %s 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |  LIMIT %s]
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | [parameters: ('maintenance', 1)]
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | (Background on this error at: http://sqlalche.me/e/13/e3q8)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | Traceback (most recent call last):
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     self.dialect.do_execute(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     cursor.execute(statement, parameters)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 206, in execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     res = self._query(query)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |           ^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 319, in _query
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     db.query(q)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/connections.py", line 254, in query
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     _mysql.connection.query(self, query)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | MySQLdb.OperationalError: (2013, 'Lost connection to server during query')
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | The above exception was the direct cause of the following exception:
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | Traceback (most recent call last):
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 2073, in wsgi_app
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     response = self.full_dispatch_request()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1519, in full_dispatch_request
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     rv = self.handle_user_exception(e)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1515, in full_dispatch_request
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     rv = self.preprocess_request()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |          ^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1857, in preprocess_request
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     rv = self.ensure_sync(before_func)()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/opt/powerdns-admin/powerdnsadmin/routes/index.py", line 70, in before_request
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     maintenance = Setting().get('maintenance')
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/opt/powerdns-admin/powerdnsadmin/models/setting.py", line 563, in get
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     result = self.query.filter(Setting.name == setting).first()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3429, in first
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     ret = list(self[0:1])
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                ~~~~^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return list(res)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return self._execute_and_instances(context)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     result = conn.execute(querycontext.statement, self._params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return meth(self, multiparams, params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return connection._execute_clauseelement(self, multiparams, params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     ret = self._execute_context(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |           ^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1316, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     self._handle_dbapi_exception(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     util.raise_(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     raise exception
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1276, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     self.dialect.do_execute(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/default.py", line 608, in do_execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     cursor.execute(statement, parameters)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 206, in execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     res = self._query(query)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |           ^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/cursors.py", line 319, in _query
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     db.query(q)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/lib64/python3.11/site-packages/MySQLdb/connections.py", line 254, in query
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     _mysql.connection.query(self, query)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2013, 'Lost connection to server during query')
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | [SQL: SELECT setting.id AS setting_id, setting.name AS setting_name, setting.value AS setting_value 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | FROM setting 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | WHERE setting.name = %s 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |  LIMIT %s]
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | [parameters: ('maintenance', 1)]
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | (Background on this error at: http://sqlalche.me/e/13/e3q8)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | During handling of the above exception, another exception occurred:
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | Traceback (most recent call last):
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1202, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     conn = self._revalidate_connection()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 469, in _revalidate_connection
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     raise exc.InvalidRequestError(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | sqlalchemy.exc.InvalidRequestError: Can't reconnect until invalid transaction is rolled back
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | The above exception was the direct cause of the following exception:
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | Traceback (most recent call last):
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 2091, in __call__
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return self.wsgi_app(environ, start_response)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/werkzeug/middleware/proxy_fix.py", line 187, in __call__
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return self.app(environ, start_response)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 2076, in wsgi_app
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     response = self.handle_exception(e)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                ^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 1432, in handle_exception
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     server_error = self.ensure_sync(handler)(server_error)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/opt/powerdns-admin/powerdnsadmin/routes/base.py", line 34, in handle_internal_server_error
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return render_template('errors/500.html', code=500, message=e), 500
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/templating.py", line 153, in render_template
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     ctx.app.update_template_context(context)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib/python3.11/site-packages/flask/app.py", line 750, in update_template_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     context.update(func())
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                    ^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/opt/powerdns-admin/powerdnsadmin/__init__.py", line 92, in inject_sitename
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     setting = Setting().get('site_name')
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/opt/powerdns-admin/powerdnsadmin/models/setting.py", line 563, in get
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     result = self.query.filter(Setting.name == setting).first()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3429, in first
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     ret = list(self[0:1])
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |                ~~~~^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return list(res)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return self._execute_and_instances(context)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     result = conn.execute(querycontext.statement, self._params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1011, in execute
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return meth(self, multiparams, params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     return connection._execute_clauseelement(self, multiparams, params)
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1124, in _execute_clauseelement
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     ret = self._execute_context(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |           ^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1206, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     self._handle_dbapi_exception(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1510, in _handle_dbapi_exception
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     util.raise_(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/util/compat.py", line 182, in raise_
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     raise exception
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 1202, in _execute_context
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     conn = self._revalidate_connection()
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |   File "/usr/local/lib64/python3.11/site-packages/sqlalchemy/engine/base.py", line 469, in _revalidate_connection
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |     raise exc.InvalidRequestError(
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | sqlalchemy.exc.StatementError: (sqlalchemy.exc.InvalidRequestError) Can't reconnect until invalid transaction is rolled back
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | [SQL: SELECT setting.id AS setting_id, setting.name AS setting_name, setting.value AS setting_value 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | FROM setting 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | WHERE setting.name = %s 
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    |  LIMIT %s]
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | [parameters: [immutabledict({})]]
dns-server_pdns-admin-uwsgi.1.y0xhuzyxp5zs@pdns-node-03    | [pid: 22|app: 0|req: 218/218] 10.0.0.2 () {46 vars in 933 bytes} [Tue May  9 18:31:59 2023] GET /logout => generated 0 bytes in 46 msecs (HTTP/1.1 500) 0 headers in 0 bytes (0 switches on core 0)

@asow25
Copy link
Author

asow25 commented May 9, 2023

When my services were all in replicas 1, there was no problem. I made a scale of the services pdns-master and pdns-recursor replicas=3

@pschiffe
Copy link
Owner

pschiffe commented May 9, 2023

It's all about MySQL connections - MySQLdb.OperationalError: (2013, 'Lost connection to server during query')
Do you have maybe some limits on MySQL side? There is limit for simultaneous connections in MySQL, maybe some other. It could explain why there's a problem with more containers, and more connections.
See - https://mariadb.com/kb/en/handling-too-many-connections/ and also check logs for galera and dblb

@asow25
Copy link
Author

asow25 commented May 9, 2023

I removed galera and dldb, I bump into https://github.com/colinmollenhour/mariadb-galera-swarm
It use two services, first, initialize and then add nodes to cluster.

@asow25
Copy link
Author

asow25 commented May 9, 2023

| 2023-05-05 17:43:54 140387251173120 [Warning] Aborted connection 10 to db: 'unconnected' user: 'system' host: '127.0.0.1' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 17:54:02 140387250558720 [Warning] Aborted connection 18 to db: 'powerdns' user: 'root' host: '175.0.7.31' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:14:04 140386845738752 [Warning] Aborted connection 42 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.31' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:14:05 140386845431552 [Warning] Aborted connection 41 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.38' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:17:29 140387251173120 [Warning] Aborted connection 17 to db: 'powerdns' user: 'root' host: '175.0.7.31' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:17:29 140386846045952 [Warning] Aborted connection 34 to db: 'powerdns' user: 'root' host: '175.0.7.40' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:17:29 140386846353152 [Warning] Aborted connection 33 to db: 'powerdns' user: 'root' host: '175.0.7.40' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:17:29 140387247548160 [Warning] Aborted connection 32 to db: 'powerdns' user: 'root' host: '175.0.7.38' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:17:29 140387250558720 [Warning] Aborted connection 31 to db: 'powerdns' user: 'root' host: '175.0.7.38' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:17:30 140387250251520 [Warning] Aborted connection 26 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.40' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:28:59 140386846045952 [Warning] Aborted connection 46 to db: 'powerdns' user: 'root' host: '175.0.7.40' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 18:56:51 140387251173120 [Warning] Aborted connection 45 to db: 'powerdns' user: 'root' host: '175.0.7.40' (Got an error reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 19:09:16 140387251173120 [Warning] Aborted connection 51 to db: 'powerdns' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 19:09:16 140386846045952 [Warning] Aborted connection 52 to db: 'powerdns' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-05 19:50:37 140386846045952 [Warning] Aborted connection 58 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.31' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09  7:47:40 140386846045952 [Warning] Aborted connection 397 to db: 'powerdns' user: 'root' host: '175.0.7.40' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09  8:17:33 140386846045952 [Warning] Aborted connection 403 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09  8:30:40 140386846045952 [Warning] Aborted connection 416 to db: 'powerdns' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09  8:30:40 140386845431552 [Warning] Aborted connection 417 to db: 'powerdns' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09  8:30:59 140386845738752 [Warning] Aborted connection 420 to db: 'powerdns' user: 'root' host: '175.0.7.31' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09  9:45:14 140386846045952 [Warning] Aborted connection 429 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09 10:20:45 140386846045952 [Warning] Aborted connection 452 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09 12:08:12 140386846045952 [Warning] Aborted connection 476 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09 12:18:13 140386846045952 [Warning] Aborted connection 480 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09 16:28:21 140386846045952 [Warning] Aborted connection 529 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09 16:38:22 140386845738752 [Warning] Aborted connection 535 to db: 'powerdnsadmin' user: 'root' host: '175.0.7.38' (Got timeout reading communication packets)
galera_node.1.nhg2su1wj6k5@pdns-node-01    | 2023-05-09 17:57:36 140387251173120 [Warning] Aborted connection 412 to db: 'powerdns' user: 'root' host: 'localhost' (Got timeout reading communication packets)

@asow25
Copy link
Author

asow25 commented May 9, 2023

MariaDB [(none)]> SHOW VARIABLES LIKE 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 100   |
+-----------------+-------+
1 row in set (0.00 sec)

@pschiffe
Copy link
Owner

It seems like mysql connections are idling. You can try to increase the max_connections variable, or even better decrease wait_timeout to 60 or 30 seconds, so the idle connections are closed faster and they don't take up space.

@asow25
Copy link
Author

asow25 commented May 10, 2023

I have executed the commands SET GLOBAL max_connections = 500; and SET GLOBAL wait_timeout = 60;
max_connections is now 500 for all, and I noticed wait_timeout, there are session level and at the global level

MariaDB [(none)]> SHOW VARIABLES LIKE 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 500   |
+-----------------+-------+
1 row in set (0.00 sec)

MariaDB [(none)]> SHOW VARIABLES LIKE 'wait_timeout';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| wait_timeout  | 28800 |
+---------------+-------+
1 row in set (0.00 sec)

MariaDB [(none)]> SHOW GLOBAL VARIABLES LIKE 'wait_timeout';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| wait_timeout  | 60    |
+---------------+-------+

@pschiffe
Copy link
Owner

I think that means that wait_timeout can be overridden by client, but those pdns clients shouldn't be doing that.
After you set GLOBAL wait_timeout, try to reconnect and run SHOW VARIABLES LIKE 'wait_timeout'; again. It should show the correct value I guess. If not, it's best to set it directly in config file (or via env vars to the mariadb container)

@asow25
Copy link
Author

asow25 commented May 10, 2023

I found this

        command: [
            '--wait_timeout=28800',
        ]

@pschiffe
Copy link
Owner

For the mariadb server? Feel free to try it.

@asow25
Copy link
Author

asow25 commented May 10, 2023

I looked at the official site of mariadb, I read the doc about wait_timeout

SET GLOBAL interactive_timeout=60;
SET GLOBAL wait_timeout=30;

The value has changed, even when I log out and log in again, I have launched pdns-admin, I am waiting to see

@asow25
Copy link
Author

asow25 commented May 10, 2023

Thank you so much, I can't thank you enough, you don't know anything about me and yet you give me your time, you help me a lot and that's very nice!

@pschiffe
Copy link
Owner

you are welcome ❤️

@asow25
Copy link
Author

asow25 commented May 10, 2023

I have the same mistakes

@pschiffe
Copy link
Owner

You can try to take a look in the pdns admin project, or ask there: https://github.com/search?q=repo%3APowerDNS-Admin%2FPowerDNS-Admin+Lost+connection+to+server+during+query&type=issues
You can also check open mysql connections on the mariadb server, I know they are visible in phpmyadmin, probably via some mysql cli command as well. Other than that, I'm not sure.

@asow25
Copy link
Author

asow25 commented May 10, 2023

Thank you, I will try

@asow25
Copy link
Author

asow25 commented Jun 6, 2023

Hi @pschiffe I have attempted to set up pdns on k8s, but I'm encountering issues with the configuration.

Jun 06 10:19:42 Created local state directory '/var/run/pdns/'
Jun 06 10:19:42 Listening on controlsocket in '/var/run/pdns/pdns.controlsocket'
Jun 06 10:19:42 Guardian is launching an instance
Jun 06 10:19:42 Loading '/usr/lib/pdns/pdns/libgmysqlbackend.so'
Jun 06 10:19:42 This is a guarded instance of pdns
Jun 06 10:19:42 Fatal error: Trying to set unknown setting 'default-soa-name'
Jun 06 10:19:43 Our pdns instance exited with code 1, respawning

When I comment out the line PDNS_default_soa_name, I encounter the same error for PDNS_default_ttl

Here's my manifest file

kind: ConfigMap
apiVersion: v1
metadata:
  name: master-config
data:
  PDNS_gmysql_host: "db"
  PDNS_gmysql_user: "root"
  PDNS_gmysql_port: "3306"
  PDNS_gmysql_dbname: "powerdns"
  PDNS_default_soa_name: "ns.sysops.com"
  PDNS_api: "yes"
  PDNS_webserver: "yes"
  PDNS_webserver_address: "0.0.0.0"
  PDNS_webserver_allow_from: "10.244.0.0/16"
  PDNS_version_string: anonymous
  PDNS_default_ttl: "1500"
 
---  
kind: Secret
apiVersion: v1
metadata:
  name: master-secret
data:
  apikey: MTIzNDU2Nzg5MA==
---  
kind: Service
apiVersion: v1
metadata:
  name: master-api
spec:
  ports:
  - port: 8081
    targetPort: 8081
  type: ClusterIP
  selector:
    app: pdns-master
---  
apiVersion: apps/v1
kind: Deployment
metadata:
  name: pdns-master
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pdns-master
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app: pdns-master
    spec:
      containers:
      - image: pschiffe/pdns-mysql:alpine
        name: pdns-master
        env:
        - name: PDNS_gmysql_password
          valueFrom:
            secretKeyRef:
              name: mysql-secret
              key: password
        - name: PDNS_api_key
          valueFrom:
            secretKeyRef:
              name: master-secret
              key: apikey
        envFrom:
        - configMapRef:
            name: master-config
        # resources:
        #   limits:
        #     cpu: "1"
        #     memory: "1Gi"
        #   requests:
        #     cpu: "0.5"
        #     memory: "512Mi"
        ports:
        - name: dns-udp
          containerPort: 53
          protocol: UDP
          hostPort: 53
        - name: dns-tcp
          containerPort: 53
          protocol: TCP
          hostPort: 53
        - containerPort: 8081
          protocol: TCP

@pschiffe
Copy link
Owner

pschiffe commented Jul 4, 2023

@asow25, is this also resolved?

@asow25 asow25 closed this as completed Aug 3, 2023
@asow25
Copy link
Author

asow25 commented Aug 3, 2023

@pschiffe, yes thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants