PANDAS 4

The fourth-generation of the PANDAS web archiving workflow system. While open source this is not yet documented or packaged for use outside our (NLA) infrastructure.

PANDAS provides a user interface for curators to perform website selection, collection building, scheduled crawling and quality assuarance using various web crawlers (Heritrix, Browsertrix, HTTrack).

External Dependencies

PANDAS requires the following tools to be installed:

• JDK 17
• Maven
• pywb
• Ghostscript
• Bamboo (the NLA web archive collection manager)
• An SQL database (see next section)

On RHEL/CentOS 7 install with:

yum install -y epel-release
yum install -y ghostscript httrack java-11-openjdk-devel python36 python36-devel python36-setuptools
pip install pywb

Database Support

PANDAS is known to work with Oracle, Postgresql and MariaDB. It may work with other databases that are supported by Hibernate and support sequences and recursive CTEs.

MySQL is assumed to currently not work due to the use of sequences. H2 is used for automated tests but is not recommended for production due to bugs in its CTE support.

Building and Running

Compile with:

mvn package

Run the package jar with:

java -jar ui/target/pandas-admin.jar

Configuration

Set the following environment variables:

## Database details
#PANDAS_DB_URL=
#PANDAS_DB_USER=
#PANDAS_DB_PASSWORD=

## Webapp details
# PORT=3001
# CONTEXT_PATH=/admin

## Path to store lucene indexes
#DATA_PATH=/tmp/data

## OpenID Connect authentication (optional)
#OIDC_URL=
#OIDC_CLIENT_ID=
#OIDC_CLIENT_SECRET=

## Browsertrix backend
#BROWSERTRIX_WORKERS=4
#BROWSERTRIX_PAGE_LIMIT=1000
#BROWSERTRIX_USER_AGENT_SUFFIX=

Keycloak integration

PANDAS can use Keycloak for single sign on. It may be possible to adapt it other OpenID Connect auth servers although account editing and access roles currently make use of Keycloak-specific features.

Versions known to work:

• Red Hat Single Sign-On 7.4
• Keycloak 14

Configure Keycloak in this way:

1. Create a new realm called 'pandas'.

2. Create a new client for each application (e.g. pandas-ui, pandas-gatherer, pandas-delivery, bamboo etc).

   • Access type: confidential
   • Standard Flow Enabled: on
   • Implicit Flow Enabled: off
   • Direct Access Grants Enabled: off
   • Service Accounts Enabled: on
   • Valid Redirect URIs: user-facing URL of the application followed by /*

   Press save.

3. Go to the Credentials tab and copy the client secret. In each app's own configuration set OIDC_URL to the realm URL (e.g. https://localhost:8443/auth/realms/pandas and OIDC_CLIENT_ID and OIDC_CLIENT_SECRET so they match the values in Keycloak.

4. Create the following realm-level roles:

   • sysadmin
   • panadmin
   • agadmin
   • stduser
   • infouser

5. For each role set:

   • Composite Roles: on
   • Realm Roles - Associated Roles: the next lower access level (e.g. panadmin -> agadmin, agadmin -> stduser)

To have pandas-ui to create and edit user using the Keycloak REST API configure it with SAVE_USERS_TO_KEYCLOAK=true