pull from upstream (#6)

* Add the utf-8 byte order marker to simplify issues with loading to Excel

* Brought tests suites inline with models.py, handle the utf-8 BOM, and expect bytes over the wire.

* Whoops. params where they should have been.

* Minor changes to cache invalidation to get rid of write access rqmt.

* ugh tests.

* get_cache should be type-hinting a str return, not bool. Also, I was returning both a datetime, or a str. Whoops.

* sigh. tests. remember the tests.

* - removed Beta banner
- removed Bold links in some pages
- add Terms and Conditions in footers

* - removed temporary Google Analytics
- add Content Security Policy on header
- moved some inline javascript call to a external file

* forgot one inline onclick javascript

* - implemented a whitelist for report names that can be call via the app URL.

  for now : only one report name is allowed : compliance

* - forgot one file

* build package for public app

* fix syntax errors

* fire new job names

* added logic to only display the donut for Public users

* forgot to remove bold for links for modal (How to read this table?)

* removed some unwanted space

* put back Beta Banner

* Minor tweaks to config to enable usage of Azure Managed Service Identities in combination with Azure KeyVault.

* this time with updated req's

* local ci would be great when you're sleep deprived.

* removed secret name out of code

* Removed headers due to duplication..

The upstream servers are also placing these headers, so removing from here.

* Security Update: pyyaml bump to pull in safe_load

Fixes this yaml/pyyaml#74.

Note we were already using safe_load.

* Security Update: pyyaml version bump

yaml/pyyaml#74

* Paginate scroll to top

* add semi-colon

* - Implementation of Google Tag Manager

   GTM ID is stored in Environment variable called GOOGLE_TAG_MANAGER

* fix typo

* fix data-domain, can't use comma to enclose value, break if value have comma in domain name

* removed CSP policies from HTML header. CSP is now implemented on Nginx server.

* - some cleanup before merge to Master branch

* - to fix Alerts from LGTM

* Compatibility with kubernetes  (cds-snc#127)

* Modification for deploying on k8s

* Small fix on dockerfile

* Added CI workflow file

* Ignore pip pinning in CI

* defer datatable render (cds-snc#129)

* Changed worker type and worker amount (cds-snc#130)

* Added PR review app configuration;

* Actually hit the right container

* Take 2

* Upgraded deps (cds-snc#132)

Bump dependencies for pymongo and flask_pymongo. Fixes time based connection issues.

* Task default organizations (cds-snc#136)

* - set default view to Organizations instead of Domains
- removed logic to public and internal view since now we will have same view for internal/public users

* - fix some accessibilities issues

* - put back role=row for TR. If not present, Mobile view doesnt display the green plus button
  in By Organizations page

* - for Accessibility : implement "Skip to main content" link at top of pages ( visible when Tab into focus)

* update content for the Guidance page (cds-snc#137)

.circleci/config.yml

@@ -47,7 +47,7 @@ jobs:
               docker push "${DOCKER_REGISTRY}/${DOCKER_NAMESPACE}/${CIRCLE_PROJECT_REPONAME}:latest"
           name: "Build and Deploy Website Docker Image"
     working_directory: ~/repo
-  build_package:
+  build_internal_package:
     docker:
       - image: cdssnc/track-web-build:0.1.0
     working_directory: /opt/apps/track-web
@@ -58,12 +58,25 @@ jobs:
       - store_artifacts:
           path: /opt/apps/track-web/track-web.tar.gz
           destination: track-web.tar.gz
-
+  build_external_package:
+    docker:
+      - image: cdssnc/track-web-build:0.1.0
+    working_directory: /opt/apps/track-web-public
+    steps:
+      - checkout
+      - run:
+          command: sh deploy/build-env-public.sh
+      - store_artifacts:
+          path: /opt/apps/track-web-public/track-web-public.tar.gz
+          destination: track-web-public.tar.gz
 workflows:
   version: 2
   tracker:
     jobs:
       - track_web
-      - build_package:
+      - build_internal_package:
+         requires:
+            - track_web
+      - build_external_package:
          requires:
             - track_web

.github/main.workflow

@@ -0,0 +1,11 @@
+workflow "CI" {
+  on = "push"
+  resolves = [
+    "Dockerfile lint"
+  ]
+}
+
+action "Dockerfile lint" {
+  uses = "docker://cdssnc/docker-lint"
+  args = "--ignore DL3013"
+}

Dockerfile

@@ -1,27 +1,19 @@
-MAINTAINER David Buckley <david.buckley@cds-snc.ca>
+FROM python:3.5 as python-base
 LABEL Description="Track Web Security Compliance" Vendor="Canadian Digital Service"
 
-FROM python:3.5 as python-base
 COPY requirements.txt /opt/track-web/requirements.txt
 COPY setup.py /opt/track-web/setup.py 
 COPY track /opt/track-web/track
 COPY MANIFEST.in /opt/track-web/MANIFEST.in
 
-# Build wheels to install into production image
-# Force a build with --no-binary to get around the case where a wheel is available for python:3.5 but not python:3.5-alpine
 RUN pip install --upgrade pip && mkdir wheels && pip wheel --no-binary :all: -r /opt/track-web/requirements.txt -w wheels && pip wheel --no-deps /opt/track-web/ -w wheels
-
-FROM python:3.5-alpine
-MAINTAINER David Buckley <david.buckley@cds-snc.ca>
-LABEL Description="Track Digital Security Compliance" Vendor="Canadian Digital Service"
-
-COPY --from=python-base /wheels /wheels
 
 RUN pip install /wheels/* && rm -rf /wheels /root/.cache/pip && \
-    addgroup -S track-web && adduser -S -G track-web track-web && \
+    addgroup --system track-web && adduser --system --group track-web && \
     mkdir -p /opt/track-web/.cache && \
     chown -R track-web /opt/track-web
+
 USER track-web:track-web
 
 EXPOSE 5000
-ENTRYPOINT ["gunicorn", "track.wsgi:app", "--bind=0.0.0.0:5000", "--worker-class=gthread", "--access-logfile=-", "--error-logfile=-", "--capture-output"]
+ENTRYPOINT ["gunicorn", "track.wsgi:app", "--bind=0.0.0.0:5000", "--worker-class=sync", "--access-logfile=-", "--error-logfile=-", "--log-level=debug", "--workers=4"]

deploy/build-env-public.sh

@@ -0,0 +1,9 @@
+WORKDIR=${1:-"/opt/apps/track-web-public"}
+mkdir -p $WORKDIR
+cd $WORKDIR
+python3 -m venv .venv
+. .venv/bin/activate
+pip install --upgrade pip
+pip install -r requirements.txt
+tar -czvf track-web-public.tar.gz .venv track
+rm -rf .venv

elenchos.json

@@ -0,0 +1,6 @@
+{
+    "dockerfiles": {
+      "grc.io/cdssnc/track-web": "."
+    },
+    "overlay": "manifests/overlays/elenchos"
+  }

manifests/overlays/elenchos/app-deployment.yaml

@@ -0,0 +1,28 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  labels:
+    app: track-web
+  name: track-web
+spec:
+  selector:
+    matchLabels:
+      app: track-web
+  template:
+    metadata:
+      labels:
+        app: track-web
+    spec:
+      containers:
+        - image: gcr.io/cdssnc/track-web
+          imagePullPolicy: Always
+          name: track-web
+          env:
+            - name: TRACKER_MONGO_URI
+              value: mongodb://track-ro:0D^GEPgF52d&2S@ds113692.mlab.com:13692/trackweb
+      dnsPolicy: ClusterFirst
+      restartPolicy: Always
+      schedulerName: default-scheduler
+      securityContext: {}
+      terminationGracePeriodSeconds: 30
+status: {}

manifests/overlays/elenchos/app-service.yaml

@@ -0,0 +1,14 @@
+apiVersion: v1
+kind: Service
+metadata:
+  labels:
+    name: track-web 
+  name: track-web 
+spec:
+  type: NodePort
+  ports:
+    - port: 5000
+      targetPort: 5000
+  selector:
+    app: track-web 
+

manifests/overlays/elenchos/kustomization.yaml

@@ -0,0 +1,10 @@
+resources:
+  - app-deployment.yaml
+  - app-service.yaml
+  - traefik-ingress-controller-cluster-role-binding.yaml
+  - traefik-ingress-controller-cluster-role.yaml
+  - traefik-ingress-controller-deployment.yaml
+  - traefik-ingress-controller-service-account.yaml
+  - traefik-ingress.yaml
+  - traefik-ingress-service.yaml
+

manifests/overlays/elenchos/traefik-ingress-controller-cluster-role-binding.yaml

@@ -0,0 +1,12 @@
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRoleBinding
+metadata:
+  name: traefik-ingress-controller
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: traefik-ingress-controller
+subjects:
+  - kind: ServiceAccount
+    name: traefik-ingress-controller
+    namespace: kube-system

manifests/overlays/elenchos/traefik-ingress-controller-cluster-role.yaml

@@ -0,0 +1,24 @@
+apiVersion: rbac.authorization.k8s.io/v1beta1
+kind: ClusterRole
+metadata:
+  name: traefik-ingress-controller
+  namespace: kube-system
+rules:
+  - apiGroups:
+      - ""
+    resources:
+      - services
+      - endpoints
+      - secrets
+    verbs:
+      - get
+      - list
+      - watch
+  - apiGroups:
+      - extensions
+    resources:
+      - ingresses
+    verbs:
+      - get
+      - list
+      - watch

manifests/overlays/elenchos/traefik-ingress-controller-deployment.yaml

@@ -0,0 +1,37 @@
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  labels:
+    k8s-app: traefik-ingress-lb
+  name: traefik-ingress-controller
+  namespace: kube-system
+spec:
+  template:
+    metadata:
+      labels:
+        k8s-app: traefik-ingress-lb
+    spec:
+      containers:
+        - args:
+            - --api
+            - --kubernetes
+            - --debug
+            - --defaultentrypoints=http
+            - --entrypoints=Name:http Address::80
+          image: traefik:1.7
+          name: traefik-ingress-lb
+          ports:
+            - containerPort: 80
+              hostPort: 80
+              name: http
+            - containerPort: 8080
+              hostPort: 8080
+              name: admin
+          securityContext:
+            capabilities:
+              add:
+                - NET_BIND_SERVICE
+              drop:
+                - ALL
+      serviceAccountName: traefik-ingress-controller
+      terminationGracePeriodSeconds: 60

manifests/overlays/elenchos/traefik-ingress-controller-service-account.yaml

@@ -0,0 +1,5 @@
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: traefik-ingress-controller
+  namespace: kube-system

manifests/overlays/elenchos/traefik-ingress-service.yaml

@@ -0,0 +1,13 @@
+apiVersion: v1
+kind: Service
+metadata:
+  name: traefik-ingress-service
+  namespace: kube-system
+spec:
+  ports:
+  - name: http
+    port: 80
+    protocol: TCP
+  selector:
+    k8s-app: traefik-ingress-lb
+  type: LoadBalancer

manifests/overlays/elenchos/traefik-ingress.yaml

@@ -0,0 +1,14 @@
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  annotations:
+    kubernetes.io/ingress.class: traefik
+  name: traefik-ingress
+spec:
+  rules:
+    - http:
+        paths:
+          - path: /
+            backend:
+              serviceName: track-web
+              servicePort: 5000

requirements.txt

@@ -2,11 +2,11 @@ flask==0.12.4
 gunicorn==19.6.0
 pyyaml==3.13
 python-slugify==1.2.1
-Flask-PyMongo==0.5.1
+Flask-PyMongo==2.2.0
 flask-compress==1.4.0
 click==6.7
 Babel==2.6.0
 Flask-Caching==1.4.0
-pymongo==3.7.0
+pymongo==3.7.2
 azure-keyvault==1.1.0
 msrestazure==0.5.1

setup.py

@@ -22,8 +22,8 @@
         'gunicorn==19.6.0',
         'pyyaml==3.13',
         'python-slugify==1.2.1',
-        'pymongo==3.7.0',
-        'Flask-PyMongo==0.5.1',
+        'pymongo==3.7.2',
+        'Flask-PyMongo==2.2.0',
         'flask-compress==1.4.0',
         'click==6.7',
         'Babel==2.6.0',

track/config.py

@@ -13,13 +13,15 @@
 class Config:
     DEBUG = False
     TESTING = False
-    MONGO_URI = "mongodb://localhost:27017/track"
+    MONGO_URI = os.environ.get("TRACKER_MONGO_URI", "mongodb://localhost:27017/track")
     CACHE_TYPE = "null"
 
     @staticmethod
     def init_app(app):
         pass
 
+
+
 class ProductionConfig(Config):
 
     CACHE_TYPE = "filesystem"

track/helpers.py

@@ -1,6 +1,7 @@
 import pkg_resources
 import yaml
 import datetime
+import os
 from track import models
 from track.data import FIELD_MAPPING
 from babel.dates import format_date
@@ -56,3 +57,7 @@ def percent(num, denom):
     @app.template_filter("percent_not")
     def percent_not(num, denom):
         return (100 - round((num / denom) * 100))
+
+    @app.template_filter("fetch_env")
+    def fetch_env(value):
+        return os.getenv(value)

track/static/js/dataTables.downloads.js

@@ -36,9 +36,7 @@ $.fn.dataTable.Download = function ( inst ) {
     if (drawnOnce) return;
 
     var elem = "" +
-      "<a onClick=\"gtag('event', 'download', { event_category: 'Download / Télécharger', event_action: 'Download / Télécharger CSV'});\" class=\"text-https-blue hover:text-black font-bold\" href=\"" + csv + "\" download>" +
-        text +
-      "</a>";
+      "<a class=\"text-https-blue hover:text-black font-bold\" href=\"" + csv + "\" download>" + text +"</a>";
 
     container.html(elem);
     drawnOnce = true;