Skip to content

Commit

Permalink
SUBMARINE-857. Image should be built locally in github-action
Browse files Browse the repository at this point in the history
### What is this PR for?
The current image is pulled from docker hub so the change in images will not be reflected immediately.
I revised the workflow files and build the images before deploying the cluster.
Also, I used Github action cache to cache docker layer to accelerate.

### What type of PR is it?
[Bug Fix]

### Todos

### What is the Jira issue?
https://issues.apache.org/jira/browse/SUBMARINE-867

### How should this be tested?

### Screenshots (if appropriate)

### Questions:
* Do the license files need updating? No
* Are there breaking changes for older versions? No
* Does this need new documentation? No

Author: ByronHsu <byronhsu1230@gmail.com>
Author: jeff-901 <b07901052@ntu.edu.com>

Signed-off-by: byronhsu <byronhsu@apache.org>

Closes #613 from ByronHsu/build-image-locally and squashes the following commits:

fd25858 [ByronHsu] fix path
bbe3e86 [ByronHsu] debug maven build
0fe62e1 [ByronHsu] debug maven build
cf5f655 [ByronHsu] debug maven build
76d1b7a [ByronHsu] debug maven build
9196726 [ByronHsu] debug maven build
eb32872 [ByronHsu] debug maven build
8fcd7e8 [ByronHsu] debug maven build
8fc3f6f [ByronHsu] debug maven build
fb9d6a6 [ByronHsu] monitor error
216f208 [ByronHsu] monitor error
b37e13c [ByronHsu] fix typo
96d4d81 [ByronHsu] try to use docker cache
ac60687 [ByronHsu] try to use docker cache
4dc1145 [ByronHsu] build before deploy
5ba3884 [ByronHsu] ensure the script is ran
35f1e94 [jeff-901] for travis
e0293de [jeff-901] for travis
eb0adac [jeff-901] edit remain url
85650f0 [jeff-901] edit mlflow dockerfile s3 url
ed1cfe2 [jeff-901] delete fix ip
  • Loading branch information
ByronHsu committed Jun 29, 2021
1 parent e8e2711 commit 1c77153
Show file tree
Hide file tree
Showing 2 changed files with 81 additions and 2 deletions.
30 changes: 30 additions & 0 deletions .github/scripts/build-image-locally.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

SUBMARINE_VERSION="0.6.0-SNAPSHOT"
FOLDER_LIST=("database" "mlflow" "submarine")
IMAGE_LIST=("apache/submarine:database-${SUBMARINE_VERSION}" "apache/submarine:mlflow-${SUBMARINE_VERSION}" "apache/submarine:server-${SUBMARINE_VERSION}")

for i in "${!IMAGE_LIST[@]}"
do
echo "Build Image ${IMAGE_LIST[i]}"
echo "Execute ./dev-support/docker-images/${FOLDER_LIST[i]}/build.sh"
./dev-support/docker-images/"${FOLDER_LIST[i]}"/build.sh
kind load docker-image "${IMAGE_LIST[i]}"
done

53 changes: 51 additions & 2 deletions .github/workflows/python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ jobs:
env:
PYTHON: ${{ matrix.python }}
steps:
- uses: actions/checkout@v1
- name: Checkout project
uses: actions/checkout@v1
- name: Setup python environment
uses: actions/setup-python@v1
with:
Expand All @@ -38,7 +39,20 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 120
steps:
- uses: actions/checkout@v1
- uses: actions/checkout@v2
- name: Set up JDK 1.8
uses: actions/setup-java@v1
with:
java-version: '1.8'
- name: Set up Maven 3.6.3
uses: stCarolas/setup-maven@v4
with:
maven-version: 3.6.3
- name: Check version
run: |
mvn --version
java -version
- name: Create the kind config
run: |
cat <<EOF > ./kind-config-kind.yaml
Expand Down Expand Up @@ -73,6 +87,41 @@ jobs:
kubectl get pods -n kube-system
echo "current-context:" $(kubectl config current-context)
echo "environment-kubeconfig:" ${KUBECONFIG}
# Cache docker layer
- name: Cache docker layer
uses: satackey/action-docker-layer-caching@v0.0.11
# Ignore the failure of a step and avoid terminating the job.
continue-on-error: true

# Cache maven (for submarine server)
- uses: actions/cache@v1
with:
path: ~/.m2/repository/com
key: ${{ runner.os }}-maven-com-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-com-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/org
key: ${{ runner.os }}-maven-org-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-org-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/net
key: ${{ runner.os }}-maven-net-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-net-
- uses: actions/cache@v1
with:
path: ~/.m2/repository/io
key: ${{ runner.os }}-maven-io-${{ hashFiles('**/pom.xml') }}
restore-keys: |
${{ runner.os }}-maven-io-
- name: Build Image locally
run: |
.github/scripts/build-image-locally.sh
- name: Start submarine
run: |
# Fix submarine-database start failed in kind. https://github.com/kubernetes/minikube/issues/7906
Expand Down

0 comments on commit 1c77153

Please sign in to comment.