Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Save test results for each test spec to improve test results reporting #5449

Merged
Merged
Changes from 28 commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
e6e6254
PoC to improve test results reporting
rnapoles-rh Feb 7, 2022
afbd934
PoC to improve test results reporting
rnapoles-rh Feb 7, 2022
05afd87
PoC to improve test results reporting
rnapoles-rh Feb 7, 2022
4a12966
PoC to improve test results reporting
rnapoles-rh Feb 7, 2022
0e590e7
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
2b5a36a
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
ce9058d
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
7a75a2f
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
68f84ef
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
8895ca9
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
f205870
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
74a9afe
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
f440708
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
717a95f
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
d519894
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
64b56e7
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
6287935
PoC to improve test results reporting
rnapoles-rh Feb 8, 2022
6d2952f
PoC to improve test results reporting
rnapoles-rh Feb 9, 2022
7b0d393
PoC to improve test results reporting
rnapoles-rh Feb 9, 2022
b8ab487
PoC to improve test results reporting
rnapoles-rh Feb 9, 2022
9cb1ca6
Save test results for each spec to improve test results reporting
rnapoles-rh Feb 9, 2022
858ca5d
Incorporated feedback from review
rnapoles-rh Feb 10, 2022
260ad6c
Incorporated feedback from review
rnapoles-rh Feb 10, 2022
24bb502
Removed blank spaces after commas in data row
rnapoles-rh Feb 10, 2022
f57bdaf
Incorporated additional feedback
rnapoles-rh Feb 24, 2022
fe85ae5
Updated comment to rerun tests
rnapoles-rh Mar 2, 2022
f17a19d
Incorporated feedback to fix unit tests failure and check for err dur…
rnapoles-rh Mar 2, 2022
83650b1
Incorporated additional feedback on close command
rnapoles-rh Mar 5, 2022
2225212
Edited comment to rerun tests
rnapoles-rh Mar 9, 2022
a9431f8
Edited comment to rerun tests
rnapoles-rh Mar 9, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
42 changes: 41 additions & 1 deletion tests/helper/helper_generic.go
Original file line number Diff line number Diff line change
Expand Up @@ -285,6 +285,11 @@ type CommonVar struct {
// original values to get restored after the test is done
OriginalWorkingDirectory string
OriginalKubeconfig string
// Ginkgo test realted
testFileName string
testCase string
testFailed bool
testDuration float64
}

// CommonBeforeEach is common function runs before every test Spec (It)
Expand Down Expand Up @@ -312,12 +317,47 @@ func CommonBeforeEach() CommonVar {

// CommonAfterEach is common function that cleans up after every test Spec (It)
func CommonAfterEach(commonVar CommonVar) {
// Get details, including test result for each test spec and adds it to local testResults.txt file
// Ginkgo test related variables.
commonVar.testFileName = strings.Replace(CurrentGinkgoTestDescription().FileName[strings.LastIndex(CurrentGinkgoTestDescription().FileName, "/")+1:strings.LastIndex(CurrentGinkgoTestDescription().FileName, ".")], "_", "-", -1) + ".go"
commonVar.testCase = CurrentGinkgoTestDescription().FullTestText
commonVar.testFailed = CurrentGinkgoTestDescription().Failed
commonVar.testDuration = CurrentGinkgoTestDescription().Duration.Seconds()
Comment on lines +324 to +325
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't these things go in CommonAfterEach()? How can you know the duration the test took to run and it's pass/fail status before it is run?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, I moved them to CommonAfterEach()


var prNum string
var resultsRow string
prNum = os.Getenv("GIT_PR_NUMBER")
passedOrFailed := "PASSED"
if commonVar.testFailed {
passedOrFailed = "FAILED"
}
clusterType := "OCP"
if IsKubernetesCluster() {
clusterType = "KUBERNETES"
}
testDate := strings.Split(time.Now().Format(time.RFC3339), "T")[0]
resultsRow = prNum + "," + testDate + "," + clusterType + "," + commonVar.testFileName + "," + commonVar.testCase + "," + passedOrFailed + "," + strconv.FormatFloat(commonVar.testDuration, 'E', -1, 64) + "\n"
testResultsFile := filepath.Join("/", "tmp", "testResults.txt")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are we doing with this file? I assume this is not uploaded on any cloud storage bucket.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not yet, once this PR gets merged I will update the pipeline script to upload it to a bucket


f, err := os.OpenFile(testResultsFile, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
fmt.Println("Error when opening file: ", err)
} else {
_, err = f.WriteString(resultsRow)
if err != nil {
fmt.Println("Error when writing to file: ", err)
}
if err = f.Close(); err != nil {
fmt.Println("Error when closing file: ", err)
}
}

// delete the random project/namespace created in CommonBeforeEach
commonVar.CliRunner.DeleteNamespaceProject(commonVar.Project)

// restores the original kubeconfig and working directory
Chdir(commonVar.OriginalWorkingDirectory)
err := os.Setenv("KUBECONFIG", commonVar.OriginalKubeconfig)
err = os.Setenv("KUBECONFIG", commonVar.OriginalKubeconfig)
Expect(err).NotTo(HaveOccurred())

// delete the temporary context directory
Expand Down