add support for linking tests to Jira issues (e.g. stories, requirements), using Xray

[bitcoder]

This PR adds support for "linking" mocha tests to existing Jira issue, such as user stories, right from the test code.
For that, users just need to add a "requirement" attribute on the "it" block, specifying the Jira issue key to link that test to.
This ultimately will add a "requirement" attribute on the generated JUnit XML report that can be processed by Xray (https://www.getxray.app/), a well-known test management tool for Jira, whenever the report is submitted to it.

The code was tested with basic mocha tests and also using Cypress (which uses mocha).
This will benefit many teams using Jira and Xray and doesnt interfere with existing behaviour.

Example of how to link a test to a user story, having the issue key CALC-123, using just mocha:

var assert = require('assert');
describe('Array', function() {
  describe('#indexOf()', function() {
    it('should return -1 when the value is not present', { requirement: 'CALC-123' }, function() {
      assert.equal([1, 2, 3].indexOf(4), -1);
    });

    it('should return >= 0 when the value ispresent', { tags: 'CALC2' }, function() {
      assert.equal([1, 2, 3].indexOf(1), 0);
    });


  });
});

Example of how to link a test to a user story, having the issue key CALC-123, using Cypress:

describe('My second test', function() {
  it('Gets, types and asserts', { requirement: 'CALC-123' }, function() {
    cy.visit('https://example.cypress.io')

    cy.contains('type').click()

    // Should be on a new URL which includes '/commands/actions'
    cy.url().should('include', '/commands/actions')

    // Get an input, type into it and verify that the value has been updated
    cy.get('.action-email')
      .type('fake@email.com')
      .should('have.value', 'fake@email.com') // error added on the test itself just to make it fail
  })
})

[bitcoder]

if you agree, I can then update also the doc on this PR and add some test(s)

[bitcoder]

This needs to be updated to match latest developments and the format produced by https://github.com/Xray-App/xray-junit-extensions

[bitcoder]

@michaelleeallen is this project still active btw?

[ypicard]

This pull request would be greatly appreciated by Jira / Xray users. Do we have any updates on the state of it?
Thanks for taking the time to develop this!

[bitcoder]

I'm still waiting feedback from this project' maintainers to see how to move forward. Team what are your thoughts?

[michaelleeallen]

@bitcoder thanks for submitting this PR. This reporter is intended to support any test runner in a generic fashion. I see this implementation as being specific to Xray, and cannot accept it as-is. I would suggest updating this PR to accommodate any reporting metadata:

it('should do something valuable', { reportMeta: { requirement: 'CALC-123' } }, () => { ... });

Where reportMeta is an Object containing any attributes the user wishes to include on the generated test case XML element. This would allow future metadata fields to be included to support other test runners' features.

Also, there should be a unit test for this and some comments in the code to explain why the code is needed.

Thanks!

[bitcoder]

Thanks @michaelleeallen , I will work on that and come back to you

[bitcoder]

I would swear it worked whenever I implemented this but now that I was picking it once again I cannot get it working (my fault, I should have it with tests).
I was looking over the docs and mochajs code and I think following syntax is currently not supported by mocha. I checked with the mochajs community on gitter and I was told that mocha never supported that; it seems that Mochawesome is offering a similar feature with addContext on their API.

it('should do something valuable', { reportMeta: { requirement: 'CALC-123' } }, () => { ... });

Therefore, unless I'm mistaken, we would need to keep the syntax `it('test_title', function())`` and use another way to add the metainformation.
@michaelleeallen, would perhaps an approach like this be valid, where I add it to the context ctx of test object and then would look at it from inside the mocha-junit-reporter?

 beforeEach(function() {
    this.reportMeta = {};
  })

it('should return -1 when the value is not present', function() {
  // this.test.ctx.reportMeta= { requirements: 'CALC-13' };
  this.reportMeta= { requirements: 'CALC-13' };

  ...
});

As there will be some specific logic to add some custom properties to the testcase elements, maybe a xrayMode boolean should be used to enable those specifics. What do you think?

[michaelleeallen]

I think adding the metadata as you demonstrated would be fine, but I do not like the idea of an "xrayMode" flag since again this library is runner agnostic. Looking at your example, I think it makes more sense to call the property testCaseMetadata:

it('should return -1 when the value is not present', function() {
  this.testCaseMetadata = { requirements: 'CALC-13' };
  ...
});

[bitcoder]

@michaelleeallen , I've implemented your suggestion and updated the PR.
This approach works with Xray but can also be by other tools that can take advantage of this mechanism for setting custom metadata on testcase. It's flexible enough, up to a point, to not make it too much complex.
Kindly ask for reviewing it.
I've also updated the README.md file and added tests.

[michaelleeallen]

This looks good to me. I'm going to include this in a release with some dependency updates.

[michaelleeallen]

@bitcoder there is a failing test case. Can you have a look?

[bitcoder]

@bitcoder there is a failing test case. Can you have a look?

sure, going to check it and I'll let you know

[michaelleeallen]

@bitcoder i just noticed that this is using a properties element within a testcase which is not valid for Jenkins: https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

Jenkins only allows property elements under a testsuite element. I could be wrong about this, but we could prove it with a test case that runs this output against the junit XSD. There are a couple of tests in the spec file that will show you how to do this.

[bitcoder]

Well, properties are usually an element that appear under testsuite, that's correct. junit has not an official XSD though AFAIK. I had checked before with the Junit project leader Marc Philipp , about the feasibility of having also propertiesunder the testcase element and we didn't see a problem with it; however, some tools may have a more strict validation and support only what ant provided back then, on the first releases of the Junit report format.
The current code only adds properties if testCaseMetadata is provided, which ensures we keep backward compatible; this is independent of the "ant/jenkins" mode flags. However, I understand if you want to have to inhibit this feature if those modes are active.
Having said that, shall we keep the current logic or do you prefer to change it to only make it applicable if not in Jenkins mode and also not in ant mode?

[bitcoder]

Just to complement, Datadog also supports properties at testcase level, which
will also be supported by this PR.

[bitcoder]

@michaelleeallen , I've merged the code from upstream and tested it on the forked project (as there I can run GH actions) and it seems to be working now. Unsure why it was not working before... perhaps some merging issue?
Just waiting for your feedback so we can move forward :)

[patelrid12]

This will be really helpful feature for our automation framework. can someone merge this ?

[fuantomu]

Any updates on this? This is exactly what I need right now to automate test reports with Jira

[bitcoder]

@michaelleeallen can you please check my last comments above?

[Nahkag]

Hello, this does not work for me with Cypress and Xray :(
There is no new property created in the XML report file.

=> I understand now, this version is not merged. So I supposed, I won't be alble to use it in my CI ?

[bitcoder]

Hello, this does not work for me with Cypress and Xray :( There is no new property created in the XML report file.

=> I understand now, this version is not merged. So I supposed, I won't be alble to use it in my CI ?

@Nahkag you can require directly the code from https://github.com/bitcoder/mocha-junit-reporter in your package.json

I need to work with @michaelleeallen to get this merged

[Nahkag]

Hello, this does not work for me with Cypress and Xray :( There is no new property created in the XML report file.
=> I understand now, this version is not merged. So I supposed, I won't be alble to use it in my CI ?

@Nahkag you can require directly the code from https://github.com/bitcoder/mocha-junit-reporter in your package.json

I need to work with @michaelleeallen to get this merged

[bitcoder]

Glad it worked @asub927 :)
Waiting for final merge or feedback from the maintainers @michaelleeallen @clayreimann

[InduKrish]

just fYI, we are planning to use cypress plugin for reporting all cypress tests to Xray, and we have been waiting for the fix as well Please let us know once it is ready to use. Thank you so much

[InduKrish]

Hi, Can you please clarify how to map more than one requirement to the test?

describe('template spec', () => {
//it('passes', () => {
it('passes', { testCaseMetadata: { test_key: 'CALC-1', requirements: 'CALC-123', tags: 'ui,core' } }, function() {

cy.visit('https://example.cypress.io')

})
})

[InduKrish]

@asub927 , Can you please assist me with the sample testCaseMetaData object that you use in the cypress test?

[bitcoder]

just use comma as a delimeter.

it('passes', { testCaseMetadata: { test_key: 'CALC-1', requirements: 'CALC-123,CALC-567', tags: 'ui,core' } }, function() {

By curiosity, this is how that information gets into the generated JUnit XML report.

[InduKrish]

Thank you , i would like to generate the xml report, I do have a UI repository with 270+ test cases, Can you please guide me with the steps that i should follow to be able to use the latest code, meanwhile we wait for the code to be merged?

do i have to place run_tests.sh script in my repository and update package.json with mocha-junit-reporter dependency and run the shell script that will produce the xml? Can you please confirm?

Should i add updated cypress.json with reporter name or anything? This is what I have cuurently.

{
"video": false,
"baseUrl": "http://localhost:3000",
"reporter": "mochawesome",
"reporterOptions": {
"reportDir": "cypress/reports",
"reportFileName": "report",
"overwrite": false,
"html": false,
"json": true,
"charts": true,
"timestamp": "mmddyyyy-HHMMss"
},
"retries": {
"runMode": 2
}
}

[bitcoder]

Thank you , i would like to generate the xml report, I do have a UI repository with 270+ test cases, Can you please guide me with the steps that i should follow to be able to use the latest code, meanwhile we wait for the code to be merged?

do i have to place run_tests.sh script in my repository and update package.json with mocha-junit-reporter dependency and run the shell script that will produce the xml? Can you please confirm?

Should i add updated cypress.json with reporter name or anything? This is what I have cuurently.

{ "video": false, "baseUrl": "http://localhost:3000", "reporter": "mochawesome", "reporterOptions": { "reportDir": "cypress/reports", "reportFileName": "report", "overwrite": false, "html": false, "json": true, "charts": true, "timestamp": "mmddyyyy-HHMMss" }, "retries": { "runMode": 2 } }

Please see a sample repository I created with Cypress here, installing cypress from scratch and without any customization whatsoever :)

You need to:

1. on package.json add an entry to "devDependencies": "mocha-junit-reporter": "bitcoder/mocha-junit-reporter" ; this will make your project use the customized mocha-junit-reporter
2. npm install (to install the customized mocha-junit-reporter package)
3. run cypress and specify the custom mocha-junit-reporter: ./node_modules/cypress/bin/cypress run -s cypress/e2e/dummyspec.cy.js --reporter mocha-junit-reporter

As simple as that.

[InduKrish]

After running npm install and add testCaseMetadata to the test , I see an error that says "No overload matches this call" error"

this is the version i have in package.json
"cypress": "^11.2.0",
"mocha-junit-reporter": "bitcoder/mocha-junit-reporter"

[bitcoder]

After running npm install and add testCaseMetadata to the test , I see an error that says "No overload matches this call" error"

this is the version i have in package.json "cypress": "^11.2.0", "mocha-junit-reporter": "bitcoder/mocha-junit-reporter"

probably that's due to the nodejs version you're using. I'm on v16.10.0

[asub927]

After running npm install and add testCaseMetadata to the test , I see an error that says "No overload matches this call" error"

this is the version i have in package.json "cypress": "^11.2.0", "mocha-junit-reporter": "bitcoder/mocha-junit-reporter"

probably that's due to the nodejs version you're using. I'm on v16.10.0

@InduKrish - That error is due to type-definitions not being available for this package. You can ignore the error or add a custom type definition locally. The code should compile and upload works as expected. Let me know if that helps.

[InduKrish]

@asub927 @bitcoder I am using Javascript , and Can you please advise if i should still add custom type definition and also use only node version v16.10.0?

[bitcoder]

@asub927 @bitcoder I am using Javascript , and Can you please advise if i should still add custom type definition and also use only node version v16.10.0?

I'm using Javascript only.. as you can see on the project/sample code that I've shared. I know it works with nodejs v16.10.0, as it is my default but I dont think there are any specifics of nodejs 16.10 so..
On the custom type definitions, I don't use them nor have that much experience with them so far.

[InduKrish]

Hi @bitcoder , i ran all my tests -270+ test cases, and all tests are executed, however in the final test-results.xml, only 53 tests were reported, not sure why the other test results are not captured Can you please advise?

[bitcoder]

I would need more info in order to be able to help out otherwise it's impossible. Can you try to make a working example showing that behavior? Maybe simplifying what you have until you can replicate that problem? Then please share the code or a github repo with it?

[InduKrish]

I realized, that it is running only the tests from the last file , attached here is the hierarchy inside the folder named cypress/e2e, and here is the command that I use to run the cypress test

"scripts": {
"start": "react-scripts start",
"build": "rollup -c",
"build:yalc": "rollup -c && yalc publish && yalc push --sig",
"build:functional": "react-scripts build",
"test:functional": "concurrently --kill-others --success first "npm run build:functional && npx serve -s build" "wait-on http://localhost:3000/ && npm run cy:functional"",
"cy:functional": "TZ=America/Chicago cypress run --headless --spec "cypress/e2e/**/" --reporter mocha-junit-reporter",
"cy:clean-reports": "rm -rf cypress/reports",
"cy:functional:merge-reports": "mochawesome-merge cypress/reports/mochawesome.json > cypress/reports/functional_test_report.json",
"cy:functional:generate-report": "npm run cy:functional:merge-reports && marge cypress/reports/functional_test_report.json --reportDir ./cypress/reports/ --inline",
"cy:generate-xml-report": "jsonjunit --json cypress/reports/ --junit cypress/reports/",
"tsc:check": "tsc --noEmit --project "./tsconfig.json"",
"tsc:check:cypress": "tsc --noEmit --project "./cypress/tsconfig.json""
},

I run all tests using this command ie ---> npm run test:functional

here is what i have in the cypress.config.ts file
import {defineConfig} from 'cypress';

export default defineConfig({
e2e: {
baseUrl: 'http://localhost:3000',
video: false,
// reporter: 'mochawesome',
reporterOptions: {
reportDir: 'cypress/reports',
reportFileName: 'report',
overwrite: false,
html: false,
json: true,
charts: true,
timestamp: 'mmddyyyy-HHMMss',
},
},
});

and the xml has the result of only the last cypress test file, although all tests from all folders under cypress/e2e are executed.

[InduKrish]

Hi @bitcoder, Can you please review the comment above and assist me with the resolution? eagerly awaiting your response. Thank you!

[bitcoder]

Hi @bitcoder, Can you please review the comment above and assist me with the resolution? eagerly awaiting your response. Thank you!

I've seen it but I dont have opportunity at this time. However, that seems to be not a problem with my implementation but more with the Cypress<>mocha-junit-reporter integration. Have you tried to generate the JUnit XML report without using my fork, using the builtin JUnit XML reporting that comes with Cypress?

[InduKrish]

Hi @bitcoder , yes, I see the results of all the tests in the xml report when i ran with builtin mochaawesome reporter, below is the config file,

and removed --reporter mocha-junit-reporter and ran all my tests with the below configuration and xml has the results of all the tests that are executed.

import {defineConfig} from 'cypress';

export default defineConfig({
e2e: {
baseUrl: 'http://localhost:3000',
video: false,
reporter: 'mochawesome',
reporterOptions: {
reportDir: 'cypress/reports',
reportFileName: 'report',
overwrite: false,
html: false,
json: true,
charts: true,
timestamp: 'mmddyyyy-HHMMss',
},
},
});

"scripts": {
"start": "react-scripts start",
"build": "rollup -c",
"build:yalc": "rollup -c && yalc publish && yalc push --sig",
"build:functional": "react-scripts build",
"test:functional": "concurrently --kill-others --success first "npm run build:functional && npx serve -s build" "wait-on http://localhost:3000/ && npm run cy:functional"",
"cy:functional": "TZ=America/Chicago cypress run --headless --spec "cypress/e2e/**/*"",
"cy:clean-reports": "rm -rf cypress/reports",
"cy:functional:merge-reports": "mochawesome-merge cypress/reports/mochawesome.json > cypress/reports/functional_test_report.json",
"cy:functional:generate-report": "npm run cy:functional:merge-reports && marge cypress/reports/functional_test_report.json --reportDir ./cypress/reports/ --inline",
"cy:generate-xml-report": "jsonjunit --json cypress/reports/ --junit cypress/reports/",
},

[bitcoder]

@InduKrish not sure of what is happening in your scenario.
This PR doesn't change the logic of handling the tests (as seen here).

[anwari666]

@InduKrish I had the same problem as yours, where the resulting xml file contains only the last testsuite. Apparently the file is replaced all the time, so I put [hash] in the filename, where it will be replaced dynamically.

This worked for me:

{
  reporterOptions: {
    mochaFile: 'results/test-results.[hash].xml',
  }
}

Source: https://www.npmjs.com/package/mocha-junit-reporter#results-report

[InduKrish]

Hi @anwari666 Thank you so much for letting me know, I have a question. Can you please advise if you have added the line below to cypress.config.ts file?
reporterOptions: {
mochaFile: 'results/test-results.[hash].xml',
}
}

1. As per @bitcoder , I thought i should remove reporter: 'mochawesome' and i removed it already. Can you please confirm what i did is right? i have only done the following as stated below. Can you please advise if i should add now mochaFile: 'results/test-results.[hash].xml', - this line in cypress.config.ts file?

comment from @bitcoder below:
Please see a sample repository I created with Cypress here, installing cypress from scratch and without any customization whatsoever :)
on package.json add an entry to "devDependencies": "mocha-junit-reporter": "bitcoder/mocha-junit-reporter" ; this will make your project use the customized mocha-junit-reporter
npm install (to install the customized mocha-junit-reporter package)
run cypress and specify the custom mocha-junit-reporter: ./node_modules/cypress/bin/cypress run -s cypress/e2e/dummyspec.cy.js --reporter mocha-junit-reporter

Can you please tell me how your cypress.config.ts looks like?

2. and Will you be merging all xmls at the end to make a rest call to Xray to update with the final xml? Can you please tell me how you are merging all the xmls?

This is how mine looks like:

here is what i have in the cypress.config.ts file
import {defineConfig} from 'cypress';

export default defineConfig({
e2e: {
baseUrl: 'http://localhost:3000/',
video: false,
// reporter: 'mochawesome', ---> i commented out mochaawesome reporter line.
reporterOptions: {
reportDir: 'cypress/reports',
reportFileName: 'report',
overwrite: false,
html: false,
json: true,
charts: true,
timestamp: 'mmddyyyy-HHMMss',
},
},
});

Can you please advise what i should add here?

[anwari666]

@InduKrish

Can you please advise if you have added the line below to cypress.config.ts file?

Yes I added it to my cypress.config.ts file. In your case, just put the mochafile within the reporterOptions object.

I thought i should remove reporter: 'mochawesome' and i removed it already.

You can, as long as when you invoke the command, you invoke it with --reporter mocha-junit-reporter.

Will you be merging all xmls at the end to make a rest call to Xray to update with the final xml? Can you please tell me how you are merging all the xmls?

Yes as well. My plan is to use sed, but I haven't done that part programmatically. Should be straightforward.

[debo07]

Thanks @bitcoder for this. I am using cypress, this worked out best for me and quite flexible.

I wish if this could be merged soon!

I don't have any issue to use it from "bitcoder/mocha-junit-reporter", just need to make sure this fork is continuously pulling from the main repo for any feature or fixes.

If someone wants to use the main repo and still want to have this feature, something like this can be done.

const MochaJUnitReporter = require("mocha-junit-reporter");

function CustomJunitReporter(runner, options) {
    MochaJUnitReporter.call(this, runner, options);
}

CustomJunitReporter.prototype = Object.create(MochaJUnitReporter.prototype);
CustomJunitReporter.prototype.constructor = CustomJunitReporter;

function getTestCaseMetadata(test) {
    // in pure mocha this is obtained from the test object context; in Cypress, test metadata is stored in the ._testConfig.unverifiedTestConfig
    return test._testConfig &&
        test._testConfig.unverifiedTestConfig &&
        test._testConfig.unverifiedTestConfig.testCaseMetadata
        ? test._testConfig.unverifiedTestConfig.testCaseMetadata
        : test.ctx && test.ctx.testCaseMetadata
        ? test.ctx.testCaseMetadata
        : null;
}

CustomJunitReporter.prototype.getTestcaseData = function (test, err) {
    // Refererence - https://github.com/michaelleeallen/mocha-junit-reporter/pull/153/files
    let testcase = MochaJUnitReporter.prototype.getTestcaseData.call(this, test, err);
    var testCaseMetadata = getTestCaseMetadata(test);
    if (testCaseMetadata) {
        /*
            Additional testcase metadata can be processed and embed on the JUnit XML report, as properties on the <testcase> element
            Xray provides support for processing some of these information (https://docs.getxray.app/display/XRAYCLOUD/Taking+advantage+of+JUnit+XML+reports), others may follow.
            The following code provides a more or less generic approach to provide properties for the testcase:
            - based on a property name and value
            - based on a property name and bulk content
            - based on a property name and a array of elements, each one with attributes
        */

        var properties = [];
        for (var key in testCaseMetadata) {
            if (typeof testCaseMetadata[key] === "string") {
                // if the metadata has a key whose value contains a string, then the property is named by the key
                // and has a "value" attribute based on its value
                //  e.g. { test_key: 'CALC-1' }

                properties.push({
                    property: { _attr: { name: key, value: testCaseMetadata[key] } },
                });
            } else if (testCaseMetadata[key] instanceof Array) {
                /*
                    if the metadata has a key whose value contains a array, then each element is in turn a object,
                    of a specific type/name, with attributes and cdata
                    e.g.
                    {
                        testrun_evidence: [
                        { item: { name: "dummy-evidence1.txt", _cdata: 'aGVsbG8=' }},
                        { item: { name: "dummy-evidence2.txt", _cdata: 'd29ybGQ=' }}
                        ]
                    }
                */

                // array with properly formatted elements that can be serialized to XML
                var items = [];

                // process all objects and convert each one to a XML element
                // object attributes are mapped to attributes on the XML element, except for "_cdata" which is mapped to cdata
                for (var idx = 0; idx < testCaseMetadata[key].length; idx++) {
                    var item = testCaseMetadata[key][idx];

                    // the XML element tag name is the only attribute on the object belonging to the array based property
                    var itemTag = Object.keys(item)[0];

                    // prepare a object that can be serialized to XML in the format of:
                    //   <itemTag attr1="..." attrN="...""><![CDATA[...]]></itemTag>
                    var itemElem = {};
                    itemElem[itemTag] = {};
                    itemElem[itemTag]._attr = {};
                    for (var attr in item[itemTag]) {
                        if (attr === "_cdata") {
                            itemElem[itemTag]._cdata = item[itemTag][attr];
                        } else {
                            itemElem[itemTag]._attr[attr] = item[itemTag][attr];
                        }
                    }

                    // save it to the list of XML elements
                    items.push(itemElem);
                }
                var multiElemProperty = { property: items };
                multiElemProperty.property.unshift({ _attr: { name: key } });
                properties.push(multiElemProperty);
            } else if (typeof testCaseMetadata[key] === "object") {
                // if it's an object, non-Array nor string, then it's a property with built-in content to be embed as cdata
                // note: a mix of cdata and standard attributes is not supported
                // e.g. { test_description: { _cdata: 'a sample test' } }

                var _cdata = testCaseMetadata[key]._cdata;
                if (_cdata) {
                    properties.push({
                        property: {
                            _attr: { name: key },
                            _cdata: this.removeInvalidCharacters(_cdata),
                        },
                    });
                }
            }
        }

        if (properties.length > 0) {
            testcase.testcase.push({ properties: properties });
        }
    }
    return testcase;
};

module.exports = CustomJunitReporter;

This CustomJunitReporter can now be used in Cypress reporter option.
And this will have the risk, if you make any fixes or update the logic to make it any better :)

[cheuk0324]

any update for this PR? would love to have this merge in

[InduKrish]

Hi , Can you please tell us when this PR is going to get merged? we have been waiting to update our cypress framework to report issues to Jira. Can you please expedite? Thank you in advance.

[lostdog19]

Hi @bitcoder. Can you please merge in changes from the main repo while we wait for the MR to be resolved? I'd like to use your additions to the code, but also the changes for token replacement in the results filename (more than just [hash]).

[bitcoder]

hi @lostdog19 . done

[bitcoder]

@clayreimann do you intend to merge this or any thoughts about this long standing request with so many comments? :)