Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add support for linking tests to Jira issues (e.g. stories, requirements), using Xray #153

Open
wants to merge 8 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
88 changes: 88 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,94 @@ var mocha = new Mocha({
})
```

### Append properties to testcase

You can provide metadata for each testcase, i.e., properties for each `testcase`.
Some tools (e.g., Xray) may take advantage of this to provide additional information related to the execution of each testcase or to perform other operations while importing test results (e.g., linking results to existing test cases or stories on some tool, such as Jira).

Zero, one or more properties may be provided using this metadata. You can use whatever name and attributes for the properties.

Properties can be defined by:

- a name (string) and an inline value (string)
- a name (string) and a content (multiline string)
- or a name (string), and a set of objects having one or more attributes, and optionally content

```xml
<testsuites>
<testsuite>
<testcase name="should do some stuff" time="1.7390" classname="demo">
<properties>
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bitcoder i just noticed that this is using a properties element within a testcase which is not valid for Jenkins: https://github.com/jenkinsci/xunit-plugin/blob/master/src/main/resources/org/jenkinsci/plugins/xunit/types/model/xsd/junit-10.xsd

Jenkins only allows property elements under a testsuite element. I could be wrong about this, but we could prove it with a test case that runs this output against the junit XSD. There are a couple of tests in the spec file that will show you how to do this.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, properties are usually an element that appear under testsuite, that's correct. junit has not an official XSD though AFAIK. I had checked before with the Junit project leader Marc Philipp , about the feasibility of having also propertiesunder the testcase element and we didn't see a problem with it; however, some tools may have a more strict validation and support only what ant provided back then, on the first releases of the Junit report format.
The current code only adds properties if testCaseMetadata is provided, which ensures we keep backward compatible; this is independent of the "ant/jenkins" mode flags. However, I understand if you want to have to inhibit this feature if those modes are active.
Having said that, shall we keep the current logic or do you prefer to change it to only make it applicable if not in Jenkins mode and also not in ant mode?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to complement, Datadog also supports properties at testcase level, which
will also be supported by this PR.

<property name="test_key" value="CALC-1"/>
<property name="test_description"><![CDATA[a sample test]]></property>
<property name="testrun_evidence">
<item name="dummy-evidence1.txt"><![CDATA[aGVsbG8=]]></item>
<item name="dummy-evidence2.txt"><![CDATA[d29ybGQ=]]></item>
</property>
</properties>
</testcase>
</testsuite>
</testsuites>
```

To do so, set them on the test object using `testCaseMetadata`.
The syntax is pretty straighforward: the root keys on the `testCaseMetadata` object are mapped to properties, where the key is mapped directly to the `name` attribute. If it holds a string value, then it's mapped to the `value` attribute on the corresponding `property` element; if it holds an object having `_cdata`, then the content will be stored as cdata on the corresponding `property. A more complex scenario is supported, in the case you want to pass a property containing an array of objecs; in this case, the root key will be mapped to a XML element and its keys will be mapped with to attributes on that element.

`_cdata` is a special attribute name that will be used internally as a way to signal that we want to embed its value as cdata content on the corresponding XML element, instead of being mapped to an attribute.

As an example that showcases the different type of properties, in mocha this would be something like:

```javascript
it('should do some stuff', function() {
this.testCaseMetadata = {
test_key: 'CALC-1',
test_description: { _cdata: 'a sample test' },
testrun_evidence: [
{ item: { name: "dummy-evidence1.txt", _cdata: 'aGVsbG8=' }},
{ item: { name: "dummy-evidence2.txt", _cdata: 'd29ybGQ=' }}
]
};
...
});
```

If using Cypress, this could be added directly as the second argument (i.e., the test configuration) on the `it()` block; mocha by itself doesn't provide this facility as shown above though.

```javascript
it('should do some stuff', { testCaseMetadata: {
test_key: 'CALC-1',
test_description: { _cdata: 'a sample test' },
testrun_evidence: [
{ item: { name: "dummy-evidence1.txt", _cdata: 'aGVsbG8=' }},
{ item: { name: "dummy-evidence2.txt", _cdata: 'd29ybGQ=' }}
]
}}, function() {

cy.visit('http://www.example.com');
...
});
```

Most probably, your calls will be much simpler as you may need just to simple pass a property and its value, as shown in the following examples.

mocha:

```javascript
it('should do some stuff', function() {
this.testCaseMetadata = { test_key: 'CALC-1'};
...
});
```

Cypress:

```javascript
it('should do some stuff', { testCaseMetadata: { test_key: 'CALC-1'} }, function() {
cy.visit('http://www.example.com');
...
});
```

### Results Report

Results XML filename can contain `[hash]`, e.g. `./path_to_your/test-results.[hash].xml`. `[hash]` is replaced by MD5 hash of test results XML. This enables support of parallel execution of multiple `mocha-junit-reporter`'s writing test results in separate files. In addition to this these placeholders can also be used:
Expand Down
85 changes: 85 additions & 0 deletions index.js
Original file line number Diff line number Diff line change
Expand Up @@ -314,6 +314,11 @@ MochaJUnitReporter.prototype.getTestsuiteData = function(suite) {
return testSuite;
};

function getTestCaseMetadata (test) {
// in pure mocha this is obtained from the test object context; in Cypress, test metadata is stored in the ._testConfig.unverifiedTestConfig
return (test._testConfig && test._testConfig.unverifiedTestConfig && test._testConfig.unverifiedTestConfig.testCaseMetadata) ? test._testConfig.unverifiedTestConfig.testCaseMetadata : ((test.ctx && test.ctx.testCaseMetadata) ? test.ctx.testCaseMetadata : null);
}

/**
* Produces an xml config for a given test case.
* @param {object} test - test case
Expand All @@ -335,6 +340,86 @@ MochaJUnitReporter.prototype.getTestcaseData = function(test, err) {
}]
};

var testCaseMetadata = getTestCaseMetadata(test);
if (testCaseMetadata) {
/*
Additional testcase metadata can be processed and embed on the JUnit XML report, as properties on the <testcase> element
Xray provides support for processing some of these information (https://docs.getxray.app/display/XRAYCLOUD/Taking+advantage+of+JUnit+XML+reports), others may follow.
The following code provides a more or less generic approach to provide properties for the testcase:
- based on a property name and value
- based on a property name and bulk content
- based on a property name and a array of elements, each one with attributes
*/

var properties = [];
for (var key in testCaseMetadata) {
if (typeof testCaseMetadata[key] === 'string') {
// if the metadata has a key whose value contains a string, then the property is named by the key
// and has a "value" attribute based on its value
// e.g. { test_key: 'CALC-1' }

properties.push({ property: { _attr: { name: key, value: testCaseMetadata[key] } } });
} else if (testCaseMetadata[key] instanceof Array) {
/*
if the metadata has a key whose value contains a array, then each element is in turn a object,
of a specific type/name, with attributes and cdata

e.g.
{
testrun_evidence: [
{ item: { name: "dummy-evidence1.txt", _cdata: 'aGVsbG8=' }},
{ item: { name: "dummy-evidence2.txt", _cdata: 'd29ybGQ=' }}
]
}
*/

// array with properly formatted elements that can be serialized to XML
var items = [];

// process all objects and convert each one to a XML element
// object attributes are mapped to attributes on the XML element, except for "_cdata" which is mapped to cdata
for (var idx = 0; idx < testCaseMetadata[key].length; idx++) {
var item = testCaseMetadata[key][idx];

// the XML element tag name is the only attribute on the object belonging to the array based property
var itemTag = Object.keys(item)[0];

// prepare a object that can be serialized to XML in the format of:
// <itemTag attr1="..." attrN="...""><![CDATA[...]]></itemTag>
var itemElem = {};
itemElem[itemTag] = {};
itemElem[itemTag]._attr = {};
for (var attr in item[itemTag]) {
if (attr === '_cdata') {
itemElem[itemTag]._cdata = item[itemTag][attr];
} else {
itemElem[itemTag]._attr[attr] = item[itemTag][attr];
}
}

// save it to the list of XML elements
items.push(itemElem);
}
var multiElemProperty = { property: items };
multiElemProperty.property.unshift({ _attr: { name: key} });
properties.push(multiElemProperty);
} else if (typeof testCaseMetadata[key] === 'object') {
// if it's an object, non-Array nor string, then it's a property with built-in content to be embed as cdata
// note: a mix of cdata and standard attributes is not supported
// e.g. { test_description: { _cdata: 'a sample test' } }

var _cdata = testCaseMetadata[key]._cdata;
if (_cdata) {
properties.push({ property: { _attr: { name: key }, _cdata: this.removeInvalidCharacters(_cdata) } });
}
}
}

if (properties.length > 0) {
testcase.testcase.push({properties: properties});
}
}

// We need to merge console.logs and attachments into one <system-out> -
// see JUnit schema (only accepts 1 <system-out> per test).
var systemOutLines = [];
Expand Down
63 changes: 63 additions & 0 deletions test/mocha-junit-reporter-spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -854,5 +854,68 @@ describe('mocha-junit-reporter', function() {
});
});
});

describe('custom testCaseMetadata information', function () {
var reporter;
var rootSuite;
var suite1;

beforeEach(function() {
reporter = createReporter({jenkinsMode: true, attachments: true});
rootSuite = reporter.runner.suite;
suite1 = Suite.create(rootSuite, 'suite1');
});

it('maps a string value based atribute from the testCaseMetadata info to a custom property', function(done) {
var test = createTest('mytest', function() {
this.testCaseMetadata = { test_key: 'CALC-1'};
});
suite1.addTest(test);

runRunner(reporter.runner, function() {
expect(reporter._xml).to.include('<properties>');
expect(reporter._testsuites[1].testsuite[1].testcase[1].properties).to.have.lengthOf(1);
expect(reporter._xml).to.include('<property name="test_key" value="CALC-1"/>');
done();
});
});

it('maps a content value based atribute from the testCaseMetadata info to a custom property', function(done) {
var test = createTest('mytest', function() {
this.testCaseMetadata = { test_description: { _cdata:'a sample test' } };
});
suite1.addTest(test);

runRunner(reporter.runner, function() {
expect(reporter._xml).to.include('<properties>');
expect(reporter._testsuites[1].testsuite[1].testcase[1].properties).to.have.lengthOf(1);
expect(reporter._xml).to.include('<property name="test_description"><![CDATA[a sample test]]></property>');
done();
});
});

it('maps array based atribute from the testCaseMetadata info to a custom property, with inner elements', function(done) {
var test = createTest('mytest', function() {
this.testCaseMetadata = {
testrun_evidence: [
{ item: { name: "dummy-evidence1.txt", _cdata: 'aGVsbG8=' }},
{ item: { name: "dummy-evidence2.txt", _cdata: 'd29ybGQ=' }}
]
};
});
suite1.addTest(test);

runRunner(reporter.runner, function() {
expect(reporter._xml).to.include('<properties>');
expect(reporter._testsuites[1].testsuite[1].testcase[1].properties).to.have.lengthOf(1);
expect(reporter._xml).to.include('<property name="testrun_evidence">');
expect(reporter._xml).to.include('<item name="dummy-evidence1.txt"><![CDATA[aGVsbG8=]]></item>');
expect(reporter._xml).to.include('<item name="dummy-evidence2.txt"><![CDATA[d29ybGQ=]]></item>');
done();
});
});

});

});
});