Include tests for examples in docs

[dvd101x]

Just today I started working with embedded docs. At some point I thought of testing if my examples on the embedded docs worked. And ended up doing something like this.

math.evaluate(math.help(math.add).doc.examples)

A few times I have found that the examples throw an error. So I tested something automatic and found the following functions have errors in their examples

function	example	error
diff	diff([1, 2, 4, 7, 0], 0)	RangeError: Cannot have negative dimension
diff	diff([[1, 2], [3, 4]], 0)	RangeError: Cannot have negative dimension
evaluate	evaluate("sqrt(" + 4 + ")")	Error: Cannot convert "sqrt(" to a number
invmod	invmod(8, 12)=NaN	SyntaxError: Invalid left hand side of assignment operator = (char 14)
invmod	invmod(7, 13)=2	SyntaxError: Invalid left hand side of assignment operator = (char 14)
invmod	invmod(15151, 15122)=10429	SyntaxError: Invalid left hand side of assignment operator = (char 21)
lyap	lyap(A,Q)	Error: Undefined symbol Q
numeric	numeric("4", "Fraction)	SyntaxError: End of string " expected (char 24)
numeric	numeric(fraction(2, 5), "number)	SyntaxError: End of string " expected (char 33)
polynomialRoot	a = polynomialRoot(-6, 11, -6 1)	SyntaxError: Parenthesis ) expected (char 31)
schur	schur(A)	TypeError: Unexpected type of argument in function schur (expected: Array or Matrix, actual: Unit | identifier, index: 0)
simplifyConstant	simplifyConatant("(3-3)*x")	Error: Undefined function simplifyConatant
sylvester	sylvester(A,B,C)	TypeError: Unexpected type of argument in function sylvester (expected: Array or Matrix, actual: Unit | identifier, index: 0)
typed	double("hello")	Error: Cannot convert "hello" to a number

To replicate this you only need to go into the parser of mathjs home and type help for the specific function in the parser

Now, some of these are simple typos or maybe a copy paste from the not embedded docs.

I would like to discuss the possibility of automatic testing for the examples.

[josdejong]

Thanks David! It it a good idea to implement something to automatically test the embedded docs for errors. There is something similar in place to test the comments that are used to generate the docs on the website. Is this something you would like to give a go?

[josdejong]

Thanks for sharing. We could work this script out in more detail and add it as a file in the folder ./tools. Then, we can add an npm script "test:embedded-docs" in package.json that runs the script. Is that something you would like to work out in a PR?

[dvd101x]

It would be something similar to #2453 and #2452 ?

[dvd101x]

I would like to try this in a PR, maybe I can have it ready for next month

[gwhitney]

Switching to here from the discussion in #2452. I agree that at least as a stopgap a script that verifies that each example expression in an embedded doc executes without generating an error is a good idea. But wouldn't it be simpler just to add the test script as say test/node-tests/embedded-docs.test.js and let it run under the existing test harness, rather than add a fifth "variety" of test in addition to test:src, test:node, test:generated, and test:types? It doesn't seem like this test script has to do anything that couldn't easily be performed in a test:node script. Like the existing doc.test.js it can import the mathjs object and then loop through all the symbols, running the embedded doc check on each.

[josdejong]

Good idea Glen, I think you're right and we can create this as a regular test since it does not rely on the build process before it can run.

[dvd101x]

I was checking if all functions that are in the parser have docs and found the following that does not:

• addScalar
• compile
• divideScalar
• equalScalar
• multiplyScalar
• parse
• parser
• replacer
• reviver

I'm not familiar with some of them, does it make sense for them to be in the parser without docs?

To replicate type on the Demo

compile       # yields function
help(compile) # yields Error: No documentation found on "compile"

[josdejong]

Good finds. I think these functions originally where not exported, but now they are. We'll need to write embedded docs for them. This is mostly a copy past from the documentation in the comment of the function itself, though I see they can use some attention too (adding syntax, examples, seealso).

[josdejong]

The fixes of #2976 are published now in v11.9.0