History

The message: OK (XXX tests) should only be printed if ALL tests pass, just as it is the case with AnsiColourResultPrinter. The stylesheet currently prints this messages even though tests have failed. The stylesheet seems to behave correctly if there are errors as well as test failures.

need to restructure regression testing to allow for easy, seperate testing of different UTF-X components

UTF-X tests in the 'test' directory should be broken up into two categories 1. correct tests (expected to pass) 2. incorrect tests (expected to fail or generate error) correct tests should then be included in the full regression test and executed once for each transformer factory impl supported just as the samples

TestGenerator prints incorrect usage message; pls improve the documentation

need to be able to normalise white space insode elements, so more complex source and expected fragments can be tested even after auto formatting by an XML editor. For example, one may want to consider the following two paragraphs as equivalent: <para>This is an example of some long pragraph with text which will wrap if formatted within an XML editor such as oXygen</para> <para>This is an example of some long pragraph with text which will wrap if formatted within an XML editor such as oXygen</para> This feature is over and above the Canonical XML 1.0 specification

--- build.xml~ 2006-08-23 11:34:36.000000000 -0400 +++ build.xml 2006-09-02 16:26:14.984075200 -0400 @@ -77,15 +77,15 @@ <!--~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~--> <!-- Java 1.5 bundled Xalan XSLTC processor - <property name="tarnsformer_factory_impl" value="com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl" /> + <property name="transformer_factory_impl" value="com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl" /> Xalan XSLTC processor - <property name="tarnsformer_factory_impl" value="org.apache.xalan.xsltc.trax.TransformerFactoryImpl" /> + <property name="transformer_factory_impl" value="org.apache.xalan.xsltc.trax.TransformerFactoryImpl" /> Xalan Interpretive processor - <property name="tarnsformer_factory_impl" value="org.apache.xalan.processor.TransformerFactoryImpl" /> + <property name="transformer_factory_impl" value="org.apache.xalan.processor.TransformerFactoryImpl" /> Saxon 8B XSLT 2.0 processor - <property name="tarnsformer_factory_impl" value="net.sf.saxon.TransformerFactoryImpl" /> + <property name="transformer_factory_impl" value="net.sf.saxon.TransformerFactoryImpl" /> --> - <property name="tarnsformer_factory_impl" value="net.sf.saxon.TransformerFactoryImpl" /> + <property name="transformer_factory_impl" value="net.sf.saxon.TransformerFactoryImpl" /> @@ -216,7 +216,7 @@ <!--~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~--> <target name="rt" depends="jar"> <java fork="true" classname="${testrunner}"> - <jvmarg value="-Djavax.xml.transform.TransformerFactory=${tarnsformer_factory_impl}" /> + <jvmarg value="-Djavax.xml.transform.TransformerFactory=${transformer_factory_impl}" /> <jvmarg value="-enableassertions" /> <jvmarg value="-Xmx512m" /> <classpath refid="runtime.classpath" /> @@ -279,7 +279,7 @@ <java classname="${testrunner}" fork="yes"> - <jvmarg value="-Djavax.xml.transform.TransformerFactory=${tarnsformer_factory_impl}" /> + <jvmarg value="-Djavax.xml.transform.TransformerFactory=${transformer_factory_impl}" /> <jvmarg value="-Dutfx.test.dir=samples" /> <!-- <jvmarg value="-Dutfx.test.file=samples/utfxdoc/test/utfxdoc_xhtml_test.xml" /> --> <jvmarg value="-enableassertions" /> @@ -356,7 +356,7 @@ <target name="tests" depends="jar"> <java classname="${testrunner}" fork="yes"> - <jvmarg value="-Djavax.xml.transform.TransformerFactory=${tarnsformer_factory_impl}" /> + <jvmarg value="-Djavax.xml.transform.TransformerFactory=${transformer_factory_impl}" /> <jvmarg value="-Dutfx.test.dir=tests" /> <jvmarg value="-enableassertions" /> <arg value="utfx.framework.XSLTRegressionTest" /> @@ -377,7 +377,7 @@ <target name="testfile-has-error" depends="jar"> <java classname="${testrunner}" fork="yes"> - <jvmarg value="-Djavax.xml.transform.TransformerFactory=${tarnsformer_factory_impl}" /> + <jvmarg value="-Djavax.xml.transform.TransformerFactory=${transformer_factory_impl}" /> <jvmarg value="-Dutfx.test.file=tests/test/broken_stylesheet_test.xml" /> <jvmarg value="-enableassertions" /> <arg value="utfx.framework.XSLTRegressionTest" />

jvm 1.5.0_06

Found in branch_0_0_x STEPS TO REPRODUCE: * define ant target in build.xml <target name="test" depends="jar"> <java classname="${testrunner}" fork="yes"> <jvmarg value="-Djavax.xml.transform.TransformerFactory=$ {tarnsformer_factory_impl}" /> <jvmarg value="-Dutfx.test.file=tests/test/broken_stylesheet_test.xml" /> <jvmarg value="-enableassertions" /> <arg value="utfx.framework.XSLTRegressionTest" /> <classpath refid="runtime.classpath" /> </java> </target> * ant test ACTUAL RESULT: [java] OK (0 tests) EXPECTED RESULT: [java] Tests run: 1, Errors: 1 [java] Java Result: 1

need to manage history of changes in an XML file; the history.xml file should show releases with changelog of most significant enhancemts/bug fixes etc. This history should be integrated with issue tracking and CVS logs. It should be a UTF-X sample just as UTF-X Doc.

jvm 1.5.0_06

XSL:FO stylesheet does not work as it is out of synch with the DTD

As well as chosing a result printer to render test run result, one should be able to tell the framework to render the output into HTML and open the file in browser after test run. This HTML rendition should done be via the XMLResultPrinter, which can be done now, but transformation is an additional manual step.

the manual gives examples of using UTF-X from a command line but does not talk about CLASSPATH. If utfx.jar has all the required libraries in its manifest then it will be much easier to run it from the command line

very ugly output

When createing test summary for errors, the XMLResultPrinter does not output the name of the test that vaused the error

Currently the TestGenerator does not support named templates. It just inserts a comment into the generated TDF: <!-- ignoring name teplate XXX (not yet supported) -->

TDF should allow one to specify a CSS stylesheet which will be used to style the <utfx:expected> fragment. This feature will allow to produce more realistic HTML output. DTD will also need to be modified.

Found in branch_0_0_x STEPS TO REPRODUCE: * define ant target in build.xml <target name="test" depends="jar"> <java classname="${testrunner}" fork="yes"> <jvmarg value="-Djavax.xml.transform.TransformerFactory=$ {tarnsformer_factory_impl}" /> <jvmarg value="-Dutfx.test.file=tests/test/broken_stylesheet_test.xml" /> <jvmarg value="-enableassertions" /> <arg value="utfx.framework.XSLTRegressionTest" /> <classpath refid="runtime.classpath" /> </java> </target> * ant test ACTUAL RESULT: [java] OK (0 tests) EXPECTED RESULT: [java] Tests run: 1, Errors: 1 [java] Java Result: 1

The root element passed to stylesheet under test is <utfx-wrapper> and not the child of the <utfx:source> element in the TDF. Therefore absolute XPath expressions can't be used in the stylesheet under test. ============================== STEPS TO REPRODUCE: Stylesheet under test: <xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0"> <xsl:template name="absolute-copy-of"> <xsl:copy-of select="/"/> </xsl:template> </xsl:stylesheet> TDF: <?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="../../../src/xsl/test_definition_xhtml.xsl"?> <utfx:tests xmlns:utfx="http://utfx.org/test-definition"> <utfx:stylesheet src="absolute_xpath_tests.xsl"/> <utfx:test> <utfx:name>absolute xpath test</utfx:name> <utfx:call-template name="absolute-copy-of"/> <utfx:assert-equal> <utfx:source> <root> <branch> <leaf></leaf> </branch> </root> </utfx:source> <utfx:expected> <root> <branch> <leaf></leaf> </branch> </root> </utfx:expected> </utfx:assert-equal> </utfx:test> </utfx:tests> Actual result: [java] [1] absolute xpath test [java] actual: [java] <utfx-wrapper>< [ DIFF ] utfx-wrapper><root><branch><leaf></leaf></branch></root></utfx-wrapper></utfx-wrapper> [java] expected: [java] <utfx-wrapper>< [ DIFF ] root><branch><leaf></leaf></branch></root></utfx-wrapper> Expected result: Passed

invalid utf-8 character (one byte encoding). This can be reproduced on Windows with the current version (1.6.2.1) of address_book_xhtml.xml TDF. My guess is that the non braking character 0x160 is causing the problem. Oxygen claims that the document is well formed so my guess that this is caused by some intermediate DOM serialisation into a platform specific encoding (which I don't believe is utf-8 on windows). It works for me on Linux where the default platform encoding is utf-8.

We have some stylesheets that use <xsl:param> (both top level and template level). Can there be some way in the test definition to set the value of these parameters?

Need to complete both XSLT stylesheets and the corresponding tests. Also need a larger sample XML document. You may want to add other elements to the DTD. I don't really care what you do with as it is a stand alone samples and does not have to comply with any utf-x structure/style rules. In fact it would be better if it was different to other stylesheets and TDF in the framework to show a slightly different way of using UTF-X.

Xalan fails to correctly import import the stylesheet from the wrapper when running under Windows. Possibly '/' vs '\' issue.

currently the test generator parsers the stylesheet and generates tests based on the templates found in the stylesheet. for each template test generator creates an element with some text, passes the element through the stylesheet and uses source and output to form a utf-x test. It would be helpful if a user could also pass a sample XML document (or W3C Schema) to the generator (as well as the XSLT) and use that document/schema as the source of element structure. Generated tests would be then much more meaningful

currently all tests (and samples) use XSLT 1.0 stylesheets; we should ensure the framework works with XSLT 2.0 as well.

TDF rendition does not output source fragment attributes. Will be fixed in 0.0.7

utf-x sample unit tests should be executed as part of the regression test. furthermore, all samples should be executed once for each XSLT processor supported ; currently j2se5 buil in, xalan and saxon8 utf-x tests that are executed as part of the 'tests' target should also become part of the regression test but this is little bit more tricky as for some tests correct behaviour is to either fail or throw an exception. Some sort of a wrapper test case would have to be written.

If the xsl or xml is invalid, you just get a message saying "0 tests" and you have to view the log file to see any errors. It would be useful to see any validation errors in the test summary.

if there are TDFs at the same level as test directories with TDFs then the printer will print incorrect test suite name. This is because suite names are only set upon entering a suite and assumption was made that every suite name will be set correctly upon starting that suite. this does not work if a TDF follows a suite at the same level (ie a test case is a sibling of a test suite). Test suite names will have to be stored on a stack. This may also apply to other result printers.

need a stylesheet to render output generated by the XMLResultPrinter so test results can be displayed in a HTML page

path problems on windows; need to write windows specific unit tests

if there is a <utfx:source-builder> element at the top level, it is always invoked even if use-source-parser="no" is specified on the <utfx:source> element.

this happens if the named template assumes that the current context node is what it expects. currently all named templates are called from the root node (i.e. utfx-wrapper), which is not always correct, instead the current context should be set to the first child element of <utfx-wrapper>.

this can happens when validate="no" and use-source-parser="yes". the root element is not initialised, so an invalid DOCTYPE is generated. also rawSource is not set under this situation, so an empty fragment is passed to the SourceBuilder.

this stylesheet needs to be updated so its usable; this is an important part of the system

if a test definition file does not have <utfx:source-validation> element but validate="true" attribute is present on a <utfx:source> element then the framework attempts to validate the fragment ... and obviously fails cos no DTD is defined. validate="true" should be ignored on <utfx:source> elements unles <utfx:source-validation> is specified. Same applies to <utfx:expected> !

for some reason <utfx:message> content is not used when test fails.

need to be able to generate UTF-X test definition file skeleton from a an XSLT stylesheet

currently ANSIColourResultPrinter is the default result printer, but it does not work well on windows platforms as they do not suppport ANSI colour escaping. default result printer should be different for Linux and other platforms so result printing works well out of the box.

if <expected> fragment is not well formed then the test fails where error should be generated.

unit tests are required for all assert*() methods defined in UTFXUnitTest class