Issues in digesting boost test results

[shishirmisra]

Hi.
I'm trying to get Sonarqube to understand boost test results from some production code. My code is laid out like this due to some internal recommendations -
Base -> Lib
-> Test

The actual code resides in the Lib directory. Based on information from various Confluence pages here, I've created a sonar-project.properties file at the level of the Base directory with these entries -

sonar.projectBaseDir=.
sonar.sources=Lib/
sonar.tests=Test/
sonar.cxx.xunit.xsltURL=boosttest-1.x-to-junit-1.0.xsl
sonar.cxx.xunit.reportPath=boost*.xml

As per https://github.com/wenns/sonar-cxx/wiki/Test-runners, I generated a boost result file like this -'mytest.exe --output_format=XML --log_sink=boostlog.xml --log_level=all --report_level=no'. I then ran sonar-runner -X to see what was happening. The runner created a transformed file for the test report with entries like this -

+[Info] - check writeHandler->writeData(str, strlen(str), vec) == 18 passed - == [File] - BlobStore_test.cpp - == [Line] - 56 + .........................................

The sonar-runner section for the parsing of the test report contained this -

12:51:12.180 DEBUG - Using pattern 'boost*.xml' to find reports
12:51:12.181 INFO - Processing report 'Z:\src\lib\BlobStorage.\boostlog.xml'
12:51:12.181 DEBUG - Transforming the report using xslt 'boosttest-1.x-to-junit-1.0.xsl'
12:51:12.311 INFO - Parsing report 'Z:\src\lib\BlobStorage.\boostlog.xml.after_xslt'
12:51:12.374 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobReadsuite.', creating a dummy one
12:51:12.375 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobSparseWritesuite.BlobStore_test', creating a dummy one
12:51:12.375 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobStoragePresentShare.BlobStore_test', creating a dummy one
12:51:12.375 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobStoragePresentShare.', creating a dummy one
12:51:12.376 DEBUG - Cannot find the source file for test 'BlobStore_test.blobStoragePresentRemoveShareAndLinks.BlobStore_test', creating a dummy one
12:51:12.376 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobDeletesuite.BlobStore_test', creating a dummy one
12:51:12.376 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobWritesuite.BlobStore_test', creating a dummy one
12:51:12.377 INFO - Sensor CxxXunitSensor done: 403 ms

  End result - no test results show in the relevant Sonarqube project. Am I doing something wrong? Or can boost results not be parsed right now? These are the tool versions that I'm using in my setup -

Sonarqube - 4.3
C++ (community) - 0.9.1
Boost - 1.44

Since the logs pertain to production code, I'm not attaching them wholesale, but can email them if needed.

Thanks,
Shishir

[wenns]

Hi Shishir,
please prevent the wiki from interpreting the contents of your report. Format it as code or something.

[shishirmisra]

Thanks Walerie. 2nd try -

    I'm trying to get Sonarqube to understand boost test results from some production code.
My code is laid out like this due to some internal recommendations - 

Base -> Lib
     -> Test

    The actual code resides in the Lib directory. Based on information from various 
Confluence pages here, I've created a sonar-project.properties file at the level of the
Base directory with these entries -

sonar.projectBaseDir=.
sonar.sources=Lib/
sonar.tests=Test/
sonar.cxx.xunit.xsltURL=boosttest-1.x-to-junit-1.0.xsl
sonar.cxx.xunit.reportPath=boost*.xml

    As per https://github.com/wenns/sonar-cxx/wiki/Test-runners, I generated a boost result
file like this -'mytest.exe --output_format=XML --log_sink=boostlog.xml --log_level=all
--report_level=no'. I then ran sonar-runner -X to see what was happening. The runner created
a transformed file for the test report with entries like this -

<?xml version="1.0" encoding="UTF-8"?><testsuite tests="23" errors="4" failures="2" name="MergedTestSuite" skipped="0">
<testcase classname="BlobStore_test.BlobWritesuite.BlobStore_test" name="TestWriteData_dataLen" time="0.001">
<system-out>
+[Info] - check writeHandler-&gt;writeData(str, strlen(str), vec) == 18 passed
+ == [File] - BlobStore_test.cpp
+ == [Line] - 56
+</system-out>
</testcase>
.........................................

The sonar-runner section for the parsing of the test report contained this -

12:51:12.180 DEBUG - Using pattern 'boost*.xml' to find reports
12:51:12.181 INFO - Processing report 'Z:\src\lib\BlobStorage.\boostlog.xml'
12:51:12.181 DEBUG - Transforming the report using xslt 'boosttest-1.x-to-junit-1.0.xsl'
12:51:12.311 INFO - Parsing report 'Z:\src\lib\BlobStorage.\boostlog.xml.after_xslt'
12:51:12.374 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobReadsuite.', creating a dummy one
12:51:12.375 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobSparseWritesuite.BlobStore_test', creating a dummy one
12:51:12.375 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobStoragePresentShare.BlobStore_test', creating a dummy one
12:51:12.375 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobStoragePresentShare.', creating a dummy one
12:51:12.376 DEBUG - Cannot find the source file for test 'BlobStore_test.blobStoragePresentRemoveShareAndLinks.BlobStore_test', creating a dummy one
12:51:12.376 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobDeletesuite.BlobStore_test', creating a dummy one
12:51:12.376 DEBUG - Cannot find the source file for test 'BlobStore_test.BlobWritesuite.BlobStore_test', creating a dummy one
12:51:12.377 INFO - Sensor CxxXunitSensor done: 403 ms

    End result - no test results show in the relevant Sonarqube project. Am I doing 
something wrong? Or can boost results not be parsed right now? These are the tool versions
that I'm using in my setup -
Sonarqube - 4.3
C++ (community) - 0.9.1
Boost - 1.44

Since the logs pertain to production code, I'm not attaching them wholesale, but can email 
them if needed.

Thanks,
Shishir

[guwirth]

Only a general hint V0.9.1 and SonarQube 3.7.4 with boost tests is working fine with real and virtual files.

Cannot find the source file for test 'BlobStore_test.BlobReadsuite.
Cannot find the source file for test 'BlobStore_test.BlobSparseWritesuite.BlobStore_test', creating a dummy one

• Does the file BlobStore_test.BlobReadsuite. exist?
• file has no file extension?
• is the file below root or outside?

[wenns]

OK, now I see the problem. Your test results arent visible because the plugin cannot find the according test files. They cannot be found because your report doesnt provide a path for them in 'classname=...'. Now, the plugin tries to create a dummy resource instead of the real one but this has been disabled in API's of recent SQ versions (see also #263 (comment)).

So you have the following options:

• (somehow) make sure that your reports provide paths to the test files, relative to projects root, in the 'classname' properties, or
• wait until we patch the plugin to save the overall test results in context of the project itself.

[shishirmisra]

@wenns - Thanks for the tips. Do you have a timeline in mind for the patch? I'm trying to generate a suitable XML file anyway but the patch would help for projects with multiple test files. Also, I'm not clear about the content of the 'classname' property. The inbuilt XSLT file populates that with the file name and the test-suite name. What would be a good value for this property? Any example?

@guwirth - to answer your questions -

• all tests are in a single file called BlobStore_test.cpp. The classname field is being filled by the boosttest-1.x-to-junit-1.0.xsl transform

• the file has a valid extension

• the file is present in the directory specified by sonar.tests.

  If you have a working setup, can you paste a few entries from your boost XML output and your xslt output? Are you using the same command that I'm using - mytest.exe --output_format=XML --log_sink=boostlog.xml --log_level=all --report_level=no'?

Thanks for your help thus far!
Regards,
Shishir

[wenns]

On 09/03/2014 11:26 AM, shishirmisra wrote:

@wenns https://github.com/wenns - Thanks for the tips. Do you have a
timeline in mind for the patch? I'm trying to generate a suitable XML
file anyway but the patch would help for projects with multiple test
files. Also, I'm not clear about the content of the 'classname'
property. The inbuilt XSLT file populates that with the file name and
the test-suite name. What would be a good value for this property? Any
example?

The most reliable method is to provide a FS path to the test file,
relative to projects root. So, for example, if your project is
/home/joe/mycoolcode
and the test file is
/home/joe/mycoolcode/src/component1/tests/test_foo.cc
the property classname should be set to:
src/component1/tests/test_foo.cc

(I know, its does not correspond with the name of the property, but you
got the plot...)

Just classnames can also work (the plugin uses the AST for lookups
then), the obvious precondition is that the files can be parsed.

@guwirth https://github.com/guwirth - to answer your questions -

• all tests are in a single file called BlobStore_test.cpp. The
  classname field is being filled by the
  boosttest-1.x-to-junit-1.0.xsl transform

• the file has a valid extension
  *

  the file is present in the directory specified by sonar.tests.

  If you have a working setup, can you paste a few entries from your
  boost XML output and your xslt output? Are you using the same
  command that I'm using - mytest.exe --output_format=XML
  --log_sink=boostlog.xml --log_level=all --report_level=no'?

Thanks for your help thus far!
Regards,
Shishir

—
Reply to this email directly or view it on GitHub
#276 (comment).

[shishirmisra]

Just classnames can also work (the plugin uses the AST for lookups
then), the obvious precondition is that the files can be parsed.

That's an interesting point actually. The sonar output shows a set of errors for the test CPP file. I'm not sure if I should open another issue for that. Basically, BlobStore_test.cpp includes the main header file present under the sonar.sources directory, and sonar-runner complains about missing headers from that main header -

14:53:52.918 WARN  - [Z:\src\lib\BlobStorage\BlobStorage_lib\BlobStorage.h:19]: cannot find the sources for '#include <boost/uuid/uuid.hpp>'
14:53:52.918 WARN  - [Z:\src\lib\BlobStorage\BlobStorage_lib\BlobStorage.h:20]: cannot find the sources for '#include <boost/uuid/uuid_generators.hpp>'

Similar errors are also seen for BlobStore_test.h header. This leads to these syntax errors during parsing -

14:53:53.086 WARN  - [Z:\src\lib\BlobStorage\BlobStorage_test\BlobStore_test.cpp:42]: syntax error, skip 'BOOST_FIXTURE_TEST_SUITE'

Basically, it looks like sonar-runner can't use sonar.cxx.includeDirectories while parsing code under sonar.tests. The same include paths definitely work while parsing code under sonar.sources. Any idea what is going on? Does this merit another bug report?

[wenns]

On 09/03/2014 01:36 PM, shishirmisra wrote:

Just classnames can also work (the plugin uses the AST for lookups
then), the obvious precondition is that the files can be parsed.

That's an interesting point actually. The sonar output shows a set of
errors for the test CPP file. I'm not sure if I should open another
issue for that. Basically, BlobStore_test.cpp includes the main header
file present under the sonar.sources directory, and sonar-runner
complains about missing headers from that main header -

|14:53:52.918 WARN - [Z:\src\lib\BlobStorage\BlobStorage_lib\BlobStorage.h:19]: cannot find the sources for '#include <boost/uuid/uuid.hpp>'
14:53:52.918 WARN - [Z:\src\lib\BlobStorage\BlobStorage_lib\BlobStorage.h:20]: cannot find the sources for '#include <boost/uuid/uuid_generators.hpp>'
|

Similar errors are also seen for BlobStore_test.h header. This leads to
these syntax errors during parsing -

|14:53:53.086 WARN - [Z:\src\lib\BlobStorage\BlobStorage_test\BlobStore_test.cpp:42]: syntax error, skip 'BOOST_FIXTURE_TEST_SUITE'
|

Basically, it looks like sonar-runner can't use
sonar.cxx.includeDirectories while parsing code under sonar.tests. The
same include paths definitely work while parsing code under
sonar.sources. Any idea what is going on? Does this merit another bug
report?

Hmm... Looks like.

—
Reply to this email directly or view it on GitHub
#276 (comment).

[guwirth]

Please have also a look to https://github.com/wenns/sonar-cxx/wiki/Path-and-path-separator-issues

[wenns]

The issue should be handled with #313. Marking it as a duplicate. Please reopen if not.