Cxx community plugin sonar scan analysis taking too long to complete

[fayaz988]

i am using sonar community 9.9 LTS
sonar community cxx 4.1.3 jar
it is taking too long to complete sonar scan analysis after report. xml generation because of it big project but
is there any we can make it faster.

Run sensors on project

INFO: Sensor CXX [cxx]
INFO: Sensor CXX [cxx] (done) | time=9859297ms
INFO: Sensor CXX Cppcheck report import [cxx] 

[guwirth]

Hi @fayaz988,

from your description I understand that the overall time is high.

can you please turn debug info on to see in which sensor the time is used up?
https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Get-Debug-Information

Sensor Sensor CXX [cxx] is the squid sensor and not the Cppcheck sensor.

because of it big project but

How big is your project (LoC)?

is there any we can make it faster

You can try https://github.com/SonarOpenCommunity/sonar-cxx/wiki/sonar.cxx.squid.disabled

Regards,

[Nekto89]

@guwirth Is there any difference how plugin analyzes input depending on SonarQube version? On SQ 8.9 local analysis takes about 9 minutes, and on SQ 9.9 it's almost 3 hours. I'll try enabling debug information on next run.

[Nekto89]

My project is 1.5M lines of code.

00:15:02.892 INFO: Sensor CXX [cxx] (done) | time=9602093ms
00:15:05.005 INFO: Sensor CXX Cppcheck report import [cxx] (done) | time=2113ms
00:16:12.573 INFO: Sensor CXX Visual C++ compiler report import [cxx] (done) | time=67568ms
00:16:19.235 INFO: Sensor CXX Bullseye coverage report import [cxx] (done) | time=6662ms

From log I don't see any specific place that takes lots of time individually. There are lots of
"Not enough content in '.......h' to have CPD blocks, it will not be part of the duplication detection"
"CPD error in file '......h' at line:XXXX, column:XX"

I've tried enabling profiling.xml
-X -Dsonar.verbose=true -Dsonar.showProfiling=true
but I can't find it anywhere.

[guwirth]

Is there any difference how plugin analyzes input depending on SonarQube version? On SQ 8.9 local analysis takes about 9 minutes, and on SQ 9.9 it's almost 3 hours. I'll try enabling debug information on next run.

@Nekto89
Are you using the same cxx plugin version with both SQ versions?
Is only the SQ version different?
At least on our integration server the time is not that different, sample
https://github.com/SonarOpenCommunity/sonar-cxx/actions/runs/4797926536

[guwirth]

"Not enough content in '.......h'

@Nekto89 means typically only that the file is too small.

[Nekto89]

Is there any difference how plugin analyzes input depending on SonarQube version? On SQ 8.9 local analysis takes about 9 minutes, and on SQ 9.9 it's almost 3 hours. I'll try enabling debug information on next run.

@Nekto89 Are you using the same cxx plugin version with both SQ versions? Is only the SQ version different? At least on our integration server the time is not that different, sample https://github.com/SonarOpenCommunity/sonar-cxx/actions/runs/4797926536

I've also updated sonar.scanner.appVersion from 4.6.2 to 4.8, can it be because of this? I think that I've tried it once with Java 11 and next with Java 17 with the same result.

8.9 context:

SonarQube plugins:
  - Dependency-Check 2.0.8 (dependencycheck)
  - C++ (Community) 2.1.0.428 (cxx)
  - Python Code Quality and Security 3.4.1.8066 (python)
  - CSS Code Quality and Security 1.4.2.2002 (cssfamily)
  - Go Code Quality and Security 1.8.3.2219 (go)
  - JaCoCo 1.1.1.1157 (jacoco)
  - Kotlin Code Quality and Security 1.8.3.2219 (kotlin)
  - JavaScript/TypeScript Code Quality and Security 7.4.4.15624 (javascript)
  - TFVC 2.3.1 (scmtfvc)
  - Ruby Code Quality and Security 1.8.3.2219 (ruby)
  - Scala Code Quality and Security 1.8.3.2219 (sonarscala)
  - C# Code Quality and Security 8.22.0.31243 (csharp)
  - Java Code Quality and Security 6.15.1.26025 (java)
  - HTML Code Quality and Security 3.4.0.2754 (web)
  - Flex Code Quality and Security 2.6.1.2564 (flex)
  - XML Code Quality and Security 2.2.0.2973 (xml)
  - PHP Code Quality and Security 3.17.0.7439 (php)
  - VB.NET Code Quality and Security 8.22.0.31243 (vbnet)
Global server settings:
  - email.from=...
  - email.prefix=[SONARQUBE]
  - email.smtp_host.secured=******
  - email.smtp_password.secured=******
  - email.smtp_port.secured=******
  - email.smtp_username.secured=******
  - sonar.core.id=...
  - sonar.core.serverBaseURL=...
  - sonar.core.startTime=2023-04-14T11:21:23+0200
  - sonar.cxx.errorRecoveryEnabled=true
  - sonar.exclusions=**/Reference.cs,**/Strings.Designer.cs,**/jquery*.js,**/knockout*.js,**/*.min.js,**/bootstrap*.js,**/*knockout.js
  - sonar.forceAuthentication=true
  - sonar.html.file.suffixes=.html,.aspx,.ascx,.cshtml
  - sonar.lf.logoUrl=...
  - sonar.plugins.risk.consent=ACCEPTED
  - sonar.scm.disabled=true
  - sonar.technicalDebt.ratingGrid=0.005,0.01,0.02,0.03
  - sonar.tfvc.collectionuri=...
Project server settings:
  - sonar.cxx.metric.api.file.suffixes=.h
Project scanner properties:
  - sonar.cxx.bullseye.reportPaths=../overall.xml
  - sonar.cxx.cppcheck.reportPaths=../cppcheck.xml
  - sonar.cxx.defines=SDK_DLL_IMPORT
SDK_DLL_EXPORT
PATCHBASEDSEG_BEGIN=namespace SDK { namespace PatchBasedSegmentation {
PATCHBASEDSEG_END=}}
  - sonar.cxx.file.suffixes=.hxx,.hpp,.hh,.h,.txx,.inl,.cxx,.cpp,.cc,.c
  - sonar.cxx.metric.api.file.suffixes=.h
  - sonar.cxx.vc.encoding=UTF-8
  - sonar.cxx.vc.regex=(.*>)?(?<file>.*)\((?<line>\d+)\):\x20warning\x20(?<id>C\d+):(?<message>.*)
  - sonar.cxx.vc.reportPaths=C:/Software/Jenkins/workspace/Git_SonarQube_Microcores/src/build.log
  - sonar.dependencyCheck.htmlReportPath=../dependency-check-report.html
  - sonar.dependencyCheck.reportPath=../dependency-check-report.xml
  - sonar.host.url=...
  - sonar.language=cxx
  - sonar.login=******
  - sonar.projectBaseDir=C:\Software\Jenkins\workspace\Git_SonarQube_Microcores\src\Libraries
  - sonar.projectDate=2023-04-29T13:42:49+0000
  - sonar.projectKey=SDK:master
  - sonar.projectName=SDK:Microcores
  - sonar.projectVersion=9c5a95a8dc3d3250d961dcfbf2c2be0425ea6b8c
  - sonar.scanner.app=ScannerCLI
  - sonar.scanner.appVersion=4.6.2.2472
  - sonar.sourceEncoding=UTF-8
  - sonar.working.directory=C:\Software\Jenkins\workspace\Git_SonarQube_Microcores\src\Libraries\.scannerwork

9.9 context:

Plugins:
  - Dependency-Check 3.1.0 (dependencycheck)
  - C++ (Community) 2.1.0.428 (cxx)
Bundled analyzers:
  - Python Code Quality and Security 3.24.0.10784 (python)
  - Go Code Quality and Security 1.11.0.3905 (go)
  - JaCoCo 1.3.0.1538 (jacoco)
  - Kotlin Code Quality and Security 2.12.0.1956 (kotlin)
  - IaC Code Quality and Security 1.11.0.2847 (iac)
  - JavaScript/TypeScript/CSS Code Quality and Security 9.13.0.20537 (javascript)
  - Ruby Code Quality and Security 1.11.0.3905 (ruby)
  - Scala Code Quality and Security 1.11.0.3905 (sonarscala)
  - C# Code Quality and Security 8.51.0.59060 (csharp)
  - Java Code Quality and Security 7.16.0.30901 (java)
  - HTML Code Quality and Security 3.7.1.3306 (web)
  - Flex Code Quality and Security 2.8.0.3166 (flex)
  - XML Code Quality and Security 2.7.0.3820 (xml)
  - PHP Code Quality and Security 3.27.1.9352 (php)
  - Text Code Quality and Security 2.0.2.1090 (text)
  - VB.NET Code Quality and Security 8.51.0.59060 (vbnet)
  - Configuration detection fot Code Quality and Security 1.2.0.267 (config)
Global server settings:
  - email.from=...
  - email.prefix=[SONARQUBE]
  - sonar.core.id=...
  - sonar.core.serverBaseURL=...
  - sonar.core.startTime=2023-05-02T16:52:54+0200
  - sonar.cxx.errorRecoveryEnabled=true
  - sonar.exclusions=**/Reference.cs,**/Strings.Designer.cs,**/jquery*.js,**/knockout*.js,**/*.min.js,**/bootstrap*.js,**/*knockout.js
  - sonar.forceAuthentication=true
  - sonar.html.file.suffixes=.html,.aspx,.ascx,.cshtml
  - sonar.lf.logoUrl=...
  - sonar.plugins.risk.consent=ACCEPTED
  - sonar.scm.disabled=true
  - sonar.technicalDebt.ratingGrid=0.005,0.01,0.02,0.03
Project server settings:
  - sonar.cxx.metric.api.file.suffixes=.h
Project scanner properties:
  - sonar.cxx.bullseye.reportPaths=../overall.xml
  - sonar.cxx.cppcheck.reportPaths=../cppcheck.xml
  - sonar.cxx.defines=SDK_DLL_IMPORT
SDK_DLL_EXPORT
PATCHBASEDSEG_BEGIN=namespace SDK { namespace PatchBasedSegmentation {
PATCHBASEDSEG_END=}}
  - sonar.cxx.file.suffixes=.hxx,.hpp,.hh,.h,.txx,.inl,.cxx,.cpp,.cc,.c
  - sonar.cxx.metric.api.file.suffixes=.h
  - sonar.cxx.vc.encoding=UTF-8
  - sonar.cxx.vc.regex=(.*>)?(?<file>.*)\((?<line>\d+)\):\x20warning\x20(?<id>C\d+):(?<message>.*)
  - sonar.cxx.vc.reportPaths=C:/Software/Jenkins/workspace/Git_SonarQube_Microcores/src/build.log
  - sonar.dependencyCheck.htmlReportPath=../dependency-check-report.html
  - sonar.dependencyCheck.reportPath=../dependency-check-report.xml
  - sonar.host.url=...
  - sonar.language=cxx
  - sonar.login=******
  - sonar.projectBaseDir=C:\Software\Jenkins\workspace\Git_SonarQube_Microcores\src\Libraries
  - sonar.projectDate=2023-04-29T13:42:49+0000
  - sonar.projectKey=SDK:master
  - sonar.projectName=SDK:Microcores
  - sonar.projectVersion=9c5a95a8dc3d3250d961dcfbf2c2be0425ea6b8c
  - sonar.scanner.app=ScannerCLI
  - sonar.scanner.appVersion=4.8.0.2856
  - sonar.sourceEncoding=UTF-8
  - sonar.working.directory=C:\Software\Jenkins\workspace\Git_SonarQube_Microcores\src\Libraries\.scannerwork

Input should be the same, because I just reran sonar scanner again on the same folder with different server.

[Nekto89]

First 21:35:00.946 DEBUG: process unit
Last 21:37:56.186 DEBUG: process unit 

So it's just 3 minutes?
Together there are 13753 entries "process unit" which seems to be just the number of C++ files.

Then there are only CPD error in file/Not enough content/Highlighting error until
00:15:02.892 INFO: Sensor CXX [cxx] (done) | time=9602093ms
Does this somehow help? Should I run with some additional verbose options?

[guwirth]

Together there are 13753 entries "process unit" which seems to be just the number of C++ files.

@Nekto89 please have a look to our Wiki. At the end there is an explanation how to interpret the LOG files.

• https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Scan-Source-Code
• https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Scan-Report-Files

[guwirth]

Hi @Nekto89,

think there are many combinations:

• OS: Windows or Linux
• Java: 11 or 17
• SQ: 8.9 or 9.9 LTS
• scanner: 4.6.2 or 4.8
• cxx plugin version: seems to be always C++ (Community) 2.1.0.428 (cxx)

Which combinations did you exactly test/compare?
Seems cxx plugin is the same (so something else must be the root cause - could also be changed API)?

Maybe you can compare sensor times fast/slow to see at least which sensor is slower (seems Sensor CXX)?

00:15:02.892 INFO: Sensor CXX [cxx] (done) | time=9602093ms
00:15:05.005 INFO: Sensor CXX Cppcheck report import [cxx] (done) | time=2113ms
00:16:12.573 INFO: Sensor CXX Visual C++ compiler report import [cxx] (done) | time=67568ms
00:16:19.235 INFO: Sensor CXX Bullseye coverage report import [cxx] (done) | time=6662ms

You can also try to disable SQUID to see if Sensor CXX is faster:
https://github.com/SonarOpenCommunity/sonar-cxx/wiki/sonar.cxx.squid.disabled

Interesting is also if there are many syntax errors (which could slow done the parser):
https://github.com/SonarOpenCommunity/sonar-cxx/wiki/Detect-and-fix-parsing-errors

Regards,

[Nekto89]

without heavy squid part as expected it only takes 3+ minutes

15:50:55.504 INFO: EXECUTION SUCCESS
15:50:55.504 INFO: ------------------------------------------------------------------------
15:50:55.504 INFO: Total time: 3:24.549s
15:50:55.547 INFO: Final Memory: 25M/2048M

But the result on server doesn't have lots of useful things so it is not a solution for my usecase.

[guwirth]

@Nekto89

• means SQ 9.9, scanner 4.8, Java 17, cxx plugin 2.1.0.428 is slow. Which OS are you using?
• means SQ 8.9, scanner 4.6.2, Java 11, cxx plugin 2.1.0.428 is fast?
  Correct?

[Nekto89]

I've reran on SQ 8.9 again now to recheck. I've also sorted both log files without timestamps to try and see any differences (regex r"\d{2}:\d{2}:\d{2}.\d{3}(.*)" to get rid of timestamps). There is nothing useful related to C++. All data/metrics are completely the same, just in different order.

Java 17.0.7 Azul Systems, Inc. (64-bit)
sonar scanner 4.8.0.2856
C++ (Community) 2.1.0.428 (cxx)
Windows Server 2019 (Version 1809, OS Build 17763.3887)
SONAR_SCANNER_OPTS=-Xmx5g -Xms2g

Community 8.9.5.50698 - INFO: Sensor CXX [cxx] (done) | time=256122ms
Community 9.9.1.69595 - INFO: Sensor CXX [cxx] (done) | time=9602093ms

[guwirth]

Hi @Nekto89,

maybe you can try with latest SQ 9.9.1, they fixed two performance issues:

• https://community.sonarsource.com/t/sonarqube-9-9-1-lts-released/90049
• https://sonarsource.atlassian.net/browse/SONAR-18872?jql=project%20%3D%2010139%20AND%20fixVersion%20%3D%2014051
• https://sonarsource.atlassian.net/browse/SONAR-18472?jql=project%20%3D%2010139%20AND%20fixVersion%20%3D%2014051

Because rest seems to be the same it must be something with new SQ 9.9 LTS. (#2518)

Regards,