Skip to content

[Merge-on-Red] - Implement Test Process Watcher #78742

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ivdiazsa merged 35 commits into from
Mar 10, 2023
Merged

[Merge-on-Red] - Implement Test Process Watcher #78742

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
be9867e
Started the draft for the watch tower.
ivdiazsa Nov 10, 2022
e79ae66
Parsing command-line arguments done.
ivdiazsa Nov 11, 2022
b17eef2
Removed auto-generated file.
ivdiazsa Nov 11, 2022
d40ad9e
Added an example I made on how to run processes from C/C++ :)
ivdiazsa Nov 12, 2022
6cc9e9e
Perfect arg parsing done!
ivdiazsa Nov 16, 2022
fcd9f1a
Seems to be working?
ivdiazsa Nov 16, 2022
1c7f46c
Watchdog is functional now!
ivdiazsa Nov 17, 2022
ec90d1e
Fixed a nuisance with the Windows compiler messing with Linux code, b…
ivdiazsa Nov 22, 2022
b157904
Got the Windows draft working, at long last.
ivdiazsa Nov 23, 2022
36a75f0
Added initial linking of the watcher to the repo's build scripts. Not…
ivdiazsa Dec 6, 2022
c4f2009
Moved the watcher to the CoreCLR project. Not yet functional though...
ivdiazsa Dec 6, 2022
e8b13a5
Got it to build on Windows!
ivdiazsa Dec 7, 2022
b476ad2
Builds and Works on Linux!
ivdiazsa Dec 7, 2022
f74fb2e
Fixed a slowdown with the watcher.
ivdiazsa Dec 8, 2022
958dc29
Forgot to not leak memory :)
ivdiazsa Dec 8, 2022
67e25fe
This is C++, not C.
ivdiazsa Dec 8, 2022
7ef09b2
Bash scripts now call the watcher, but watcher still crashes when bui…
ivdiazsa Dec 14, 2022
226ea36
Renamed variable.
ivdiazsa Dec 14, 2022
5fbd787
FIXED THE LINUX PART!
ivdiazsa Dec 14, 2022
0ff6b8d
Started implementing the constant flow of logs. Had to save my progress.
ivdiazsa Dec 16, 2022
8f21b9c
TEMP LOG BUILDS!
ivdiazsa Dec 17, 2022
c7fc541
TEMP LOGS WORK!
ivdiazsa Dec 17, 2022
4f658e2
Updated with main.
ivdiazsa Feb 15, 2023
a88e48c
Test Watcher seems to be working on Linux and MacOS.
ivdiazsa Feb 22, 2023
8fceb44
Updated the Helix Commands with the watcher now included, and removed…
ivdiazsa Feb 22, 2023
064b9b0
Fixed build issue in the watcher's Windows version.
ivdiazsa Feb 22, 2023
2cb9270
Apparently works on Windows now too?
ivdiazsa Feb 23, 2023
450eba9
Fixed build issue with Windows x86.
ivdiazsa Feb 23, 2023
eba2bcd
Fixed issue where we were not building the watcher in certain subsets…
ivdiazsa Feb 24, 2023
d68d16c
Fixed yet another Windows issue.
ivdiazsa Feb 27, 2023
998965c
Used ExeSuffix instead of recomputing it and excluded Browser-Wasm
ivdiazsa Feb 27, 2023
7cb4e89
Changed the Helix Correlation Payloads to bundle the whole watchdog d…
ivdiazsa Mar 1, 2023
2ba916a
Changed the scripts to use the test watcher in the Core_Root, instead…
ivdiazsa Mar 3, 2023
a712352
Fixed undeclared variable in Windows test scripts.
ivdiazsa Mar 3, 2023
0cc8875
Addressed PR feedback comments.
ivdiazsa Mar 10, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Changed the scripts to use the test watcher in the Core_Root, instead… 
… of the object artifacts.
  • Loading branch information
@ivdiazsa
ivdiazsa committed Mar 3, 2023
commit 2ba916a2218dae300c4c3b5a52642b78352d66c8
31 changes: 12 additions & 19 deletions src/tests/Common/CLRTest.Execute.Bash.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -180,19 +180,6 @@ fi
<Description>Set CORE_ROOT to the specified value before running the test.</Description>
</BashCLRTestExecutionScriptArgument>

<BashCLRTestExecutionScriptArgument Include="watcher">
<HasParam>true</HasParam>
<ParamText>=*</ParamText> <!-- Bash specific -->
<ParamName>watcherFullPath</ParamName>
<Command><![CDATA[ export _WatcherFullPath="${i#*=}"
if [ ! -f "$_WatcherFullPath" ]
then
echo "The Watcher FullPath %5C%22${_WatcherFullPath}%5C%22 does not exist"
usage
fi]]></Command>
<Description>Path to the test watcher that will look over the test.</Description>
</BashCLRTestExecutionScriptArgument>

<BashCLRTestExecutionScriptArgument Include="e;env">
<HasParam>true</HasParam>
<ParamText>=*</ParamText> <!-- Bash specific -->
Expand All @@ -206,6 +193,12 @@ fi
export __DotEnvArg=-e ${__DotEnv}]]></Command>
<Description>A dotenv file to pass to corerun to set environment variables for the test run.</Description>
</BashCLRTestExecutionScriptArgument>

<BashCLRTestExecutionScriptArgument Include="usewatcher">
<HasParam>false</HasParam>
<Command><![CDATA[ _RunWithWatcher=1]]></Command>
<Description>Run the tests using the test watcher.</Description>
</BashCLRTestExecutionScriptArgument>
</ItemGroup>

<PropertyGroup>
Expand Down Expand Up @@ -263,11 +256,11 @@ then
exit 1
fi

# Copy CORECLR native binaries and the test watcher (if provided) to $LinkBin,
# Copy CORECLR native binaries and the test watcher to $LinkBin,
# so that we can run the test based on that directory
cp $CORE_ROOT/*.so $LinkBin/
cp $CORE_ROOT/corerun $LinkBin/
cp $_WatcherFullPath $LinkBin/
cp $CORE_ROOT/watchdog $LinkBin/

# Copy some files that may be arguments
for f in *.txt;
Expand All @@ -277,7 +270,6 @@ then

ExePath=$LinkBin/$(InputAssemblyName)
export CORE_ROOT=$PWD/$LinkBin
export _WatcherFullPath=$PWD/$LinkBin/watchdog
fi
]]>
</BashLinkerTestLaunchCmds>
Expand All @@ -298,7 +290,7 @@ fi
</PropertyGroup>
<PropertyGroup>
<CLRTestRunFile Condition="'$(CLRTestIsHosted)'=='true'">"$CORE_ROOT/corerun" $(CoreRunArgs) ${__DotEnvArg}</CLRTestRunFile>
<WatcherRunFile>"$_WatcherFullPath" 300</WatcherRunFile>
<WatcherRunFile>"$CORE_ROOT/watchdog" 300</WatcherRunFile>

<!-- Note that this overwrites CLRTestBashPreCommands rather than adding to it. -->
<CLRTestBashPreCommands Condition="'$(CLRTestKind)' == 'BuildAndRun' and '$(TargetArchitecture)' == 'wasm'"><![CDATA[
Expand Down Expand Up @@ -334,7 +326,7 @@ fi
if [ ! -z "$CLRCustomTestLauncher" ];
then
LAUNCHER="$CLRCustomTestLauncher $PWD/"
elif [ ! -z "$_WatcherFullPath" ]
elif [ "$_RunWithWatcher" == 1 ];
then
LAUNCHER="$(WatcherRunFile) $(CLRTestRunFile)"
else
Expand Down Expand Up @@ -365,7 +357,7 @@ $(BashLinkerTestLaunchCmds)
if [ ! -z "$CLRCustomTestLauncher" ];
then
LAUNCHER="$CLRCustomTestLauncher $PWD/"
elif [ ! -z "$_WatcherFullPath" ]
elif [ "$_RunWithWatcher" == 1 ];
then
LAUNCHER="$(WatcherRunFile) $(CLRTestRunFile)"
else
Expand Down Expand Up @@ -556,6 +548,7 @@ ReleaseLock()
}
cd "$%28dirname "${BASH_SOURCE[0]}")"
LockFile="lock"
_RunWithWatcher=0


# The __TestEnv variable may be used to specify a script to source before the test.
Expand Down
23 changes: 8 additions & 15 deletions src/tests/Common/CLRTest.Execute.Batch.targets
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,18 +217,12 @@ Exit /b 0
<Description>Set CORE_ROOT to the specified value before running the test.</Description>
</BatchCLRTestExecutionScriptArgument>

<BatchCLRTestExecutionScriptArgument Include="watcher">
<HasParam>true</HasParam>
<ParamName>watcherFullPath</ParamName>
<BatchCLRTestExecutionScriptArgument Include="usewatcher">
<HasParam>false</HasParam>
<Command><![CDATA[
IF EXIST "%2" (
set _WatcherFullPath=%2
) ELSE (
ECHO The Watcher FullPath "%2" does not exist
GOTO :USAGE
)
set /A _RunWithWatcher=1
]]></Command>
<Description>Path to the test watcher that will look over the test.</Description>
<Description>Run the tests using the test watcher.</Description>
</BatchCLRTestExecutionScriptArgument>
</ItemGroup>

Expand Down Expand Up @@ -274,19 +268,18 @@ IF defined DoLink (
Exit /b 1
)

REM Copy CORECLR native binaries and the test watcher (if provided) to %LinkBin%, so that we can run the test based on that directory
REM Copy CORECLR native binaries and the test watcher to %LinkBin%, so that we can run the test based on that directory
copy %CORE_ROOT%\clrjit.dll %LinkBin% > nul 2> nul
copy %CORE_ROOT%\coreclr.dll %LinkBin% > nul 2> nul
copy %CORE_ROOT%\mscorrc.dll %LinkBin% > nul 2> nul
copy %CORE_ROOT%\CoreRun.exe %LinkBin% > nul 2> nul
copy %_WatcherFullPath% %LinkBin% > nul 2> nul
copy %CORE_ROOT%\watchdog.exe %LinkBin% > nul 2> nul

REM Copy some files that may be arguments
copy *.txt %LinkBin% > nul 2> nul

set ExePath=%LinkBin%\$(InputAssemblyName)
set CORE_ROOT=%scriptPath%\%LinkBin%
set _WatcherFullPath=%scriptPath%\%LinkBin%\watchdog.exe
)
]]>
</BatchLinkerTestLaunchCmds>
Expand All @@ -305,7 +298,7 @@ if defined DoLink (
</PropertyGroup>
<PropertyGroup>
<CLRTestRunFile Condition="'$(CLRTestIsHosted)'=='true'">"%CORE_ROOT%\corerun.exe" $(CoreRunArgs) %__DotEnvArg%</CLRTestRunFile>
<WatcherRunFile>"%_WatcherFullPath%" 300</WatcherRunFile>
<WatcherRunFile>"%CORE_ROOT%\watchdog.exe" 300</WatcherRunFile>

<BatchCopyCoreShimLocalCmds Condition="'$(CLRTestScriptLocalCoreShim)' == 'true'"><![CDATA[
REM Local CoreShim requested - see MSBuild property 'CLRTestScriptLocalCoreShim'
Expand All @@ -319,7 +312,7 @@ $(BatchCopyCoreShimLocalCmds)

IF NOT "%CLRCustomTestLauncher%"=="" (
set LAUNCHER=call %CLRCustomTestLauncher% %scriptPath%
) ELSE IF NOT "%_WatcherFullPath%"=="" (
) ELSE IF %_RunWithWatcher% EQU 1 (
set LAUNCHER=$(WatcherRunFile) $(CLRTestRunFile)
) ELSE (
set LAUNCHER=%_DebuggerFullPath% $(CLRTestRunFile)
Expand Down
13 changes: 2 additions & 11 deletions src/tests/Common/helixpublishwitharcade.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -376,9 +376,6 @@

<!-- On Windows, you need to "call" a script or else the cmd.exe instance it is running in will exit, and the HelixPostCommands will not execute. -->
<_MergedWrapperRunScriptPrefix Condition="'$(TestWrapperTargetsWindows)' == 'true'">call </_MergedWrapperRunScriptPrefix>

<_MergedWrapperRunScriptWatcherArgs Condition="'$(TestWrapperTargetsWindows)' != 'true'">-watcher=./watchdog</_MergedWrapperRunScriptWatcherArgs>
<_MergedWrapperRunScriptWatcherArgs Condition="'$(TestWrapperTargetsWindows)' == 'true'">/watcher=.\watchdog.exe</_MergedWrapperRunScriptWatcherArgs>
</PropertyGroup>

<ItemGroup Condition="'@(_MergedWrapperMarker)' != ''" >
Expand Down Expand Up @@ -414,8 +411,8 @@
<HelixCommandLines Condition="'$(TestWrapperTargetsWindows)' != 'true'" Include="export TEST_HARNESS_STRIPE_TO_EXECUTE=.0.1" />
<HelixCommandLines Condition="'$(TestWrapperTargetsWindows)' != 'true'" Include="chmod +x $(_MergedWrapperRunScriptRelative)" />
<!-- Force assemblies to lazy-load for LLVM AOT test runs to enable using tests that fail at AOT time (and as a result can't be AOTd) -->
<HelixCommandLines Condition="'$(RuntimeVariant)' == 'llvmfullaot'" Include="$(_MergedWrapperRunScriptPrefix)$(_MergedWrapperRunScriptRelative) $(_MergedWrapperRunScriptWatcherArgs) --aot-lazy-assembly-load" />
<HelixCommandLines Condition="'$(RuntimeVariant)' != 'llvmfullaot'" Include="$(_MergedWrapperRunScriptPrefix)$(_MergedWrapperRunScriptRelative) $(_MergedWrapperRunScriptWatcherArgs)" />
<HelixCommandLines Condition="'$(RuntimeVariant)' == 'llvmfullaot'" Include="$(_MergedWrapperRunScriptPrefix)$(_MergedWrapperRunScriptRelative) -usewatcher --aot-lazy-assembly-load" />
<HelixCommandLines Condition="'$(RuntimeVariant)' != 'llvmfullaot'" Include="$(_MergedWrapperRunScriptPrefix)$(_MergedWrapperRunScriptRelative) -usewatcher" />
<HelixCommandLines Include="$(XUnitLogCheckerCommand)" />
</ItemGroup>

Expand Down Expand Up @@ -721,17 +718,11 @@
<XUnitRunnerDll>$CORE_ROOT/xunit/xunit.console.dll</XUnitRunnerDll>
</PropertyGroup>

<PropertyGroup>
<TestWatcherDirectory>$(ArtifactsDir)obj/coreclr/$(OSPlatformConfig)/test-watchdog/</TestWatcherDirectory>
<TestWatcherDirectory>$([MSBuild]::NormalizePath($(TestWatcherDirectory)))</TestWatcherDirectory>
</PropertyGroup>

<ItemGroup Condition=" '$(UsesHelixSdk)' == 'true' ">
<HelixCorrelationPayload Include="$(CoreRootDirectory)" />

<!-- Browser-Wasm follows a very different workflow, which is currently out of scope of the Log Checker and Test Watcher. -->
<HelixCorrelationPayload Include="$(XUnitLogCheckerDirectory)" Condition="'$(TargetsBrowser)' != 'true'" />
Copy link
Member

@lewing lewing Mar 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this change the current behavior for wasm logs then?

Copy link
Contributor Author

@ivdiazsa ivdiazsa Mar 10, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not at all. Wasm is out of scope of this project for the time being, hence we are excluding it. So wasm tests will remain unaffected.

<HelixCorrelationPayload Include="$(TestWatcherDirectory)" Condition="'$(TargetsBrowser)' != 'true'" />

<LegacyPayloads Include="$([System.IO.Directory]::GetDirectories($(LegacyPayloadsRootDirectory)))" Condition="Exists('$(LegacyPayloadsRootDirectory)')" />
<LegacyPayloads Update="@(LegacyPayloads)">
Expand Down