Skip to content

Test | Add flaky test quarantine zone #3856

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mdaigle wants to merge 15 commits into
base: main
Choose a base branch
from
Open

Test | Add flaky test quarantine zone #3856

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
fa22414
Skip flaky tests. Try removing platform filters which should already …
mdaigle Dec 19, 2025
dba39e7
Make filter configurable in build.proj
mdaigle Dec 19, 2025
13b684c
Add quarantined testing steps.
mdaigle Dec 19, 2025
8a4f8e0
Fix closing quote.
mdaigle Dec 19, 2025
4d14a1e
Fix closing quote.
mdaigle Dec 19, 2025
ae19c47
Fix quote wrapping for filter argument.
mdaigle Dec 19, 2025
29ecebc
Fix continueOnError indent.
mdaigle Dec 19, 2025
3413a96
Tag flaky tests.
mdaigle Dec 20, 2025
4c4cf1e
Tag xevent test as flaky.
mdaigle Dec 20, 2025
2fe2941
Fix indentation.
mdaigle Jan 5, 2026
c56e1b0
Mark DiagnosticTest as flaky.
mdaigle Jan 5, 2026
bde6d56
Merge branch 'main' of github.com:dotnet/SqlClient into dev/mdaigle/f…
mdaigle Jan 5, 2026
c846ec8
Remove stray preprocessor directive.
mdaigle Jan 5, 2026
a4a1864
Don't collect code coverage for flaky tests.
mdaigle Jan 5, 2026
13661d3
Add preliminary cleanup in case previous runs left files behind.
mdaigle Jan 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 22 additions & 13 deletions build.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,15 @@
<NugetPackProperties Condition="'$(TF_BUILD)' == 'true'">$(NugetPackProperties);ContinuousIntegrationBuild=true;</NugetPackProperties>
</PropertyGroup>

<!-- Test configuration properties -->
<PropertyGroup>
<FilterStatement>$(Filter)</FilterStatement>
<FilterStatement Condition="'$(FilterStatement)' == ''">category!=failing&amp;category!=flaky</FilterStatement>
<!--Collect code coverage unless explicitly disabled-->
<CollectCodeCoverage Condition="'$(CollectCodeCoverage)' == ''">true</CollectCodeCoverage>
<CollectStatement Condition="'$(CollectCodeCoverage)' == 'true'">--collect "Code coverage"</CollectStatement>
</PropertyGroup>

<!-- Release Build properties must be turned on for Release purposes, and turned off for Code Coverage calculations -->
<PropertyGroup>
<BuildForRelease Condition="$(BuildForRelease) == ''">true</BuildForRelease>
Expand Down Expand Up @@ -230,15 +239,15 @@
-p:Configuration=$(Configuration)
-p:Target$(TFGroup)Version=$(TF)
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=failing%26category!=flaky"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Unit-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
</PropertyGroup>
<Message Text=">>> Running unit tests for Windows via command: $(TestCommand)"/>
<Exec ConsoleToMsBuild="true" Command="$(TestCommand)"/>

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22 net8_0_AnyCPU_ManagedSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net462_AnyCPU_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_Named_Instance net9_0_AnyCPU_NativeSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_Named_Instance net8_0_AnyCPU_ManagedSNI_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net8_0_AnyCPU_ManagedSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net9_0_AnyCPU_NativeSNI_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net462_AnyCPU_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_Named_Instance net8_0_AnyCPU_NativeSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net9_0_AnyCPU_ManagedSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Sql22_x86 net462_AnyCPU_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         C:\x86\dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net9_0_AnyCPU_ManagedSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net8_0_AnyCPU_ManagedSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net9_0_AnyCPU_NativeSNI_2)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_ARM64_Azure_Sql net9_0_AnyCPU_NativeSNI_1)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetcoreapp-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net462_AnyCPU_3)

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.

Check failure on line 250 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project

build.proj#L250 

build.proj(250,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\UnitTests\Microsoft.Data.SqlClient.UnitTests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Unit-Windowsnetfx-1,2,3"       " exited with code 1.
</Target>

<!-- Run all unit tests applicable to Unix. -->
Expand All @@ -251,11 +260,11 @@
-p:Configuration=$(Configuration)
-p:TargetNetCoreVersion=$(TF)
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=failing%26category!=flaky"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Unit-Unixnetcoreapp-$(TestSet)"
</TestCommand>
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
</PropertyGroup>
<Message Text=">>> Running unit tests for Unix via command: $(TestCommand)"/>
Expand All @@ -277,9 +286,9 @@
-p:ReferenceType=$(ReferenceType)
-p:TestSet=$(TestSet)
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Functional-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand All @@ -300,9 +309,9 @@
-p:ReferenceType=$(ReferenceType)
-p:TestSet=$(TestSet)
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Functional-Unixnetcoreapp-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand All @@ -326,15 +335,15 @@
-p:ReferenceType=$(ReferenceType)
-p:TestSet=$(TestSet)
-p:TestTargetOS=Windows$(TargetGroup)
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=non$(TargetGroup)tests&amp;category!=failing&amp;category!=nonwindowstests"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Manual-Windows$(TargetGroup)-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
</PropertyGroup>
<Message Text=">>> Running Manual test for Windows via command: $(TestCommand)" />
<Exec ConsoleToMsBuild="true" Command="$(TestCommand)" />

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win22_Sql22_Named_Instance net8_0_AnyCPU_ManagedSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net9_0_AnyCPU_ManagedSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win22_Azure_Sql net9_0_AnyCPU_ManagedSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win22_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net8_0_AnyCPU_ManagedSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win11_Azure_Sql net8_0_AnyCPU_ManagedSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win11_Azure_Sql net9_0_AnyCPU_NativeSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net9.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net8_0_AnyCPU_NativeSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp         --collect "Code coverage"         --results-directory TestResults         --filter "category!=failing&category!=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Project (Win11_Azure_Sql net462_AnyCPU_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetfxVersion=net462         -p:ReferenceType=Project         -p:TestSet=3         -p:TestTargetOS=Windowsnetfx                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetfx-3"       " exited with code 1.

Check failure on line 346 in build.proj

View check run for this annotation

Azure Pipelines / PR-SqlClient-Package (Win11_ARM64_Azure_Sql net8_0_AnyCPU_ManagedSNI_3)

build.proj#L346 

build.proj(346,5): Error MSB3073: The command "         dotnet test "src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj"         --no-build         -v n         -p:Configuration=Debug         -p:TargetnetcoreVersion=net8.0         -p:ReferenceType=Package         -p:TestSet=3         -p:TestTargetOS=Windowsnetcoreapp                  --results-directory TestResults         --filter "category=flaky"         --logger:"trx;LogFilePrefix=Manual-Windowsnetcoreapp-3"       " exited with code 1.
</Target>

<!-- Run all Manual tests applicable to Unix. -->
Expand All @@ -349,9 +358,9 @@
-p:ReferenceType=$(ReferenceType)
-p:TestSet=$(TestSet)
-p:TestTargetOS=Unixnetcoreapp
--collect "Code coverage"
$(CollectStatement)
--results-directory $(ResultsDirectory)
--filter "category!=nonnetcoreapptests&amp;category!=failing&amp;category!=nonlinuxtests&amp;category!=nonuaptests"
--filter "$(FilterStatement)"
--logger:"trx;LogFilePrefix=Manual-Unixnetcoreapp-$(TestSet)"
</TestCommand>
<TestCommand>$(TestCommand.Replace($([System.Environment]::NewLine), " "))</TestCommand>
Expand Down
20 changes: 20 additions & 0 deletions eng/pipelines/common/templates/jobs/ci-code-coverage-job.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,20 @@ jobs:
dotnet tool install --global dotnet-reportgenerator-globaltool
displayName: Install dotnet tools

- pwsh: |
Write-Host "Removing leftover coverage files from previous runs..."
Remove-Item $(netFxDir) -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item $(netCoreDir) -Recurse -Force -ErrorAction SilentlyContinue

Write-Host "Removing leftover merged XML files from previous runs..."
Remove-Item coverageNetFxXml -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coverageNetCoreXml -Recurse -Force -ErrorAction SilentlyContinue

Write-Host "Removing leftover reports from previous runs..."
Remove-Item coveragereportNetFx -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coveragereportNetCore -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coveragereportAddOns -Recurse -Force -ErrorAction SilentlyContinue

- ${{ each targetFramework in parameters.targetFrameworks }}:
- task: DownloadPipelineArtifact@2
displayName: 'Download Coverage Reports [${{ targetFramework }}]'
Expand Down Expand Up @@ -196,3 +210,9 @@ jobs:
./codecov --verbose upload-process --fail-on-error -t $(CODECOV_TOKEN) -f "coveragereportNetCore\Cobertura.xml" -F netcore
./codecov --verbose upload-process --fail-on-error -t $(CODECOV_TOKEN) -f "coveragereportAddOns\Cobertura.xml" -F addons
displayName: Upload to CodeCov

- pwsh: |
Write-Host "Removing reports..."
Remove-Item coveragereportNetFx -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coveragereportNetCore -Recurse -Force -ErrorAction SilentlyContinue
Remove-Item coveragereportAddOns -Recurse -Force -ErrorAction SilentlyContinue
21 changes: 19 additions & 2 deletions eng/pipelines/common/templates/steps/build-and-run-tests-netcore-step.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,12 +73,29 @@ steps:
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky"'

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Functional Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky"'
Comment on lines +76 to +83
Copy link

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The functional tests for .NET Core are missing the code coverage collection flag. The corresponding .NET Framework version (build-and-run-tests-netfx-step.yml line 75) includes --collect "Code Coverage" but this line doesn't. This inconsistency means code coverage won't be collected for .NET Core functional tests, which could impact the completeness of coverage reports.

Suggested change
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky"'
- task: DotNetCoreCLI@2
displayName: 'Run Flaky Functional Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'
- task: DotNetCoreCLI@2
displayName: 'Run Flaky Functional Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'

Copilot uses AI. Check for mistakes.
continueOnError: true

- task: DotNetCoreCLI@2
displayName: 'Run Manual Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter category!=nonnetcoreapptests&category!=failing&category!=nonwindowstests --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Manual Tests for ${{parameters.TargetNetCoreVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
Copy link

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flaky tests are configured to collect code coverage (--collect "Code Coverage"), but this may not align with the intention to disable code coverage collection for flaky tests as specified in build.proj and run-all-tests-step.yml where -p:CollectCodeCoverage=false is set. This inconsistency could lead to unnecessary coverage data being collected for tests that are intentionally quarantined and may skew coverage metrics.

Suggested change
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetCoreVersion=${{parameters.TargetNetCoreVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky"'

Copilot uses AI. Check for mistakes.
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}
continueOnError: true
21 changes: 19 additions & 2 deletions eng/pipelines/common/templates/steps/build-and-run-tests-netfx-step.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,12 +72,29 @@ steps:
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Functional Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\FunctionalTests\Microsoft.Data.SqlClient.FunctionalTests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
Copy link

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flaky tests are configured to collect code coverage (--collect "Code Coverage"), but this may not align with the intention to disable code coverage collection for flaky tests as specified in build.proj and run-all-tests-step.yml where -p:CollectCodeCoverage=false is set. This inconsistency could lead to unnecessary coverage data being collected for tests that are intentionally quarantined and may skew coverage metrics.

Suggested change
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} -p:CollectCodeCoverage=false --no-build -v n --filter "category=flaky"'

Copilot uses AI. Check for mistakes.
continueOnError: true

- task: DotNetCoreCLI@2
displayName: 'Run Manual Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=nonnetfxtests&category!=failing&category!=nonwindowstests" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category!=failing&category!=flaky" --collect "Code Coverage"'
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}

- task: DotNetCoreCLI@2
displayName: 'Run Flaky Manual Tests for ${{parameters.TargetNetFxVersion }}'
inputs:
command: test
projects: 'src\Microsoft.Data.SqlClient\tests\ManualTests\Microsoft.Data.SqlClient.ManualTesting.Tests.csproj'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
Copy link

Copilot AI Jan 6, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flaky tests are configured to collect code coverage (--collect "Code Coverage"), but this may not align with the intention to disable code coverage collection for flaky tests as specified in build.proj and run-all-tests-step.yml where -p:CollectCodeCoverage=false is set. This inconsistency could lead to unnecessary coverage data being collected for tests that are intentionally quarantined and may skew coverage metrics.

Suggested change
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky" --collect "Code Coverage"'
arguments: '-p:Platform=${{parameters.platform }} -p:TestTargetOS="${{parameters.TestTargetOS }}" -p:TargetNetFxVersion=${{parameters.TargetNetFxVersion }} -p:ReferenceType=${{parameters.referenceType }} -p:Configuration=${{parameters.buildConfiguration }} -p:TestMicrosoftDataSqlClientVersion=${{parameters.NugetPackageVersion }} --no-build -v n --filter "category=flaky"'

Copilot uses AI. Check for mistakes.
retryCountOnTaskFailure: ${{parameters.retryCountOnManualTests }}
continueOnError: true
Loading
Loading