Skip to content

Conversation

@jviau
Copy link
Contributor

@jviau jviau commented Jul 13, 2023

Issue describing the changes in this PR

resolves #1740

This PR attempts to solve frequent pain points with our SDK build steps which cause file locked issues. I believe the issue lies with how we perform an inner-build with the WorkerExtensions.csproj to generate collect host extensions. To mitigate this inner-build causing file lock issues, two primary changes have been made:

  1. The WorkerExtension.csproj is now generated & built in the intermediate output path of the outer-project (typically obj/{Configuration}/{TargetFramework}) under the WorkerExtensions folder.
    • Importing of Directory.Build.props and Directory.Build.targets is disabled for the inner-build, to avoid unintentional consequences of moving to intermediate output path.
    • The goal of moving here is that using the temp directory can be problematic as it is not part of the source root on a CI machine. This means anti-virus may be scanning it aggressively, it may be on a different drive altogether, or many other factors may impact the behavior and performance of using the temp directory. Switching to intermediate directory removes these concerns and unknowns outright. It also has the added benefit of we can now easily re-use the generated csproj between builds.
    • If customers have an issue with this, the MSBuild property that controls this path has been restructured in a way that customers can override this path if they want. Including reverting back to the tmp directory.
  2. Incremental build support has been improved
    • Copy tasks now use SkipUnchanged to avoid copying files which already exist and are up to date at the target destination
    • Generation of the csproj itself now checks for an existing csproj, and only writes if there are differences.

These changes combined reduces how much happens on re-builds of a functions worker project, which as a result means less file interactions and thus less file locked issues.

I believe there is still one file-locked issue we will be unable to resolve without help from the NuGet team. Our inner-build restore is not part of the outer-build restore. Instead, it happens as part of the AfterBuild stage of the outer-build. I have seen instances of files in the NuGet package cache being locked at this point in time, causing build issues. Unfortunately, this is out of our control and can only be addressed by participating in the outer-build restore.

Pull request checklist

TODO

  • My changes do not require documentation changes
    • Otherwise: Documentation issue linked to PR
  • My changes should not be added to the release notes for the next release
    • Otherwise: I've added my notes to release_notes.md
  • My changes do not need to be backported to a previous version
    • Otherwise: Backport tracked by issue/PR #issue_or_pr
  • I have added all required tests (Unit tests, E2E tests)

@jviau jviau requested a review from fabiocav July 13, 2023 21:06
@jviau jviau changed the base branch from main to feature/ext-proj July 18, 2023 21:50
@jviau
Copy link
Contributor Author

jviau commented Aug 15, 2023

/azp run

@azure-pipelines
Copy link

You have several pipelines (over 10) configured to build pull requests in this repository. Specify which pipelines you would like to run by using /azp run [pipelines] command. You can specify multiple pipelines using a comma separated list.

@jviau jviau merged commit f672526 into Azure:feature/ext-proj Aug 16, 2023
@jviau jviau deleted the ext-proj branch August 16, 2023 18:16
jviau added a commit that referenced this pull request Jan 18, 2024
…d nuget feed issues (#1946)

* Worker Extension project incremental build support (#1749)

* Merge main to feature/ext-proj (#1832)

* Refactor SDK targets to address multiple writes and flaky builds (#1861)

* Ensure ExtensionsCsProjDirectory is a full path (#1938)

* Pass RestoreSources to inner-build restore task (#1937)

* Rev SDK version to 1.16.0-preview1 (#1941)

* Skip func targets during design time build (#1954)

* Correct merge mistakes

* Update sln

* Fix properties

* update worker.config.json generation (#2122)

* [SDK] Remove properties from inner-build (#2161)

* Expand on RemoveProperties for inner build. Fix source gen default

* Add PublishProfile to remove props

* Un-remove some properties from inner build
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SDK: Incremental build support, build in intermediate dir

2 participants