Locked mode not working as hash is calculated incorrectly for previously cached packages

[alefranz]

Details about Problem

I've been struggling with the new "locked mode" of dotnet restore since release and I've finally been able to create repro steps.

The problems seems related to using different version of the .NET Core SDK on the same machine, in particular versions before the locked mode introduction alongside the new ones.
As the NuGet Package Cache is shared between all versions, if the package as been cached by an old version of dotnet.exe/nuget, it looks like the new .nupkg.metadata get created with an incorrect signature.

Example:
Newtonsoft.Json 12.0.1 has signature:

{
  "version": 1,
  "contentHash": "jmVyoEyk0In8r+AObYQyFKVFm7uSRzE0XSHSbEtBJcZDMV6DqJoyB4FLcHwprPVhAh826so0db3DIKXVnpGoPA=="
}

however if it was already in the Nuget Package Cache, when using the new dotnet.exe/nuget it get this signature

{
  "version": 1,
  "contentHash": "pBR3wCgYWZGiaZDYP+HHYnalVnPJlpP1q55qvVb+adrDHmFMDc1NAKio61xTwftK3Pw5h7TZJPJEEVMd6ty8rg=="
}

which obviously cause the locked mode feature to fail.

This behaviour seems consistent in different Windows versions and machines.
Not tested on Linux/MaxOs.

I believe it is a common scenario to use different versions of the SDK on a dev machine as well on a build agent (not everyone use disposable docker agents yet) e.g. via global.json in different projects

Happy to help investigate more but I will need someone to point me in the right direction.

Thank you

Product Versions

NuGet product used: dotnet.exe

Version pre locked mode used:
❯ dotnet --version
2.1.403
❯ dotnet nuget --version
NuGet Command Line
4.8.1.0

Version with locked mode used:
❯ dotnet --version
2.2.102
❯ dotnet nuget --version
NuGet Command Line
4.9.2.0

OS version: Win10 17763

Worked before? No

Detailed repro steps so we can see the same problem

1. git clone https://github.com/alefranz/DotnetRestoreLockedMode
2. Delete package from Nuget package cache in %userprofile%.nuget\packages\newtonsoft.json\12.0.1
3. cd DotnetRestoreLockedMode\NetSdk22
4. dotnet restore --force --force-evaluate
5. dotnet restore --force --locked-mode pass!
6. Delete package from Nuget package cache in %userprofile%.nuget\packages\newtonsoft.json\12.0.1
7. cd ..\NetSdk21
8. dotnet restore --force
9. cd ..\NetSdk22
10. dotnet restore --force --locked-mode fail! :(

Sample Project

https://github.com/alefranz/DotnetRestoreLockedMode

Note the global.json is at the project level for simplicity, but it doesn't make any difference having different solutions.

[rrelyea]

/cc: @anangaur

[rrelyea]

@jainaashish - I think we meant to put this in 4.9.3, not 4.9.x - when we discussed yesterday, right?

[jainaashish]

we discussed #7667 and #7646 to be in 4.9.3. We didn't discuss this one but from the description it seems the issue is packages cache having wrong package in there which gets wrong hash in lock file. And the current solution for this is to clean packages cache or that specific package so I'm afraid we can address this for 4.9.3 or even 4.9.x

[anangaur]

issue is packages cache having wrong package in there which gets wrong hash in lock file. A

@jainaashish I dont think the package is wrong. I see different SHA being generated when the package is downloaded from nuget.org vs. when the package is already there but .nupkg.metadata file is not there and is generated in-situ. For Newtonsoft.json 12.0.1, when downloaded from internet (nuget.org), the SHA is jmVyoEyk0In8r+AObYQyFKVFm7uSRzE0XSHSbEtBJcZDMV6DqJoyB4FLcHwprPVhAh826so0db3DIKXVnpGoPA==

but when generated, it is: pBR3wCgYWZGiaZDYP+HHYnalVnPJlpP1q55qvVb+adrDHmFMDc1NAKio61xTwftK3Pw5h7TZJPJEEVMd6ty8rg==

[jainaashish]

this is weird, I'll look into this. Does this also happen with other packages as well or other version of Newtonsoft.json?

[anangaur]

Yes. Same with all packages. When a package is freshly downloaded, the contentHash in .nuspec.metadata file matches with the hash stored in .nupkg.SHA512 file. Now if you delete the .nuspec.metadata file and restore again, the newly generated .nuspec.metadata file has a different hash than what was before. Seems wrong to me.

[jainaashish]

this is happening with dotnet restore... so assuming something is fishy with xplat path. Works fine inside VS.

[anangaur]

I guess we should test out all the paths including NuGet.exe as well. The priority of fixing this issue is high IMO.

[alefranz]

yes, confirming this happens with dotnet restore and for all packages.
@anangaur thanks for appreciating the importance of this issue.
@jainaashish if you can point me in the right direction I'll be happy to try to have a look over the weekend