From 4ab1dbfc20a526551d9d40438a78e244e9119af4 Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Thu, 14 Nov 2019 09:50:11 -0800 Subject: [PATCH 01/12] Add files via upload --- 1-Draft/RFCxxxx-Secrets-Management.md | 284 ++++++++++++++++++++++++++ 1 file changed, 284 insertions(+) create mode 100644 1-Draft/RFCxxxx-Secrets-Management.md diff --git a/1-Draft/RFCxxxx-Secrets-Management.md b/1-Draft/RFCxxxx-Secrets-Management.md new file mode 100644 index 000000000..1fa0055dd --- /dev/null +++ b/1-Draft/RFCxxxx-Secrets-Management.md @@ -0,0 +1,284 @@ +--- +RFC: RFCxxxx +Author: Sydney +Status: Draft +SupercededBy: N/A +Version: 0.1 +Area: Security +Comments Due: July 31st, 2019 +Plan to implement: Yes, PS7 +--- + +# Simplify and secure use of secrets in scripts + +Advanced scripts that touch many systems require multiple secrets and +types of secrets particularly when orchestrating across different clouds. +Best practice is to not hard code any secrets in scripts, but currently +this requires custom code on different platforms to handle this securely. + +The secrets management module provides commands for registering vault extensions, and accessing vault secrets. This greatly reduces the temptation to hard code secrets directly into production source code, and instead use the Secrets Management module to dynamically retrieve secrets at runtime. + +## Motivation + +> As a PowerShell script author, +> I can securely use multiple secrets, +> so that I can automate complex orchestration across multiple remote resources. + +## High Level Design + +This is a new independent module called `Microsoft.PowerShell.SecretsManagement`. +Secrets are stored securely in a local vault. +The local vault is expected to only allow access to the user who owns that +vault. +Secrets required to access remote vaults are stored in the local vault and used by the secrets management +cmdlets to retrieve remote secrets. + +`User Context` --> `Local Vault` --> `SecretsVault` --> `Remote Vault` + +## User Experience + +Registering and using remote secrets: + +```powershell +# Install the AzKeyVault Extension module to be used in this example. +Install-Module AKVaultExtension + +# In this example, we explicitly register this extension +Register-SecretsVault -Name Azure -ModuleName AKVaultExtension -VaultParamters "_SPT_Parameters_AzKeyVault_" + +# Once the vault is registered, and populated with secrets, the secret can be retrieved using Get-Secret +$azDevOpsToken = Get-Secret -Name AzDevOps +Invoke-RestMethod https://dev.azure.com/… -Credential $azDevOpsToken -Authentication Basic +``` + +Registering and using local secret: + +```powershell +# For local vault, and vaults that implement the Add-Secret cmdlet we can register custom secrets +# In this example, we store a PSCredential object +Add-Secret -Secret $cred -Name MyCreds +New-PSSession -ComputerName myServer -Credential (Get-Secret -Name MyCreds) +``` + +## Specification + +### Secrets Vault + +There are two types of vaults: local and remote. +A local vault, by default, is already created and named `Default`. +`Default` on Windows is [CredMan](https://docs.microsoft.com/en-us/windows/desktop/SecAuthN/credentials-management). +`Default` on macOS is [KeyChain](https://developer.apple.com/documentation/security/keychain_services). + +>[!NOTE] +>KeyChain support is unlikely to be available in the first release of this feature. + +`Default` on Linux is [Gnome Keyring](https://wiki.gnome.org/Projects/GnomeKeyring/). + +>[!NOTE] +>Linux has many options for local credential management. The choice of Gnome Keyring +>is that it's a simple local vault we can easily test against and validate that our +>design works with a well-known Linux local vault. Since the extension model is +>open, we expect additional vault support to come from owners of those vaults or +>the community. + +### Credential Vault Extensions + +Vault extensions are PowerShell modules that provide implementations of four required functions: + +- SetSecret - Adds a secret to the vault + +- GetSecret - Retrieves a secret from the vault + +- RemoveSecret - Removes a secret from the vault + +- GetSecretInfo - Returns information about one or more secrets (but not the secret itself) + +The extension module can expose the above functions either through a C# class implementing an abstract type, or by publicly exporting script cmdlet functions. +Each function implementation takes a set of parameter arguments needed for secret manipulation and error reporting. +In addition, each function implementation takes an optional parameter argument that is a dictionary of additional parameters, specific to that implementation. +When registering the vault extension, the additional parameters are stored securely in the built-in local vault as a hashtable. +At runtime the additional parameters are read from the built-in local vault and passed to the vault implementing functions. + +### C# class implementation + +The PowerShell module must include a 'RequiredAssemblies' entry in the module manifest which provides the name of the binary that implements the abstract type. + +```powershell +@{ + ModuleVersion = '1.0' + RequiredAssemblies = @('AKVaultExtension') +} +``` + +```C# +// AKVaultExtension implements these abstract methods +public abstract bool SetSecret( + string name, + object secret, + IReadOnlyDictionary parameters, + out Exception error); + +public abstract object GetSecret( + string name, + IReadOnlyDictionary parameters, + out Exception error); + +public abstract bool RemoveSecret( + string name, + IReadOnlyDictionary parameters, + out Exception error); + +public abstract KeyValuePair[] GetSecretInfo( + string filter, + IReadOnlyDictionary parameters, + out Exception error); +``` + +When PowerShell loads the module, the required assembly will be loaded and the implementing type becomes available. + +### PowerShell script implementation + +For a script implementation, the PowerShell module must include a subdirectory named 'ImplementingModule' in the same directory containing the module manifest file. +The ImplementingModule subdirectory must contain PowerShell script module files named 'ImplementingModule' that implements the required script functions. + +ImplementingModule.psd1 + +```powershell +@{ + ModuleVersion = '1.0' + RootModule = '.\ImplementingModule.psm1' + FunctionsToExport = @('Set-Secret','Get-Secret','Remove-Secret','Get-SecretInfo') +} +``` + +ImplementingModule.psm1 + +```powershell +function Set-Secret +{ + param ( + [string] $Name, + [object] $Secret, + [hashtable] $AdditionalParameters + ) +} + +function Get-Secret +{ + param ( + [string] $Name, + [hashtable] $AdditionalParameters + ) +} + +function Remove-Secret +{ + param ( + [string] $Name, + [hashtable] $AdditionalParameters + ) +} + +function Get-SecretInfo +{ + param ( + [string] $Filter, + [hashtable] $AdditionalParameters + ) +} +``` + +## Vault registration cmdlets + +The following cmdlets are provided for vault extension registration. + +```powershell +Register-SecretsVault +Get-SecretsVault +Unregister-SecretsVault +``` + +`Register-SecretsVault` registers a PowerShell module as an extension vault for the current user context. +Validation is performed to ensure the module either provides the required binary with implementing type or the required script commands. +If a dictionary of additional parameters is specified then it will be stored securely in the built-in local vault, assuming that some parameter arguments likely contain secrets. + +`Get-SecretsVault` returns a list of extension vaults currently registered in the user context. + +`Unregister-SecretsVault` un-registers an extension vault. + +`Get-SecretsVault` will enumerate registered extensions returning +the module name and cmdlet used (if appropriate) and the friendly name. + +A `SecretsVaultInfo` object will contain properties for all supported +parameters by registration. + +```output +Name Module Cmdlet ScriptBlock +---- ------ ------ ----------- +AzKeyVault AzKeyVault Get-AzKeyVaultSecret +myVault param($Name) +``` + +### Storing Secrets + +The `Add-Secret` cmdlet is used to store a secret. +The `-Name` must be unique within a vault. +The `-Vault` parameter defaults to the local vault. +A `-NoClobber` parameter will cause this cmdlet to fail if the secret already exists. +A `-Secret` parameter accepts one of the supported types outlined below. + +### Retrieving Secrets + +The `Get-Secret` cmdlet is used to retrieve secrets as the same type as they +were originally added. +The `-Name` parameter retrieves the secret associated with that name. +The `-Vault` parameter defaults to the local vault. +The returned object will be the original stored secret type. + +`Get-Secret` returns a single secret and will result in an error if passed an empty string. + +### Removing Secrets + +The `Remove-Secret` cmdlet iremoves a secret by name from a given vault. + +### Authorization + +Access to the credential vault is always using the current process security context. +In the case of remote vaults, the remote credential is stored within the local +default vault and retrieved as needed when accessing secrets from that remote +vault. + +### Secrets Supported + +Secret objects supported by this module are currently limited to: + +- byte[] - Blob secret + +- string - String secret + +- SecureString - Secure string secret + +- PSCredential - PowerShell credential secret + +- Hashtable - Hash table of name value pairs, where values are restricted to the above secret types. + +## Alternate Proposals and Considerations + +In this release, the following are non-goals that can be addressed in the future: + +- Provision to rotate certs/access tokens +- Sharing local vaults across different computer systems +- Sharing local vault across different user contexts +- A PSProvider for a Secrets: PSDrive +- C# interface for extensions +- C# API to retrieve secrets for C# based modules +- Delegation support +- Local vault requiring to be unlocked automatically +- Ubiquitous `-Secret` parameter taking a hashtable to automatically populate + a cmdlet's parameter taking a secret from the vault: + +```powershell +Invoke-WebRequest -Secret @{Credential="GitHubCred"} +# this retrieves the secret called GitHubCred and passes it to the `-Credential` +# parameter of Invoke-WebRequest +``` From bb8ae2de4d29fa0b09fd8fd57a5fa65752b58365 Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Thu, 14 Nov 2019 11:57:13 -0800 Subject: [PATCH 02/12] Update RFCxxxx-Secrets-Management.md --- 1-Draft/RFCxxxx-Secrets-Management.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/1-Draft/RFCxxxx-Secrets-Management.md b/1-Draft/RFCxxxx-Secrets-Management.md index 1fa0055dd..e7d879b7a 100644 --- a/1-Draft/RFCxxxx-Secrets-Management.md +++ b/1-Draft/RFCxxxx-Secrets-Management.md @@ -226,6 +226,8 @@ The `-Name` must be unique within a vault. The `-Vault` parameter defaults to the local vault. A `-NoClobber` parameter will cause this cmdlet to fail if the secret already exists. A `-Secret` parameter accepts one of the supported types outlined below. +When a plain text secret is added it will be stored as a secure string. +For all other secret types the secret will be stored as object type of the secret specified. ### Retrieving Secrets @@ -233,7 +235,9 @@ The `Get-Secret` cmdlet is used to retrieve secrets as the same type as they were originally added. The `-Name` parameter retrieves the secret associated with that name. The `-Vault` parameter defaults to the local vault. -The returned object will be the original stored secret type. +The `-AsPlainText` switch will return a string secret as plain text. +By default a string secret will be returned as a secure string. +For all other secret types the returned object will be the original stored secret type. `Get-Secret` returns a single secret and will result in an error if passed an empty string. From 4566cf9d873dcdc2c21db58eea624778ad8669aa Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Thu, 14 Nov 2019 12:00:35 -0800 Subject: [PATCH 03/12] Update RFCxxxx-Secrets-Management.md --- 1-Draft/RFCxxxx-Secrets-Management.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/1-Draft/RFCxxxx-Secrets-Management.md b/1-Draft/RFCxxxx-Secrets-Management.md index e7d879b7a..048d98e83 100644 --- a/1-Draft/RFCxxxx-Secrets-Management.md +++ b/1-Draft/RFCxxxx-Secrets-Management.md @@ -226,7 +226,7 @@ The `-Name` must be unique within a vault. The `-Vault` parameter defaults to the local vault. A `-NoClobber` parameter will cause this cmdlet to fail if the secret already exists. A `-Secret` parameter accepts one of the supported types outlined below. -When a plain text secret is added it will be stored as a secure string. +When a string secret is added it will be stored as a secure string. For all other secret types the secret will be stored as object type of the secret specified. ### Retrieving Secrets From 3089647f887458fc98180d1d6b6b1118c36796fb Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Fri, 22 Nov 2019 12:25:07 -0800 Subject: [PATCH 04/12] Delete RFCxxxx-Secrets-Management.md --- 1-Draft/RFCxxxx-Secrets-Management.md | 288 -------------------------- 1 file changed, 288 deletions(-) delete mode 100644 1-Draft/RFCxxxx-Secrets-Management.md diff --git a/1-Draft/RFCxxxx-Secrets-Management.md b/1-Draft/RFCxxxx-Secrets-Management.md deleted file mode 100644 index 048d98e83..000000000 --- a/1-Draft/RFCxxxx-Secrets-Management.md +++ /dev/null @@ -1,288 +0,0 @@ ---- -RFC: RFCxxxx -Author: Sydney -Status: Draft -SupercededBy: N/A -Version: 0.1 -Area: Security -Comments Due: July 31st, 2019 -Plan to implement: Yes, PS7 ---- - -# Simplify and secure use of secrets in scripts - -Advanced scripts that touch many systems require multiple secrets and -types of secrets particularly when orchestrating across different clouds. -Best practice is to not hard code any secrets in scripts, but currently -this requires custom code on different platforms to handle this securely. - -The secrets management module provides commands for registering vault extensions, and accessing vault secrets. This greatly reduces the temptation to hard code secrets directly into production source code, and instead use the Secrets Management module to dynamically retrieve secrets at runtime. - -## Motivation - -> As a PowerShell script author, -> I can securely use multiple secrets, -> so that I can automate complex orchestration across multiple remote resources. - -## High Level Design - -This is a new independent module called `Microsoft.PowerShell.SecretsManagement`. -Secrets are stored securely in a local vault. -The local vault is expected to only allow access to the user who owns that -vault. -Secrets required to access remote vaults are stored in the local vault and used by the secrets management -cmdlets to retrieve remote secrets. - -`User Context` --> `Local Vault` --> `SecretsVault` --> `Remote Vault` - -## User Experience - -Registering and using remote secrets: - -```powershell -# Install the AzKeyVault Extension module to be used in this example. -Install-Module AKVaultExtension - -# In this example, we explicitly register this extension -Register-SecretsVault -Name Azure -ModuleName AKVaultExtension -VaultParamters "_SPT_Parameters_AzKeyVault_" - -# Once the vault is registered, and populated with secrets, the secret can be retrieved using Get-Secret -$azDevOpsToken = Get-Secret -Name AzDevOps -Invoke-RestMethod https://dev.azure.com/… -Credential $azDevOpsToken -Authentication Basic -``` - -Registering and using local secret: - -```powershell -# For local vault, and vaults that implement the Add-Secret cmdlet we can register custom secrets -# In this example, we store a PSCredential object -Add-Secret -Secret $cred -Name MyCreds -New-PSSession -ComputerName myServer -Credential (Get-Secret -Name MyCreds) -``` - -## Specification - -### Secrets Vault - -There are two types of vaults: local and remote. -A local vault, by default, is already created and named `Default`. -`Default` on Windows is [CredMan](https://docs.microsoft.com/en-us/windows/desktop/SecAuthN/credentials-management). -`Default` on macOS is [KeyChain](https://developer.apple.com/documentation/security/keychain_services). - ->[!NOTE] ->KeyChain support is unlikely to be available in the first release of this feature. - -`Default` on Linux is [Gnome Keyring](https://wiki.gnome.org/Projects/GnomeKeyring/). - ->[!NOTE] ->Linux has many options for local credential management. The choice of Gnome Keyring ->is that it's a simple local vault we can easily test against and validate that our ->design works with a well-known Linux local vault. Since the extension model is ->open, we expect additional vault support to come from owners of those vaults or ->the community. - -### Credential Vault Extensions - -Vault extensions are PowerShell modules that provide implementations of four required functions: - -- SetSecret - Adds a secret to the vault - -- GetSecret - Retrieves a secret from the vault - -- RemoveSecret - Removes a secret from the vault - -- GetSecretInfo - Returns information about one or more secrets (but not the secret itself) - -The extension module can expose the above functions either through a C# class implementing an abstract type, or by publicly exporting script cmdlet functions. -Each function implementation takes a set of parameter arguments needed for secret manipulation and error reporting. -In addition, each function implementation takes an optional parameter argument that is a dictionary of additional parameters, specific to that implementation. -When registering the vault extension, the additional parameters are stored securely in the built-in local vault as a hashtable. -At runtime the additional parameters are read from the built-in local vault and passed to the vault implementing functions. - -### C# class implementation - -The PowerShell module must include a 'RequiredAssemblies' entry in the module manifest which provides the name of the binary that implements the abstract type. - -```powershell -@{ - ModuleVersion = '1.0' - RequiredAssemblies = @('AKVaultExtension') -} -``` - -```C# -// AKVaultExtension implements these abstract methods -public abstract bool SetSecret( - string name, - object secret, - IReadOnlyDictionary parameters, - out Exception error); - -public abstract object GetSecret( - string name, - IReadOnlyDictionary parameters, - out Exception error); - -public abstract bool RemoveSecret( - string name, - IReadOnlyDictionary parameters, - out Exception error); - -public abstract KeyValuePair[] GetSecretInfo( - string filter, - IReadOnlyDictionary parameters, - out Exception error); -``` - -When PowerShell loads the module, the required assembly will be loaded and the implementing type becomes available. - -### PowerShell script implementation - -For a script implementation, the PowerShell module must include a subdirectory named 'ImplementingModule' in the same directory containing the module manifest file. -The ImplementingModule subdirectory must contain PowerShell script module files named 'ImplementingModule' that implements the required script functions. - -ImplementingModule.psd1 - -```powershell -@{ - ModuleVersion = '1.0' - RootModule = '.\ImplementingModule.psm1' - FunctionsToExport = @('Set-Secret','Get-Secret','Remove-Secret','Get-SecretInfo') -} -``` - -ImplementingModule.psm1 - -```powershell -function Set-Secret -{ - param ( - [string] $Name, - [object] $Secret, - [hashtable] $AdditionalParameters - ) -} - -function Get-Secret -{ - param ( - [string] $Name, - [hashtable] $AdditionalParameters - ) -} - -function Remove-Secret -{ - param ( - [string] $Name, - [hashtable] $AdditionalParameters - ) -} - -function Get-SecretInfo -{ - param ( - [string] $Filter, - [hashtable] $AdditionalParameters - ) -} -``` - -## Vault registration cmdlets - -The following cmdlets are provided for vault extension registration. - -```powershell -Register-SecretsVault -Get-SecretsVault -Unregister-SecretsVault -``` - -`Register-SecretsVault` registers a PowerShell module as an extension vault for the current user context. -Validation is performed to ensure the module either provides the required binary with implementing type or the required script commands. -If a dictionary of additional parameters is specified then it will be stored securely in the built-in local vault, assuming that some parameter arguments likely contain secrets. - -`Get-SecretsVault` returns a list of extension vaults currently registered in the user context. - -`Unregister-SecretsVault` un-registers an extension vault. - -`Get-SecretsVault` will enumerate registered extensions returning -the module name and cmdlet used (if appropriate) and the friendly name. - -A `SecretsVaultInfo` object will contain properties for all supported -parameters by registration. - -```output -Name Module Cmdlet ScriptBlock ----- ------ ------ ----------- -AzKeyVault AzKeyVault Get-AzKeyVaultSecret -myVault param($Name) -``` - -### Storing Secrets - -The `Add-Secret` cmdlet is used to store a secret. -The `-Name` must be unique within a vault. -The `-Vault` parameter defaults to the local vault. -A `-NoClobber` parameter will cause this cmdlet to fail if the secret already exists. -A `-Secret` parameter accepts one of the supported types outlined below. -When a string secret is added it will be stored as a secure string. -For all other secret types the secret will be stored as object type of the secret specified. - -### Retrieving Secrets - -The `Get-Secret` cmdlet is used to retrieve secrets as the same type as they -were originally added. -The `-Name` parameter retrieves the secret associated with that name. -The `-Vault` parameter defaults to the local vault. -The `-AsPlainText` switch will return a string secret as plain text. -By default a string secret will be returned as a secure string. -For all other secret types the returned object will be the original stored secret type. - -`Get-Secret` returns a single secret and will result in an error if passed an empty string. - -### Removing Secrets - -The `Remove-Secret` cmdlet iremoves a secret by name from a given vault. - -### Authorization - -Access to the credential vault is always using the current process security context. -In the case of remote vaults, the remote credential is stored within the local -default vault and retrieved as needed when accessing secrets from that remote -vault. - -### Secrets Supported - -Secret objects supported by this module are currently limited to: - -- byte[] - Blob secret - -- string - String secret - -- SecureString - Secure string secret - -- PSCredential - PowerShell credential secret - -- Hashtable - Hash table of name value pairs, where values are restricted to the above secret types. - -## Alternate Proposals and Considerations - -In this release, the following are non-goals that can be addressed in the future: - -- Provision to rotate certs/access tokens -- Sharing local vaults across different computer systems -- Sharing local vault across different user contexts -- A PSProvider for a Secrets: PSDrive -- C# interface for extensions -- C# API to retrieve secrets for C# based modules -- Delegation support -- Local vault requiring to be unlocked automatically -- Ubiquitous `-Secret` parameter taking a hashtable to automatically populate - a cmdlet's parameter taking a secret from the vault: - -```powershell -Invoke-WebRequest -Secret @{Credential="GitHubCred"} -# this retrieves the secret called GitHubCred and passes it to the `-Credential` -# parameter of Invoke-WebRequest -``` From cd2b6ef18418a3c3c647efca95f8be9c9c8ff701 Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Fri, 22 Nov 2019 13:57:02 -0800 Subject: [PATCH 05/12] Add files via upload --- 1-Draft/RFCxxxx-PowerShellGet-3.0.md | 446 +++++++++++++++++++++++++++ 1 file changed, 446 insertions(+) create mode 100644 1-Draft/RFCxxxx-PowerShellGet-3.0.md diff --git a/1-Draft/RFCxxxx-PowerShellGet-3.0.md b/1-Draft/RFCxxxx-PowerShellGet-3.0.md new file mode 100644 index 000000000..6dee3aeed --- /dev/null +++ b/1-Draft/RFCxxxx-PowerShellGet-3.0.md @@ -0,0 +1,446 @@ +--- +RFC: RFCxxxx +Author: Sydney Smith +Status: Draft +SupercededBy: N/A +Version: 0.3 +Area: PowerShellGet +Comments Due: 6/30 +Plan to implement: Yes, PS7 +--- + +# PowerShellGet 3.0 Module + +PowerShellGet has enabled an active community to publish and install PowerShell +Modules. +However, it has some architectural and design limitations that prevent it from +moving forward more quickly. +This RFC proposes some significant changes to address this. + +> [!NOTE] +> There are no intentions to break existing PSGet v2 users so existing +> users can stay on PSGet v2 and that will continue to work. +## Motivation + + As a PowerShell user, + I can discover how to install missing cmdlets, + so that I can be more productive. + + As a PowerShell user, + I can easily install modules without specifying lots of switches, + so that I can be more productive. + + As a PowerShellGet contributor, + I can more easily contribute code, + so that PowerShellGet evolves more quickly. + +## User Experience + +```powershell +PS> Get-Satisfaction +Get-Satisfaction : The term 'Get-Satisfaction' is not recognized as the name of a cmdlet, function, script file, or operable program. +Check the spelling of the name, or if a path was included, verify that the path is correct and try again. +At line:1 char:1 ++ Get-Satisfaction ++ ~~~ ++ CategoryInfo : ObjectNotFound: (Get-Satisfaction:String) [], CommandNotFoundException ++ FullyQualifiedErrorId : CommandNotFoundException +Suggestion: You can obtain `Get-Satisfaction` by running `Install-PSResource HappyDays`. +PS> Install-PSResource HappyDays +``` + +## Specification + +### Rewrite of PowerShellGet + +PowerShellGet is currently written as PowerShell script with a dependency on PackageManagement. +Proposal is to write PSResource in C# to reduce complexity and make easier to maintain. +In addition, remove dependency on PackageManagement completely as well as dependency on +nuget.exe. +This module will depend on https://www.nuget.org/packages/NuGet.Client. +This module would be shipped in the PSGallery supporting older +versions of PowerShell. +This module will ship in PowerShell 7.1. + +### Side-by-side with PowerShellGet + +Since the current PowerShellGet 2.x version is a script module and this new one +is a C# based module, they can coexist side-by-side. + +Script and module metadata will retain the same format as it exists with v2. + +### Local cache + +Instead of always connecting to a repository to perform an online search, +`Find-PSResource` (see below) works against a local cache. +This will also enable changes in PowerShell to use `Find-PSResource -Type Command` to look +in this cache when it can't find a command and suggest to the user the module to install to +get that command. +This will be a local json file (one per repository) containing sufficient metadata +for searching for resources and their dependencies. +The cache will be stored in the user path. +There is no system cache that is shared. +A system wide cache would require elevation or sudo to create and update preventing +it from being useful. + +Example cache entry: + +```json +{ + "name": "This is my module", + "exportedFunctions": [ + "Get-One", + "Set-Two" + ], + "version": "1.0.0.0", + "type": "module", + "tags": [ + "Linux", + "PSEdition_Core", + "PSEdition_Desktop", + "AzureAutomation" + ] +} +``` + +An example cache with 5000 resources (approximately the number of unique packages +currently published to PowerShellGallery.com) is ~700KB in compressed json form. + +The cache would have both latest stable and latest prerelease versions of resources. + +>[!NOTE] +>Need to experiment if it makes sense to have a single cache file for all repositories +>or a different cache per repository for size and perf reasons. +>Perf tests will determine if the cache needs to be in binary form. + +### Automatic updating of the cache + +On any operation that reaches out to a repository, a REST API will be called to +see if the hash of the cache matches the current cache and if not, a new one +is downloaded. + +>[!NOTE] +>In the implementation we'll need to handle the case where the module attempts +>to update the cache while it is being read. +If the repository doesn't support this new API, it falls back to current behavior +in PSGet v2. +This means that there is no local cache of that repository and operations will +always connect to that repository. + +### Repository management + +`Register-PSResourceRepository` will allow for registering additional repositories. +A `-PSGallery` switch enables registering PSGallery should it be accidentally removed. +This would be in a different parameter set from `-URL` and `-Repositories`. + +The `-URL` will accept the HTTP address without the need to specify `/api/v3` as +that will be assumed and discovered at runtime (trying v3 first, then falling +back to v2, then the literal URL). +Support for local filesystem repositories must be maintained. +A `-Trusted` switch indicates whether to prompt the user when installing resources +from that repository. +Trusted repositories automatically have a priority of 0. +`-Trusted` is a different parameter set from `-Priority`. + +By default, if this switch is not specified, the repository is untrusted. +A `-Repositories` parameter will accept an array of hashtables equivalent to +the parameter names (like splatting). + +```powershell +Register-PSResourceRepository -Repositories @( + @{ URL = "https://one.com"; Name = "One"; Trusted = $true; Credential = $cred } + @{ URL = "https://powershellgallery.com"; Name = "PSGallery"; Trusted = $true; Default = $true } + @{ URL = "\\server\share\myrepository"; Name="Enterprise"; Trusted = $true } +) +``` + +A `-Name` parameter allows for setting a friendly name. + +A `-Priority` parameter allows setting the search order of repositories. +A lower value has higher priority. +If not specified, the default value is 50. +PSGallery will have a default value of 50. + +`Get-PSResourceRepository` will list out the registered repositories. + +`Set-PSResourceRepository` can be used to update a repository URL, trust level, +or priority. + +`Unregister-PSResourceRepository` can be used to un-register a repository. + +### Finding resources + +Current design of PowerShellGet is to have different cmdlets for each of the +different resources types that it supports: + +`Find-Command` +`Find-DscResource` +`Find-Module` +`Find-RoleCapability` +`Find-Script` + +Instead, these will be abstracted by a `Find-PSResource` cmdlet. +A `-Type` parameter will accept an array of types to allow filtering. + +With support of the generic `PSResource`, this means we can also find and +install arbitrary nupkgs that may only contain an assembly the user wants to +use in their script. + +For each repository, if the local cache does not exist, the cmdlet will call +`Update-PSResourceCache` first and then attempt a search. +However, this cmdlet does not update the cache if one exists. + +A `-Version` paramter will accept a string of a [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). + +A `-Prerelease` switch will allow include prerelease resources in the results. + +A `-ModuleName` paramter will accept a string, and will return commands, DSC resources, and Role Capabilities found in modules with that name. + +A `-Tags` parameter will accept a string array and will filter results based on PSResource tags. + +A `-Filter` parameter will accept a string array and will filter results based on all PSResource metadata (including resource descriptions). + +A `-Proxy` parameter will accept a uri. + +A `-ProxyCredential` parameter will accept a PSCredential + +A `-Credential` paramter will accept a PSCredential. + +An `-IncludeDependencies` switch will include dependent resources in the returned results. + +`Find-Module` will be retained to provide compatibility with v2. + +### Installing resources + +`Install-PSResource` will only work for modules unless `-DestinationPath` is +specified which works for all resource types. +A `-Prerelease` switch allows installing prerelease versions. +Other types will use `Save-PSResource` (see below). +A `-Repository` parameter accepts a specific repository name or URL to the repository: + +```powershell +Install-PSResource myModule -Repository 'https://mygallery.com' +# or +Install-Module myModule -Repository 'https://mygallery.com' +``` + +If `-Repository` is not specified and the same resource is found in multiple +repositories, then the resource is automatically installed quietly from the +trusted repository with the highest version matching the `-Version` parameter +(if specified, otherwise newest non-prerelease version unless `-Prerelease` +is used). +If there are no trusted repositories matching the query, then the newest version +fulfilling the query will be prompted to be installed from the highest priority +repositories. +If there are multiple repositories with the same priority level containing the same +version, the first one is used and will be prompted to install. + +`-TrustRepository` can be used to suppress being prompted for untrusted sources. +`-IgnoreDifferentPublisher` can be used to suppress being prompted if the publisher +of the module is different from the currently installed version. +`-Reinstall` can be used to re-install a module. + +A `-DestinationPath` parameter allows specifying the target directory instead +of the default one. +This will be in a different parameter set than `-Scope`. + +A `-Version` paramter will accept a string of a [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). + +A `-Tags` parameter will accept a string array and will filter based on PSResource tags. + +A `-Filter` parameter will accept a string array and will filter based on all PSResource metadata (including resource descriptions). + +A `-Proxy` parameter will accept a uri. + +A `-ProxyCredential` parameter will accept a PSCredential + +A `-Credential` parameter will accept a PSCredential. + +An `-IncludeDependencies` switch will include dependent resources in the returned results. + +A `-NoClobber` switch will prevent installing modules that have the same cmdlets +as a differently named module already on the system. + +A `-Quiet` switch will suppress progress information. + +`-Scope` with `AllUsers` and `CurrentUser` will work the same as it does in v2. +Default is `CurrentUser`. + +`Install-Module` cmdlet will be retained for compatibility with v2. + +v3 will continue to use the versioning folder format as v2 in that it will be +Major.Minor.Patch and not contain the semver prerelease label. + +### Dependencies and version management + +`Install-PSResource` can accept a path to a psd1 or json file (using `-RequiredResourcesFile`), +or a hashtable or json (using `-RequiredResources`) where the key is the module name and the +value is either the required version specified using Nuget version range syntax or +a hash table where `repository` is set to the URL of the repository and +`version` contains the [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). + +```powershell +Install-PSResource -RequiredResources @{ + "Configuration" = "[1.3.1,2.0)" + "Pester" = @{ + version = "[4.4.2,4.7.0]" + repository = "https://www.powershellgallery.com" + credential = $cred + allowPrerelease = $true + } +} +``` + +The json format will be the same as if this hashtable is passed to `ConvertTo-Json`: + +```json +{ + "Configuration": "[1.3.1,2.0)", + "Pester": { + "version": "[4.4.2,4.7.0]", + "credential": null, + "repository": "https://www.powershellgallery.com", + "allowPrerelease": true + } +} +``` + +>[!NOTE] +>Credentials management will be using [Secrets Management RFC](https://github.com/PowerShell/PowerShell-RFC/pull/208) +> for generalized credential management. +The `repository` property can be the name of the registered repository or the URL +to a repository. +If no `repository` is specified, it will use the `default` repository. + +The older `System.Version` four part version type will be supported to retain +compatibility with existing published modules using that format. + +If the resource requires dependencies that are not already installed, then +a prompt will appear _before anything is installed_, listing all the resources +including module name, version, size, and repository to be installed unless +`-IncludeDependencies` is specified which will install without prompting. +Rejecting the prompt will result in nothing being installed. + +Declared dependencies are searched and installed using the same trust algorithm +as described for `Install-PSResource` above. + +>[!NOTE] +>Installing dependencies is following the `apt` experience in that it prompts +>by default instead of automatically installing dependencies. Since dependencies +>can be quite large, this would be equivalent to `ConfirmImpact=High` which +>would prompt by default. +>Dependency installation works with `-DestinationPath` parameter. + +### Saving resources + +This cmdlet uses `Install-PSResource -DestinationPath`. + +With the removal of PackageManagement, there is still a need to support saving +arbitrary nupkgs (assemblies) used for scripts. + +`Save-PSResource -Type Library` will download nupkgs that have a `lib` folder. +The dependent native library in `runtimes` matching the current system runtime +will be copied to the root of the destination specified. +A `-IncludeAllRuntimes` can be used to explicitly retain the `runtimes` directory +hierarchy within the nupkg to the root of the destination. + +>[!NOTE] +>Library support may not be available in 3.0. +A `-Prerelease` switch allows saving prerelease versions. +If the `-Version` includes a prerelease label like `2.0.0-beta4`, then this +switch is not necessary and the prerelease version will be installed. + +A `-AsNupkg` switch will save the resource as a nupkg (if it was originally a +nupkg) instead of expanding it into a folder. + +### Updating resources + +`Update-PSResource` will update all resources to most recent minor version by default. + +A `-UpdateTo` parameter has values `MinorVersion` (as default), `MajorVersion`, `PatchVersion`. +`MajorVersion` allows updating to latest major version. +`PatchVersion` allows updating to latest patch version (e.g. 6.2.1 to 6.2.4). + +If the installed resource is a pre-release, it will automatically update to +latest prerelease or stable version (if available). + +### Publishing resources + +`Publish-PSResource` will supporting publishing modules and scripts. +It will follow the same model as `Publish-Module` in v2. + +A `-DestinationPath` can be used to publish the resulting nupkg locally. +`Publish-PSResource` can accept a path to the nupkg to publish to a repository. + +A `-Nuspec` parameter can be used to specify a nuspec file rather than relying +on this module to produce one. + +A `-SkipDependenciesCheck` switch can be used to bypass the default check that +all dependencies are present. + +### Listing installed resources + +`Get-PSResource` will list all installed resources with new `Type` column. + +```output +Version Name Type Repository Description +------- ---- ---- ---------- ----------- +5.0.0 VSTeam Module PSGallery Adds functionality for working with Visual … +0.14.94 PSGitHub Module PSGallery This PowerShell module enables integration … +0.7.3 posh-git Module PSGallery Provides prompt with Git status summary … +1.0.1 PSAutoMute Script PSGallery Powershell script to Auto Mute you sound devices … +``` + +### Uninstalling resources + +`Uninstall-PSResource` will be available to remove installed resources. + +>[!NOTE] +>Uninstalling dependencies automatically will be something to consider in the future. + +### Proxy support + +Each cmdlet will have `-Proxy` and `-ProxyCredential` parameters. + +### PowerShellGallery status + +Upon failure to connect to PSGallery, the cmdlets should retrieve status from +a well known REST API and return a more descriptive error message to the user. + +## Alternate Proposals and Considerations + +These are items are outside the scope of this RFC and version 3.0. +Many of these items can be addressed in a future version of PowerShellGet 3.x without +introducing a breaking change: + +- This RFC does not cover the module authoring experience on publishing a cross-platform + module with multiple dependencies and supporting multiple runtimes. + +- If there is a desire to explicitly update the local cache (like `apt`), we can introduce a + `Update-PSResourceCache` cmdlet in the future and a property on PSRepository registrations + indicating whether it auto-updates or not. + +- The ability to set policy for PSGet + +- Automatic cleanup of old resources when upgrading or removing resources + +- Signing incorporated into `Publish-PSResource` + +- `-Trusted` switch to `Install-PSResource` and `Find-PSResource` that pre-validates + the signing cert is trusted on the system + +- Notify users that they have outdated versions of a module (just patch versions?) + +- A way to filter resources to those allowed to be installed on the system + +- Enterprise management of local cache + +- Full semver support won't happen until there is a semver type in .NET + +- Consider using a merkle tree to validate modules + +PowerShell module loading needs to be updated to [understand semver](https://github.com/PowerShell/PowerShell/issues/2983). + +Instead of `Find-PSResource`, call it `Find-Resource`. +Context may be understandable since the cmdlet comes from PowerShellGet, but +`Resource` as a noun is pretty general. From b49170148ab35b220fe596ffaeb58fd95498447a Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Wed, 29 Jan 2020 10:26:52 -0800 Subject: [PATCH 06/12] Update RFCxxxx-PowerShellGet-3.0.md --- 1-Draft/RFCxxxx-PowerShellGet-3.0.md | 52 ++++++++++++++++++++-------- 1 file changed, 38 insertions(+), 14 deletions(-) diff --git a/1-Draft/RFCxxxx-PowerShellGet-3.0.md b/1-Draft/RFCxxxx-PowerShellGet-3.0.md index 6dee3aeed..8d13a6642 100644 --- a/1-Draft/RFCxxxx-PowerShellGet-3.0.md +++ b/1-Draft/RFCxxxx-PowerShellGet-3.0.md @@ -1,4 +1,4 @@ ---- +--- RFC: RFCxxxx Author: Sydney Smith Status: Draft @@ -20,6 +20,7 @@ This RFC proposes some significant changes to address this. > [!NOTE] > There are no intentions to break existing PSGet v2 users so existing > users can stay on PSGet v2 and that will continue to work. + ## Motivation As a PowerShell user, @@ -60,7 +61,7 @@ nuget.exe. This module will depend on https://www.nuget.org/packages/NuGet.Client. This module would be shipped in the PSGallery supporting older versions of PowerShell. -This module will ship in PowerShell 7.1. +This module will ship in PowerShell 7.1 alongside PowerShellGet 2.x. ### Side-by-side with PowerShellGet @@ -69,6 +70,10 @@ is a C# based module, they can coexist side-by-side. Script and module metadata will retain the same format as it exists with v2. +We will introduce an `Enable-PowerShellGetAlias` cmdlet that will allow users to use known cmdlets like Install-Module +from the PowerShellGet 3.0 implementation. +This additional cmdlet may not ship in PowerShellGet 3.0 but is planned as a part of the transition effort. + ### Local cache Instead of always connecting to a repository to perform an online search, @@ -113,11 +118,14 @@ The cache would have both latest stable and latest prerelease versions of resour >or a different cache per repository for size and perf reasons. >Perf tests will determine if the cache needs to be in binary form. -### Automatic updating of the cache +### Updating of the cache -On any operation that reaches out to a repository, a REST API will be called to -see if the hash of the cache matches the current cache and if not, a new one -is downloaded. +In the initial implementation of the module we will use introduce a cmdlet `Update-PSCache` +in order to update the cache. + +We will continue to explore automatic updating of the cache in which on any operation that reaches out to a repository, +a REST API will be called to see if the hash of the cache matches the current cache and if not, a new one +is downloaded. This system may also take into consideration how recently the cache has been updated to avoid constant updates. >[!NOTE] >In the implementation we'll need to handle the case where the module attempts @@ -131,7 +139,7 @@ always connect to that repository. `Register-PSResourceRepository` will allow for registering additional repositories. A `-PSGallery` switch enables registering PSGallery should it be accidentally removed. -This would be in a different parameter set from `-URL` and `-Repositories`. +This would be in a different parameter set from `-URL` and `-Repository`. The `-URL` will accept the HTTP address without the need to specify `/api/v3` as that will be assumed and discovered at runtime (trying v3 first, then falling @@ -143,7 +151,7 @@ Trusted repositories automatically have a priority of 0. `-Trusted` is a different parameter set from `-Priority`. By default, if this switch is not specified, the repository is untrusted. -A `-Repositories` parameter will accept an array of hashtables equivalent to +A `-Repository` parameter will accept an array of hashtables equivalent to the parameter names (like splatting). ```powershell @@ -209,12 +217,13 @@ A `-Credential` paramter will accept a PSCredential. An `-IncludeDependencies` switch will include dependent resources in the returned results. `Find-Module` will be retained to provide compatibility with v2. +If the prerelease switch is not used, and only a prerelease version of the module is found a warning will be emitted. ### Installing resources -`Install-PSResource` will only work for modules unless `-DestinationPath` is +`Install-PSResource` will mirror the parameter set for `Find-PSResource`. +The cmd will only work for modules unless `-DestinationPath` is specified which works for all resource types. -A `-Prerelease` switch allows installing prerelease versions. Other types will use `Save-PSResource` (see below). A `-Repository` parameter accepts a specific repository name or URL to the repository: @@ -244,7 +253,9 @@ A `-DestinationPath` parameter allows specifying the target directory instead of the default one. This will be in a different parameter set than `-Scope`. -A `-Version` paramter will accept a string of a [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). +A `-Type` parameter will accept an array of types to allow filtering. + +A `-Version` parameter will accept a string of a [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). A `-Tags` parameter will accept a string array and will filter based on PSResource tags. @@ -266,7 +277,7 @@ A `-Quiet` switch will suppress progress information. `-Scope` with `AllUsers` and `CurrentUser` will work the same as it does in v2. Default is `CurrentUser`. -`Install-Module` cmdlet will be retained for compatibility with v2. +If aliases are enabled `Install-Module` will call to `Install-PSResource -Type Module`. v3 will continue to use the versioning folder format as v2 in that it will be Major.Minor.Patch and not contain the semver prerelease label. @@ -398,6 +409,14 @@ Version Name Type Repository Description >[!NOTE] >Uninstalling dependencies automatically will be something to consider in the future. +If a package is unable to be uninstalled because it is currently loaded, the cmdlet will emit a warning +which enumerates the packages that were unable to be installed. +We will explore a dependency clean parameter in a future version of PowerShellGet. + +When uninstalling the dependency check will be performed not only on the RequiredModules.Name, +but also on RequiredModules.Version, which should be less-than-or-equal to the version of the module being +uninstalled in order to be considered a dependency. + ### Proxy support Each cmdlet will have `-Proxy` and `-ProxyCredential` parameters. @@ -407,6 +426,11 @@ Each cmdlet will have `-Proxy` and `-ProxyCredential` parameters. Upon failure to connect to PSGallery, the cmdlets should retrieve status from a well known REST API and return a more descriptive error message to the user. +### Open Sourcing the module + +In order to open source PowerShellGet 3.0 we will fork the current PowerShellGet repo, rename it as PowerShellGet2, +and use the existing PowerShellGet repo as the primary support and development contact for PowerShellGet 3+. + ## Alternate Proposals and Considerations These are items are outside the scope of this RFC and version 3.0. @@ -416,7 +440,7 @@ introducing a breaking change: - This RFC does not cover the module authoring experience on publishing a cross-platform module with multiple dependencies and supporting multiple runtimes. -- If there is a desire to explicitly update the local cache (like `apt`), we can introduce a +- [Should be the opposite] If there is a desire to explicitly update the local cache (like `apt`), we can introduce a `Update-PSResourceCache` cmdlet in the future and a property on PSRepository registrations indicating whether it auto-updates or not. @@ -433,7 +457,7 @@ introducing a breaking change: - A way to filter resources to those allowed to be installed on the system -- Enterprise management of local cache +- Enterprise management of local cache/the ability to configure a PSRepository on a system-wide level - Full semver support won't happen until there is a semver type in .NET From fb20ffd98cbe241b32ecc733bba76f2f22ec3364 Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Wed, 29 Jan 2020 10:28:53 -0800 Subject: [PATCH 07/12] Update RFCxxxx-PowerShellGet-3.0.md --- 1-Draft/RFCxxxx-PowerShellGet-3.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/1-Draft/RFCxxxx-PowerShellGet-3.0.md b/1-Draft/RFCxxxx-PowerShellGet-3.0.md index 8d13a6642..b4024e562 100644 --- a/1-Draft/RFCxxxx-PowerShellGet-3.0.md +++ b/1-Draft/RFCxxxx-PowerShellGet-3.0.md @@ -284,14 +284,14 @@ Major.Minor.Patch and not contain the semver prerelease label. ### Dependencies and version management -`Install-PSResource` can accept a path to a psd1 or json file (using `-RequiredResourcesFile`), -or a hashtable or json (using `-RequiredResources`) where the key is the module name and the +`Install-PSResource` can accept a path to a psd1 or json file (using `-RequiredResourceFile`), +or a hashtable or json (using `-RequiredResource`) where the key is the module name and the value is either the required version specified using Nuget version range syntax or a hash table where `repository` is set to the URL of the repository and `version` contains the [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). ```powershell -Install-PSResource -RequiredResources @{ +Install-PSResource -RequiredResource @{ "Configuration" = "[1.3.1,2.0)" "Pester" = @{ version = "[4.4.2,4.7.0]" From b5e81c930eab63b0a5ee08a2d234d37e83e0b863 Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Tue, 11 Feb 2020 12:15:55 -0800 Subject: [PATCH 08/12] Update RFCxxxx-PowerShellGet-3.0.md --- 1-Draft/RFCxxxx-PowerShellGet-3.0.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/1-Draft/RFCxxxx-PowerShellGet-3.0.md b/1-Draft/RFCxxxx-PowerShellGet-3.0.md index b4024e562..a9ac00dd4 100644 --- a/1-Draft/RFCxxxx-PowerShellGet-3.0.md +++ b/1-Draft/RFCxxxx-PowerShellGet-3.0.md @@ -301,7 +301,7 @@ Install-PSResource -RequiredResource @{ } } ``` - +In this case the modules named "Configuration", and "Pester" will be installed. The json format will be the same as if this hashtable is passed to `ConvertTo-Json`: ```json @@ -342,6 +342,9 @@ as described for `Install-PSResource` above. >would prompt by default. >Dependency installation works with `-DestinationPath` parameter. +We will also introduce a `New-RequiredResourceFile` cmdlet which will create a template file. +If the switch `-AsPSD1` is used it will create a psd1 file, otherwise it will default to json. + ### Saving resources This cmdlet uses `Install-PSResource -DestinationPath`. From 2171bd358ab159fc8bd29d1435b5456d06f8ad5a Mon Sep 17 00:00:00 2001 From: Steve Lee Date: Tue, 10 Mar 2020 13:41:50 -0700 Subject: [PATCH 09/12] Rename 1-Draft/RFCxxxx-PowerShellGet-3.0.md to 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md Moving to Draft-Accepted folder --- {1-Draft => 2-Draft-Accepted}/RFCxxxx-PowerShellGet-3.0.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {1-Draft => 2-Draft-Accepted}/RFCxxxx-PowerShellGet-3.0.md (100%) diff --git a/1-Draft/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md similarity index 100% rename from 1-Draft/RFCxxxx-PowerShellGet-3.0.md rename to 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md From b118eca8edb56a4988dd1072b208cd8dbbfebe76 Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Mon, 18 May 2020 15:39:10 -0700 Subject: [PATCH 10/12] Update RFCxxxx-PowerShellGet-3.0.md --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 67 ++++++++++--------- 1 file changed, 35 insertions(+), 32 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index a9ac00dd4..9014d6887 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -38,15 +38,6 @@ This RFC proposes some significant changes to address this. ## User Experience ```powershell -PS> Get-Satisfaction -Get-Satisfaction : The term 'Get-Satisfaction' is not recognized as the name of a cmdlet, function, script file, or operable program. -Check the spelling of the name, or if a path was included, verify that the path is correct and try again. -At line:1 char:1 -+ Get-Satisfaction -+ ~~~ -+ CategoryInfo : ObjectNotFound: (Get-Satisfaction:String) [], CommandNotFoundException -+ FullyQualifiedErrorId : CommandNotFoundException -Suggestion: You can obtain `Get-Satisfaction` by running `Install-PSResource HappyDays`. PS> Install-PSResource HappyDays ``` @@ -58,17 +49,16 @@ PowerShellGet is currently written as PowerShell script with a dependency on Pac Proposal is to write PSResource in C# to reduce complexity and make easier to maintain. In addition, remove dependency on PackageManagement completely as well as dependency on nuget.exe. -This module will depend on https://www.nuget.org/packages/NuGet.Client. -This module would be shipped in the PSGallery supporting older -versions of PowerShell. -This module will ship in PowerShell 7.1 alongside PowerShellGet 2.x. +This module will depend on Nuget Server and Client packages including https://www.nuget.org/packages/NuGet.Client. +This module would be shipped in the PSGallery supporting PowerShell 5.1+. +This module will likely ship in PowerShell 7.1 alongside PowerShellGet 2.x. ### Side-by-side with PowerShellGet Since the current PowerShellGet 2.x version is a script module and this new one is a C# based module, they can coexist side-by-side. -Script and module metadata will retain the same format as it exists with v2. +Script and module metadata will retain compatible formatting as it exists with v2. We will introduce an `Enable-PowerShellGetAlias` cmdlet that will allow users to use known cmdlets like Install-Module from the PowerShellGet 3.0 implementation. @@ -141,9 +131,7 @@ always connect to that repository. A `-PSGallery` switch enables registering PSGallery should it be accidentally removed. This would be in a different parameter set from `-URL` and `-Repository`. -The `-URL` will accept the HTTP address without the need to specify `/api/v3` as -that will be assumed and discovered at runtime (trying v3 first, then falling -back to v2, then the literal URL). +The `-URL` will accept the HTTP address with support for v2 and v3 protocols. Support for local filesystem repositories must be maintained. A `-Trusted` switch indicates whether to prompt the user when installing resources from that repository. @@ -155,7 +143,7 @@ A `-Repository` parameter will accept an array of hashtables equivalent to the parameter names (like splatting). ```powershell -Register-PSResourceRepository -Repositories @( +Register-PSResourceRepository -Repository @( @{ URL = "https://one.com"; Name = "One"; Trusted = $true; Credential = $cred } @{ URL = "https://powershellgallery.com"; Name = "PSGallery"; Trusted = $true; Default = $true } @{ URL = "\\server\share\myrepository"; Name="Enterprise"; Trusted = $true } @@ -217,7 +205,6 @@ A `-Credential` paramter will accept a PSCredential. An `-IncludeDependencies` switch will include dependent resources in the returned results. `Find-Module` will be retained to provide compatibility with v2. -If the prerelease switch is not used, and only a prerelease version of the module is found a warning will be emitted. ### Installing resources @@ -242,7 +229,7 @@ If there are no trusted repositories matching the query, then the newest version fulfilling the query will be prompted to be installed from the highest priority repositories. If there are multiple repositories with the same priority level containing the same -version, the first one is used and will be prompted to install. +version, the first one registered is used and will be prompted to install. `-TrustRepository` can be used to suppress being prompted for untrusted sources. `-IgnoreDifferentPublisher` can be used to suppress being prompted if the publisher @@ -292,10 +279,10 @@ a hash table where `repository` is set to the URL of the repository and ```powershell Install-PSResource -RequiredResource @{ - "Configuration" = "[1.3.1,2.0)" - "Pester" = @{ - version = "[4.4.2,4.7.0]" - repository = "https://www.powershellgallery.com" + 'Configuration' = '[1.3.1,2.0)' + 'Pester' = @{ + version = '[4.4.2,4.7.0]' + repository = 'https://www.powershellgallery.com' credential = $cred allowPrerelease = $true } @@ -306,12 +293,12 @@ The json format will be the same as if this hashtable is passed to `ConvertTo-Js ```json { - "Configuration": "[1.3.1,2.0)", - "Pester": { - "version": "[4.4.2,4.7.0]", - "credential": null, - "repository": "https://www.powershellgallery.com", - "allowPrerelease": true + 'Configuration': '[1.3.1,2.0)', + 'Pester': { + 'version': '[4.4.2,4.7.0]', + 'credential': null, + 'repository': 'https://www.powershellgallery.com', + 'prerelease': true } } ``` @@ -380,7 +367,7 @@ latest prerelease or stable version (if available). ### Publishing resources -`Publish-PSResource` will supporting publishing modules and scripts. +`Publish-PSResource` will supporting publishing modules, scripts and nupkgs. It will follow the same model as `Publish-Module` in v2. A `-DestinationPath` can be used to publish the resulting nupkg locally. @@ -431,7 +418,7 @@ a well known REST API and return a more descriptive error message to the user. ### Open Sourcing the module -In order to open source PowerShellGet 3.0 we will fork the current PowerShellGet repo, rename it as PowerShellGet2, +In order to open source PowerShellGet 3.0 we will fork the current PowerShellGet repo, rename it as PowerShellGetv2, and use the existing PowerShellGet repo as the primary support and development contact for PowerShellGet 3+. ## Alternate Proposals and Considerations @@ -464,8 +451,24 @@ introducing a breaking change: - Full semver support won't happen until there is a semver type in .NET +- Consider if the prerelease switch is not used, and only a prerelease version of the module is found a warning will be emitted. + - Consider using a merkle tree to validate modules +- Consider adding support for automatic searches from Find-PSResource -Type Command when a command in not found in module discovery +```powershell +PS> Get-Satisfaction +Get-Satisfaction : The term 'Get-Satisfaction' is not recognized as the name of a cmdlet, function, script file, or operable program. +Check the spelling of the name, or if a path was included, verify that the path is correct and try again. +At line:1 char:1 ++ Get-Satisfaction ++ ~~~ ++ CategoryInfo : ObjectNotFound: (Get-Satisfaction:String) [], CommandNotFoundException ++ FullyQualifiedErrorId : CommandNotFoundException +Suggestion: You can obtain `Get-Satisfaction` by running `Install-PSResource HappyDays`. +PS> Install-PSResource HappyDays +``` + PowerShell module loading needs to be updated to [understand semver](https://github.com/PowerShell/PowerShell/issues/2983). Instead of `Find-PSResource`, call it `Find-Resource`. From 881d07f91ebc8bf9a8866fe9f2480e66893f8720 Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Mon, 14 Sep 2020 11:55:33 -0700 Subject: [PATCH 11/12] Update RFCxxxx-PowerShellGet-3.0.md Updated based on current scope for PowershellGet 3.0 release. Includes the following updates: - Plans to ship in PowerShell 7.2 - CompatPowerShellGet compatibility module - No `-Filter` parameter for find and install - Integration with SecretManagement moved to post-GA - Install-PSResource installs dependencies by default - New-RequiredResourceFile cmdlet moved to post-GA - Save-PSResource `-Type Library` and `-IncludeAllRuntimes` moved to post-GA - Update-PSResoure `-UpdateTo` parameter moved to post-GA - Local Cache support moved to post-GA --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 188 ++++++------------ 1 file changed, 62 insertions(+), 126 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index 9014d6887..9b985cbda 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -23,10 +23,6 @@ This RFC proposes some significant changes to address this. ## Motivation - As a PowerShell user, - I can discover how to install missing cmdlets, - so that I can be more productive. - As a PowerShell user, I can easily install modules without specifying lots of switches, so that I can be more productive. @@ -51,7 +47,7 @@ In addition, remove dependency on PackageManagement completely as well as depend nuget.exe. This module will depend on Nuget Server and Client packages including https://www.nuget.org/packages/NuGet.Client. This module would be shipped in the PSGallery supporting PowerShell 5.1+. -This module will likely ship in PowerShell 7.1 alongside PowerShellGet 2.x. +This module will likely ship in PowerShell 7.2 alongside PowerShellGet compatibility module. ### Side-by-side with PowerShellGet @@ -60,70 +56,9 @@ is a C# based module, they can coexist side-by-side. Script and module metadata will retain compatible formatting as it exists with v2. -We will introduce an `Enable-PowerShellGetAlias` cmdlet that will allow users to use known cmdlets like Install-Module +We will introduce a compatibility module that will allow users to use known cmdlets like Install-Module from the PowerShellGet 3.0 implementation. -This additional cmdlet may not ship in PowerShellGet 3.0 but is planned as a part of the transition effort. - -### Local cache - -Instead of always connecting to a repository to perform an online search, -`Find-PSResource` (see below) works against a local cache. -This will also enable changes in PowerShell to use `Find-PSResource -Type Command` to look -in this cache when it can't find a command and suggest to the user the module to install to -get that command. -This will be a local json file (one per repository) containing sufficient metadata -for searching for resources and their dependencies. -The cache will be stored in the user path. -There is no system cache that is shared. -A system wide cache would require elevation or sudo to create and update preventing -it from being useful. - -Example cache entry: - -```json -{ - "name": "This is my module", - "exportedFunctions": [ - "Get-One", - "Set-Two" - ], - "version": "1.0.0.0", - "type": "module", - "tags": [ - "Linux", - "PSEdition_Core", - "PSEdition_Desktop", - "AzureAutomation" - ] -} -``` - -An example cache with 5000 resources (approximately the number of unique packages -currently published to PowerShellGallery.com) is ~700KB in compressed json form. - -The cache would have both latest stable and latest prerelease versions of resources. - ->[!NOTE] ->Need to experiment if it makes sense to have a single cache file for all repositories ->or a different cache per repository for size and perf reasons. ->Perf tests will determine if the cache needs to be in binary form. - -### Updating of the cache - -In the initial implementation of the module we will use introduce a cmdlet `Update-PSCache` -in order to update the cache. - -We will continue to explore automatic updating of the cache in which on any operation that reaches out to a repository, -a REST API will be called to see if the hash of the cache matches the current cache and if not, a new one -is downloaded. This system may also take into consideration how recently the cache has been updated to avoid constant updates. - ->[!NOTE] ->In the implementation we'll need to handle the case where the module attempts ->to update the cache while it is being read. -If the repository doesn't support this new API, it falls back to current behavior -in PSGet v2. -This means that there is no local cache of that repository and operations will -always connect to that repository. +This module will ship separately from PowerShellGet 3.0 but is planned as a part of the transition effort. ### Repository management @@ -194,8 +129,6 @@ A `-ModuleName` paramter will accept a string, and will return commands, DSC res A `-Tags` parameter will accept a string array and will filter results based on PSResource tags. -A `-Filter` parameter will accept a string array and will filter results based on all PSResource metadata (including resource descriptions). - A `-Proxy` parameter will accept a uri. A `-ProxyCredential` parameter will accept a PSCredential @@ -246,16 +179,12 @@ A `-Version` parameter will accept a string of a [Nuget version range syntax](ht A `-Tags` parameter will accept a string array and will filter based on PSResource tags. -A `-Filter` parameter will accept a string array and will filter based on all PSResource metadata (including resource descriptions). - A `-Proxy` parameter will accept a uri. A `-ProxyCredential` parameter will accept a PSCredential A `-Credential` parameter will accept a PSCredential. -An `-IncludeDependencies` switch will include dependent resources in the returned results. - A `-NoClobber` switch will prevent installing modules that have the same cmdlets as a differently named module already on the system. @@ -303,35 +232,12 @@ The json format will be the same as if this hashtable is passed to `ConvertTo-Js } ``` ->[!NOTE] ->Credentials management will be using [Secrets Management RFC](https://github.com/PowerShell/PowerShell-RFC/pull/208) -> for generalized credential management. -The `repository` property can be the name of the registered repository or the URL -to a repository. -If no `repository` is specified, it will use the `default` repository. - The older `System.Version` four part version type will be supported to retain compatibility with existing published modules using that format. -If the resource requires dependencies that are not already installed, then -a prompt will appear _before anything is installed_, listing all the resources -including module name, version, size, and repository to be installed unless -`-IncludeDependencies` is specified which will install without prompting. -Rejecting the prompt will result in nothing being installed. - Declared dependencies are searched and installed using the same trust algorithm as described for `Install-PSResource` above. ->[!NOTE] ->Installing dependencies is following the `apt` experience in that it prompts ->by default instead of automatically installing dependencies. Since dependencies ->can be quite large, this would be equivalent to `ConfirmImpact=High` which ->would prompt by default. ->Dependency installation works with `-DestinationPath` parameter. - -We will also introduce a `New-RequiredResourceFile` cmdlet which will create a template file. -If the switch `-AsPSD1` is used it will create a psd1 file, otherwise it will default to json. - ### Saving resources This cmdlet uses `Install-PSResource -DestinationPath`. @@ -339,28 +245,12 @@ This cmdlet uses `Install-PSResource -DestinationPath`. With the removal of PackageManagement, there is still a need to support saving arbitrary nupkgs (assemblies) used for scripts. -`Save-PSResource -Type Library` will download nupkgs that have a `lib` folder. -The dependent native library in `runtimes` matching the current system runtime -will be copied to the root of the destination specified. -A `-IncludeAllRuntimes` can be used to explicitly retain the `runtimes` directory -hierarchy within the nupkg to the root of the destination. - ->[!NOTE] ->Library support may not be available in 3.0. -A `-Prerelease` switch allows saving prerelease versions. -If the `-Version` includes a prerelease label like `2.0.0-beta4`, then this -switch is not necessary and the prerelease version will be installed. - A `-AsNupkg` switch will save the resource as a nupkg (if it was originally a nupkg) instead of expanding it into a folder. ### Updating resources -`Update-PSResource` will update all resources to most recent minor version by default. - -A `-UpdateTo` parameter has values `MinorVersion` (as default), `MajorVersion`, `PatchVersion`. -`MajorVersion` allows updating to latest major version. -`PatchVersion` allows updating to latest patch version (e.g. 6.2.1 to 6.2.4). +`Update-PSResource` will update all resources to most recent version by default. If the installed resource is a pre-release, it will automatically update to latest prerelease or stable version (if available). @@ -411,11 +301,6 @@ uninstalled in order to be considered a dependency. Each cmdlet will have `-Proxy` and `-ProxyCredential` parameters. -### PowerShellGallery status - -Upon failure to connect to PSGallery, the cmdlets should retrieve status from -a well known REST API and return a more descriptive error message to the user. - ### Open Sourcing the module In order to open source PowerShellGet 3.0 we will fork the current PowerShellGet repo, rename it as PowerShellGetv2, @@ -424,6 +309,8 @@ and use the existing PowerShellGet repo as the primary support and development c ## Alternate Proposals and Considerations These are items are outside the scope of this RFC and version 3.0. +This list includes a number of features that were included in the inital versions +of the RFC but did not meet the final scope of PowerShellGet 3.0. Many of these items can be addressed in a future version of PowerShellGet 3.x without introducing a breaking change: @@ -451,11 +338,11 @@ introducing a breaking change: - Full semver support won't happen until there is a semver type in .NET -- Consider if the prerelease switch is not used, and only a prerelease version of the module is found a warning will be emitted. +- If the prerelease switch is not used, and only a prerelease version of the module is found a warning will be emitted. -- Consider using a merkle tree to validate modules +- Using a merkle tree to validate modules -- Consider adding support for automatic searches from Find-PSResource -Type Command when a command in not found in module discovery +- Support for automatic searches from Find-PSResource -Type Command when a command in not found in module discovery ```powershell PS> Get-Satisfaction Get-Satisfaction : The term 'Get-Satisfaction' is not recognized as the name of a cmdlet, function, script file, or operable program. @@ -469,8 +356,57 @@ Suggestion: You can obtain `Get-Satisfaction` by running `Install-PSResource Hap PS> Install-PSResource HappyDays ``` -PowerShell module loading needs to be updated to [understand semver](https://github.com/PowerShell/PowerShell/issues/2983). +- PowerShell module loading needs to be updated to [understand semver](https://github.com/PowerShell/PowerShell/issues/2983). + +- Local cacheing,instead of always connecting to a repository to perform an online search, +`Find-PSResource` (see below) works against a local cache. +This will also enable changes in PowerShell to use `Find-PSResource -Type Command` to look +in this cache when it can't find a command and suggest to the user the module to install to +get that command. +This will be a local json file (one per repository) containing sufficient metadata +for searching for resources and their dependencies. +The cache will be stored in the user path. +There is no system cache that is shared. +A system wide cache would require elevation or sudo to create and update preventing +it from being useful. + +Example cache entry: -Instead of `Find-PSResource`, call it `Find-Resource`. -Context may be understandable since the cmdlet comes from PowerShellGet, but -`Resource` as a noun is pretty general. +```json +{ + "name": "This is my module", + "exportedFunctions": [ + "Get-One", + "Set-Two" + ], + "version": "1.0.0.0", + "type": "module", + "tags": [ + "Linux", + "PSEdition_Core", + "PSEdition_Desktop", + "AzureAutomation" + ] +} +``` + +An example cache with 5000 resources (approximately the number of unique packages +currently published to PowerShellGallery.com) is ~700KB in compressed json form. + +The cache would have both latest stable and latest prerelease versions of resources. + +- Updating of the local cache + +- Credentials management using [Secrets Management RFC](https://github.com/PowerShell/PowerShell-RFC/pull/208) for generalized credential management. +The `repository` property can be the name of the registered repository or the URL +to a repository. +If no `repository` is specified, it will use the `default` repository. + +- `Save-PSResource -Type Library` will download nupkgs that have a `lib` folder. +The dependent native library in `runtimes` matching the current system runtime +will be copied to the root of the destination specified. +A `-IncludeAllRuntimes` can be used to explicitly retain the `runtimes` directory +hierarchy within the nupkg to the root of the destination. + +- Upon failure to connect to PSGallery, the cmdlets should retrieve status from +a well known REST API and return a more descriptive error message to the user. From 17ecbf7c091dca0c00bad7eb5054491d9d5a0639 Mon Sep 17 00:00:00 2001 From: Sydney Smith <43417619+SydneyhSmith@users.noreply.github.com> Date: Wed, 12 Jan 2022 08:34:53 -0800 Subject: [PATCH 12/12] Update RFCxxxx-PowerShellGet-3.0.md --- 2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md | 33 ++++--------------- 1 file changed, 7 insertions(+), 26 deletions(-) diff --git a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md index 9b985cbda..e472721df 100644 --- a/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md +++ b/2-Draft-Accepted/RFCxxxx-PowerShellGet-3.0.md @@ -117,10 +117,6 @@ With support of the generic `PSResource`, this means we can also find and install arbitrary nupkgs that may only contain an assembly the user wants to use in their script. -For each repository, if the local cache does not exist, the cmdlet will call -`Update-PSResourceCache` first and then attempt a search. -However, this cmdlet does not update the cache if one exists. - A `-Version` paramter will accept a string of a [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). A `-Prerelease` switch will allow include prerelease resources in the results. @@ -142,9 +138,6 @@ An `-IncludeDependencies` switch will include dependent resources in the returne ### Installing resources `Install-PSResource` will mirror the parameter set for `Find-PSResource`. -The cmd will only work for modules unless `-DestinationPath` is -specified which works for all resource types. -Other types will use `Save-PSResource` (see below). A `-Repository` parameter accepts a specific repository name or URL to the repository: ```powershell @@ -165,20 +158,14 @@ If there are multiple repositories with the same priority level containing the s version, the first one registered is used and will be prompted to install. `-TrustRepository` can be used to suppress being prompted for untrusted sources. -`-IgnoreDifferentPublisher` can be used to suppress being prompted if the publisher +`-SkipPublisherCheck` can be used to suppress being prompted if the publisher of the module is different from the currently installed version. `-Reinstall` can be used to re-install a module. -A `-DestinationPath` parameter allows specifying the target directory instead -of the default one. -This will be in a different parameter set than `-Scope`. - A `-Type` parameter will accept an array of types to allow filtering. A `-Version` parameter will accept a string of a [Nuget version range syntax](https://docs.microsoft.com/en-us/nuget/reference/package-versioning#version-ranges-and-wildcards). -A `-Tags` parameter will accept a string array and will filter based on PSResource tags. - A `-Proxy` parameter will accept a uri. A `-ProxyCredential` parameter will accept a PSCredential @@ -252,9 +239,6 @@ nupkg) instead of expanding it into a folder. `Update-PSResource` will update all resources to most recent version by default. -If the installed resource is a pre-release, it will automatically update to -latest prerelease or stable version (if available). - ### Publishing resources `Publish-PSResource` will supporting publishing modules, scripts and nupkgs. @@ -263,9 +247,6 @@ It will follow the same model as `Publish-Module` in v2. A `-DestinationPath` can be used to publish the resulting nupkg locally. `Publish-PSResource` can accept a path to the nupkg to publish to a repository. -A `-Nuspec` parameter can be used to specify a nuspec file rather than relying -on this module to produce one. - A `-SkipDependenciesCheck` switch can be used to bypass the default check that all dependencies are present. @@ -274,12 +255,12 @@ all dependencies are present. `Get-PSResource` will list all installed resources with new `Type` column. ```output -Version Name Type Repository Description -------- ---- ---- ---------- ----------- -5.0.0 VSTeam Module PSGallery Adds functionality for working with Visual … -0.14.94 PSGitHub Module PSGallery This PowerShell module enables integration … -0.7.3 posh-git Module PSGallery Provides prompt with Git status summary … -1.0.1 PSAutoMute Script PSGallery Powershell script to Auto Mute you sound devices … +Version Name Repository Description +------- ---- ---------- ----------- +5.0.0 VSTeam PSGallery Adds functionality for working with Visual … +0.14.94 PSGitHub PSGallery This PowerShell module enables integration … +0.7.3 posh-git PSGallery Provides prompt with Git status summary … +1.0.1 PSAutoMute PSGallery Powershell script to Auto Mute you sound devices … ``` ### Uninstalling resources