[Feature] Add device filter for selecting or excluding a subset of devices

[manhtukhang]

Fix

• Is it Possible To Hide GPUs? #197
• Is it possible to hide GPUs on a per-node basis? #236
• How can I hide a subset of GPUs? #435
• How to exclude some specific GPUs? #935
• Limit devices visible in a Kubernetes node gpu-operator#612 (related project)

Description

Selecting and excluding some specific devices on specific nodes is a common use case when users want to share GPUs for other purposes outside K8S or for testing/debugging.

This PR introduces new flags (and configurations) for the above feature.
By using combine with Per-Node Configuration With a Node Label, it allows more granular control on selecting/excluding devices on some specific nodes

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[manhtukhang]

Hi @elezar @ArangoGutierrez,

We are an Nvidia customer and have the real use case of filtering devices on some specific nodes.
Could you please review this Feature PR?

Many thanks!