You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: CONTRIBUTING.md
+16-2
Original file line number
Diff line number
Diff line change
@@ -40,7 +40,21 @@ Your contributions will fall into two categories:
40
40
41
41
Once you finish implementing a feature or bug-fix, please send a Pull Request to https://github.com/pytorch/serve.
42
42
43
-
For more non-technical guidance about how to contribute to PyTorch, see the Contributing Guide.
43
+
New features should always be covered by at least one integration test.
44
+
For guidance please have a look at our [current suite of pytest tests](https://github.com/pytorch/serve/tree/master/test/pytest) and orient yourself on a test that covers a similar use case as your new feature.
45
+
A simplified version of an example test can be found in the [mnist template test](https://github.com/pytorch/serve/blob/master/test/pytest/test_mnist_template.py) which shows how to create a mar file on the fly and register it with TorchServe from within a test.
46
+
You can run most tests by simply executing:
47
+
```bash
48
+
pytest test/pytest/test_mnist_template.py
49
+
```
50
+
To have a look at the TorchServe and/or test output add `-s` like this:
51
+
```bash
52
+
pytest -s test/pytest/test_mnist_template.py
53
+
```
54
+
To run only a subset or a single test from a file use `-k` like this:
55
+
```bash
56
+
pytest -k test/pytest/test_mnist_template.py
57
+
```
44
58
45
59
### Install TorchServe for development
46
60
@@ -50,7 +64,7 @@ Ensure that you have `python3` installed, and the user has access to the site-pa
50
64
51
65
Run the following script from the top of the source directory.
52
66
53
-
NOTE: This script force reinstalls`torchserve`, `torch-model-archiver` and `torch-workflow-archiver`if existing installations are found
67
+
NOTE: This script force re-installs`torchserve`, `torch-model-archiver` and `torch-workflow-archiver`if existing installations are found
Copy file name to clipboardexpand all lines: README.md
+3-3
Original file line number
Diff line number
Diff line change
@@ -79,7 +79,7 @@ Refer to [torchserve docker](docker/README.md) for details.
79
79
* Microsoft [DeepSpeed](examples/large_models/deepspeed), [DeepSpeed-Mii](examples/large_models/deepspeed_mii)
80
80
* Hugging Face [Accelerate](examples/large_models/Huggingface_accelerate), [Diffusers](examples/diffusers)
81
81
* Running large models on AWS [Sagemaker](https://docs.aws.amazon.com/sagemaker/latest/dg/large-model-inference-tutorials-torchserve.html) and [Inferentia2](https://pytorch.org/blog/high-performance-llama/)
82
-
* Running [Llama 2 Chatbot locally on Mac](examples/LLM/llama2)
82
+
* Running [Meta Llama Chatbot locally on Mac](examples/LLM/llama)
83
83
* Monitoring using Grafana and [Datadog](https://www.datadoghq.com/blog/ai-integrations/#model-serving-and-deployment-vertex-ai-amazon-sagemaker-torchserve)
84
84
85
85
@@ -90,8 +90,8 @@ Refer to [torchserve docker](docker/README.md) for details.
90
90
91
91
92
92
## 🏆 Highlighted Examples
93
-
*[Serving Llama 2 with TorchServe](examples/LLM/llama2/README.md)
94
-
*[Chatbot with Llama 2 on Mac 🦙💬](examples/LLM/llama2/chat_app)
93
+
*[Serving Meta Llama with TorchServe](examples/LLM/llama/README.md)
94
+
*[Chatbot with Meta Llama on Mac 🦙💬](examples/LLM/llama/chat_app)
95
95
*[🤗 HuggingFace Transformers](examples/Huggingface_Transformers) with a [Better Transformer Integration/ Flash Attention & Xformer Memory Efficient ](examples/Huggingface_Transformers#Speed-up-inference-with-Better-Transformer)
TorchServe does not prevent users from configuring the address to be any value, including the wildcard address `0.0.0.0`.
28
+
TorchServe does not prevent users from configuring the address to be of any value, including the wildcard address `0.0.0.0`.
27
29
Please be aware of the security risks of configuring the address to be `0.0.0.0`, this will give all addresses(including publicly accessible addresses, if any)
28
-
on the host, access to the TorchServer endpoints listening on the ports shown above.
29
-
2. TorchServe's Docker image is configured to expose the ports `8080`, `8081`, `8082`, `7070`, `7071` to the host by [default](https://github.com/pytorch/serve/blob/master/docker/Dockerfile). When starting the container,
30
-
make sure to map the ports exposed by the container to `localhost` ports or a specific IP address as shown in this [security guideline](https://github.com/pytorch/serve/blob/master/docker/README.md#security-guideline).
30
+
on the host, access to the TorchServe endpoints listening on the ports shown above.
31
+
2.By [default](https://github.com/pytorch/serve/blob/master/docker/Dockerfile), TorchServe's Docker image is configured to expose the ports `8080`, `8081`, `8082`, `7070`, `7071` to the host. When starting the container,
32
+
map the ports exposed by the container to `localhost` ports or a specific IP address, as shown in this [security guideline](https://github.com/pytorch/serve/blob/master/docker/README.md#security-guideline).
31
33
32
34
3. Be sure to validate the authenticity of the `.mar` file being used with TorchServe.
33
35
1. A `.mar` file being downloaded from the internet from an untrustworthy source may have malicious code, compromising the integrity of your application.
34
-
2. TorchServe executes arbitrary python code packaged in the `mar` file. Make sure that you've either audited that the code you're using is safe and/or is from a source that you trust.
35
-
3.Torchserve supports custom [plugins](https://github.com/pytorch/serve/tree/master/plugins) and [handlers](https://github.com/pytorch/serve/blob/master/docs/custom_service.md).
36
+
2. TorchServe executes the arbitrary python code packaged in the `mar` file. Make sure that you've either audited that the code you're using is safe and/or is from a source that you trust.
37
+
3.TorchServe supports custom [plugins](https://github.com/pytorch/serve/tree/master/plugins) and [handlers](https://github.com/pytorch/serve/blob/master/docs/custom_service.md).
36
38
These can be utilized to extend TorchServe functionality to perform runtime security scanning using tools such as:
4. Running Torchserve inside a container environment and loading an untrusted `.mar` file does not guarantee isolation from a security perspective.
41
-
4. By default TorchServe allows you to register models from all URLs. Make sure to set `allowed_urls` parameter in config.properties to restrict this. You can find more details in the [configuration guide](https://pytorch.org/serve/configuration.html#other-properties).
42
+
4. Running TorchServe inside a container environment and loading an untrusted `.mar` file does not guarantee isolation from a security perspective.
43
+
4. By default, TorchServe allows you to register models from all URLs. Make sure to set `allowed_urls` parameter in config.properties to restrict this. You can find more details in the [configuration guide](https://pytorch.org/serve/configuration.html#other-properties).
42
44
-`use_env_allowed_urls=true` is required in config.properties to read `allowed_urls` from environment variable.
43
45
5. Enable SSL:
44
46
@@ -57,9 +59,6 @@ TorchServe as much as possible relies on automated tools to do security scanning
57
59
7. If you intend to run multiple models in parallel with shared memory, it is your responsibility to ensure the models do not interact or access each other's data. The primary areas of concern are tenant isolation, resource allocation, model sharing and hardware attacks.
58
60
8. TorchServe supports token authorization: check [documentation](https://github.com/pytorch/serve/blob/master/docs/token_authorization_api.md) for more information.
59
61
60
-
61
-
62
-
63
62
## Reporting a Vulnerability
64
63
65
-
If you find a serious vulnerability please report it to https://www.facebook.com/whitehat and [email protected]
64
+
If you find a vulnerability please report it to https://www.facebook.com/whitehat and [email protected]
### Configure TorchServe gRPC listening addresses, ports and max connection age
97
97
The inference gRPC API is listening on port 7070, and the management gRPC API is listening on port 7071 on localhost by default.
98
98
99
99
To configure different addresses use following properties
@@ -106,7 +106,15 @@ To configure different ports use following properties
106
106
*`grpc_inference_port`: Inference gRPC API binding port. Default: 7070
107
107
*`grpc_management_port`: management gRPC API binding port. Default: 7071
108
108
109
-
Here are a couple of examples:
109
+
To configure [max connection age](https://grpc.github.io/grpc-java/javadoc/io/grpc/netty/NettyServerBuilder.html#maxConnectionAge(long,java.util.concurrent.TimeUnit)) (milliseconds)
110
+
111
+
*`grpc_inference_max_connection_age_ms`: Inference gRPC max connection age. Default: Infinite
112
+
*`grpc_management_max_connection_age_ms`: Management gRPC max connection age. Default: Infinite
113
+
114
+
To configure [max connection age grace](https://grpc.github.io/grpc-java/javadoc/io/grpc/netty/NettyServerBuilder.html#maxConnectionAgeGrace(long,java.util.concurrent.TimeUnit)) (milliseconds)
115
+
116
+
*`grpc_inference_max_connection_age_grace_ms`: Inference gRPC max connection age grace. Default: Infinite
117
+
*`grpc_management_max_connection_age_grace_ms`: Management gRPC max connection age grace. Default: Infinite
0 commit comments