You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardexpand all lines: CODE_OF_CONDUCT.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Copy file name to clipboardexpand all lines: CONTRIBUTING.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Copy file name to clipboardexpand all lines: README.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
# ❗ANNOUNCEMENT: Security Changes❗
2
6
TorchServe now enforces token authorization enabled and model API control disabled by default. These security features are intended to address the concern of unauthorized API calls and to prevent potential malicious code from being introduced to the model server. Refer the following documentation for more information: [Token Authorization](https://github.com/pytorch/serve/blob/master/docs/token_authorization_api.md), [Model API control](https://github.com/pytorch/serve/blob/master/docs/model_api_control.md)
Copy file name to clipboardexpand all lines: SECURITY.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Copy file name to clipboardexpand all lines: benchmarks/README.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
# Torchserve Model Server Benchmarking
2
6
3
7
The benchmarks measure the performance of TorchServe on various models and benchmarks. It supports either a number of built-in models or a custom model passed in as a path or URL to the .mar file. It also runs various benchmarks using these models (see benchmarks section below). The benchmarks are executed in the user machine through a python3 script in case of jmeter and a shell script in case of apache benchmark. TorchServe is run on the same machine in a docker instance to avoid network latencies. The benchmark must be run from within `serve/benchmarks`
Copy file name to clipboardexpand all lines: benchmarks/add_jmeter_test.md
+9-5
Original file line number
Diff line number
Diff line change
@@ -1,16 +1,20 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
## Adding a new test plan for torchserve
2
6
3
7
A new Jmeter test plan for torchserve benchmark can be added as follows:
4
8
5
9
* Assuming you know how to create a jmeter test plan. If not then please use this jmeter [guide](https://jmeter.apache.org/usermanual/build-test-plan.html)
6
10
* Here, we will show you how 'MMS Benchmarking Image Input Model Test Plan' plan can be added.
7
-
This test plan does following:
8
-
11
+
This test plan does following:
12
+
9
13
* Register a model - `default is resnet-18`
10
14
* Scale up to add workers for inference
11
15
* Send Inference request in a loop
12
16
* Unregister a model
13
-
17
+
14
18
(NOTE - This is an existing plan in `serve/benchmarks`)
15
19
* Open jmeter GUI
16
20
e.g. on macOS, type `jmeter` on commandline
@@ -63,7 +67,7 @@ You can create variables or use them directly in your test plan.
63
67
* input_filepath - input image file for prediction
64
68
* min_workers - minimum workers to be launch for serving inference request
65
69
66
-
NOTE -
70
+
NOTE -
67
71
68
72
* In above, screenshot, some variables/input box are partially displayed. You can view details by opening an existing test cases from serve/benchmarks/jmx for details.
69
-
* Apart from above argument, you can define custom arguments specific to you test plan if needed. Refer `benchmark.py` for details
73
+
* Apart from above argument, you can define custom arguments specific to you test plan if needed. Refer `benchmark.py` for details
Copy file name to clipboardexpand all lines: benchmarks/jmeter.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Copy file name to clipboardexpand all lines: benchmarks/sample_report.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Copy file name to clipboardexpand all lines: binaries/README.md
+20-16
Original file line number
Diff line number
Diff line change
@@ -1,4 +1,8 @@
1
-
# Building TorchServe and Torch-Model-Archiver release binaries
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
5
+
# Building TorchServe and Torch-Model-Archiver release binaries
## Uploading packages to production torchserve account
159
163
160
-
As a first step binaries and docker containers need to be available in some staging environment. In that scenario the binaries can just be `wget`'d and then uploaded using the instructions below and the docker staging environment just needs a 1 line code change in https://github.com/pytorch/serve/blob/master/docker/promote-docker.sh#L8
164
+
As a first step binaries and docker containers need to be available in some staging environment. In that scenario the binaries can just be `wget`'d and then uploaded using the instructions below and the docker staging environment just needs a 1 line code change in https://github.com/pytorch/serve/blob/2a0ce756b179677f905c3216b9c8427cd530a129/docker/promote-docker.sh#L8
161
165
162
166
### pypi
163
167
Binaries should show up here: https://pypi.org/project/torchserve/
Binaries should show up here: https://hub.docker.com/r/pytorch/torchserve
184
188
185
-
Change the staging org to your personal docker or test docker account https://github.com/pytorch/serve/blob/master/docker/promote-docker.sh#L8
189
+
Change the staging org to your personal docker or test docker account https://github.com/pytorch/serve/blob/2a0ce756b179677f905c3216b9c8427cd530a129/docker/promote-docker.sh#L8
186
190
187
191
188
192
### Direct upload
@@ -197,7 +201,7 @@ For an official release our tags include `pytorch/torchserve/<version_number>-cp
197
201
## Direct upload Kserve
198
202
To build the Kserve docker image follow instructions from [kubernetes/kserve](../kubernetes/kserve/README.md)
199
203
200
-
When tagging images for an official release make sure to tag with the following format `pytorch/torchserve-kfs/<version_number>-cpu` and `pytorch/torchserve-kfs/<version_number>-gpu`.
204
+
When tagging images for an official release make sure to tag with the following format `pytorch/torchserve-kfs/<version_number>-cpu` and `pytorch/torchserve-kfs/<version_number>-gpu`.
Copy file name to clipboardexpand all lines: binaries/conda/README.md
+5-2
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
# Building conda packages
2
6
3
7
1. To build conda packages you must first produce wheels for the project, see [this readme](../README.md) for more details on building `torchserve` and `torch-model-archiver` wheel files.
@@ -21,4 +25,3 @@ The built conda packages are available in the `output` directory
21
25
Anaconda packages are both OS specific and python version specific so copying them one by one from a test/staging environment like https://anaconda.org/pytorch/torchserve/ to an official environment like https://anaconda.org/torchserve-staging can be fiddly
22
26
23
27
Instead you can run `anaconda copy torchserve-staging/<package>/<version_number> --to-owner pytorch`
Copy file name to clipboardexpand all lines: cpp/README.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Copy file name to clipboardexpand all lines: docker/README.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
## Security Changes
2
6
TorchServe now enforces token authorization enabled and model API control disabled by default. Refer the following documentation for more information: [Token Authorization](https://github.com/pytorch/serve/blob/master/docs/token_authorization_api.md), [Model API control](https://github.com/pytorch/serve/blob/master/docs/model_api_control.md)
Copy file name to clipboardexpand all lines: docs/FAQs.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Copy file name to clipboardexpand all lines: docs/README.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
# ❗ANNOUNCEMENT: Security Changes❗
2
6
TorchServe now enforces token authorization enabled and model API control disabled by default. These security features are intended to address the concern of unauthorized API calls and to prevent potential malicious code from being introduced to the model server. Refer the following documentation for more information: [Token Authorization](https://github.com/pytorch/serve/blob/master/docs/token_authorization_api.md), [Model API control](https://github.com/pytorch/serve/blob/master/docs/model_api_control.md)
Copy file name to clipboardexpand all lines: docs/Troubleshooting.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
## Troubleshooting Guide
2
6
Refer to this section for common issues faced while deploying your Pytorch models using Torchserve and their corresponding troubleshooting steps.
Copy file name to clipboardexpand all lines: docs/batch_inference_with_ts.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Copy file name to clipboardexpand all lines: docs/code_coverage.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
# Code Coverage
2
6
3
7
## To check branch stability run the sanity suite as follows
Copy file name to clipboardexpand all lines: docs/configuration.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
# Advanced configuration
2
6
3
7
The default settings form TorchServe should be sufficient for most use cases. However, if you want to customize TorchServe, the configuration options described in this topic are available.
Copy file name to clipboardexpand all lines: docs/custom_service.md
+9-5
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
Torchserve returns the captum explanations for Image Classification, Text Classification and BERT models. It is achieved by placing the below request:
258
262
`POST /explanations/{model_name}`
259
263
260
-
The explanations are written as a part of the explain_handle method of base handler. The base handler invokes this explain_handle_method. The arguments that are passed to the explain handle methods are the pre-processed data and the raw data. It invokes the get insights function of the custom handler that returns the captum attributions. The user should write his own get_insights functionality to get the explanations
264
+
The explanations are written as a part of the explain_handle method of base handler. The base handler invokes this explain_handle_method. The arguments that are passed to the explain handle methods are the pre-processed data and the raw data. It invokes the get insights function of the custom handler that returns the captum attributions. The user should write his own get_insights functionality to get the explanations
261
265
262
-
For serving a custom handler the captum algorithm should be initialized in the initialize functions of the handler
266
+
For serving a custom handler the captum algorithm should be initialized in the initialize functions of the handler
263
267
264
268
The user can override the explain_handle function in the custom handler.
265
-
The user should define their get_insights method for custom handler to get Captum Attributions.
269
+
The user should define their get_insights method for custom handler to get Captum Attributions.
266
270
267
271
The above ModelHandler class should have the following methods with captum functionality.
268
272
@@ -292,7 +296,7 @@ The above ModelHandler class should have the following methods with captum funct
Copy file name to clipboardexpand all lines: docs/default_handlers.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
# TorchServe default inference handlers
2
6
3
7
TorchServe provides following inference handlers out of box. It's expected that the models consumed by each support batched inference.
Copy file name to clipboardexpand all lines: docs/genai_use_cases.md
+5-1
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
4
+
1
5
# TorchServe GenAI use cases and showcase
2
6
3
7
This document shows interesting usecases with TorchServe for Gen AI deployments.
@@ -8,4 +12,4 @@ In this blog, we show how to deploy a RAG Endpoint using TorchServe, increase th
This Multi-Image Generation Streamlit app is designed to generate multiple images based on a provided text prompt. Instead of using Stable Diffusion directly, this app chains Llama and Stable Diffusion to enhance the image generation process. This multi-image generation use case exemplifies the powerful synergy of cutting-edge AI technologies: TorchServe, OpenVINO, Torch.compile, Meta-Llama, and Stable Diffusion.
15
+
This Multi-Image Generation Streamlit app is designed to generate multiple images based on a provided text prompt. Instead of using Stable Diffusion directly, this app chains Llama and Stable Diffusion to enhance the image generation process. This multi-image generation use case exemplifies the powerful synergy of cutting-edge AI technologies: TorchServe, OpenVINO, Torch.compile, Meta-Llama, and Stable Diffusion.
Copy file name to clipboardexpand all lines: docs/getting_started.md
+4
Original file line number
Diff line number
Diff line change
@@ -1,3 +1,7 @@
1
+
# ⚠️ Notice: Limited Maintenance
2
+
3
+
This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.
0 commit comments