Skip to content

Commit 9d10087

Browse files
set model_snapshot_path to None to prevent unbound local error (#3336)
* set model_snapshot_path to None to prevent unbound local error * address PR comments with pythonic usage, fix README * small change * revert formatting changes
1 parent 6bdb1ba commit 9d10087

File tree

2 files changed

+5
-4
lines changed

2 files changed

+5
-4
lines changed

README.md

+2-2
Original file line numberDiff line numberDiff line change
@@ -65,10 +65,10 @@ Refer to [torchserve docker](docker/README.md) for details.
6565
#### VLLM Engine
6666
```bash
6767
# Make sure to install torchserve with pip or conda as described above and login with `huggingface-cli login`
68-
python -m ts.llm_launcher --model_id meta-llama/Meta-Llama-3.1-8B-Instruct --disable_token_auth
68+
python -m ts.llm_launcher --model_id meta-llama/Llama-3.2-3B-Instruct --disable_token_auth
6969

7070
# Try it out
71-
curl -X POST -d '{"model":"meta-llama/Meta-Llama-3.1-8B-Instruct", "prompt":"Hello, my name is", "max_tokens": 200}' --header "Content-Type: application/json" "http://localhost:8080/predictions/model/1.0/v1/completions"
71+
curl -X POST -d '{"model":"meta-llama/Llama-3.2-3B-Instruct", "prompt":"Hello, my name is", "max_tokens": 200}' --header "Content-Type: application/json" "http://localhost:8080/predictions/model/1.0/v1/completions"
7272
```
7373

7474
#### TRT-LLM Engine

ts/llm_launcher.py

+3-2
Original file line numberDiff line numberDiff line change
@@ -168,8 +168,9 @@ def main(args):
168168

169169
model_store_path = Path(args.model_store)
170170
model_store_path.mkdir(parents=True, exist_ok=True)
171-
if args.engine == "trt_llm":
172-
model_snapshot_path = download_model(args.model_id)
171+
model_snapshot_path = (
172+
download_model(args.model_id) if args.engine == "trt_llm" else None
173+
)
173174

174175
with create_mar_file(args, model_snapshot_path):
175176
if args.engine == "trt_llm":

0 commit comments

Comments
 (0)