merging master

Author: udaij12

.github/workflows/ci-cpu-cpp.yml

@@ -0,0 +1,34 @@
+name: CI CPU - CPP Build
+
+on:
+  push:
+    branches:
+      - master
+  pull_request:
+    branches:
+      - master
+
+concurrency:
+  group: ci-cpu-${{ github.workflow }}-${{ github.ref == 'refs/heads/master' && github.run_number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  ci-cpu-cpp:
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, macOS-latest]
+    steps:
+      - name: Checkout TorchServe
+        uses: actions/checkout@v2
+      - name: Install libtorch - macOS
+        if: matrix.os == 'macOS-latest'
+        run: |
+            brew install libtorch
+      - name: Install dependencies
+        run: |
+          python ts_scripts/install_dependencies.py --environment=dev --cpp
+      - name: Build
+        run: |
+          cd cpp && ./build.sh --install-dependencies

.github/workflows/kserve_cpu_tests.yml

@@ -29,6 +29,10 @@ jobs:
         with:
           python-version: 3.8
           architecture: x64
+      - name: Install grpcurl
+        run: |
+          sudo curl -sSL https://github.com/fullstorydev/grpcurl/releases/download/v1.8.0/grpcurl_1.8.0_linux_x86_64.tar.gz | sudo tar -xz -C /usr/local/bin grpcurl
+          sudo chmod +x /usr/local/bin/grpcurl
       - name: Checkout TorchServe
         uses: actions/checkout@v3
       - name: Checkout kserve repo
@@ -37,5 +41,5 @@ jobs:
           repository: kserve/kserve
           ref: v0.11.1
           path: kserve
-      - name: Validate torchserve-kfs
+      - name: Validate torchserve-kfs and Open Inference Protocol
         run: ./kubernetes/kserve/tests/scripts/test_mnist.sh

.gitignore

@@ -41,3 +41,7 @@ instances.yaml.backup
 ## Models
 *.pt
 *.onnx
+
+# cpp
+cpp/_build
+cpp/third-party

.gitmodules

@@ -1,3 +1,9 @@
 [submodule "third_party/google/rpc"]
 	path = third_party/google/rpc
 	url = https://github.com/googleapis/googleapis.git
+[submodule "cpp/third-party/llama.cpp"]
+	path = cpp/third-party/llama.cpp
+	url = https://github.com/ggerganov/llama.cpp.git
+[submodule "cpp/third-party/llama2.c"]
+	path = cpp/third-party/llama2.c
+	url = https://github.com/karpathy/llama2.c

MANIFEST.in

@@ -1,4 +1,7 @@
 include ts/frontend/model-server.jar
+include ts/cpp/bin/*
+include ts/cpp/lib/*
+include ts/cpp/resources/logging.config
 include PyPiDescription.rst
 include ts/configs/*
 include ts/version.txt

benchmarks/README.md

@@ -2,13 +2,15 @@
 
 The benchmarks measure the performance of TorchServe on various models and benchmarks. It supports either a number of built-in models or a custom model passed in as a path or URL to the .mar file. It also runs various benchmarks using these models (see benchmarks section below). The benchmarks are executed in the user machine through a python3 script in case of jmeter and a shell script in case of apache benchmark. TorchServe is run on the same machine in a docker instance to avoid network latencies. The benchmark must be run from within `serve/benchmarks`
 
-We currently support benchmarking with JMeter & Apache Bench. One can also profile backend code with snakeviz.
+We currently support benchmarking with JMeter, Apache Bench and Locust. One can also profile backend code with snakeviz.
 
 * [Benchmarking with Apache Bench](#benchmarking-with-apache-bench)
 * [Auto Benchmarking with Apache Bench](#auto-benchmarking-with-apache-bench)
 * [Benchmarking and Profiling with JMeter](jmeter.md)
 
-# Benchmarking with Apache Bench
+⚠️**Please Note**⚠️ In future we will deprecate Apache Bench in favor of locust.
+
+# Benchmarking with Locust/Apache Bench
 
 ## Installation
 
@@ -43,6 +45,8 @@ Refer [parameters section](#benchmark-parameters) for more details on configurab
 
 `python benchmark-ab.py`
 
+To switch between Apache Bench and Locust you can set the --benchmark-backend/-bb parameter to either "ab" or "locust".
+
 ### Run benchmark using a config file
 The config parameters can be provided using cmd line args and a config json file as well.
 This command will use all the configuration parameters given in config.json file.