Add corgi script and leashed mode

Author: TylerZeroMaster

README.md

@@ -48,7 +48,40 @@ The full explanation of Enki it is out of scope for this documentation. To learn
 > After installing Docker, navigate to the Docker Desktop GUI preferences and increase the `Memory` value to at least `8GiB`.
 > [Here's where you can find the Docker Desktop GUI settings](https://docs.docker.com/docker-for-windows/#resources)
 
-### Local development
+
+
+## Local development 
+
+**General Workflow**
+1. Start stack with or without GitHub OAuth (see below)
+1. Visit http://localhost/ to see the frontend
+1. Make modifications
+1. Wait for automatic reload
+
+**NOTE**: You might get 502 when visiting http://localhost/ at first: this is normal. Wait a few seconds and try again.
+
+### GitHub OAuth Disabled (Leashed Mode)
+
+To start corgi without GitHub OAuth, in the project root directory, run:
+
+```bash
+./corgi start-leashed
+```
+
+or
+
+```bash
+./corgi start
+```
+
+
+In leashed mode, corgi only uses data that is stored locally.
+
+Currently, that data is stored in `backend/app/tests/unit/data`.
+
+**NOTE:** By default, you will only be able to use `tiny-book` as the repository and `book-slug1` as the book when queuing jobs. This data can be updated: see [Run backend unit tests](#run-backend-unit-tests) for more information.
+
+### GitHub OAuth Enabled
 
 **Prerequisites**
 * You will need access to a GitHub OAuth app. You can [create your own](https://docs.github.com/en/developers/apps/building-oauth-apps/creating-an-oauth-app) if you do not have one already.
@@ -57,9 +90,11 @@ The full explanation of Enki it is out of scope for this documentation. To learn
     2. `GITHUB_OAUTH_SECRET` (your oauth app client secret)
     3. `SESSION_SECRET` (secret used to encrypt session cookies)
 
-Build/start the stack with Docker Compose:
+Build/start the stack with the Corgi script:
 
-    docker-compose -f docker-compose.stack.dev.yml up -d
+```bash
+./corgi start-dev
+```
 
 View the API Docs here:
 
@@ -68,15 +103,20 @@ View the API Docs here:
 
 To check the logs run:
 
-    docker-compose logs backend/frontend/etc.
+```bash
+./corgi logs backend/frontend/etc.
+```
+
+### Functions Patched in Leashed Mode
 
+Leashed mode attempts to patch any `app.github.api` function functions starting with "get_". It looks for the associated mock functions in `backend/app/tests/unit/init_test_data.py`.
+
+### Hot Reloading
 
 Using the development stack, the Svelte frontend is rebuilt inside the container 
 as you make changes: no restarts required. The page should reload automatically
-as well.
-
-NOTE: The dev frontend server can take a while to start up. You might get 502
-responses at first: this is normal. Wait a few seconds and try again.
+as well. The same is true for the backend: as you make modifications, the
+backend server should reload with your latest changes.
 
 
 ### View the Docs
@@ -119,19 +159,25 @@ Note: Can be done in container or outside the container, with installed requirem
 
 To run unit tests:
 
-    cd backend/app
-    poetry run pytest tests/unit
+```bash
+cd backend/app
+poetry run pytest tests/unit
+```
 
-The unit tests use vcr to store response from GitHub. To initialize test data:
+The unit tests use vcr to store response from GitHub. To update the test data:
 
-    cd backend/app
-    poetry run pytest tests/unit --init-test-data --github-token "<token>"
+```bash
+cd backend/app
+poetry run pytest tests/unit --init-test-data --github-token "<token>"
+```
 
 ### Run integration and UI tests 
 
 To run the tests execute:
 
-    ./scripts/tests.ci.sh
+```bash
+./scripts/tests.ci.sh
+```
 
 ### How to develop UI tests
 
@@ -143,7 +189,9 @@ In the [./scripts/tests.ci.local](./scripts/tests.ci.local) file comment out the
 
 In order to view the browser first list all the containers for the docker-stack.yml file:
 
-    $ docker-compose -f docker-stack.yml ps
+```bash
+docker-compose -f docker-stack.yml ps
+```
 
 A table will be displayed with column names. Find the one labeled PORTS for the backend-tests container.
 

backend/app/app/leashed.py

@@ -0,0 +1,63 @@
+import logging
+
+from fastapi.responses import RedirectResponse
+
+
+def get_server():
+    from app.main import server
+
+    return server
+
+
+logger = logging.getLogger("uvicorn")
+
+
+def patch_github_api():
+    import app.github
+    import app.github.api
+    import tests.unit.init_test_data
+
+    github_getters = [s for s in dir(app.github.api) if s.startswith("get_")]
+    mock_map = []
+    for getter in github_getters:
+        mock_function = getattr(
+            tests.unit.init_test_data, f"mock_{getter}", None
+        )
+        if mock_function is None:
+            logger.warn(
+                f"Missing mock function for {getter}. "
+                "This is probably fine, but I thought you should know."
+            )
+        else:
+            mock_map.append((getter, mock_function))
+
+    for module in (app.github.api, app.github):
+        for getter, mock_function in mock_map:
+            setattr(module, getter, mock_function)
+
+
+def disable_auth():
+    import app.api.endpoints.auth
+    import app.core.auth
+
+    class MockOAuth:
+        async def authorize_redirect(self, _, redirect_uri):
+            return RedirectResponse(redirect_uri)
+
+        async def authorize_access_token(self, *_):
+            from uuid import uuid4
+
+            return {"access_token": str(uuid4())}
+
+    async def nop(*_args, **_kwargs):
+        pass
+
+    app.api.endpoints.auth.github_oauth = MockOAuth()
+    app.api.endpoints.auth.sync_user_repositories = nop
+
+
+patch_github_api()
+disable_auth()
+
+# Delay the loading of the server module until after everything is patched
+server = get_server()

backend/app/bin/live-reload.sh

@@ -2,4 +2,7 @@
 
 cd /app/app || exit
 
-uvicorn main:server --host 0.0.0.0 --port 80 --debug
+MODULE_NAME=${MODULE_NAME:-main}
+CALLABLE_NAME=${CALLABLE_NAME:-server}
+
+uvicorn "${MODULE_NAME}:${CALLABLE_NAME}" --host 0.0.0.0 --port 80 --debug

backend/app/bin/prestart-leashed.sh

@@ -0,0 +1,12 @@
+#! /usr/bin/env bash
+
+# Let the DB start
+python ./bin/db_wait.py
+
+# Run migrations
+alembic upgrade head
+
+export MODULE_NAME=leashed
+
+# shellcheck source=SCRIPTDIR/live-reload.sh
+source ./bin/live-reload.sh

backend/backend.dockerfile

@@ -54,7 +54,6 @@ COPY ./docker/start.sh /start.sh
 RUN chmod +x /start.sh
 
 COPY ./docker/gunicorn.conf /gunicorn.conf
-RUN echo "reload = True" >> /gunicorn.conf
 
 COPY ./app /app
 WORKDIR /app

corgi

@@ -0,0 +1,60 @@
+#!/usr/bin/env bash
+
+set -eo pipefail
+
+log() {
+    echo $* >&2
+}
+
+die() {
+    log $*
+    exit 1
+}
+
+usage() {
+    cat <<EOF
+Manage your local corgi stack
+Usage:
+
+    corgi <COMMAND>
+
+The commands are:
+    start               Alias for start-leashed
+    start-dev           Start corgi in dev mode (requires oauth app, see README
+                        for more details)
+    start-leashed       Start corgi in leashed mode (uses mocked GitHub API)
+    logs                See logs (supports same options as docker-compose logs)
+    stop                Stop corgi
+
+EOF
+
+    if [[ -n "$1" ]]; then
+        log $*
+    fi
+    exit 1
+}
+
+COMMAND="$1"
+[[ -n "$COMMAND" ]] || usage
+shift
+
+
+here="$(cd "$(dirname "$0")" && pwd)"
+dev_stack="$here/docker-compose.stack.dev.yml"
+leash_stack="$here/docker-compose.stack.leashed.yml"
+readonly here
+readonly dev_stack
+readonly leash_stack
+
+
+if [[ "$COMMAND" == "start-dev" ]]; then
+    docker-compose -f "$dev_stack" up -d
+elif [[ "$COMMAND" == "start-leashed" || "$COMMAND" == "start" ]]; then
+    docker-compose -f "$dev_stack" -f "$leash_stack" up -d
+elif [[ "$COMMAND" == "logs" ]]; then
+    docker-compose -f "$dev_stack" logs $*
+elif [[ "$COMMAND" == "stop" ]]; then
+    docker-compose -f "$dev_stack" down
+else
+    usage "Unknown command \"$COMMAND\""
+fi

docker-compose.stack.dev.yml

@@ -32,16 +32,13 @@ services:
     ports:
       - '35729:35729'
 
-  backend-tests:
+  backend:
     build:
       context: ./backend
-      dockerfile: tests.dockerfile
+      dockerfile: backend.dockerfile
+      target: dev-runner
     volumes:
-      - /tmp/test-results:/tmp/test-results
       - ./backend/app:/app
-    ports:
-      - '5900'
-    command: bash -c "while true; do sleep 1; done"
     env_file:
       - env-backend.env
     environment: &testenv
@@ -55,32 +52,21 @@ services:
       - TAG=${TAG}
       - STACK_NAME=${STACK_NAME}
       - DEPLOYED_AT=${DEPLOYED_AT}
-  backend:
-    build:
-      context: ./backend
-      dockerfile: backend.dockerfile
-      target: dev-runner
-    volumes:
-      - ./backend/app:/app
-    env_file:
-      - env-backend.env
-    environment: *testenv
     labels:
       - traefik.frontend.rule=PathPrefix:/api,/docs,/redoc,/openapi.json
       - traefik.enable=true
       - traefik.port=80
       - traefik.tags=${TRAEFIK_TAG}
-# Uncomment lines below to create Infinite loop to keep container live doing nothing
-#   backend:
-#     command: bash -c "while true; do sleep 1; done"
     command: ./bin/prestart-dev.sh
     networks:
       default:
         aliases:
           - ${DOMAIN}
+
   db:
     image: postgres:12
     environment: *testenv
+
   docs:
     build:
       context: ./docs

docker-compose.stack.leashed.yml

@@ -0,0 +1,4 @@
+version: '3.7'
+services:
+  backend:
+    command: ./bin/prestart-leashed.sh