lectures/tdd/Testing_and_design.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4ce9ac47",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Packages, testing & principles of design\n",
    "\n",
    "![exploits](figures/exploits_of_a_mom.png)\n",
    "\n",
    "Timothy E. Holy\n",
    "\n",
    "Washington University in St. Louis"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca4f3fd",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Topics for today\n",
    "\n",
    "- Julia packages\n",
    "- testing, reliability, and design\n",
    "- how to write tests\n",
    "- test coverage"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b446101",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Julia packages\n",
    "\n",
    "Packages are the main way to distribute code. Packages encapsulate the *reusable* portions of code:\n",
    "\n",
    "```julia\n",
    "function compute_something(args...)\n",
    "    # If this function is generally useful, this probably belongs in a package\n",
    "end\n",
    "\n",
    "\n",
    "# this probably belongs in an \"analysis script\" on your local machine\n",
    "data = load(\"/home/me/project1/mydatafile.ext\")      # this path is specific to my computer\n",
    "result = compute_something(data)\n",
    "```\n",
    "\n",
    "You can also store analysis scripts as a repository on GitHub, but they aren't distributed or managed by `Pkg`."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6a83cf2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Finding Julia packages\n",
    "\n",
    "![juliahub](figures/juliahub.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b479ed1",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Structure of Julia packages\n",
    "\n",
    "Packages are `git` repositories combining:\n",
    "\n",
    "- code: `src/`\n",
    "- tests: `test/`\n",
    "- documentation: `docs/`\n",
    "- version metadata: `Project.toml`\n",
    "- \"maintenance\" scripts (e.g., GitHub Actions): `.github/workflows`\n",
    "- (optional) external dependencies required for the package and/or custom build code: [`Artifacts.toml`](https://pkgdocs.julialang.org/v1/artifacts/) and/or `deps/`\n",
    "\n",
    "Strongly recommended: create packages with the [PkgTemplates](https://github.com/JuliaCI/PkgTemplates.jl) package"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28ba0f1a",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Reasons to be concerned about code-quality\n",
    "\n",
    "- science is increasingly reliant on computation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67990f85",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- experimental science has widely agreed-upon standards about positive and negative controls, etc.; there is nothing comparable for computation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5d8e4daa",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- many estimates for error rates are scary: a widely-cited statistic (*Code Complete* by Steve McConnell) gives an industry average of 15-50 bugs per 1000 lines of *delivered* code."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "584d461b",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- it's widely agreed that carefully-designed and tested software can beat this by one or more orders of magnitude"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "167ea8e9",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# One way to help: testing\n",
    "\n",
    "- *static testing*: \"code that analyzes code\" (in Julia: [JET.jl](https://github.com/aviatesk/JET.jl) and [SnoopCompile.jl](https://github.com/timholy/SnoopCompile.jl))\n",
    "- *dynamic testing*: run the code and check the results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5b7bfa82",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "## Testing can do more than just reduce bugs\n",
    "\n",
    "Testing can help encourage development *processes* that results in better outcomes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8e085c6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Testing is under-utilized in most scientific software ecosystems\n",
    "\n",
    "Next lecture will cover **Continuous Integration**; for now, think of this as a systematic way to run tests.\n",
    "Based on whether packages use a continuous integration provider, one can infer the fraction of packages (mostly Python, R, and Matlab) published in the Journal of Open Source Software that have testing:\n",
    "\n",
    "![JOSS](figures/ci-joss.png)\n",
    "\n",
    "\n",
    "Analysis by [Jamie Quinn](http://blog.jamiejquinn.com/analysing-ci-in-joss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "babfc003",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Testing is nearly ubiquitous in the Julia ecosystem\n",
    "\n",
    "![julia-pkg-testing](figures/fraction-tests.png)\n",
    "\n",
    "(Over 95% of packages have Continuous Integration)\n",
    "\n",
    "The ubiquity of testing is probably mostly attributable to the consistent structure of Julia packages & PkgTemplates.\n",
    "\n",
    "This still doesn't guarantee that test coverage is adequate."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34e57bdc",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Challenges to writing good code: smart people\n",
    "\n",
    "*(Acknowedgments to Evan Dorn, https://www.youtube.com/watch?v=HhwElTL-mdI)*\n",
    "\n",
    "Many people who write code are so smart they can juggle a lot in their heads\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "77dbe9a4",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\"Sit down and start typing\" is like an engineer who says \"I don't need plans, just give me the bricks and I'll start construction.\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5f7adcc",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "May work for small projects, but limits you with complex projects"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc53bf86",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "(story time)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6556fcfb",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Advantages of *having* tests\n",
    "\n",
    "- documents what you expect your code to do"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a304b4a",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- validates your initial implementation (to you and others)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ab650b2",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- *it frees you to improve the code later*, largely without fearing that you'll trigger new bugs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76039aba",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "(interactive demo)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "122c0c89",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Test-driven development (TDD)\n",
    "\n",
    "![TDD](figures/TDD.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c344f15",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Advantages of *writing* tests\n",
    "\n",
    "- \"exercise\" your design before going to the trouble to write it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f48d1578",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- testability encourages good design: testable code tends to be \"good code\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e8a20de",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Code that might be hard to test\n",
    "\n",
    "```julia\n",
    "function calculate(A, B, C, x, y, z; option1=nothing, option2=nothing, option3=nothing, option4=nothing...)\n",
    "    \n",
    "    # 100 lines of \"preliminaries\" that set options, convert arguments into standard formats, etc.\n",
    "    \n",
    "    # Now the real computation starts\n",
    "    \n",
    "end\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4858ddb9",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Matlab is particularly bad about encouraging designs like these (one callable function per file, lack of namespaces)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db7a416d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "But Python (& R?) are not free of such problems (one callable *method* per function => many options/function)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10ba0dc4",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Code that might be easier to test\n",
    "\n",
    "```julia\n",
    "# Special cases\n",
    "calculate(A::AbstractVector, x) = ...   # in one dimension, only `A` and `x` are relevant\n",
    "calculate(A::AbstractMatrix, B, C, x::Bool, y::AbstractVector, z::Real) = ...   # options unused if `x::Bool`\n",
    "calculate(A::AbstractMatrix, B, C, x::AbstractVector{Bool}, y::AbstractVector, z::Real; option2=mean(x)/2) = ...\n",
    "\n",
    "# General case\n",
    "function calculate(A::AbstractMatrix, B, C, x, y, z; option1=nothing, option2=nothing, option3=nothing, option4=nothing...)\n",
    "    zz = part_of_calculation(B, C; option2)\n",
    "    return different_part_of_calculation(A, zz; option1)\n",
    "end\n",
    "\n",
    "part_of_calculation(B, C; option2=nothing) = ...\n",
    "\n",
    "different_part_of_calculation(A, x; option1=size(A, 2)) = ...\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a273abfe",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Interaction between testing & development: a sketch\n",
    "\n",
    "Suppose you have a huge directory of data files:\n",
    "\n",
    "```\n",
    "simulation_8-17-21_10000points_random_a=5.hdf5\n",
    "simulation_8-21-21_10000points_random_a=50.hdf5\n",
    "simulation_8-27-21_10000points_grid_a=5.hdf5\n",
    "simulation_8-28-21_10000points_grid_a=50.hdf5\n",
    "simulation_9-13-21_25000points_random_a=7.5_algorithm2.hdf5\n",
    "simulation_10-4-2021_25000points_random_a=37.0_algorithm2.hdf5\n",
    "...\n",
    "```\n",
    "\n",
    "You want to analyze the overall conclusions that can be drawn from these different executions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21b26db9",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "# Pieces of your code\n",
    "- load the data (from one file or many)\n",
    "- perform analysis and synthesis\n",
    "- plot the results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ee8666",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "All in one function: testing is hard (you have to inspect the plots manually)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49a08511",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Modularization\n",
    "\n",
    "Idea: split the analysis & synthesis into separate pieces\n",
    "\n",
    "```julia\n",
    "function analysis1(filename)\n",
    "    ...\n",
    "end\n",
    "\n",
    "function analysis2(filename1, filename2)\n",
    "    ...\n",
    "end\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3082f204",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Modularization fail\n",
    "\n",
    "Scary thought: if I keep the same *data*, and just tweak the filename (how it encodes the parameters), will I get the same result?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d99744c4",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Example: somewhere, your code assumes `a` is an integer, but then you analyze a file like\n",
    "```\n",
    "simulation_9-13-21_25000points_random_a=7.5_algorithm2.hdf5\n",
    "```\n",
    "Will your code treat `a` as if it were 7?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4305716",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Problem with this design: `analysis1` has to be tested with:\n",
    "- all the different kinds of `data` you need to handle\n",
    "- AND all the different ways you might encode the parameters as a string in the filename\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83ee4d72",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Will you notice the mistake just from looking at the plots?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87de0996",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Aphorisms for *better* modularization\n",
    "\n",
    "Single responsibility principle (SRP): code should do \"one thing\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85088f06",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Don't repeat yourself (DRY) (bad design: write parsing code in each of `analysis1`, `analysis2`, etc.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "493557f1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Other aphorisms that interact with testing:\n",
    "- KISS (Keep it simple, stupid!)\n",
    "- YAGNI (You aren’t going to need it) (don't implement it if it's not necessary)\n",
    "- Hide implementation details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b573cbc",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## A better design\n",
    "\n",
    "```julia\n",
    "struct Simulation\n",
    "    date::Date              # from the `Dates` standard library\n",
    "    npoints::Int\n",
    "    initialization::Symbol\n",
    "    a::Float64\n",
    "    version::Int\n",
    "end\n",
    "\n",
    "function Simulation(filename)\n",
    "    # handle the parsing, give user-friendly errors, ...\n",
    "end\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b41925aa",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "```julia\n",
    "sim = Simulation(\"simulation_8-27-21_10000points_grid_a=5.hdf5\")\n",
    "@test month(sim.date) == 8\n",
    "@test sim.a == 5\n",
    "sim = Simulation(\"simulation_8-27-21_10000points_grid_a=7.5.hdf5\")\n",
    "@test sim.a == 7.5\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1c35ccd",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Get the parsing working to the point of confidence, *make commits*, then move on to the next task."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17cc73f0",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "Strict TDD would say write the tests first, and that would be great, but keeping *testability* in mind when you write the code does almost as much good.\n",
    "\n",
    "\"Test-centered mindset\" is more important than any particular workflow."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e6365f5",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Cons and pros of writing tests\n",
    "\n",
    "- writing tests takes time\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a90bbce",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- writing bad code and fixing it takes even longer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df95f6fb",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- sometimes, having a good process for development makes you faster (but it takes practice)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c85dc5e2",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Limits to testing\n",
    "\n",
    "- some things are so hard to test, it may not be worth it\n",
    "- don't test trivial things (\"add 1 to x\")\n",
    "- playing around is OK (but write tests for the parts you keep)\n",
    "- when fixing bugs, sometimes you can't even write a MWE until you verify that you've understood the bug by fixing it\n",
    "\n",
    "Remember, the goal is good code no matter how you get it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea286501",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Test coverage\n",
    "\n",
    "You're thinking about contributing to an external package. Are you \"coding without a net\"?\n",
    "\n",
    "![without a net](figures/without_a_net.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67aec612",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Coverage examples\n",
    "\n",
    "To avoid embarassing anyone, let me pick examples of my own:\n",
    "- encouraging case: https://github.com/timholy/FlameGraphs.jl\n",
    "- discouraging case: https://github.com/timholy/ProfileView.jl\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b44d53e1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "In discouraging cases, an exemplary workflow would be to:\n",
    "1. submit PR(s) that improve coverage\n",
    "2. see whether the maintainer merges them\n",
    "3. once confident you won't break anything, start making the changes you want"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca3305b7",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Special topics (1/2): errors, warnings, logs, & broken tests\n",
    "\n",
    "Homework: read the documentation for Julia's `Test` standard library\n",
    "\n",
    "- `@test_throws`: check that code produces an error\n",
    "- `@test_warn` and `@test_logs`: check that code prints notices for the user\n",
    "- `@test_broken`: your TODO list!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "602c9137",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Special topics (2/2): tests for GUIs\n",
    "\n",
    "Observables (also called reactive variables, signal/slots)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7309d8e3",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "using Observables\n",
    "obs = Observable(3)\n",
    "on(obs) do val\n",
    "    println(\"obs has value $val\")\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c8241b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "obs[] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eff976a6",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Strategy:\n",
    "- user interaction does nothing but update the values of observables\n",
    "- all program code depends only on observables\n",
    "- test your code by having the tests set new values for the observables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "41726093",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Summary\n",
    "\n",
    "- testing is an integral part of modern software development, and essential if you want to produce high-quality code"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4361f174",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- leverage testing as a *systematic process* that you use to help improve the quality of your design"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c43bd72c",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- tools for analyzing test coverage can help you identify & catch potential weaknesses before they bite you"
   ]
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Julia 1.9.0-beta3",
   "language": "julia",
   "name": "julia-1.9"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}