Tests as Policy Automation

Feb 02, 2021 – Filed as: Django6 Software70

Automated tests are usually used for testing functional requirements of your product code. But they can also be used to enforce other policies and coding practices as well.

If writing a web application it’s likely you already have a rule like “all automated tests must pass before any new version of the web application can be deployed to customers on the production environment”. In that case any new policies you want to enforce regularly can be added to your standard automated test suite and they will necessarily have to be satisfied on every deployment!

Here are some example of special policies I’ve enforced from the automated test suite of a large web application I work on (which uses Django, Python+mypy, and JavaScript+TypeScript as core technologies):

  • Ensure the typechecker reports no errors
    • test_type_checker_reports_no_errors_in_python 1
    • test_type_checker_reports_no_errors_in_typed_javascript 2
  • Ban unsafe coding patterns by inspecting source code
    • test_no_new_fragile_test_suites3
    • test_ensure_all_directories_containing_py_files_have_init_file4
  • Ensure hard-coded debug modes are turned off
    • test_the_debug_toolbar_is_disabled 5
    • test_that_compress_is_enabled 6
  • Ensure test-only environmental settings are correctly configured
    • test_that_tests_do_not_send_real_emails
  • Ensure invariants that should apply to all types of a large/unbounded number of domain objects are satisfied:
    • test_all_django_add_and_edit_admin_pages_render 7 👈 especially useful and powerful
    • test_every_block_type_satisfies_all_block_standards 8
      • test_c_blocks_for_python_blocks_must_use_consistent_indent_width
      • test_every_field_id_must_use_underscore_case
      • test_every_block_type_whose_codegen_always_references_x_library_must_import_x_library
      • … (9 more)
  • Prevent certain configuration settings from changing without triggering a discussion with Product Management, the Business, or your Dev Lead
    • test_max_redirect_count_is_5 9

Hopefully these examples give you some ideas of some special policies you might enforce in your own automated test suite. Happy coding!

Related Articles

  • Shitlist Driven Development - Gives techniques for how to effectively apply automated policy changes of the type discussed in this article at large scale.

  1. TechSmart’s backend Python and Django code is typechecked using the mypy type checker.↩

  2. TechSmart’s frontend JavaScript code is typechecked using the TypeScript compiler, tsc.↩

  3. In this context a “fragile test suite” corresponds to a subclass of Django’s StaticLiveServerTestCase or TestCase whose setUpClass method fails to use a try-finally to invoke super().tearDownClass() explicitly if something goes wrong partway through the test suite setup. This fragile-detection metatest walks through all test suite classes and uses Python’s inspect.getsourcelines to read the source code of all test classes to look for the absense of the proper kind of try-finally.↩

  4. It is important for any directory containing Python source files (*.py) to contain an __init__.py file so that the directory is marked properly as a Python package and is recognized correctly by Python typecheckers like mypy.↩

  5. The Django Debug Toolbar is an amazingly useful tool to profile the database queries that your Django-rendered page is making.↩

  6. TechSmart’s frontend JavaScript code is concatentated and minified using the excellent Django Compressor app. At some point we may migrate instead to using Snowpack, a lightweight module bundler.↩

  7. The “all Django Add and Edit pages must render” metatest automatically discovers all Django models and related pages that exist in Django’s admin site. Then it tries to navigate to each such admin page and ensures that it renders completely. This metatest has caught cases where some of our more complex custom admin pages broke due to a code change elsewhere.↩

  8. The “blocks” referred to here are the lego-like blocks which snap together to form programs in the visual Skylark language.↩

  9. The Python IDE in the TechSmart Platform contains an API-compatible reimplementation of the popular Requests HTTP client library so that students can write programs that access the internet, in a controlled fashion.↩