DaFoster (Software)

AI Attribution in Git

2026-04-17T00:00:00+00:00

When making a git commit I’ve been looking for a way to record what AI model or coding agent harness I used to help me write the commit, ideally in a machine-readable way.

For AI-drafted code that I’ve reviewed or self-drafted code with significant AI revisions, I add one of the following trailers to my git commit message:

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
Co-authored-by: GPT-5.3-Codex <noreply@openai.com>
Co-authored-by: Google Gemini 3.1 Flash Image (Nano Banana 2)
- TODO: Identify an email address to use for Gemini

Tip: I use text expansion macros like coa → "Co-authored-by: " and aco → "Claude Opus 4.6 <noreply@anthropic.com>" to type those trailers quickly.

For AI-drafted code that I’ve NOT reviewed (rare; mostly for internal or one-off tools):

Author = one of the emails above, via git commit --author="AI Model <noreply@example.com>".
Co-authored-by = me:
- Co-authored-by: David Foster <david@dafoster.net>

If I ever want to make such unreviewed or minimally-reviewed AI-drafted code public, I can use git blame to easily identify which lines are principally AI-drafted, so that I can go back and review them carefully; I never publish AI-written code that I haven’t reviewed myself.

Tip: I use the Git Spotlight tool to quickly see which lines of code are written by AI (C4 = Claude Opus 4.6) or by me (DF = David Foster). See below:

Why the Co-authored-by pattern?

Claude Code introduced the convention of adding a Co-Authored-By: Claude <noreply@anthropic.com> trailer to any AI-authored commits by telling the AI to use such trailers in its system prompt, much to the annoyance of some users. GitHub recognizes that Co-Authored-By trailer and displays Claude as a co-author of the related commit.

I think that kind of attribution is lightweight and makes sense.

Oops, I wrote a database

2026-02-23T00:00:00+00:00

Never write your own database or filesystem. It’s way harder than you expect.
— some wise programmer¹

Databases and filesystems are hard to write in part because they have very strong requirements around never losing data (durability) even when unusual events happen such as unexpected process termination, hardware failure, out of disk space, or out of memory. I realized recently that Crystal - a program of mine for downloading websites - should strive to provide the same high reliability guarantees when saving websites to disk in its project format. And so began my ~3-week quest to shore up the ACID (atomicity, consistency, isolation, and durability) properties of how Crystal writes to its projects. Here are a few things I learned along the way.

Background

A Crystal project is an ACID-compliant storage system for holding downloaded website data: multiple-reader, single-writer.

It contains a SQLite table of downloaded page revisions, each row of which corresponds to an on-disk file with the revision’s content.

Adding a revision requires creating both a database row and file at the same time.

Strategy: Write file then rename to commit, rather than (re)write in place

It’s not safe to write a file directly to its final location because sudden process termination or disk disconnection could abort the write in the middle, leaving an incompletely written file in its final “committed” location.

Instead it’s safer to write a file first to a temporary location on the same filesystem as its committed location. After the write is complete, move it to its final place.

If sudden process termination stops the write, when Crystal reopens the project it will delete the temporary file. Additionally Crystal will check whether the highest numbered revision row in the database is missing its corresponding revision file, and delete the incomplete revision row if so.

Strategy: Write file then replace to commit, if the file may have concurrent readers

Crystal projects using the Pack16 format have a more complex challenge: Each revision row now corresponds to an entry in a zip file that can be shared by up to 16 revisions. Thus writing one revision may require altering a zip file containing a different revision that is being concurrently read.

Similar to the write+rename strategy, we can use a write+replace strategy: Write a new revision to a new zip file and then replace the old zip file. Existing readers of the old zip file will not be disrupted; they will still be able to read from the copy of the zip file that they opened even though it is no longer visible in the filesystem.

Windows-only: Move-aside a file before replacing it

On macOS and Linux the above strategy just works; it is possible to replace a file that is open for reading with a different file without doing anything special. However on Windows files are opened in an exclusive mode by default that prevents reading or writing the file while it is still open. You must explicitly open a file with FILE_SHARE_READ, FILE_SHARE_WRITE, or FILE_SHARE_DELETE modes on Windows if you want to allow concurrent readers, writers, or deleters. Crystal defines its own open_nonexclusive function to get this same non-exclusive open behavior across all operating systems.

However Windows presents another challenge: It does not allow you to replace a file that is opened for reading even if it is opened with all of FILE_SHARE_READ, FILE_SHARE_WRITE, and FILE_SHARE_DELETE, despite allowing you to delete it or rename it.

So on Windows, Crystal renames an old zip file to a transitional (but crash-durable) “moved-aside” location before renaming the new zip file to the original location of the old zip file. So the total process looks like:

Write new zip file to temporary location
Move aside the old zip file to a moved-aside location
- This is permitted even if the old zip file has readers, when opened with FILE_SHARE_WRITE
Rename the new zip file to its permanent location, where the old zip file was
Delete the old zip file in its moved-aside location
- This is permitted even if the old zip file has readers, when opened with FILE_SHARE_DELETE

The above (non-atomic) process introduces several complexities:

A. If a new concurrent reader appears between steps 2 and 3, when the old zip file has been moved-aside but not yet replaced with a new zip file, the reader must know to look for the old zip file at the moved-aside location in addition to its normal location.
B1. If sudden process termination happens between steps 2 and 3, some kind of repair process after reopening the project must move the old zip file from its moved-aside location back to its normal location.
B2. If sudden process termination happens between steps 3 and 4, the old zip file in the moved-aside location should be deleted.

In case B1, Crystal performs the repair at read time, when an entry from the zip file is first read after reopening the project.

Read Repair: A time to clean up incomplete operations

When Crystal performs a read on a Pack16 project, looks for a zip file corresponding to a revision, and does not find it, it will then look for a moved-aside zip file that was left behind during scenario B1 above. If it finds such a moved-aside zip file it will repair it by moving it back to its normal location.

Note that this kind of repair is itself a kind of write which might happen during a high-level read operation. Since Crystal allows concurrent reads, the possibility of a repair happening during a read means that concurrent writes are now possible too… Yikes.

Concurrent writes to the same file are unsafe by default but can be made safe with mutual exclusion. Crystal maintains an in-memory mutex lock for each file it is in the middle of repairing to prevent concurrent writes from corrupting the project.

Zooming Out: The Big Picture

I expected this work to take 1–1.5 weeks. It took about three. Most of the overrun came from a cascade of complications I didn’t see coming:

Windows opens files exclusively by default, so even allowing concurrent readers required FILE_SHARE_READ.
Windows then refused to replace a file that was still open for reading — even with all three share modes set — so writes required the move-aside dance and FILE_SHARE_WRITE / FILE_SHARE_DELETE on the reader side.
The move-aside dance introduced two new crash-recovery cases (B1 and B2), which became read-repair.
Read-repair quietly turned reads into potential writes, which forced in-memory per-revision locks to keep the multiple-reader/single-writer contract honest.

Each layer was reasonable on its own; the surprise was how much each one cost in aggregate.

Of the four ACID properties, atomicity and durability drove essentially all of this work. Consistency was largely a freebie given the architecture: the database lives in one place, only one process ever opens a project, and that process has at most one writer. Isolation was similarly cheap given the single-writer rule and the absence of in-place rewrites — with the wrinkle that read-repair turned a “read” into a {read + limited write}, requiring introducing per-revision locks.

It was a fun exercise in polishing one aspect of Crystal to a high standard. But I must say the caution around hand-rolling your own storage code is well earned: you may very well double or triple the time you expect it to take. 🙂

I can’t find a specific original person who said “Never write your own database or filesystem”. The closest discussion I was able to locate comes from Never Write Your Own Database. So perhaps this specific quote just comes from me now 🙂↩

Issue counts always go up

2025-08-20T00:00:00+00:00

I’ve noticed that the GitHub issue count on my software project Crystal always seems to go up, even though I’m the only one who ever files issues (for now) and even though I’m actually completing issues occasionally.

I think I’ve figured out why: Often when I complete an issue, I only complete the first 80% of the issue which is most valuable to me and file smaller issues for the remaining 20% that is left. So I end up with 1 big issue completed and 2+ smaller issues created. Net effect is that more issues are opened than closed even though the missing functionality goes down.

It’s like a game of asteroids, where blowing up the big asteroid just creates a bunch of smaller asteroids.

So I can’t easily see progress by watching the issue count go down. Instead I need to think of a way to measure the amount of missing functionality so that I can see that go down instead and feel good about completing issues. 🙂

Designing Software in the Large

2025-07-22T00:00:00+00:00

A Philosophy of Software Design is my favorite book I’ve read to date about designing large long-lived maintainable software programs. Here’s what I learned:

Complexity

Complexity is anything related to the structure of a software system that makes it hard to understand & modify the system.

Symptoms of complexity:
- Change Amplification - A seemingly simple change requires code modifications in many different places.
- High Cognitive Load - High amount of information a developer needs to know to complete a task.
- Unknown Unknowns - When it is not obvious which pieces of code must be modified to complete a task, or what information a developer must have to carry out the task successfully.
Causes of complexity:
- Dependencies - A dependency exists when a given piece of code cannot be understood and modified in isolation; the code relates in some way to other code.
- Obscurity - Obscurity occurs when important information is not obvious.

To keep a software system maintainable, you must strive to keep the complexity of the system low as you & others make changes to it.

The big picture

Concepts related to complexity, which the remaining sections of this article will zoom in on:

Dependencies: A Cause of Complexity

A dependency exists when a given piece of code cannot be understood in isolation; the code relates in some way to other code. Numerous and strong dependencies between modules of a system make it difficult to change one module without changing others (Change Amplification) and make it difficult to understand modules in isolation (High Cognitive Load).

Key contributors to dependency-complexity are:

Duplication - Duplication occurs when the same knowledge is used in multiple places. This can be direct repetition of the same piece of code appearing in multiple places. Or it can be more subtle, such as multiple classes understanding the details of the same file format or network protocol.
Exceptions - An exception is a particularly complex element of an interface which can propagate not just from a method to its direct caller but also to higher level callers (and their interfaces).
Inheritance - Implementation inheritance in object-oriented systems creates dependencies between the parent class and each of its subclasses.
Temporal Decomposition - A system exhibits temporal decomposition when its structure corresponds to the time order in which operations will occur.

The opposite of dependency-complexity is cohesive code, created primarily by focusing on designing deep modules.

Deep modules allow a lot of functionality to be accessed through a small interface:

Keeping module interfaces small lowers the number and strength of dependencies between modules, resulting in lower overall system complexity:

Chapters 4-9 of the book are related to techniques for forming deeper modules. My article How to Design Large Programs with Abstraction and Encapsulation also discusses deep modules in the context of encapsulation & abstraction.

Obscurity: A Cause of Complexity

Obscurity occurs when important information is not obvious.

Obscurity creates Unknown Unknowns and contributes to High Cognitive Load:

Key contributors to obscurity:

Vague Names
Inconsistency
Inadequate Documentation.

Smaller contributors to obscurity:

indirection via listeners
indirection via polymorphism
using generic containers to hold structured data

The opposite of obscurity is obvious code. Obvious code

uses precise names,
is consistent,
is well-documented, and
makes good use of whitespace.

There are many techniques for creating consistency in code:

Consistent use of names, consistent code style, and consistent implementation patterns are hallmarks of consistency in general.
Tools such as autoformatters, linters, and automated consistency tests are great for enforcing consistency.
Documentation of conventions can be captured in coding style guides and through references to documented design patterns.

Strategic vs. Tactical Mindset

Complexity accumulates naturally in a software system if left unchecked.

And once complexity has accumulated it is hard to eliminate.

Therefore the book recommends taking a “zero-tolerance” stance toward the incremental introduction of complexity.

Specifically, it advocates taking a strategic approach to programming tasks, where you intentionally invest time to produce clean designs and fix problems, in addition to getting new features working.

Many programmers, by contrast, take a purely tactical approach that focuses only on getting new features working, disregarding the long-term costs of adding incremental complexity and other forms of cutting corners.

Takeaway

Working code (alone) is not enough.

To keep a software system maintainable, you must strive to keep the complexity of the system low as you & others make changes to it.

The trouble with the Symbol type

2025-03-16T00:00:00+00:00

There are a few languages that offer a symbol type, notably JavaScript, Ruby, Lisp, and Erlang. A symbol is similar to a string but guarantees that only one copy of the symbol exists with the same value. In JavaScript you can create a symbol with the code:

Symbol.for('hello')
// Symbol(hello)

Symbols can be compared with each other for equality very quickly because only the object references need to be compared; the values do not need to be compared. This high speed of comparison is the main advantage that symbols offer over strings.

However:

Every symbol that is created must be added to a global registry of all symbols to ensure that it is unique. Since all symbols live in this global registry and there is no way to remove a symbol from the registry, a symbol may never be garbage collected.
Because symbols live forever, it is unwise to create a symbol from an arbitrary (unknown) string, because doing so repeatedly will fill up memory with immortal symbol objects.
Similarly it is unwise to deserialize a symbol from (untrusted) input because - again - doing so repeatedly will fill up memory with immortal symbol objects.

To avoid the above problems I generally prefer using strings instead of symbols. Strings can be freely serialized/deserialized safely and they can be garbage collected. Although strings cannot be compared with each other as fast as symbols can, they still compare very fast, especially for short strings.

Related: Interned Strings

Some languages don’t have a symbol type but do have a string type that can be “interned”, notably Java and Python. When a string is interned it is added to the global registry of all interned strings (if it wasn’t there already) and the copy from the registry is returned.

Java’s interned strings are immortal and are never garbage collected.

Python’s interned strings are NOT immortal and will be garbage collected if no references to an interned string exist outside of the global registry.

Dumping a traceback when an error message is printed

2025-03-13T00:00:00+00:00

When a program I’m debugging prints something unexpected like an error message, I can usually search for a fragment of the message in the program’s code to figure out why it was printed.

However sometimes I’m not able to locate the error message in the code at all! In that case one big hammer I have for isolating the offending code is to alter print(), sys.stdout.write(), and sys.stderr.write() to dump a traceback whenever a fragment of the error message is about to be printed:

altair (https://github.com/vega/altair)
+ <type_comment>:1: SyntaxWarning: invalid escape sequence '\('
+ 
+ *** Keyword 'SyntaxWarning' printed at:
+   File "/new_mypy/bin/mypy", line 10, in <module>
+     sys.exit(console_entry())
+   File "/new_mypy/mypy/__main__.py", line 15, in console_entry
+     main()
+   File "/new_mypy/mypy/main.py", line 156, in main
+     res, messages, blockers = run_build(sources, options, fscache, t0, stdout, stderr)
+   File "/new_mypy/mypy/main.py", line 240, in run_build
+     res = build.build(sources, options, None, flush_errors, fscache, stdout, stderr)
+   File "/new_mypy/mypy/build.py", line 191, in build
+     result = _build(
+   File "/new_mypy/mypy/build.py", line 267, in _build
+     graph = dispatch(sources, manager, stdout)
+   File "/new_mypy/mypy/build.py", line 2937, in dispatch
+     process_graph(graph, manager)
+   File "/new_mypy/mypy/build.py", line 3335, in process_graph
+     process_stale_scc(graph, scc, manager)
+   File "/new_mypy/mypy/build.py", line 3430, in process_stale_scc
+     mypy.semanal_main.semantic_analysis_for_scc(graph, scc, manager.errors)
+   File "/new_mypy/mypy/semanal_main.py", line 94, in semantic_analysis_for_scc
+     process_functions(graph, scc, patches)
+   File "/new_mypy/mypy/semanal_main.py", line 252, in process_functions
+     process_top_level_function(
+   File "/new_mypy/mypy/semanal_main.py", line 291, in process_top_level_function
+     deferred, incomplete, progress = semantic_analyze_target(
+   File "/new_mypy/mypy/semanal_main.py", line 351, in semantic_analyze_target
+     analyzer.refresh_partial(
+   File "/new_mypy/mypy/semanal.py", line 668, in refresh_partial
+     self.accept(node)
+   File "/new_mypy/mypy/semanal.py", line 7320, in accept
+     node.accept(self)
+   File "/new_mypy/mypy/nodes.py", line 818, in accept
+     return visitor.visit_func_def(self)
+   File "/new_mypy/mypy/semanal.py", line 926, in visit_func_def
+     self.analyze_func_def(defn)
+   File "/new_mypy/mypy/semanal.py", line 1007, in analyze_func_def
+     self.analyze_function_body(defn)
+   File "/new_mypy/mypy/semanal.py", line 1601, in analyze_function_body
+     defn.body.accept(self)
+   File "/new_mypy/mypy/nodes.py", line 1285, in accept
+     return visitor.visit_block(self)
+   File "/new_mypy/mypy/semanal.py", line 5272, in visit_block
+     self.accept(s)
+   File "/new_mypy/mypy/semanal.py", line 7320, in accept
+     node.accept(self)
+   File "/new_mypy/mypy/nodes.py", line 1641, in accept
+     return visitor.visit_with_stmt(self)
+   File "/new_mypy/mypy/semanal.py", line 5424, in visit_with_stmt
+     e.accept(self)
+   File "/new_mypy/mypy/nodes.py", line 1997, in accept
+     return visitor.visit_call_expr(self)
+   File "/new_mypy/mypy/semanal.py", line 5837, in visit_call_expr
+     self.try_parse_as_type_expression(a)
+   File "/new_mypy/mypy/semanal.py", line 7719, in try_parse_as_type_expression
+     t = self.expr_to_analyzed_type(maybe_type_expr)
+   File "/new_mypy/mypy/semanal.py", line 7355, in expr_to_analyzed_type
+     typ = self.expr_to_unanalyzed_type(expr)
+   File "/new_mypy/mypy/semanal.py", line 7418, in expr_to_unanalyzed_type
+     return expr_to_unanalyzed_type(
+   File "/new_mypy/mypy/exprtotype.py", line 202, in expr_to_unanalyzed_type
+     return parse_type_string(expr.value, "builtins.str", expr.line, expr.column)
+   File "/new_mypy/mypy/fastparse.py", line 327, in parse_type_string
+     _, node = parse_type_comment(f"({expr_string})", line=line, column=column, errors=None)
+   File "/new_mypy/mypy/fastparse.py", line 288, in parse_type_comment
+     typ = ast3_parse(type_comment, "<type_comment>", "eval")
+   File "/new_mypy/mypy/fastparse.py", line 142, in ast3_parse
+     return ast3.parse(
+   File "/usr/lib/python3.12/ast.py", line 52, in parse
+     return compile(source, filename, mode, flags,
+   File "/usr/lib/python3.12/warnings.py", line 113, in _showwarnmsg
+     _showwarnmsg_impl(msg)
+   File "/usr/lib/python3.12/warnings.py", line 30, in _showwarnmsg_impl
+     file.write(text)
+   File "/new_mypy/mypy/main.py", line 98, in write
+     traceback.print_stack(file=self.wrapped)

Notice that the dumped traceback points directly at the offending code.

Here’s Python code which monkeypatches prints to dump a traceback whenever a particular substring is printed:

import sys
import traceback

def print_traceback_whenever_printed_line_contains(needle: str) -> None:
    """
    Replaces sys.stdout with a wrapper that inspects each write. If the output
    contains 'needle', a traceback is printed to the original sys.stdout.
    The monkeypatch remains in place after this function returns.
    """
    class PatchedStdStream:
        def __init__(self, wrapped):
            self.wrapped = wrapped

        def write(self, text: str) -> None:
            # Write text to the original stdout
            self.wrapped.write(text)

            # Check if 'needle' is in the text being written
            if needle in text:
                # Print a traceback to the original stdout
                self.wrapped.write(f'\n*** Keyword {needle!r} printed at:\n')
                traceback.print_stack(file=self.wrapped)

        def __getattr__(self, name: str):
            """
            Forward all other attributes (e.g., flush, fileno, isatty, etc.)
            to the original stdout.
            """
            return getattr(self.wrapped, name)

    sys.stdout = PatchedStdStream(sys.stdout)
    sys.stderr = PatchedStdStream(sys.stderr)

if __name__ == "__main__":
    # Activate the patch to watch for "ERROR" in any output
    print_traceback_whenever_printed_line_contains("ERROR")

    print("Everything is normal here...")
    print("Some random text with ERROR inside it!")
    print("Continuing after error message...")

Enjoy!

Effective Code Reviews

2024-08-18T00:00:00+00:00

Software engineering research has identified¹ some techniques and rules of thumb for making code reviews more effective at identifying defects:

Limit code reviews to 1 hour or less.
- After 60-90 minutes into a code review the ability to find further defects drops off dramatically due to focus fatigue.
- If a code review in progress is dragging on past 1 hour, consider taking a break and continuing the remainder later in a “part 2”.
Limit code reviews to 400 lines of code or less.
- Code reviews performed faster than 400-500 lines of code per hour show a dramatic drop in density of detected defects.
- Combining that result with the earlier result that reviews lasting longer than 1 hour are problematic, we can conclude that no more than 400 lines of code should be reviewed in the same session.
Identify the most complex/risky/controversial changes to review first when there is the least focus fatigue.²
Link to related “context” (i.e. files, documentation, classes, etc) alongside the immediate changes of a code review, for a 15-30% increase in defect detection.
Self-review changes before posting them for external review, because 50% of defects can be found in a self-review at half the time cost of an external review.
An individual code review should be either asynchronous or synchronous but not both.
- For a code review posted asynchronously, there is little gain in following up with an additional full synchronous review: Only an additional 4% of defects are likely to be found.

All results in this article originate from the “Modern Code Review” chapter of “Making Software: What Really Works, and Why We Believe It” by Jason Cohen, unless otherwise specified.↩
This result comes from another source which I have since forgotten.↩

PyCon US 2024 Highlights

2024-06-16T00:00:00+00:00

I was happy to be able to attend PyCon US 2024 this past year, a prominent conference for the Python programming language. I got to meet in person a number of folks I’ve interacted with online, from the Python Typing community and elsewhere.

Favorite Talks

I particularly liked the following talks, which I’ve organized by topic.

Most video links below are to the Hublio platform, which only conference attendees can access. Unfortunately the talk recordings have not yet been shared more widely on YouTube.

Performance

Unlocking the Parallel Universe: Subinterpreters and Free-Threading in Python 3.13 - Anthony Shaw
- Shows many ways to run Python programs in parallel: in different processes, different interpreters, and different threads. There are tradeoffs with each strategy.
- Insight: “Subinterpreters will be a better solution than Multiprocessing for long-running workers.”
- Insight: Free threading is a good solution for workers that need the fastest data exchange between workers. But free threading is difficult to program correctly, without race conditions.
- 📽 Video (Hublio)
Overcoming GIL with subinterpreters and immutability - Yury Selivanov
- Introduces memhive, a new library for safely sharing state across multiple Python interpreters in the same process, by providing an immutable/persistent map structure (HAMT) that can be efficiently updated and shared between interpreters
- Insight: Subinterpreters are a better solution than Multiprocessing for workers that need to share state - such as a cache - between workers, because subinterpreter workers share memory. Subinterpreters are easier to program than free threading.
- 📽 Video (Hublio)

Debugging

Profiling at the speed of light - Pablo Galindo Salgado
- Showed how to use the perf tool on a Python 3.12 and 3.13 process to look at low-level performance metrics like CPU-level and instruction-level counters.
- Showed how to make flame graphs to see where a mixed Python+C program is spending its time visually.
- 📽 Video (Hublio)
Rewind: Python Time-Travel Debugger - Toby Ho
- Showed pyrewind, a time-travel reverse-debugger for Python. Looks like a promising tool for pure Python code.
- It’s unclear whether the tool supports debugging in the presence of C extensions, which may have side effects that the debugger does not track.
- 📽 Video (Hublio)

API Design

The Design of Everyday APIs - Lynn Root
- Outlined several principles for improving the design of APIs.
- Gives “before” and “after” examples when applying these principles to an example API.
- 📽 Video (Hublio)
- 🎞 Slides

AI

Keynote - Simon Willison
- Great talk about AI and Machine Learning. Many interesting quotes from this presentation.
- “You shouldn’t need a CS degree to automate stuff with a computer”
- 📽 Video (Hublio)
Lies, damned lies and large language models - Jodie Burchell
- Describes different types of hallucinations that Large Language Models (LLMs) can experience.
- Shows how to measure how much a particular model hallucinates.
- 📽 Video (Hublio)

Typing System

Theory of Type Hints, Revisited - Carl Meyer
- Covers in-progress efforts to increase the formality of the underlying theory behind Python’s type system, especially with respect to what the Any type means. These efforts have been partially motivated by ongoing work (for over a year) to attempt to introduce an Intersection type into Python’s typing system.
- 📽 Video pending
TypeForm: Type Hint for Type Expressions - David Foster
- Covers TypeForm, an in-progress Python type system feature for spelling a “type expression object” at runtime. This was my own talk. More details in PEP 747.
- 📽 Video (YouTube)

Fun

Build in-browser 3D experiences with WebGL and PyScript - Łukasz Langa
- Showed how to use Python to draw 3D graphics in the browser with WebGL.
- Showed how to take data from a data processing library in Python and directly visualize it by talking to JavaScript visualization libraries.
- 📽 Video (Hublio)
Connecting Old to New with CircuitPython: Retrocomputer input devices on modern PCs - Jeff Epler
- Shows how to use Python to build connectors to old keyboards, mice, and other hardware devices.
- 📽 Video (Hublio)

Redis relicensing: Why is this a problem?

2024-03-24T00:00:00+00:00

I find the fire & heat around Redis changing licensing surprising. Maintaining Redis takes effort which cannot be free in a sustainable fashion.

Consider this post which complains Redis isn’t “uphold[ing] the ideals of Free and Open Source Software”, as if it was created as a vehicle to evangelize the FSF and OSI missions. But it was not, it was created to be useful, as a data structure server.

I also disagree with that post’s assertion that Redis contributions would have been withheld if it started with a different license. I expect contributions would come from existing users of Redis who wanted new features themselves and were willing to sponsor the related effort, regardless of the license in use.

Certainly if Redis has started under its new license and I was using Redis on Azure - which Microsoft has already bought a commercial license for - then I wouldn’t care that Redis was licensed under the SSPL.

Frankly I think it would be a win for software sustainability if there was a trend of new software projects being offered under source-available licenses requiring large cloud providers like AWS, Azure, and GCP to give back financially to the core maintainers.

I plan to be announcing a project of my own soon under a source-available license restricting unlicensed commercial use. I’ve already put several years of effort into this project. No way in hell do I want to “get Jeff'ed”, with someone else selling what I’ve put so much of my own effort into creating without giving anything back.

Bulkheads: A pattern for handling unexpected errors in software

2024-02-28T00:00:00+00:00

Some seafaring ships divide their body into multiple watertight compartments, so that if one compartment becomes flooded the rest of the ship will remain floodfree and intact. I got the idea of using a similar pattern of “bulkheads” in software to limit the damage caused by an unhandled exception.

Normally an unhandled exception will cause its thread to print diagnostic information to the console and then stop. This is an acceptable error-handling strategy for many command-line programs, but it is not a good error-handling strategy for a graphical program where the console is invisible.

Goals

I’ve been working on a graphical program where I want all exceptions and errors to be reported to the user in some way, rather than failing silently and presenting an unresponsive user interface when something goes wrong.

Additionally, there are certain critical sections of the program where an unhandled exception could leave a nearby data structure in an invalid state, causing subsequent operations on that structure to fail and/or propagate the corruption to other parts of the program state. It would be better if an unhandled exception occurring in a critical section marked any affected data structures as corrupt, such that further operations attempting to access those structures would refuse to run.

The Bulkheads Pattern can be used to meet both of those goals:

How it Works

I’ve implemented a family of @capture_crashes_to... function decorators in Python which mark a critical section of code and define which Bulkhead the critical section is related to.

Any exception raised in the critical section which bubbles up to the decorator will be caught at the last moment and set as the crash_reason for the associated bulkhead.

If a function calls into a critical section marked with @capture_crashes_to... whose bulkhead has been marked as crashed, the critical section will refuse to run and will instead return to the caller immediately, possibly returning a default value.

Example

Below, the child_task_did_complete listener method is marked with @capture_crashes_to_self, so any unexpected exception raised inside the listener won’t escape and crash the thread which called the listener. Instead the exception will be stored in the Task’s crash_reason.

class Task(Bulkhead):
    crash_reason: Optional[BaseException] = None

class DownloadResourceTask(Task):
    @capture_crashes_to_self
    def child_task_did_complete(self, task: Task) -> None:
        if task is self._download_body_task:
            ...
        elif task is self._parse_links_task:
            links = self._parse_links_task.future.result()
            embedded_resources = [
                Resource(
                    self._resource.project,
                    urljoin(self._resource.url, link.relative_url))
                for link in links if not link.embedded
            ]
            for resource in embedded_resources:
                self.append_child(resource.create_download_task(...))
        else:
            ...

And indeed an unhandled exception did end up getting raised in the above code: The urljoin operation was observed to raise an unexpected exception for certain unusual inputs even though it generally never raises.

Try it!

The infrastructure for using bulkheads currently lives as a utility module in the Crystal project. You can install it to your local Python virtual environment using:

python -m pip install crystal-web

Here is a small program you can run in the Python REPL that demonstrates the use of bulkheads:

>>> from crystal.util.bulkheads import BulkheadCell, capture_crashes_to
>>> 
>>> divide_bulkhead = BulkheadCell()
>>> 
>>> @capture_crashes_to(divide_bulkhead)
... def divide(x, y):
...     return x / y
... 
>>> print(divide(10, 2))
5.0
>>> print(divide(10, 0))
Exception in bulkhead:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/me/Projects/crystal-web/crystal/util/bulkheads.py", line 292, in bulkhead_call
    return func(*args, **kwargs)  # cr-traceback: ignore
  File "<stdin>", line 3, in divide
ZeroDivisionError: division by zero
None
>>> divide_bulkhead.crash_reason
ZeroDivisionError('division by zero')
>>> print(divide(10, 5))
None

See the API for more detailed information about how to use bulkheads.

Appendix

Effective Critical Sections and Bulkheads
Enforcing that an error-handling strategy is defined
API

Effective Critical Sections and Bulkheads

For critical sections, I’ve found it effective to protect:

listener methods, since they usually operate in a different conceptual context than their caller, and it is useful to prevent crashes inside listeners from also crashing their callers;
the run function in new threads, which can capture any exception that is about to crash the thread; and
the main function in the main thread, which can capture any exception that is about to crash the thread.

For bulkheads, I’ve found it effective to protect:

Component structures¹ (which can be arranged in a tree, forming a larger Composite structure),
certain “top-level” structures² in the program

Enforcing that an error-handling strategy is defined

It can be useful for a caller to enforce that a callee has an explicitly defined error-handling strategy (via a @capture_crashes_to* decorator). For example, a function that is dispatching events to listeners may want to enforce that each listener is marked with @capture_crashes_to* before calling it. Such enforcement can be implemented with the following utilities:

run_bulkhead_call(func) – Calls a function after verifying it is marked with @capture_crashes_to*. Raises if the function is not marked.
ensure_is_bulkhead_call(func) – Raises if the specified function is not marked with @capture_crashes_to*.
is_bulkhead_call(func) – Returns whether the specified function is marked with @capture_crashes_to*.

Example of an event dispatcher which enforces its listeners are marked with @capture_crashes_to*:

def _resource_did_instantiate(self, resource: Resource) -> None:
    for lis in self.listeners:
        run_bulkhead_call(lis.resource_did_instantiate, resource)

See API > Error-handling enforcement utilities for more information.

API

Everything in this section is imported from crystal.util.bulkheads. For example:

from crystal.util.bulkheads import Bulkhead, capture_crashes_to_stderr

Bulkhead API

First, define locations where unhandled exceptions can be captured to by implementing the Bulkhead protocol:

class Bulkhead(Protocol):  # abstract
    """
    A sink for unhandled exceptions (i.e. crashes).
    """
    crash_reason: Optional[CrashReason]

CrashReason = BaseException

Or if you don’t want to implement your own bulkhead type, you can use the included BulkheadCell:

class BulkheadCell(Bulkhead):
    """
    A concrete Bulkhead which stores any crash that occurs,
    but takes no special action to report such crashes.
    """
    crash_reason: Optional[CrashReason]

    def __init__(self, value: Optional[CrashReason]=None) -> None:
        self.crash_reason = value

Critical Section API

Then, mark critical sections in your code with an appropriate @capture_crashes_to* decorator. There are several to choose from:

For critical sections defined as a method on a sensitive data structure that is its own bulkhead, @capture_crashes_to_self is a good choice:

def capture_crashes_to_self(
        bulkhead_method: Optional[Callable[Concatenate[_B, _P], _RT]]=None,
        *, return_if_crashed=None  # _RF
        ):
    """
    A Bulkhead method that captures any raised exceptions to itself,
    as the "crash reason" of the bulkhead.

    If the bulkhead was already crashed (with a non-None "crash reason") then
    this method will immediately abort, returning `return_if_crashed`.

    Examples:
        class MyBulkhead(Bulkhead):
            @capture_crashes_to_self
            def foo_did_bar(self) -> None:
                ...

            @capture_crashes_to_self(return_if_crashed=Ellipsis)
            def calculate_foo(self) -> Result:
                ...
    """

For critical sections defined as a method which takes a sensitive data structure (i.e. a bulkhead) as an argument, @capture_crashes_to_bulkhead_arg is a good choice:

def capture_crashes_to_bulkhead_arg(
        method: Optional[Callable[Concatenate[_S, _B, _P], _RT]]=None,
        *, return_if_crashed=None  # _RF
        ):
    """
    A method that captures any raised exceptions to its first Bulkhead argument,
    as the "crash reason" of the bulkhead.

    If the bulkhead was already crashed (with a non-None "crash reason") then
    this method will immediately abort, returning `return_if_crashed`.

    Examples:
        class MyClass:
            @capture_crashes_to_bulkhead_arg
            def other_foo_did_bar(self, other: Bulkhead) -> None:
                ...

            @capture_crashes_to_bulkhead_arg(return_if_crashed=Ellipsis)
            def calculate_baz(self, other: Bulkhead) -> Result:
                ...
    """

Sometimes you may have an outer function that is already operating on a sensitive data structure (i.e. a bulkhead) which wants to define (and call) an inner function that also operates on the same structure. In that case it may be useful to protect the inner function with @capture_crashes_to(bulkhead), specifying the bulkhead to use at the time of decoration:

def capture_crashes_to(
        bulkhead: Bulkhead,
        return_if_crashed=None  # _RF
        ) -> Callable[[Callable[_P, _RT]], Callable[_P, Union[_RT, _RF]]]:
    """
    A method that captures any raised exceptions to the specified Bulkhead,
    as the "crash reason" of the bulkhead.

    If the bulkhead was already crashed (with a non-None "crash reason") then
    this method will immediately abort, returning `return_if_crashed`.

    Examples:
        @capture_crashes_to(bulkhead)
        def foo_did_bar() -> None:
            ...

        @capture_crashes_to(bulkhead, return_if_crashed=Ellipsis)
        def calculate_foo() -> Result:
            ...
    """

As a last resort, you may have a critical section where there is no reasonable bulkhead available to route an unhandled exception to. For example if your core error reporting infrastructure code (like a display_error function) fails, the only reasonable choice may be to fallback to printing the exception to the standard error stream (stderr):

def capture_crashes_to_stderr(
        func: Optional[Callable[_P, _RT]]=None,
        *, return_if_crashed=None  # _RF
        ):
    """
    A method that captures any raised exceptions, and prints them to stderr.

    Examples:
        @capture_crashes_to_stderr
        def foo(self) -> None:
            ...

        @capture_crashes_to_stderr(return_if_crashed=Ellipsis)
        def calculate_foo(self) -> Result:
            ...
    """

Critical Sections (in Crystal) always print to the standard error stream

The current implementation of the @capture_crashes_to* decorators in Crystal always prints unhandled exceptions to stderr as a side effect, to indicate that a crash happened.

Most of the decorators print in yellow warning text because the crash was presumably handled when it was routed to a bulkhead.
However the @capture_crashes_to_stderr decorator prints in red error text to signal that the only handling of the crash was to print it to stderr.

In the future, if/when the bulkheads.py module is extracted to its own package on PyPI, independent of Crystal, I expect I’ll alter the @capture_crashes_to* decorators to NOT print anything to stderr by default since I don’t think that side-effect is appropriate for all programs that may want to use these decorators.

Error-handling enforcement utilities

def run_bulkhead_call(
        bulkhead_call: Callable[_P, _R],
        /, *args: _P.args,
        **kwargs: _P.kwargs
        ) -> '_R':
    """
    Calls a function marked as @capture_crashes_to*,
    which does not reraise exceptions from its interior.

    Raises AssertionError if the specified function is not actually
    marked with @capture_crashes_to*.
    """

def ensure_is_bulkhead_call(callable: Callable) -> None:
    """
    Raises AssertionError if the specified function is not actually
    marked with @capture_crashes_to*.
    """

def is_bulkhead_call(callable: Callable) -> bool:
    """
    Returns whether the specified function is marked with @capture_crashes_to*.
    """

Component bulkhead structures in my website downloader Crystal include (1) Task (which compose to form the Task Tree) and (2) entitytree.Node (which compose to form the Entity Tree)↩
Top-level bulkhead structures in my website downloader Crystal include (1) the Task Scheduler itself (by using the root Task as the associated bulkhead), (2) the Task Tree UI (by again using the root Task as the associated bulkhead), and (3) the Entity Tree UI (which is its own bulkhead that reports errors to the Task Tree UI).↩

Debugging a deadlock in Python

2023-10-20T00:00:00+00:00

I recently encountered a deadlock while running the automated tests of my website downloader, Crystal. The process for investigating and fixing this deadlock was interesting, so I thought I’d share it:

What does a deadlock look like?

One day after making some changes to my program, when I ran the program’s automated tests the program just completely froze, becoming unresponsive to all input and not providing any further output:

$ crystal --test
...
======================================================================
RUNNING: test_some_tasks_may_complete_immediately (crystal.tests.test_tasks)
----------------------------------------------------------------------
    "GET /_/https/example.com/ HTTP/1.1" 404 -
    *** Requested resource not in archive: https://example.com/
    "GET /_/https/xkcd.com/1/ HTTP/1.1" 200 -
⚠️ No further output printed. No further input accepted. Program is stuck.

To investigate why the program was frozen I wanted to see exactly which lines of code the program was stuck on. Python has a useful faulthandler module in the standard library that, among other things, can start listening for various Unix signals and print out a traceback showing the line of code being executed on all threads.

To enable the faulthandler when running a program you can use the PYTHONFAULTHANDLER environment variable:

$ PYTHONFAULTHANDLER=1 crystal --test

Then after the program becomes stuck, you can open a new terminal window and send a SIGABRT signal to the stuck program to trigger the faulthandler:

$ kill -SIGABRT 7984  # 7984 = PID of the python process

When triggered on the Crystal program, the faulthandler printed tracebacks like:

faulthandler traceback

======================================================================
RUNNING: test_some_tasks_may_complete_immediately (crystal.tests.test_tasks)
----------------------------------------------------------------------
    "GET /_/https/example.com/ HTTP/1.1" 404 -
    *** Requested resource not in archive: https://example.com/
    "GET /_/https/xkcd.com/1/ HTTP/1.1" 200 -
Fatal Python error: Aborted

Thread 0x0000700058d9b000 (most recent call first):
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/std.py", line 111 in acquire
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/std.py", line 134 in __enter__
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/std.py", line 577 in __new__
  File "/Users/me/Crystal/src/crystal/util/progress.py", line 143 in __init__
  ...

Thread 0x0000700057d98000 (most recent call first):
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/std.py", line 111 in acquire
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/std.py", line 134 in __enter__
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/_monitor.py", line 66 in run
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930 in _bootstrap

(... 🤔 62 copies of the above paragraph ...)

Thread 0x0000700017c55000 (most recent call first):
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/selectors.py", line 416 in select
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socketserver.py", line 232 in serve_forever
  File "/Users/me/Crystal/src/crystal/tests/test_download_body.py", line 129 in do_serve_forever
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910 in run
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930 in _bootstrap

Thread 0x000070000ab28000 (most recent call first):
  File "/Users/me/Crystal/src/crystal/tests/util/runner.py", line 130 in run
  File "/Users/me/Crystal/src/crystal/util/xthreading.py", line 101 in wrapper
  File "/Users/me/Crystal/src/crystal/tests/util/runner.py", line 49 in run_test
  File "/Users/me/Crystal/src/crystal/util/xthreading.py", line 101 in wrapper
  File "/Users/me/Crystal/src/crystal/tests/index.py", line 107 in _run_tests
  File "/Users/me/Crystal/src/crystal/tests/index.py", line 80 in run_tests
  File "/Users/me/Crystal/src/crystal/util/xthreading.py", line 101 in wrapper
  File "/Users/me/Crystal/src/crystal/__main__.py", line 314 in bg_task
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910 in run
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973 in _bootstrap_inner
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930 in _bootstrap

Current thread 0x000000011b0405c0 (most recent call first):
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/wx/core.py", line 2262 in MainLoop
  File "/Users/me/Crystal/src/crystal/__main__.py", line 323 in _main
  File "/Users/me/Crystal/src/crystal/__main__.py", line 45 in main
  File "/Users/me/Crystal/venv3.9/bin/crystal", line 6 in <module>

Notice in the traceback above that several of the stuck threads are at tqdm/std.py", line 111 in acquire, which is trying to acquire a low-level threading.RLock.

When there is at least one thread trying to acquire a lock and not making any progress, there’s probably a deadlock. So the program definitely seems to be deadlocked.

Isolating and fixing a deadlock

To debug further, it would be useful for the program to detect when it cannot acquire the lock it wants and then immediately print out:

the traceback which is trying to acquire the lock and
the traceback which is currently holding the lock.

So I made some modifications around the code that acquires/releases the lock to log that information:

Code to add deadlock detection and logging to lock acquire & release

✚ import threading
✚ import traceback

class TqdmDefaultWriteLock(object):
    th_lock = TRLock()
✚   th_lock_holders = []  # type: List[Tuple[threading.Thread, traceback.StackSummary]]

    def acquire(self, *a, **k):
✚       if 'timeout' not in k:
✚           k['timeout'] = 5.0  # seconds

        for lock in self.locks:
            ok = lock.acquire(*a, **k)
✚           if not ok:
✚               print(
✚                   f'*** TqdmDefaultWriteLock: Failed to acquire lock in '
✚                   f'{k["timeout"]} seconds. '
✚                   f'Lock holder is {TqdmDefaultWriteLock.th_lock_holders[-1][0]} at:\n'
✚                   f'{"".join(TqdmDefaultWriteLock.th_lock_holders[-1][1])}')
✚               print(f'*** New acquirer is:\n{"".join(traceback.format_stack())}')
✚               raise AssertionError('Failed to acquire TqdmDefaultWriteLock')

✚       TqdmDefaultWriteLock.th_lock_holders.append((
✚           threading.current_thread(),
✚           traceback.format_stack()
✚       ))

    def release(self):
✚       TqdmDefaultWriteLock.th_lock_holders.pop()

        for lock in self.locks[::-1]:  # Release in inverse order of acquisition
            lock.release()

With those modifications in place, I can pinpoint the smoking gun:

======================================================================
RUNNING: test_project_opens_as_readonly_when_project_is_on_readonly_filesystem (crystal.tests.test_readonly_mode)
----------------------------------------------------------------------
*** TqdmDefaultWriteLock: Failed to acquire lock in 5.0 seconds. Lock holder is <_MainThread(MainThread, started 4446586304)> at:
  ...
  File "/Users/me/Crystal/src/crystal/model.py", line 679, in _apply_migrations
    self._process_table_rows(
  File "/Users/me/Crystal/src/crystal/model.py", line 785, in _process_table_rows
    progress_bar.update()
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/std.py", line 1263, in update
    self.refresh(lock_args=self.lock_args)
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/std.py", line 1367, in refresh
    self._lock.acquire()

*** New acquirer is:
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 930, in _bootstrap
    self._bootstrap_inner()
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/_monitor.py", line 66, in run
    with self.tqdm_cls.get_lock():
  File "/Users/me/Crystal/venv3.9/lib/python3.9/site-packages/tqdm/std.py", line 134, in __enter__
    self.acquire()

Here, I notice that the traceback [_apply_migrations() → _process_table_rows() → update()] is holding the global tqdm._lock, which is strange because there shouldn’t be a reason for this code to hold a lock at all.

Culprit: Inappropriate locking

Originally I thought update() just called display() directly:

sequenceDiagram participant Project participant ProgressBar as ProgressBar
(extends tqdm) participant lock as tqdm._lock
«global» participant ProgressListener Project->>+Project: _process_table_rows loop each table row Project->>+ProgressBar: update ProgressBar->>+ProgressBar: display [about once per second] ProgressBar->>ProgressListener: update_gui_progress_bar_func deactivate ProgressBar deactivate ProgressBar end deactivate Project

But actually update() calls a refresh() method first, and that call to refresh() grabs the global tqdm._lock!

sequenceDiagram participant Project participant ProgressBar as ProgressBar
(extends tqdm) box rgb(191, 223, 255) participant lock as tqdm._lock
«global» end participant ProgressListener Project->>+Project: _process_table_rows loop each table row Project->>+ProgressBar: update rect rgb(191, 223, 255) note over ProgressBar,ProgressListener: ⚠️ Intermediate refresh() method acquires tqdm._lock! ProgressBar->>+ProgressBar: refresh [about once per second] ProgressBar->>lock: acquire rect white ProgressBar->>+ProgressBar: display ProgressBar->>ProgressListener: update_gui_progress_bar_func deactivate ProgressBar end ProgressBar->>lock: release deactivate ProgressBar end deactivate ProgressBar end deactivate Project

It makes sense for the update() method on a normal tqdm instance to use a lock because it prints its progress bar to a shared terminal, possibly at the same time that other instances are also trying to print to the same terminal. However the custom ProgressBar subclass of tqdm which is used here is displaying a graphical progress bar in an unshared dialog and so it does not need to do additional locking.

So we need to avoid the locking done by refresh().

When looking at the code for that method, I notice that it’s possible to pass a nolock=True argument to disable its locking behavior:

class tqdm(...):
    def refresh(self, nolock=False, lock_args=None):
        if not nolock:
            self._lock.acquire(...)
        self.display()
        if not nolock:
            self._lock.release()
        return True

So it should be possible to override refresh() to always assume nolock=True:

class ProgressBar(tqdm):
✚   @overrides
✚   def refresh(self, nolock=False, lock_args=None) -> bool:
✚       # Never grab the global tqdm._lock,
✚       # because we're NOT printing a CLI progress bar to a shared terminal
✚       return super().refresh(nolock=True, lock_args=lock_args)

    # Override tqdm's normal behavior of printing a CLI progress bar
    # to instead update a GUI progress bar
    @overrides
    def display(self, *args, **kwargs) -> None:
        update_gui_progress_bar_func(self.n, self.miniters)

Lo and behold, the deadlocking behavior is no longer observed! 🎉

Common culprit: Acquiring multiple locks in an inconsistent order

Although not a problem in the above example, another common cause of deadlocks happens when a section of code acquires multiple locks at the same time but a different section of code tries to acquire the same locks in a different order.

A common fix for deadlock related to acquiring locks in an inconsistent order is to give each lock a unique ID upon construction. Whenever a section of code wants to acquire multiple locks at a time, it must acquire each individual lock in increasing order of ID.

Fin

So, that was my recent adventure in tracking down and fixing a deadlock in Python. Perhaps some of the techniques mentioned above will be helpful for you in debugging your own programs. Happy coding!

Custom file icons, folder icons, and app icons on different operating systems

2023-10-12T00:00:00+00:00

I recently extended my website downloader, Crystal, so that the projects it creates have a proper icon on macOS, Windows, and Linux. It was a lot more challenging than I expected!

Crystal organizes a group of downloaded web pages into a project, which is a special folder containing a particular arrangement of files:

$ tree xkcd.crystalproj
xkcd.crystalproj      ← project
├── database.sqlite   ← web page metadata
└── revisions
    ├── 1   ← web page #1 content
    ├── 2   ← web page #2 content
    ├── 3   ← web page #3 content
    ├── 4   ← web page #4 content
    └── 5   ← web page #5 content

My goal was to make each of these projects appear as a file, rather than as a folder, and to open in Crystal automatically when it is double-clicked on:

Looks and behaves like a file, but is secretly a folder!

Icons in macOS

On macOS it’s easy to get projects to behave the way I wanted. It’s actually common on macOS for certain types of folders - called bundles - to behave like files. For example apps on macOS are themselves stored as bundles with the .app extension.

To tell macOS that folders ending with .crystalproj are bundles I added the following to Crystal’s Info.plist file, which is included in all macOS applications:

<key>CFBundleDocumentTypes</key>
<array>
    <dict>
        <key>CFBundleTypeExtensions</key>
        <array>
            <string>crystalproj</string>
        </array>
        <key>CFBundleTypeIconFile</key>
        <string>DocIconMac.icns</string>
        <key>CFBundleTypeName</key>
        <string>Crystal Project</string>
        <key>CFBundleTypeRole</key>
        <string>Editor</string>
        <key>LSTypeIsPackage</key>
        <true/>
    </dict>
</array>

The important parts of this Info.plist are:

CFBundleTypeExtensions = [‘crystalproj’]
- Defines the extension used to recognize a project
LSTypeIsPackage = true
- Says that a project is a bundle and therefore should be treated as a file even though it is a folder
CFBundleTypeIconFile = “DocIconMac.icns”
- Defines the name of the .icns icon file to use for a project

Setting the app icon requires similar changes to the Info.plist:

<key>CFBundleIconFile</key>
<string>AppIconMac.icns</string>

Simple.

Icons in Windows

On Windows there is no concept of a “bundle”. However it is possible to give a folder a custom icon by putting a specially crafted desktop.ini file inside of it:

$ cat xkcd.crystalproj/desktop.ini
[.ShellClassInfo]
DirectoryClass=crystalproj
ConfirmFileOp=0
IconFile=icons\docicon.ico
IconIndex=0
InfoTip=Crystal Project

It’s even possible to tell Windows to open Crystal when the folder is double-clicked on by setting some registry keys:

However it’s still possible to navigate inside of a such a folder, notably from Open and Save dialogs, so we need another way to open projects in those contexts. My solution was to add a special “opener” file inside of a project folder which could be used to open the enclosing project:

Of course that .crystalopen file itself needs an icon. Again you can set some registry keys to tell Windows about this new file extension and its associated icon:

And if a .crystalopen file is double-clicked we want it to open Crystal, which can be configured with more registry keys:

All of these registry keys should be set by the installer for the app.

Icons in Linux

Oh Linux… Icons for files, folders, and apps are displayed by the desktop environment, and multiple such desktop environments exist for Linux. The most common ones (in 2023) seem to be GNOME 3 and KDE, but there’s also xfce, MATE (GNOME 2), and Budgie, among others. Most of these desktop environments strive to implement the Freedesktop Specifications for defining file types & icons, but with varying degrees of compatibility.

According to the Icon Theme Specification, which is one of the Freedesktop Specifications, an application developer should be able to install icons for file types to a special “hicolor” theme which all other themes must eventually inherit from. But there are some problems:

GNOME doesn’t seem to implement theme inheritance correctly, so icons installed in the “hicolor” theme aren’t picked up by other themes. So instead of installing there I wrote a script that installed icons to every theme on the user’s system. Nobody on StackOverflow has been able to find a better solution so far.
GNOME stores custom icon references for folders in the GIO database but KDE (and the Freedesktop Specification) use .directory files.
GNOME has not one but two GIO attributes for storing custom icon references, and not all programs recognize both of them:
- The metadata::custom-icon-name key references the name of an icon installed to a theme.
  - However the Desktop Icons GNOME Shell Extension, which is responsible for displaying icons on the desktop, does not understand this key!
- The metadata::custom-icon key references a single icon file.
  - All programs in GNOME seem to understand this kind of key so long as you use an absolute file:// URL rather than an absolute or relative file path.
Some programs only recognize PNG icons and other programs only recognize SVG icons, so you must install both PNG and SVG versions of each icon.
- In particular some programs in KDE only seem to recognize SVG icons even though the Icon Theme Specification implies that support for PNG icons is mandatory.
- GNOME by contrast seems to prefer PNG icons.
- If you started with only a raster version of an icon, as I did, you’ll have to vectorize the icon to make it an SVG. I had some success using the Adobe Express PNG to SVG converter tool to make an initial conversion and Inkscape for touching it up.
Some programs don’t notice new icons immediately after installing them.
- KDE’s desktop, Plasma, must be restarted before it notices new icons.
- KDE’s file manager Dolphin doesn’t seem to notice new icons until it is restarted. Unfortunately I haven’t found a safe way to restart Dolphin without losing information about the windows and tabs it has open.
Some programs - notably Open and Save dialogs - aren’t able to scale icons to the size they expect, so you must provide prescaled versions of each icon.
- Scaling SVG icons isn’t particularly straightforward, but I managed to do so with the rconvert-svg tool.
Icons for apps on the desktop don’t show up correctly in GNOME unless you set the metadata::trusted GIO attribute on it.
There are probably even more issues in desktop environments other than GNOME 3 and KDE, but I did not test any additional desktop environments.

In short, my experience in getting icons to work in Linux has been a big poorly-documented mess. But after 2 weeks I did succeed!

Conclusions

When creating a desktop application for end users it’s important to get icons for your app and its documents configured correctly to provide a good user experience, but it getting those configurations correct can be challenging.

I’d like to file bugs on several Linux desktop environments so that future Linux application developers don’t have such a hard time getting icons setup, but right now I just don’t have the energy. Hopefully at least this blog post will help other application developers configure icons more quickly.

How to make a binary .deb Debian package using Docker

2023-08-31T00:00:00+00:00

The official instructions for making a .deb Debian package¹ I found to be rather long-winded and fiddly, so I came up with a hopefully more-straightforward method of building a .deb package, using Docker.

What’s in a binary .deb package?

A binary .deb package is kind of like a .zip file whose contents are unpacked to / during the installation process. For example, a simple .deb package that installs a hello program to /usr/bin might have the following structure:

$ dpkg -c hello_1.0-1_darwin-amd64.deb  # show contents of .deb package
./usr/
./usr/bin/
./usr/bin/hello

A .deb package also contains a control file with metadata describing the package:

$ cat ./DEBIAN/control
Package: hello
Version: 1.0-1
Section: misc
Priority: optional
Architecture: all
Maintainer: David Foster <david@example.com>
Description: A program that prints a hello world message and exits.

How can Docker be used to build a binary .deb package?

Let’s say you make a Dockerfile that builds and installs the hello program, based on the official build and install instructions for the program. Such a Dockerfile might look like:

FROM debian:latest

# Install build dependencies
RUN apt-get update
RUN apt-get install build-essential -y  # install gcc

# Download source code
COPY ./hello.c /usr/src/

# Compile the binary (AKA: make)
RUN gcc -o /tmp/hello /usr/src/hello.c

# Install the binary (AKA: make install)
RUN cp /tmp/hello /usr/bin/hello

When you build a Docker image from such a Dockerfile, each capitalized command in the Dockerfile (like RUN and COPY) creates a separate layer which contains the files created by the command.

Notably, the last RUN command above which actually installs the hello binary creates a layer containing exactly those files which should be installed.

If we could extract the files from that “install” layer then we’d have exactly the set of files that a hello .deb package should install. It turns out we can!

Extracting the installed files from a Docker image layer

First build the Docker image from the Dockerfile:

$ docker build -t hello-build:latest .

Then extract the contents of all Docker image layers:

$ mkdir build
$ docker save -o build/hello-build.tar hello-build:latest
$ cd build
$ tar xf hello-build.tar

You should see a directory structure like:

$ tree
.
├── 05bc8a33fbd971aa8c6d6e3613ea41f0477e4507229e6072c43f3ef61b549b82
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── 2745707919ffe930cb00def7c34e4e9bbac8e30b99f66e631edb2eb6b3a5b89c
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── 3bda020d2a872426189c67863e8add029463c14957b9fa17690e591a6a1455e2
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── 440d2890e2b9b3ce91db819ee42fb77a807daf2f46f02579a4c0df9274eebc13
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── 8c7b379494075edb4563162a9e844d548d797a470cb55c3f49f0969b9437b1e4.json
├── 9bc167af8f88fb2b70b47cc2b2be68a8be5862e49572353c13075eb5ce8845bc
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── b6100c2b22f88d1698c24d418c989bfe0174d7a4e20ca7d60824dc36d1013638
│   ├── VERSION
│   ├── json
│   └── layer.tar
├── hello-build.tar
├── manifest.json
└── repositories

Each of the directories with a very long name corresponds to a Docker image layer. The contents of each directory is the set of files the layer contains.

The manifest.json contains information about which layer corresponds to which directory:

$ cat manifest.json | jq
[
  {
    "Config": "8c7b379494075edb4563162a9e844d548d797a470cb55c3f49f0969b9437b1e4.json",
    "RepoTags": [
      "hello-build:latest"
    ],
    "Layers": [
      "05bc8a33fbd971aa8c6d6e3613ea41f0477e4507229e6072c43f3ef61b549b82/layer.tar",
      "b6100c2b22f88d1698c24d418c989bfe0174d7a4e20ca7d60824dc36d1013638/layer.tar",
      "9bc167af8f88fb2b70b47cc2b2be68a8be5862e49572353c13075eb5ce8845bc/layer.tar",
      "440d2890e2b9b3ce91db819ee42fb77a807daf2f46f02579a4c0df9274eebc13/layer.tar",
      "2745707919ffe930cb00def7c34e4e9bbac8e30b99f66e631edb2eb6b3a5b89c/layer.tar",
      "3bda020d2a872426189c67863e8add029463c14957b9fa17690e591a6a1455e2/layer.tar"
    ]
  }
]

Now remember, we want to find the files in the layer for the install step in the Dockerfile, which happens to be the last step and thus also the last layer:

# Install the binary (AKA: make install)
RUN cp /tmp/hello /usr/bin/hello

So we just need to look at the last item in the "Layers" key from the manifest.json file to find the directory we want:

"3bda020d2a872426189c67863e8add029463c14957b9fa17690e591a6a1455e2/layer.tar"

So let’s unpack the files for that layer:

$ export LAYER_TARFILE="3bda020d2a872426189c67863e8add029463c14957b9fa17690e591a6a1455e2/layer.tar"
$ mkdir layer
$ tar Cxf layer $LAYER_TARFILE
$ cd layer

And indeed we see the installed files:

$ tree
.
└── usr
    └── bin
        └── hello

Building a .deb package from a set of files

Remember that a binary .deb package is built from a directory of files to install plus a DEBIAN directory containing at least a control file.

Write a `control` file

Since we already have a directory of files to install from the last step, we just need to add a DEBIAN directory and write a control file:

$ mkdir DEBIAN
$ cat > DEBIAN/control <<EOF
Package: hello
Version: 1.0
Section: misc
Priority: optional
Architecture: all
Maintainer: David Foster <david@example.com>
Description: Prints a hello world message and exits.
EOF

Here are some important parameters in a typical control file:

Package (required) – Name of the package. Used to refer to it as a dependency.
Version (required) – Version of the package.
Section (recommended) – misc is fine, although a more targeted category may make your package more discoverable if you upload it to a package archive.
Priority (recommended) – optional is probably what you want.
Architecture (required) – all disables machine architecture checking. Get a list of valid architectures for your current machine using dpkg-architecture -L.
Maintainer (required) – The package maintainer’s name and email address.
Description (required) – The short description and long description of the package.
Depends – Names other packages to install alongside this package.
Recommends – Names other packages to install by default alongside this package unless --no-install-recommends is passed to apt-get install.
Conflicts – Names other packages whose presence will prevent this package from being installed.

Build the `.deb`

With the control file written, we now have all files that we want to put into the .deb package. So let’s actually bundle it into a .deb:

$ docker run -it -v .:/home --name deb-builder debian:latest
$$ dpkg-deb --root-owner-group --build home

Inspect the `.deb`

You can inspect the contents of the .deb to verify that it looks reasonable:

$$ dpkg -c home.deb | more  # (Optional) verify contents look OK
drwxr-xr-x root/root         0 2023-09-08 18:50 ./
drwxr-xr-x root/root         0 2023-07-25 00:00 ./usr/
drwxr-xr-x root/root         0 2023-09-08 18:46 ./usr/bin/
-rwxr-xr-x root/root     15952 2023-09-08 18:46 ./usr/bin/hello

Find and fix errors with Lintian

You can also use the Lintian tool to look for problems with your package:

$$ apt-get update && apt-get install lintian -y
$$ lintian --tag-display-limit 0 home.deb  # (Optional) verify output looks OK; some ERRs are OK
E: hello: arch-independent-package-contains-binary-or-object [usr/bin/hello]
E: hello: extended-description-is-empty
E: hello: no-changelog usr/share/doc/hello/changelog.gz (native package)
E: hello: no-copyright-file
E: hello: unstripped-binary-or-object [usr/bin/hello]
W: hello: no-manual-page [usr/bin/hello]
W: hello: undeclared-elf-prerequisites (libc.so.6) [usr/bin/hello]

Each line starting with E is an error, and each line starting with W is a warning.

You can get a detailed description of each error by adding running lintian with the -i option:

$$ lintian -i --tag-display-limit 0 home.deb
E: hello: arch-independent-package-contains-binary-or-object [usr/bin/hello]
N: 
N:   The package contains a binary or object file but is tagged Architecture: all.
N:   
N:   If this package contains binaries or objects for cross-compiling or binary blobs for other purposes independent of the
N:   host architecture (such as BIOS updates or firmware), please add a Lintian override.
N: 
N:   Visibility: error
N:   Show-Always: no
N:   Check: binaries/architecture
...

It’s quite possible the .deb may work as-is even if there are errors. However it’s best to fix as many of the errors and warnings as possible.

For example, to fix the following error:

W: hello: undeclared-elf-prerequisites (libc.so.6) [usr/bin/hello]

You can add Depends: libc6 to the control file.

Don’t install to `/usr/local`!

A common type of error you do need to fix happens when you try to install something into /usr/local, because Debian reserves the local directory for its own purposes. For example if we installed hello to /usr/local, we’d see Lintian errors that look like:

E: hello: dir-in-usr-local [usr/local/bin/]
E: hello: file-in-usr-local [usr/local/bin/hello]

Fix those errors by altering the build process in your Dockerfile to install to /usr/bin rather than to /usr/local/bin. For many Linux programs built using a line like ./configure, you can usually alter that line to be ./configure --prefix=/usr to configure the install process to install to /usr/bin.

Copy out the `.deb`

Once you’re happy with the .deb package, copy it out of your build container, give it a proper name, and exit the build container:

$$ cp home.deb /home/hello_1.0_all.deb
$$ exit
$ ls
hello_1.0_all.deb
...

If you need to reenter the build container later, use: console $ docker start -i deb-builder

Testing your .deb package

It’s a good idea to verify that package is installable. You can do that in another fresh container:

$ docker run -it -v .:/home --name deb-tester debian:latest
$$ apt-get update
$$ apt-get install ./home/hello_1.0_all.deb -y
$$ which hello  # ensure installed
/usr/bin/hello
$$ hello  # ensure runs
Hello world!
$$ exit

If you need to reenter the testing container later, use: console $ docker start -i deb-tester

Putting it all together

A complete example of all the files to build the hello .deb package is in the following GitHub repository:

hello-deb-package

https://wiki.debian.org/Packaging ↩

Debugging bash scripts with the bashdb debugger

2023-02-22T00:00:00+00:00

Sometimes when I have a problem with a bash script I want the ability to run each line of the script one by one to see what is wrong. There is a debugger called bashdb that can be used in this way. It behaves similarly to gdb (used for debugging C and C++ programs) and pdb (used for debugging Python programs).

How to install bashdb

First you need to determine what version of bash you are running, since you need to use a corresponding version of bashdb:

# bash --version
GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

This version of bash is 5.1.16, so we need a bashdb version 5.1.x. Let’s download it:

For versions of bashdb <= 5.0.x, download from: https://sourceforge.net/projects/bashdb/files/bashdb/
For versions of bashdb >= 5.1.x, you must download from the associated Git branch. For example bashdb 5.1.x is in git branch bash-5.1

NOTE: The following commands are written for an Ubuntu- or Debian-based Linux distribution. Also, you may need to prefix some commands with sudo to avoid permission errors.

# apt-get update
# apt-get install git -y
# git clone https://git.code.sf.net/p/bashdb/code bashdb-code
# cd bashdb-code/
# git checkout bash-5.1

For bashdb >= 5.1.x, you’ll need to build the configure and bashdb files:

# apt-get update
# apt-get install autoconf -y  # to run autogen.sh
# apt-get install binutils -y  # to install "strings" tool, for autogen.sh
# apt-get install make -y  # to install "make" tool
# apt-get install texinfo -y  # to install "makeinfo" tool, for make
# ./autogen.sh

You should now have a bashdb file in the current directory:

# ls | grep bashdb
bashdb
bashdb-main.inc
bashdb-main.inc.in
bashdb-part2.sh
bashdb-trace
bashdb-trace.in
bashdb.in

(Optional) If you have root permissions, you can install bashdb:

# make install

How to run bashdb

Let’s create an example script to debug:

# cat << EOF > /tmp/hello.sh
echo 1
echo 2
echo 3
echo 4
EOF

If you did install bashdb in the step above, start the debugger on a Bash script at /tmp/hello.sh with:

# bash --debugger /tmp/hello.sh 
...
(/tmp/hello.sh:1):
1:  echo 1
bashdb<0>

If you did not install bashdb, you can run the debugger by calling the bashdb script manually:

# bash ./bashdb /tmp/hello.sh 
...
(/tmp/hello.sh:1):
1:  echo 1
bashdb<0>

Common commands in bashdb

See all debugger commands using help:

bashdb<0> help
Available commands:
-------------------
  action     condition  edit     frame    load     run     source  unalias  
  alias      continue   enable   handle   next     search  step    undisplay
  backtrace  debug      eval     help     print    set     step+   untrace  
  break      delete     examine  history  pwd      shell   step-   up       
  clear      disable    export   info     quit     show    tbreak  watch    
  commands   display    file     kill     return   signal  trace   watche   
  complete   down       finish   list     reverse  skip    tty   

Readline command line editing (emacs/vi mode) is available.
Type "help" followed by command name for full documentation.
bashdb<1>

See where you are in the script using list:

bashdb<1> list
  1: => echo 1
  2:    echo 2
  3:    echo 3
  4:    echo 4
  5:    
bashdb<2>

You can step over the current line using next or n:

bashdb<3> next
1
(/tmp/hello.sh:2):
2:  echo 2
bashdb<4> list
  1:    echo 1
  2: => echo 2
  3:    echo 3
  4:    echo 4
  5:    
bashdb<5>

You can step into the current line using step or s. This is useful to enter into a function call or a line that uses source to call another script:

bashdb<6> step
2
(/tmp/hello.sh:3):
3:  echo 3
bashdb<7> list
  1:    echo 1
  2:    echo 2
  3: => echo 3
  4:    echo 4
  5:    
bashdb<8>

You can step out of the current function with finish.

Other common commands include:

eval STATEMENT - Run a statement.
- Example: eval X=1
print EXPR - Print an expression, such as a variable.
- Example: print $X
break - Set a breakpoint.
- Example: break 3 (set breakpoint on line 3)
- Example: break /tmp/hello.sh:4 (set breakpoint on line 4 of file /tmp/hello)
continue, cont, c - Step continuously until a breakpoint is hit or the script ends.
exit - Stop the script and exit the debugger.

Crystal Web Archiver 1.3.0b Released!

2022-07-10T00:00:00+00:00

Crystal is a website downloader program that is intended to save websites for long-term archival, even after the original site has fallen off the internet.

Today’s release brings the ability to download sites requiring login, better-support for infinite-scrolling sites, a read-only mode, and a CLI shell. See the release notes for more information.

Download Crystal for Windows 7, 8, and 10
Download Crystal for macOS 10.14 and later
- You will need to right-click or Control-click on the application and select “Open” to open it for the first time.
Quickstart guide for downloading your first website

Substack sites in particular can be successfully downloaded with this release of Crystal.

The next release of Crystal will probably be the first final (non-beta) release, with an improved visual appearance, tutorials, and lots of documentation. See the roadmap for more information.

Previous Releases

Crystal 1.1.0b and 1.2.0b

CompressedTextField for Django & MySQL is released!

2022-07-07T00:00:00+00:00

I’m proud to release django-mysql-compressed-fields: a new library that provides a compressed version of Django’s TextField for use with a MySQL ¹ database! This is the first library from TechSmart to be open-sourced. 🎉

In particular you can replace a TextField or CharField like:

from django.db import models

class ProjectTextFile(models.Model):
    content = models.TextField(blank=True)

with:

from django.db import models
from mysql_compressed_fields import CompressedTextField

class ProjectTextFile(models.Model):
    content = CompressedTextField(blank=True)

such that the text value of the field is actually compressed in the database.

String-based lookups are supported:

html_files = ProjectTextFile.objects.filter(content__contains='<html')
html_files = ProjectTextFile.objects.filter(content__startswith='<!DOCTYPE')
html_files = ProjectTextFile.objects.filter(content__endswith='</html>')
empty_html_files = ProjectTextFile.objects.filter(content__in=['', '<html></html>'])

Advanced manipulations with MySQL’s COMPRESS(), UNCOMPRESS(), and UNCOMPRESSED_LENGTH() functions are also supported:

from django.db.models import F
from mysql_compressed_fields import UncompressedLength

files = ProjectTextFile.objects.only('id').annotate(
    content_length=UncompressedLength(F('content'))
)

For more information, including how to migrate to use CompressedTextField, please see the documentation.

My understanding is that the Postgres database - another database type that works well with Django - transparently compresses all text fields automatically, so a library like this one wouldn’t be useful for users of that database.↩

Crystal Web Archiver 1.1.0b Released!

2021-03-23T00:00:00+00:00

Crystal is a website downloader program that is intended to save websites for long-term archival, even after the original site has fallen off the internet.

Today I’m pleased to announce Crystal’s first beta release, bringing support for downloading more complex static sites than ever before, and generally being stable enough for a full public release.

Download Crystal for Windows 7, 8, and 10
Download Crystal for macOS 10.14 and later
- You will need to right-click or Control-click on the application and select “Open” to open it for the first time.
Quickstart guide for downloading your first website

Update: These download links have been updated to point to the 1.2.0b release, which has a number of fixes related to first-time-launch issues and improvements for downloading large websites (with 10,000+ URLs).

Many new types of sites can be successfully downloaded with this release of Crystal, including DeviantArt portfolio websites at *.daportfolio.com (which are scheduled to fall off the internet in about a week after this post), Otaku World, and 16Personalities.

Digital preservation is important

I personally care a lot about digital preservation, which makes it possible to access and enjoy digital content even after it was originally released, substantially beyond the usual lifetime of such works. I’m planning to work on a few projects this year to advance the state of the art in preserving both online sites and old Macintosh programs from the 1990’s. If this topic is also of interest to you, please consider following me on Twitter or RSS. I like hearing from people who have similar interests.

Related Projects

ClassicBox - (Pre-alpha) Makes it easy to play old Mac games from the 1990s on modern hardware, similar to the DOSBox project.

You might not need centralized continuous integration

2021-03-16T00:00:00+00:00

The general availability of easy-to-use centralized continuous integration¹ (CI) solutions in recent years - from GitHub Actions² and Travis as hosted solutions, and from Jenkins³ and Hudson as on-premises solutions - has been wonderful for allowing software to be tested continuously throughout development, catching errors early before changes are merged to shared development and mainline branches.

Today I’d like to explore the now perhaps-controversial idea of “When might it not be worth the effort to setup a centralized continuous integration service?” In particular I’ve somehow managed to not setup a CI server for the primary web app I’ve been developing and running in production even after several years.

The wonders of continuous integration

First, I want to dispel any notion that I’m against the idea of continuous integration in general. Nothing could be further from the truth: Continuous integration - automatically running a project’s test suite against smaller branches in version control that are pending merge to a larger branch - is extremely valuable to detect new bugs as early as possible, before they reach customers and before they even reach other members of your development team. Detecting bugs early helps you ship high-quality software faster.

The trouble with centralized continuous integration

That being said, centralized continuous integration, where there’s a single dedicated service or server that is responsible for running CI jobs, can be complex to setup. If you’re setting up CI for a project that has only a command-line interface, is an API, or is otherwise functional in design, then you should just stop here and go setup centralized CI for your project. For those cases it’s generally not that hard to setup these days. On the other hand if you’re trying to setup CI for a graphical program or something like a website that requires UI testing, then getting CI setup can be much more tricky.

Now it is absolutely possible to get CI setup for a project that requires graphical testing, it’s just a lot of work: You’ll probably need to automate the creation of a VM with the target operating system (through a Docker container for Linux, or a true VM for Mac or Windows), build & install the latest project version from source, install related web browsers and other UI tools, and setup some kind of remote graphical access (through VNC for Linux or Mac, or Remote Desktop for Windows) to debug any test failures that only occur during the CI process.

Alternative: Distributed ⁴ continuous integration

Here’s another idea: You already have a test suite that runs on your development machines right? What if you were to just lean on that existing process for your CI needs?

In particular the thought is that you would make it a policy to regularly run the full test suite at certain intervals, perhaps before merging a feature branch in version control to the development branch. Or perhaps before merging the development branch to the mainline branch released to end-users and production.

How might you enforce such a policy? Well, you likely already have a custom tool that helps you merge & release the development branch to end-users that already runs on your local development machine. What if you just add a step to that tool that runs the test suite, and have the tool only continue if the tests pass? Easy right?

Of course any sufficiently large test suite - especially one interacting with a GUI or a live network - will have at least a few flaky tests that fail occasionally and cannot be fixed immediately. So there should still be a way for a developer to quickly rerun any (flaky) tests that fail so that they hopefully pass, and the developer can manually certify that all tests did eventually pass. To manually certify that all tests did pass on a particular commit, a developer could be required to manually “sign off” on the testing by typing out part of the commit’s hash, perhaps just the first few characters of it.

Putting it all together An example distributed continuous integration system

Given the design above, what might an implementation of a distributed continuous integration system look like in practice? Here’s what it looks like for my primary web app, a Django-based rich web application:

Let’s say I’m on the tip of our development branch (develop) and want to merge into the mainline branch (main) and deploy to production. My Git history might look like:

* a8affa5f7 <David Foster> - (main) Hotfix
| * bd8821196 <David Foster> - (HEAD -> develop) Feature D
| * 26adfda55 <David Foster> - Feature C
|/  
* b005bfee9 <David Foster> - Deploy to production.
|\  
| * 77e0fc2a1 <David Foster> - Feature B
| * d6f29d264 <David Foster> - Feature A
|/ 
* 588e5e61b <David Foster> - Deploy to production.

To merge I’d run our special-purpose mergedeploy command:

$ python3 manage.py mergedeploy
Merging: main <- develop
Pulling trunk and feature branches...
Creating merge commit...
Running tests...
  Log: /var/folders/vm/nd6nhd948xj4ds001s76srj80000gn/T/mergedeploy-fabs0_mt.log
CommandError: Tests failed.
Fix, commit the fix, and run: pm mergedeploy --continue
Or abort with: pm mergedeploy --abort

Inspecting the log I see:

Creating test database for alias 'default'...
System check identified no issues (1 silenced).
......ssssssssssssssssssssssssssssssssssssssssssssF..........
----------------------------------------------------------------------
Ran 61 tests in 16.154s

FAILED (failures=1)
Destroying test database for alias 'default'...

Rerun failed tests with:
$ python3 manage.py test \
    ide.tests.PerformanceTests.test_mouse_move_near_top_of_large_program_is_fast_enough

Looks like there’s a single failing non-deterministic performance test that’s likely to be flaky. Let’s rerun it:

$ python3 manage.py test \
    ide.tests.PerformanceTests.test_mouse_move_near_top_of_large_program_is_fast_enough
Creating test database for alias 'default'...
System check identified no issues (1 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 15.562s

OK

Great it passed! Now all of our tests are passing.

So let’s try the mergedeploy again, using the first few characters of merge commit’s hash as our value to “sign off” that all tests did eventually pass:

$ git rev-parse HEAD
4dff0746ae59de693b7c5cd3a6550cad47a1e58d

$ python3 manage.py mergedeploy --continue --token=4dff07
Skipping tests because token already available.
Committing the merge...
Deploying...

Fetching latest code from Github...
Fast-forwarding deployed code to target branch...
Fetching latest submodules from Github...
Updating virtual environment...
Checking for deployment issues...
Collecting static files...
Compressing JS and CSS assets...
Building deployment archive...
Applying migrations to database...
Operations to perform:
  Apply all migrations: admin, auth, in_school, ...
Running migrations:
  Applying in_school.0073_auto_20210308_1248... OK
Uploading static files...
  Uploading: CACHE
  Uploading: everything else
Uploading deployment archive to AWS and starting deployment...
Optimizing...
  Clearing expired sessions...
  Auditing user permissions...
  Auditing primary keys...
Deployment STARTED

And that’s it! A really easy distributed continuous integration system.

Tests as Policy Automation - Has ideas for creatively using automated tests to enforce additional (non-functional) properties in your application.
Database clamps - Shows how to write deterministic performance tests for database-dependent code.

https://docs.microsoft.com/en-us/azure/devops/learn/what-is-continuous-integration ↩
GitHub Actions is my favorite hosted continution integration solution at the moment, replacing the formerly-excellent Travis, which has become unfriendly to open-source development in recent years.↩
Jenkins is the venerated and popular self-hosted continuous integration server that’s been around for the last decade or so.↩
I call this approach distributed continuous integration because it is distributed across your team’s development machines, rather than relying on being run on a central CI server or “build box”.↩

Reliable rendering of web pages that view concurrently modified data

2021-03-09T00:00:00+00:00

Any time a backend Django or Rails function calculates something complex from the database to send to the frontend as part of a view, there is a chance the database will be modified concurrently before the view is displayed to the visitor, causing the site visitor to see outdated information.

In many cases displaying stale information is fine. After all refreshing the page will bring the latest information to a visitor who is temporarily seeing stale information. But in other cases showing stale information can be a major problem:

For example if the stale data is some editable text that is shared with other site visitors and the new visitor tries to edit it, they’ll clobber the last set of changes made by the previous visitor! Data loss is not OK.
Or perhaps the stale information controls whether the visitor is allowed to access a piece of content that they are actively interested in. For example a user may be requesting to join a chat room and is waiting to be let in. Presenting stale access information saying that the visitor is locked out when they really aren’t will cause them to wait forever and eventually give up! Abandonment is not OK.

How can we avoid serving stale data to the frontend to avoid such problems? This article presents one technique to both avoid serving stale data from the backend in the first place and repair the stale data quickly after page load if necessary.

Although this article is written from the perspective of a Django developer, the concepts apply to any web page that is rendered dynamically based on data from a database or other data source that is subject to concurrent modification.

The Problem

Let’s start by reviewing the problem: Consider a web page that is templated by a backend server such as Django or Rails which uses information from a database. If visitor A reads from the database while templating such a page, but visitor B then alters the part of the database that A read, A will see outdated information when her page finishes loading:

We want to ensure that A sees the freshest data either immediately or as quickly as possible after initial page load.

Designing a Solution

How can visitor A detect the change made by visitor B? One idea is to double-check the freshness of information from the database just before returning information to the frontend:

Although such a strategy does narrow the time window in which a race condition could occur, it is still possible that a concurrent modification is made while the templated page is in transit to the frontend:

Another idea is for the frontend to explicitly double-check the freshness of the information it receives from the backend shortly after it initially renders. If the frontend happens to be displaying outdated information it can repair its state immediately with the new information from the backend:

This strategy is almost perfect (for getting the most up-to-date state reliably to the frontend). However there’s still a narrow time window in which visitor B can make a modification after visitor A’s final request.

What now? To detect these types of concurrent changes it is necessary for the backend to push any new changes to the frontend in real-time (via WebSocket) and the frontend needs to be prepared to receive these kinds of updates to its displayed state continuously:

Note: My last article explains in depth how to setup such real-time updates over WebSocket in Django using Channels. In Rails you’d use ActionCable.

Additionally the backend must be prepared to push even those changes that occur between when the backend templates a view and before the frontend has connected a WebSocket:

Note: Neither Django’s Channels nor Rails' ActionCable provide out-of-the box support for observing events that occur during this critical time period. In the Implementation section below I outline a technique of tracking the “timepoint” an event was generated at so that it can be buffered and delivered later reliably.

Great! We’ve got a bulletproof design to quickly observe an accurate up-to-date state on the frontend. But it does seem to be rather complex…

Alternative Design: No backend templating

It is possible to remove the initial logic that templates information initially on the backend and rely entirely on the WebSocket established after frontend page load to receive both the initial page state and any updates to that state in real-time. Indeed this is the approach taken by many Single Page Applications (SPAs):

However removing backend templating entirely will cause the initial page load to contain no content (beyond an annoying spinner) and raise your time-to-first-render. Content won’t be delivered until after JavaScript is loaded - which can take a while on mobile devices - and after a socket connection is established, disproportionately slowing the browsing experience of any site visitor who isn’t on a high-end device with a fast low-latency internet connection.

I’d like to reach users who are on mobile devices, in rural areas, and from faraway countries where minimizing latency and bandwidth is important to avoid an unacceptable user experience. So we’re back to the more complex design with both backend templating and real-time updates…

Implementation (using Django)

Let’s actually sketch our design for reliable rendering in code!

I’ll use the example of a User Home Page that lists a set of associated Chat Rooms from my last article.

Please review the User, ChatRoom, and ChatRoomJoinRequest models.

Let’s define a cacheable calculation for the set of available rooms given a particular user:

from typing import TypedDict

IntStr = str  # parseable as an int

class RoomInfo(TypedDict):
    id: IntStr
    title: str
    pending: bool

def calculate_chat_room_list(user: User) -> List[RoomInfo]:
    joined_rooms = [
        dict(
            id=str(room.id),
            title=room.title,
            pending=False,
        )
        for room in user.joined_room_set.all()
    ]
    pending_rooms = [
        dict(
            id=str(join_request.room.id),
            title=join_request.room.title,
            pending=True,
        )
        for join_request in ChatRoomJoinRequest.filter(user=user, status='pending')
    ]
    return list(sorted(
        joined_rooms + pending_rooms,
        key=lambda room_info: (room_info['title'], int(room_info['id']))
    ))

Then in the backend view function that templates the page’s HTML, we’ll prepopulate the latest data in that HTML:

# chat_project/chat/views.py

@login_required
def user_home_page(request: HttpRequest) -> HttpResponse:
    ...

    last_updated = UserConsumer.create_timepoint()  # capture
    chat_room_list = calculate_chat_room_list(request.user)  # capture

    return render(request, 'chat/user_home_page.html', dict(
        user_id=str(request.user.id),
        last_updated=last_updated,
        chat_room_list=chat_room_list,
    ))

Then on the frontend JavaScript will wake up and establish a WebSocket connection to the backend, to see if any concurrent changes were made to the room list since the version in the HTML was calculated:

# chat_project/chat/static/chat/user_home_page.js

/*public*/ function setupUserHomePage() {
    const userId = JSON.parse(
        document.querySelector('#user-id').innerText);
    const lastUpdated = JSON.parse(
        document.querySelector('#last-updated').innerText);
    setupUserSocket(userId, lastUpdated);
}

Whenever a user’s room join request is updated, the backend must notify the frontend appropriately through the socket by creating, buffering, and forwarding an event stamped with a new timepoint¹:

# chat_project/chat/models.py

class ChatRoomJoinRequest(models.Model):
    ...  # fields

    def save(self, *args, **kwargs) -> None: ...

    def notify_did_alter(self,
            action: Literal['create', 'admit', 'deny']) -> None:
        ...

        from chat.consumers import UserConsumer  # avoid circular import
        UserConsumer.notify_did_alter_join_request(self, action)

# chat_project/chat/consumers.py

class UserConsumer(WebsocketConsumer):
    ...

    # Send message to group
    @classmethod
    def notify_did_alter_join_request(cls,
            join_request: ChatRoomJoinRequest,
            action: Literal['create', 'admit', 'deny']) -> None:
        update_timepoint = cls.create_timepoint()  # capture
        async_to_sync(get_channel_layer().group_send)(
            cls.user_group_for(join_request.user_id),
            {
                'timepoint': update_timepoint,
                'type': 'did_alter_join_request',
                'kwargs': { ... }
            }
        )

When the frontend first connects a socket to the backend, the backend will attempt to immediately forward any events that were buffered since the timepoint passed in the connect call:

# chat_project/chat/static/chat/sockets.js

/*public*/ function setupUserSocket(userId, lastUpdated) {
    new WebSocketClient(
        '/ws/user/' + userId + '/' + lastUpdated, {
            onmessage: function(e) {
                didReceiveUserSocketMessage(JSON.parse(e.data));
            },
        }
    );
}

function didReceiveUserSocketMessage(data) {
    var type = data['type'];
    var kwargs = data['kwargs'];
    if (type === 'did_alter_join_request') {
        console.log(
            'User socket: Did alter join request: ' + 
            kwargs['action'] + ' ' + kwargs['id']);
        if (kwargs['action'] === 'create') {
            // TODO: Create a new row on User Home Page in the Rooms section
        } else if (kwargs['action'] === 'admit') {
            // TODO: Promote the row in the Rooms section to be a full room,
            //       removing the "Waiting to be admitted" banner
        } else if (kwargs['action'] === 'deny') {
            // TODO: Remove the row in the Rooms section
        }
    } else {
        console.warn(
            'User socket: Did receive unexpected message type: ' + type);
    }
}

That’s it! Hopefully you found this technique useful for reliably rendering the freshest information to visitors, even in the presence of concurrent modifications.

In Django⁶ :

Real-time updates in Django with WebSockets, Channels, and pub-sub
Building web apps with Vue and Django - The Ultimate Guide - Planning how to integrate Vue with a new or existing Django app

A timepoint represents a point in time relative to a shared clock. If you have a single shared Redis instance which you’re already using to forward socket messages, as is the case with Django’s default Channels configuration, it is convenient to use Redis’s TIME command to generate a timepoint, perhaps as part of a stored procedure.↩

Real-time updates in Django with WebSockets, Channels, and pub-sub

2021-03-02T00:00:00+00:00

It’s easy to build a simple chat server in Channels with real-time updates¹ but it’s a bit more complicated to design a system for a more realistic (and complex) data model that has real-time updates. Here, I will show a publish-subscribe (or “pub-sub”) pattern using WebSockets and Channels that can be used by your frontend to watch elements of your backend data model for updates in real-time.

Topics and Messages
Simple Example: Public Chat Rooms
Extended Example: Private Chat Rooms
Implementation in Django
Challenge: Lost events

Topics and Messages

In a publish-subscribe design, frontend pages subscribe to topics that are managed by the backend, by connecting a WebSocket to a wss:// backend endpoint corresponding to the topic.

The backend sends messages to a topic when taking certain actions and those messages are then forwarded on by the topic to all of its subscribers.

Simple Example: Public Chat Rooms

In the context of a simple chat app, there exist Chat Rooms and Chat Messages as models in the database. There also exists a frontend Chat Room Page which is populated with the initial set of Chat Messages on page load. This page wants to observe any new Chat Messages created related to the Chat Room it’s displaying.

To observe newly created Chat Messages, the Chat Room Page subscribes to a Chat Room Topic by immediately opening a WebSocket to the topic endpoint as soon as its JavaScript starts running. That topic receives messages like

{
    'type': 'chat_message_created',
    'kwargs': {
        'id': '42',
        'message': 'Hello world!'
    }
}

which the Chat Room Page then also receives and uses to update its local copy of Chat Messages on the page.

Extended Example: Private Chat Rooms

Let’s say you wanted to extend your chat app such that each chat room is owned by a particular user and that other users must get permission from that owning user before they are allowed to join the room:

Now there are Users, each of which has their own User Home Page which lists the set of Chat Rooms they are already members of and all Chat Rooms that they have submitted a request to join. From this page a user can navigate to a chat room or submit a request to join a new room, perhaps by providing some kind of join code associated with the room.

Additionally the Chat Room Page is extended so that if the person viewing the Chat Room is also the owner of the Chat Room, the page will display not just the list of users already in the room but also a list of users that have created a join request and are waiting to join the room. The owner can then choose to admit a waiting user to the room or to deny a waiting user from entering.

Now things are a bit more complicated:

There is now an additional User Topic that the User Home Page must subscribe to.
There is a new Chat Room Join Request model to track which rooms a user has requested to join. Creating a new request needs to be observed by the Chat Room Page in real-time so that it can display the new waiting user to the room owner. When the owner chooses to admit or deny the join request, that action needs to be observed in real-time on the User Home Page of the requesting user.
A single model (Chat Room Join Request) can have various actions applied to it (create, admit, deny) that are observed by multiple types of pages.
Topics need to authenticate its subscribers:
- In particular the Chat Room Topic will only allow members of the Chat Room to subscribe to it.
- And the User Topic will only allow its related User to subscribe to it.
Some Topics choose to forward certain types of messages to only a subset of their subscribers. In particular the Chat Room Topic should forward Chat Room Join Request messages to the room owner only and not to the other room members.

Implementation in Django

Models, Consumers, and Routes

Here’s a sketch of what an implementation of the above extended chat app might look like in Django with the Channels library.

First we need some models:

# django/contrib/auth/models.py

class User(models.Model): ...  # built-in to Django

# chat_project/chat/models.py
from django.db import models

class ChatRoom(models.Model):
    owner = models.ForeignKey(
        'User', related_name='owned_room_set', on_delete=models.PROTECT)
    title = models.CharField(max_length=50)
    member_set = models.ManyToManyField(
        'User', related_name='joined_room_set', blank=True)

class ChatMessage(models.Model):
    room = models.ForeignKey(
        'ChatRoom', related_name='message_set', on_delete=models.CASCADE)
    sender = models.ForeignKey(
        'User', related_name='message_set' on_delete=models.PROTECT)
    posted = models.DateTimeField(auto_now_add=True)
    message = models.CharField(max_length=200)

class ChatRoomJoinRequest(models.Model):
    user = models.ForeignKey(
        'User', related_name='join_request_set', on_delete=models.CASCADE)
    room = models.ForeignKey(
        'ChatRoom', related_name='join_request_set', on_delete=models.CASCADE)

Then we need some topics. In Django Channels the backend object that manages a topic is called a “consumer”, so let’s create some consumers:

# chat_project/main/routing.py
import chat.routing
from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter
from django.conf.urls import url

application = ProtocolTypeRouter({
    # (http->django views is added by default)
    'websocket': AuthMiddlewareStack(
        URLRouter(
            chat.routing.websocket_urlpatterns +
            # (add websocket_urlpatterns from more apps here)
            []
        )
    ),
})

# chat_project/chat/routing.py
from django.conf.urls import url

from . import consumers

websocket_urlpatterns = [
    url(r'^ws/user/(?P<user_id>\d+)$', consumers.UserConsumer),
    url(r'^ws/room/(?P<chat_room_id>\d+)$', consumers.ChatRoomConsumer),
]

# chat_project/chat/consumers.py
from channels.generic.websocket import WebsocketConsumer

class UserConsumer(WebsocketConsumer): ...

class ChatRoomConsumer(WebsocketConsumer): ...

Connect Frontend to Backend

Let’s subscribe to a topic from the frontend:

// chat_project/chat/static/chat/room.js

/*public*/ function setupChatRoomPage() {
    const roomId = ...;

    setupChatRoomSocket(roomId);  // create early to avoid missing events

    ...
}

// chat_project/chat/static/chat/sockets.js

/*public*/ function setupChatRoomSocket(roomId) {
    new WebSocketClient(
        '/ws/room/' + roomId, {
            onmessage: function(e) {
                didReceiveChatRoomSocketMessage(JSON.parse(e.data));
            },
        }
    );
}

function didReceiveChatRoomSocketMessage(data) {
    ...
}

WebSocketClient opens and manages a WebSocket, transparently handling connect failures and temporary disconnections, retrying with exponential backoff and jitter. Please substitute your favorite WebSocket management library here.

Now let’s fill out the consumer on the backend so that it can accept a connection from the frontend:

# chat_project/chat/consumers.py
from asgiref.sync import async_to_sync
from channels.generic.websocket import WebsocketConsumer
from channels.layers import get_channel_layer
from chat.models import ChatRoom
from django.contrib.auth.models import User
import json

class ChatRoomConsumer(WebsocketConsumer):
    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.user_group = None

    def connect(self) -> None:
        # Authenticate
        if self.user.is_anonymous:
            self.close_during_connect(code=4401)  # Unauthorized
            return

        room_id = self.scope['url_route']['kwargs']['room_id']

        # Identify requested objects
        try:
            self.room = ChatRoom.objects.get(id=room_id)
        except ChatRoom.DoesNotExist:
            self.close_during_connect(code=4404)  # Not Found
            return

        # Authorize
        if (self.user.id != self.room.owner_id and 
                self.user not in self.room.member_set.all()):
            self.close_during_connect(code=4403)  # Forbidden
            return

        # Join group
        self.room_group = self.room_group_for(room_id)
        async_to_sync(self.channel_layer.group_add)(
            self.room_group,
            self.channel_name
        )

        self.accept()

    def close_during_connect(*, code: int) -> None:
        self.accept()  # must accept before closing with custom code
        self.close(code=code)

    def disconnect(self, close_code) -> None: ...

    # === Group ===

    @staticmethod
    def room_group_for(room_id: int) -> str:
        return 'room_%s' % room_id

    # === Utility ===

    @property
    def user(self) -> User:
        return self.scope['user']

In a regular Django view:

We’d normally use the @login_required decorator to authenticate a user but here we need to do that manually by checking user.is_anonymous and returning WS 4401 Unauthorized manually if the user hasn’t logged in.
We’d normally receive parameters from the URL as arguments to the view function but here we have to look up the arguments manually from self.scope['url_route']['kwargs']['PARAMETER_NAME'] instead.
When authorizing we’d normally check properties of request.user against the room ownership and membership, but here we must check self.user (which is a shortcut for self.scope['user']) instead. A failure would normally be reported using an HttpResponseForbidden but here we must manually return a WS 4403 Forbidden code.

At the end of the connection sequence the consumer adds itself to the Channels “group” for all consumers related to the same room topic.

Upon disconnection the consumer also need to unregister from the group:

class ChatRoomConsumer(WebsocketConsumer):
    def __init__(self, *args, **kwargs) -> None: ...

    def connect(self) -> None: ...

    def disconnect(self, close_code) -> None:
        # Leave group
        if self.room_group is not None:
            async_to_sync(self.channel_layer.group_discard)(
                self.room_group,
                self.channel_name
            )

    ...

This consumer does not expect to receive any incoming messages from the frontend; it only expects to send messages to the frontend later:

class ChatRoomConsumer(WebsocketConsumer):
    def __init__(self, *args, **kwargs) -> None: ...

    def connect(self) -> None: ...

    def disconnect(self, close_code) -> None: ...

    # Receive message from page via WebSocket
    def receive(self, text_data) -> None:
        pass

    ...

Backend Can Send Messages to Frontend

Okay now we have the frontend connecting to the backend. How about we trigger an event on the backend such that it gets observed by the frontend in real-time?

Let’s create a helper method on the consumer that sends an event to all other consumers in the same topic, and forwards it on to the frontend through the consumer’s connected socket:

# chat_project/chat/consumers.py
...
from chat.models import ChatRoomJoinRequest
from typing import Literal

class ChatRoomConsumer(WebsocketConsumer):
    def __init__(self, *args, **kwargs) -> None: ...

    def connect(self) -> None: ...

    def disconnect(self, close_code) -> None: ...

    def receive(self, text_data) -> None: ...

    # Send message to group
    @classmethod
    def notify_did_alter_join_request(cls,
            join_request: ChatRoomJoinRequest,
            action: Literal['create', 'admit', 'deny']) -> None:
        async_to_sync(get_channel_layer().group_send)(
            cls.room_group_for(join_request.room_id),
            {
                'type': 'did_alter_join_request',
                'kwargs': {
                    'id': join_request.id,
                    'action': action,
                    'requesting_user': {
                        'id': join_request.user_id,
                        'first_name': join_request.user.first_name,
                        'last_name': join_request.user.last_name,
                    } if action == 'create' else None
                }
            }
        )

    # Receive message from group
    def did_alter_join_request(self, event) -> None:
        kwargs = event['kwargs']

        # Only the room owner may observe join request events so suppress for other users
        if self.user.id == self.room.owner_id:
            # Forward message to page via WebSocket
            self.send(text_data=json.dumps({
                'type': 'did_alter_join_request',
                'kwargs': {
                    'id': str(kwargs['id']),
                    'action': kwargs['action'],
                    'requesting_user': kwargs['requesting_user'],
                }
            }))

Notice that only the room owner (and not its other members) are notified of events related to join-requests, since the room owner is the only one with the ability to admit or deny folks trying to join the room.

Now we need to receive that forwarded event on the frontend:

// chat_project/chat/static/chat/sockets.js

/*public*/ function setupChatRoomSocket(roomId) { ... }

function didReceiveChatRoomSocketMessage(data) {
    var type = data['type'];
    var kwargs = data['kwargs'];
    if (type === 'did_alter_join_request') {
        console.log(
            'Chat room socket: Did alter join request: ' + 
            kwargs['action'] + ' ' + kwargs['id']);

        if (kwargs['action'] === 'create') {
            // TODO: Create a new row on Chat Room Page in the Members section
        } else if (kwargs['action'] === 'admit') {
            // TODO: Promote the row in the Members section to a full member of
            //       the room, removing the "Deny" and "Allow" buttons
        } else if (kwargs['action'] === 'deny') {
            // TODO: Remove the row in the Members section
        }
    } else {
        console.warn(
            'Chat room socket: Did receive unexpected message type: ' + type);
    }
}

We’ve now carved out a path for events related to join-requests to be forwarded all the way to the frontend in real-time.

Backend Triggers Event to Send to Frontend

We still need to actually trigger the event in the backend by calling ChatRoomConsumer.notify_did_alter_join_request() somewhere.

Let’s make a ChatRoomJoinRequest model fire a create event automatically when it is created by an initial save:

# chat_project/chat/models.py

class ChatRoomJoinRequest(models.Model):
    ...  # fields

    def save(self, *args, **kwargs) -> None:
        change = self.id is not None  # capture
        super().save(*args, **kwargs)
        if not change:  # add
            self.notify_did_alter('create')

    def notify_did_alter(self,
            action: Literal['create', 'admit', 'deny']) -> None:
        from chat.consumers import ChatRoomConsumer  # avoid circular import
        ChatRoomConsumer.notify_did_alter_join_request(self, action)

        # NOTE: In the future we'll want to notify the User Home Page here too
        #from chat.consumers import UserConsumer  # avoid circular import
        #UserConsumer.notify_did_alter_join_request(self, action)

Now if a ChatRoomJoinRequest is created in the Django admin site, its creation should be observed in real-time by the Chat Room Page. Great!

Now let’s alter ChatRoomJoinRequest such that an admit or deny action fires an admit or deny event appropriately. We can temporary add a status field to the model, with a “pending”, “admitted”, or “denied” state, and listen for state changes to determine whether an admit or deny action has been taken:

class ChatRoomJoinRequest(models.Model):
    ...  # original fields
    status = models.CharField(
        max_length=10,
        choices=[
            ('pending', '⏳ Pending'),
            ('admitted', '✅ Admitted'),
            ('denied', '✖️ Denied'),
        ],
        default='pending',
    )

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self._initial_status = self.status

    def save(self, *args, **kwargs) -> None:
        change = self.id is not None  # capture
        super().save(*args, **kwargs)
        if not change:  # add
            self.notify_did_alter('create')
        else:  # change
            if self.status != self._initial_status:
                if self.status == 'admitted':
                    self.notify_did_alter('admit')
                elif self.status == 'denied':
                    self.notify_did_alter('deny')

    def notify_did_alter(self, ...): ...

Now if the status field of a ChatRoomJoinRequest is changed in the Django admin site, the related admit or deny event should be observed in real-time by the Chat Room Page. Fantastic!

Now all actions that can be taken on a ChatRoomJoinRequest can be observed by the Chat Room Page in real-time. ✅ But what the User Home Page? It also wants to be notified of events related to join-requests.

Wiring up the User Home Page

Recall that the User Home Page also wants to observe events related to ChatRoomJoinRequests so that it can notify a user whether their request to join a room was approved or denied by the room owner, in real-time.

Similar to how the Chat Room Page observes events in real-time, we’ll need to create a User Topic that is backed by a frontend socket and a backend consumer that manages that socket (UserConsumer).

The code for setupUserSocket is in the same pattern as that for setupChatRoomSocket so I won’t bother listing it.

Ditto for UserConsumer, whose code is similar to that for ChatRoomConsumer.

Now we need to link ChatRoomJoinRequest.notify_did_alter to notify not just the Chat Room Topic of changes, but also the User Topic via UserConsumer.notify_did_alter_join_request:

# chat_project/chat/models.py

class ChatRoomJoinRequest(models.Model):
    ...  # fields

    def save(self, *args, **kwargs) -> None:
        change = self.id is not None  # capture
        super().save(*args, **kwargs)
        if not change:  # add
            self.notify_did_alter('create')

    def notify_did_alter(self,
            action: Literal['create', 'admit', 'deny']) -> None:
        from chat.consumers import ChatRoomConsumer  # avoid circular import
        ChatRoomConsumer.notify_did_alter_join_request(self, action)

        from chat.consumers import UserConsumer  # avoid circular import
        UserConsumer.notify_did_alter_join_request(self, action)

Review of Implementation

Congratulations! You’ve wired up a non-trivial chat server that can admit and deny requests to join private chat rooms! 🎉

Let’s review how a particular sequence of actions might be taken on the backend and observed on the frontend in real-time:

A staff member logs into the Django admin site and creates a ChatRoomJoinRequest object:
- ChatRoomJoinRequest.save() triggers and detects a create action.
- ChatRoomJoinRequest.notify_did_alter() does forward the message to all interested consumers.
- *Consumer.notify_did_alter_join_request() does forward the message to the Channels group corresponding to the * topic (either a Chat Room Topic or a User Topic).
- *Consumer.did_alter_join_request() for all related consumer instances (on all backend servers) does receive the message, and forwards it on to its connected socket.
- didReceive*SocketMessage on either the Chat Room Page or User Home Page does receive the message, and takes action to update the page immediately. ✅
The staff member changes the status field from its default ⏳ Pending value to be ✅ Admitted, and presses the “Save and continue” button on the admin page:
- ChatRoomJoinRequest.save() triggers and detects an admit action.
- (… same intermediate calls as above …)
- didReceive*SocketMessage on either the Chat Room Page or User Home Page does receive the message, and takes action to update the page immediately. ✅
The staff member does press the “Delete” button on the admin page.
- The ChatRoomJoinRequest model is deleted.

Challenge: Lost events

The above implementation does not handle a particular race condition that can occur if an event of interest (such as the creation of a Chat Message) happens while a page (like the Chat Room Page) is loading. Imagine:

User Alice is on the Chat Room Page and has already posted a first message to the room.
User Bob requests the Chat Room Page from Django, which renders to HTML the current set of Chat Messages in the Chat Room. Bob’s Chat Room Page hasn’t yet loaded its JavaScript yet…
Alice posts a second message to the room, which fires a create action to every loaded page that is listening to the room. However because Bob’s page hasn’t finished loading its JavaScript yet, it has not established a socket connection to the backend yet and misses the event.
Bob’s JavaScript finally loads and establishes a socket connection to the backend.
Alice posts a third message to the room, which get observed by Bob, now that his socket is connected.

By the end of this story Alice has posted three messages to the chat room but Bob only sees the first and third message that Alice posted. Bob missed the second message because it was fired in between when Bob started and finished loading the Chat Room Page.

How can we avoid losing events like this? Join me next week as I sketch some solutions to this tricky scenario.

Happy coding!

In Django⁶ :

Building web apps with Vue and Django - The Ultimate Guide - Planning how to integrate Vue with a new or existing Django app
Database clamps - Writing deterministic performance tests for database-dependent code in Django
Tests as Policy Automation - Has ideas for creatively using automated tests to enforce various (non-functional) properties in your Django web app.

The official Channels tutorial has a nice example of building a chat server.↩

Building web apps with Vue and Django (2024) - The Ultimate Guide

2021-02-16T00:00:00+00:00

1 server or 2 servers?
1-server approach
2-server approach
Conclusion

Vue and Django are both fantastic for building modern web apps - bringing declarative functional reactive programming to the frontend, and an integrated web app platform, ecosystem, and battle-hardened ORM to the backend. However they can be somewhat tricky to use together.

Here I’d like to show some approaches for setting Vue and Django up in combination for both new web apps and existing Django-based web apps. I’ve been building web apps with Django for ~9 years and with Vue for ~6 years, and in particular I’ve extensively tested the 1‑server concatenated bundling approach described below in production.

1 server or 2 servers?

The first question to consider when planning to use Django and Vue together is whether to use a single server that serves both backend endpoints and frontend assets, or to use two different servers, one to host the frontend and a separate “API server” to host the backend.

Factors in favor of a 1-server setup:

Operational maintenance costs are significantly simplified with only a single server to deploy, monitor, and manage.
Relegating Django’s role to only that of an API server - in the 2-server setup - will complicate or even prevent you from using any Django features or third-party apps that rely on server-side rendering by Django, such as its famous built-in administration interface. Instead you’ll be doing most of your server-side rendering with some JavaScript framework that is Node-compatible. And Node-based server-side rendering is generally rather complex to setup.

Factors in favor of a 2-server setup:

If you want to use the latest and greatest version of JavaScript, with transpilation and module bundling, the 2-server setup has great community documentation for setting up a frontend server and build pipeline that supports all of these new JavaScript features.
- However most of those features can still be obtained in a single-server setup with some effort, as described later in this article.
If your organization already has separate frontend and backend teams, it may be easier to have 2 servers - one managed by the frontend team, and one managed by the backend team - to allow each team to focus on each server separately (and satisfy Conway’s Law ¹ 🙂).
If your organization already has a separate operations team that has the capacity to manage the additional server implied by a 2-server setup, then the additional operational overhead may be acceptable.

At my company we went with the single server option because our engineering team is small (< 5 engineers) - so we care a lot about minimizing operational overhead - and is composed of full-stack generalists who are familar with both backend and frontend technologies - so we have engineers that can work anywhere in the stack but aggressively prefer simple architechures that don’t require extreme specialization in multiple domains.

In the next section I’ll drill down into the 1-server approach, but you can also skip to the 2-server approach if that’s what you’re leaning toward.

1-server approach

When using a single server, you’ll need to:

pick an asset bundling strategy,
render SEO-friendly HTML server-side with critical navigation and content available immediately on page load, and
enhance that HTML client-side with Vue after page load with additional dynamic content.

Bundling strategies

When using a single integrated Django server to not only host your backend but also serve your frontend, you’ll need to decide how you want to bundle your JavaScript assets on the frontend together. Bundling of some kind is necessary for a production deployment that is fast enough to be mobile-friendly and usable by distant international customers who may not have fast network connectivity to your server.

There are multiple bundling techniques that can be used with Django, with various pros and cons:

Concatenated Bundling,
Import-Traced Bundling, and
Transpiled Bundling.

Concatenated Bundling

This is by far the easiest bundling technique to use with Django and is a good technique to start with.

With concatenated bundling, you write JavaScript files that consist entirely of top-level function definitions and other non-executable code. Your HTML template served by Django contains <script>-includes of every JavaScript file that the current page requires, directly or transitively. And after all JavaScript files are included the HTML page uses an inline <script> to call the root JavaScript function which sets up the rest of the page:

<!-- todo/templates/todo/todo.html -->
<!DOCTYPE html>
<html>
<head>...</head>
<body>...</body>
{% compress js %}
    <script src="{% static 'todo/todo.js' %}"></script>
    <script src="{% static 'todo/todo/list.js' %}"></script>
    <script src="{% static 'todo/todo/item.js' %}"></script>
    <script>
        setupTodoPage();
    </script>
{% endcompress %}
</html>

Note: The {% compress js %} tag above assumes that you are using the excellent Django Compressor library which integrates well with Django as your concatenating bundler.

Pros of concatenated bundling:

Very easy to setup. Leverages excellent documentation from the Django Compressor library.
Zero bundle build times during development.
Fastest bundle build times during deployment, compared with other approaches.²

Cons of concatenated bundling:

You must manage your JS dependencies in HTML manually:
- Whenever adding a new JS module you must remember to add it to the HTML for the appropriate page(s).
- If you alter an existing JS module to depend on a new JS module, you must find all page HTMLs that include the first module and update them to include the second module if needed.
- If you want to avoid managing JS dependencies manually, consider import-traced bundling instead.
JavaScript files are not transformed or transpiled in any way, so if you want to use newer JavaScript features that aren’t supported by your customers' browsers then you’re out of luck. If you need transpilation consider the transpiled or the 2-server approaches instead.
It is awkward to define a class in a JavaScript module that inherits from a base class in a different module, because that requires <script>-including the base class’s module first, breaking the usual rule that “the order that JS files are imported should not matter”. (However if you just limit inheritance to be within a single module or avoid it entirely, then this restriction doesn’t matter in practice.)

Note that choosing concatenated bundling does not prevent you from using TypeScript-based type checking in your JS files, which I do recommend for long-lived projects. In short, you can put a special // @ts-check comment at the top of a JS file to enable type-checking of JS with the TypeScript compiler (tsc).

Import-Traced Bundling

With the advent of the Vite bundler we can get all the benefits of the concatenated bundling approach while eliminating the need to manage JS dependencies manually, at the cost of a slightly-more-complex deployment process.

With import-traced bundling, you write a root JavaScript file for each page that uses regular JavaScript import statements to bring in related modules. Your HTML template then only needs to include the root JavaScript file:

<!-- todo/templates/todo/todo.html -->
<!DOCTYPE html>
<html>
<head>...</head>
<body>...</body>
<!-- js -->
    <script type="module">
        import { setupTodoPage } from "@app/todo/todo.js";
        setupTodoPage();
    </script>
<!-- endjs -->
</html>

The root JavaScript file might look like:

// static/todo/todo.js
import { defineTodoList } from "@app/todo/list.js";

export function setupTodoPage() {
    const app = Vue.createApp(...);
    
    defineTodoList(app);
    
    app.mount(...);
}

And that root JS file can include other modules like:

// static/todo/list.js
import { defineTodoItem } from "@app/todo/item.js";

export function defineTodoList(app) {
    app.component('todo-list', {
        props: {
            ...
        },
        template: `
            ...
        `,
        methods: {
            ...
        }
    });
    
    defineTodoItem(app);
}

Note: The import statements above assume that Vite has been configured to resolve @app to Django’s root static directory in vite.config.js.

Some key differences between import-traced bundling and concatenated bundling:

The HTML page only needs to include the root JS file for the page and not any of its indirect JS dependencies.
JS modules must use import to declare other JS modules that they depend on, and export any functions that they want to be importable by other modules.
JS files for all Django apps in the Django project should be put in a common static directory rather than using per-app static directories. Having a common directory for all JS files will make it easier to configure Vite to build combined JS bundles for production deployments.
- It should still be possible to configure Vite to use app-specific static directories, with some custom build system modifications, but I haven’t yet tried to do so myself.
Django Compressor is no longer necessary.

For a production deployment, you’ll need to alter your deployment script to run Vite on the root static directory for the Django project, enumerating the set of root JS files, and generating same-named files for deployment to your static asset server.

Pros of import-traced bundling:

Easy to setup for development. (Trickier to setup for deployment.)
Zero bundle build times during development.
Fast bundle build times during deployment.
JS dependencies are automatically managed through regular import statements in JS.

Cons of import-traced bundling:

JavaScript files are not transpiled at development time (although they are at deployment time), so if you want to use the most bleeding-edge JavaScript features that aren’t even supported by your development browser then you’re out of luck. If you need transpilation during development consider the transpiled or the 2-server approaches instead.

To see a full example of combining Django and Vite together using the above strategy, take a look at the hello-django-vite prototype I put together.

Transpiled Bundling

2024 Update: The import-traced bundling approach used with newer bundlers like Vite gives all the advantages of transpiled bundling without any of the downsides. Therefore I would not recommend a traditional transpiler approach in 2024.

If you want the latest and greatest JavaScript features that haven’t made it even to your latest development browser (ex: the latest Chrome or Firefox) then you’ll need to pay the cost of needing to transpile during development time:

The transpiled bundling approach generally uses the same kind of HTML, JS, and filesystem structure as the import-traced bundling approach but you have additional flexibility depending on the particular set of bundler and transpiler tools you select.

Common choices for transpiler tools as of mid 2024 are Vite/Rollup, Webpack, Babel, and TypeScript.

Pros of transpiled bundling:

Latest bleeding-edge JavaScript features are available.
JS dependencies are automatically managed.
- You can also manage dependencies automatically with the simpler import-traced bundling approach.

Cons of transpiled bundling:

Moderate-to-slow bundle build times during development.
- You can eliminate build times during development entirely using either the import-traced bundling or concatenated bundling approaches.
Slow bundle build times during deployment, assuming you enable aggressive optimizations.

For a sketch of how you might wire these tools together, take a look at Jacob Kaplan-Moss’s thoughts on the transpiled bundling approach at PyCon 2019.

Render baseline HTML with Django

Now that you’ve picked a bundling approach, we can move on to rendering our first page with Django and Vue.

When a browser (or search engine crawler) first requests a page from your site, it will only be able to immediately render the initial HTML served by Django. In particular, browsers will take some time to start running any JavaScript on your HTML page, so it’s important that the initial HTML served by Django contains your most important page content.

For example if we were building the product page of an online store, we’d want to ensure the initial HTML rendered by Django immediately contained things like:

the site logo and top navigation links,
the product image,
the product description,
placeholders for other less-important panels that are okay to load later in JavaScript, with appropriate animated spinner icons or other loading indicators.

So the HTML rendered by Django might look something like:

<!-- store/templates/store/product.html -->
<!DOCTYPE html>
<html>
<head>...</head>
<body>
    <!-- Top navigation -->
    <nav class="navbar navbar-inverse">
        <div class="container-fluid">
            <div class="navbar-header">
                <!-- Site logo -->
                <a class="navbar-brand" href="/">Acme Store</a>
            </div>
            <div class="navbar-collapse collapse">
                <!-- Top navigation links -->
                <ul class="nav navbar-nav">
                    <li><a href="/computer-systems/">Computer Systems</a></li>
                    <li><a href="/components/">Components</a></li>
                    <li><a href="/electronics/">Electronics</a></li>
                    ...
                </ul>
            </div>
        </div>
    </nav>
    
    <!-- Main content -->
    <div class="container">
        <div class="content-container">
            <!-- Primary panel -->
            <img alt="Product image" href="{{ product.image_url }}" style="float: left;" />
            <h1>{{ product.title }}</h1>
            <div>
                {{ product.description }}
            </div>
            
            <!-- Secondary panel; a placeholder filled out by JS later -->
            <div id="recommendations-panel">
                <h2>Recommended for You</h2>
                <img
                    v-if="loading" 
                    alt="Loading spinner"
                    src="{% static 'store/loading-spinner.gif' %}" />
                <template
                    v-else
                    v-cloak>
                    ...
                </template>
            </div>
        </div>
    </div>
</body>

{{ recommendations_panel_data|json_script:"recommendations-panel-data" }}

{% compress js %}
    <script src="{% static 'store/product.js' %}"></script>
    <script src="{% static 'store/product/recommendations.js' %}"></script>
    <script>
        setupProductPage();
    </script>
{% endcompress %}
</html>

Notice that most of the page is initially rendered server-side with Django’s built-in templating system and not with Vue. This server-side rendered content loads fast and can be appropriately indexed by search engines for better SEO.

However certain less-important panels like the Recommendations panel don’t need to be loaded immediately and will be given life by Vue later after the page’s JavaScript starts running. Let’s consider how that works:

Enhance baseline HTML with Vue

When the browser (or search engine) first loads the Recommendations panel it will initially see just a loading spinner:

<div id="recommendations-panel">
    <h2>Recommended for You</h2>
    <img
        v-if="loading" 
        alt="Loading spinner"
        src="{% static 'store/loading-spinner.gif' %}" />
    <template
        v-else
        v-cloak>
        ...
    </template>
</div>

Note: The v-cloak directive is associated with the CSS rule [v-cloak] { display: none; } so nothing marked by that directive will be displayed initially.

Later when the page’s JavaScript starts running, it will call setupProductPage():

// store/product.js
/*public*/ function setupProductPage() {
    setupRecommendationsPanel();
}

which will call setupRecommendationsPanel():

// store/product/recommendations.js
/*public*/ function setupRecommendationsPanel() {
    Vue.createApp({
        data() {
            return JSON.parse(document.querySelector('#recommendations-panel-data').innerText);
        },
        computed: {
            loading() {
                ...
            },
            ...
        },
        methods: {
            ...
        }
    }).mount('#recommendations-panel');
}

which will use Vue to render the interior of the panel, perhaps after performing an Ajax request back to a Django endpoint to fetch more data.

Notice that some data for the panel can be prepopulated in HTML via the {{ ...|json_script:"..." }} template tag by Django and then fetched later via document.querySelector(...).innerText in JavaScript.

Done! Skip to the conclusion.

2-server approach

When using a separate frontend server for Vue and a separate backend server for Django, you’ll need to:

setup a standalone Vue server and a standalone Django server using official documentation from each project,
alter the frontend Vue server to support server-side rendering or prerendering using official documentation,
create some pages in Vue, some endpoints in Django (perhaps using the Django REST Framework), and have those pages access those routes using plain old window.fetch() (or something shinier like Axios).

Conclusion

Hopefully this guide has been useful in helping you setup Vue inside your new or existing Django web app. Happy coding!

Database clamps - Writing deterministic performance tests for database-dependent code in Django
Tests as Policy Automation - Has ideas for creatively using automated tests to enforce various (non-functional) properties in your Django web app.
Other Django⁶ articles

Related Projects

TechSmart Platform - Large web app that I work on that uses Django and Vue. (Sorry it’s closed-source!)

Conway’s Law: The technical architecture of a system tends to mirror the organizational and communication structure of the people that build it. So if you have 2 teams building a compiler, they’re likely to build a 2-pass compiler. Similarly if you have a frontend and a backend team, they’re likely to build separate frontend and backend servers if left to themselves.↩
At the time of writing it takes 2.1 seconds for me to bundle 100,650 lines (5,304 KiB) of JS and 17,617 lines (699 KiB) of CSS using Django Compressor’s default settings which uses rJSMin to minify JS (via regex no less!).↩

Database clamps: Deterministic performance tests for database-dependent code

2021-02-09T00:00:00+00:00

If you’ve got a moderate-sized Django web application then you’re probably already writing automated tests to make sure none of its pages break unexpectedly when you’re making changes to them. That is, you’re testing page functionality.

However another way that pages can break is that they take too long to display, or otherwise don’t have enough performance. The very first Django application I deployed to customers got crushed with only 12 concurrent users! How embarassing! 🤭 I hadn’t even bothered to do basic performance testing before that initial deployment because I didn’t think it was possible for such a small number of expected users to bring down my site. I now know better and require that any new web page have automated performance tests before being deployed to customers.

There are many kinds of performance tests, but right now I’d like to focus on automated database performance tests, or what I like to call database clamps:

What are they?

A database clamp measures the number of database queries issued when a web page is being rendered server-side. For example:

from django.test import TestCase

class TodoListPageTests(TestCase):
    ...

    def test_todo_list_mdp(self):  # mdp = maintains database performance
        self.client.login(username='user', password='password')
        with self.assertNumQueries(3):  # <-- DATABASE CLAMP
            response = self.client.get(reverse('todo:list'))
            self.assertEqual(200, response.status_code)

Here, assertNumQueries is used to clamp the number of database queries issued when the todo:list page is rendered server-side. If any changes are made to the the page that increases (or otherwise changes) the number of database queries issued, then the test will detect the change and fail.

Why are they useful?

Database clamps are particularly useful for web applications because server-side rendering time is typically dominated by database query time. If you get your database access patterns under control then it’s likely the remaining server-side rendering time will be negligible.

Also, unlike most other kinds of performance tests, database clamps are fully deterministic and always give consistent results no matter how fast the machine the test is being run on. Very useful!

Conclusion

Avoid the embarassment of your site falling over when only a handful of customers try to use it. Use database clamps!

Appendix: Better database clamps

A database clamp which uses Django’s assertNumQueries function will fail not just when the number of database queries increases (which is usually a problem) but will also fail when the number of queries decreases (which is usually okay, and even desirable).

In my own Django web application I use a custom version of assertNumQueries that still fails if the number of queries increases but only issues a warning (via warnings.warn(...)) if the number of queries decreases:

$ python3 manage.py test gradebook
System check identified no issues (0 silenced).
.
----------------------------------------------------------------------
Ran 1 test in 0.758s

OK

Warnings:
gradebook/tests/test_gradebook.py:425: UserWarning: 14 database queries executed, no more than 15 expected. Consider reducing the expected query count to match.

In addition, my version of assertNumQueries expects to be called from an automated test method whose name contains the word mdp (“maintains database performance”) and warns if it is being called from a test lacking that acronym. This restriction allows my engineering team to easily search for and run exactly those tests which use database clamps when making large scale changes that may break many database clamps at once:

$ python3 manage.py test $(python3 manage.py list_tests mdp -s)
System check identified no issues (0 silenced).
..............s..s..................................................
----------------------------------------------------------------------
Ran X tests in Ys

OK (skipped=Z)

Performance Testing - Details a few of the big guns of performance testing.
Tests as Policy Automation - Has more ideas for creatively using automated tests to enforce additional (non-functional) properties in your backend web application.

Tests as Policy Automation

2021-02-02T00:00:00+00:00

Automated tests are usually used for testing functional requirements of your product code. But they can also be used to enforce other policies and coding practices as well.

If writing a web application it’s likely you already have a rule like “all automated tests must pass before any new version of the web application can be deployed to customers on the production environment”. In that case any new policies you want to enforce regularly can be added to your standard automated test suite and they will necessarily have to be satisfied on every deployment!

Here are some example of special policies I’ve enforced from the automated test suite of a large web application I work on (which uses Django, Python+mypy, and JavaScript+TypeScript as core technologies):

Ensure the typechecker reports no errors
- test_type_checker_reports_no_errors_in_python ¹
- test_type_checker_reports_no_errors_in_typed_javascript ²
Ban unsafe coding patterns by inspecting source code
- test_no_new_fragile_test_suites³
- test_ensure_all_directories_containing_py_files_have_init_file⁴
Ensure hard-coded debug modes are turned off
- test_the_debug_toolbar_is_disabled ⁵
- test_that_compress_is_enabled ⁶
Ensure test-only environmental settings are correctly configured
- test_that_tests_do_not_send_real_emails
Ensure invariants that should apply to all types of a large/unbounded number of domain objects are satisfied:
- test_all_django_add_and_edit_admin_pages_render ⁷ 👈 especially useful and powerful
- test_every_block_type_satisfies_all_block_standards ⁸
  - test_c_blocks_for_python_blocks_must_use_consistent_indent_width
  - test_every_field_id_must_use_underscore_case
  - test_every_block_type_whose_codegen_always_references_x_library_must_import_x_library
  - … (9 more)
Prevent certain configuration settings from changing without triggering a discussion with Product Management, the Business, or your Dev Lead
- test_max_redirect_count_is_5 ⁹

Hopefully these examples give you some ideas of some special policies you might enforce in your own automated test suite. Happy coding!

Shitlist Driven Development - Gives techniques for how to effectively apply automated policy changes of the type discussed in this article at large scale.

Database clamps: Deterministic performance tests for database-dependent code - Shows how to clamp the database performance of server-side rendering using automated tests.

TechSmart’s backend Python and Django code is typechecked using the mypy type checker.↩
TechSmart’s frontend JavaScript code is typechecked using the TypeScript compiler, tsc.↩
In this context a “fragile test suite” corresponds to a subclass of Django’s StaticLiveServerTestCase or TestCase whose setUpClass method fails to use a try-finally to invoke super().tearDownClass() explicitly if something goes wrong partway through the test suite setup. This fragile-detection metatest walks through all test suite classes and uses Python’s inspect.getsourcelines to read the source code of all test classes to look for the absense of the proper kind of try-finally.↩
It is important for any directory containing Python source files (*.py) to contain an __init__.py file so that the directory is marked properly as a Python package and is recognized correctly by Python typecheckers like mypy.↩
The Django Debug Toolbar is an amazingly useful tool to profile the database queries that your Django-rendered page is making.↩
TechSmart’s frontend JavaScript code is concatentated and minified using the excellent Django Compressor app. At some point we may migrate instead to using Snowpack, a lightweight module bundler.↩
The “all Django Add and Edit pages must render” metatest automatically discovers all Django models and related pages that exist in Django’s admin site. Then it tries to navigate to each such admin page and ensures that it renders completely. This metatest has caught cases where some of our more complex custom admin pages broke due to a code change elsewhere.↩
The “blocks” referred to here are the lego-like blocks which snap together to form programs in the visual Skylark language.↩
The Python IDE in the TechSmart Platform contains an API-compatible reimplementation of the popular Requests HTTP client library so that students can write programs that access the internet, in a controlled fashion.↩

Python's type checking renaissance

2021-01-26T00:00:00+00:00

You may have heard that TypeScript has been taking the web development space by storm in the last few years, bringing to it static types. I believe the same thing is starting to happen in the world of Python, where type checkers like mypy, Pyre, and Pyright are increasingly used, at least where Python is used by companies to write large systems.

For the last several releases of Python, there have been an increasing number of type checking features added to each release by various PEPs:

Python releases, and related type checking features

Python 3.5 - 3.9

Python 3.5 (Released Sep 2015)
- Type Hints (PEP 484) - The original introduction of type checking type annotations into the Python language.
Python 3.6 (Released Dec 2016)
- Syntax for variable annotations (PEP 526)
Python 3.7 (Released Jun 2018)
- Deferred Evaluation of Annotations with Strings (PEP 563) - Most type checking annotations no longer need to be in runtime context.
- New kinds of types:
  - TypedDict is available in mypy_extensions - Typed dictionaries have specific named keys mapped to specific value types. Such dictionaries are ubiquitous in JSON.
  - Data Classes (PEP 557) - Data classes are analogous to typed dictionaries.
Python 3.8 (Released Oct 2019)
- New kinds of types:
  - TypedDict is standardized (PEP 589)
  - Literal types are standardized (PEP 586)
Python 3.9 (Released Oct 2020)
- Fewer imports from the typing module are needed:
  - Type Hinting Generics In Standard Collections (PEP 585) - Can use list[T], dict[K, V], etc in place of List[T] and Dict[K, V].

Python 3.10 (Released Oct 2021)
- New kinds of types:
  - Parameter Specification Variables: ParamSpec (PEP 612)
  - User-Defined Type Guards: TypeGuard (PEP 647)
- New syntax useful for types:
  - Explicit Type Aliases (PEP 613)
  - ~~Support for indexing with keyword arguments (PEP 637)~~ (Rejected 😞)
- Fewer imports from the typing module are needed:
  - Union types are shortened to X | Y (PEP 604)
    - Optional types shortened to X | None
  - ~~Optional types shortened to X? (PEP 645)~~ (Withdrawn)
Python 3.11 (Scheduled for Oct 2022)
- New kinds of types:
- New syntax useful for types:
  - ~~Callable Type Syntax (PEP 677)~~ (Rejected 😞)
- Extensions to existing typing features:
  - Required[] and NotRequired[] items for TypedDict (PEP 655)
Python 3.12 (Forecasted for Oct 2023)
- New kinds of types:
  - (??) Type Hint for Runtime Type Annotation Objects: TypeForm (planned)
- New syntax useful for types:
  - (??) Deferred Evaluation Of Annotations Using Descriptors (PEP 649) - Most type checking annotations no longer need to be in runtime context.

The traffic on typing-sig, the mailing list where most major new typing features are proposed and designed, has also been seeing increasing traffic year over year.¹

I personally have found type checking to be very useful when working on large Python applications². It helps me find numerous small errors that are introduced during regular development. Sure, most of these errors would be caught by the automated tests that I write for each new feature anyway, but the type checker can find a whole bunch of errors all at once without even running the program, which decreases cycle times when adding new features:

Without type checking I tend to:

Write a bunch of new code.
Repeat 4-6 times, in rapid succession:
- Run program to manually test.
- Find basic error (like a missing import).
- Fix basic error.
Debug/fix deeper errors in the new code.

But with type checking I can:

Write a bunch of new code.
Run the type checker. Get a report of 4-6 basic errors. Fix basic errors immediately.
Debug/fix deeper errors in the new code.

I appreciate this increased productivity that type checking gives me when working on large programs. And I hope that more Python users try type checking for themselves, especially as Python gains an increasing number of related features in recent releases. It’s an exciting time to be a Python developer!

Related Discussion

Hacker News

Related Projects

TypedDict - Python typechecker support for recognizing structured dictionaries with specific named keys mapped to specific value types.
trycast — Parses JSON-like values whose shape is defined by typed dictionaries (TypedDicts) and other standard Python type hints.

Update History

2022-03-29:
- Update final PEPs for Python 3.10, which has now been released.
- Update in-progress PEPs for Python 3.11, which is now in development.
- Update speculative PEPs for Python 3.12, which is in the future.

Increasing traffic on typing-sig is reflected in an increasing number of discussions year-over-year. In 2019 there were 63 discussions. In 2020 there were 113 discussions (79% increase). And 2021 is just beginning.↩
I personally use type checking in a large Django web application that I work on.↩

I no longer trust The Great Suspender

2021-01-20T00:00:00+00:00

I know a number of folks use The Great Suspender to automatically suspend inactive browser tabs in Chrome. Apparently recent versions of this extension have been taken over by a shady anonymous entity and is now flagged by Microsoft as malware. Notably the most recent version of the extension (v7.1.8) has added integrated analytics that can track all of your browsing activity across all sites. Yikes.

Recommendations for users of The Great Suspender (7.1.8):

Temporary easy fix

Disable analytics tracking by opening the extension options for The Great Suspender and checking the box “Automatic deactivation of any kind of tracking”.
Pray that the shady developer doesn’t issue a malicious update to The Great Suspender later. (There’s no sensible way to disable updates of an individual extension.)

Permanent harder fix (👈 Recommended!)

Close as many unneeded tabs as you can.
Unsuspend all remaining tabs. ⏳
- ⚠️ Any tabs that you forget to unsuspend will be lost when uninstalling The Great Suspender in the next step.
Uninstall The Great Suspender.
Download the latest good version of The Great Suspender (7.1.6) from GitHub, and move it to some permanent location outside your Downloads folder. (It should be commit 9730c09.)
Load your downloaded copy as an unpacked extension. (This copy will not auto-update to future untrusted versions of the extension.)
All done! 🎉

Caveat: My understanding is that installing an unpacked extension in this way will cause Chrome to issue a new kind of security prompt every time it is launched, which you’ll have to ignore. 😕 I see no security prompt for using an unpacked extension at least on macOS 10.14 Mojave with Chrome 88 and Developer Mode left on.

Other options

Other browser extensions for suspending tabs exist, as mentioned in the Hacker News discussion for this article. However I have not conducted my own security review on any of those other extensions, so buyer beware.

2021-02-06 Update: Wow. Google has pulled The Great Suspender from its web store. It is still possible to install the latest clean version of The Great Suspender using the “Permanent harder fix” instructions above. Or you might consider one of the alternatives suggested by Lifehacker.

In Productivity¹³ :

In Software⁷⁸ :

How to Design Large Programs with Abstraction and Encapsulation

OS Abstractions are Failing Us

2019-06-27T00:00:00+00:00

In recent years I’ve increasingly noticed software being written that cannot get the performance it needs unless it bypasses usual operating system services.

Concurrency

For example let’s consider a program that wants to do many tasks at the same time. Traditionally you would either create multiple threads or multiple processes for each parallel line of execution. But threads and processes have a lot of overhead - in particular they take up a lot of memory - so it’s unwise to have more than a few hundred of them. So if you have a web service handling hundreds or thousands of connections per second, you cannot effectively serve all those requests on one machine using a thread or process per connection.

So some developers turn to writing their web servers with asynchronous I/O and green threads so that they can multiplex multiple concurrent tasks onto a single OS thread. You can run millions of these green threads concurrently per machine on modern hardware. That’s pretty slick, at least until one of those green threads starts hogging the CPU or accidentally performs a blocking operation. Such a misbehaving green thread will block all other green threads from running, since they use cooperative multitasking and cannot be preempted.

Other developers turn to microthreads in languages like Go, Erlang, or Elixir¹. These environments have an in-process scheduler that multiplexes microthreads onto a single OS thread. Again you can run millions of microthreads concurrently on modern hardware. Happily microthreads can be preempted by the scheduler and so a misbehaving microthread won’t interfere with other microthreads, although it may cause your server to burn CPU wastefully.

Disk I/O throughput

These days our disks are solid-state drives with access times similar to RAM rather than the slower rotating magnetic platters of earlier years. Yet my understanding is that the common filesystem and socket abstractions require per-operation overhead comparable to the time spent actually performing the I/O. Yikes. Research seems to be underway considering ways to bypass various OS abstractions to get faster results.²

Exciting times

It’s neat to be working in computing at the time when some of the fundamental abstractions are being called into question. When practitioners are receptive to change there’s an opportunity to contribute new ideas and actually have them tried and used.

Dependent Types: Impressions of a software practitioner

2019-01-06T00:00:00+00:00

Dependent types are a feature of certain programming language type systems that are unusually powerful, expressive, and precise, compared with other kinds of types. Dependently-typed programming languages such as Coq, Agda, and Idris appear occasionally in academia but not at all in mainstream languages used by software practitioners.

I have been curious about dependent types for some time because of their potential for writing programs that are more correct and have fewer bugs. If you look at my hierarchy of programming languages you’ll notice dependently typed languages way at the bottom beyond even the “high-typed languages”. I even wrote a provably-correct implementation of insertion sort in Idris a few years ago.

Recently my interest in dependent types was rekindled by the release of a new book, The Little Typer, that aims to be an accessible introduction to dependently typed programming. I just finished reading that book. Below are my impressions of the applicability of dependent types to a software practitioner such as myself.

What is a dependent type anyway?

A type describes the shape of a particular value or the shape of a variable that holds values. Being able to talk about the shape of something in a programming language is useful because most kind of operations always want to be given things of the same shape.

There are simple types like Integer, String, and List in all typed languages.

There are generic types like List of Integer and Map of String to Integer that are defined in terms of other types. Generic types are extremely useful for describing collection values (i.e. lists, sets, maps, etc), which are pervasive in programming.

Then there are dependent types like List of Integer of length 5 or IsSorted [1,2,3] that are defined in terms of values. Since values are more precise things than types, a dependent type (defined with a value as a parameter) is a more precise shape than a simple unparameterized type or a generic type (defined with a type as a parameter).

What are dependent types useful for?

More precise types in general

Since dependent types allow defining more precise shapes than other kinds of types, and because type systems can be used by a type checker to verify the consistency of a program, dependent types allow defining functions in a program with more precise input and output restrictions that are automatically checked for correctness by the type checker. Such automatic checking is powerful for providing confidence that the program is correct (i.e. self-consistent).

With dependent types, you can define a known-length collection type such as List of Integer of length 5. This is clearly more precise than List of Integer or List. However I don’t think this additional precision is generally useful: very few functions in my programs care about how long their lists are at compile type. At most some functions would prefer to disallow empty lists.

With the known-length collection type mentioned above, you can further define a function first that takes a non-empty list of values (that the type checker can verify as definitely non-empty) and can guarantee that it will return an output value without having any kind of error. In most languages the best you can do is define such a function that takes any list that will fail at runtime if it is given an empty list. With dependent types you can flag such uses at compile time.

Proofs; Propositions as types

Dependent types allow you to construct proof types such as (IsPositive 5) and their associated proof values. The ability to write arbitrary logical statements as types in a dependently typed programming language is sometimes called “propositions as types”. This capability is particularly useful for human mathematicians, theorem provers, and academics.

It can be useful to use proof types to write functions that not only compute something useful but also return a proof that they calculated the correct value. For example it is possible to write a sorting function that is proven correct in that it returns not only an output sorted list but also a proof that the output list is sorted and also a proof that the output list has the same elements as the input list. That’s pretty cool. Unfortunately these proof values can be devilishly difficult to construct and manipulate.

That’s it?

These are all the applications I’ve been able to discover for dependent types so far. In particular I haven’t been able to find any specific “killer application” of dependent types to any common problem that I see as a software practitioner.

What costs arise from using dependent types?

Complexity

You must learn, remember, and understand several new kinds of language constructs to manipulate dependent types. In particular in addition to regular function application (λ) you must also learn about how to construct and manipulate dependent pairs using Π and Σ. This creates a steeper initial learning curve and constant mental overhead while writing dependently-typed programs.

Verbosity

Dependent types have additional parameters that need to be passed between functions, transformed inside functions, and output by functions. Thus any function manipulating a dependently typed value will require a lot more code to manipulate these additional parameters than an equivalent function using simpler-typed values will not.

More code takes longer to write, provides more opportunities for introducing bugs, and is more time-consuming to maintain.

Lack of encapsulation

Using dependent types frequently seems to involve pattern matching on the low-level structure of values. For example it is common for a function manipulating the natural number type (Nat) to pattern-match on the specific value constructors of Nat, revealing that it is implementated as a 0 base value wrapped inside some number of add1s, similar to a linked list.

If you wanted to change the natural number type to instead be implemented using a more-efficient list of bits, you wouldn’t be able to do so without breaking every function using the original definition of natural numbers.

Restrictions on loops and recursion

Several dependently-typed languages I’ve encountered so far (at least Pie and Agda) disallow function definitions that cannot be proven by the compiler to always terminate, presumably to guard against the type checker going into an infinite loop if presented with a problematic program. In particular this means no generalized loops and no generalized recursion are allowed. These language constructs are extremely useful and it is frequently a pain to work around their absence.

Conclusion

There are no killer apps of dependent types that I can identify for software practitioners, and dependent types have significant costs. Thus in general I feel the costs of using dependent types outweigh the benefits for practical software applications outside the domain of theorem-proving.

Why isn't the external link symbol in Unicode?

2018-11-24T00:00:00+00:00

I have frequently wanted to use the external link symbol ( ) in text that I’m writing, without having to resort to including an image. Normally the solution there would be to find the graphical symbol in Unicode and figure out how to type it.

Apparently at least one proposal for the external link symbol was submitted to the Unicode Consortium as far back as August 2006. It was reviewed at a meeting on November 2006 and it was decided to post a public review.

Later in April 2012 someone noticed that the public review had never been posted because the symbols subcomittee (which was tasked with posting the review) had been folded into the Unicode Technical Committee group and that task had gotten lost in the shuffle.

Then in June 2012 that proposal was finally rejected on the following basis:

2012-2, EXTERNAL LINK SIGN, 2012-June-06

Proposals to encode a character for the “external link sign”, which is often seen as a graphic element indicating a link to a document located external to the website where the page using the external link sign resides. (See L2/06-268, L2/12-143, L2/12-169.)

Disposition: The UTC rejected the proposals to add “external link sign”, most recently in L2/12-169. It is unclear that the entity in question is actually an element of plain text, given the inevitable connection to its function in linking to other documents, and thus its coexistence with markup for links. Furthermore, the existing widespread practice of representing this sign on web pages using images (often specified via CSS styles) would be unlikely to benefit from attempting to encode a character for this image. (This notice of non-approval should not be construed as precluding alternate proposals which might propose encoding a simple shape-based symbol or symbols similar in appearance to the images used for external link signs, should an appropriate plain-text argument for the need to encode such a simple graphic symbol be forthcoming.) References to UTC Minutes: [131-C26], May 10, 2012.

I find this rejection disappointing.

Its main rationale appears to be that the external link icon is not an element of plain text. I would agree that is the case. However I would like to point out that emoji and other similar useful symbols are not plain text either yet they have been accepted and continue to be accepted.

I wonder if the external link symbol would be accepted as an emoji? I don’t have enough energy right now to pursue that myself.

2021 Update: These days I use the ⎋ symbol to approximate the external link symbol in my own informal writing. I’ve also used 🔗 in the past.

The Flat Module Pattern in JavaScript

2018-09-23T00:00:00+00:00

There are several patterns for structuring modules in JavaScript. The most common ones I see talked about is the original JavaScript Module Pattern, AMD Modules, CommonJS Modules, and the emerging Harmony Modules.

The large JavaScript application that I work on at work - an in-browser IDE - does not presently use any of these well-known patterns but instead uses what I’m going to call the Flat Module Pattern.

The Flat Module Pattern

Here’s what the Flat Module Pattern looks like, for an excerpt of a real module called save.js:

/*public*/ function setupSave() {
    if (isMainOrStandaloneWindow()) {
        setupSaveToken();
    }
}

// -----------------------------------------------------------------------------
// Autosave

var DELAY_AFTER_TYPING_STOPS_TO_AUTOSAVE = 2000;

/*public*/ function autosave(fileInfo) {
    ...
}

// -----------------------------------------------------------------------------
// Save Token

function setupSaveToken() {
    ...
}

/*public*/ function isUnsafeToSave() {
    ...
}

// -----------------------------------------------------------------------------

Notice that:

Only function and constant declarations exist at the top-level. No other executable code may run at the top-level.
Public functions are marked with the keyword-comment /*public*/. A module should not attempt to reference a non-public method in a different module.
Modules do not import each other or declare cross-module dependencies explicitly.

Consequences of this pattern:

Modules can be <script>-included in any order, since there are no side-effects of including a module.
All functions are dumped into the global namespace.
- In practice this is not a problem since well-named application functions have distinct names from other application functions, and libraries don’t generally dump functions directly into the global namespace.

All such modules on a particular page are directly included by the HTML page:

{% compress js %}
    <!-- Library -->
    <script src="{% static 'tsk_platform/student_survey.js' %}"></script>
    <script src="{% static 'tsk_platform/util/dialog.js' %}"></script>
    <script src="{% static 'tsk_platform/util/markdown.js' %}"></script>
    <script src="{% static 'tsk_platform/util/slider.js' %}"></script>
    <script src="{% static 'tsk_platform/util/url.js' %}"></script>
    <script src="{% static 'tsk_platform/util/websocket.js' %}"></script>
    <script src="{% static 'tsk_platform/vendor/jquery.min.js' %}"></script>
    <script src="{% static 'tsk_platform/vendor/mdl/material.min.js' %}"></script>

    <!-- Page-Specific -->
    <script src="{% static 'ide/code/__init__.js' %}"></script>
    <script src="{% static 'ide/code/panels/__init__.js' %}"></script>
    <script src="{% static 'ide/code/panels/assignment/__init__.js' %}"></script>
    <script src="{% static 'ide/code/panels/assignment/code_comprehension.js' %}"></script>
    <script src="{% static 'ide/code/panels/assignment/code_writing.js' %}"></script>
    <script src="{% static 'ide/code/panels/code_instructions.js' %}"></script>
    <script src="{% static 'ide/code/panels/project.js' %}"></script>
    <script src="{% static 'ide/code/panels/slides.js' %}"></script>
    <script src="{% static 'ide/code/run/__init__.js' %}"></script>
    <script src="{% static 'ide/code/run/engine/java.js' %}"></script>
    <script src="{% static 'ide/code/run/engine/python.js' %}"></script>
    <script src="{% static 'ide/code/run/run_window.js' %}"></script>
    <script src="{% static 'ide/code/tabs/__init__.js' %}"></script>
    <script src="{% static 'ide/code/tabs/editor/blocks.js' %}"></script>
    <script src="{% static 'ide/code/tabs/editor/font.js' %}"></script>
    <script src="{% static 'ide/code/tabs/editor/image.js' %}"></script>
    <script src="{% static 'ide/code/tabs/editor/scene.js' %}"></script>
    <script src="{% static 'ide/code/tabs/editor/sound.js' %}"></script>
    <script src="{% static 'ide/code/tabs/editor/text.js' %}"></script>
    <script src="{% static 'ide/code/tabs/save.js' %}"></script>
    <script>
        setupCodePage();
    </script>
{% endcompress %}

In considering how modules may reference members of each other:

Page-specific modules may freely reference members from each other and from library modules.
Most library modules are self-contained and do not reference members from anybody, although some library modules may reference members from other library modules.

The Flat Module Pattern is incredibly simple to follow, with almost no boilerplate. It is also easy to concatenate and minify.

My team has been using this pattern successfully over the last 4 years or so with a codebase that is now approximately 55,000 lines of JavaScript. I’m pretty happy with it.

The Encapsulated Flat Module Pattern

The one major drawback of the above Flat Module Pattern is that public functions are advisory-only since they are just marked with /*public*/ comments. It is still possible (and easy) to accidentally improperly reference a non-public function from outside the module it is declared in.

So here’s an enhanced variation I call the Encapsulated Flat Module Pattern that doesn’t have that problem:

(function() {
    function public(func) { window[func.name] = func; }

    public(function setupSave() {
        if (isMainOrStandaloneWindow()) {
            setupSaveToken();
        }
    });

    // -----------------------------------------------------------------------------
    // Autosave

    var DELAY_AFTER_TYPING_STOPS_TO_AUTOSAVE = 2000;

    public(function autosave(fileInfo) {
        ...
    });

    // -----------------------------------------------------------------------------
    // Save Token

    function setupSaveToken() {
        ...
    }

    public(function isUnsafeToSave() {
        ...
    });

    // -----------------------------------------------------------------------------
})();

Notice that each /*public*/ advisory comment has been replaced with a call to a special public(...) function that actually exports the provided function. Any function that is not marked as public is truly private and not available for reference by other modules.

Benefits of this pattern:

Public functions are easy to distinguish from private functions at their definition site.
Public functions and private functions can be freely interleaved. In particular there is no need to group public functions in the module together.
Public and private functions in the same module can call each other naturally and with the same syntax. In particular public functions do not have a special calling convention.
No function names are duplicated by the boilerplate. In particular there’s no need to repeat function names in a public exports list, as is common with other JavaScript module patterns.
It is easy to convert a function from private to public or visa-versa. In particular no references need to be updated since public and private functions are referenced the same way.

Conclusions

I’ve found these JavaScript module patterns to be very useful in structuring JavaScript application code I’ve written over the last few years.

I think these patterns would work well in any large JavaScript application that avoids class inheritance¹ and has reasonable non-colliding public function names.

I do not think these patterns would be a good idea for JavaScript libraries that need to avoid polluting the global namespace. For such libraries, falling back on the Universal Module Definition pattern seems more prudent.

Supporting class inheritance requires serializing the order of top-level definitions since the definition of a subclass depends of the definition of its superclass, and therefore subclasses must be declared after its related superclass. The module patterns in this article do not serialize top-level definition order and therefore cannot reliably support subclasses being declared in a different module than its superclass.↩

Performance Testing

2018-06-02T00:00:00+00:00

In this article I will describe the theory of performance testing¹ and how we conduct such testing in practice at my company TechSmart, in the context of rich web applications and web services.

By the end of this article you should understand how we define performance testing and perhaps get some ideas for implementing or customizing your own tools for performance-testing your own web service.

Theory
Gatling Concepts
- Scenarios and simulations
- Simulation parameters
Load generation
Stress testing
End

Theory

Performance testing of a web service typically involves verifying non-functional requirements such as whether it is:

Responsive - responds quickly to requests
Resilient - handles a large number of requests; avoids crashing outright; rejects or queues requests as needed if too many in progress at once
Fault-Tolerant - whether the service self-heals if it crashes

Performance testing aims to answer the following kinds of questions:

How responsive is the service under a particular amount of load?
(See § “perftest run” below)
What is the maximum load that the service can handle?
(See § “perftest maxload” below)
How responsive is the service under its maximum load?
What is the bottleneck that is saturated when the service is at its maximum load?
(See § “Bottleneck isolation” below)
How does the service behave when stressed beyond its maximum load?
(See § “Overload behavior” below)

The core of performance testing is load generation. You write a program that generates a specified amount of load on your target web service and measure what happens when various amounts of load are applied.

There are many load generation tools that exist:

One of the oldest load generation tools is JMeter, which uses a GUI to define simulation programs in an XML-based file format. JMeter uses a separate OS thread for each concurrent HTTP connection which severely limits the number of concurrent HTTP connections it can host on a single box.
At my company TechSmart we use Gatling as our load generation tool. Compared with JMeter, Gatling is far more efficient at generating load because it uses multiplexed async I/O on a single OS thread to handle all HTTP connections. Also, Gatling simulations are written in an actual programming language (Scala) rather than XML, so you can write your simulations with rich abstractions and do more-advanced customizations.

Gatling Concepts

Scenarios and simulations

A scenario describes a pattern of HTTP requests that a single user makes against a web service. For example in the ViewCodePage scenario a user performs the {LoginPage.loginWithoutRedirect, CodePage.view} subscenarios which consist of individual HTTP requests.

A simulation describes an aggregate pattern of HTTP requests that multiple users make against a web service. For example the ViewCodePage(X, Y) simulation simulates X users arriving over Y seconds that each perform the actions described in the ViewCodePage scenario.

At TechSmart we have written several simulations that exercise each of the major pages on our platform website.

Simulation parameters

Most simulations written at TechSmart vary their behavior based on parameters that are passed in as environment variables. Simulations read these environment variables upon initialization using code like:

class LoginAsStudent extends Simulation {
    val X = sys.env.getOrElse("X", "__missing__").toInt
    val Y = sys.env.getOrElse("Y", "__missing__").toInt

Thus the set of parameters that a particular simulation expects can be deduced by reading the top of the simulation’s source code.

Most simulations at TechSmart support X and Y parameters to inject X users over Y seconds during the simulation.

Load generation

The “gatling” management command

The “gatling” management command is a low-level command we’ve implemented at TechSmart that invokes the Gatling tool, sets up various required paths automatically, and runs a Gatling simulation script.

A typical invocation of the “gatling” management command looks like:

$ X=4 Y=1 pm gatling --simulation tsplatform.LoginAsStudent

Note: The pm command above is an alias for python3 manage.py which is the Django task runner.

This invocation is equivalent to the more-verbose:

$ X=4 Y=1 $GATLING_HOME/bin/gatling.sh --simulation tsplatform.LoginAsStudent --simulations-folder $PERFORMANCE_HOME/simulations --data-folder $PERFORMANCE_HOME/data --bodies-folder $PERFORMANCE_HOME/bodies

The Gatling tool emits lots of output in the console while it is running and eventually generates an HTML report with detailed statistics about what HTTP requests were made during the simulation, response times for individual and aggregated requests, and other information.

The “perftest run” management command

Typically the most important information in the Gatling HTML report generated by running a simulation is the maximum response time² for a particular type of HTTP request that was made during the simulation.

Consequently we’ve created a “perftest run” management command that behaves similarly to the “gatling” command but automatically presents the maximum response time for the most important request type³ after running the simulation.

With the “perftest run” management command, we can easily answer the question:

How responsive is the service under a particular amount of load?
- When R requests/second are made continuously, for some R = X/Y:
  - What is the maximum response time?
  - What does the response time distribution look like?

Here is an example of running a simple simulation, using “perftest setup” to create test data and then using “perftest run” to get the maximum response time for a particular amount of load:

$ pm perftest setup students 8
Deleting test students...
Creating 8 test student(s)...

$ pm perftest setup calendars 1
Deleting test calendars...
Creating 1 test calendar(s)...
Associating 8 user(s) with 1 calendar(s)...

$ pm perftest run LoginAsStudent 4 1
Running simulation with 4 user(s) over 1 second(s)...
When 4 user(s) over 1 second(s), max response time for request 'submit_login' is 393 ms.
  Report: /Users/me/pkgs/gatling-charts-highcharts-bundle-2.2.2/results/loginasstudent-1497049484195/index.html

(We also have a “perftest teardown” command that deletes all test data created by “perftest setup”.)

Targeting remote environments

Our simulations are written by default to target the website running on the developer’s local machine (127.0.0.1). For real testing you’ll want to run tests on a remote version of the website such as the one on a dedicated perf environment (perf.example.com).

For example, to run a command on our performance environment we would first setup the environment with:

$ pm on perf perftest setup students 8
$ pm on perf perftest setup calendars 1

Note: The “on” management command runs some other command in the context of a particular remote environment.

And then we’d run the performance test by typing:

$ pm on perf perftest run LoginAsStudent 4 1

The “on” command sets the GATLING_BASE_URL environment variable (among other things) and the “perftest run” subcommand passes that environment variable to the underlying Gatling simulation. The GATLING_BASE_URL variable specifies the base URL that all HTTP requests are prefixed with. For example the above command is equivalent to:

$ # (Change environment to "perf", defaulting to its database tier, cache tier, etc)
$ X=4 Y=1 GATLING_BASE_URL=http://perf.example.com pm gatling --simulation tsplatform.LoginAsStudent

All simulations support the GATLING_BASE_URL parameter to change the base URL because they all use a common Gatling HTTP Protocol object that defines its base URL from GATLING_BASE_URL:

class LoginAsStudent extends Simulation {
    ...
    val httpProtocol = Common.httpProtocol
    ...
}

object Common {
    private val baseUrl = sys.env.getOrElse(
        "GATLING_BASE_URL", "http://127.0.0.1:8000")

    val httpProtocol = http
        .baseURL(baseUrl)
        ...
}

Stress testing

The “perftest maxload” management command

The “perftest maxload” management command can be used to automatically perform “perftest run” several times to determine the maximum number of users (X) over a given period of time (Y) such that the maximum response time for the HTTP request of interest is less than a particular threshold (1,200 ms by default).

With the “perftest maxload” management command, we can answer the questions:

What is the maximum load that the service can handle?
- What is the maximum requests/second that can be made before the requests start backing up and response times go hockeystick?
How responsive is the service under its maximum load?

An example invocation of “perftest maxload” looks like:

$ pm perftest maxload LoginAsStudent --preserve-calendars
Determining baseline response time...

Deleting test students...
Creating 1 test student(s)...
Associating 1 user(s) with 1 calendar(s)...
Running simulation with 1 user(s) over 5 second(s)...
When 1 user(s) over 5 second(s), max response time for request 'submit_login' is 138 ms.
  Report: /Users/davidf/pkgs/gatling-charts-highcharts-bundle-2.2.2/results/loginasstudent-1497567171908/index.html

Seeking shaft of hockeystick...

Deleting test students...
Creating 2 test student(s)...
Associating 2 user(s) with 1 calendar(s)...
Running simulation with 2 user(s) over 5 second(s)...
When 2 user(s) over 5 second(s), max response time for request 'submit_login' is 124 ms.
  Report: /Users/davidf/pkgs/gatling-charts-highcharts-bundle-2.2.2/results/loginasstudent-1497567182571/index.html

(... ditto for 4 users over 5 seconds ... OK ...)
(... ditto for 8 users over 5 seconds ... OK ...)
(... ditto for 16 users over 5 seconds ... OK ...)
(... ditto for 32 users over 5 seconds ... OK ...)
(... ditto for 64 users over 5 seconds ... OK ...)
(... ditto for 128 users over 5 seconds ... FAIL ...)

Seeking knee of hockeystick... About 6 step(s) or 1:41.

Deleting test students...
Creating 96 test student(s)...
Associating 96 user(s) with 1 calendar(s)...
Running simulation with 96 user(s) over 5 second(s)...
When 96 user(s) over 5 second(s), max response time for request 'submit_login' is 3754 ms.
  Report: /Users/davidf/pkgs/gatling-charts-highcharts-bundle-2.2.2/results/loginasstudent-1497567312703/index.html

(... ditto several times, performing a binary search ...)

num_users,response_time,num_seconds
1,138,5
2,124,5
4,120,5
8,157,5
16,123,5
32,182,5
64,920,5
128,6844,5
96,3754,5
80,2407,5
72,1449,5
68,1227,5
66,1232,5
65,939,5

Maximum load is 65 user(s) over 5 second(s) (13.0 users/second) with a response time of 939 ms.

Notice how “maxload” used a binary search to automatically find the maximum load that the service could handle before the maximum response times went out of bounds. Neat.

Also notice that “maxload” outputs a CSV of every sample taken. This CSV is useful to graph as a scatterplot to see graphically how the response time varies depending on the number of users. We generate these scatterplot graphs often enough that we’ll probably extend “maxload” in the future to just generate a scatterplot image automatically.

Bottleneck isolation

Once you’ve determined the maximum load that your service can support, you can ask yourself whether that load is good enough, based on the level of traffic that you forecast your site will receive in the near future.

Should the maximum load not be good enough, you’ll want to isolate the bottleneck in the system that is constraining the maximum load. Then you can focus on optimizing that bottleneck so that the maximum load increases to be good enough. Note that if you optimize a bottleneck in one area sufficiently, you may find that the bottleneck moves to a different location.

For a web service, the most common bottlenecks in our experience are:

CPU
- Is the CPU usage 100% on some box?
Network
- Are the number of synchronous database requests made by the application tier to the database tier very high (i.e. more than a dozen or so)?
Memory
- Is there no available memory remaining on some box?
- Is there paging activity on some box?
IOPS
- Are the IOPS the maximum for the storage type on some box?
Concurrency Configuration
- Is the maximum number of worker processes active? (Gunicorn)
- Is the maximum number of connections per worker process being reached? (Nginx)
Disk Usage
- Has the maximum quota on the /tmp filesystem been reached?

To isolate the bottleneck, use “perftest run” to start generating a load on the service equivalent to the maximum load it can handle (as measured previously by “perftest maxload”) for a long time interval, say 10 minutes. While that load is being generated, use monitoring tools to look at the usage of CPU, Memory, IOPS, and other resources on each box in the system to look for saturation. When you find the saturated resource, that’s the bottleneck.

Once you’ve located the bottleneck, there are usually a few options to optimize it away. To give you some ideas, here are some bottlenecks that the TechSmart website has hit in its performance testing (from least to most recently):

Not enough frontend workers
- Saturated resource: All frontend Gunicon workers busy 100% of the time, yet not 100% of the CPU utilized
- Fix: Reconfigure Gunicorn to increase the worker count to (2*N + 1), where N is the number of CPU cores.
Too many database requests
- Saturated resource: Frontends busy waiting on the network for dozens of synchronous database queries to complete for a single page load
- Fixes:
  - Optimize frontend logic to reduce the number of database queries per page load to less than a dozen.⁴
  - Use automated tests to clamp the database query count so that it doesn’t increase.
Not enough frontend CPU
- Saturated resource: 100% CPU on each box in the frontend cluster
- Fix: Increase frontend cluster size from 2 up to 5 boxes. At this point the bottleneck moves elsewhere.
Not enough database IOPS
- Saturated resource: IOPS is maximum for the database storage type (i.e. Magnetic or SSD).
- Fixes:
  - Shrink database contents with compression to reduce persisted data volume.
  - Switch database storage type from Magnetic to SSD to increase the maximum IOPS.
- Other potential fixes:
  - Increase RAM on database servers so that the database contents fit into memory, making IOPS irrelevent.
  - Scale reads. Create read replicas of the database.
  - Scale writes. Shard the database to multiple database masters.
Not enough remaining concurrent requests quota
- Saturated resource: Maximum number of concurrent requests to Nginx
- Fix: Increase configured maximum number of concurrent requests.

And here are some bottlenecks that our performance testing has identified we will hit under much higher loads than what we currently experience:

Not enough database CPU
- Saturated resource: 100% CPU on the single master database server
- Potential fixes:
  - Cache more aggressively, so that requests are diverted from the database tier to the cache tier.
  - Scale reads. Create read replicas of the database.
  - Scale writes. Shard the database to multiple database masters.

Overload behavior

It is useful to determine what happens when your service is subjected to greater than its maximum load. In particular:

Does it reject requests loudly?
Does it drop requests silently?
Does it queue requests? Up to a particular soft or hard limit?
Does it crash?
If it crashes, does it automatically restart?

You should decide what desired behavior you want your system to exhibit when receiving a temporary over-maximum load (i.e. a spike) or a sustained over-maximum load (i.e. a flood). Then verify whether the actual behavior matches the desired behavior.

If your service is generally written with infinitely-flexible buffers, it’s likely that receiving sustained over-maximum load will causes requests to queue up until memory is exhausted, and the out-of-memory condition will cause the service to crash.

On the other hand if your service is generally written with fixed-size buffers, it’s likely that receiving any over-maximum load will cause requests to be rejected or dropped.

End

Hopefully this article has provided some insight into concepts around performance testing and given you some ideas about how to implement or improve tooling to perform performance testing.

If you have any improvements or other comments on the contents of this article I’d love to hear from you.

Updates:

2022-02-06:
- Mention more types of bottlenecks I’ve encountered at TechSmart since 2018.
- Bold common patterns of overload behavior.
- Add table of contents for easier navigation.

For the purposes of this article I define performance testing to include load testing and stress testing.↩
Many other performance testing tools focus not on the maximum response time but rather on other measures such as the 99th percentile response time or the mean response time, which are inappropriate to use.↩
The most important request type for a particular simulation is determined by inspecting the source code for a simulation file and looking for a line like val PROFILED_REQUEST_NAME = "view_code".↩
In Django, an easy way to eliminate a bunch of database queries is to make appropriate use of select_related() and prefetch_related() to pre-fetch a group of relationships all at once (with a constant number of queries) rather than one at a time (with a variable number of queries, depending on how many relationships there are).↩

Unsound type systems are still useful

2018-04-07T00:00:00+00:00

The conventional wisdom in the academic community appears to be that a type system is not useful if it cannot be proven to be sound¹.

However as a software practitioner I definitely find unsound type systems to still be quite useful. The principal benefits that I get out of a type system include:

Identification of many common types of errors, in particular:
- misspelled names of functions and variables,
- missing import statements,
- null-pointer dereference errors,
- type mismatch errors, and
- inexhaustive case handling of type codes and algebraic data types.
API documentation that is machine-checked for consistency and correctness that cannot silently get outdated.

Common unsound type systems such as mypy and TypeScript typically provide all of these benefits.

A sound type system can additionally provide a few additional benefits that I don’t miss much in practice:

All runtime types will be consistent with the static types assigned by the type system.

This property permits the elimination of most or all runtime type checks, which can improve the constant-factor performance of programs by a factor of 100 - 1,000x. I don’t miss this type of performance reduction because the programs I write - typically web applications - are usually I/O-bound rather than CPU-bound. Therefore I care far more about the algorithmic performance of programs I write rather than the constant-factor performance.²
It is safe to fully “lean on the type system” to locate and fix consistency errors when performing common refactorings such as renames, moves, and function signature changes.

In absence of a sound type system it becomes necessary to lean on tests instead of the type system to locate and fix consistency errors.

In short, I get a lot of mileage out of unsound type systems, so I would encourage those who research and implement type systems to still consider type systems that are unsound as worthy of study, such as the Gradual Typing family of type systems.

Soundness means that a type system never misjudges the runtime type of a value. In practice this means that (1) the type system defines a static type for all variables and expressions and (2) type casts that override the type system are not permitted.↩
I use the term algorithmic performance to refer to the Big-Oh average case and worst case CPU performance of a particular program. I use the term constant-factor performance to refer to differences in program CPU performance due to differences in constant factors only.
For example in comparing an implementation of merge sort on integers written in a statically-typed language such as C to an implementation written in a dynamically-typed language such as Python, they will be equivalently fast in terms of Big-Oh notation (so they have the same algorithmic performance) but the C version will be faster in terms of constant-factor performance due to having no runtime type checks.
↩

How to implement a large software feature

2018-01-20T00:00:00+00:00

I just started implementing my first big new software feature since the start of the New Year. I thought I’d outline my process since I don’t think I’ve written it down before.

Let’s begin: A feature request comes in. In my case:

Dynamic Calendar for Teachers: Extend our existing Plan page, which shows a calendar of daily activities, such that classroom teachers (our customers) can move activities around on the calendar without engaging our support team over live chat.

I’m working in a large existing codebase and this feature needs to be integrated with existing code. This is a common situation.

1. Review the current behavior

We already have a Dynamic Calendar for Staff feature implemented that our support staff can use to rearrange calendars. It just hasn’t been released directly to teachers because there a few rough edges that need to be polished off first.

Therefore this new feature is primarily a UI change, publicizing existing functionality, along with some backend adjustments to the functionality to make it bulletproof.

I annotate a screenshot of the current Dynamic Calendar dropdown that staff can use to manipulate days on the calendar:

2. Design the new behavior. Write out the delta between the old and new behaviors.

I mock up proposed new UI. In this case I just use a text file:

--------------------
Add empty day / Add teaching day...
Remove empty day
-------------------- ▲ teacher sees
Go to admin page ❏
Add empty day (without relayout)
Remove empty day (without relayout)
Relayout
-------------------- ▲ staff sees

In other cases I actually create a visual mockup using a diagram on paper or a graphics program.

I also write up a summary of behavioral changes:

* Remove teacher actions (because deprecated):
    - Set as non-teaching day
* Add teacher actions:
    - Add teaching day...
        - Needs to prompt for start/end times.
            > Suggest specific options based on existing schedule.
            > Allow full customization as well.
* Publicize and change actions:
    - Add empty day
        - May now perform a relayout. Is slow. Can fail.
        - Adds day at the current day rather than after it.
    - Remove empty day
        - May now perform a relayout. Is slow. Can fail.
        - Continues to require day to be empty of skipped objects too.
          Teacher can now see skipped objects and move them elsewhere
          before removing an otherwise empty day.
* Remove staff actions (because deprecated):
    - Domino step
    - Domino cascade
* Add staff actions:
    - Add empty day (without relayout)
    - Remove empty day (without relayout)
    - Relayout

These behavior changes also imply the addition of some new popups, which I also mock up in another text file.

With the new behavior now clearly written out, it’s time to review it with a member of the User Experience team. They approve the proposed design with some changes.

I also review the new behavior with the member of the Engineering team most familar with the affected subsystems. In this example that person is myself, so there is no further action.

3. Write out engineering tasks that perform the deltas

For this example, here’s a simplified version of the tasks I write out:

[ ] Rip out all DC actions that involve a domino or other magic. #EasyWin
[ ] Merge "can manage" and "can admin" permissions to just "can admin".
=== Familiar with DC logic. Prior system complexity minimized. [1]
[ ] Add "Relayout" action that redirects to the existing "Recreate class calendar suffix" page
[ ] Retitle:
    - Add empty day -> Add empty day (without relayout)
    - Remove empty day -> Remove empty day (without relayout)
[ ] Alter "Add empty day" to add *at* the specified day rather than *after*
=== Staff-facing actions are ready for release [1]
[ ] Add "newdc" feature flag
[ ] Implement new DCD action: Add empty day, behind "newdc" flag
    - ... (details elided)
[ ] Implement new DCD action: Remove empty day, behind "newdc" flag
    - ... (details elided)
=== Teacher-facing actions are ready for release. [3]
[ ] Announce the DC feature to teachers
    - ... (details elided)
[ ] Remove "newdc" feature flag
=== Released. [1]

I may review the engineering tasks with another member of the Engineering team if I uncover any additional details that merit review. In this example I do not.

I update the estimated time for the feature on the Project Management calendar. If the new time estimate has significantly increased then I contact Project Management to see whether any external deadlines are impacted and to negotiate if they are.

4. Perform the engineering tasks

This is the easy part: Develop, test, code review, merge, repeat.

The Trouble with Global Variables

2017-04-09T00:00:00+00:00

You’ve probably heard that global variables are bad. Today I want to explain why they are problematic, with examples, and give some design alternatives where you might be tempted to use a global variable.

What is a global?

A global is a variable from the environment that a function can directly read or write without going through a function parameter.

For example, here is a warn function that adds a warning message to a global warnings list.

warnings = []

def warn(message):
    warnings.append(message)

When is it tempting to use a global?

Globals are tempting to use when there are a large number of related functions that need to read or write a common piece of information.

For example consider a parser that has a root parse_calendar_file function that calls a tree of subordinate functions to parse data from a YAML document into domain objects. Any of these subordinate functions may emit warnings.

Why are globals problematic?

Globals create hidden dependencies that are difficult to discover and easy to propagate.

Globals are hidden

Globals act as a kind of hidden parameter or return value of any function that uses them, which callers may be unaware of. Such global-dependent functions are fragile and easy to misuse.

A function that reads a global expects it to be in a particular state before the function is called. If it is not in the correct state, the function will misbehave. And yet you don’t see the global mentioned in the function’s parameter list.

A function that writes a global relies on the global to return information to the caller or an ancestor of the caller. And yet you don’t see the global mentioned in the function’s return value.

In fact to see whether a function depends on a global you must crack it open and examine its implementation. This breaks encapsulation, requiring you to inspect the function’s private implementation and not just its public interface to discern how to use it properly. Thus functions that manipulate globals are easy to misuse.

Globals are viral

Globals silently infect both direct and indirect callers such that they have a dependency on the global, which new callers may be unaware of. Globals create spooky action at a distance.

In the parsing example above consider what would happen if you were to invoke a subordinate parsing function like parse_bell_schedule directly, without going through the root function that initializes the warnings global. Since the call tree of the subordinate function eventually reaches warn, which expects warnings to be initialized, calling the subordinate function directly will crash!

It is not possible to read only the implementation of the subordinate function parse_bell_schedule to discern that it depends on the warnings global: you have to read the implementation of every function that it calls, directly or indirectly, to see whether it uses the global! This is a massive encapsulation violation of the entire call tree that leads to the global! Every function in the call tree becomes easy to misuse.

Globals are thread-hostile

Functions that depend on mutable globals directly or indirectly cannot be used safely in a multi-threaded program. Attempting to do so will create race conditions, garbage output, and crashes.

Consider what would happen if you had two threads that independently and simultaneously called the root function parse_calendar_file.

Say thread A gets halfway through parsing when B just starts parsing. Thread B will clobber thread A’s version of the warnings global when it starts parsing. As thread A and B continue parsing, warnings from both threads will be mixed together into a nonsensical mishmash. After the first-finishing thread destroys the global, the other thread may crash when it tries to doubly destroy the global. Either way both threads will either return bogus warnings or crash outright.

To make multi-threaded usage of warnings safe, it is necessary for there to be a separate version of warnings owned by each call tree or thread. Globals by definition are global in scope and are not themselves divisible into separate versions per call tree or thread.¹

What can I use instead of a global?

Considering that the big issue with globals is that they are invisible and, well, globally scoped, let’s consider alternatives that are actually visible and more-tightly scoped.

Explicit Parameters

One way to eliminate a global variable is to just pass its value around as an explicit parameter.²

This approach provides no encapsulation around the shared variable and doesn’t allow you to easily add additional shared variables to the same function group. This approach is thus suited to when you explicitly don’t want encapsulation, don’t anticipate adding new shared variables, and don’t want to change the variable itself, such as when you’re passing a Whole Object around.

For our parsing example, we’ll create a warnings parameter on all parsing functions:

Our parsing example really wants some encapsulation around the shared warnings variable and is likely to want more shared variables later (like errors) so the explicit parameter approach is not appropriate here.

Context Objects

A Context Object bundles together a bunch of variables that are shared among a set of related functions as fields. The related functions then take the Context Object as an explicit parameter.

For our parsing example, we’ll create a WarningsContext class which is then passed around among all parsing functions:

It now becomes explicit as to which functions depend on the ability to issue warnings, at the cost of adding noise to the function signatures.

In our parsing example if a new caller was introduced that wanted to call the subordinate parse_bell_schedule function directly it would now be obvious that the caller would need to create and initialize a WarningsContext object first and pass it to the subordinate function. No more crashes. Nice.

Also notice that WarningsContext is sufficiently encapsulated to be usable by not just the calendar file parser code but also reusable by other code that wishes to generate warnings. Cool.

If there is a desire to add another variable that the parsing functions all depend on (like errors), it is easy to add it to the WarningsContext class which is already being passed around everywhere, and then rename the class to something more specific like ParsingContext or CalendarFileParsingContext. However if you are using a very specific context name you probably want a Method Object instead…

Method Objects

A Method Object is like a Context Object but is even more cohesive: it bundles together not only the shared variables but the functions themselves into a single class. This new class is instantiated internally by the root function and lives only for the duration of the original function call.³

For our parsing example we’ll create a CalendarFileParser class:

A Method Object is especially useful when there are many variables that are used by the same set of cohesive functions and these variables are tightly bound to the functions themselves.

If we wanted to add an errors variable to the parsing functions here in addition to warnings, we’d just declare it as another field on CalendarFileParser. Easy.

Ende

Hopefully this discussion has been useful in explaining why global variables are to be avoided and what design techniques can be used instead.

I anticipate the next few articles will continue on the theme of considerations and techniques when designing large software systems. Stay tuned.

You can get around the natural thread-hostility of a global variable by wrapping its value in a thread local. A thread local is a little-known special kind of Cell object that stores an independent value depending on which thread it is accessed from. In Python see threading.local. In Java see java.lang.ThreadLocal.↩
If you need to modify the base of a shared variable when it is being passed around as an explicit parameter then you must also pass back the altered variable through the return value. However this gets unwieldly very fast. In such cases you probably want to either wrap the shared variable in a Cell so that you don’t have to modify the variable itself or use a Context Object. If you truly must use explicit parameters and return values, such as in a functional language that disallows direct mutation of values, consider using the Monad pattern to reduce syntactic overhead.↩
A Method Object gets its name from the fact that it exposes only a single public method and all of the other private methods and fields exist to serve that method.↩

How to Design Large Programs with Abstraction and Encapsulation

2017-03-25T00:00:00+00:00

I spend a lot of time as a professional coder working on very large programs, attempting to grow them while also keeping them from collapsing under their own weight. This is hard.

Abstraction and Encapsulation

The Challenge

A large program has a lot of behavior to specify. The complexity of the behavior specified by a program is roughly proportional to its size.¹

However a coder, being only human, can only hold a fixed amount of concepts in their head at once.

Chunking

The number of concepts can be reduced by chunking them into larger concepts that are less numerous. For example a bundle of lines can be chunked into a single function or a method.

Chunking allows you to reduce the total number of concepts that you need to keep in your head at once.

There are many kind of chunks. The kinds I normally think about are:

Lines
Paragraphs
Functions, Methods²
Classes
Modules³
Packages⁴
Assemblies, Libraries
Binaries
Services

Abstractions

A larger unit that is created by bundling smaller units is called an abstraction. An abstraction typically has:

a name
a public surface area, and
a private interior.

For a function unit, it has a function name, some number of parameters and a return value which serves as the public surface area, and a body of code lines which serve as the private interior.

Encapsulation

An abstraction reduces global complexity by hiding (or encapsulating) its large private interior, exposing only the smaller public surface area to the rest of the system. One only needs to understand the smaller public surface area when using the function and not the details of the private interior.

A well-designed abstraction thus strives to minimize its public surface area while maximizing its private interior.⁵ Such an abstraction takes up only small mental space relative to its interior complexity.

As an example of minimizing public surface area, let’s consider the example of a larger kind of unit, a class:

By extension of the principle above, everything should be private by default; only those methods that need to be public (because they are used externally) should be made public.

If a method on a class is declared public when it is only used internally, you can perform a Refactor Privatize to make it private.

The same principle applies yet again to modules, a larger kind of unit:

Again, everything should be private by default.

Combining Encapsulated Abstractions

Encapsulated abstractions really shine in reducing program complexity when you combine them together.

Without abstractions everything can depend on everything else which creates a potential combinatorial explosion of complexity. By contrast with abstractions, the local complexity is bounded much more tightly.

Within an abstraction the maximum local complexity is proportional to the size of that particular abstraction’s private interior plus the size of the public surface areas of all other abstractions.

Names Matter

It is worth noting that a name by itself serves as an kind of abstraction. A good function name can tell you what the function does without you needing to crack it open and read its implementation. A good class name tells you what it represents, what it is responsible for, what it is, and what it is not.

If you have to look behind a name to figure out what something does, the name needs to be improved via a Refactor Rename. Refactor Rename is probably the most common refactoring I apply out of all the kinds of refactorings I use.

For simple names, coding conventions related to names often can tell you a lot. For example:

Methods:
- setFoo - Sets the “foo” property on a class. Take a single value as a parameter and returns nothing.
- getFoo - Gets the “foo” property on a class. Takes no parameters and returns the value.
- initFoo, setupFoo - Designed to be called once.
- updateFoo - Designed to be called multiple times.
Variables
- curFoo - The current element when iterating over a collection of foos.
- i, j, k - An uninteresting loop counter.
- e, f - An uninteresting exception or event.
- minFoo, maxFoo - A minimum or maximum permissive value for foo, inclusive.
- limFoo - A maximum permissive value for foo, exclusive.
- fooIndex - A 0-based position of something.
- fooOrdinal - A 1-based position of something.
- LOUD_CASE - A constant.

To Be Continued

I’ve got many other techniques for designing large programs. In future articles I hope to share some of these with you.

I’m intentionally oversimplifying: Program complexity actually tends to grow with the square of its size rather than linearly because all pieces of the program can depend on the other pieces and these dependencies contribute to the program complexity.↩
A method is kind of function that is attached to a class.↩
A module contains functions, classes, and variables, among other things.↩
A package is a module that contains submodules.↩
A Philosophy of Software Design makes the same recommendation under the guidance “Modules should be deep”, where a “deep module” is an abstraction whose public surface area is small compared to the volume of its private interior.↩

Glue in Functional Programming Languages

2016-05-21T00:00:00+00:00

Why Functional Programming Matters is a famous paper on the merits of functional programming (FP). It argues that FP has two big special tools for glueing programs together:

Higher-order functions can be easily composed with other functions to create powerful composite functions.
Lazy evaluation allows efficient processing of streams and large data structures.

Higher-Order Functions, meet List Comprehensions

I would like to note that Python — a modern imperative language that also supports functional and object-oriented paradigms — has not only higher order functions but something even better: list comprehensions.

List comprehensions allow very succinct composition of user functions with the most important higher-order functions: map, and filter.

Compare the following paired examples, showing a list comprehension first, followed by a composition of map and filter:

[n * 2 for n in nums]
nums.map(lambda n: n * 2)
[n for n in nums if n % 2 == 0]
nums.filter(lambda n: n % 2 == 0)
[item.name for item in items if item is not None]
items.filter(lambda item: item is not None).map(lambda item: item.name)

I would argue that in all of these cases the list comprehension is both (1) easier to understand and (2) more succinct.

Lazy Evaluation, meet Generators and Coroutines

Lazy evaluation can certainly be very expressive for processing large data structures, compared with greedy evaluation. However I would like to note the following:

Lazy evaluation significantly complicates using a traditional step-by-step debugger tool to step through a program:¹ the current instruction pointer constantly warps up and down the call stack, which can be very confusing.
Lazy evaluation interacts poorly with error handling, I/O, and other side effects.²

This limits lazy evaluation’s advantages in processing large data structures to only those structures which are in-memory or procedurally generated rather than the (IMHO more-common) structures that reside on disk or must be fetched over the network - which must be read with I/O and may be malformed, thus requiring error handling.
Pervasive use of lazy evaluation introduces unnecessary context switching overhead that degrades the constant-factor performance of algorithms for which greedy evaluation would be acceptable. This makes pervasive lazy evaluation unsuitable for certain domains, such as systems domains.

Instead of pervasive lazy evaluation, I find it sufficient to have lazy evaluation on demand in the form of Python’s generators, generator comprehensions, and coroutines when needed.

Presumably this is why Haskell does not ship with a traditional debugger. (Or at least I am not able to locate documentation for one.)↩
Presumably this is one reason why Haskell does not allow unconstrained side effects, although there are many other reasons for having such constraints.↩

Abandonment vs. Unchecked Exceptions for Error Handling

2016-05-17T00:00:00+00:00

Read a very interesting article about error handling in Midori recently, and it got me thinking about errors again. I’ve thought about errors a lot in the past, as you can see in my old Error Handling article from 2013.

Midori mentions a few mechanisms for handling errors:

error codes,
unchecked exceptions,
checked exceptions, and
abandonment.

Midori has chosen to run with

abandonment for bugs, and
checked exceptions for most regular errors,
error codes for other regular errors.

These seem to be reasonable choices given the special constraints of a language intended for systems, namely:

correctness is more important than convenience,
constant-factor performance¹ is important,

However I work mostly in application domains rather than system domains, so I wouldn’t make these same choices. In particular abandonment really gets under my skin.

Abandonment

Abandonment in Midori is used for bugs and serious errors where there is no expectation that the error condition can be handled sensibly. Abandonment tears down the entire process.

I have a few problems with this approach:

Abandonment assumes that the process is the primary isolation boundary.

In a web server, for example, a single request being handled would be a more appropriate boundary. It would be better to abort only the current request rather than bring down the entire web server.
Abandonment provides no opportunity for its error condition to be handled in an alternative fashion.

For example, code that intentionally allocates large blocks of memory on a regular basis may in fact be prepared to handle out of memory conditions, which would normally trigger abandonment.

Granted, you can often support this scenario by bifurcating the API, providing one allocate-memory function that returns an explicit error code on failure and a different allocate-memory function that abandons on failure.

But there is one big advantage to using abandonment:

Better constant-factor performance.

Using abandonment in place of unchecked exceptions means that there is no need to pepper functions everywhere with the low-level code needed to propagate unchecked exceptions. Such code globally degrades constant-factor performance of all functions in the language.

Unchecked Exceptions

In most modern programming languages intended for applications (as opposed to systems), unchecked exceptions are used for reporting bugs. They have the following advantages:

Unchecked exceptions fail fast, terminating the process by default.

This is no different than abandonment so far, however…
Unchecked exceptions can be caught by a higher level error handler.

This additional flexibily is useful for web servers, parsers, and other applications that expect to encounter errors but don’t want to crash entirely in the presence of errors.

But:

Unchecked exceptions degrade constant-factor performance..

However this typically doesn’t matter in application domains. Only algorithmic performance matters.

Conclusion

I think abandonment is a reasonable approach for systems languages but I still prefer unchecked exceptions myself for application languages. I prefer the additional flexibility you get with unchecked exceptions and don’t mind the performance difference.

Error Handling
- Discusses considerations and specific implementation techniques for writing code that is robust in the presence of errors.
Error handling styles in programming
- Summarizes the most prominent strategies for handling runtime errors.

As used in this article, constant-factor performance refers to being sensitive to high constant factors in program time costs. This is in contrast to algorithmic performance which cares primarily about avoiding, for example, an O(n²) algorithm where an O(n) algorithm would suffice.↩

Notes on “The Clean Coder”

2016-04-22T00:00:00+00:00

I recently picked up a copy of The Clean Coder: An excellent book about the non-technical aspects of being a senior software engineer.

Below are my notes taken while reading the book.

Introduction

xvi.

Management sometimes doesn’t view software engineers as professionals in the same way, for example, they treat lawyers as professionals.

In this situation management is likely to babysit the engineers, micromanage them, and be more prone to asking them to sacrifice their personal lives for the current project.

As an engineer this situation is undesirable. This book intends to show you how to present yourself and interact as a professional.

xxii.

Specifically:

What is a software professional?
How does a professional behave?
How does a professional deal with conflict, tight schedules, and unreasonable managers?
When, and how, should a professional say “no”?
How does a professional deal with pressure?

Values

Responsible

10.

“Upon reflection I realized that shipping without testing the routine had been irresponsible. The reason I neglected the test was so I could say I had shipped on time. It was about me saving face. I had not been concerned about the customer, nor about my employer. I had only been concerned about my own reputation.”

“I should have taken responsibility early and told Tom that the tests weren’t complete and that I was not prepared to ship the software on time. That would have been hard, and Tom would have been upset. But no customers would have lost data, and no service managers would have called.”

Accountable

12.

It is the lot of a professional to be accountable for errors even though errors are virtually certain. Practice apologizing.

Apologies are necessary but insufficient. As you mature, your error rate should rapidly decrease toward the asymptote of zero.

Bugs that Escape

12.

QA should find nothing.

If a bug reaches QA, figure out why those bugs managed to escape your notice and do something to prevent it from happening again.

Every time QA, or worse a user, finds a problem, you should be surprised, chagrined, and determined to prevent it from happening again.

Strive for Quality

13.

100% line coverage is a minimum bar of quality, in the opinion of Robert C. Martin.

15.

Merciless refactoring: Every time you look at a module you make small, lightweight changes to it to improve its structure.

Learning & Teaching

16.

Train yourself outside of work. For your career.

Buy books.
Goto conferences.
Spend time each week practicing.
Know the history of computer science. Many gems are there.

20.

Train your juniors! The best way to learn is to teach.

Know Your Business Domain

21.

Know your business domain well enough to be able to recognize and challenge specification errors.

Estimates & Saying No

25.

Stick to your guns RE schedule estimates. Professionals speak truth to power. Professionals have the courage to say no to their managers.

26-28. 30.

Negotiate!

30.

Being a team player means player your position as well as you possibly can, and helping out your teammates when they get into a jam.

32-36.

There is no “trying”.

If you are not holding back some energy in reserve, if you don’t have a new plan, if you aren’t going to change your behavior, and if you are reasonably confident in your original estimate, then promising to try is fundamentally dishonest. You are lying. And you are probably doing it to save face and to avoid a confrontation.

34.

(Passive aggression is when you let someone else hang themselves through inaction on your part.)

How and When to Code

58.

Confidence and error-sense is the key to mastery.

59.

Write code to reveal intent.

Worrying? Tired? Don’t code!

62-64.

The author avoids the flow zone. I find it useful for routine code, and less useful for novel code.

63.

Music can be a plus or minus when writing code. It only works well for me when I’m writing routine code (and not novel code).

63.

Interruptions happen. Try not to be rude.

Refinding your place after an interruption can be tricky. I personally find having an active card useful, as I can just review the card and the checklist on it that I was working through.

Automated Tests

Acceptance Tests

Acceptance tests are a special kind of automated test that specifically test the stakeholder-desired behaviors of a feature. Once the acceptance tests pass, the feature is “done” from the perspective of the stakeholders. This is extremely powerful.

100.

Acceptance tests should be written as a collaboration of the stakeholders and the programmers.

109.

Unit tests and acceptance tests (which are not the same thing) are documents first and tests second. Their primary purpose is to formally document the design, structure, and behavior of the system. The fact that they automatically verify the design, structure, and behavior that they specify is wildly useful, but the specification is the true purpose.

110.

Make sure that all your unit tests and acceptance tests are run several times per day in a continuous integration system.

It is very important to keep the CI tests running at all times. A broken build in the CI system should be viewed as an emergency, a “stop the presses” event.

Other kinds of automated tests

114.

Automated tests can act to:

Specify the desired behavior or properties of a component.
Characterize the current behavior of a component. Useful to detect unexpected changes. Unlike a specifying test, a characterizing test does not worry as much whether the behavior being tested is actually the intended behavior.

116-117.

Automated tests can be classified based on the size of the things they put under test:

Unit tests - Test individual functions or classes.
Component tests - Test individual components or features.
Integration tests - Test how multiple components or features interact.
System tests - Test the entire system. Throughput and performance tests are commonly of this variety.

Time Management

Meetings

There are two truths about meetings:

Meetings are necessary.
Meetings are huge time wasters.

Be very careful about which meetings you attend and which you politely refuse.

One of the most important duties of your manager is to keep you out of meetings.

Have an agenda and a goal: If you are asked to go to a meeting, make sure you know what discussions are on the table, how much time is allotted for them, and what goal is to be achieved. If you can’t get a clear answer on these things, then politely decline to attend.

Focus: A scarce resource

127-128.

Focus is a scarce resource, rather like manna.

It is wise to manage one’s time to take advantage of when focus-manna is available. Focus-manna is a decaying resource so it often needs to be used when it is available.

Sleep is paramount to recharge focus.

Power naps also work well for me personally.

Blind alleys

131.

The Rule of Holes: When you are in one, stop digging.

Messes

132.

Messes slow you down but don’t stop you. Messes are worse than blind alleys because you can always see the way forward, and it always looks shorter than the way back (but it isn’t).

Estimates

138.

Businesses like to view estimates as commitments. Developers like to view estimates as guesses.

A commitment is something you must achieve.
An estimate is a guess. No commitment is implied.

Commitment is about certainty. Other people are going to make plans based on your commitments.

An estimate is not number, despite it often being presented as such. Rather, it is a distribution.

141.

There is a system of estimation called “PERT” or “trivariate analysis”. You give an estimate as a combination of {Optimistic Estimate, Nominal Estimate, Pessimistic Estimate}, which creates a beta distribution. These distributions can be added together to determine the combined time distribution for performing a series of tasks.

141.

Estimating tasks with a group of team-members is more accurate than estimating by oneself. There are many techniques for this. Probably the most famous is “planning poker”. There is also “affinity estimation” and “flying fingers”.

Pressure

153.

You know what you believe by observing yourself in a crisis.

If in a crisis you follow your disciplines, then you truly believe in those disciplines. On the other hand, if you change your behavior in a crisis, then you don’t truly believe in your normal behavior.

Choose disciplines that you feel comfortable following in a crisis. Then follow them all the time.

154.

Avoid creating surprises. Nothing makes people more angry and less rational than surprises.

Teams

168.

When assigning work to folks, there’s no such thing as half a person. You cannot sensibly tell a programmer to devote half their time to project A and half their time to project B.

169.

For a 12-person team, here’s a suggested team composition:

7 devs
2 testers – Who write acceptance tests, focusing on correctness, failure, boundary cases.
2 analysts – Who write acceptance tests, focusing on business value and the happy path.
1 project manager

171.

It’s reasonable to assign multiple projects to a single team. Particularly if the team has gelled.

On the other hand assigning multiple projects to a single person is not a great idea.

Mentoring, Apprenticeship, and Craftsmanship

182.

Software apprenticeship: A very interesting idea. Consider the following levels of proficiency:

Master
Journeymen
Apprentice / Intern

(See the descriptions in the book.)

184.

A craftsman is someone who works quickly, but without rushing, who provides reasonable estimates and meets commitments. A craftsman knows when to say no, but tries hard to say yes. A craftsman is a professional.

School can teach the theory of computer programming. But school does not, and cannot teach the discipline, practice, and skill of being a craftsman.

Roles on Software Teams

2016-04-02T00:00:00+00:00

I currently work at a software startup and wear a lot of hats. Here are some of the hats I wear and some of the hats I interact with.

Product Manager

A product manager envisions new things to create.

Is either a stakeholder proxy or a direct stakeholder:
- Stakeholder Proxy: Represents the desires of external stakeholders, such as customers. This is an outbound role.
- Direct Stakeholder: Represents the desires of internal stakeholders, such as the engineering team. This is an inbound role.
Has an area of focus.
Discovers or envisions new items (within the stakeholder’s area of focus).
Prioritizes items (within the stakeholder’s area of focus).
Talks with the relevant Project Manager to get items scheduled.
Advocates for one’s items and area of focus.

Project Manager

A project manager schedules things to be created.

Knows what all the items in the backlog are.
- Story: A particular small outcome that is meaningful.
- Epic: A large outcome. Associated with a collection of Stories.
  - Specifying Epic: Generates new Stories.
  - Characterizing Epic: Groups together existing Stories.
- Area of Focus: Typically a product. Sometimes a special initiative.
Knows, for each area of focus or important stakeholder, the relative priority of items.
- Talks with Product Managers to obtain relative priorities.
Decides on the absolute priority of items, balancing different areas of focus and stakeholders.
Costs items (indirectly).
- Talks with Engineers to obtain costs.
- Negotiates with Engineers and Product Managers to obtain valuable features with reasonable time cost in a timely manner.
Schedules items to releases and assigns items to Engineers.
Monitors progress of items, looking for slippage.

Engineer

An engineer directly creates new things.

Breaks down items to smaller items.
- More commonly, breaks down a Story to Tasks (or “workitems”).
- Less commonly, reclassifies a Story as an Epic and breaks the Epic down to Stories.
Costs items (directly).
- Negotiates with Project Manager and/or Product Managers, perhaps splitting Stories or Epics across releases, to obtain valuable features and parts of features in a timely manner.
Completes stories by implementing the associated Tasks.
Is sufficiently skilled to do any Task required for the completion of Stories they are assigned.
Is familar enough with the business domain to be able to recognize and challenge specification errors.

There are several kinds of engineers, which vary based on skillset:

Software Engineer (Full-Stack)

Skilled in:
- Frontend application logic. UI implementation.
- Backend application logic. Abstraction design.
- Data(base) schema design.
- Data(base) administration and operations.
Produces:
- Software. Code commits.

UX Engineer

Skilled in:
- Interaction design and usability.
- Graphic design. UI design.
May also be skilled in:
- UI implementation.
Produces:
- UI wireframes, both high-fidelity and low-fidelity.
- Use cases. Scenarios. Interaction designs. Given-When-Then specifications.

Content Engineer

Produces content for the business domain. Skills vary depending on what the domain is.

For example:

Curriculum Engineer - Designs and writes curriculum to teach kids.
Writer - Plans and writes long-form compositions such as technical documentation, manuals, blog posts, or books.

Salesperson

A salesperson finds specific prospects, introduces them to products created by engineering, and converts them to customers.

Sales activity helps to acquire customers.

Marketer

A marketer creates marketing content that creates general awareness for products and product categories among a target audience that contains prospects.

Awareness in the target audience makes it easier for sales to convert individual prospects from the audience to customers.

Marketing activity helps to acquire customers.

Support-person

A support person assists customers (and sometimes prospects) with operating products.

Support activity helps to retain customers.

State of the Union in Programming Languages (2015)

2015-09-13T00:00:00+00:00

Some programming languages are better at some tasks than others. Below I have presented my own assessment of how various languages stack up against each other for the following common classes of tasks:

Programmer-bound: Degree of expressive power. Magnitude of what you can implement with a small development staff. Affected by design simplicity, platform stability, and library availability.
I/O-bound: Suitedness for programs that spend most of their time performing I/O.
CPU-bound: Suitedness for programs that spend most of their time calculating.

Additionally, some programming languages are on the rise or on the decline, for a wide variety of reasons. So I have also provided an assessment of:

Platform longevity: Degree to which the language is likely to attract and retain people and resources.

The rating scale is:

Excellent (+2)
Good (+1)
Okay (0)
Poor (-1)
Fail (-2)

Application Domain / General Purpose

	Python	Java	C#	C	C++
Programmer-bound	Excellent	Okay	Okay¹	Poor	Okay²
I/O-bound	Good	Good	Good	Good	Good
CPU-bound	Poor	Good	Good	Excellent	Excellent
Platform longevity	Good	Okay³	Excellent	Excellent⁴	Good

Web Domain

	JavaScript⁵	PHP⁶	NodeJS	Ruby	Go
Programmer-bound	Okay⁷	Okay⁸	Poor	Good⁹	Okay
I/O-bound	Okay?	Okay?	Excellent	Good	Excellent
CPU-bound	Good¹⁰	Okay?	Good¹¹	Poor	Excellent
Platform longevity	Excellent¹²	Good¹³	Good¹⁴	Good¹⁵	Okay

Other Domains

	Objective-C	Haskell	Perl	Fortran	Lua
Domain	OS X, iOS	Academia	SysAdmin	Computation	Game Mods
Programmer-bound	Good¹⁶	Okay¹⁷	Good¹⁸	?	Good
I/O-bound	Good	Poor¹⁹	Good?	?	Good
CPU-bound	Excellent	Good	Poor?	Excellent	Good
Platform longevity	Good	Good²⁰	Poor²¹	Good	Okay

Other Worthy-of-Consideration Languages

Swift
Clojure - a Lisp dialect
OCaml
Standard ML
Ada
Rust
Idris
Forth

Normally Good, but still leans verbose, similar to Java.↩
Normally Poor, based on the language design, but has a good standard library and massive corporate support for tooling.↩
Normally Good, but Oracle is a poor steward for the language.↩
Cemented as the just-above-assembly language of all CPUs.↩
Specifically JavaScript used on the frontend in a web browser.↩
Is actually a Web Framework masquerading as a Programming Language.↩
Normally Poor, but has a huge number of external libraries and massive corporate support for tooling around inefficiencies in the language.↩
Normally Poor, but has easiest deployment model of any web framework.↩
Normally Excellent, due to massive community support, but degraded by platform instability and lack of language specification, which retards tooling development.↩
Normally Poor, but has highly performant VMs due to massive corporate support.↩
Normally Poor, but has highly performant VMs due to massive corporate support.↩
Cemented as the assembly language of the web browser.↩
Normally Okay, but being propped up by cheap web hosting providers and an army of amateurs.↩
Drafts off of JavaScript’s popularity.↩
Normally Okay, but being propped up by startups.↩
Normally Okay, but has a top-class standard library.↩
Normally Good, but degraded by impenetrable documentation and pervasive speculative generality.↩
Normally Excellent, but degraded by excessive redundant syntax.↩
Normally Good, but degraded by making I/O difficult to perform.↩
Normally Okay, but is being propped up by academia.↩
Normally Fail, but lives on as a preinstalled scripting language for Linux.↩

Algorithms 101 for Software Applications

2015-08-08T00:00:00+00:00

In general I feel that deep algorithm knowledge is overrated in the software industry. In the early 90’s, one needed to know about common algorithms because you needed to actually implement them. Today in the 00’s and 10’s one mainly needs to know which algorithms are appropriate to use, since most common algorithms are already implemented in the standard library of modern programming languages. Therefore I would like to give a summary of the minimum that I think modern coders actually need to know to do their job effectively.

All advice in this article is directly towards those who write applications: primarily rich web applications, desktop applications, and server-side tools. In this context the most advanced type of program you would be asked to write would be a compiler or other program transformation tool.

If on the other hand you work in a low-level environments close to the hardware¹, are performing scientific calculations², or are regularly processing insanely large data sets³, you will need to know more about algorithms that what I cover here.

Collections

In general the main algorithmic work you’ll be doing when writing programs will involve the use of collections included in your language’s standard library. Collections hold values. The algorithms for manipulating these collections will also typically be included in the standard library.

Basic collections

The following kinds of collections are so common as to be fundamental. Practically all programs involve the efficient manipulation of data stored in these collections.

List - An ordered sequence of elements.
Set - An unordered set of elements, disallowing duplicates.
Map - A structure that associates keys with values.

Sets and Maps in particular come in different flavors depending on what kind of ordering guarantee (if any) they provide:

Unordered Set/Map
- Elements can be in any order.
Stable Set/Map
- Elements appear in the order that they were added.
Sorted Set/Map
- Elements are always sorted via either their natural ordering or a custom comparison function.

Other useful more-advanced collections exist such as Bidirectional Maps, Counted Bags, Partitioning Bags, and Priority Queues, but they will not be considered here.⁴

Basic operations

All collections support operations for adding elements, removing elements, and iterating over elements.⁵ However they do so at different speeds. It is important to choose the best kind of collection to use based on which operations your program will perform on it most frequently.

Below are the operations that each collection type supports most efficiently, along with the operation’s speed in Big-Oh notation⁶ when using the best⁷ implementation available:

List
- Append element to end - O(1)
- Get element at random index - O(1)
- Set element at random index to new element - O(1)
- Iterate over all elements - O(n)
- Contains a specific element? - O(n)! (consider using a Set instead)
- Remove first occurrence of an element - O(n)! (consider using a Bag instead)
- Remove all occurrences of an element - O(n)! (consider using a Set instead)
Set
- Add element (if not already present) - O(1)†
- Remove element - O(1)†
- Contains a specific element? - O(1)†
- Iterate over all elements - O(n)
Map
- Set value associated with key - O(1)†
- Get value associated with key - O(1)†
- Contains a specific key? - O(1)†
- Iterate over all key-value pairs - O(n)

† = Assumes using an Unordered or Stable implementation with a well-designed hash function. If using a Sorted implementation then performance degrades from O(1) to O(log(n)). If using a poor hash function then performance degrades from O(1) to as low as O(n).

Common algorithms exist for performing these operations. As mentioned previously it is rare that you will need to code such collection-manipulating algorithms manually unless you are working in a constrained environment. Instead you just need to be a smart shopper when picking a collection implementation to use from your language’s standard library.

Common collection implementations

Generally speaking the collections available in the standard library of a programming language will be include one or more of the following specific implementations, possibly using different names:

List
- Array-Based List (recommended)
  - O(1) amortized⁸ append
  - O(1) random get and set
  - But O(n) insert, remove, and prepend
- Linked List
  - O(1) prepend
  - O(1) append if tail pointer is tracked, otherwise O(n) append
  - But O(n) random get, set, insert, and remove
  - But causes memory fragmentation
- Array
  - O(1) random get and set
  - But does not support resizing after construction
Set
- Linked Hash Set (recommended) - a stable set
  - O(1) add, remove, and contains
  - O(n) iteration in consistent (but not sorted) order
- Hash Set - an unsorted set
  - O(1) add, remove, and contains
  - O(n) iteration, but in random order
- Tree Set - a sorted set
  - O(log(n)) add, remove, and contains
  - O(n) iteration in sorted order
Map
- Set-Based Map (recommended)
  - Uses an underlying set to hold the keys.
  - O(1) get, set, and contains-key for stable and unsorted sets
  - O(log(n)) get, set, and contains-key for sorted sets
  - O(n) iteration and contains-value
- Association List
  - Is just a list of key-value pairs.
  - O(n) iteration and contains-value
  - But O(n) get, set, and contain-key

Here is a table of implementations for various common programming languages:

Sorting

The only advanced operation you commonly need to perform on a collection is to sort it.

The following sorting algorithms are worth knowing about for actual use:

Insertion Sort
- O(n²) average and worst case performance
- Usually faster in practice than other algorithms for small numbers of elements.
Merge Sort
- O(n⋅log(n)) average and worse case performance, optimal for a comparison-based sort.
- Can be done on disk rather than in RAM, for very large data sets.
Quicksort
- O(n⋅log(n)) average case performance
- But O(n²) worst case performance
- But is not a stable sort
Heap Sort
- O(n⋅log(n)) average and worse case performance, optimal for a comparison-based sort.
- But is not a stable sort
- But is complex to implement since it involves heaps.

It’s also worth knowing that there are even faster sort algorithms that are not based on comparisons. For example:

Radix Sort
- O(w⋅n) average and worst case performance, where w is usually a constant

In interviews, knowing about and being able to implement merge sort, quicksort, and insertion sort has always been sufficient for me.

Standard libraries vary in what specific sorting algorithms they offer:

Python uses Timsort, a hybrid of Merge Sort and Insertion Sort.
Java previously used either Merge Sort or Insertion Sort depending on the size of the input list but later switched to Timsort.
Perl previously used Quicksort but later switched to Merge Sort.
C++ does not mandate a particular algorithm but does specify that whatever algorithm provided will have worst case performance of O(n⋅log(n)) or better.

Trees

Beyond collections, it’s useful to know how to both to implement tree data structures - particularly N-Ary Trees - how to traverse them, and how to manipulate them.

N-Ary Trees are made of Nodes, starting at a root node, where each node contains a list of children nodes. A node with no children is a leaf. A node can be related to another node as a child, parent, descendant, ancestor, or cousin. A node may or may not track who its parent is, depending on how it is implemented.

Trees can be traversed in either Depth-First Order or Breadth-First Order. Most programs use Depth-First Order because it is easy to implement via recursion. Breadth First-Order traversal may be useful, however, in writing algorithms related to the shortest-path from the tree root.

In interviews, I’ve found useful the ability to improvise new tree-related algorithms. For example an algorithm to find the nearest common ancestor.

Graphs

It is rare that you will need to use graphs in real-world application development except for the most demanding kinds of applications, such as compilers and program transformation tools. Therefore I will only cover them briefly here.

A Graph is made of Nodes (sometimes called “vertices”) and Edges. Edges connect pairs of nodes together. Beyond that:

In a “directed” graph (or “digraph”), each edge has a head and tail. By contrast in an “undirected” graph, each edge is unoriented and just joins two nodes together.
Some graphs are “weighted”, meaning that nodes and/or edges are annotated with numerical weights.
Some graphs are “multi-graphs”, meaning that they explicitly allow multiple edges between the same pair of nodes.

Graphs are useful for representing and manipulating connected structures, such as the intersections and roads on a map, the set of statements and related next-statements in source code, and the set of squares on a game board plus its neighbors.

Algorithms for working with graphs are similar to those for working with trees but are more complicated, since each node in a graph can be typically be reached along multiple paths.

Knowing how to traverse a graph in both Depth-First Order and Breadth-First Order without revisiting a node twice is extremely useful, both in practice and for interviews. For example how could you implement the paint bucket tool in a drawing program, coloring in a region of neighboring pixels without coloring in any pixel twice?

Odds and Ends

There are a few other miscellaneous concerns worth mentioning:

Strings in many languages (Java, Python, JavaScript) are immutable. Therefore appending to a string takes O(n) time. By itself this is not a problem, however you need to be careful not to repeatedly append to a string inside a loop because then your runtime will be closer to O(n²). Instead, append the string parts to a temporary list and then join the list together afterwards using the appropriate standard library function. That will get you back to O(n) time.
Implementing binary search in a list correctly is a lot harder than you think. The first couple of attempts in the academic literature actually had flawed algorithms.
Algorithms involving array rearrangement and index manipulation are good to practice for interviews, since some interviewers ask for low-level tricky algorithms of this sort. For example the Dutch National Flag problem is a classic.

Series

This article is part of the Programming for Perfectionists series.

Low-level environments typically lack rich standard libraries. So you’re back to implementing linked lists and friends by hand.↩
Scientific calculations involve lots of floats, which have a host of additional considerations to worry about. Chiefly float imprecision and numerical stability. The data sets involved are also typically very large.↩
Web services that have the popularity of Facebook, Amazon, Apple, Microsoft, or Google are dealing with unbelievable data volumes that merit using specialized algorithms. Other websites, although perhaps still popular, don’t receive anywhere near the same level of traffic.↩
Further, other useful more-advanced non-collection data structures exist such as Trees, Directed Graphs, and Undirected Graphs, but they also will not be covered here.↩
The runtimes quoted here are for operations that are performed in-place, modifying the original collection value. If you require operations that efficiently return modified copies of the original collection, look into persistent data structures, such as used by Rich Hickey in the Clojure standard library.↩
In Big-Oh notation, O(1) is a constant amount of time, O(log(n)) is time proportional to the order of magnitude of the collection’s size, and O(n) is time proportional to the size of the collection. Other common runtimes include O(n⋅log(n)) and O(n²).↩
Considerably worse performance can result if you use a poor implementation. For example if you attempt to append (not prepend) an element to a list in a functional language that uses the default singly-linked list implementation, it will take O(n) time rather than the usual O(1) amortized time.↩
O(1) amortized time means that in aggregate the per-operation time cost will average out to O(1). However individual operations may take O(1) in the average case or O(n) in the worst case.↩

Unicode 101

2015-06-27T00:00:00+00:00

Handling international and Unicode text correctly in modern programming languages remains a poorly understood topic.

Optional Structural Static Typing in Python

2015-04-26T00:00:00+00:00

I’m working on a new project to build a typechecking linter for Python, in the same vein as mypy. The big difference between mypy and what I’m hoping to build is that my type system will be a structural type system rather than a nominal type system.

Structural vs. Nominal Type Systems

Let’s consider the following program:

interface Foo { public int length(); }
interface Bar { public int length(); }

Foo foo = ...
Bar bar = foo;   // Will this work?

In a structural type system (ex: Go, OCaml) the assignment would be allowed because both interfaces have exactly the same set of methods.¹ If it looks like a duck then it is a duck.

In a nominal type system (ex: Java, C#) the assignment would be rejected because there is no declared subtype relationship between the names Foo and Bar. The actual structure of the interfaces is completely ignored.

Why not Nominal Typing?

It generally requires explicit type annotations for all function parameters, and sometimes function return types too. I don’t want to write any type annotations for the majority of programs that I write, and certainly not annotations for every single function.

There is a serious risk that Python code written with a nominal type system in mind will start introducing large numbers of new abstract base classes for the sole purpose of making the typechecker happy. This is exactly the kind of pollution that I’m trying to escape from when working in dynamic languages such as Python.

Why Structural Typing?

The philosophy behind structural typing meshes a lot better with the “duck typing” that you see in classic Python with no type annotations. If you try to invoke a method on an object and the object has a method with a matching name, the program will work. This is exactly the properly that a structural type system checks.

Loosely defined protocols such as “file-like objects” continue to work.
More complex protocols such as “file-like objects that support fileno()” (as needed by the subprocess module) also work without needing to introduce further abstract base classes.
Lightweight JSON-based objects continue to work and benefit from type checking.

More importantly, structural type systems are much more amenable to global type inference, which can be used to eliminate the need for explicit type annotations. This would allow Python programs to still be type-checked without the need to specify type annotations in the usual case. Imagine if your programs today, without any type annotations, could suddenly become type checked with no additional effort from you, the programmer.

Challenges with Structural Typing

Of course structural typing isn’t all good. It has one major sticking point: Structural type systems tend to generate huge verbose error messages that aren’t particularly actionable.

This happens because the type checker only knows that two (complex) inferred types are incompatible. It doesn’t know which of the two types is correct. And the error messages containing the derivation of the inferred types typically involves lots of locations in the code unrelated to where the actual error lies. Imagine the same level of verbosity as C++ template errors.

Mission: Difficult

So in summary the goal is to make a type checking linter for Python with a structural type system and global type inference which nevertheless presents comprehensible and actionable error messages in a majority of error scenarios.

I see some original research in my near future.

References

Uncovering the Unknown: Principles of Type Inference (1h3m)
- Excellent presentation outlining the difference between nominal vs. structural subtyping, typing inference, and some pros/cons between both kinds of type systems.
Types and Programming Languages (645 pages)
- Excellent broad overview of type systems and type theory. Leans toward covering type systems similar to those used in mainstream programming languages. Has only limited information about type reconstruction and type inference.

To be more precise, the assignment would be allowed because Foo structure contains everything that Bar requires: the methods on Bar are a subset of the methods on Foo.↩

Semantic Versioning vs. Romantic Versioning

2015-03-14T00:00:00+00:00

In recent years the larger software development community has started to pay more attention to version numbers.

Semantic Versioning

The Node community in particular has introduced specific guidelines for what it means when you increment various parts of a version number. Generally:

A version number is a 3-part value X.Y.Z where X, Y, and Z are the major, minor, and patch components respectively.
When making a new release that contains bug fixes only, you may increment the patch component.
- X.Y.Z -> X.Y.(Z+1)
When making a new release that contains new features (and possibly some bug fixes) but not breaking changes, you may increment the minor component (and reset the patch component to zero).
- X.Y.Z -> X.(Y+1).0
When making a new release that contains breaking changes (and possibly some new features or bug fixes), you may increment the major component (and reset both the minor and patch components to zero).
- X.Y.Z -> (X+1).0.0

This scheme for version numbering, called semantic versioning (or SemVer), is easy for automatic dependency management systems to understand, making it highly useful for communities that encourage liberal use of third-party libraries.

When to use Semantic Versioning

Use of semantic versioning is highly recommended for projects that are intended to be depended on and automatically upgraded, such as:

libraries (ex: jQuery, Underscore, requests, libxml2).

Semantic versioning is also useful for slower-moving projects that may require some manual upgrade steps, such as:

frameworks (ex: Django, Ruby on Rails, Spring);
language platforms (Java, C#, Python, Ruby);
runtime platforms (ex: JVM, CLR, CPython, MRI); and
OS platforms (ex: OS X, Windows, Debian).

Using semantic versioning makes it easy for high-level projects to regularly upgrade their lower-level dependencies, either automatically or manually.

Romantic Versioning

By contrast, I use the term romantic versioning¹ to refer to all other version numbering schemes that adhere to the following fuzzy guidelines:

A version number is a multi-part value such as X, X.Y, or X.Y.Z; with numeric components.
A conceptually major change bumps the first component of the value.
A conceptually minor change or bug fix bumps a later component of the value.

Romantic versioning is meant primarily for humans to understand, at the possible expense of easy understanding by a computer. This makes it suitable for marketing purposes (since marketing is directed at humans) but often not suitable for automated dependency management (which is performed by a computer).

When to use Romantic Versioning

As mentioned before, romantic versioning is mainly suitable for projects that are actively marketed and updated over time but not depended on by other projects, such as:

applications (Microsoft Word, Firefox, Spotify, Evernote).

Some projects are not updated over time (such as many console games) and so have no version number at all.

Hybrid Romantic & Semantic Versioning

There are many projects that maintain two different version numbers simultaneously: a romantic version for marketing purposes and a semantic version for developer purposes. For example:

Java
- Currently at “product version” 8.40 (“Java SE 8u40”), which is the romantic version; and “developer version” 1.8.40, which is the semantic version.
Windows
- At marketing version 7 (“Windows 7”), which is the romantic version; and at internal version 6.1, which is the best approximation of its semantic version.

There are also some libraries that advertise a romantic version as if it were a semantic version which causes lots of problems when automated dependency management systems try to consume the library.

Final Observations

Projects typically² have a romantic version, a semantic version, or both.
Applications tend to have a romantic version only, since they are marketed to consumers rather than developers.
Libraries tend to have a semantic version only, since they are used only by automated tooling and developers.
Other types of projects like frameworks and platforms have more variation, often providing both a romantic and semantic version since they are often both marketed to consumers and used by developers.

Enjoy this article? Follow my Twitter account for notifications of new articles. 😃

Unique Features of Various Programming Languages
- Describes programming languages and their unique features.
Visual Guide to Programming Language Properties
- Visualizes how various programming language properties interact.

I have seen other people use different terms to refer to romantic versioning. For example Dominic Tarr calls such versioning schemes “sentimental versioning”. However he mainly mocks such schemes, highlighting the negative aspects of romantic versioning as perceived by a developer without considering the positive aspects for marketing to consumers.↩
Projects that receive no updates are unversioned, with no version number, such as many console games. And a small number of projects use an esoteric versioning scheme, with version numbers that are hard to interpret by either human or computer. For example successive version numbers of TeX are closer and closer approximations of π.↩

Proof terms in Idris

2015-02-27T00:00:00+00:00

Recently I’ve been experimenting with the language Idris, one of the rare dependently-typed programming languages.¹ For example you can write a sorting function that is proven correct by the compiler at compile time:

sort : (inputList:Vect n e) -> 
       (outputList:Vect n e ** (IsSorted outputList) ** 
                               (ElemsAreEq inputList outputList))

The preceding is a function that takes an input list (or “vector”) of elements and returns a tuple containing:

an output list,
a proof that the output list is sorted (IsSorted), and
a proof that the input list and output list have the same elements (ElemsAreEq).

Since the type signature is verified at compile time, any function implementing this signature is a provably correct sort function.²

The notion that a proof can be represented explicitly in a program is fascinating and a unique capability of dependently-typed languages.

In this article I will focus specifically on what proof terms are, how to define them, and how to construct them.

I assume that the reader has basic familiarity with Haskell-like syntax.

A simple built-in proof type: `Data.So`

Proof types are tricky to grasp at first. Thus let’s start by examining the simplest proof type in the Idris standard library³, the So type.

Here is the definition of the Data.So type:

data So : Bool -> Type where 
    Oh : So True

The So type represents a proof that a particular boolean test has been performed and that it evaluated to True. For example it is possible to construct a proof value (Oh) of type So (1 < 2) but it is not possible to construct a value of type So (1 > 2).

You can construct a value of So (1 < 2) directly on the command-line:

$ idris --nobanner
Idris> :module Data.So
*So> the (So (1 < 2)) Oh
Oh : So True

The function (the FooType fooValue) tells the compiler that fooValue should be of type FooType, which is occasionally useful. In this case I am asserting to the compiler that So (1 < 2) is so obvious that it should just try to construct the proof value Oh immediately.

However you can’t make a So (1 > 2):

$ idris --nobanner
Idris> :module Data.So
*So> the (So (1 /= 1)) Oh
(input):1:5:When elaborating argument x to function Prelude.Basics.the:
        Can't unify
                So True
        with
                So (fromInteger 1 > fromInteger 2)

        Specifically:
                Can't unify
                        True
                with
                        fromInteger 1 > fromInteger 2

Here the error message indicates that the compiler isn’t convinced that True == (1 > 2), which is absolutely correct.

In a real program the main way to construct a So is to use the choose function, whose definition is:

choose : (b : Bool) -> Either (So b) (So (not b))
choose True  = Left Oh
choose False = Right Oh

You might have to stare at this definition for a while to understand how it works. I know it took me a while to grok it. Let me try to explain:

The choose function is performing a pattern match on the Bool argument it receives. If it successfully pattern matches on True, the compiler itself understands that argument b is equivalent to True, so it permits the construction of the value Oh (of type So b) on the right-hand-side of the choose True definition. The choose False definition works via similar reasoning to construct a value Oh of type So (not b), since the compiler can deduce that not b is True when b is False.

A simple derived proof type: `IsLte`

What if we wanted to prove specifically that one element is less than or equal to another? We could write a proof type derived directly from Data.So.

IsLte : Ord e => (x:e) -> (y:e) -> Type
IsLte x y = So (x <= y)

Here the IsLte type is defined directly in terms of the So proof type that was described previously. Note that it is necessary to add the Ord e type constraint so that the compiler will allow the use of <= to compare x and y in (x <= y).

Strictly speaking the definition of IsLte here doesn’t create a new type, but rather a type alias to the underlying So. Therefore you must use the Oh proof value to actually make an IsLte:

$ idris --nobanner IsLte.idr 
*IsLte> the (IsLte 1 2) Oh
Oh : So True
*IsLte> the (IsLte 2 1) Oh
(input):1:5:When elaborating argument x to function Prelude.Basics.the:
        Can't unify
                So True
        with
                IsLte (fromInteger 2) (fromInteger 1)

        Specifically:
                Can't unify
                        True
                with
                        fromInteger 2 <= fromInteger 1

Now it’s fine and all to create a So or IsLte directly on the command line. But how would you create one in an actual program?

mkIsLte : Ord e => (x:e) -> (y:e) -> Maybe (IsLte x y)
mkIsLte x y =
    case (choose (x <= y)) of 
        Left proofXLteY =>
            Just proofXLteY
        Right proofNotXLteY =>
            Nothing

This function takes an x and a y at runtime and constructs a proof value of type (IsLte x y) if x <= y. If x is not <= y then it fails by returning a Nothing.

On the Idris CLI, let’s try to construct a valid proof that (IsLte 1 2):

$ idris --nobanner InsertionSort.idr
*InsertionSort> mkIsLte 1 2
Just Oh : Maybe (So True)

Nice. How about a proof that (IsLte 2 1)?

$ idris --nobanner InsertionSort.idr
*InsertionSort> mkIsLte 2 1
Nothing : Maybe (So False)

Fails as expected.

A simple standalone proof type: `HeadIs`

Let’s implement our own simple proof type. How about a proof that a vector begins with a particular value?

data HeadIs : Vect n e -> e -> Type where
    MkHeadIs : HeadIs (x::xs) x

A value of type HeadIs xs y represents a proof that a vector xs begins with (“has the head”) y.

Based on the definition above, the compiler will only permit the construction of the atomic MkHeadIs proof value if it can destructure its first vector argument and see that its head matches the second argument.

A complex proof type: `IsSorted`

How about we try to make a more complex proof type? How about a proof that a list of elements is sorted? Such a proof type would be useful in the sort function mentioned at the beginning of this article.

Let’s take a first stab at thinking about the initial signature of IsSorted. We want to be able to write IsSorted xs, so we need IsSorted to take a vector xs argument and return a type:

data IsSorted : (xs:Vect n e) -> Type where
    ...

Okay. Now, since proof terms work entirely by pattern matching on the arguments received, let’s think about what cases exist for matching on the single xs vector argument:

Nil – Definitely sorted, since it’s an empty vector. (Case 1)
x::xs – Not enough information to determine whether sorted. Let’s decompose xs:
- x::Nil – Definitely sorted, since it’s a vector with only one element. (Case 2A)
- x::(y::ys) – Sorted if: (Case 2B)
  - (1) we can prove (x <= y) and that
  - (2) (y::ys) is sorted.⁴

So let’s write out the preceding cases in code:

data IsSorted : (xs:Vect n e) -> Type where
    IsSortedZero :
        ... ->
        IsSorted Nil
    IsSortedOne  :
        ... ->
        IsSorted (x::Nil)
    IsSortedMany :
        ... ->
        IsSorted (x::(y::ys))

Straightforward. Now let’s fill in the holes for the parameter types:

data IsSorted : (xs:Vect n e) -> Type where
    IsSortedZero :
        IsSorted Nil
    IsSortedOne  :
        (x:e) ->
        IsSorted (x::Nil)
    IsSortedMany :
        (x:e) -> (y:e) -> (ys:Vect n'' e) ->     -- (n'' == (n - 2))
        (IsLte x y) -> IsSorted (y::ys) ->
        IsSorted (x::(y::ys))

The IsSortedZero case didn’t require any parameters at all. Easy enough.
The IsSortedOne case had only the variable x in its return type, so we just need to declare the parameter (x:e) so that it knew what x to use.
The IsSortedMany case is a lot more interesting:
- It needs the three x, y, and ys objects that appear in the return type (x::(y::ys)) to be defined.
- It needs a proof that x <= y. So let’s use an (IsLte x y).
- It needs a proof that y::ys is sorted, so let’s recursively use the IsSorted proof type that we’re in the middle of defining!
  - Yay recursively defined types. Just like recursively defined functions. And sometimes just as mind-bending.
  - I believe Idris will only allow recursive definitions of types in this way because the compiler itself can determine that the IsSorted (y::ys) proof type is “smaller than” the IsSorted (x::(y::ys)) proof type whose definition it feeds into.

Let’s try to compile this:

$ idris -o Scratch Scratch.idr 
Scratch.idr:17:18:When elaborating type of Main.IsSortedMany:
Can't resolve type class Ord e

Whoops. The IsSortedMany definition doesn’t know what to fill in for the implicit Ord e parameter to the IsLte type. So let’s add that parameter to IsSortedMany:

data IsSorted : (xs:Vect n e) -> Type where
    IsSortedZero :
        IsSorted Nil
    IsSortedOne  :
        (x:e) ->
        IsSorted (x::Nil)
    IsSortedMany :
        Ord e =>
        (x:e) -> (y:e) -> (ys:Vect n'' e) ->    -- (n'' == (n - 2))
        (IsLte x y) -> IsSorted (y::ys) ->
        IsSorted (x::(y::ys))

Now this actually compiles and I could stop here. However I know based on painful experience⁵ that we will run into type unification errors down the road if we don’t additionally put the Ord e constraint on the IsSorted type itself. So let’s just do that:

data IsSorted : Ord e => (xs:Vect n e) -> Type where
    IsSortedZero :
        Ord e =>
        IsSorted Nil
    IsSortedOne  :
        Ord e =>
        (x:e) ->
        IsSorted (x::Nil)
    IsSortedMany :
        Ord e => 
        (x:e) -> (y:e) -> (ys:Vect n'' e) ->    -- (n'' == (n - 2))
        (IsLte x y) -> IsSorted (y::ys) ->
        IsSorted (x::(y::ys))

Boom. Done.

Now, similar to the mkIsLte function from before, let’s create a mkIsSorted function that, given a list, checks whether it is sorted and returns an IsSorted sortedness proof if it is.

mkIsSorted : Ord e => (xs:Vect n e) -> Maybe (IsSorted xs)
mkIsSorted Nil =
    Just IsSortedZero
mkIsSorted (x::Nil) =
    Just (IsSortedOne x)
mkIsSorted (x::(y::ys)) =
    case (mkIsLte x y) of
        Just proofXLteY =>
            case (mkIsSorted (y::ys)) of
                Just proofYYsIsSorted =>
                    Just (IsSortedMany x y ys proofXLteY proofYYsIsSorted)
                Nothing =>
                    Nothing
        Nothing =>
            Nothing

This implementation is interesting. Note how the proof values IsSortedZero, IsSortedOne, and IsSortedMany are constructed directly.

Also notice in particular that to construct an IsSortedMany value that the subproof values proofXLteY and proofYYsIsSorted needed to be constructed first. And that the latter subproof required a recursive invocation of mkIsSorted. Lots of recursion in Idris.

Let’s try calling the function with a valid sorted list:

$ idris --nobanner InsertionSort.idr
*InsertionSort> mkIsSorted [1,2,3]
Just (IsSortedMany 1
                   2
                   [3]
                   Oh
                   (IsSortedMany 2
                                 3
                                 []
                                 Oh
                                 (IsSortedOne 3))) : Maybe (IsSorted [1, 2, 3])

Nice. See how the proof term embeds the (IsLte 1 2) and (IsLte 2 3) values (both Oh) which represent the comparison checks made to determine whether the list was sorted. In this fashion a proof value effectively embeds an execution trace of the program that generated it. Neato.

We can generalize this observation: The contents of a big proof value is a nested structure of smaller proof values that taken together are used to derive the big proof value. Here, an IsSorted [1,2,3] value contains both an IsLte 1 2 and an IsLte 2 3 value, which is enough to deduce that [1,2,3] is sorted.

Now let’s try calling mkIsSorted with a non-sorted list:

$ idris --nobanner InsertionSort.idr
*InsertionSort> mkIsSorted [3,2,1]
Nothing : Maybe (IsSorted [3, 2, 1])

As expected, we couldn’t construct a proof that an unsorted list was actually sorted.

Conclusion

Hopefully you now have a better grasp of proof types and values, and also have some practice in constructing them.

In an upcoming article I hope to demonstrate how proof types like IsSorted and ElemsAreEq can be used to create a provably correct implementation of insertion sort. Stay tuned.

Agda is another dependently typed language that I’ve looked at. However I was not impressed.↩
Actually to prove the correctness of the sort function you need to verify one more thing in addition to the function signature: You must verify that the definitions of the related proof types (in this case IsSorted and ElemsAreEq) correctly implement what their name implies.

Usually the definitions of proof types are fairly compact so inspection isn’t much of a burden. By contrast the correct construction of proof terms within an algorithm is considerably more complex. Happily the compiler checks your work ruthlessly within the algorithm itself, so you don’t have to worry about messing up term construction.
↩
The Idris community appears to prefer the term “the standard library” to refer specifically to the prelude part of the library code that ships with Idris and “the library” to refer to the non-prelude part. I think this distinction is unimportant and confusing so I will use the term “standard library” to refer to all library code that ships with Idris, which is consistent with terminology usage in other languages (Python, Java, C#, Perl, PHP, etc).↩
In my initial attempt to define IsSorted I incorrectly required only the subproof IsSorted ys and not IsSorted (y::ys) when proving IsSorted x::(y::ys). But that restriction allows you to construct IsSorted [9,10,1,2] which is clearly bogus. Pay attention to how your proof types are defined.↩
If you fail to add the Ord e constraint to the other type constructors, you will eventually run into errors like Can't unify "So (<=) {{constrarg1}} x y" with "So (<=) {{constrarg}} x y" where the compiler loses type information about the specific Ord e instance when trying to pull type information through the IsSortedZero or IsSortedOne type constructors during type unification. That probably didn’t make much sense - especially the part about “type unification” - but you’ll probably have to debug a similar issue at some point.↩

Spectrum of Languages by Hardware Distance

2014-12-20T00:00:00+00:00

Programming languages (and the ecosystems that surround them) are incredibly interesting. They can be sliced in many different ways based on syntactic similarity, type system similarity, suitedness for various problem domains, and many other attributes.

Below I present an spectrum of various languages, arranged by distance from the underlying hardware¹:

TLDR
Full

A few interesting patterns become evident in this diagram:

Expressiveness ∝ Distance from Hardware
- Usually the closer you are to the hardware, the more you have to think like a computer and less like a human, decreasing expressiveness.
- There are some outlier languages (such as Go, Lua, and Forth) which are more expressive than their distance from the hardware would suggest.
Type systems tend to be similar for languages with similar hardware-distance.
- Bare Metal languages like machine code and assembly have no type system at all. Every piece of data is just a machine word and no type checking is done at all.
- Almost Bare Metal languages like C and C++ have a type system that is just strong enough to tell the compiler how to compile operations efficiently, but these type systems are not specifically designed for the elimination of bugs or expressiveness.
- High Level languages care more about expressiveness, and they tend to divide into two camps: statically-typed (Java, C#) and dynamically-typed (Python, Ruby, Lisps).
- Anything that is Intermediate Typed or below has a stronger focus on the type system, usually for preventing bugs or just having very precise types.
“PhD required below this line”
- I have observed generally two large groups of people that work with programming languages, the Practitioners and the Academics. Each group uses their own specialized jargon² and tends not to talk to each other that much.
- Languages inside the “PhD required below this line” area are generally used primarily by Academics, and therefore their documentation (often only in the form of research papers) is difficult to read by Practitioners.
- I currently group myself as a Practitioner who is trying to read Academic.
- Of course some so-called Academic languages are sometimes used in practice for commercial systems. The Java bytecode verifier, for example, is defined in Prolog.
- Many languages in or near this region are lumped under the vague term functional programming, which roughly translates to a programming style that avoids mutable data structures & side effects by default, or just “a language that feels like Haskell” (the canonical “functional” language).
Middle of the spectrum = Sweet spot
- There are a quite a lot of languages in the middle layers (High Level and Intermediate Typed). Probably because they intersect most with expressive languages, which are the most fun to program in.
- Programming near the hardware is unfun and generally unnecessary unless you’re writing an OS, an embedded system, a CPU-intensive scientific computation, or a graphics shader.
- I suspect there aren’t as many languages at the bottom in Academic-land because there are fewer Academics than Practitioners, such languages are harder to write (because of more complex type systems), or because I simply am not aware of many such languages as a non-Academic.
Declarative Languages that are general-purpose are rare.
- Most Declarative Languages are domain-specific. Arguably Prolog is general since it answers arbitrary questions from logic. And arguably SQL is general because most data stores (and every SQL database) is queryable with it.
- Other examples of declarative languages not listed here are VHDL and Verilog (used for hardware design), and ReceiptTally³ (used for personal finance).

Visual Guide to Programming Language Properties
- Shows programming language features, how they are related, and which features are present in popular languages.
- Visualizes some of the languages discussed in this article.
Unique Features of Various Programming Languages
- Discusses several programming languages and their unique features.

Series

This article is part of the Programming for Perfectionists series.

When speaking of “hardware distance”, the hardware I am referring to is the common commodity hardware architectures such as x86 and ARM. It should be noted that there are also specialized architectures where Forth, Lisp, and Prolog are the assembly language. And some architectures like the Burroughs actually have typed memory and pointers, which eliminates a lot of the dangers of working with untyped assembly.↩
Examples of Practitioner-jargon: “inheritance”, “generics”, “dependency injection”, “covariance”. Examples of Academic-jargon: “monad”, “functor”, “reduce”, “dependent type”, “currying”. More examples from this presentation on functional programming.↩
ReceiptTally is a custom language of my own designed to describe household receipts. With this description it can be calculated who owes whom money for reimbursement purposes. I haven’t bothered to make this project public at this time.↩

Error handling styles in programming

2014-11-22T00:00:00+00:00

There are many ways that programs can handle errors at runtime. The excellent book Exercises in Programming Style proposes an interesting categorization of common error handling patterns. Fascinatingly, the programming language that you work in often actively encourages one particular error handling pattern over another.

Reinterpret and Continue (Constructivist)

In this style, a function that receives a bad input or encounters an error always returns a sensible result to its caller, silently reinterpreting the error to a valid value. Such a function will never crash the program but may produce unexpected results in the presence of bad inputs or errors.

Languages whose standard libraries and syntax regularly reinterpret errors and questionable input make it difficult to write programs that are not in this style.

Environments that encourage this style:

PHP
- Very few PHP errors will actually crash the program - instead you get strange results and error text inserted into rendered pages.
- Errors are printed to a server log file that nobody reads.
HTML, CSS
- Web browsers reinterpret bad markup as best they can to valid markup. Of course different browsers do this in different ways, so the only way to get a consistent rendering is to use valid markup.
- Rendering errors are not generally logged anywhere.
JavaScript

Developers that wish to use a more Fail Fast style of error handling (see below) but are forced to write in one of these languages may use special validation and linting tools to look for errors in the program text before they can be reinterpreted.

Fail Fast (Tantrum, Passive Aggressive)

In this style, a function that receives a bad input or encounters an error refuses to continue, returning an error back to its caller.

Exercises in Programming Style makes the additional distinction that functions in some programs try to handle errors immediately at their point of detection no matter what (“Tantrum”), whereas functions in other programs prefer to bubble up errors to callers until a caller with enough context is reached that decides to actually handle the error (“Passive Aggressive”).

Programs in the Tantrum substyle have lots of error handling code scattered throughout the program text, whereas programs in the Passive Aggressive substyle tend to consolidate most error handling code in top-level functions and let lower-level code just bubble up errors.

Languages that have built-in exceptions make it easy to implement programs in the Passive Aggressive substyle.

Environments that encourage this style:

Java
C#
Python

Error Handling
- More comprehensively discusses considerations and specific implementation techniques for writing code that is robust in the presence of errors.
Abandonment vs. Unchecked Exceptions for Error Handling
- Describes abandonment, an uncommon error-handling style.

Submitting a Python App to the Mac App Store

2014-06-24T00:00:00+00:00

Recently I needed to submit a Python app to the Mac App Store. Since there seemed to be no good documentation online for doing this, I had to figure it out myself. Here’s what I learned:

Building a Python app for the Mac App Store

Below I will explain how to make a Python script into a Mac App Store app. I would recommend that you download my example Python app that can be submitted to the Mac App Store and examine its build system while reading the rest of this article.

Get a Python script

First, I assume you already have a Python script that you’d like to package as an application and submit.

My example app uses src/HelloAppStore.py as the main script.

Make it into a regular app

A Python script, along with its included Python modules and dependencies, can be bundled into a regular Mac app using py2app.

I won’t repeat the py2app documentation, but you generally need to create a setup.py file that looks something like this:

from setuptools import setup

APP = ['src/HelloAppStore.py']
DATA_FILES = []
OPTIONS = {
    'argv_emulation': True,
    #'iconfile': 'src/Icon.icns',  # optional
    #'plist': 'src/Info.plist',    # optional
}

setup(
    app=APP,
    data_files=DATA_FILES,
    options={'py2app': OPTIONS},
    setup_requires=['py2app'],
)

Then you can build dist/HelloAppStore.app into a nice double-clickable app by running the command:

python setup.py py2app

My example app runs the above command as part of the ./build-app.sh script.

Alter the app to conform to Mac App Store constraints

There are several additional nontrivial restrictions that must be satisfied before an app can be submitted to the Mac App Store:

It must be able to run in sandboxed mode.
It must be code-signed.
It must not depend on any deprecated APIs (like QuickTime), even from included libraries.
It must not include any PowerPC code, even from included libraries.
It must have a large app icon (with sizes up to 1024x1024).
It must have a specified app store category.
It must not have any content that Apple finds offensive in its sole discretion.

An exhaustive list of restrictions can be found in the Mac App Store Review Guidelines.

Sandboxing

Mac App Store apps must be sandboxed. Sandboxed apps are restricted in several ways, but the most significant restriction is the inability to read/write arbitrary files.

In particular an app cannot write to a file outside of its sandbox container unless a native¹ open/save dialog is used to prompt for the location of the file or the file already resides within the app’s container directory.

It’s a good idea to read the App Sandboxing documentation to understand the full set of restrictions and how to overcome them when necessary (and when possible).

My example app enables sandboxing by specifying that com.apple.security.app-sandbox = true in the src/app.entitlements file. This entitlements file is used in the code-signing process described in the next section.

Code-signing

Mac App Store apps must be code-signed. This means you must go through some extra hoops to generate signing certificates, download them to your dev machine, and alter your build script to sign the final app package with it.

These certificates have to be generated by Apple as part of your Mac Developer Program subscription ($99/year). And renewed annually if you wish to continue making app updates.

Since Python apps are built outside of Xcode, you’ll have to use the codesign tool manually to sign your app.

My example app runs the codesign tool as part of the ./build-app.sh script, recursively signing the inner frameworks, helper tools, and finally the outer application binaries. As part of the signing process the application binaries are also embedded with entitlements, which are used to enable sandboxing.

See the Code Signing Guide for more information about manual code-signing.

Deprecated APIs

Your Python script probably won’t be directly using any deprecated APIs. However your Python script might depend on other libraries that do. Such dependencies can be difficult or impossible to eliminate. Typically you have to modify and recompile the dependency manually.

In particular the very popular wxPython GUI toolkit depends on deprecated QuickTime APIs at the time of writing, making any Python app that depends on it inadmissible to be submitted to the Mac App Store.

My example app has no workarounds for deprecated APIs.

PowerPC Code

Your Python script probably won’t be directly using any old PowerPC code. However, again, your Python script might depend on other libraries that do. In fact Python 2.7 itself includes PowerPC code.

Luckily any PowerPC code can be stripped out easily using the lipo tool, so you just need to add some extra lipo commands to your build script.

My example app uses lipo in the ./build-app.sh script to remove PowerPC code from the python2.7 library.

Large App Icon

If you don’t have an app icon you’ll have to create one. If you do already have an app icon, I’ll bet you it doesn’t meet the minimum 1024x1024 pixel size requirement.

Creating an icon entails creating several images for your icon at various specific sizes, and then using the iconutil command to generate a final .icns icon file.

My example app contains a ./build-icon.sh script that can be used to generate src/Icon.icns from images in the src/Icon.iconset directory.

See the Icon Design Guidelines in the OS X Human Interface Guidelines for more information about creating icons.

App Store Category

The Info.plist file inside a Mac App Store app is required to specify what category on the Mac App Store it belongs to under the LSApplicationCategoryType key.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    ...
    <key>LSApplicationCategoryType</key>
    <string>public.app-category.reference</string>
    ...
</dict>
</plist>

So far as I can tell, the set of valid values for this key is not documented anywhere. Therefore to find new values, I created a new Cocoa Application project in Xcode (where I could specify the app category in a dropdown), compiled the app, and opened its Info.plist file to see what value it had for the LSApplicationCategoryType key.

Submitting your app to the store

Regular apps written in Objective-C for the Mac App Store are usually submitted directly from within Xcode. This is not an option for Python apps that you build outside of Xcode.

Instead you have to use an older app submission tool called Application Loader to upload your app. However Application Loader doesn’t submit a .app package directly; it requires an installer .pkg instead.

An installer package can be built from your .app using the productbuild tool.

My example app runs the productbuild tool as part of the ./build-pkg.sh script which creates an installer package containing the app.

For more information about the productbuild tool, see its man page.

Fin

Hopefully you should now have a good idea of what is involved in submitting a Python app to the Mac App Store.

If you’d like to actually try out the process of submitting an app, try building and submitting my example app using the instructions in its README.

Please send comments and corrections to David Foster.

Therefore if you are using a GUI toolkit that uses a simulated open/save dialog rather than a native one, your app won’t be able to access the files the user selects!↩

Agda: Second Impressions

2014-02-17T00:00:00+00:00

These are my second impressions of the Agda programming language after writing (or trying to write) a few simple Agda programs.

If you only want the highlights, just read the Executive Summary. If you also want to see supporting material and notes, continue reading further.

Executive Summary
Programs

Executive Summary

Agda is simultaneously touted as (1) a dependently-typed programming language and (2) a proof assistant. I find that Agda is a considerably better proof assistant than it is a programming language at this time.

A few issues I found when trying to use Agda as a programming language to get real work done:

I/O is painful:
- The Agda standard library lacks any functions for reading from standard input. Clearly the ability to write basic command-line programs is not a priority for the Agda library implementors.
  - You must implement such functions yourself as native functions that call into Haskell.
- The I/O that does exist is based on Haskell’s I/O monad, which by itself is difficult enough to learn.
- It appears that one has to learn what coinduction is in order to use Agda’s native IO module. Yet another abstract academic construct.
  - I can work around this by restricting myself to the IO.Primitive module. This module directly mirrors Haskell’s IO system and thus only requires knowledge of monads.
Some basic numerical operations are very inefficient.
- The div and mod operators for the default implementation of natural numbers (Data.Nat) run in linear time relative to the magnitude of the divisor.¹
  - Consequently it takes about 16 seconds to print the number 4000 because of division inefficiency when formatting the number for display.
- At least ≤?, ≟, +, and * appear to be logarithmic- or constant-time.
The “Hello World” program compiles to a 14 MB binary. Unacceptably large.
The “Hello World” program takes 3 seconds to compile. Unacceptably slow.
It takes quite a lot of effort to get a basic environment running. You have to:
- Install Agda.
- Install the Agda standard library separately.
- Compile the Agda IO.FFI module manually.
- Install and learn Emacs.
- Configure Emacs to work with Agda.²

Agda: First Impressions
- Previous article in this series.

Programs

The following are all the simple programs I’ve written with Agda so far.

Peano

A manual reimplementation of natural numbers (i.e. non-negative integers) just so that I can get my Agda environment up and running.³

Peano.agda

module Peano where

data ℕ : Set where
  zero : ℕ
  suc : ℕ → ℕ

_+_ : ℕ → ℕ → ℕ
zero + zero  = zero
zero + n     = n
(suc n) + n' = suc (n + n')

I resent being forced to use Emacs as opposed to my favorite text editor.
- Particularly since the installation instructions are not bulletproof.

HelloNumber

A program that prints the number 5.

No more reimplementing silly stuff like numbers. Let’s use the standard library.

HelloNumber.agda

module HelloNumber where

-- Agda standard library 0.7
open import Data.Nat
open import Data.Nat.Show using (show)
open import Data.Unit using (⊤)
open import IO
import IO.Primitive as Prim

five : ℕ
five = 5

main' : IO ⊤
main' = putStrLn (show five)

main : Prim.IO ⊤
main = run main'

The Agda standard library is not installed with Agda by default. WTF?
- It also has not yet reached v1.0.
Standard library IO appears to be defined equivalently to Haskell’s IO monad.
I am amused that compiling the IO module also implies compiling the entirety of Algebra.
I’m sad that declaration order in Agda matters. In particular items must be declared before they can be used.
- This forces me to write programs in the annoying bottom-to-top order, requiring lots of extra up-down eye scanning for large programs.
Surprise! The main function doesn’t have type IO ⊤ (which would be equivalent to the Haskell IO ()). Instead it has type Prim.IO ⊤ which can be obtained by passing an IO ⊤ through the IO.run function.
Even after installing the standard library, I have to manually compile the IO.FFI library to perform basic IO. *grumble*
My test program that just prints the number 5 takes up 14 MB. What a waste of space…
- I assume those 14 MB are primarily the bulk added by the standard library.

HelloWorld

Now let’s try printing strings, which should allow writing the standard “Hello world” program.

HelloWorld.agda

module HelloWorld where

-- Agda standard library 0.7
open import Data.String
open import Data.Unit using (⊤)
open import IO
import IO.Primitive as Prim

greeting : String
greeting = "Hello World"

main' : IO ⊤
main' = putStrLn greeting

main : Prim.IO ⊤
main = run main'

It takes 3 seconds to type-check my Hello World program when compiling to an executable. That’s really slow.
- I hope those 3 seconds are just fixed overhead and not proportional to the size of the program.

PrintTwoThings

Now let’s try stringing multiple IO actions together.

PrintTwoThings.agda

module PrintTwoThings where

-- Agda standard library 0.7
open import Coinduction using (♯_)
open import Data.String
open import Data.Unit using (⊤)
open import IO
import IO.Primitive as Prim

doThis : IO ⊤
doThis = putStr "Everybody say coinductive deep embedding! "

thenThat : IO ⊤
thenThat = putStrLn "Coinductive deep embedding!"

main' : IO ⊤
main' = (♯ doThis) >> (♯ thenThat)

main : Prim.IO ⊤
main = run main'

Surprise! The >> operator used to join two IO actions isn’t IO A -> IO B -> IO B. Rather it is ∞ (IO A) -> ∞ (IO B) -> IO B.
- This means I need to wrap IO actions inside the ∞ type before I can chain them together.
- What is the ∞ type? No idea. It has something to do with “coinduction”, which Wikipedia failed to explain to me coherently.
  - Information here suggests that a ∞ A represents a lazy unevaluated computation of type A that might not terminate when it is actually evaluated.
  - Representing the possibility of nontermination explicitly is probably required since Agda normally requires all of its function definitions to provably terminate.
- According to the Coinduction library, I can make a ∞ using the ♯ operator which takes a value of any type A and makes a ∞ A out of it.
- Therefore instead of action1 >> action2 I need ♯ action1 >> ♯ action2

EchoInput

Read a line from standard input. Write it back to the screen.

EchoInput.agda

module EchoInput where

-- Agda standard library 0.7
open import Data.String
open import Foreign.Haskell using (Unit)
open import IO.Primitive

postulate
  getLine : IO Costring

{-# COMPILED getLine getLine #-}

doThis : IO Costring
doThis = getLine

thenThat : Costring → IO Unit
thenThat = λ s → putStrLn s

main : IO Unit
main = doThis >>= thenThat

There is no function in the standard library to read from standard input. Nor is there a reference to stdin itself. Unbelievable.
- I can define getLine manually using the foreign function interface. But I really shouldn’t have to define such a simple function. This should be in the standard library…
When writing native IO functions it’s a lot easier to directly use Agda’s Primitive.IO module instead of the regular IO module. I no longer need to use ∞ or ♯ either, which is nice.

PromptForName

PromptForName.agda

module PromptForName where

-- Agda standard library 0.7
open import Data.String
open import Foreign.Haskell using (Unit)
open import IO.Primitive

postulate
  getLine : IO Costring

{-# COMPILED getLine getLine #-}

main : IO Unit
main = 
  putStrLn (toCostring "What's your name?") >>= (λ _ → 
  getLine >>= (λ s → 
  putStr (toCostring "Hello ") >>= (λ _ → 
  putStrLn s)))

Surprise! Primitive.IO defines the >>= operator but not >>. Asymmetric with the regular IO module.
Native IO functions require Costring objects. So a string literal has to be passed to toCostring to be used.

EchoInputReverse

Read a line from standard input. Reverse it in Agda. Write the reversed line back to the screen.

EchoInputReverse.agda

module EchoInputReverse where

-- Agda standard library 0.7
open import Data.List using (reverse)
open import Data.String
open import Foreign.Haskell using (Unit)
open import IO.Primitive

postulate
  getLine : IO String

{-# COMPILED getLine getLine #-}

main : IO Unit
main = 
  getLine >>= (λ s → 
  return (toCostring (fromList (reverse (toList s)))) >>= (λ s' → 
  putStrLn s'))

Problem: The line from standard input is a Costring. I can’t reverse it or do anything useful with it without first converting it to a String. But there is no function in the standard library to do this conversion, even as a runtime cast.
Surprise workaround: Even though the Haskell type String maps to the Agda type Costring in the general case, it is still possible to postulate that it maps directly to the Agda type String for certain functions where you can assert that the string won’t be infinite.
- So the Agda FFI must be capable of performing implicit coercion between the native Costring and String datatypes. This is not documented.
- This approach has the disadvantage that it marks the result of getLine as being a finite string when it may not in fact be. See discussion at StackOverflow.

ParseInt

Prompt the user for a number, parse it, and output it. If an invalid number was input, output zero.

ParseInt.agda

module ParseInt where

-- Agda standard library 0.7
open import Data.Char
open import Data.List
open import Data.Maybe
open import Data.Nat
open import Data.Nat.Show
open import Data.String
open import Foreign.Haskell using (Unit)
open import IO.Primitive

postulate
  getLine : IO String

{-# COMPILED getLine getLine #-}

parseInt : String → Maybe ℕ
parseInt s = 
  then? (unwrap (parseDigits s)) (λ s' → 
  just (digitsToℕ s'))
  where
    parseDigits : String → List (Maybe ℕ)
    parseDigits s = map toDigit (toList s) where
      toDigit : Char → Maybe ℕ
      toDigit '0' = just 0
      toDigit '1' = just 1
      toDigit '2' = just 2
      toDigit '3' = just 3
      toDigit '4' = just 4
      toDigit '5' = just 5
      toDigit '6' = just 6
      toDigit '7' = just 7
      toDigit '8' = just 8
      toDigit '9' = just 9
      toDigit _   = nothing

    -- TODO: There's probably a standard-library function that does this
    --       but I can't find it.
    unwrap : List (Maybe ℕ) → Maybe (List ℕ)
    unwrap xs = unwrap' (just []) xs where
      unwrap' : Maybe (List ℕ) → List (Maybe ℕ) → Maybe (List ℕ)
      unwrap' (just xs) (just y ∷ ys) = unwrap' (just (Data.List._++_ xs [ y ])) ys  -- makes unwrap O(N^2)!
      unwrap' (just xs) (nothing ∷ _) = nothing
      unwrap' (just xs) []            = just xs
      unwrap' nothing   _             = nothing

    -- TODO: Look for a monadic _>>=_ that does the same thing
    then? : {A : Set} → {B : Set} → Maybe A → (A → Maybe B) → Maybe B
    then? nothing _ = nothing
    then? (just r1) op2 = op2 r1

    digitsToℕ : List ℕ → ℕ
    digitsToℕ xs = digitsToℕ' (reverse xs) where
      digitsToℕ' : List ℕ → ℕ
      digitsToℕ' []       = 0
      digitsToℕ' (x ∷ xs) = x + (10 * (digitsToℕ' xs))

ℕ? : Maybe ℕ -> ℕ
ℕ? (just x) = x
ℕ? nothing  = 0

main : IO Unit
main = 
  getLine >>= (λ s → 
  return (show (ℕ? (parseInt s))) >>= (λ s' → 
  putStrLn (toCostring s')))

Agda type checking errors are really hard to read:

Actual:

List Char !=< _A_10 s → _B_11 s of type Set
when checking that the expression toList s has type
_A_10 s → _B_11 s

Better:

Expected type (A → B) but found type (List Char).

Agda’s show function that converts a natural number to a string has quadratic performance!
- It takes 0.3s to print 1000, 2.2s to print 2000, 6.9s to print 3000, and 15.9s to print 4000.
- This is crazy! It probably means that at least one of {+, *, mod} is not constant time. Not cool.
  - Indeed the _divMod_ operator runs in linear time with respect to the magnitude of the dividend. Madness.
Surprise! Equality comparison between two numbers is not = or == but rather ≟.
- It doesn’t return a Bool either. Instead it returns a fancy boolean called a Decidable with an embedded equality proof.
- A Decidable can be converted to a regular boolean with the ⌊_⌋ operator. Weird.
I suspect that (arbitrary precision) natural numbers from Data.Nat take linear memory to represent rather than logarithmic.
- There is Data.Bin which represents natural numbers using binary, but most of the standard library doesn’t use it.

There is another module called Data.Bin which provides a representation of natural numbers which uses an efficient binary representation. But most of the standard library does not use it.↩
I found configuring Emacs for Agda to be nontrivial. I had to learn enough Emacs Lisp to debug my .emacs configuration file.↩
Kudos to Learn You an Agda > Hello, Peano for helping me implement this first Agda program.↩

Agda: First Impressions

2014-01-24T00:00:00+00:00

These are my first impressions of the Agda programming language after researching it in some depth but before writing any actual Agda programs.

If you only want the highlights, just read the Executive Summary. If you also want to see supporting material and notes, continue reading further.

Executive Summary
Getting Started
Environment
Libraries
Papers
Books
Other Resources

Executive Summary

Agda, by virtue of it possessing a dependent type system, makes it a very powerful programming language. Agda is distinguished from its cousins like Coq in that it makes more of an effort at being a programming language as opposed to being merely a proof assistant. Certainly worthy of study for the purpose of learning what a truly powerful type system is capable of expressing.

However Agda I judge to not presently be worthy as a full production-quality programming language, suitable for long term development. My reasons:

Agda lacks a package manager
- Acts as a barrier for the creation of 3rd party libraries. A language cannot be used effectively without 3rd party libraries.
- Library discoverability is also impaired.
The standard library has a version number less than 1.0, namely 0.7.
- A lack of a forward-compatibility promise from the standard library is not acceptable for long term development.
I can find no books about Agda development, let alone real-world Agda development.

Other tripping points:

Agda does not have a (traditional-style) documentation generator tool, which discourages somewhat the creation of API documentation and especially the creation of guide-level documentation.
- Agda API documentation is instead issued via hyperlinked versions of Agda source code, which appear to emulate the Agda Emacs mode that Agda is traditionally developed with.
- Such API documentation can be used effectively but it is not the format that the larger programming community expects (i.e. styled HTML) and is lacking in aesthetic appeal.
Agda only appears to have integration with the Emacs code editor. In particular there is no obvious integration with any GUI code editors, like Sublime Text. And such editor integration is necessary at minimum for the Unicode-style input that Agda requires.
Agda’s main website and “marketing” has very little polish.
- It’s a shame that this matters, but it makes Agda appear unfinished and not ready for prime time. In practice this will scare away users.
- The main website, for example, is just a stock wiki.
- Agda has no logo.

I expect that many of these shortcomings will be addressed over time.

In the meantime, as I mentioned before, I think Agda is still worthy of investigation for expanding one’s knowledge of what is possible in a powerful type system. In fact, any language where you can do something like write a sorting function that both works and proves itself to be a correct implementation in the same stroke deserves a closer look.¹

Agda: Second Impressions
- Next article in this series.

Getting Started

Official website of Agda is http://wiki.portal.chalmers.se/agda/pmwiki.php:

Yes, it’s a wiki. Automatically lowers the quality. :-(
(1) “Agda is a dependently typed functional programming language.”
(2) “Agda is a proof assistant.”

Uses Unicode operators all over the place. Nice to read. Tricky to type. Even library developers use them.

Environment

Has 3 backends: Haskell, JavaScript, and Epic

Only the Haskell one should be taken seriously, since the latest version of the standard library (0.7) only supports the Haskell backend.

Agda should not be used on Windows. None of the core release maintainers run Windows.²

Strongly expects that you use Emacs, as Agda provides its own interactive Emacs mode.

Libraries

Apparently the Agda community is sufficiently small that all libraries (of significance?) can be listed comfortably on the main Agda website at http://wiki.portal.chalmers.se/agda/pmwiki.php?n=Main.Libraries.

Informal sampling shows several libraries to be outdated, not updated within the last 2 years.
Therefore the number of actual usable up-to-date libraries is expected to be quite small, in the 4-7 range. Not promising.

Only two libraries under the https://github.com/agda umbrella have been updated within the last year. One of them is the Agda standard library. Not a good sign of activity.

Libraries do not carry external documentation. Instead all documentation is within the source code itself. The expectation is that you will use the Agda emacs mode to jump around between modules and explore to find what you want. Not ideal but serviceable.

There seems to be some tool used to compile HTML that acts like the Agda Emacs mode. In particular hyperlinking between modules is supported.

Readers of the documentation are assumed to have high mathematical training. I see terms like “communative semiring”, “coinductive”, “lemma”, and “decidable”.

agda-frp-js (last activity 2 years ago)

This is an FRP implementation in Agda. I’ll look at it because I’ve been working in FRP systems recently (i.e. Elm). And because I’d like to see how well libraries are packaged by the Agda community.

“Beh” is short for “behavior” and is equivalent to an Elm Signal.

I note that there is no API documentation whatsoever for the module. You have to browse through the examples in the source tree manually. This suggests that either Agda has no doc generator tool (likely!) or this library author simply didn’t bother to generate docs for the public to read.

Virtually no comments.

Has own definition of Bool (booleans), RSet (function types), Int (integer types), and possibly other super primitive notions. This appears to be related to the use of the JavaScript backend, as most of these primitives have native JS code defined for many of these definitions.

agda-frp-ltl (last activity 2 years ago)

Not going to look at this closely due to the lack of activity.

Papers

Agda is used in quite a number of academic papers. Certainly more papers than actual libraries, so far as I can tell. There’s a whole list of papers that use Agda.

Safe Functional Reactive Programming through Dependent Types (2009)

Describes an FRP system that is slightly more powerful than that of Yampa. It defines a type system around FRP that guarantees that FRP programs written within the type system will necessarily continue to make progress (i.e. not deadlock). (It is possible for an FRP system to deadlock if its signal graph get reconfigured into certain kinds of loops.) It also eliminates the distinction between continuous and discrete signals, reducing one class of errors. Furthermore it allows programs to include feedback loops and uninitialised signals in an FRP program, whilst guaranteeing evaluation progress at the type level.

Books

Hard to find books on Agda. I get nothing.

However I do see some books on Coq, which Agda is inspired by. Let’s not get distracted by those at the moment.

Other Resources

Dependently Typed Programming in Agda (2008)
- Appears to be the original paper that introduced the Agda programming language. Or at the very least a fairly complete description of the language along with tutorial-style content.
The Power of Pi (2008)
- Despite the indescriptive name, promises to show a number of design patterns and techniques that apply to dependently types languages, particularly Agda.
- “Dependent types matter. And not just for program verification and proof assistants: dependent types matter to programmers.”
- “This paper demonstrates how to program with dependent types. In particular, we present three case studies. Each case study describes a domain-specific language that is difficult to embed in conventional functional languages such as Haskell.”
- “design patterns of dependently-typed programming”
- “domain-specific embedded type systems”
Brutal (Meta)Introduction to Dependent Types in Agda
- Aims to describe at a high-level how dependently typed languages work at the level of theory, without going into too many details. Uses Agda as a concrete language to work with.
Learn You an Agda
- Already read this. Good high level introduction to the promise (or taste) of what’s possible in dependently typed languages.
- Difficult to browse, since the rendered HTML of these markdown pages fell off the internet a while ago.

I first recognized that you could write this kind of provably correct sorting function when I read these slides at swan.ac.uk. You end up having a sorting function with a signature like sort : (inList : List a) -> (outList : List a, IsSorted outList, EqElements inList outList).

The IsSorted outList type represents proofs that outList is sorted. Thus the existence of a member of the IsSorted outList type proves that outList is sorted. Similarly the EqElements inList outList is a proof type where inList and outList are proven to have equal elements.
See also the dedicated article at mazzo.li
↩
None of the people who currently release Agda are running Windows according to the latest Agda Implementors' Meeting.↩

Impressions of Haskell

2013-11-12T00:00:00+00:00

I finally found an online tutorial for Haskell that involves writing a non-trivial program:

Write Yourself a Scheme in 48 Hours

It’s an excellent tutorial and has given me a good impression of Haskell for building a medium-sized non-trivial program: a LISP interpreter.

This impression, however, has been more negative than positive. I would consider Haskell for use for doing programs that do heavy computation with pure functions alone.¹ However if there’s any significant amount of I/O, error handling, or state, using Haskell is just bloody painful.

Monads are painful

I/O must be done inside an I/O monad.²
Error handling must be done inside an error monad.³
Passing around shared state must be done inside a state monad.⁴

The preceding by itself would not be such a problem except that, in addition:

Code that is pure uses a different syntax than code that is in a monad.
Code that needs to be in multiple monads at the same time is a royal pain to write. It requires the use of monad transformers and explicit converting (or “lifting”) between individual monads and combined monads.

Having monadic code be in a different syntax than pure code requires the programmer to learn and use two different syntaxes. And these syntaxes aren’t even similar: they read in different directions. Mixing code that is both pure and monadic becomes confusing. Upgrading pure code to be monadic, which is not uncommon during development and maintenance, requires a lot of non-trivial syntactic transformations.

Pure code - Reads inside-out, generally bottom-to-top and right-to-left.
Monadic code - Reads top-to-bottom (with do-notation) and left-to-right (with bind operations).
Pure + monadic code - Reads in multiple directions.

Using a pure value within monadic code requires you to “lift” the pure value into a monad value with an explicit conversion. Similarly using a monadic value in multi-monadic code (created by a monad transformer) requires you to lift the monadic value into a multi-monad value first. These explicit conversions pepper and obscure the main program logic. One has to do an annoying amount of gymnastics to make the type system happy.

Code golf erodes readability

Another factor complicating the readability of code is the Haskell community’s propensity for code golf: the tendancy to write code that is especially terse to the point of making it difficult to understand. There are many language and library features in Haskell that enable this:

Overuse of operator overloading and custom operators:
- The ability to mix custom infix and prefix operators in the same expression. Confusing.
- The ability to treat an infix operator as prefix or visa-versa, when using certain syntax. Inconsistent.
- The usage of custom operators in ambiguous fashion requiring knowledge of either the operator associativity or precedence to disambiguate. Requires non-local knowledge to parse expressions.
  - For example the $ operator appears to be a precedence hack to avoid the need to parenthesize an expression properly. I don’t think it worth introducing tricky syntax to avoid typing a single close paren.
Implicit partial application of functions. Infix operator sections.
- You must know the arity of a function to determine what an expression containing that function means.⁵ Requires non-local knowledge to parse expressions.
Point-free style.
- Discourages the use of explaining variables.

Terseness at the expense of readability was a hallmark of Perl. I don’t find it any more attractive in Haskell.

Here’s a short example containing monadic code, point-free style, and an operator section.

-- Parses the expression in the first argument to the program, evaluates it, and prints the result.
main :: IO ()
main = getArgs >>= putStrLn . show . eval . readExpr . (!! 0)

I argue that the preceding code is manifestly confusing for any newcomer to Haskell, yet this type of code I see frequently in Haskell written by others.

It is possible to rewrite the code to be more explicit:

main = do args <- getArgs
          putStrLn (show (eval (readExpr (args !! 0))))

Although I think the preceding would not be considered idiomatic Haskell.

The end?

Haskell’s strong type system and heavy restrictions on mutable state provide good assurance that the programs you write are correct, assuming you manage to write the program in the first place.

However I find Haskell difficult to write (due to monads and the type system) and difficult to read (due to code golf). I also do not require such a high assurance of correctness in the domains I work in. I typically find that a disciplined minimization of mutable state in other languages is sufficent to provide the necessary level of correctness I require, and with more flexibility during maintenance.

Types of programs I might consider doing in Haskell include, for example, maze generators, Sudoku solvers (non-visual), parsers, and compilers. Games involving lots of state such as platformers, adventure games, or anything with multiple screens would be a pain. GUIs and other interactive programs based on events would be a pain.↩
A monad is a particular pattern of sequencing computation involving the type system. The use of monads is unique to Haskell so far as I am aware.↩
Granted it is possible to handle errors using throw but this is discouraged because it yields non-deterministic ordering with respect to I/O operations. It is also possible to use throwIO but only if the function is already in the IO monad. Should the function be otherwise pure, you’d be forced to “infect” it with the IO monad in order to use throwIO.↩
Of course you could just incorporate the shared state into the parameters and return type of all functions, but this gets annoying fast. Granted it is possible to use IORefs if the function using state is already in the IO monad. Should the function be otherwise pure, you’d be forced to “infect” it with the IO monad in order to use IORefs.↩
I wrote an entire article on just this point of implicit partial application being confusing. Unfortunately I think r/haskell was confused by my argument.

Consider the expression f 3 by itself, possibly part of a larger expression like f 3 $ g. In a typical language I consider functions to be entities that take some fixed number of arguments and then produce a single result. This means that if I encounter a well-named function f that I haven’t seen before, I can usually guess from its name what it does and what its return type is. I can also assume that an expression like f 3 will evaluate to this assumed return type.

In Haskell I cannot make that assumption: Support for implicit partial function application might mean that an expression f 3 might not evaluate to the return type of f. I have to ask myself “Does it look like this function needs more arguments here to perform the action that its name suggests?”. If the answer is yes then I assume that the expression evaluates to a new function with some unknown number of arguments with the return type I expect. That’s very confusing. I object to the possibility that two function applications that begin with f 3 ... could return different types of values.
↩

Strange Loop 2013 Highlights

2013-09-30T00:00:00+00:00

This year I attended Strange Loop, a conference on emerging tools, languages, and trends in software. I found the following sessions to be most interesting.

Session videos are available for attendees now, and for the general public closer to March 2014.

Emerging Languages Camp

Noether: Symmetry in Programming Language Design - Daira Hopwood - 45 min
- A language system, composed of several different language levels, each level breaking successively more symmetries to gain expressive power at the expense of static reasoning about the program’s behavior. Very flexible.
- This is the only language presented during the Emerging Languages Camp that had a strong theoretical foundation.
- I will definitely be looking into this language family when it has been implemented.
Nimrod: A new approach to meta programming - Andreas Rumpf - 25 min
- A new statically-typed systems language. Full-featured.
- Already has non-trivial tooling, such as an IDE.
- Has a realtime garbage collector, macros, whole program dead code elimination, and generics.
- Can automatically promote sideeffect-free functions to compile-time execution.
- Compile-time optimizations on the AST can be coded directly in Nimrod. This is similar to macros but acts like a global peephole optimization. Very powerful.
The J Programming Language - Tracy Harms - 27 min
- An array-based data language.
- Successor to the famous APL language, which allows very succinct representations of numeric computations. Although extremely hard to read for those unfamiliar with the syntax:
  - Find all prime numbers: (~R∊R∘.×R)/R←1↓ιR
  - Conway’s Game of Life: life←{↑1 ⍵∨.∧3 4=+/,¯1 0 1∘.⊖¯1 0 1∘.⌽⊂⍵}
- If you’ve never used an array-based numerical language before, this is a great beginner’s introduction.
BODOL, or How To Accidentally Build Your Own Language - Bodil Stokke - 42 min
- A guide to implementing your own language on top of Clojure.
- A very entertaining presentation.

Honorable Mention:

Qbrt Bytecode: Interface Between Code and Execution - Matthew Graham - 26 min
- A bytecode assembly language with builtin primitives for concurrency and inline asynchronous I/O.
- I liked the presentation itself although I’m not sure I’d want to use Qbert for any of my own projects.

Unsessions

QuickCheck - no video available
- QuickCheck is a testing tool that automatically generates test cases for a program given a description of its inputs, outputs, and expected semantics. Quite powerful.
- When a failing test case is found, QuickCheck is able to automatically reduce the test case to the minimal reproducing test case. Extremely useful.
- Although QuickCheck was originally written for Haskell, there are now ports to several other languages.
- There a commercial versions of QuickCheck that are superior to the open-source versions in a few ways.

Sessions & Themes

Async (Reducing callback hell)

If you only watch one of these presentations, pick the Clojure one.

Async in C# and F#
- Although not presented at Strange Loop, the implementation of the async and await keywords in C# and F# seems to have greatly inspired similar mechanisms in other languages.
- async/await avoids the callback hell problem often seen in JavaScript, where a series of nested callbacks just keeps increasing the indentation level of the code. And callbacks in the midst of complex control flow like loops are even more problematic.
Async in Scala - Philipp Haller - 44 min
- Similar to async/await in C# and F#.
- Implemented in Scala as a macro, using the experimental support for Scala macros.
Async in Clojure: Channels & Queues - Rich Hickey - 44 min
- Appears to be different in spirit to the async-implementation in other languages: Seems to be intended for implementing channels (from the Go language), rather than attempting to target the callback hell problem.
- The presentation itself has some interesting general thoughts on concurrency in software.
Async in Python
- Again, not presented at Strange Loop, but it appears that the yield from expression in Python 3 can be used to implement C#/F#-style async/await. I need to look into this.

Creativity and Artificial Intelligence

Creative Machines - Joseph Wilk - 41 min
- Interesting philosohpical discussion on how computers can create creative things.
- Teaching computers to play music.
- Joseph’s program for creating music compositions makes, in his opinion, better music than he could compose himself. This brings up the interesting observation that a student, in this case a computer, can surpass the teacher. This is not unheard of when both teacher and student are human, but it’s a bit uncomfortable to conventional thinking when the student is a “soulless” or “non-understanding” computer and the teacher is a “sentient” human.
Learnfun and Playfun: A Nintendo automation system - Tom Murphy VII - 28 min
- Teaching computers to play video games.
- I found it particularly amusing that the computer learned to manipulate the random number generator (by pausing the game for calculated periods) to create its own luck. In pinball games this allowed the computer to play perfectly.

Theory Meets Practice

The Trouble With Types - Martin Odersky - 49 min
- A balanced description of static vs. dynamic types, a classic debate in computing.
- Martin himself, being the creator of Scala, is firmly in the static typing camp but still provides a good overview of when dynamic typing is useful as well.
Add ALL the things: abstract algebra meets analytics - Avi Bryant - 37 min
- A good introduction to parts of category theory, using analytics systems for concrete examples throughout the talk.
- There was another presentation on category theory at Strange Loop but I found this talk far easier to follow.
Finding a way out - Chris Granger - 33 min
- Thoughts on the future of programming and how it can be made better.
- The second half of the presentation walks through a demo of Aurora, a new alpha-stage tool that Chris is working on, incorporating some of Chris’s new thoughts on the programming process.
- Chris Granger is the creator of Light Table, a visual programming environment that attempts to bring improved interactivity and feedback to the development process. Light Table is inspired by Bret Victor’s excellent talk Inventing on Principle.

Tools

Native Speed on the Web: JavaScript and asm.js - Alon Zakai - 30 min
- Useful overview of asm.js, which can compile C/C++ to fast JavaScript.
- This capability is very exciting since it can connect programs written for the traditional desktop environment to the web deployment platform. For example my efforts at making old Mac software easy to emulate on modern hardware, which currently requires installation on a desktop computer, could be altered to be widely deployable to the web.
The Birth & Death of JavaScript - Gary Bernhardt - 20 min (without comments) or 40 min (with)
- A very creative presentation on how asm.js could be used to make major changes to computing, bringing all traditional desktop applications to the browser.
- Uses the narrative style of presenting as if from a different time period, also used to great effect in Bret Victor’s talk on The Future of Programming.

Minorities in Computing

Women

It’s been known for some time that women are strongly underrepresented in computing. I know they’re falling out of the training pipeline early as I receive almost no women applicants to the Splunk Seattle office.

So far the primary discouraging factors I’ve been able to identify include a lack of female role models in software and a hostile learning environment created by guys when girls try to enter the field.

Making Software Development Make Sense to Everyone - Jen Myers - 58 min
- Very informative. Full of data and ideas.
Rails Girls: Empowering women through code - Adriana Palacio, Laura Garcia - 13 min
- Key takeaway is that there are a lack of female role models in computing.
The History of Women in Technology - Sarah Dutkiewicz - 40 min
- Reviews a number of important women in computing. Unfortunately very few from the modern era.

Kids

It is also difficult to get kids into computing. No longer are computers widely shipped with BASIC interpreters, HyperCard, or other well-designed introductory programming environments that kids can just jump into. Additionally the bar for what is considered an acceptable game or program has risen considerably, making it harder to get excited about simple programs.

How to teach your kid to code with Hopscotch
- Hopscotch is a visual programming language for the iPad designed to help kids get started with programming.
- Another system for introducing kids to programming is Scratch.
- Notes:
  - Finding good ideas for programs is tricky. For kids, ideas have to be applicable to their cultural context. Drawing things is more interesting than scraping Twitter.
  - Making all available tools and commands visible is important for a system designed for learning.
  - The game of “program your parent” is amusing.
  - Traditional editors, code, and IDEs actually look rather intimidating to new users. It’s like a non-aviation operator being confronted with the full control panel of an airplane cockpit.
  - Poor typing skills is a non-trivial barrier for young kids.

Other

Exercises in Style - Crista Lopes - 39 min
- A fascinating experient where the same program is written in several different styles.
- This experiment makes it obvious that not only are there lots of ways of writing the same program, but also that different languages encourage different styles.
- The material in this presentation will eventually be incorporated into a book Exercises in Programming Style that will be released in Spring of 2014. I’m definitely getting a copy.
Taking PHP Seriously - Keith Adams - 40 min
- Reviews the good parts of PHP.
- Introduces Hack, a gradually-typed version of PHP that in production use at Facebook.
- The HipHop project also includes a lot of useful tools such as a debugger, profiler, and IDE integration.

Regular Expressions

2013-09-04T00:00:00+00:00

Regular expressions are a concise way to search and transform strings using patterns. They are available in numerous text editors and programming languages.

How I learn new programming languages

2013-08-25T00:00:00+00:00

I don’t learn new programming languages very often. It simply requires an enormous amount of time to properly understand the local philosophy, community, tools, and ecosystem. But every so often I see a glimmer in an unfamilar language that merits me picking it up.

Very recently I decided to learn Clojure to fulfill my future needs for a high-power language¹. Before embarking on this journey, I thought it would be valuable to reflect on how I’ve learned other languages in the past and why I learned them.

My Journey
Patterns of Learning
Appendix

My Journey

HyperTalk & HyperCard – Play
- This was my first programming language, first used at age 6.
- The HyperCard environment was sufficiently well designed that I could play with existing programs, copy bits and pieces to assemble my own, and eventually build components from scratch.
- The physicality of the system, the straightforward syntax, and the composability of program elements enabled this kind of play. I do not know of any programming environments today in which this would be possible.
  - Visual Basic is promising, although I’ve never used it. It has physicality and native graphics.
  - DarkBASIC is also promising for making 3D games, although I haven’t used it for anything serious. It has native 3D graphics.
  - Python has a REPL that you can experiment with and has reasonably straightforward and consistent syntax. However it has no native graphics and no physicality.
  - Processing has a built-in graphics. But it’s not designed for composability at all, meaning that you can’t mix and match parts from different programs. Its function naming is also haphazard and inconsistent.
- My game Dungeons was the most advanced program I wrote in HyperCard. Other kids at National Computer Camp would line up at my computer who wanted to play it. I was very proud.
BASIC, Pascal – Instructional Setting
- At age 8 I picked these languages up at National Computer Camp because the instructors insisted on learning them before going on to other languages.
- These were my first compiled languages.
- Neither of these languages were distinctive enough for me to keep using.
  I did no major projects in these languages.
C, C++ – Brute Force, Instructional Setting
- I learned these to interface directly with the Mac operating system and to do programs in color, since HyperCard only supported black & white graphics.
- I also had some instructional assistance at National Computer Camp during the summer.
- I wish I could remember how I internalized pointers and manual memory management, since these are some of the most confusing bits of these languages for beginners. And most annoying for experts.
Java – Brute Force
- I initially learned Java in 2001 when I tried to write RealMediaFixer in C++.² When doing this I discovered that CodeWarrior’s buffered I/O implementation was busted on Mac OS, behaving in a nondeterministic fashion. So I picked the most similar-looking language to C++ I’d heard of, namely Java, and ported RealMediaFixer to use it instead. It worked on the first try and I didn’t look back to C++ for several years. (After all, why would I use a language where I/O was broken?)
- I just wrote tons and tons of programs in Java for the next 8 years or so, through the rest of middle school, high school, and college.³
- I also had some early instructional assistance at National Computer Camp. I am forever grateful to whatever instructor introduced me to the Java API documentation, allowing me to teach myself arbitrary parts of the Java standard library without relying on books or tutorials. Once you can teach yourself, the learning game is really on.
- My Java decompiler decomp4 was the most advanced program I wrote in Java.
Python – Targeted Large Projects
- While interning at Google in 2008 I noticed that they had three primary languages: Java, C++, and Python. Since Google engineers were the smartest group of developers I’d ever run into, I figured they might be on to something with Python. So I started reading about Python and its capabilities.
- My first attempt at a real program was the implementing of DiskSurveyorX in late 2008, a disk space visualizer.⁴ Unfortunately the best GUI toolkit available in Python, wxPython, I found to be far too clunky to use. So I ended up reverting to implementing in Java using the Swing toolkit.
- My second and successful attempt to write a substantial Python program was Crystal Web Archiver in late 2011, a website downloader and archival system.
  - Again I had to use wxPython. This time though I ended up writing an abstraction layer over the worst parts to remove much of the pain.
- Python is currently my favorite language for general purpose programming. Particularly for scripting, data analysis, and other kinds of exploratory work.⁵
JavaScript – Socialization, Brute Force (via Employment)
- I’ve been using JavaScript, HTML, and CSS since mid-2012 to create rich web applications, primarily in the form of Splunk’s app development framework. JavaScript was chosen because it is the lingua-franca of web development and because of its enormous leveragable ecosystem of libraries such as JQuery, Backbone, and Bootstrap.
- Advanced JavaScript is almost impossible to learn on your own because there is no patron company that controls it nor is there a Benevolent Dictator for Life to provide officially blessed documentation or tutorials for how to use the language effectively. And this language has a sufficiently large number of design flaws that you need to learn to use it in a disciplined fashion.
- I wouldn’t have been able to get far in JavaScript without help from my coworker Itay who was already proficient. Should you not have access to such a resource probably the next-best thing is to look for non-trivial projects on Github and read them. For example I learned a lot by playing with BrowserQuest and reading its source code.

Patterns of Learning

The primary patterns observed in my learning journey above include:

Play – undirected learning, by oneself
- This only works well with exceptionally well-designed programming environments. Sadly there are very few modern environments that fit the bill.
  - HyperCard is dead. Visual Basic is dying. DarkBASIC is not well known.
  - Processing and Racket work okay but have some issues.
Brute Force – writing lots and lots of programs
- This method will certainly give you the most understanding. It also takes the most time.
- Having access to a mentor to check your work is useful but not required.
Targeted Large Projects – writing a small number of projects geared toward getting the most experience
- This is similar to the “brute force” approach but is more tactical. It also requires more mental effort.

These primary patterns are coupled with a few secondary patterns:

Instructional Setting – taking a class to learn basic and intermediate skills
- I only found taking classes to be useful when my programming foundations were weak. Such classes provide very little value to me today.
- Note that taking a class isn’t enough by itself. You have to practice on your own outside the class and after the class is finished.
Socialization – working with other developers to learn intermediate and advanced skills
- Some things you can only learn efficiently from other developers.
- Socialization is particularly useful for learning things in a highly fragmented ecosystem (such as JavaScript) or in settings with advanced concepts (such as overall program architecture or the functional programming paradigm).

Appendix

There are a few languages I learned enough of to decide they weren’t worth using in new projects:

PHP – most widely deployed server-side web language
- Just say no.
Ruby – used primarily as a server-side web language
- Insufficiently distinct from Python.
- Poor backward compatibility guarantees. No release notes.
- Sloppy design: Non-formal grammar. Syntax schizophrenia. Monkeypatching accepted as a valid practice. Mutable strings.
- But superior packaging, dependency management, and isolation systems.

Unique Features of Various Programming Languages
- Describes unique features of several languages mentioned in this article.

Paul Graham’s essays on Lisp and language power are a thought-provoking read. By my assessment the current most-powerful languages out there are Haskell and various members of the Lisp family (ex: Scheme, Racket, Clojure). I’ve decided to pass over Haskell for the time being for a number of reasons, particularly because of its high learning curve. I will probably return to it eventually.↩
RealMediaFixer was a program I wrote that repaired RealMedia (.rm) files playable in RealPlayer. Such files downloaded over dialup would often get subtlely corrupted. Since downloading a 30-minute video could easily take a day it was more practical to try repairing the broken file rather than redownloading it.↩
I have a huge trove of Java programs from this time period I’d like to publish some day. A very small subset of these programs are listed on my old projects page.↩
DiskSurveyorX was a disk space visualizer program that used similar visualizations as the original DiskSurveyor program for classic Mac OS. It however had some nice usability improvements pulled from my learnings from the information visualization class I took at the Technische Universität München. A modern disk space visualizer that uses a similar visualization is DaisyDisk, which I highly recommend.↩
For desktop GUI development, the Java + Swing combination is still my favorite. Followed closely by the Objective-C + Cocoa + Mac combination. I don’t care about GUI programs on Windows or Linux. However now practically all GUI development is going to the web, where solutions based on HTML + CSS + JavaScript reign supreme.↩

Interactive Computing

2013-08-19T00:00:00+00:00

I spent a considerable amount of time yesterday going through the work of Bret Victor, who appears to be on a personal mission to bring interactive computing to mainstream developers. The tools he’s prototyping and building are just incredible.

What the heck do I mean by interactive computing you ask? This is perhaps best demonstrated by examples from Bret’s portfolio:

Inventing on Principle (54 minutes)
- Shows 5 different demos where Bret has programs or other complex objects where he can tweak the object at runtime and get immediate feedback. This is incredibly empowering.
- It makes me feel that a lot of the developer tools I’ve been using are obsolete. Particularly for advanced applications that involve the analysis and exploration of data.
Up and Down the Ladder of Abstraction (13 pages)
- Shows how an algorithm based on tunable parameters can be inspected and modified at runtime with immediate feedback.

By happenstance I’ve been fighting the problem of lacking immediate feedback recently: I’ve been writing a number of tools for processing complex semi-structured information where the algorithms needed a lot of tweaking.¹ To gain a limited form of live tweaking I wrote a function decorator that allowed me to modify and reload functions at runtime without interrupting the program. The tools that Bret is prototyping are considerably more advanced than this, allowing changes to individual expressions without even a reload step.

Other small steps in the direction of interactivity in mainstream environments include the display of non-text results in interactive prompts. The IPython environment, for example, allows image values to be displayed directly in interactive prompts. This is great for statistical visualizations on complex datasets and for image processing functions. The Racket environment (a Scheme/Lisp dialect) also supports direct display of image values. It would be nice if HTML elements, sound waveforms, and video objects were also directly displayable.

The recent work on the Light Table web IDE is directly inspired by Bret’s work, which I didn’t realize the first time I heard about it. And now I notice that Light Table has support for Python² in addition to the original Clojure and JavaScript/HTML languages. Sweet.

Exciting developments.

In case you’re wondering, these tools are designed to scrape various kinds of data from popular websites. I’m experimenting with the idea of making it easy to extract structured information out of semi-structured websites. If it was trivially easy to get data out of websites, what could you do with it? What new mashups and applications would arise that you didn’t think of before?↩
Sadly it seems that Light Table’s current Python support is limited to allowing reevaluation of individual expressions upon request. Its support for Clojure is better, including an “instarepl” which evaluates all expressions as they are typed and displays the results of all intermediate variables and data flows.↩

Fixing a function at runtime without restarting the program

2013-08-13T00:00:00+00:00

This weekend I spent some time writing programs that scrape the contents of various websites to extract information. One challenge when doing this is that webpages are a mess and the functions I wrote to parse the webpages would frequently crash due to some kind of unexpected input. So I decided I wanted a way to fix these functions at runtime without needing to restart the entire program.

So after hacking around with the Python debugger and employing some magic, I created a new function decorator called @retriable. This decorator drops into the debugger if a decorated function raises an exception, allowing you to inspect what caused the crash. Then you could fix the function in the source file and exit the debugger. Afterwards the function would be reloaded and called again with the same input as before, all without interrupting the overall program.

Demo

Let’s illustrate the decorator’s use with the following program:

# flakey.py

from retriable import retriable

@retriable
def identity(input):
    if input in [5, 7]:
        raise ValueError('Could not understand input: %d' % input)
    return input

This contrived function is intended to return its input unmodified, but fails with certain inputs.

The above function will be exercised with the following main program:

# main.py

import flakey

for i in xrange(1, 10 + 1):
    print flakey.identity(i)

Let’s try to run the main program:

$ python main.py
1
2
3
4
> /Users/davidf/Projects/scrape-tools/flakey.py(7)identity()
-> raise ValueError('Could not understand input: %d' % input)
(Pdb)

Uh oh. The function crashed. Let’s see what the input was:

(Pdb) input
5
(Pdb)

Okay. Examining the function in flakey.py we see the if-statement that’s causing 5 to fail. So we change it to read if input in [7]: to fix it and resave the file.

Having made our fix, we tell the debugger to exit, and instruct the next prompt to retry the function.

(Pdb) quit
[R]etry after reload, Rais[E], or [Q]uit? [R] r
5
6
> /Users/davidf/Projects/scrape-tools/flakey.py(7)identity()
-> raise ValueError('Could not understand input: %d' % input)

Now the function handled 5 but crashed on input 7. Again we fix the function by removing the condition completely, leaving only: def identity(input): return input. Resaving and retrying the function gives:

(Pdb) quit
[R]etry after reload, Rais[E], or [Q]uit? [R] r
7
8
9
10
$

Nice. The program completed without interruption and we were able to fix the identity function along the way.

Although this example is contrived you could imagine saving a lot of time if the function to be fixed was moderately expensive and needed to be invoked hundreds of times by the original program instead of just a handful of times. In such cases restarting the entire program is rather expensive.

The `@retriable` decorator

So, without further ago, here the @retriable decorator:

# retriable.py

import pdb
import sys

def retriable(func):
    """
    Marks a moderately expensive function that needs to support being
    debugged and fixed at runtime without interrupting the calling program.

    A function marked with this decorator that crashes by raising an exception
    will cause the Python debugger to be started at the point where the
    exception was thrown. When the debugger exits, the user will be given the
    option to reload the function and reinvoke it with the original input.

    This decorator only works for top-level functions in a module that are
    idempotent, meaning that they return the same result regardless of how many
    times they are invoked with the same input.

    Functions marked in this way should only be invoked from external modules
    via its module object. So instead of `from module import retriable_function`
    use `import module` and invoke the function using
    `module.retriable_function(...)`.

    NOTE: When a decorated function (and its containing module) is reloaded,
          any callers on the stack that reside within the reloaded module will
          continue to execute the pre-reloaded version of themselves. Therefore
          it is less confusing if functions marked as @retriable are not
          directly or indirectly invoked by methods in the same module.

    Author: David Foster (dafoster.net)
    License: MIT
    Tested On: CPython 2.7
    """

    if sys._getframe().f_back.f_code.co_name != '<module>':
        raise AssertionError(
            'The @retriable decorator can only be safely applied to ' +
            'top-level functions within a module.')

    module = sys.modules[func.__module__]
    func_name = func.__name__

    def decorated_func(*args, **kwargs):
        while True:
            try:
                return func(*args, **kwargs)
            except:
                # Start debugger at point where exception was raised
                tb = sys.exc_info()[2]
                pdb.post_mortem(tb)

                # Prompt user whether to try again or not
                while True:
                    choice = raw_input(
                        '[R]etry after reload, Rais[E], or [Q]uit? [R] ')
                    if len(choice) == 0 or choice[0].lower() == 'r':
                        # Retry after reload
                        break
                    elif choice[0].lower() == 'e':
                        # Raise
                        raise
                    elif choice[0].lower() == 'q':
                        # Quit
                        sys.exit(1)
                    else:
                        # Not understood... ask again
                        continue

                # Retry after reload, recursively
                reload(module)
                return getattr(module, func_name)(*args, **kwargs)
    return decorated_func

Some limitations and caveats about this decorator’s use are described in its docstring.

Enjoy.

Sending email from command line scripts

2013-07-27T00:00:00+00:00

This weekend I open-sourced a script called notifymail which I have been using for the past few years to send myself emails from automated scripts, particularly Python scripts.

It is very easy to configure notifymail for the first time:

$ pip install notifymail
$ notifymail.py --setup
SMTP Server Hostname: smtp.gmail.com
SMTP Server Port [465]: 587
SMTP Server Uses TLS (y/n) [n]: yes
SMTP Username: robot@gmail.com
SMTP Password: ********
From Address [robot@gmail.com]: robot@gmail.com
From Name (optional) []: notifymail
To Address: admin@example.com

Verifying connection to SMTP server... OK

Check your mail provider’s documentation to get the SMTP settings mentioned above. For example I made an internet search for “gmail SMTP settings” to find Gmail’s SMTP settings:

Gmail SMTP Server: smtp.gmail.com
Gmail SMTP Port: 587 (for TLS)
Gmail SMTP Uses TLS? yes

Once you have notifymail installed, you can send an email to yourself in a Python script with as little code as:

import notifymail
notifymail.send('Subject', 'Hello World', from_name='greeting_script')

Or you can invoke notifymail from the command line:

$ echo "Hello World" | notifymail.py -s "Subject"

Full documentation is available on the notifymail project page.

Hasn’t this been done before?

I reinvented my own wheel to send email principally because of the poor documentation of other alternatives:

mail and postfix were so complicated I couldn’t figure out how to set them up.
ssmtp didn’t work after I tried to configure it and there was no good documentation to help me debug why it wasn’t working.

For reference, here’s some information for setting up those alternatives.

Fun things to do with `notifymail`

I have a script called heartbeat that periodically attempts to connect to all of my servers via SSH. If it cannot connect to a server it sends me an email with notifymail. If it cannot access email it displays a sticky Growl notification locally.
I have another script called meetupfilter that tracks incoming “New Meetup Group” emails from Meetup. It waits until all such announced groups have at least 3 events on their calendar before sending me a notification at my personal email with notifymail. That way I don’t hear about Meetup groups that appear but then fizzle out immediately, which is a surprising number. I may open-source this script eventually if I hear there is interest.

Error Handling

2013-07-13T00:00:00+00:00

Most of the time programs operate happily inside their main scenario. Occasionally they need to cope with unusual circumstances, such as being no longer able to read data from the network because the user has turned off their WiFi. The process by which a program responds to an error is called error handling.

Behaviors Upon Failure
Reporting Errors
Error Seriousness
Guarantees after Failure
Failing Fast (and Error Locality)
- assert vs. if
Implementation Considerations for Exceptions
Summary

Behaviors Upon Failure

At the time an error occurs in a function there are a few things it can do:

Delegate to its caller
- Typically a function doesn’t have enough information to decide how to handle the error itself. In such a case the error is reported to the calling function, which is typically higher-level and has more context.
- For example if an I/O error occurs while parsing data from a file, the parser will typically just report the I/O error to the code that invoked the parser. The parser itself only knows how to parse things and not what should be done in the case of an I/O error.
- The particular error handling strategy of a function reporting an error to its caller is so common that many programming languages provide special mechanisms to support it, particularly in the form of exceptions.
Display the error to the user and continue
- This is a common pattern for routine errors in interactive programs.
- For example a web browser that cannot access a particular webpage will display an error page explaining why the page could not be accessed.
Display the error to the user and exit
- This is a common pattern for unexpected errors in contexts where total program failure is deemed acceptable.
- In languages that use exceptions, any unhandled exceptions that bubble out of the main function or to the initial function in a thread will terminate the program or the thread. Programs can avoid this fate by adding a top-level exception handler that takes alternate action.
- Examples:
  - In PHP it is an idiom to use the die() function upon the failure of many operations:
    - ```
    $connection = mysql_connect('localhost', 'dbuser', 'password') or die('Could not connect to MySQL server.');
```
- The Linux kernel makes great use of the panic() function when it gets into a bad state, stopping the entire operation of the computer.
Log the error and continue
- This is a common pattern for non-interactive programs and programs that need to keep running continuously, such as web servers.
- Naturally there needs to be somebody who goes through the log occasionally to look for problems if this error handling strategy is used.
Ignore the error implicitly and continue blindly
- This is almost always done out of laziness.
- In C, this just means ignoring the result of a function that returns an error code.
Ignore the error explicitly and continue boldly
- This is usually done out of laziness.
- One legitimate case when this is done is ignoring I/O errors when closing a resource during cleanup, since there isn’t anything sensible that can be done at that point. A Java example:
  - ```
  r = new Resource();
  try {
      ...
  } finally {
      try {
          r.close();
      } catch (IOException e) {
          // ignore
      }
  }
```

Less common handling strategies include:

Prompt the user to decide what to do
Attempt recovery and retry (or resume) the original operation, perhaps with different inputs

Reporting Errors

When a function wants to report an error, it has to pass information about the error to its caller. There are a few ways that error information can be encapsulated:

Error Codes
- When a function is designed to report errors using error codes, it is declared to return an integer or other integer alias (ex: OSErr). Upon success the function returns 0 (or noErr). Upon failure the function returns some other value that describes the type of error encountered. Should the function need to return other values upon success it must declare out parameters to pass them back to the caller.
- Error codes have several disadvantages that make their use uncommon in modern high level languages:
  - An error code only describes the type of error encountered, but not any additional contextual information that would be useful in handling the error. Typically only the function that is one hop away from the error site has enough additional context to actually handle the error nicely.
    - For example if you try to open a nonexistent file on classic Mac OS, you’ll get back the error code fnfErr which indicates that the specified file was not found. Note however that this error code alone doesn’t include information about which file couldn’t be found, which is necessary context to display a meaningful error message.
  - A function that uses error codes relies on its caller to check whether it returns an error. Lazy callers might not actually check, causing any reported errors to be ignored silently.
    - For example I have never seen a C program actually check the result of printf(), even though it can return an error code upon I/O error. This behavior is reasonable in this case because there isn’t much a program can do to alert the user if it can’t print to the screen.
    - More problematic are C programs that fail to check the result of malloc() for NULL when allocating objects. Such programs will crash when malloc returns NULL during an out-of-memory condition and the caller tries to manipulate it.
  - When an error code occupies the return value of a function, that return value can’t be used for “normal” function output. This typically means that multiple functions returning error codes cannot be chained together on the same line, increasing verbosity.
- The one advantage of error codes is that they are very fast to create and return, improving performance in fragments of code that experience a large number of errors. However this performance-over-usability tradeoff is almost never worth it in modern programs.
- These days error codes are only used commonly in programming languages like C that don’t support the use of exceptions.
Error Sentinels (such as null, -1, or 0)
- A function that reports errors using error sentinels uses its return value to return the normal output of the function most of the time. However when an error occurs, a special invalid¹ value (a sentinel) is returned instead.
  - For example if you read() from an InputStream in Java, usually the resultant byte value is returned. But if the end-of-file is reached, the special value of -1 is returned instead.
- Error sentinels suffer from the same disadvantages as error codes. And they have the same performance advantage.
- Unfortunately the use of error sentinels, especially null, remains common in many modern programs despite their disadvantages. Even in cases where a Null Object could be used to better effect.
- Some statically typed languages like Haskell and SML explicitly represent the potential of a sentinel being returned using a datatype such as Maybe or Option. Typically this is done so that the compiler can automatically flag callers that aren’t checking for sentinels properly, eliminating their largest disadvantage.
Exceptions
- An exception is an object that encapsulates information about an error. This includes not only the type of error but also additional contextual information that can be used to generate a reasonable error message and assist in debugging by an end-user or developer.
  - Exceptions also typically include a stack trace which can be used to pinpoint the location in the program where the error occurred originally, which is valuable for debugging.
  - Some languages like Java and Python provide exception chaining (or wrapping), which is special support for higher-level exceptions to reference any lower-level exceptions that caused the higher-level exception to occur.
- A function that uses exceptions for error reporting can throw (or raise) an exception when an error occurs. When this happens the runtime looks for an enclosing exception handler that knows how to deal with the exception. If it finds a handler the handler is run. If no such handler is found the function terminates and the same exception is rethrown at the point in the caller function where the throwing function was called. Should there be no caller, such as in the case of the main function or the initial function of a thread, the program or thread terminates.
- Exceptions are the preferred method for error handling in most modern languages. They have the principal advantages of preserving detailed information about the error that occurred and being difficult to accidentally ignore.
- One disadvantage of using exceptions is that they are relatively expensive to create and throw which can be a problem if the related error is relatively common, such as attempting to read past the end of a file. They also typically require special language support for throwing them, which rules out their use in simpler languages like C.
- Some languages such as Java additionally have the notion of a checked exception. When a function throws a checked exception, it is required to declare the type of the exception in its signature. When a caller tries to invoke a function that declares a checked exception in its signature, the compiler enforces that the caller actually handles the exception - either with an exception handler or by delegating to a higher-level function.
  - It is challenging to design APIs that use checked exceptions since they propagate virally through function declarations. I will probably write a future article that specifically deals with using checked exceptions effectively.
  - Some statically typed languages like Haskell implement the semantics of checked exceptions using a datatype such as Either. Compilers in such languages enforce that callers explicitly unpack the Either value and handle any contained exception object explicitly.

In summary:

Exceptions should be used for most error reporting needs.
Error sentinels should be used to report common errors where the performance penalty of exceptions is too much.
Error codes should be avoided entirely unless the programming language does not support exceptions, in which case error codes should be used instead.

Error Seriousness

Often the seriousness of an error is related to how it is handled. Generally speaking, errors are either:

Common
- For example reading past the end of a file falls into this category, as it will always happen (once) whenever a program attempts to read an entire file into memory.
- Very few errors fall into this category.
Expected
- These errors are likely to occur in normal operation, but not commonly.
- I/O errors related to reading from a file or from a network socket are an example.
Unexpected
- Unexpected errors are ones for which the programmer did not plan any explicit handling.
- This includes dividing by zero, dereferencing a null pointer, attempting to access an object off the end of an array, reading from a file that was closed, and other “programmer errors”.
Fatal
- Fatal errors are a rare and particularly nasty subset of unexpected errors for which there is no good recovery strategy. Such errors almost always terminate the program when they occur.
- Fatal errors include running out of memory², the inability to locate a method or library which the program was linked against, and other nasty conditions.

Common errors are typically reported using error sentinels or error codes for performance reasons. Other error categories are usually reported using exceptions unless the language in use doesn’t support exceptions, in which case error codes are used instead.

The standard libraries of programming languages typically distinguish between expected, unexpected, and fatal errors by using different exception base classes for each. This makes it easy to write exception handlers that can catch all thrown exceptions in a particular category. For example:

Java uses Error as the base class for fatal errors, RuntimeException for unexpected errors, and all other subclasses of Exception for expected errors.
C# uses SystemException (informally) as the base class for unexpected and fatal errors. All other subclasses of Exception are used for expected errors.³
Python uses StandardError (informally) as the base class for unexpected and fatal errors. All other subclasses of Exception are used for expected errors.⁴

If a language supports checked exceptions, the expected exceptions should generally be marked as checked to force callers to handle them appropriately. Conversely, unexpected and fatal exceptions should not be marked as checked since this burdens the caller unnecessarily.⁵

Guarantees after Failure

When an error occurs in the middle of multi-step function, the function has to make a decision about what kind of state it wants to leave the program in when it returns to its caller. In general a function can leave the program:

in its original state,
- This usually involves rolling back or reversing any actions performed prior to the error.
- Functions that make this guarantee are called atomic or transactional.
in a different but valid state, or
- If a full rollback is not possible, it is still usually possible to partially rollback to a valid state.
in a different and potentially illegal state.
- Naive functions may simply return immediately upon error, leaving whatever resources they were using in a bad state.
- This is particularly a problem for programs that use exceptions for error handling, as the default behavior when an exception is thrown is to return immediately, without performing any cleanup.

Any complex fault-tolerant function should document which of these guarantees it makes.⁶ If it makes no guarantees at all, the caller may have to assume that whatever resource the function was operating on is in a bad state if the function returns an error.

Consider the concrete example of a program that copies a comma-separated-value (CSV) file to a new file. This generally involves the steps:

Open the source CSV file.
Open the destination file, replacing any previously existing file.
Loop over each row (or byte) of the source CSV file and write each row to the destination.
Close the destination file.
Close the source file.

Here is a naive implementation that makes no guarantees to its caller in the event of an error.

/** Copies the source CSV file to the destination file. */
// (I didn't think about error handling at all.)
public static void copyCsvFile(File sourceFile, File destFile) throws IOException {
    InputStream fileIn = new FileInputStream(sourceFile);
    OutputStream fileOut = new FileOutputStream(destFile);
    int b;
    while ((b = fileIn.read()) != -1) {
        fileOut.write(b);
    }
    fileOut.close();
    fileIn.close();
}

Now imagine what happens if an I/O error occurs in the middle of copying bytes: The write method will throw an IOException and since copyCsvFile has no matching exception handler, the copyCsvFile function itself will stop and rethrow the IOException. Notably, the destination file is left with incomplete and invalid CSV contents. And neither the source nor the destination file is closed, leaking those resources from the operating system.

We can at least avoid leaking resources by adding logic that ensures that resources are always closed when the function completes:

/** Copies the source CSV file to the destination file. */
// (No open file handles will be leaked even in the event of an error.)
public static void copyCsvFile(File sourceFile, File destFile) throws IOException {
    InputStream fileIn = new FileInputStream(sourceFile);
    try {
        OutputStream fileOut = new FileOutputStream(destFile);
        try {
            int b;
            while ((b = fileIn.read()) != -1) {
                fileOut.write(b);
            }
            fileOut.flush();
        } finally {
            try {
                fileOut.close();
            } catch (IOException e) {
                // Ignore I/O errors upon close since nothing can be done
            }
        }
    } finally {
        try {
            fileIn.close();
        } catch (IOException e) {
            // Ignore I/O errors upon close since nothing can be done
        }
    }
}

This improved function will no longer leak open file handles in the event of an error but it will still leave an invalid destination CSV file.

If we document the additional guarantee that the function performs an atomic file copy, we’d want to explicitly code the function to delete the destination file in the event that it couldn’t be fully copied. Here’s an implementation:

/** Copies the source CSV file to the destination file atomically. */
public static void copyCsvFile(File sourceFile, File destFile) throws IOException {
    InputStream fileIn = new FileInputStream(sourceFile);
    try {
        OutputStream fileOut = new FileOutputStream(destFile);
        boolean finishedCopying = false;
        try {
            int b;
            while ((b = fileIn.read()) != -1) {
                fileOut.write(b);
            }
            fileOut.flush();
            finishedCopying = true;
        } finally {
            try {
                fileOut.close();
            } catch (IOException e) {
                // Ignore I/O errors upon close since nothing can be done
            }

            if (!finishedCopying) {
                boolean deleteSuccess = destFile.delete();
                // If the delete fails then the rollback failed.
                // Since there's nothing that can be done in that case,
                // we ignore deletion failures.
            }
        }
    } finally {
        try {
            fileIn.close();
        } catch (IOException e) {
            // Ignore I/O errors upon close since nothing can be done
        }
    }
}

If we wanted to get even more fancy we could document instead that the function guarantees that it will copy as much of the source CSV file to the destination CSV file, leaving the longest valid destination CSV file even in the case of an error. In particular if the entire file cannot be copied, the function will copy as many complete lines from the source CSV file as possible, stripping off any incompletely written lines.

This is actually relatively difficult to implement correctly in Java while still handling characters correctly and preserving end-of-line sequences, so here’s a Python 2 implementation instead:

def copy_csv_file(source_filepath, dest_filepath):
    """
    Copies the source CSV file to the destination file.

    If an error occurs while copying, as many rows as possible are copied,
    leaving a valid destination CSV file.
    """
    with open(source_filepath, 'rb') as file_in:
        offset_to_last_line_written = 0
        finished_copying = False
        file_out = open(dest_filepath, 'wb')
        try:
            while True:
                cur_line_bytes = file_in.readline()
                file_out.write(cur_line_bytes)
                offset_to_last_line_written = file_out.tell()
            file_out.flush()
            finished_copying = True
        finally:
            truncated_successfully = False
            if not finished_copying:
                try:
                    file_out.truncate(offset_to_last_line_written)
                    truncated_successfully = True
                except IOError:
                    # Unable to truncate. Will try to delete the file instead...
                    pass

            try:
                file_out.close()
            finally:
                if not finished_copying and not truncated_successfully:
                    try:
                        os.remove(dest_filepath)
                    except IOError:
                        # Unable to truncate or remove the destination file.
                        # Nothing else can be done.
                        pass

Actually the preceding implementation isn’t correct in the presence of output stream buffering (unless it is line-buffered), since it could be the case that the offset_to_last_line_written points to the end of a line that in fact has not been written to disk but is rather in the output buffer. A correct and performant implementation that additionally handles that case is left as a (non-trivial) exercise for the reader.

Failing Fast and Error Locality

Errors are easiest to handle when they are signaled at the exact point where a problem first occurred, or as close to it as possible. Thus functions should try to fail fast whenever possible.

It is a good idea for functions to check their inputs (especially their arguments) immediately upon invocation to see whether they conform to the expected format. This provides early warning of state corruption that could get introduced into the derived output of the function.

In addition if there are points where a function can make a non-trivial assertion about its current state, and this assertion is at risk of breaking due to modifications by maintainers, it should make an explicit check that the assertion is true.

`assert` vs. `if`

When checking assertions, a function can always use the humble if statement:

public class Registry {
    private Map<String, Object> items = new LinkedHashMap<String, Object>();

    public void register(String id, Object item) {
        if (id == null)
            throw new IllegalArgumentException(
                "Cannot register an item with a null ID.");
        if (item == null)
            throw new IllegalArgumentException(
                "Cannot register a null item with ID \"" + id + "\".");

        if (items.containsKey(id))
            throw new IllegalStateException(
                "Already have an item registered with the ID \"" + id + "\".");
        items.put(id, item);
    }

    // (... more methods ...)
}

However there is also an assert statement in many languages. The assert statement typically differs from if in that it can be compiled-out of the program automatically if desired, for a modest performance boost at the expense of safety. Therefore assert should typically only be used in performance-critical code (that has been verified as such by a profiler).

In practice I almost never use assert, preferring to rely on if instead.

Implementation Considerations for Exceptions

Exceptions are API

Expected exceptions⁷ are part of a function’s API. Consequently:

Expected exceptions should be given the same coverage in a function’s documentation as its parameters or return type. Remembering to document expected exceptions is particularly important when writing API documentation for languages lacking checked exceptions (i.e. everything other than Java, including C#).
Callers may depend on the expected exceptions in the function’s API documentation.

A function cannot remove or change the exceptions it throws without breaking callers that have been coded to expect the old set of exceptions. And new exceptions that are added will not be expected by existing callers.

Designing Exception Hierarchies

It is generally a good idea have a separate exception type for each specific type of error that a caller might want to handle distinctly. That way a caller can easily write an exception handler that catches a specific exception of interest. For example FileNotFoundException is likely to be treated distinctly from a generic IOException, so it is given a separate exception type (that inherits from the generic IOException class).

Errors that are not likely to be handled distinctly by the caller can just reuse a generic exception class directly. For example a piece of code that detected a “bad media” error when reading from a disk could just throw a plain IOException with a message instead of creating a special subclass. However note that by doing this you provide no API for any caller that does in fact want to catch this kind of exception.⁸

Designing Error Messages

An error message typically accompanies an exception, and it is this message that is typically presented verbatim to the user if high-level code doesn’t recognize the exception type itself. Therefore it is important that the message be:

well-formatted,
- Therefore basic grammar rules apply, such as beginning sentences with a capital letter and ending them with a period.
- A surprising number of real-world exception messages don’t even meet this low bar.
understandable by the user,
- This is a tricky requirement because the point where an exception is generated may not know the kind of user that will eventually receive the exception, particularly if it is a low-level error that is deeply buried or if the error occurs in a widely-used utility function.
- In general expected errors are likely to reach a normal non-programmer end-user, so they should be written with that audience in mind.
- By constrast unexpected and fatal errors are typically programmer errors and only likely to be seen by other developers when developing a program. Therefore they should be written with the developer in mind.
specific, and
- An error message without sufficient contextual information cannot be easily corrected by the person that receives it. Messages should contain enough information for the user to easily locate the problem and fix it themselves.
- For example a error encountered while parsing a text file should include information about the location of the parse error, so that the user can find the problem in the original document. Any good compiler will give you at least a line number when it encounters a syntax error and may even provide a specific column number as well.
in the correct spoken language.
- In particular if the user’s current locale is non-English (such as German), a German message should be presented if possible, not a default English one.
- Programs that are only intended to be used in a single locale can ignore this guideline and naively emit messages in the expected locale’s language.

Translating Exceptions During Handling

Wrapping Low-Level Exceptions in High-Level Exceptions

It is common for a single system to have a single high-level exception type that is thrown by most of its functions. For example a parser’s functions may all throw the high-level ParseException.

However the implementation of such a system may delegate to other systems that use a different set of exceptions. In the case of a parser, it typically has to read the characters it is parsing from an I/O stream, which may throw a low-level IOException.

Now, top-level parser functions have a few options in reporting the underlying IOException as a possible failure case to its caller:

Expose - The top-level function declares that it throws both ParseException and IOException.
- However this exposes the implementation detail that the parsing system relies on the I/O system and forces the caller to deal with the low-level IOException directly.
- Therefore this is not generally recommended for top-level functions, although it may be used by internal functions.
Wrap - The top-level function intercepts IOExceptions and translates them to a generic ParseException that wraps the original IOException.
- This allows the top-level parsing functions to present the simpler exception API of just throwing a ParseException.
- Callers of the parser remain able to intercept and extract the underlying IOException if they wish.
Map - The top-level function intercepts IOExceptions and maps them to a special subclass of ParseException (like ParseIOException), optionally wrapping the original IOException for further inspection by the caller.
- This is similar to the “wrapping” approach, but makes it even easier for the caller to catch the underlying exception since it can directly catch the mapped exception (ParseIOException).
- Thus this approach is preferable if the underlying failure case is sufficiently important to advertise prominently in the exception API.

In summary, most low-level level exceptions should be wrapped in the generic high-level exception. For prominent low-level exceptions, they should be mapped to a specific high-level exception subclass.

Here’s a Java example of a parser taking the “wrapping” approach to low-level I/O exceptions.

// The top-level methods of this class all throw ParseException in their API.
public class RuleParser {
    public static Rule parse(InputStream input) throws ParseException {
        try {
            return new RuleParser(input).readRule();
        } catch (IOException e) {
            throw new ParseException("I/O error while parsing rule.", e);
        }
    }

    private Rule readRule() throws ParseException, IOException {
        // (...)
    }

    // (...)
}

Wrapping Error Codes in Exceptions

Sometimes a high-level function using exceptions needs to report an error received from a low-level function that uses error codes. In this case the low-level error code needs to be communicated to the caller somehow via an exception.

Typically the low-level function comes from an overall subsystem of some kind which uses error codes in general for error reporting. In such a case it is typical to define a generic exception to wrap all error codes received from the subsystem. This generic exception should preserve the original error code for inspection by callers, along with whatever extra context may be available from the subsystem, typically just an error message.

For example Python uses the OSError exception to wrap error codes received from the underlying C library. It is populated with the error code received from the C errno global variable and the message received from the C function perror().

Now the high-level function may not want to report this kind of low-level subsystem exception directly, in which case the “wrap” or “map” technique discussed above in “Wrapping Low-Level Exceptions in High-Level Exceptions” should be used in addition.

Promoting Error Sentinels to Exceptions

Typically error sentinels are used to report common errors that are intended to be handled immediately by the caller. However if the caller cannot handle the error itself but needs to delegate to its second-order caller, it typically needs to promote the sentinel to an exception.

As an example, the end-of-stream condition when reading from a stream is generally considered to be a common error in Java. However a function that is parsing a data structure out of a stream does not expect an end-of-stream when it is in the middle of parsing a structure. Thus the parser wishes to report the end-of-stream condition to its caller as either an expected or unexpected error (depending on context), both of which require an exception.

public class BinaryInputStream extends FilterInputStream {
    public BinaryInputStream(InputStream in) {
        super(in);
    }

    public int readUInt8() throws IOException {
        int b = this.read();
        if (b == -1) {
            throw new EOFException("Unexpected end of stream.");
        }
        return b;
    }

    public int readUInt16() throws IOException {
        return
            (readUInt8() << 8) |
            (readUInt8() << 0);
    }
}

In the preceding example the -1 error sentinel was promoted to an EOFException.

Avoid Using Error Codes in Generic Exceptions

The use of internal error codes within a generic exception class should generally be avoided, since this makes it difficult to handle them. (One exception to this guideline is when using an exception to wrap external error codes received from another subsystem, as described above in “Wrapping External Error Codes in Exceptions”.)

Consider the following example:

// A generic exception that wraps its own set of error codes. DON'T DO THIS.
public class FetchException extends RuntimeException {
    private int code;
    private String text;

    public static final int JOB_NOTREADY = 1;
    public static final int TIMEOUT = 2;
    public static final int AMBIGUOUS_NAME = 3;

    FetchException(int code, String text) {
        super(text);
        this.code = code;
        this.text = text;
    }

    public int getCode() {
        return code;
    }

    public String getText() {
        return text;
    }
}

If you actually wanted to detect the JOB_NOTREADY case, you’d have to write code like:

public Job fetchJob(String jobName) {
    for (int triesLeft = MAX_FETCH_ATTEMPTS; triesLeft > 0; triesLeft--) {
        try {
            return service.getJobs().get(jobName);
        } catch (FetchException e) {
            if (e.getCode() == FetchException.JOB_NOTREADY) {
                // Retry again
                continue;
            } else {
                throw e;
            }
        }
    }
    throw new FetchException(TIMEOUT,
        "Job \"" + jobName + "\" was not ready after " +
            MAX_FETCH_ATTEMPTS + " fetch attempts.");
}

It’s not pleasant having to put that if-statement in the exception handler. And the throwing of the TIMEOUT-coded FetchException couldn’t save contextual information like the jobName and MAX_FETCH_ATTEMPTS in a machine-readable field since the generic FetchException didn’t have fields that were specific to the TIMEOUT code.

A better solution would be to use specific subclasses of FetchException instead:

public class FetchException extends RuntimeException { ... }

public class JobNotReadyException extends FetchException { ... }

public class FetchTimeoutException extends FetchException {
    private String jobName;
    private int numFetchAttempts;

    FetchTimeoutException(String jobName, int numFetchAttempts) {
        super("Job \"" + jobName + "\" was not ready after " +
            MAX_FETCH_ATTEMPTS + " fetch attempts.");
        this.jobName = jobName;
        this.numFetchAttempts = numFetchAttempts;
    }

    // (... Accessors for jobName and numFetchAttempts ...)
}

public class AmbiguousNameException extends FetchException { ... }

Then the code could be simplified to just read:

public Job fetchJob(String jobName) {
    for (int triesLeft = MAX_FETCH_ATTEMPTS; triesLeft > 0; triesLeft--) {
        try {
            return service.getJobs().get(jobName);
        } catch (JobNotReadyException e) {
            // Retry again
            continue;
        }
    }
    throw new FetchTimeoutException(jobName, MAX_FETCH_ATTEMPTS);
}

Avoid Using Exceptions for Flow Control

Sometimes programs report common errors in the form of an exception instead of using a more appropriate mechanism such as an error sentinel. This is inefficient since throwing exceptions is slow in the common case. And it is awkward for the caller who must have an explicit exception handler around every invocation to deal with the common case. Don’t do it.

Using Exceptions for Thread Termination

However one case where exceptions are useful as a means of flow control is to force (or recommend) that a thread terminate. Such exceptions are classified as fatal errors so that most exception handlers ignore them.

For example Java uses the ThreadDeath exception (a subclass of the fatal Error) to terminate a thread. And Python uses the KeyboardInterrupt exception (a subclass of the fatal BaseException) to kill the main thread when the user presses Control-C.

Summary

Error handling is hard. But you’ll provide a better experience by properly handling and communicating errors back the user.

Don’t be that guy who provides useless generic error messages:

(I doubt even the program itself knows what went wrong.)

Error handling styles in programming
- Summarizes the most prominent strategies for handling runtime errors.
Abandonment vs. Unchecked Exceptions for Error Handling
- Describes abandonment, an uncommon error-handling style.

Series

This article is part of the Programming for Perfectionists series.

Flawed functions that return sentinel values that are actually valid place extra onus on the caller to perform additional checks to determine whether an error actually occurred. For example PHP’s stream_get_contents function can return '' or FALSE upon failure. But it can also return '' upon success. See my insane workaround.↩
Most programs treat out-of-memory errors as fatal, although there are a few hardened programs like SQLite that treat out-of-memory as an expected error.↩
The C# exception hierarchy is illustrated in “C# exception hierarchy”.↩
The Python exception hierarchy is documented in “Exception Hierarchy”.↩
Java has a few annoying examples where unexpected exceptions were marked as checked, burdening all subsequent callers. In particular Object.clone() throws the checked CloneNotSupportedException, making it hard to use. And Java’s reflection library throws the checked IllegalAccessException and InvocationTargetException whenever you try to invoke() a method, neither of which are expected errors. And Thread.sleep() throws the checked InterruptedException. Now IOException, thrown by all I/O functions, is legitimately a checked exception because it is an expected exception.↩
Instead of documenting the guarantees after failure for individual functions, it also common to document an overall failure handling strategy for the entire system. For example most databases are documented as generally operating in a transactional fashion, with failed operations leaving the database in its original state. Of course some functions may deviate from the general policy, in which case the deviation should be documented.↩
Unexpected and fatal exceptions, on the other hand, are not typically part of a function’s API. As such, callers should not write exception handlers that depend on them.↩
In this circumstance a caller is forced to guess the exception type by parsing the exception’s message. However this solution is brittle since the message isn’t part of the API and could change in the future or vary depending on the current locale.↩

Handling Text Correctly

2013-06-01T00:00:00+00:00

A remarkable number of programs do not handle text in a reasonable fashion, which causes those programs to break when confronted with non-English characters and symbols. Even the standard libraries of most programming languages are not immune.¹

This chapter describes the foundation of how text handling works in software and gives examples of common pitfalls when working with text in real code. Upon completion you should be able to evaluate your favorite programming environment’s built-in support for handling text and be able to write programs that handle text correctly and consistently across operating systems.

Characters, Sets, and Encodings

Important Character Sets
Unicode

Unicode Character Encodings

Common Mistakes & Practical Tips

You cannot interpret a byte array as a string without knowing its encoding.

Reading Text Files
Converting Between Bytes and Strings

A "char" (in your favorite language) is probably not a character.

8-bit chars
16-bit chars
32-bit chars and variable-bit chars

End of Line Sequences
Summary

Characters, Sets, and Encodings

Text is made up of multiple characters (or codepoints²), each of which represents a different letter, symbol or punctuation mark. The word “Hello”, for example is made of the characters “H”, “e”, “l”, “l”, and “o”. This collection of characters is called a string. Each character is assigned a number using a character set (sometimes called a code page or a charset).

For example the ASCII character set assigns numbers in the range (0-127) for characters in most Western European languages.

H → 72

e → 101
 
l → 76

l → 76

o → 111

ASCII is one of the oldest character sets. Most other sets use the same mappings as ASCII, while defining additional mappings of their own for numbers above 127.

After mapping a character to a number, that number is converted to an actual byte sequence for storage. The entire process of converting a character to a byte sequence is defined by a character encoding.

Since many character sets provide mappings to numbers in the range 0-255 (which fit in 8-bit bytes), you can output the character number as a single byte. For example, to output “Hello” in ASCII, you would emit the byte sequence:

H	e	l	l	o
72	101	76	76	111

Many English-speakers, being only familiar with the Western European character sets which are always encoded with individual 8-bits bytes, often use the terms character set and character encoding interchangably.

However the East Asian languages such as Chinese, Japanese, and Korean (CJK) have many more than 256 characters, making it impossible to fit them in only 8-bit bytes. Therefore each character must be encoded using multiple bytes, possibly a variable number of bytes. Thus either a fixed-width encoding (often 16-bits per character) or a variable width encoding (with differing numbers of bytes per character) may be used.

Important Character Sets

The following are the most prevalent non-Unicode character sets you are likely to encounter. All of these character sets are encoded with a single byte per character.

Character Set	Character Encoding
ASCII	single byte (7-bit)
Windows-1252 (Windows Latin 1)	single byte
ISO 8859-1 (ISO Latin 1)	single byte
Mac OS Roman	single byte

Informally (and in practice), these may also be described as character encodings. So a text file “in ASCII encoding” refers to a file in the ASCII character set with its standard single-byte encoding.

ASCII
- 7-bit character set for representing characters common to most Western European languages.
- ⚠ Many programs and APIs that claim to input or output “ASCII” (or “ANSI”³) are actually unaware of character sets and will accept strings in whatever the operating system’s default character set happens to be (often Windows-1252).
Windows-1252 (AKA Windows Latin 1)
- This is the default character set on English Windows systems.
ISO 8859-1 (AKA ISO Latin 1)
- The default character set for web pages and other HTTP traffic.
- ⚠ In practice this is very frequently confused with Windows-1252, which differs in only a handful of characters. In fact this confusion is to such a degree that the draft HTML 5 specification requires that documents advertised as ISO-8859-1 actually be parsed as Windows-1252.⁴
Mac OS Roman
- This is the default character set on most Macintosh systems prior to Mac OS X.

Not every character set can represent every character in the world. For example Windows-1252 cannot represent any Chinese or Japanese characters, although GBK and Shift JIS respectively can. This means that you cannot in general mix text from two sources that uses a different character sets without converting to a new character set (that can represent all characters in both sets). Typically the Unicode character set (discussed below) is used for this purpose.

Character Set	Character Encoding
Windows-932 (“Shift JIS”)	variable width, 1-2 bytes
Windows-936 (“GBK”)	variable width, 1-2 bytes

Unicode

Unicode is a character set just like ASCII or Windows-1252: it maps characters to numbers.⁵ However it is sufficiently important to justify special mention.

The Unicode character set is designed to support all characters in all character sets that came before it, plus many more. If you can think of a character, it’s almost certainly in Unicode. (And if it isn’t, it’s likely not in any character set.) In this sense, Unicode can be viewed as the universal character set.

Any program written today that wishes to represent characters correctly should be using the Unicode character set and one of its associated encodings.

Originally, Unicode mapped characters to numbers in the range 0x0000-0xFFFF, requiring only 16 bits to represent each character. At that time it was possible to encode each Unicode character using a 2-byte fixed width encoding, known as UCS-2. Many early Unicode-aware systems, such as Java, were designed around this original specification.

In 1996, Unicode was extended to map characters to numbers in the larger range of 0x00000-0x10FFFF, requiring up to 20 bits per character. Thus UCS-2, being limited to 16-bits, was no longer capable of representing all Unicode characters. In its place arose the UTF-16 encoding which, like UCS-2, uses a single 16-bit value to represent characters in range 0x0000-0xFFFF (the basic multilingual plane) and two 16-bit values to represent characters in range 0x10000-0x10FFFF (the supplemental planes). And so many UCS-2 systems were retroactively upgraded to use UTF-16 in place of UCS-2.

Today, UTF-16 is the most common in-memory representation for Unicode strings. However, many programs incorrectly treat individual 16-bit values from UTF-16 directly as characters, due to ignorance of UTF-16’s variable-width nature. In many cases this causes no problems, since most programs operate on strings and substrings opaquely, as opposed to working with individual characters. However problems will arise if unaware programs attempt to manipulate characters directly, such as by counting the number of characters in a string or by filtering individual characters.

Unicode Character Encodings

Unlike the other character sets discussed previously, the Unicode character set has multiple different encodings.

Character Set	Character Encoding	Character Encoding Scheme
Unicode	UTF-8	variable width, 1-4 bytes
	UCS-2	fixed width, 2 bytes
	UTF-16	variable width, 2 or 4 bytes
	UTF-32 / UCS-4	fixed width, 4 bytes

UTF-8
- This is the default on-disk encoding for most modern Unicode-aware programs.
  - Many text editors save in this format by default.
  - Python 2.5+ source files are assumed to be UTF-8 by default.
- It can represent any character in the Unicode character set, and thus any character in the world.
- It is compact, representing all ASCII characters as single bytes and most other characters as two bytes.
- ⚠ Programs may optionally prepend a byte-order-mark (BOM) at the beginning of a file to mark it as UTF-8. Most Windows programs do this, for example. Programs that input UTF-8 files should be prepared to handle BOMs.
UCS-2
- This is the native encoding used by early Unicode-aware systems, such as Java 1.4 and below.
- It can only represent Unicode characters in the basic plane (0x0000-0xFFFF), but no higher.
  In particular supplementary characters in the supplementary planes (0x10000-0x10FFFF) cannot be represented. These are sometimes called astral characters.
- Many APIs that originally only supported UCS-2 were retroactively upgraded to use UTF-16.
UTF-16
- This is the default in-memory encoding for most modern Unicode-aware systems, including Windows, Mac OS X, the Java 1.5+ runtime⁶, the .NET runtime (including C#), and Python 2.x⁷.
- It can represent any character in the Unicode character set.
- Characters in the basic plane (0x0000-0xFFFF) are encoded as a single 16-bit value.
  Supplementary characters (0x10000-0x10FFFF) are encoded as two 16-bit surrogate values.
- ⚠ It is common for code to incorrectly manipulate UTF-16 data as if it were fixed-width UCS-2 instead.
- ⚠ Programs may optionally prepend a byte-order-mark (BOM) at the beginning of a file to mark it as UTF-16 or to specify a byte-ordering other than big-endian. Programs that input UTF-16 files should be prepared to handle BOMs.
- ⚠ Some outdated documentation and APIs may refer to the UTF-16 encoding as the “Unicode encoding”. Notably C#’s UnicodeEncoding, Mac OS X’s NSUnicodeStringEncoding or Python 2.2-3.2’s unicode type on “narrow” builds, which are the default.
UTF-32 / UCS-4
- This is a fixed width encoding capable of representing any Unicode character directly as a 32-bit value.
- Because it is space inefficient, this encoding is rarely seen in practice.
  - Python 2.2-3.2 offers “wide” builds that use UTF-32.
- ⚠ Programs may optionally prepend a byte-order-mark (BOM) at the beginning of a file to mark it as UTF-32 or to specify a byte-ordering other than big-endian. Programs that input UTF-32 files should be prepared to handle BOMs.

Common Mistakes & Practical Tips

You cannot interpret a byte array as a string without knowing its encoding.

Reading Text Files

You cannot read a text file correctly without knowing its encoding.

If you do not specify an encoding explicitly when opening a text file, your language’s standard library or operating system will usually pick a default encoding, which depends on the spoken language it is running in, among other factors.

Unfortunately most filesystems do not store the encoding of a text file.⁸ So there are a few options for determining an encoding:

Define a particular encoding as the expected input.
- For example, a receipt-processing program may explicitly document UTF-8 as the encoding for its input files.
Look at the contents of the input file to auto-detect the encoding:
- You could detect UTF byte-order-marks at the beginning of a file to automatically assume one of the UTF encodings.
- You could define a special syntax at the beginning of the input files to indicate the encoding.
  - XML uses a prelude at the top of a file to indicate what encoding it uses. For example <?xml version="1.0" encoding="windows-1252"?> specifies that the document is in Window-1252 encoding. (Of course to even read this initial text, you have to make the working assumption that the top of the file is some encoding that is a superset of ASCII.)
  - Python source files use a line such as # -*- coding: utf-8 -*- to indicate the encoding is other than the default. (Python 2.0-2.4 uses Windows Latin 1 as the default encoding; Python 2.5-2.7 uses ASCII, Python 3.x uses UTF-8.)
- Failing these options, you could fall back to a defined encoding (such as ASCII for Python source files) or to the operating system’s default encoding (which can vary).
Always use the operating system’s default encoding.
- This is often the behavior if you use your favorite programming language’s default mechanism for reading a file, such as Java’s FileReader class. (Interestingly, C#’s StreamReader and StreamWriter classes always use UTF-8 instead of the operating system default.)

Converting Between Bytes and Strings

You cannot correctly convert a byte array to a string without specifying the encoding to use.

Unfortunately many languages allow you to omit the encoding, and then will try to guess the encoding (usually incorrectly) if you fail to specify it.

Consider the following Java program:

byte[] footBytes = {'f', 'u', (byte)0xC3, (byte)0x9F};
String footString = new String(footBytes);                    // WRONG: OS-dependent

This program will decode different strings on different operating systems! On Mac OS X and Linux where the platform’s default encoding is UTF-8, the correct result (“fuß”) will be obtained since the original bytes were encoded in UTF-8. However on Windows the bogus result “fuÃŸ” will be decoded because the default encoding is Windows-1252.

Here’s the fixed program, which specifies the UTF-8 encoding explicitly:

byte[] footBytes = {'f', 'u', (byte)0xC3, (byte)0x9F};
String footString = new String(footBytes, "UTF-8");           // CORRECT

As another example, consider the Java InputStreamReader and FileReader classes, both of which convert from byte streams to character streams.

byte[] footBytes = {'f', 'u', (byte)0xC3, (byte)0x9F};
InputStream footStream = new ByteArrayInputStream(footBytes);
Reader footReader = new InputStreamReader(footStream);        // WRONG: OS-dependent

Or the even more innocent-looking:

Reader footReader = new FileReader("foot.txt");               // WRONG: OS-dependent

Both of these examples are wrong for the same reason: they don’t specify the encoding.

The former example can be fixed by adding "UTF-8" as the second constructor argument:

byte[] footBytes = {'f', 'u', (byte)0xC3, (byte)0x9F};
InputStream footStream = new ByteArrayInputStream(footBytes);
Reader footReader = new InputStreamReader(footStream, "UTF-8");    // CORRECT

Fixing the second example unfortunately requires using an entirely different class since FileReader has no constructer with an encoding parameter.

Reader footReader = new InputStreamReader(
    new FileInputStream("foot.txt"), "UTF-8");                // CORRECT

Of course the same problems happen when encoding a string to a byte stream:

String footString = "fu\u00DF";
byte[] footBytes = footString.getBytes();                     // WRONG: OS-dependent

And when encoding a character stream to a byte stream:

ByteArrayOutputStream footStream = new ByteArrayOutputStream();
Writer footWriter = new OutputStreamWriter(footStream);       // WRONG: OS-dependent
footWriter.write("fu\u00DF");
byte[] footBytes = footStream.toByteArray();

And when writing to text files:

Writer footWriter = new FileWriter("foot.txt");               // WRONG: OS-dependent
footWriter.write("fu\u00DF");

A “char” (in your favorite language) is probably not a character.

Many programming languages have a “char” datatype that is intended for representing a character. Usually this “char” datatype could do this effectively at the time the language was written but not in the present day, as the notion of a character has been extended over time.

8-bit chars (C/C++)

In C/C++, a “char” holds one byte. When C was first invented, 8-bit fixed-width character encodings were the norm. Therefore a single “char” was able to represent a single character precisely. However with the advent of CJK languages and multi-byte encodings, this no longer worked. Therefore a C string by itself today can only be safely interpreted as a raw byte stream. As mentioned above, you can only process it properly if you know what encoding it is in.

Without any further information, C string is often assumed to be in the operating system’s default encoding, although you cannot be sure. The correct encoding to use depends on where the string was input from.

A program can work with strings in a few ways:

Choose a particular in-memory encoding that all functions should use.
- All foreign strings will be converted to this encoding at the time of input (regardless of source). And upon output, strings will be converted to the appropriate proper output encoding.
- UTF-8 and UTF-16 are both good candidates for such an in-memory encoding since they can both represent the full repertoire of Unicode characters. Therefore you won’t lose any data by converting to/from them.
  - UTF-8 is compact and a superset of ASCII, so you can pass UTF-8 strings to brain-dead functions that are encoding unaware and get correct behavior as long as only ASCII characters are being used.
  - UTF-16 is convenient because functions that are unaware of supplementary characters will still get correct behavior as long as basic-plane Unicode characters are used, which are the most common.
Pass around the text encoding around with the underlying byte array, possibly with a custom string datatype.
- No encoding conversion overhead with reading the input stream.
- Cannot mix text with different encodings.
- Ruby takes this approach with its built-in String type.
Blithely ignore encoding issues altogether and get unexpected results when working with international characters.
- Many programs written in languages where the default string type is not Unicode take this option out of ignorance. In particular this includes many programs in C, PHP, and Python 2.x.

Gotchas:

Beware of methods that take exactly one char or return exactly one char.
- They almost certainly aren’t aware of non-ASCII or Unicode characters.
Beware of any libraries that don’t document what text encoding it assumes.
- They almost certainly aren’t aware of non-ASCII or Unicode characters.

16-bit chars (Java, C#, Objective-C, Python 2.2-3.2, C/C++’s `wchar`)

Java and C#’s char are 16-bits wide. So are C/C++’s wchar and Mac OS X’s unichar. And so are the elements of a Python 2.x string when it is compiled in the default “narrow” mode.

16-bits is sufficient to hold a Unicode character in the basic plane (0x0000-0xFFFF) but not an supplementary character in a supplemental plane (0x10000-0x10FFFF). In the case of these languages, a char represents a single UTF-16 code unit (i.e. either a character in the basic plane or a surrogate) as opposed to an actual character.

Therefore text-aware programs in these languages need to be particularly careful to deal with supplementary characters correctly, since those characters cannot fit into a single char variable.

Here is a typical Java program that is unaware of supplementary characters:

String str = "Hello";
for (int i=0, n=str.length(); i<n; i++) {
    // WRONG: Does not handle characters outside the basic plane (0x0000-0xFFFF)
    char c = str.charAt(i);

    // ... Do something with the character, like filtering out invalid characters.
}

And here is a much-longer but correct version that correctly identifies surrogates and decodes them to supplementary characters correctly:

String str = "Hello";
for (int i=0, n=str.length(); i<n; i++) {
    char c1 = str.charAt(i);

    // CORRECT. Handles all Unicode characters.
    int codepoint;
    if (Character.isHighSurrogate(c1)) {
        if (i+1 < n) {
            char c2 = str.charAt(i+1);
            if (Character.isLowSurrogate(c2)) {
                // Surrogate pair
                codepoint = Character.toCodePoint(c1, c2);

                i++;
            } else {
                // High-surrogate alone
                codepoint = (int) c1;
            }
        } else {
            // High-surrogate alone at end of string
            codepoint = (int) c1;
        }
    } else {
        // Not a surrogate pair
        codepoint = (int) c1;
    }

    // ... Do something with the character, like filtering out invalid characters.
}

Gotchas:

Beware of methods that take exactly one char or return exactly one char.
- They almost certainly aren’t aware of supplementary characters.
You can’t iterate over the characters in a “string” by iterating over the chars.
- Instead you have to use a loop like the above to iterate over the true characters.
  This example stores the true character in the codepoint variable.
You can’t get the i^th character of a “string” by getting the i^th char.
- No general workaround.

32-bit chars and variable-bit chars (Python 3.3+, Haskell)

If you’re fortunate enough to work in an environment with 32-bit or variable-bit chars then your char is in fact a character. Horray!

The only popular environment I know of with real characters is Python 3.3+, or Python 2.2-3.2 when configured to be in “wide” mode (which is not the default).

End of Line Sequences

Improper handling of end-of-line (EOL) sequences is not uncommon.

There are three common ways to end a line:

Linux: \n (line feed alone)
Windows: \r\n (carriage return + line feed)
Mac OS 9: \r (carriage return alone)

It is possible for multiple styles to occur in the same file or string.

It should also be noted that the last line in a file or string might or might not be followed by an EOL sequence. Therefore you can’t assume that every line ends with an EOL sequence.

As the following examples demonstrate, you need to read your language’s documentation carefully if you want to process lines in a consistent fashion.

Java Line Reader Example

Consider the following Java program:

// Prints the specified file to standard output.
public static void main(String[] args) {
    String filePath = args[0];
    BufferedReader lineReader =
        new BufferedReader(new FileReader(filePath, "UTF-8"));
    try {
        String nextLine;
        while ((nextLine = lineReader.readLine()) != null) {
            System.out.println(nextLine);
        }
    } finally {
        lineReader.close();
    }
}

The BufferedReader class can deal with all end-of-line sequences. Therefore this program is resilient against mixed input.

However println (in the PrintWriter class) emits the OS-specific end-of-line sequence, which means that this program will have different output on different operating systems. Not necessarily what you’d expect.

Python 2.x Line Reader Example

Consider the following Python 2.x program:

import codecs

file_path = sys.argv[1]
with codecs.open(file_path, 'rb', 'utf-8') as stream:
    for line_with_terminator in stream:
        line = line_with_terminator.rstrip(u'\r\n')    # remove any trailing '\r' and '\n's
        print line

Notice that in the Python version it is necessary to explicitly remove the \r and \n characters, since Python’s line iteration behavior is to return the entire line plus the end-of-line sequence (if available).

Python’s print statement always uses \n as the end-of-line sequence, regardless of what OS it is running on. It’s nice that this is a consistent behavior, but it might not be what you expect if you are developing on Windows.

Here is perhaps a more typical program that fails to handle the last line correctly if it doesn’t end with an EOL sequence:

import codecs

file_path = sys.argv[1]
with open(file_path, 'rU') as stream:     # the U mode converts all line endings to '\n'
    for line_with_terminator in stream:
        # WRONG: If last line lack an EOL this will chop off its trailing character improperly
        line = line_with_terminator[:-1]  # remove trailing '\n'
        # WRONG: Treating a bytestring as if it were a Unicode string
        print line

Errors like this explain why lots of Unix programs warn about or get confused by files that don’t end with a final EOL.

And here is another typical variation that does not handle end-of-line sequences properly:

import codecs

file_path = sys.argv[1]
with open(file_path, 'rb') as stream:
    for line_with_terminator in stream:
        # WRONG: Assumes EOL is one byte long, which is incorrect on Windows
        line = line_with_terminator[:-1]  # remove trailing '\n'
        # WRONG: Treating a bytestring as if it were a Unicode string
        print line

This kind of program will get extra \r characters on the end of each line when run on Windows. It also fails to handle final lines that lack an EOL.

Summary

Working with text is tricky. Your programming language probably has default handling that isn’t quite what you want (or expect) so always read the documentation carefully. And if your program is intended to be usable in multiple languages, you actually should write tests that check for proper handling of Unicode characters.

Series

This article is part of the Programming for Perfectionists series.

Updates:

2014-01-01:
- Add introduction.
- Spell out how to add encoding parameter to FileReader and InputStreamReader.

See The String Type is Broken for examples of how the standard libraries of various languages mishandle text.↩
To be precise, the definition of character used here is exactly the same as a Unicode codepoint, for those readers who are already familiar with Unicode.↩
Windows documentation often refers to the default character set (or sometimes the ASCII character set) as the “ANSI encoding”. This is misleading since this is not a single concrete encoding and has nothing to do with the ANSI standards body. For example, the Save dialog in Windows Notepad (in Windows 7) and Unicode Versus ANSI String Representations use “ANSI” to refer to the default encoding, whereas Copy a Unicode File to an ANSI File uses “ANSI” to refer to the ASCII encoding.↩
HTML 5 Draft Recommendation — 12 April 2010, 8.1 Character encodings, retrieved 2010-04-12.↩
The full Unicode standard also covers a wide variety of rules related to handling characters, such as sorting, rendering, and other operations. For our purposes though, we are only concerned with the Unicode character set.↩
Compare the documentation for java.lang.String between Java 1.4 and Java 1.5. The 1.5 documentation clearly states UTF-16 as the internal string encoding.↩
PEP 261 describes Python 2.x’s Unicode handling with respect to characters in the supplementary planes. “Narrow” Python builds (the default) use UTF-16 internally; “wide” Python builds use UTF-32 internally. (The distinction between narrow and wide builds disappeared in Python 3.3.)↩
The HFS filesystem used in Mac OS 9 and Mac OS X is one of the few filesystems that stores the encoding of text files as file metadata. However I think almost no modern OS X program is aware of this.↩

RDiscount 2.1.6 released

2013-05-28T00:00:00+00:00

There are several new features in RDiscount 2.1.6. My favorite is fenced code blocks.

Update to the latest version with:

gem update rdiscount

RDiscount now has feature parity with the latest Discount release.

What’s New?

Fenced code blocks
- backtick-delimited - from GitHub Flavored Markdown
- tilde-delimited - from PHP Markdown Extra
New extensions:
- :no_superscript - Disables superscript processing.
- :no_strikethrough - Disables strikethrough processing.
License changed from 4-clause BSD to the more-permissive 3-clause BSD.
Fix -- and --- to be converted to – and — correctly.
Fix handling of tables that have leading and trailing pipe characters.
Fix generated table of contents to be valid HTML.
Handling of special characters in headings is also improved.
Fix recognition of HTML tags that contain - or _.

What’s Next?

Only David Loren Parsons knows.¹ :-)

RDiscount 2.0.7 released
- Release notes for the previous release of RDiscount.
Why I no longer use Drupal
- Why Jekyll (and not Drupal) is awesome for making a personal website or blog.
- Jekyll can use RDiscount for Markdown rendering.

David Loren Parsons is the author and maintainer of the Discount library underlying RDiscount.↩

Implicit Partial Application Considered Harmful

2013-05-12T00:00:00+00:00

Audience

I assume familiarity with the Haskell programming language or some other language that supports calling a function with less arguments than the function's signature requires. This is called "partial application of a function".

Consider the following function-call expression in Haskell:

(alpha beta gamma)

What can be deduced?

alpha is a function (either imported or in a variable).

In most languages (that disallow implicit partial application) you could additionally deduce:

alpha takes exactly 2 arguments
The expression’s return type matches that of alpha.

Neither of those two properties are necessarily true in Haskell.

Case 1: alpha takes exactly 2 arguments

In this case, the expression behaves as you’d expect in any other language. In particular:

alpha takes exactly 2 arguments
The expression’s return type matches that of alpha.

Case 2: alpha takes more than 2 arguments

In this case, the expression is a partial application of alpha, and would be perhaps better read as:

(\ ... -> alpha beta gamma ...)

Or, in a more C-like notation:

function (...) { return alpha(beta, gamma, ...); }

Thus:

alpha takes exactly 2+K arguments, for some unknown K.
The expression returns a function of K arguments.

Case 3: alpha takes less than 2 arguments

Let’s say alpha takes 1 argument. This would be intuitively read as:

((alpha beta) gamma)

After this reformulation you need to recursively examine the new expression.

In this example, we can deduce:

(alpha beta) returns an anonymous function of (at least) 1 argument.
If this anonymous function takes 1 argument, then it is just called with gamma, and the original expression’s return type matches the return type of the anonymous function.
If this anonymous function takes more than one argument (1+K), then it is partially applied to gamma, and yet another anonymous function (that takes K parameters) is returned as the result of the original expression.

Summary

What a mess. If alpha does not in fact take two arguments then I have to exert non-trivial effort to derive the type of the expression - or even what the expression semantics are.

Reducing the Ambiguity

There are two ways I can see to simplify these weird cases:

Make partial application explicit

Then we would see syntax like:

-- alpha takes 2 arguments
-- return type matches that of alpha
(alpha beta gamma)

-- alpha takes 2+K arguments;
-- return type is a partially applied K-argument function
(alpha beta gamma ...)   

-- alpha takes 1 argument and returns a 1-argument function;
-- expression's return type matches that of the 1-argument function
((alpha beta) gamma)

-- alpha takes 1 argument and returns a (1+K)-argument function;
-- expression's return type is a partially applied K-argument function
((alpha beta) gamma ...)

Notice that each conceptually different case now has a unique syntactic representation - it’s no longer just (alpha beta gamma) for all cases.

Use a grouping operator for function application

For example parentheses could be used instead of a space to signify function calls.

Using a grouping operator syntactically prohibits relying on the left-associativity of space for partial function application, since a grouping operator doesn’t have associativity.

If you combined this suggestion with the explicit syntax extension above, you would get syntax like:

-- alpha takes 2 arguments
-- return type matches that of alpha
alpha(beta, gamma)

-- alpha takes 2+K arguments;
-- return type is a partially applied K-argument function
alpha(beta, gamma, ...)   

-- alpha takes 1 argument and returns a 1-argument function;
-- expression's return type matches that of the 1-argument function
alpha(beta)(gamma)

-- alpha takes 1 argument and returns a (1+K)-argument function;
-- expression's return type is a partially applied K-argument function
alpha(beta)(gamma, ...)

Unique Features of Various Programming Languages
- Describes several other programming languages and their unique features.
Visual Guide to Programming Language Properties
- Visualizes how various programming language properties interact.

Appendix

I originally got bitten by this syntactic ambiguity when trying to decipher the meaning of:

(flip (/) 20)

I was not previously familar with flip and so I assumed that it took two arguments, a function (namely (/)) and a non-function (namely 20). And that it probably returned a function, since its surrounding context expected a function.

In fact flip takes only one argument and thus is more clearly written as:

((flip (/)) 20)

And inlined further to be the lower-order:

(\x -> x / 20)

And, if desired, further rewritten to be the more-compact:

(/ 20)

This last form uses partial application to fill in the first argument of /, which is obvious since / is a well-known built-in infix operator that requires an unspecified left argument.

Programming for Perfectionists (P4P): A Book Outline

2013-05-11T00:00:00+00:00

I’ve read a lot of programming books. Many focus on teaching a particular language or tool, or are for teaching beginner developers the ropes. Precious few are designed to bring intermediate developers up to an advanced level. It’s time to change that.

Here is an outline of a book I would like to write. I will probably try writing some of these sections and chapters as future article posts. When that happens, each section below will be replaced with a link to the associated article.

“Programming for Perfectionists” is the working title for the book and may be revised later.

Introduction

The purpose of this book is to provide a wide and shallow overview of important concepts in real-world software development. As such the treatment of various topics will not be comprehensive. Entire books have been written on the topic of each chapter, and appropriate references will be provided where appropriate.

This book will not teach specific tools, languages, or programs in depth. It deals in concepts and patterns. However references to specific tools and languages will be provided liberally in concrete examples.

Many concepts presented here are ones that I feel are missing from the traditional “computer science” curriculum, yet which I feel to be vital to understand when working with real-world software.

Fundamentals

Representations

Many developers do not understand the common ways of representing various kinds of real-world data, such as text, integers, color, and time. This leads to errors such as munging of international characters, integer overflow vulnerabilities, ignorance of color correction, and bugs that only trigger at midnight on leap years.

Numbers
- Integers
  - Fixed-precision 2’s complement
  - Arbitrary Precision Integers
- Decimals
  - Floating Point
  - Fixed Point
Text
- Characters and Codepoints
- Text Encodings
  - “ANSI”
- Unicode, UTF-8, UCS-2, UTF-16
  - Basic Multilingual Plane, Astral Characters
  - BOM
- “Characters” and “Strings” in popular programming languages
- Line-ending sequences
Color
- Color Models, “RGB”, Colorspaces, CIE, Gamma
Pictures
- Resolution
  - Magic: 72 dpi and 96 dpi
  - Resolution Independence
  - Pixel Doubling
- Non-Square Pixels (ex: NTSC)
- Anamorphic Projections
Sound
Time
Video

Collections

Collections are the backbone for representing compound data in any programming language. Specific implementations are covered in great detail in most any traditional CS textbook. However there are a number of more advanced collections which are useful for tough problems.

Basic Collections
- Arrays
  - Bit Vectors
- Lists
  - Array lists
  - Linked lists
  - Linked lists of arbitrary objects
- Sets
  - Hash set
  - Linked hash set
- Sorted Sets
  - Red & black trees
  - Heaps
- Maps
Advanced Collections
- Bags
  - Counted Bags
  - Partitioning Bags
- Multimaps
- Bidi Multimaps
Advanced Data Structures
- Trees
- Graphs (including: directed, multi)

Error Handling

There are many more ways for a program to fail than succeed.

Signaling (Error Codes vs. Exceptions vs. Sentinels)
- Sentinels - null, 0, -1, false
Guarantees after failure
- In original state (Atomic, Transactional)
- In different but valid state
- In different and potentially illegal state
Behaviors upon failure
- Delegate to caller
- Handle internally
  - Aside: Minix is the extreme of this approach.
- Exit program
  - Linux Kernel: panic()
  - Python, PHP: die()
- Display to user
Error Locality & Failing Fast
- Assertions
Designing Error Messages
Error Seriousness
- Expected - Exception
- Unexpected - RuntimeException
- Fatal - Error (esp. OutOfMemoryError)
  - Aside: Certain programs are actually designed to handle OOM errors (ex: SQLite). Most are not.

Bonus:

Checked Exceptions
Exception Wrapping

Concurrency

Early languages like C pretend that threads don’t even exist. And even amidst the multi-core systems today, many programmers would like to pretend that concurrent execution doesn’t exist. A huge number of concurrency-related bugs are found in production code today.

Many developers are familiar with the “shared memory & locks” model used by most mainstream programming languages for working with concurrency. This model unfortunately makes it very easy to introduce bugs, and these bugs are often difficult to detect since they depend on the precise ordering of randomly-ordered events. Deadlocks and livelocks also lurk to trip the unwary. This and other more-robust models will be presented here.

Terminology
- Processes, Threads, and Green Threads
- Concurrency vs. Parallelism
Shared Memory & Locks Model
- Thread Safe vs. Thread Unsafe vs. Thread Hostile
- Deadlocks, Livelocks, and Lock Hierarchies
Apartment Threading
- Thread Affinity
- Run Loops
Actor Model
Immutability
- Read-Copy-Update

Memory Management

Tradeoffs should be described for each of the following:

Manual Memory Management
- Ownership - “Own & Borrow” Model
- Hierarchy, Composites vs. Aggregates
Reference Counting
- Automatic Reference Counting
Garbage Collection
- Generational Collectors
- Mark & Sweep

Bonus:

Special-Purpose Memory Allocators (to reduce calls to system’s malloc)
- Slab Allocaters

Languages

Comparing Programming Languages (The Right Tool for the Right Job)

A programming language is a tool for getting work done. Some tools are better at certain tasks than others. Thus picking your tool wisely for the task at hand will save you effort down the road.

It should also be noted that a programming language does not stand alone; it comes with an entire ecosystem of tools and people along with it. Such ecosystems often cater to a particular domain and make it easier to work with programs in that domain.¹

Native Development (Desktop, Mobile, and Embedded)
- C/C++ - lingua franca; easiest to bind to OS; hand-optimized performance-critical code
- Java - rich cross-platform development (including GUIs); server programming
- Python - rich cross-platform CLI scripting; text & data processing
- C# - rich Windows app programming
- Objective-C - native OS X development; native iOS development
- PowerShell - Windows CLI scripting
Web Development (Server-Side)
- PHP - web application prototyping; most widely deployed server-side scripting language
  - Java & JSP - Java’s clone of PHP
  - C# & ASP.NET - Microsoft’s clone of PHP
- Ruby & Ruby on Rails - bleeding-edge full-stack web framework
- Python & Django - mature full-stack web framework; rapid development; excellent ORM
- Python & web2py - mature full-stack web framework
  - Anecdotal evidence suggests PHP refugees prefer web2py over Django, probably because it uses implicit behavior (i.e. magic) to avoid boilerplate. In contrast Django’s philosophy is “explicit is better than implicit”, which is more in line with general Python philosophy.
- Python & (lots of choices) - lightweight web framework; for maximum control
  - Pyramid
  - Flask
  - web.py
  - CherryPy
Web Development (Client-Side)
- HTML, CSS, JavaScript - assembly languages
- Haml, Sass, CoffeeScript - 2nd-order languages
Special-Purpose Languages
- Fortran - high-performance numerical programming
- Lua - embedded game programming (ex: game mods, AIs)
- Lisp (Racket) - academic programming^AcadProg; dynamic; great for embedded DSLs
- Haskell - academic programming^AcadProg; static; better compile-time safety than any mainstream language

AcadProg: This language tends to be used in academia, especially by programming language researchers. Thus it also gets various cutting-edge programming language features before mainstream languages.

Related Articles:

Programming Paradigms (Ideas that Change the World)

The following paradigms are presented in chronological order by their recognition in the wider programming community.

(Ye Olde) Structured Programming (ALGOL)
- ban goto – GOTO Considered Harmful
- single entry, single exit
Declarative Programming (Prolog, SQL, and other DSLs)
- “Tell me what you want to do and I’ll figure out how to actually do it.”
Object-Oriented Programming (Smalltalk & Java)
- Decomposing the behavior of a large system into smart modules that interact with each other through interfaces.
Functional Programming (Lisp & Haskell)
- Higher-order functions.
- Building complex data structures out of simple primitive collections (especially lists).
- Constrained side effects.

Process

Collaborative Software Development

Open Source
- Stand on the shoulders of giants. Learn new techniques.
- Work on projects you’re passionate about.
- Start your own projects to get free development resources from the community.
Version Control Systems
- Centralized (CVS, SVN, Perforce)
- Distributed (Git, Mercurial)
- Patches - create, view, apply, submit
Automated Build Systems
- configure, make (C)
- Ant (Java)
- Rake (Ruby)
Dependency Management
- Versioning
  - Semantic Versioning
- Package Managers, Environment Isolators

Code Conventions

Philosophy
- Clarity is paramount.
- Consistency is a close second.
Indentation
- 2 columns vs. 4 columns
- Tabs vs. spaces
80 column lines (or max 100)
Brace positioning (when applicable)
Whitespace around operators
Method ordering and grouping
Blank lines and paragraphs

Communicating Software Designs (UML & Other Diagrammatic Notations)

Most of the following diagram notations assume the use of object-oriented programming. Non-OO languages do not have any standard diagram types, to my knowledge.

(Ye Olde) Flow Chart
Class Diagrams
State Charts
Sequence & Communication Diagrams
Use Cases (Fully Dressed)
CRC Cards (designing classes with responsibilities in mind)

Detecting Defects Early (Testing)

Unit Testing, Code Coverage, Continuous Integration
Static Analysis
- Compiler
  - Strong static typing
  - Enabling extra warnings (-Wall, -Weverything, -Werror)
- Lint
Dynamic Analysis
- Valgrind
- Fault Injection
Test Matrixes, Environment Isolators, Virtual Machines
“Large Hammers”
- Fuzzing
- Stress Testing
- Longhaul Testing
- (many more)

Improving Code Quality (Basic Refactoring)

Transforming the structure of existing code to improve it while maintaining its original behavior is a process called refactoring. Regularly applying refactoring to your code will greatly extend its maintainable life.

Rename Method or Variable
Extract Method
Introduce Explaining Variable
Replace Magic Number with Symbolic Constant
Replace Nested Conditional with Guard Clauses

Meta:

Duplication is the root of all evil.
Prefer delegation over inheritance.
Prefer interfaces over abstract classes.

Debugging Techniques (When It Breaks)

No program is perfect. Even if one were, requirement changes will break it soon enough.

Print Statements, Logging
Debuggers
- Breakpoints, Catchpoints
- Conditional Breakpoints & Watching variables
Heap Analyzers

Profiling/Instrumentation (Making It Faster)

Meta:

Premature optimization is frequently a waste of time.
- Programmer time is more valuable that machine time.
- Algorithm choice has a larger impact on performance than almost any manual tweaking.

Maintaining Legacy Code Safely (Advanced Refactoring)

Although the concept of refactoring can be applied to any code, not just legacy code, the deliberate and careful application of refactoring techniques is mostly restricted to legacy code.

This is covered in great detail in the “Refactoring” book by Martin Fowler.

Other

The topics in this section don’t fit in any of the other major topics. Should these extra topics not form a cohesive whole, they will not be included in the book.

Persistence

Prefer the use of a standard metaformat instead of rolling your own format.
You’ll save a ton of time.
Standard Metaformats
- XML
- JSON
- YAML
- Config files
  - UNIX config file
  - Windows INI file
  - Mac Plist file
- SQLite database
Considerations with custom formats:
- Text vs. Binary
- Backward Compatibility, Forward Compatibility, Version Numbers
- Extensibility
- In-place Modifications & Appendability
  - For text formats, this means the preservation of comments and whitespace.
- Partially-loadable
Examples of custom formats:
- Hotline protocol - highly extensible, featureful, binary protocol
- Mac resource fork format - partially-loadable appendable binary file format
- QuickTime video container - highly extensible binary file format
- Matroska video container - highly extensible binary file format
- ZIP file format
Bundles - a means for combining multiple subfiles into a single virtual file

Comments?

Does any of this content sound interesting to you? If so, drop me a line. I may add/remove content and/or modify the order that I write the chapters based on feedback.

Progress

2015-08-08
- Added a section on Basic Collections.
- Retroactively link some articles under “Comparing Programming Languages”.
2013-07-13: Added the Error Handling chapter.
2013-06-01: Added the Text chapter.

For example, the early appearance of the Ruby on Rails web framework in the Ruby language caused a lot of the Ruby ecosystem to cater to web development needs. Subsequently Ruby gained the “move fast and break things” web mentality, along with the necessary tooling to support that mentality: heavy-duty testing, continuous integration, semantic versioning, package management, etc.↩

Avoid feeling like a port on the Mac

2013-05-07T00:00:00+00:00

Mac users get rather annoyed when using programs that feel like they’re just a port from another OS (usually Windows).

Here’s are a few important points for making your Mac app feel like a native app:

The user should never have to refresh local information that an application is viewing. Refresh should happen automatically when needed.
- Exception: Web browsers. (The HTTP protocol provides no means to detect whether a web page requires refresh while it is being viewed.)
- Common violation: Wait for an application to close, without auto-refreshing the application list. [violated by Adobe Flash Updater]
- Common violation: Wait for disk to be inserted, without auto-refreshing the disk list.
When altering settings, change take effect immediately. If these settings are presented in a dialog, the changes can be reverted by pressing Cancel.
- Common violation: Settings dialogs that have an “Apply” button in addition to “OK” and “Cancel”.
When dialogs are used to present information, the buttons in the dialog should have titles that are useful and easy to interpret. The default action should be presented as the right-most button.
- Example: The Log Out action presents a dialog asking the user to confirm whether they want to log out, with buttons “Cancel” and “Log Out”.
- Common violation: Any dialog with “Yes”, “No”, and “Cancel” buttons.
Don’t ask the user any information that can be detected automatically with high confidence.
- Common violation: Asking the user what language to use. [Some installers]
- Common violation: Web pages for downloading software that prompt the user to select 32-bit vs. 64-bit architecture or an OS version.
Aesthetics are important. If you make an ugly app, it probably won’t be used.
Drag & drop should be supported whenever it seems like it should make sense.

And now for some nitpicks related to command key shortcuts:

Common command key shortcuts must be supported and not modified:
- Command-Shift-Z = Redo [violated by all Microsoft programs]
- Command-Left/Right = Move to beginning and end of line [violated by all Microsoft programs]
- Command-F = Find [violated by Microsoft Outlook]
- Command-P = Print [violated by Sublime Text]
All command key shortcuts must include the command key. That is neither Control nor Option (Alt) can be the primary modifier for a shortcut sequence.
Do not make shortcuts with the function keys (F1 - F15). They are hard to remember.
All command key shortcuts must be tied to a menu item.
- Example: Command-Option-Escape (Force Quit), which is analogous to Control-Alt-Delete on Windows, can actually be found in the Apple menu.
- Exception: Command-Shift-3 and Command-Shift-4 are legacy shortcuts that take screenshots.
There is no such thing as the Apple key. You probably meant the Command key.
There is no such thing as the Backspace key. You probably meant the Delete key - which is not to be confused with the Forward Delete key, which most Windows users just call Delete.
Not all users have a right mouse button. In that case, you must instruct the user to Control-click instead of right-click.
- Exemplar: The game Black and White detects whether the primary mouse is a single-button or a multi-button mouse and provides different instructions (and mouse graphics) for each case.

For all other issues, including other philosphical concerns to consider, read and follow the OS X Human Interface Guidelines (HIG). This is one of the best descriptions of the “Zen of the Mac”. Every Mac developer should reread this every few years.

Copyrights and Licenses for Software

2013-04-30T00:00:00+00:00

This is a quick overview of copyrights and licenses for software developers who may not know their rights under United States law. I cannot speak for international copyright law.

Disclaimer

This article presents legal information but does not provide legal advise. Although all information is correct to the best of my knowledge, I will not be held responsible for the consequences of actions you take based on information found to be incorrect or misleading.

Copyrights

The first time that a piece of software is published, whether it be in binary or source form, it automatically gains copyright status. This is true even if you don’t include the magic words Copyright (c) 2013 David Foster.¹

Copyright grants the author certain exclusive rights.

That is, other people don’t get these rights. Such rights include (but are not limited to):

The right to distribute the software, even for free.
The right to sell the software.

But then you might ask how download sites like Download.com can operate: If by default only the original author has the right to distribute the software they create, a download site would need to get special permission (i.e. a license) to distribute most programs on their site, which would be time consuming (and therefore expensive).

In reality, download sites generally rely on software authors to not enforce their copyrights. Which brings us to another point:

Copyrights are useless unless they are enforced.

Enforcement takes the form of the copyright holder actively looking for infringers: other people who are distributing or selling the software without permission.

For example, the MPAA may issue a DMCA takedown request to a YouTube user who has (without permission) posted an episode of a TV show that MPAA owns. If the user does not comply with the takedown, the MPAA may sue the user for infringement.

Licenses

A copyright holder may waive or share some rights by issuing a license.

Licenses may be issued to individuals or companies. For example a particular distributor may be granted the right to market and sell the software on behalf of the original developer.²

For open source software, licenses are typically granted to the community as a whole (i.e. every recipient of the program). This is typically done by including a file called LICENSE or COPYING with the software. A license may also be found in the software’s README.

You can write your own custom license (if you feel comfortable writing in legalese). Or you can use one of the more common existing licenses from the community. (See Common Licenses below for a list.)

Dual Licensing

Sometimes the same piece of software is offered under multiple licenses.

For example the Qt GUI toolkit has been available under both the GPL license and (with payment) a commercial closed-source license for a long time. Since companies don’t like basing their own closed source software on GPL-licensed original software, this gives them an option to use a more corporate-friendly commercial license.

When creating a new piece of software that is based on or uses dual-licensed software, you should document which license the original software is being used under.

If you are the original author of a piece of software, you can issue a new version of the software under additional licenses.

Changing a Software’s License

Under rare circumstances, the license provided with software can change.

As the copyright holder of a piece of software, you may decide to release a new version with a different set of licenses than prior versions. For example you could change the license offered from BSD to GPL, a rather radical change. Such a change would only be in effect for the newly released version - it would have no effect on older released versions.

If you are not the copyright holder of a piece of software, you cannot change the license(s) under which it is offered.

If there are contributors beyond the original author whose changes have been incorporated into the software, they become partial copyright holders in the software as well unless they explicitly waive their copyright interests via a Contributor Agreeement or similar document. In the event of multiple copyright holders, all holders must agree to any proposed changes in the set of licenses under which the software is offered. Since there may be a large number of contributors (and thus copyright holders), it may be impossible to get permission from everyone, thereby making a license change infeasible.

Projects that are particularly aware of copyright concerns may require all contributors to sign a Contributor Agreement that either waives all copyright interests or explicitly assigns the copyright associated with all contributions to the original author.³

No Warranty

All software licenses I have seen explicitly disclaim all warranties. Meaning that if the software harms someone, you can’t sue the developer for it. This is great for software developers but in my opinion bad for society.

Common Licenses

Here are the most common open source licenses, arranged generally from most permissive to most restrictive.

If you don’t care about the details and only want to know when to use which license, skip to the summary.

Public Domain⁴ — ultimately permissive

Do whatever you want with this software.
This software cannot ever be placed under copyright or any license.

MIT License — super permissive, simple

Do whatever you want with this software.

BSD License — permissive, simple

You can redistribute this software.
(Other rights are not expressedly waived.)
There are several variants of this license.

Apache 2.0 License — permissive, complex

You can do anything normally restricted by copyright.
You can do anything normally restricted by patent rights.
Redistribution must be done under certain terms.
Submissions by contributors automatically fall under this license unless specified otherwise.
You may NOT use trademarks related to this software (except in a few special cases).

LGPL License 2.1 — restrictive, complex

You can redistribute this software.
You can modify this software and distribute the derived work. However the derived work must also be LGPL-licensed.
You can relicense a copy of this software under the GPL. (This is an irreversible action for works derived from the new copy.)
If you create your own software that uses this LGPL-software, the license of your source code is unaffected. However the resulting executable and (sometimes) the object code must satisfy the following terms:
- Modification of the executable and object code is permitted for personal use.
- Reverse engineering of the executable and object code is permitted.
- You must give notice that the executable and object code uses this software and that this software is licensed under the LGPL. You must also provide a copy of the LGPL license text.
- If the software displays copyright notices during its operation (such as in an About box), the LGPL copyright notice for this software must be included and a reference to the full LGPL license text must be provided.
- Either:
  - (1) Include the source code for this software (including any modifications you made to it) when distributing the executable and object code.
  - (2) Require that the user of the executable or object code have this software installed locally already, and use shared library mechanisms to link the executable against the (unmodified) local copy of this software.
- If distributing an executable, the nondistributed form of your software must include any data or utility programs necessary to compile the executable.⁵

GPL License 2.0 — restrictive, complex, “viral”

You can redistribute the source code of this software.
You can modify this software and distribute the derived work. However the derived work must also be GPL-licensed.⁶
You can distribute this software or any software you create based on it, in executable or object code form, provided that you also distribute the corresponding source code (or provide a written offer to do so).

No License — ultimately restrictive

You cannot redistribute or sell this software.
You cannot create new software that is based on this software (i.e. derivative works).

Choosing a License

It is best to pick the license whose terms convey the message you want to send:

Public Domain
- Do anything you want with this software. It is free forever.
MIT
- Do anything you want with this software. (Although I might change my mind later.)
BSD (4-clause)
- Do anything you want with this software, although I want to be attributed (i.e. mentioned as the original author). But I don’t want my name used to promote other things.
Apache
- You can do most things with this software.
  (And my lawyers want a precise definition of what “most things” means.)
LGPL
- I want the source code for this software library (and any derived software) to be freely available forever. I also want to enforce that all improvements to this software library that are incorporated into other products be available for independent inspection.
- I don’t care if these extra terms made it burdensome for commercial entities to use my library, although I would like to make it possible for them to still use it if desired.
GPL
- I want the source code for this software library and any external software that uses it to be freely available forever.
- I don’t care if this means that my software will never be incorporated into a commercial product.
Copyright Notice; No License
- Don’t even think about incorporating this software into another product, or selling it.
No Copyright Notice; No License
- I don’t know about copyright or licenses. Ask me if you want to do something with my software.

Now that I’ve figured all of this out, it’s time to update the licenses I’m using for my own open source projects.

However if you do include the magic words Copyright (c) 2013 David Foster it is easier to demonstrate willful infringement when you sue someone for infringing your copyright. Willful infringment entitles the copyright owner to higher damages (i.e. payments from the infringer) than vanilla infringement.↩
If you sell any apps on the App Store, you signed an agreement giving Apple such marketing rights.↩
For example my present company Splunk requires that all contributors to its open source projects sign a Contributor Agreement of this type.↩
Technically the state of being in the “public domain” is not a license. Any work explicitly released into the public domain is not subject to copyright law at all. In particular there is no copyright holder and can never be a copyright holder in the future.↩
At face value this doesn’t actually seem like a restriction. However the LGPL text cautions, “It may happen that this requirement contradicts the license restrictions of other proprietary libraries that do not normally accompany the operating system. Such a contradiction means you cannot use both them and the Library together in an executable that you distribute.”↩
The GPL is ambiguous with respect to whether a piece of software that you create based on the GPL-licensed software (including merely linking to it) should be considered a derivative work or not. If so, then the GPL is truly “viral”: any piece of code that interfaces with GPL-licensed software must also be licensed under the GPL. If not, the GPL still specifies unambiguously that programs based on GPL-licensed software must be distributed with all accompanying source code. Neither proposition is tolerable for most corporate software entities.↩

Making an Existing Python Program Unicode Aware

2013-04-09T00:00:00+00:00

Last week I decided to update one of my larger Python 2.7 projects to support Unicode fully and to run under Python 3.

Here are the steps that I took and some gotchas I ran into along the way.

General Strategy

Create and fully automate the unit test suite:

This is important to detect any unexpected breakage from the Unicode-related changes you’ll be making soon.
Make sure your code coverage is good since tests can’t protect you from mistakes introduced in uncovered code.

Identify all boundaries where data is being exchanged between the program and the outside environment. In my case:

Direct I/O (i.e. read and write)
Filesystem access (i.e. open and file paths)
Process manipulation (via the subprocess module)
Command-line arguments (via sys.argv)
Time access (via the time module)

At boundaries:

Reads should decode bytestrings to unicode using the proper encoding. You may need to do research to determine what the correct encoding is or how to determine the correct encoding at runtime.
- Since my program did a lot of direct I/O on classic Mac OS data structures, the correct encoding was typically MacRoman.
- sys.getfilesystemencoding() is sometimes appropriate. Be sure to test on Windows, where this value is typically not UTF-8.
Writes should encode unicode (or ASCII bytestring literals) to bytestrings using the proper encoding.
Ensure files and streams explicitly use either binary mode (i.e. rb or wb) or text mode (i.e. rt or wt).
- UsuBytesIOally you want binary mode.
- Text mode is required for certain cases, such as the output of the json module. Even then, it’s a good idea to restrict text-based output to the ASCII character set since the native text encoding may not support the full Unicode set, particularly on Windows.

Finally:

Run the 2to3 converter on your program so that it can be run by Python 3.
Run your new Python 3 source code in a Python 3 interpreter, perhaps inside a fresh Linux VM.
Run your test suite under the Python 3 interpreter. It should find any Unicode mistakes that still need to be fixed.
You will probably need to repeat this convert-test-fix cycle a couple of times.

Special Cases

Most uses of StringIO need to be migrated to BytesIO, when used as a byte buffer. If used as a string buffer, the StringIO uses need to be left at StringIO.
- Here’s the shim I used to support BytesIO and StringIO in both Python 2 and 3. I put it in a utility module and the rest of the program imports the shims instead of using the standard library.

# BytesIO presents a stream interface to an in-memory bytestring.
# 
# This is equivalent to StringIO in Python 2 and to BytesIO in Python 3.
try:
    from io import BytesIO              # Python 3
except ImportError:
    from StringIO import StringIO as BytesIO

# StringIO presents a stream interface to an in-memory string
# (which is a bytestring in Python 2 and a unicode string in Python 3).
# 
# This is equivalent to StringIO in both Python 2 and 3.
try:
    from StringIO import StringIO
except ImportError:
    from io import StringIO             # Python 3

When writing to a binary stream, replace uses of output.write(chr(byte_ordinal)) with output.write(bchr(byte_ordinal)) and use the following shim for bchr:

# bchr() converts the specified byte integer value to a single character
# bytestring.
# 
# This is equivalent to chr() in Python 2 but requires special handling in
# Python 3.
if bytes == str:
    def bchr(byte_ordinal):
        return chr(byte_ordinal)
else:
    def bchr(byte_ordinal):
        return bytes([byte_ordinal])    # Python 3

Beware of comparisons with ''. Some of them probably need to be converted to comparisons with b''. For example, I had to adjust my at_eof utility function:

def at_eof(input):
    """
    Returns whether the specified input stream is at EOF.
    """
    with save_stream_position(input):
        at_eof = input.read(1) == b''
    return at_eof

If your program uses human-readable ASCII bytestring literals, such as FourCC codes, make sure they are marked with the b prefix appropriately.

if type_code == b'APPL':

Iteration over bytestrings works differently in Python 2 vs Python 3.
- The code for c in data: will give back single-character bytestrings in Python 2 but byte ordinals in Python 3.
- Here’s my shim to let for b in iterord(data): always iterate over byte ordinals:

# iterord() iterates over the integer values of the bytes in the specified
# bytestring.
if bytes == str:
    def iterord(bytes_value):           # Python 2
        for b in bytes_value:
            yield ord(b)
else:
    def iterord(bytes_value):           # Python 3
        return bytes_value

time.strftime returns a bytestring in Python 2 and a Unicode string in Python 3. Usually you want a Unicode string. I didn’t create a shim for this since I only had one use of strftime.

now_string = time.strftime('%b %d %H:%M', time.localtime(now))
if not isinstance(now_string, unicode):
    now_string = now_string.decode('ascii')

Regular expressions that are intended to apply to bytestrings instead of Unicode strings may need updating.
- For example the pattern re.compile(r'^Volume name is "(.*)"$') which is designed to be applied to a MacRoman-encoded bytestring would need to be updated to read re.compile(br'^Volume name is "(.*)"$')
Don’t forget to update your documentation to specify where Unicode (or ASCII bytestring literals) are expected.

Beautiful Code: SQLite

2013-04-07T00:00:00+00:00

It has been said that you can learn a lot from reading other people’s high-quality software. So I gave it a try this weekend by reading the SQLite source.

SQLite is a lightweight embedded database that does not require a standalone server. It is designed to be reliable, highly portable, and require minimal administration.

SQLite has some of the most rigorous testing methodology I have seen in an open source project. Thus its true defect count is likely to be extremely low, making it high quality.

Such a high quality product probably contains useful patterns and techniques, some of which I could reuse in my own programs. So I decided to dive into the SQLite source and see what gems I could pull out. Here are my notes:

Architecture & Design

I started by reading the high-level design documentation. SQLite has exceptionally good documentation of this type.

Documentation Index - Root of all documentation.
Architecture Documentation - Excellent overview of the architecture.
Query Planner Documentation - Describes how prepared statements are executed.
- The Virtual Database Engine
- Virtual Machine Opcodes

Internal Tools of Note

Fossil

Distributed version control system used by SQLite. I’ve never heard of this. Beyond version control also provides bug tracking, a wiki, and a blog.

Lemon

A custom parser generator used by SQLite. Alternative to the old bison/yacc combination which has some improvements:

Syntax that is less error-prone.
Fast. Reentrant. Thread-safe.
Easier to write parsers that avoid leaking memory upon error conditions.
- Important since SQLite is serious about handling out of memory conditions.

VFS

OS abstraction layer. Despite the acronym (Virtual Filesystem), it is more than just filesystem manipulation routines.

Implementation Notes

table.c (sqlite3_get_table)

This is the main entry point into the SQLite API. It executes a SQL statement and returns the result.
First impression is that most methods have a lot of malloc-failed handling. And indeed if you read the testing procedures, they involve causing random malloc errors throughout the code, so there has to be handling for such errors.
Hungarion notation is used as well. Ick. But at least it is consistent.
- z = string
- a = array
- p = object pointer
- n = int
- x = function pointer
/* Assume 32-bit assignment is atomic */
- Interesting that such a property would be consciously relied on.

legacy.c (sqlite3_exec)

This is the legacy entry point into the SQLite API. It still exists and is actually used internally by the currently recommended API entry point (sqlite3_get_table).
There are some odd syntactic conventions.
- Multi-line comments use double-stars to prefix middle lines.
- No space before closing {.
- Spaces inside outer parentheses for most statements (if, while, assert) but not all (for).
It appears that functions that begin with sqlite3_ are public, whereas functions starting with sqlite3 (no underscore) are private.

vdbeapi.c (sqlite3_step)

This function executes one or more instructions from the instruction list inside of a prepared SQL statement. The instruction is executed inside of the SQLite virtual database engine (VDBE), which is a virtual machine.
- It has a lot of support code for EXPLAIN and profiling callbacks.
- The meat of instruction execution is done by sqlite3VdbeExec.
Contains own mutex implementation. (sqlite3_mutex_enter)
Deals with a somewhat crazy circumstance: If the database schema changes in the middle of executing a (prepared) SQL statement, the statement is reprepared and rerun automatically.
Generally there is good commenting for weird and backward-compatibility behaviors.
Practically every function can fail. They all return an integer error code.
- Some functions that operate on an in-out data structure will additionally store an error code (and message) in that structure.

vdbe.c (sqlite3VdbeExec)

This function executes as many instructions as possible from a prepared SQL statement. It contains the monsterous switch statement that enumerates every possible opcode that can be executed.
- Related reference documentation: Virtual Machine Opcodes
- Related guide documentation: The Virtual Database Engine

Future

If I were to continue reading into the implementation, I think the opcodes that manipulate B-trees would likely be the most interesting ones to look at.

Summary

This is a very high-quality C library.

Internal commenting is good.
Consistent syntax is used, despite being a bit odd IMHO.
The guide-level documentation is wonderful.
Error handling is air-tight and enforced by crazy amounts of test code.
The architecture provides good separation of roles and is easy to understand.
Once again, the documentation is excellent.

And its design requirements show through in the implementation:

Reliable

There is more than 1000 times as much test code as there is product code.
100% branch coverage. That’s insanely good.
Very crazy error classes such as out of memory errors, crashes, integer overflow are all considered and tested for.
- Malloc checks are prevalent and handled.
- Transactions, journaling, and related testing deal with crashes.

Portable

The chosen language is C, which is highly portable when written in the appropriate style.
The so-called VFS layer provides an OS abstraction. Thus porting to an OS mainly just involves writing a new implementation of this abstraction.
All C files are combined into one giant C file before being sent to the compiler, which provides for better compile-time optimizations for dumber C compilers, which may occur on embedded systems.
The database format does not depend on endianness or native data type sizes, making it suitable for cross-platform use.
Very low memory configurations are available, making SQLite suitable for use on embedded devices such as cellphones.
The code is in the public domain, which removes any licensing barriers.

Simple

SQLite is embedded into the program that uses it, requiring no separate server. And no administration for such a server.
Minimal schema is imposed by SQLite, distinguishing only between integers, reals, text, and blobs at a low level. And even so, a column’s type is advisory only - you can store a value of any type in any column (except an INTEGER PRIMARY KEY column).

Of course being simple sacrifices certain other properties such as high concurrency and the ability to perform fine-grained access control (which requires administrability).

Migrating from BBEdit to Sublime Text

2013-03-22T00:00:00+00:00

I’ve been using BBEdit as my primary text editor on the Mac for over 10 years. It is an extremely capable and mature editor with just about all the bells and whistles you can imagine. However the use of Sublime Text has been spreading throughout my workplace at Splunk¹ so I decided I’d give it a try.

My initial impression was “BBEdit can do everything that Sublime can, what’s the deal?” But after using Sublime full-time for about a week, I’m starting to come around to it.

Although it is true that both have comparable feature sets, Sublime’s implementation feels more polished - not just eyecandy, but actually better usability.

Incremental find
- As you start typing the substring you want to find, it automatically starts searching within the current document. Frequently you don’t have to finish typing the substring in order to get the result you want.
- This saves a surprising amount of time compared to BBEdit where you have to finish providing a full substring (that you think is long enough) before kicking off the search.
- Edit: This is actually available in BBEdit under the “Live Search” command, but this is not part of the default Find experience and not advertised.
View a file without “opening” it (as a persistent tab)
- Sublime keeps files of current interest available as tabs in the main window. If you just click on a file, it is shown in the editor but no tab is created by default. If you decide this is an important file, you can double-click on the file name to create an actual tab for it, which makes the file easy to return to.
  - Most “Goto X” actions in Sublime both show a file and create an actual tab for it, which usually is what you want.
- If you click on any file at all in BBEdit, it is “opened” and added to the “Currently Open Documents” section, which is analogous to Sublime’s set of open tabs. This can result in the “Currently Open Documents” section getting rather cluttered when rummaging around for a particular document, making it less useful.
Minidocument view on the right side of an open file.
- This makes it easier to get a bird’s eye view of where you are in a long function or file.

Additionally, Sublime’s marketing has some a much better job of exposing advanced features that turn out to be useful in daily use.

“Goto Anything”
- BBEdit has similar “Open File by Name”, but it’s slower, doesn’t have as expressive syntax, and isn’t incremental.
“Open Folder”
- This is equivalent to a BBEdit project.
Plugins!
- This is barely advertised on the BBEdit side but hugely promoted on the Sublime side. Thus lots of people are writing (publicly available) plugins for Sublime but not so many for BBEdit.

Some features in Sublime are named more obviously:

Convert Indentation to Spaces. Convert Indentation to Tabs.
- BBEdit calls these operations “Detab” and “Entab” respectively.

Some features in Sublime are easier to access:

Change the displayed tab width.
- To change this for a single file in BBEdit, you have to go to “Show Fonts”, which I find completely nonintuitive.
- BBEdit also allows you to change the default tab spacing per source language in its preferences, which makes sense given that different languages have different conventions.
- However BBEdit does not permit you to change the tab width on a per project/folder basis, which is usually what you want when working on someone else’s open source project.
Access the function list. (“Goto Symbol…”)
- This can be done in Sublime entirely through the keyboard.

Sublime also has a few features lacking in BBEdit:

Cross-platform support
- This is huge when working in a mixed OS environment. You can use the same editor everywhere.

BBEdit still is better in a few areas:

Multi-file search and replace can be done without confirming (or saving) each modified file.
Robust on slow filesystems, particularly network filesystems.
- Sublime (2.0.1) acts very poorly: times out on directory listings, silently fails searches in “Goto Anything”, and probably has other issues.
Can edit preferences in a GUI without mucking about in text files.
Can quickly split a single file vertically to edit distant sections simultaneously.
“Zap Gremlins…”
- Deletes or replaces ASCII control characters and other nasties. Very useful for text pasted from web browsers, Word, or other less-than-pure sources of text.
“Process Duplicate Lines…”
- Eliminate duplicate lines in a file. This is often useful after performing a series of text transformations.
“Process Lines Containing…”
- Delete (or preserve) all lines containing a subexpression. Exactly like the grep command-line tool.
Can edit files on an FTP/SFTP server directly.
Customer support I’ve heard is top-notch.
It can print.

I think I’m going to continue using Sublime Text. Featurewise it’s about the same as BBEdit, but the cross-platform support², active plugin community, and general polish are winning me over.

May 2013 Update: I have now switched to Sublime for my daily programming tasks. Its superior “Goto Anything” functionality is a killer feature for navigating around large projects. And its cross-platform nature is killer when operating in a mixed OS environment. Occasionally I still need to pull out BBEdit to do a large multi-file search & replace, but otherwise I don’t use it that frequently.

Splunk makes tools for analyzing large time-series datasets, such as log files. For details see the 3 minute Splunk Product Overview. I work on the Developer Platform, making it easy for other developers to build neat things on top of the Splunk core.↩
It is extremely useful to be able to use the same full-featured text editor on any platform I use. I currently maintain a few open source projects that require testing, debugging, and development on multiple platforms (notably the RDiscount Markdown processor) and it’s nice being able to use the same editor everywhere.↩

Introducing p4status: A status command for Perforce

2013-03-16T00:00:00+00:00

p4status works like git status (or svn status), but works in Perforce.

It will list changelist files and locally modified files that haven’t been added to any changelist (including unadded files) in a readable format.

Note

The 2012 version of Perforce has an official p4 status command. However this command is only available if your corporate Perforce server has been upgraded to 2012. If your company, like mine, has not upgraded their server, this script will make your life easier.

Example

[dfoster ~/dev/p4/mybranch/path/to/myproject]$ p4status
# On changelist 154055
# 
# Changes to be committed in changelist default:
#   (use "p4 revert <file>..." to discard changes in working directory)
# 
#       deleted:    server/apps/setupfx/splunkd/default/extract.conf
# 
# Changes to be committed in changelist 154076:
#   (use "p4 revert <file>..." to discard changes in working directory)
# 
#       new file:   server/apps/setupfx/splunkd/default/inputs-NEW.conf
#       modified:   server/apps/setupfx/splunkd/default/inputs.conf
# 
# Changes not staged for commit:
#   (use "p4 add/edit/delete <file>..." to update what will be committed)
#   (use "p4 sync -f <file>..." to discard changes in working directory)
# 
#       new file:   server/apps/setupfx/splunkd/default/inputs-NEW2.conf
#

Installation

Download to your local bin directory: https://github.com/davidfstr/dotfiles/blob/master/bin/p4status
Mark as executable with chmod a+x p4status

Usage

cd to the primary directory that you’ll be working under.
Run p4status.

Limitations

Only detects unstaged files under the directory in which the command is run. Therefore it is best to reserve a terminal window that is cd-ed to the correct place where this command can be run safely.
This command is too slow to run from the root of very large branches.

Alternatives

Use the excellent Git-to-Perforce bridge, git-p4.
- This allows you to work with your Perforce repository as if it were a Git repository. And you can use Git’s very fast git status command.
Convince your Perforce server administrator to upgrade to the 2012 version. This may take some time, depending on the size of your company.

Related Projects

dotfiles
- My command-line shortcuts, including several for working with Git, particularly in an OS X environment.

Killer apps for software platforms

2013-03-10T00:00:00+00:00

A killer app is any product that is so desirable that it fuels the sale/adoption of a larger technology that the product depends on.

The gaming industry has been aware of the power of killer apps for a while, and often seeks to create launch titles for their new gaming systems that are killer apps. Microsoft, for example, deliberately created the first Halo game at same time as their Xbox console. And Nintendo has repeatedly introduced new Mario titles with its new consoles.

However what may not be so obvious is that killer apps exist for learning new software platforms as well, such as for APIs, SDKs, and programming languages.

Learning a new platform (along with the associated tools and ecosystem) takes a large investment of time, and so there is a similar barrier to adoption as, say, the monetary cost of buying a new game console.

I think the platform-builders in the software industry could learn from the gaming industry and strategically introduce killer apps alongside new platform offerings. I can think of no prior case where this was done deliberately (see the Appendix below), so I think our industry (i.e. software) is really missing out.

Appendix

Examples for Programming Languages

Killer apps for languages:

Ruby on Rails -> Ruby
Django -> Python
Emacs -> Lisp

Killer ecosystems for languages:

iPhone -> Objective-C
Android -> Java
embedded scripting in games -> Lua¹

Killer features for languages:

interface with OS natively -> C, Objective-C (OS X)
garbage collection -> Java
hack together a deployable dynamic website fast -> PHP
general-purpose scripting -> Python
interprocess scripting on OS X -> AppleScript

Other Software Examples

GitHub -> Git
Outlook -> Windows²

Programming is Fun. But What Matters is Solving a Real Problem.
- More thoughts on business considerations as applied to software.

More information about the adoption of Lua in the gaming industry on StackOverflow: Why is Lua considered a game language?↩
Although Microsoft has finally created a version of Outlook for the Mac, it is buggy and missing a number of advanced features of the Windows version. Thus two of my colleagues who do a lot of planning run Outlook specifically on Windows despite the workplace being otherwise Mac-oriented.↩

Notes on Racket (PLT-Scheme)

2013-03-01T00:00:00+00:00

Generally Racket appears to be a very usable dialect of Lisp/Scheme.

Racket is particularly well suited for those who want to develop their own programming language (including those not based on S-expressions), given that the creators of Racket are programming language researchers. For example, a number of Racket subsets have been created to assist students who are learning Racket/Scheme via the How to Design Programs book.¹ The Arc language by Paul Graham is also implemented in the Racket environment.

Racket’s developers wanted to make sure it was a practical language. To that end, there is a bundled IDE (DrRacket) with syntax highlighting, parentheses matching, and integrated debugging. There is also a package management system (PLaneT).² And built-in libraries for working with several real-world systems, such as GUIs, networking, databases, JSON/XML, etc. A cross-platform executable can easily be built. And everything is well documented.

Graphics and GUI programs are supported better than the average language. For example pictures are rendered natively in the Racket REPL, including their use within expressions. And the built-in Racket GUI library is fairly decent.

The combination of a decent built-in GUI library along with the ability to compile cross-platform executables makes Racket useful for writing cross-platform GUI applications.

Racket’s GUI library (RacketGUI)

The original version of Racket’s GUI library was implemented on top of wxWidgets.³ Thus some of wxWidget’s poor design decisions leak through:

A visual component is called a “window”, not a control.
Visual components must be associated with their parent container at creation time.

However, some fixes were made too:

There is no notion of a wxWidgets “sizer” as distinct from a “container window”.
- This simplifies the mental model considerably, since it is no longer necessary to keep track of both a “window” hierarchy and a sizer hierarchy.
- Instead, a wxWidgets “sizer” is represented as a lightweight RacketGUI “pane”, which can be nested naturally inside containers along with other components.
The event handling loop is managed in the background automatically.

Sadly, RacketGUI lacks some advanced controls I often want:

trees
tables containing controls in cells
- However the list-box% control supports cells that only contain text.
tree-tables

This is not surprising since these controls are hard to implement and are not well-supported by the original underlying wxWidgets library:

The wxWidgets tree interface (wxTreeCtrl) is annoying for dynamically generated trees.
Tables (wxListCtrl) only support text and image cells, not controls as cells.

Unique Features of Various Programming Languages
- Describes several other programming languages and their unique features.
Notes on Prolog
- Describes Prolog, a highly declarative language. Useful for verification of proofs.

The HtDP book uses the dialects “Beginning Student”, “Intermediate Student”, and “Advanced Student”. These sublanguages restrict the use of certain language features (such as using functions as objects) and can provide better error messages (which avoid mentioning advanced features not supported by the sublanguage).↩
It is possible to import modules directly from the PLaneT repository, which will automatically download and install the module if it isn’t already present. Neat!↩
Racket’s GUI layer has apparently been redone a couple of times. Originally it was based on wxWidgets. Later it was rewritten to about 200,000 lines of C++ glue to Xt, Win32, and Carbon. Then in Racket 5.1 it was replaced with about 80,000 lines of Racket glue to Gtk, Win32, Cocoa, Cairo, and Pango. That’s a 60% reduction in code! More details on the Racket blog.↩

Notes on Prolog

2013-02-25T00:00:00+00:00

What is Prolog?

Prolog is a declarative programming language that, given a list of axioms and a list of implication rules, can deduce the truth (or falsehood) of a logical statement.

For example, given the following knowledge base (AKA database):

% Axiom: Mary likes Ted.
likes(mary,ted).
% Axiom: Jane likes Ted.
likes(jane,ted).
% Rule: X is jealous of Y if they both like Z (and are not the same person).
jealous(X,Y) :- likes(X,Z), likes(Y,Z), X \= Y.

I can ask the question jealous(mary,jane) and get the affirmation true. Or I can ask the question jealous(mary,mary) and get the denial false.¹

I can also ask a more complicated question containing variables and Prolog will derive the possible values (and relations) the variables must have in order to make the question posed to be true.

For example, asking the question jealous(X,Y) will give the results:

X = mary,
Y = jane ;
X = jane,
Y = mary

As another (more interesting) example, consider the knowledge base:

% Rule: A line is horizontal if its points have the same Y coordinate.
horizontal(line( point(_,Y), point(_,Y) )).
% Rule: A line is vertical if its points have the same X coordinate.
vertical(line( point(X,_), point(X,_) )).

And the questions:

(1) horizontal(line( point(0,0), point(X,Y) )).  ==>  Y = 0.
(2) horizontal(line( point(0,0), P2 )).          ==>  P2 = point(_G327, 0).

The second question is particularly interesting because it gave back a fairly complicated answer: P2 must be a point, its X coordinate can be anything, but its Y coordinate must be zero. I think it’s particularly cool that Prolog can deduce that P2 must be a point.

As a final example of a simple problem solvable by Prolog, consider the crossword puzzle in Exercise 2.4 of “Learn Prolog Now!”. You can write a Prolog program to solve crossword puzzles!

When is Prolog useful?

Prolog’s deduction capabilities makes it useful for answering questions and making deductions within systems whose concepts can be expressed in a formal hierarchy.

In academia, for example, Prolog has been used to investigate natural language formalizations and expert systems in artificial intelligence.

Case Study: Java bytecode verifier

As a more practical example, Prolog has also been used to define the semantics of the Java bytecode verifier² for Java 6.³ Prior versions of Java had a verifier whose semantics were defined by a textual specification only (namely the Java Virtual Machine Specification).

There are a few advantages to having the verifier defined in terms of Prolog:

The specification becomes formal (and thereby unambiguous and more likely to be error-free).
The execution model for performing the verification becomes well-defined (since it would use Prolog’s) and fast (since Prolog’s core unification algorithm is fast).

Writing Programs

Functions as Rules

So how do you actually write something resembling a program in Prolog? Until now we’ve been talking about axioms, rules, and questions. But in most languages a program consists of a series of expressions that are evaluated.

For example in Python, you might write:

def add(x, y):
    return x + y

print add(5, 3)    # prints 8

In Prolog, you can do something similar by defining a rule where one of the variables in the rule (typically the last one) is its “output”:

add(X, Y, Result) :- Result is X + Y.

To actually evaluate this “function” with 5 and 3 you would ask the question add(5, 3, Result) which would yield Result = 8.

Now, don’t get trapped into the idea that such as rule always needs to be evaluated in one direction. In Prolog you have the additional power to ask what the “inputs” of the function has to be to yield an already-known “output”.

For example, you could ask the question add(X, 3, 8) to deduce that “input” X must be 5.⁴ You could even try asking add(X, Y, 8) to find all values of X and Y that yield 8 as a result (although there are an infinite number of such combinations in this example).⁵

This power to flip functions on their head is a unique quality of Prolog. (And it blows my mind.)

Ultimately a rule specifies a set of relationships between its arguments. Thus arguments can be “inputs”, “outputs”, or even temporary variables (like accumulators).

Processing Recursive Data Structures

Processing recursive data structures such as lists and trees is mind bending… For examples see:

Interactive Use (as a REPL)

It is possible to run Prolog in an interactive fashion by using the assert special rule at runtime to define new statements (i.e. axioms and rules). And listing will display the set of statements that have been defined. Finally retract and retractall can be used to undefine statements.

These special rules, however, are not restricted to use in the interpreter - they can also be used at runtime by rules in programs. For example memoization⁶ is a good use for dynamic calls to assert.

Question Evaluation (Unification, Backtracking)

When evaluating a question, Prolog takes the question expression and matches it against the axioms and rules in the input knowledge base. The algorithm used for matching is called unification. While performing unification, Prolog makes guesses about what values should be assigned to each variable in the question expression. If it encounters a contradiction, it backtracks to one of its previous guesses and revises the guess. This process is described graphically in LPN §2.2 “Proof Search”.

Although Prolog is a declarative language, the precise behavior of evaluating queries depends on the order that rules and clauses are defined. In particular the performance of the same program can vary significantly depending on ordering. As another gotcha, if you define rules in a left-recursive fashion (as opposed to a right-recursive one), Prolog will go into an infinite loop when evaluating the rule.

It is possible to debug (and step through) the evaluation of a question expression using the trace command, which is described in LPN §2.4 “Practical Session”. This is useful for investigating performance issues.

Cuts

It is possible to optimize the execution of a Prolog rule by adding a cut as a clause. This is written as a bang (!). Unfortunately to use cuts effectively (and correctly), you have to understand the exact execution model used by Prolog. It is quite easy to unwittingly insert a cut that actually changes the semantics of the original rule.

There are even hacks you can do with cuts such as implementing “negation as failure” (\+). But again you have to be very careful since cuts can change your knowledge base’s semantics.

Side Effects & I/O

Some built-in clauses trigger side effects when they are examined by the unification algorithm. Programs take advantage of such “impure predicates” to do I/O and other types of side-effecting work.

For example asking the question print('Hello') will print Hello to the screen.⁷

To write to a file you might define:

printfile(Filename, Text) :-
    open(Filename, write, Stream),
    write(Stream, Text), nl(Stream),
    close(Stream).

And then pose the question printfile('hogwarts.txt', 'Hogwarts').

During evaluation, the printfile is rewritten to open(...) AND write(...) AND nl(...) AND close(...). Each of those subclauses is then each evaluated to true (performing the associated side effect) and the overall clause becomes just true.

References

LPN: Learn Prolog Now!

Unique Features of Various Programming Languages
- Describes several other programming languages and their unique features.

When posing a question in the Prolog interpreter, you must include a trailing period at the end of the question.↩
The job of the Java bytecode verifier is to examine a compiled Java program and check whether it is well-structured and therefore (reasonably) safe for the Java virtual machine to execute. For example the verifier would reject a Java program containing a command to jump to an instruction location outside of the current method.↩
See the Java Virtual Machine Specification. Particularly “Preface to the Java SE 7 Edition” and §4.10 “Verification of class Files”.↩
Unfortunately the is clause in this example cannot be run “backwards”, so this particular question will fail. (I am not sure why this particular restriction on is exists.)↩
A Prolog program that wanted to collect all combinations of inputs in this fashion (as opposed to a user at the interpreter) would probably use a combination of the findall, bagof, and setof special rules. See LPN §11.2 “Collecting Solutions” for more information.↩
Memoization is where a function that computes a value caches the result of the computation so that repeated invocations with the same input can immediately return the saved result. This can speed of performance in certain applications. See LPM §11.1 “Database Manipulation” for an example.↩
Note that 'Hello' (with single quotes) is a variable, not a string. The string "Hello" (with double quotes) is equivalent to a list of codepoints ([72, 101, 108, 108, 111]).↩

Visual Guide to Programming Language Properties

2013-02-20T00:00:00+00:00

Here’s an interactive chart showing high-level properties of various programming languages.
You can filter the chart to only show the properties that your favorite language supports.

RDiscount 2.0.7 released

2013-02-02T00:00:00+00:00

It has been just over two years since RDiscount 1.6.8 (the previous version) was posted.
As the new maintainer, I intend to provide updates a bit more frequently. :-)

Update to the latest version with:

gem update rdiscount

What’s New?

Footnotes - from PHP Markdown Extra
Superscript tweaks
- Be more picky about what comes before a ^ if we’re superscripting.
- Modify superscript grabbing so that it grabs parenthetical and alphanumeric blocks.
Other bug fixes
- Table-of-contents generation will no longer crash for header items containing links.
- Adjacent new-style [link]s are no longer incorrectly combined.

What’s Next?

RDiscount 2.1.5 will be the next release of RDiscount. Planned features include:

Fenced code blocks
- backtick-delimited - from GitHub Flavored Markdown
- tilde-delimited - from PHP Markdown Extra

Who am I?

I should also introduce myself: I am David Foster, the new maintainer for RDiscount.

Since I am using RDiscount on my personal website, you can expect continued updates as major new features are introduced in the underlying Discount library.

Why I no longer use Drupal
- Why Jekyll (and not Drupal) is awesome for making a personal website or blog.
- Jekyll can use RDiscount for Markdown rendering.

Learnings from SICP (and Lisp)

2013-02-01T00:00:00+00:00

Recently I took the liberty of reading one of the defining books in the domain of computer science: The Structure and Interpretation of Computer Programs, often abbreviated as SICP.

SICP is the computer science textbook given to undergraduates at MIT. It serves as an advanced text on software design and as an introductory text for the Lisp programming language.

Here are some interesting things I learned:

Role of Programming Languages
Abstract Data Types

Lisp vs. Pascal Design Philosophies
Polymorphism
Cross-Type Operations

Assignments, Mutable State, and Side Effects

Lazy Evaluation

Declarative Languages
Conclusion

Role of Programming Languages

Languages should:

provide the means for combining simple ideas to form complex ideas,
provide the means for abstracting units of computation, and
serve as a framework within which we organize our ideas about processes.

Units of computation can be abstracted in several forms:

procedure - Assigns names to common patterns, allowing you to work in terms of the named abstractions directly.
- AKA function, method
abstract data type - Allows the choice of representation (i.e. the implementation) to vary separately from the API (i.e. the interface).¹
- Built-in language support is often provided in the form of classes.
- See also: LISP constructors
  - AKA constructor or static factory method in other languages
- See also: LISP selectors
  - AKA accessors, mutators, or properties in other languages
module² - Groups several procedures and data types into a namespace independent from other modules.
- AKA package (Java), namespace (C++), module (Python)
- Languages that don’t have built-in support for this concept typically use prefixes to create de-facto namespaces. (C, Objective-C, PHP < 5.3.0)
assembly - Groups several modules together in a single versioned ³ package. Assemblies can depend on other assemblies, often from separate vendors.
- AKA shared library (C, C++), gem (Ruby), egg (Python)

Abstract Data Types

Lisp vs. Pascal Design Philosophies

For you youngin’s that have never used Pascal, just replace “Pascal” with “Java” in this section and you should get the right idea.

Two design philosophies:

Lisp-school: Create abstract data types by combining a small set of general-purpose data types (particularly collections). These complex structures can then be manipulated using operations on these general-purpose types.
- “Lisp is for building organisms - imposing, breathtaking, dynamic structures built by squads fitting fluctuating myriads of simpler organisms into place.”
- “Lisp programs inflate libraries with functions whose utility transcends the application that produced them.”
Pascal-school: Create many special-purpose data types (i.e. classes) and specialized operations to manipulate them.
- “Pascal is for building pyramids - imposing, breathtaking, static structures built by armies pushing heavy blocks into place.”
- “In Pascal the plethora of declarable data structures induces a specialization within functions that inhibits and penalizes casual cooperation.”

Roughly speaking, I think of the Lisp philosophy as programming with collections and the Pascal philosophy as programming with classes.⁴

Usage

The Pascal philosophy has won out in most statically typed languages such as C++ and Java, and in languages with poor (or nonexistent) built-in collections.

The Lisp philosophy is more common in dynamically typed languages that lack built-in support for classes, such as Lisp itself and JavaScript.

A hybrid approach (using both philosophies) is seen in languages that are dynamically typed, have built-in collections, and have built-in classes, such as Python and Ruby.

Polymorphism

Polymorphism is where multiple abstract data types implement a common interface, which is typically defined as a series of methods that can be called on all implementing types.

This allows client code, when given an object known only to implement a particular interface, to invoke interface methods on the object and end up calling the correct implementation of that method depending on the runtime type of the object.

Implementation Strategies

Polymorphism can be implemented in several different ways:

Switch on Typecode
- Easiest to implement. Hardest to maintain.
- Treated as an anti-pattern in many OO languages.
Virtual Lookup Tables
- AKA “data-directed programming” (in SICP) or v-tables (C++)
- Default implementation strategy for most OO languages.
Message Passing
- Most flexible.

Many languages provide a default implementation strategy as a language construct. For example C++ and Java use virtual lookup tables. Smalltalk and Objective-C use message passing. C doesn’t give you anything for free, so you have to roll your own polymorphism.

Tradeoffs

These implementation strategies for polymorphism have some tradeoffs, which are worth knowing:

Virtual lookup tables are restricted in that the total set of operations on the abstract data type must be known in advance.
Message passing, on the other hand, can be made more flexible:
- Implementing data types may choose to support more operations than the standard set on the interface.⁵
- Heck, individual instances can choose to support more operations than the standard set.
  - In such cases, you would want to interrogate (i.e. reflect on) an individual instance to determine what operations (i.e. messages) it understands/supports.

Cross-Type Operations

Introducing cross-type operations, such as add(Integer, Complex), is a very tricky design issue.

Having explicit functions that operate on all combinations of types is possible but highly verbose. With n types and m operations, you need n*m functions to implement all combinations. Impractical.

Another strategy is to use coercion to convert a value from one type to another. So instead of defining add(Integer, Complex), just define convertToComplex(Integer) : Complex, and use the existing add(Complex, Complex). To convert between all types requires at least n but no more than n² conversion functions.

Many programming languages have built-in facilities to automatically coerce types. For some languages (like JavaScript or PHP) these coersion rules are quite complex (and error-prone). Other languages (like ML) ban implicit coercion entirely.

Java only has coercion for built-in types.
C++ lets the datatype designer choose (via implicit or explicit one-argument constructors).
Scala relies on implicit coercion a lot to enable foreign methods to be introduced on types.

One wrinkle is that these conversion functions might introduce a loss in precision. For example not every integer can be represented as double of exactly the same value.

Assignments, Mutable State, and Side Effects

Introducing assignment creates lots of complications. In particular referential transparency is lost. Optimizations related to reordering and coalescing expressions need to be a lot more careful. Static reasoning of various kinds is impaired.

To reduce bugs it is best to minimize the use of mutation by using immutable objects whenever possible.⁶

Haskell takes a very aggressive stance against assignments, mutable state, and other side effects: they are banned by default. However Haskell does allow side effects within the context of a monad. This is a special construct unique to Haskell (so far as I know).

Lazy Evaluation

Normally expressions are evaluated immediately, in the order that they occur in code. Lazy evaluation changes this behavior such that expressions are only evaluated when some primitive operation (like print or add) requires the value of the expression. Until then the unevaluated expression is passed around (as a “thunk”).

Lazy evaluation enables that creation of lazy data structures, which is a useful performance optimization in some contexts.

Unrestricted mutation and lazy evaluation do not mix well in programming languages. Since unrestricted mutation is very common in mainstream programming languages, it is quite rare to be in an environment that supports lazy evaluation. Haskell is one of the few.

Declarative Languages

Logic programming languages are at the far declarative end of the imperative-declarative spectrum. They can be used to deduce answers from a set of initial set of declarative statements.

Prolog is the best-known example of a logic programming language.

Query systems for databases are a type of logic programming language.

A “query” (i.e. an expression in the language) is transformed into a “query plan” (i.e. a specific set of steps to follow) by a query planner. The implementation of these planners is quite complex.

Conclusion

So those are a few interesting things I learned from reading SICP. Taking notes while reading made it a lot easier for me to remember the content. You might try it when reading a technical book with a lot of new concepts.

Notes on Racket
- Describes Racket, a dialect of Lisp with batteries included.
  Useful for implementing other languages and creating cross-platform GUI programs.
Unique Features of Various Programming Languages
- Discusses several programming languages and their unique features.
Notes on Prolog
- Describes the Prolog language in more detail.

For example, complex numbers can be expressed in rectangular (a + bi) or polar form (r*cos(𝜽)). – Some representations are better than others for different operations. Adding works better in rectangular form. Multiplying works better in polar form.↩
SICP does not mention the notion of a module or an assembly, however these are common higher-level units for abstracting computation in languages other than Lisp.↩
Versioning assemblies effectively is a challenging topic onto itself. If done without care you get so-called “dependency hell”. Just getting a consistent version numbering scheme can be tricky. One popular versioning scheme is codified as Semantic Versioning.↩
In my opinion, a language that supports neither first-class collections nor first-class classes is non-viable for large scale general purpose software development. C, Assembly, and Fortran fall into this category.↩
For example, the Linux virtual filesystem, which is implemented in message-passing style, has a common set of operations that all filesystems are expected to support (ex: unlink). However individual filesystems may support additional operations: For example the HFS+ filesystem on Mac OS X additionally supports a delete operation, which has slightly different semantics than the standard unlink operation.↩
A common case where eliminating mutation may not be practical is when defining and working with large data structures that need many small updates made to them over time. If such a structure were made immutable, there would be a large performance penalty for recopying the entire structure whenever a small change needed to be made.↩

Unique Features of Various Programming Languages

2013-01-29T00:00:00+00:00

A language that doesn't affect the way you think about programming, is not worth knowing.

– Alan Perlis

I like learning new languages to get myself to think about problems in different ways.

Here’s a list of a few well-known languages I’ve worked with and some of the more interesting features I’ve encountered:

C
Java
C#
Python
JavaScript
Haskell
Lisp
Other

C

Undefined behavior
- The idea that a language specification would explicitly specify certain constructs as having undefined behavior is interesting. Most specifications leave things undefined by omission, not commission.
- Although this allows various compiler optimizations, many developers rely on their particular compiler’s implementation of undefined behavior without even realizing it.¹

Java

Objects are the primary unit of composition
- There are no standalone functions.
- Design Patterns can be used to describe high level object coordination patterns.
- However sometimes the community’s focus on objects can be a bit extreme.²
Checked exceptions
- Great when used sparingly, as it forces the caller to handle expected error conditions.
  - I do think that it was a usability error to make Exception checked and RuntimeException unchecked. Rather it should be Exception that is unchecked and a new CheckedException should be the base for all checked exceptions. This makes it clear that unchecked exceptions should be the default.
- Managing checked exceptions correctly is quite difficult.
Unicode strings
- Tons of other languages use “bytestrings” as the main string type which causes all kinds of problems when working with international characters.
Documentation focus
- Every library is expected to have JavaDoc documentation, which is generated from documentation comments in the source
- This is really powerful, since it makes it easy for developers to write documentation, and to do so at the same time the implementation is coded, when the desired behavior is most fresh in the mind.
Compatibility focus
- I have never seen a community so focused on maintaining backward compatibility. I can’t think of a single deprecated method in the Java library that was actually deleted.
- Even the Java Language Specification has an entire chapter devoted to binary compatibility.
- This focus may be partially attributable to the early mantra of Write once, run anywhere.
Garbage Collection
- Being freed from the confines of manual memory management makes it a ton easier to focus on more important things.

C#

Very similar to Java.³

Assemblies
- This is a level of encapsulation above the typical namespaces or modules in most languages. Assemblies are similar to the idea of static/dynamic libraries in C or JAR files in Java.
- Notably, you can mark members as internal, which makes them public within the same assembly, but private to everybody outside the assembly. This is quite useful.
Cross-language compatibility is first-class (not just for C)
- C# runs in the Common Language Runtime, which was designed from the beginning to support interoperability between languages.
Properties are first-class
- No longer need to write explicit getter and (optional) setter methods.
Listeners are first-class
- Classes can declare an event Foo with addFooListener and removeFooListener functionality built in.
- Unfortunately the implementation has some annoyances⁴.
Foreign Methods⁵ are first-class
- C# calls these extension methods.
Partial Classes
- Allows a class’s members to be defined in multiple files.
- Useful to add functionality to a generated class (for example, from a parser generator) without those modifications getting lost when the class is next regenerated.

Python

Indentation is significant
- Everyone notices this pretty fast.
- Overall I think requiring correct indentation is a Good Thing™, as it contributes directly to readable code.
- However this makes lambda expressions much less powerful than in other languages, since you can’t easily nest statements inside a lambda expression without explicit braces (or similar delimiters).
- Another side effect of having significant indentation is that the choice of tabs vs. spaces really matters. If you mix them, your program probably won’t run.
Interactive interpreter (REPL)
- Amazingly useful for prototyping quickly, running experiments, and learning the language.
Collections are first class
- You can type [1,2,3] to get a list, {'key': 'value'} to get a dictionary, and {1,2,3} to get a set. So much faster than new ArrayList<Integer>(...), new HashMap<Integer>(...), or new HashSet<Integer>(...).
Functions are first class
- You can declare them as literals, pass them around, and create higher-order functions that take functions as parameters.
- Functions can also live on their own without an enclosing class, in contrast to Java. This is often the simplest approach for a given implementation.
“Magic” marked explicitly
- Anything “magic” that the language treats specially has names surrounded by double underscores.
- For example:
  - An object’s constructor is called __init__.
  - The method implementing an operator overload for plus is called __add__.
  - A class’s metaclass is held by the __metaclass__ field.
Generators and Coroutines
- Generators enable straightforward pulling of values from a complex data source (like a parsed data structure).
- Coroutines enable straightforward pushing of values to a complex data sink.
- Python calls both constructs a generator.

JavaScript

No blocking I/O
- All I/O is non-blocking and asynchronous. This results in heavy use of continuation passing style.
Successful despite huge flaws
- Ease of deployment and ubiquity (i.e. business considerations) trump ease of use. (PHP also wins for the same reason.)
- A few flaws:
  - Everything is in the one global namespace.
  - No user-defined namespaces, modules, or importing of other files.⁶
  - No built-in facility for classes.⁷
  - Bizarre loose semantics for ==.⁸
  - Multiple illegal value sentinels: null and undefined.⁹
JSON (JavaScript Object Notation)
- A fantastically compact, readable, and portable notation for representing all kinds of data structures. Great for data interchange.

Haskell

Lazy evaluation
- Expressions are only evaluated when some primitive operation (like print or add) requires the value of the expression.
- Allows you to glue programs together in new ways. In particular, execution of multiple functions can be interleaved trivially and termination conditions can be separated from looped computation.
  - This is explained in more detail in Why Functional Programming (1990), §4 “Gluing Programs Together”.
- Allows you to extract complex expressions without fear of introducing a performance hit (since the expression will only actually be evaluated if it is needed).
- Allows you to define your own control flow operators.¹⁰
Side effects banned by default
- Mutation of data structures and I/O, both of which have order-sensitive side effects, are not allowed except within the confines of monad.
- A monad is a construct that explicitly controls evaluation order, in contrast to the usual unpredictable lazy evaluation behavior.
null banned by default.
- Unlike many languages, there is no special null value in Haskell that can be substituted anywhere.
- Instead if a function wants to return a value of type T or null, you would declare the function as returning type Option<T>, which could either have the value Some(tValue) or None. When declared in this way, callers are required to handle both possibilies.
Type inference (+ static typing)
- The benefits of static typing without the need to specify the types for everything?
  Count me in!

Lisp (Common Lisp, Scheme, Clojure)¹¹

Homoiconic
- When you write a Lisp program, the notation you use (the grammar) is equivalent to what a compiler would see (an abstract syntax tree or AST).
- Furthermore this Lisp code is represented as a nested structure of lists, symbols, and literals, all of which can be directly generated and manipulated in Lisp itself!
- This allows Lisp code to generate list structures which can then be run as Lisp code directly.
  - Generation can be done at compile time with macros.
  - Generation can be done at runtime as well, and then invoked with eval.
- However the highly uniform structure of Lisp code, devoid of operator and syntactic diversity, makes for lousy typography and thus low readability.
Macros
- A function that transforms the AST of its operands at compile time to new code.
- Macros can be used to generate arbitrary new statements and control structures.
  - Domain specific languages, in particular, are very easy to implement in Lisp thanks to macros.
- Macros can also be used to perform code optimizations at compile time¹² (similar to “template metaprogramming” in C++).
- Fluent use of macros requires the host language to be homoiconic, which is rare. Thus Lisp remains the only well-known language family that has macros.
Lisp Conditions and Restarts
- Allows bidirectional communication between different parts of the call stack. More powerful than exceptions, since conditions can not only unwind the stack but also wind it back again via a restart.
Call-with-current-continuation
- Allows you to save the current execution state of the program in a variable and jump back to it later. Multiple times, even. It’s like a friggin' time machine. You can implement fairly complex control flow operators with this function.

Other

Fexprs
- A function whose operands are passed to it at runtime without being evaluated.
- Similar to macros and lazy evaluation in terms of power.
- Has fallen out of favor since the 1980s due to being difficult for compilers to optimize. Also it is hard to provide good error messages.
- Furthermore most fexpr functions tend to rely on eval to continue evaluating its operands. The use of eval has its own problems…

Visual Guide to Programming Language Properties
- Shows programming language features, how they are related, and which features are present in popular languages.
- Visualizes some of the features discussed in this article.
Notes on Racket
- Describes Racket, a dialect of Lisp with batteries included.
  Useful for implementing other languages and creating cross-platform GUI programs.
Notes on Prolog
- Describes Prolog, a highly declarative language.
  Useful for exploring and verifying proofs.
Learnings from SICP (and Lisp)
- Discusses computer science concepts and how they manifest in various programming languages.

Series

This article is part of the Programming for Perfectionists series.

For example it is undefined what happens when you add to an integer variable and the variable overflows. In most compilers adding 1 to the largest integer wraps around to result in the smallest integer, and thus a number of programs depend on this behavior. Other compilers assume overflow is impossible.↩
Steve Yegge has a great rant on the over-focus on objects in the Java community: Execution in the Kingdom of Nouns ↩
C# is by design almost a direct copy of Java. It amazes me that Microsoft (C#’s sponsor) would spend so much effort making a copy of an existing language.↩
If you invoke an uninitialized event-property, it will throw a NullReferenceException instead of ignoring your request, as you would expect. Workaround by initializing events with an empty delegate.↩
I am referring to the Foreign Method design pattern here, not a native function from a foreign function interface.↩
Hence workarounds like RequireJS to get includes and modules.↩
Hence reimplementions of class semantics in libraries like MooTools, Prototype, and random blog posts. (That blog post has the best implementation, IMHO.)↩
Hence recommendations to always use === instead of ==.↩
Hence recommendations to only depend on the “truthy” and “falsy” values of expressions instead of direct comparisons with null or undefined.↩
Want to implement Ruby’s until loop or unless conditional? No problem.↩
There are more dialects of Lisp than any other language I can think of. These 3 (Common Lisp, Scheme, and Clojure) are just the most popular dialects. I’ve directly used Racket in the past, which is itself a dialect of Scheme.↩
Prismatic performs efficient compile-time DOM templating in ClojureScript using macros.↩

Things I Wish I Knew Before Working in Industry

2013-01-15T00:00:00+00:00

A representative from IBM came to visit me in college several years ago when I was a student and talked about several things he wished he knew before working in the software industry.¹ I’ve found his advice to be very useful at various points during my career. And would like to share it again with you now, in outline form:

Note: Bolded points I’ve found to be especially useful.

Technology

Future: integration²
Needed: ease-of-use & less complexity
Do not reinvent wheels!
- you’ll never finish the project

Technical Knowledge

Learn to learn
Don’t need to take a class to learn something
Learn a little about a lot of things³
Read news highlights from worthwhile sources⁴

Problem Solving

Build a network of people. Asking for advice is quicker than learning the hard way.
There is always more than one way to do something.
Weigh the pros/cons of each – then pick and choose.
Innovate before you execute. Think critically before you act.

Business & Technology

Technology adoption is driven by business. (practicality)
Business is nothing without technology.
Technology is nothing without business.
If you build a better mousetrap, they will not come.
A project, no matter how advanced, is doomed for failure without a paying customer.
Think about TCO (total cost of ownership) and ROI (return on investment).
You must always justify your returns.
Communication with managers and customers is vital!
If you can’t get someone to adopt your solution, what’s the point?
There is a difference between removing barriers and creating incentive (for a customer to switch to my project/product). You need both.

Promotions

It’s not what you know.
It’s not who you know.
It’s who knows what you know.⁵
Doing more of the same is not enough.
- Need to try new things – take on more responsibility.
Doing what is asked is not enough.
Ask you manager what the expectations are.

Working with Others

Credit + gratitude are not limited resources.
- Say thank you!
- Give credit where due.
Be ambitious but be humble.
Don’t make assumptions about others.
- Give people the benefit of the doubt.
“Your lack of planning does not mean I need to treat your request as urgent.”
Do what you say you will. Keep your word.
- reliable
- dependable
The best humor is self-deprecating.
There is always someone faster, smarter, and better looking.

Managers

They don’t have a magic wand. They don’t automatically fix problems.
Managers do not have a superset of the employees' (team members') knowledge.
Managers can’t read minds. Don’t assume. Communicate explicitly.
You are responsible for your own career. But ask for help.

You and Your Job

Your job is what you make of it. Don’t be confined by a job description.
Work/Life balance doesn’t have good default settings. You must do explicit actions.
- plan
- schedule for success
You are not your job. Do not define yourself like this.

Workers

We’re employed to add value, not spend money.
It’s easy to stay busy, but are you accomplishing anything worthwhile?
- don’t needlessly clutter your schedule
If you aren’t looking at it from a customer perspective, you aren’t looking at it.
You can’t do everything. Pick what you won’t do. (The world won’t end.)

Employment

Don’t expect your first job to be your dream job. Look for a path, not an endpoint.
Be willing to stretch your comfort zone.
- geographically, work type, etc.
- Otherwise you’ll stagnate.
Know your employer’s business model.
- Examples:
  - HP = selling ink;
  - Microsoft = selling Office, Windows, and cloud services;
  - Apple = ~~iPods~~ iPhones and (to some extent) Macs
  - Facebook, Google = selling ads
- Take a look at the annual reports for your company.
Get internships, even if with competitors, maybe even delaying graduation.⁶
- It gives experience.
- Increases employment opportunities.
Start looking and prepare much earlier.⁷

Successful Employment

If you want to make a difference, look for chaos.
Stable environments offer little opportunity.⁸
Good employees deal with challenges instead of pouting. Adapt.
Stay above the commodity line (with regard to skill set and type of work done).

Life

Understand other people’s motivations.
- Makes it easier to find new ways of solving the underlying goal.
Define your values and goals.
- ex: What type of job do you want?
Your behaviors should support your goals.
Expect your goals to change. Success is dynamic.
Don’t let others define success for you.
You can get rich by getting more or wanting less.
Life is one great balancing act.
- There’s no single right way of doing things.
- Balancing your priorities and goals is important.

Skills I Learned at Microsoft
Skills I Learned at Splunk
Skills I’m Learning at TechSmart - A future article. 🙂
Programming is Fun. But What Matters is Solving a Real Problem.

Sadly I have lost the name of the presenter from IBM who gave this talk originally.↩
Apple has always been smart about creating integrated products and experiences. In 2020 Apple announced the M1, an ARM system-on-chip (SOC) with incredible performance. The internet exploded.↩
I lean toward having a breadth of knowledge or being a generalist. But it’s also to have a deep specialization in one or two areas too. The T-Shaped Engineer gives more insight into the tradeoffs of having generalist vs. specialist skills.↩
For news of general interest to technologists I like Hacker News. There’s also frequently a subreddit for most topics. For example I’ve been liking r/django recently for Django-related information.↩
It’s not enough to be doing awesome things regularly. If you want that activity to pay off in increasing your luck, you’ll also need to find ways to let folks know what you’re up to, especially those with similar values and mission-alignment. Doing both will increase your luck surface area and opportunities will come your way more often. Luck is where preparation meets opportunity, but you can actually increase opportunities!↩
While in college I interned at several smallish companies (plus Google) and I can attest that my experiences there, in environments of different sizes, helped me learn what kind of work environment I was most interested in aiming for.↩
One of the most valuable classes I took in college assigned me the task of creating an online portfolo website. Although the initial version of that (and this) site was a bit rough, later editions were quite helpful in landing positions. In particular it helped get the attention of my cofounder at TechSmart. ✨↩
Google has the best actual onsite benefits of any company I’ve ever seen. (I worked there a while back in 2007.) But this cushiness means the engineers there don’t have the same incentives to be cutting edge. Multiple high-profile engineers, notably Steve Yegge and Waze’s CEO Noam Bardin, have written about why they eventually had to leave Google in order to make better headway in making a difference in the world. All of that said, Google is still very cushy and I don’t think any less of those who choose to work there.↩

Programming is Fun. But What Matters is Solving a Real Problem.

2013-01-08T00:00:00+00:00

I love programming. It tickles me when I stumble across a new programming language that lets me express concepts in a more clear and direct fashion. I love the cleanness and order that results after a good round of refactoring¹. I enjoy the challenge of hacking away at a fiddly problem and testing all the cases to get a correct and bulletproof solution.

However I have realized programming is merely an activity. It is a means, not an end. My work is not inherently valuable because I have created a piece of software. Rather it is because I have created software that solves somebody’s problem.²

Therefore it is the problem that deserves the most attention if you want to make a difference that people care about. In that vein I have been learning about the precepts of Customer Development and Customer Discovery, which deals with how to efficiently interact with customers to discover a problem that they care about.³

Direct customer interaction, such as through in-person interviews of the general public, is way outside my comfort zone. However I will persevere. I think it is a very important skill for actually making a dent in the universe.⁴

Refactoring is a systematic method for code cleanup: improving the structure of existing code without changing its behavior. I highly recommend the book Refactoring if you want a deep-dive into the topic.↩
Business majors & managers recognize early on the importance of solving business problems. After all, their training is around organizing other people to solve problems they identify. (Although I feel they often focus overly on managing and not enough on problem identification.)↩
For a great practical workboard around Customer Discovery (which is the first quarter of Customer Development), I highly recommend Ash Maurya’s Running Lean. For a full overview of Customer Development, I recommend Steve Blank’s The Four Steps to the Epiphany.↩
Steve Jobs famously said he wanted to “put a dent in the universe”. And boy did he ever.↩

Why I no longer use Drupal

2012-12-16T00:00:00+00:00

TLDR: Drupal is overly complex for a personal blog. It is hard to maintain. Simple static site generators are easier to work with in the long term.

My portfolio website was written during college using hand-coded HTML and used server-side includes to bring in common navigation.

Then in January 2010 I remade my site in Drupal. It gained lots of fancy features such as first-class support for project categories and project languages. You could subscribe to almost any page as a feed. There were project specific updates that could be commented on. From a feature point of view, it rocked.

But some problems became apparent over time:

Security updates were frequent and difficult to apply correctly.
I didn’t like writing articles on the site because I couldn’t use simple markup formats such as Markdown.
- Drupal has no good editor plugins for markup languages.
- And its visual HTML editor generates messy HTML.
The theme I used for the site was complex and hard to modify.
- And the CSS often interfered with my article markup, necessitating me to drop into HTML when editing certain articles.
There was no sane way to test structural changes to the site locally and then automatically deploy them to production.
- Normally this would be done by keeping the site structure in the filesystem and all user content in a database. This allows the filesystem contents to be easily deployed using git push, rsync, or similar techniques.
- Drupal, by contrast, keeps its site structure in both the filesystem and in the database, along with user-generated content. Updating only the parts of the database related to site structure is cumbersome and error-prone.
- Thus, I couldn’t really change the site structure after my initial deployment to production.
Nothing was in revision control, which made me nervous.
I had to hack the Drupal core to get my contact form to work with my web hosting provider.
Most of the features of the site weren’t being used by readers.

So now I am rewriting my site yet again in straight HTML (via a simple static site generator) and outsourcing all user generated content (like comments) to third party service providers.

Benefits:

Simple simple simple.
Automatic security. The web server just serves static files.
Automatic scalability, for the same reason.
Simple authoring in Markdown with powerful client-side text editors.
Instant deployment with git push.
Everything in revision control.
I can use any web hosting provider. Hell, I can just use GitHub Pages for free.
Ultimate control over the site theme and CSS. I can fix problems myself.

Retrospect vs. Time Machine

2010-11-13T00:00:00+00:00

The old Retrospect 6.0

In the past I have used Retrospect 6.0 to backup my Macs, since it was one of the few programs that reliably backed up everything correctly, with all the metadata intact. But it is looking much less attractive these days…

It no longer finds my laptop on the network reliably, due to some bug. Especially when switching between wireless and wired networks. Thus I often miss backups.
It is slow.
It does not groom backups.
It stores everything inside a single monolithic file, which is at risk of corruption.

The new Retrospect 8.0

The latest version of Retropect (8.0) is somewhat better but has some problems:

It does not seem to backup all attributes correctly. For example, volume icons do not seem to be restored correctly. Inaccurate backups are useless.
It is more complicated to use.
Automatic grooming, although supported, is so slow to be practically unusable. It takes maybe 2x as long to do one grooming operation as an incremental backup.

Time Machine

Now there is Time Machine. Since my primary systems are 10.5+, I can use it.

It should support reliable backup of all filesystem attributes, since it was made by Apple, who is in a position to know about them. Also, Apple typically makes solid products.
It has quick, automatic, incremental backups.
It also grooms automatically while it is backing up. Grooming behavior is intelligent, keeping snapshots at sensible intervals.
The single-file restore interface is precise, intuitive, and fast.

Only wrinkle is that Time Machine is not really designed to backup to network drives, however there are instructions for circumventing this. (Of course there is Time Machine + Time Capsule for network backups, but a Capsule is egregiously expensive.)

2012 Update

Time Machine misses a few filesystem attributes that Retrospect 6.0 nailed back in the day, like volume icons.

I think it may also strip HFS+ extended attributes, since applications appear to be quarantined after they are restored from a Time Machine backup.

One last nitpick is that TM has very poor handling for bad media and bad blocks. In particular if your primary hard drive develops bad blocks, which is an early sign of imminent failure, Time Machine will choke and refuse to backup the entire volume. And this is exactly at the time you need it your files backed up the most, when your hard drive is likely to fail.

DaFoster (Software)

AI Attribution in Git

Why the Co-authored-by pattern?

Oops, I wrote a database

Background

Strategy: Write file then rename to commit, rather than (re)write in place

Strategy: Write file then replace to commit, if the file may have concurrent readers

Windows-only: Move-aside a file before replacing it

Read Repair: A time to clean up incomplete operations

Zooming Out: The Big Picture

Issue counts always go up

Designing Software in the Large

Complexity

The big picture

Dependencies: A Cause of Complexity

Obscurity: A Cause of Complexity

Strategic vs. Tactical Mindset

Takeaway

The trouble with the Symbol type

Related: Interned Strings

Dumping a traceback when an error message is printed

Effective Code Reviews

PyCon US 2024 Highlights

Favorite Talks

Performance

Debugging

API Design

AI

Typing System

Fun

Redis relicensing: Why is this a problem?

Bulkheads: A pattern for handling unexpected errors in software

Goals

How it Works

Example

Try it!

Appendix

Effective Critical Sections and Bulkheads

Enforcing that an error-handling strategy is defined

API

Bulkhead API

Critical Section API

Critical Sections (in Crystal) always print to the standard error stream

Error-handling enforcement utilities

Debugging a deadlock in Python

What does a deadlock look like?

Isolating and fixing a deadlock

Culprit: Inappropriate locking

Common culprit: Acquiring multiple locks in an inconsistent order

Fin

Custom file icons, folder icons, and app icons on different operating systems

Icons in macOS

Icons in Windows

Icons in Linux

Conclusions

How to make a binary .deb Debian package using Docker

What’s in a binary .deb package?

How can Docker be used to build a binary .deb package?

Extracting the installed files from a Docker image layer

Building a .deb package from a set of files

Write a control file

Build the .deb

Inspect the .deb

Find and fix errors with Lintian

Don’t install to /usr/local!

Copy out the .deb

Testing your .deb package

Putting it all together

Debugging bash scripts with the bashdb debugger

How to install bashdb

How to run bashdb

Common commands in bashdb

Crystal Web Archiver 1.3.0b Released!

Previous Releases

CompressedTextField for Django & MySQL is released!

Crystal Web Archiver 1.1.0b Released!

Digital preservation is important

Related Projects

You might not need centralized continuous integration

The wonders of continuous integration

Write a `control` file

Build the `.deb`

Inspect the `.deb`

Don’t install to `/usr/local`!

Copy out the `.deb`

Alternative: Distributed ⁴ continuous integration