Seymour: Live Programming for the Classroom

This web essay has only been tested on the latest version of Google Chrome, and has not been optimized for mobile phones.This essay is also interactive. To interactively edit the examples shown in each figure, please visit this page with Google Chrome version 61 or greater.

We want to create a better programming experience for learning and teaching programming. This paper describes Seymour, a live programming environment that features two visualizations of a program’s execution. One shows the execution’s details and another depicts it at a high level. These two visualizations come together to create a compelling user experience that we plan to test and improve with student feedback.

Introduction

The live programming community has produced many inspiring visions of programming, but we have yet to see these ideas significantly impact how we program today. There are substantial design and engineering problems that stand in the way of creating live programming experiences fit for widespread adoption. It may be too difficult to make an experience scale to the needs of programmers at large in one shot. Perhaps we can solve a smaller version of the problem and adapt some of the techniques we develop to a broader solution. We are developing an environment that aims to provide a better programming experience for an undergraduate intro to programming course.

There are several advantages that this use case affords us. First, programs that students write are far smaller and less complex than those written for commercial purposes. They are also guided by a class curriculum, including homework assignments and projects. This allows us to create new experiences without having to design and optimize for very large programs. Furthermore, because the teacher controls class assignments, we can tailor the system to each assignment, instead of attempting to create a fully general experience. Even if our programming experience fails to scale to be useful in the ‘real world,’ it has the potential to help students overcome programming’s steep learning curve.

This paper presents Seymour, an early prototype that follows this strategy. The goal of Seymour is to help beginners develop an intuition for program semantics, and encourage experimentation. We do this by providing a visualization of a program (rendered as the student writes it) that tells a cohesive story. Students and teachers spend significant time simulating the behavior of a program (mentally, or on paper), often erroneously. An automated method to create visual explanations of these programs can significantly ease the mental burden of teachers and students alike when introduced in the right context, e.g., by having the professor and students draw the visualization before transitioning to a computer-generated version. Juha Sorva. Visual Program Simulation in Introductory Programming Education @section 11.2

This paper contributes a multiple view Wang Baldonado et.al. “Guidelines for Using Multiple Views in Information Visualization” visualization of program execution with two parts:

• a micro visualization that shows line-by-line details about the program’s execution, including readable stories for arbitrary control structures and summaries of expressions’ side effects, and
• a macro visualization that serves as an overview of the program’s execution, and can be used as a tool to focus on different points in the execution.

This is a vision paper that describes Seymour’s macro and micro visualizations and how they come together to create a compelling live programming experience. We intend to discuss the technical details of Seymour’s implementation in a subsequent paper.

The Programming Environment

Seymour’s programming environment consists of three different components, as shown in the screenshot below:

The prototype shown in this paper relies on our own language—an unholy combination of JavaScript and Smalltalk. Before using Seymour in classrooms, we plan to adapt it to use Python instead. See the future work section for more details.

These are:

• the code editor (top left);
• the micro visualization (top right), which provides details about the activation of a method or function; and
• the macro visualization (bottom), which provides an overview of the entire program’s execution.

These components help the programmer see and understand the dynamic behavior of their code even while they are writing it.

The Micro Visualization

The micro visualization displays details about the execution of the code. These details are shown next to the lines of code that produced them, and they are updated as the programmer makes changes to the code. Try changing the ‘0’ to a ‘2’ below

Each subexpression in the program is rendered as a blue dot. To see the value of a subexpression, the programmer simply hovers over its corresponding blue dot. Try hovering over the ▪’s next to avg

The micro visualization displays loops using a columnar format, where each column represents an iteration of the loop. The programmer can read through a row to see all of the effects from a line of code over time, or read through a column to see what happened in a particular iteration.

Because Seymour only shows effects for code that is executed, it is easy to “follow the flow” Bret Victor. “Learnable Programming” of the program over time. Try scrolling the micro visualization to the right

In addition to built-in loop constructs, programmers often rely on user-defined control structures, e.g., map, filter, and reduce. While these are not loops in the strict sense of the word, they are “loop-like” so the programmer would benefit from visualizing them as such. Seymour automatically detects loopy behavior and provides a columnar visualization for it, as shown below. In fact, all built-in control structures in Seymour’s language are simply message sends/method calls (as in Smalltalk) and are rendered in a columnar manner using the same loop mechanism—i.e., there is no special handling for built-in control structures.

We use emoji to identify different objects, instead of memory addresses or object IDs.

interact reset back to video

Note that the call to reduce() is rendered as a loop even though its implementation is recursive. A detailed description of our mechanism for detecting loopiness and Seymour’s micro visualization will be the subject of an upcoming paper.

While creating the micro visualization, we sought to mitigate users’ mental simulation of the program. Seymour shows every state change, without requiring interaction, so that the user does not have to simulate the program and interactively view parts of the state.

A method call, in addition to producing a result, can modify program state. If its side effects are not shown, the user must fall back to mentally simulating the program. Consider the program below: Cmd+click on (a = 7) to see the details of f()

interact reset back to video

Because the call to f() is preceded by a statement that increments a, and followed by a statement that decrements it, the user might expect a to be 5 at the end of the program. If Seymour did not show that f() set a to 7, the user would need to read f()’s source to figure out why a is not 5. Because f()’s side effects are shown, the user can confirm at a glance that a = 6 because f() set a to 7 before it was decremented.

For each call, all side effects that are relevant to the execution are summarized. The micro visualization shows the last value of variables that are written multiple times, and hides the values of local variables that do not affect computation past the duration of the call. In the example below, we only show a’s final value, and b = 7 does not appear in f()’s summary. Cmd+click on (a = 6) to see the details of f()

interact reset back to video

A call’s summary elides most of its implementation details in order to give the user ‘just enough’ information to continue reading the story of the execution. If the user is interested in these details, Seymour can show the story of any call. In fact, the whole program is treated as a call and its implementation details are displayed like those of any other call, such as the call to add5() below. Cmd+click on the ▪ next to ans to see the details of the block call

interact reset back to video

When visualizing a call, we also show all lexical scopes that the callee (i.e., a method or a block) has access to. All the information needed to calculate the call’s result is on screen, and the method only has access to the values it can ‘see’. In the program above, which visualizes the call to add5() on line 8, Seymour shows the value of this that add5() has access to, explaining why it returns 11.

The Macro Visualization

Seymour’s micro visualization can show any single call in the program. However, views of disconnected calls are not enough to completely understand the program. Programmers need a low-level understanding of individual calls and a high-level understanding of how they are related.

The macro visualization helps the user build this high-level understanding by

• showing overall patterns in the program;
• serving as a stable, global context to which the micro visualization can be related; and
• providing a user interface for focusing the micro visualization on different calls.

interact reset back to video

The visualization is a variation on the icicle plot Kruskal and Landwehr. “Icicle Plots: Better Displays for Hierarchical Clustering” —a method for visualizing hierarchical data that makes it easy to see the ‘shape’ of a computation. Each method call is drawn as a rectangular node on screen. Time progresses from left to right with clusters of calls acting as landmarks on the program’s timeline. All calls that a method makes are drawn below its node in the visualization. Calls that do not make any further calls (leaf calls) have fixed width, and all other calls are wide enough to contain all of their children; calls with more subcalls are wider than those with fewer subcalls. Google Chrome’s DevTools Google Developers. “Performance Analysis Reference”

Librato Blog. “Timeline Profiling with Chrome DevTools” uses a similar visualization to provide an overview of calls that occur in a profile, but instead of giving leaf calls a fixed width, all calls are scaled based on run time.

The macro visualization does not change as the user selects different calls, showing every call in the program. However, without further help from the system, it is difficult to connect the time-oriented macro visualization with the code-oriented micro visualization, so Seymour helps the user make this connection with parallel highlighting. When the user hovers over a call in the code, all nodes corresponding to that call in the macro visualization are highlighted . Similarly, when the user hovers over a node in the macro visualization, Seymour highlights the definition and site of the call in question.

The visualization is also a user interface for navigating between calls in the micro visualization. When the user clicks on a node in the macro visualization, Seymour focuses the micro visualization on the call associated with that node. This makes it easy for the user to switch between contexts, drill down, and gather detailed information about the program. The currently focused call is colored dark blue in the macro visualization, along with all activations whose local variables the callee can access. Since every call can access the program’s global state, the root node of the macro visualization is always highlighted

Live Programming with Seymour

In this section, we will give the reader a feel for Seymour’s live programming experience. Several examples show how the macro visualization, the micro visualization, and the ‘focus’ mechanic come together to let the user see the execution of a program as they write it. Each example consists of a video followed by commentary. Click the timestamp at the top of each paragraph to view the relevant portion of the video.

Number.fibonacci()

In the previous section, we saw how a user might interact with a fully written implementation of Number.fibonacci(). This example shows how one might implement fibonacci() by getting several concrete examples working.

The user starts by writing a call to fibonacci() and a blank method definition. Initially, the plan is to get 5.fibonacci() working.

Realizing that 5.fibonacci()’s result depends on 4 and 3’s results, the user decides to concentrate on 1.fibonacci() and 2.fibonacci() (the base cases) instead.

Focusing on 2.fibonacci(), the user writes code to handle the base cases. The calls to 1.fibonacci() and 2.fibonacci() update to show the correct answer. While the user edits the code, the micro visualization remains focused on the same call This technique is described in McDirmid and Edwards’ “Programming with Managed Time,” section 3. , allowing them to see how the edits affect the method’s execution.

Now, they have all the parts necessary to write the rest of the implementation. They focus on 3.fibonacci() and implement the recursive case. They can see the results of each recursive call and use that information to complete the implementation. Similar methods for using runtime information are described in Allen’s Anatomy of Lisp, section 6.21 and Edwards’ “Example Centric Programming”.

Array.toString()

This example shows how the user can work with loops in Seymour. They will implement Array.toString(), which must include the string representation of element of an array.

Once again, the user calls the method with a couple examples and tries to get them working.

Writing the implementation continues in much the same manner as the last example. The user realizes that the result must show each element of the array, so they loop. The loop shows the value of each element. These values will be used as building blocks to compute the answer.

The user takes advantage of the liveness of the environment and experiments with different ways to combining the elements of the array. Initially, they incorrectly prepend sx to ans. After seeing the items appended to ans in reverse order, the user corrects their mistake.

The user tries out different ways to insert commas between each element. Putting a comma before every element does not work so they do not append a comma before the first element in the array.

Finally, the user inspects the other call to make sure that the method executes as expected.

Related Work

In “Inventing on Principle,”Bret Victor. “Inventing on Principle” @16:25 Victor introduced a novel visualization of program execution that has heavily influenced the design of Seymour’s micro visualization. His visualization displays changes to program state over time with each change lined up with the line of code that produced it. Our micro visualization builds on Victor’s visualization and extends it with call summaries that enable the programmer to see the side-effects of a call. In Victor’s visualization, this information is only available via local expansion (or inlining) of the call,Bret Victor. “Showreel 2011–2012” @1:41 and because this requires interaction, these side effects (e.g., assignments to global variables) are easy to miss.

Light TableChris Granger. “Light Table” is an IDE that enables programmers to see dynamic (runtime) state inline with source code. It supports watches that can be attached to expressions in the code to reveal their latest value as the program is executed. However, if a statement or expression is evaluated many times—e.g., as part of the body of a function that is called more than once—Light Table does not let the programmer select the context or activation that is associated with a watch. YinYangSean McDirmid. “Usable Live Programming” solves this problem with its probes that also enable the programmer to view the values of expressions. Probes are similar to Light Table’s watches, but they are associated with an implicit execution context that must be activated through navigation or through a bookmark (created when a print statement is executed). By activating an execution context, the programmer can see the corresponding probe values. With Seymour, the programmer can see the details of any context by selecting it via the macro visualization. But unlike Light Table and YinYang, Seymour shows all of the side effects in the selected context automatically, minimizing interaction required to see program state.

DejaVu is a domain-specific IDE extension for “interactive camera-based programs”Kato et. al. “DejaVu: integrated support for developing interactive camera-based programs” that features a low-level canvas view that shows values of variables in a ‘frame of interest’, and a high-level timeline view that shows variable values for many frames. These views complement each other in a similar manner to the micro and macro visualizations in Seymour. But whereas DejaVu is designed specifically for the domain of camera-based programs and features visualizations tailored for displaying image data, Seymour’s visualizations are designed to explain the semantics of generic programs. We discuss ways to integrate domain specific visualizations with Seymour in the future work section.

Online Python Tutor Philip J. Guo. “Online Python Tutor: Embeddable Web-Based Program Visualization for CS Education”

Philip J. Guo. “Live Programming Mode - Python Tutor” , a visualization tool for teaching beginner programmers, features a visualization that shows the entire program’s state at a single point in time. This includes the program counter, the stack, the heap, and relations between objects. Objects are drawn as boxes and are connected by arrows, as a professor might depict them in a classroom. This type of visualization is complementary to those provided by Seymour, and we believe programmers would benefit from seeing them together.

Future Work

Classroom Trials

We envision Seymour being used as a tool to assist students and professors in an undergraduate intro to programming class. However, Seymour has not been user tested as of the time of this writing, and we have many improvements to make before it is ready to be used to teach a class. We will be showing our programming environment in classrooms in the fall in order to test it on students and adjust our designs based on their feedback. We have access to an intro to programming class and a programming languages class at UCLA, where we will be using Seymour to help students learn Python.

To support these trials, we plan to adapt Seymour to use Python as its underlying language. To simplify the process of changing the underlying language, we have developed a language-agnostic library for creating micro and macro visualizations which can work with any imperative language. With this library, adapting Seymour to a new language becomes a simple matter of connecting the language’s runtime to the library. We will discuss this library in detail in an upcoming paper.

Better Support for Larger Programs

While Seymour’s visualizations work well for small programs, they become harder to use as programs grow in size. With larger programs, such as those students write for a class project, we have found that the micro visualization shows too many low level details, while the macro visualization shows too few. We are interested in improving Seymour’s visualizations to better explain such programs.

The macro visualization can provide a global view of small programs. However, as program size grows, it becomes too large to read and understand. We are interested in using techniques such as trace pruning Johannes Bohnet. Visualization of Execution Traces and its Application to Software Maintenance , fisheye, and minimaps to help the user process medium and large programs.

We would also like to show more low level details in the macro visualization in order to help users better tie the macro and micro visualizations together, and to understand where to focus next. In a previous project (shown below), we visualized the execution of JavaScript methods and let the user annotate the visualization based on low level details (e.g. the value of a variable for a particular call).

In this example, which shows a parser generator, characters being consumed are shown in green, parse errors are shown with red text, and calls where the parser backtracked are shown in blue. To create an annotation, the user writes a query that selects the calls they want to annotate (e.g. calls that backtrack, calls that consume a character), and modifies their nodes’ appearance in the visualization. These modifications let the user read the macro visualization for high-level, program-specific patterns, instead of having to interact to find the information they are looking for.

Another solution we are exploring is to make it easy to create domain specific visualizations from Seymour’s run-time data. These visualizations can display program information at the right level of abstraction to help the user understand the program. Instructors can use such a system to great effect as visualizations can be reused as part of course materials and can be shared with students as explanatory tools. Having computer-generated visualizations tied into the system also eases the tedious and error-prone process of illustrating algorithms on the blackboard.

Below is a screenshot of an initial attempt at visualizing Dijkstra’s algorithm using information from a running program. We use a timeline to show how a graph is traversed as the algorithm progresses. Red nodes are nodes that have not yet been traversed, and each node’s distance from node a is shown next to the node’s label. This visualization is live and updates as the user changes the code.

Language Features for Improved User Understanding

Seymour’s user experience can be further improved by adding features to the programming language underlying the system. We are exploring several language extensions that could help the user better understand their programs.

Micro visualization summaries can become large if the method call being summarized has many side effects. The example below creates an array with all elements from 1 to 10, showing 10 side effects on line 1.

We have begun experimenting with modular side effects, a language feature that allows an object to specify how its side effects should be presented to the user. As illustrated above, the micro visualization currently shows all side effects when working with an object. For example, when inserting an element into a red-black tree, the user sees all the rotations the tree conducts to maintain its invariant. With modular side effects, an object can choose how to present its internal state. A red-black tree might present itself as a set, only showing that an element has been inserted, sparing the user an excess of irrelevant details.

Together with our colleague Jonathan Edwards, we have also started exploring interventions—a mechanism that lets programmers temporarily modify program execution to aid development. In addition to displaying the effects of a call, call summaries can be used to change the program’s execution. If a method call produces an incorrect answer, the user can intervene and assert that the method call return the correct value. This assertion can then become a unit test, and its value can be substituted for the call’s return value, allowing the user to make progress on other parts of the program and address the broken method at a later time.

Conclusion

Seymour is a live programming environment that visualizes program execution as the user types. It features a micro visualization that shows details of the program’s execution, and a macro visualization that puts the micro visualization in context, letting the user focus on different parts of the program execution. These visualizations come together to give the user a helpful live programming experience.

We are excited about the prospect of using this environment in the classroom and learning from student feedback. In doing so, we hope to make Seymour a valuable learning aid for students, and ultimately, create a better user experience for all programmers.

Acknowledgements

We would like to thank Patrick Dubroy, Aran Lunzer, Sean McDirmid, Todd Millstein, Yoshiki Ohshima, and Marko Röder for their valuable feedback on this paper. Also, thanks to Bret Victor, Toby Schachman, Daniel Windham, Chaim Gingold, Jonathan Edwards, Glen Chiacchieri and the rest of the gang at HARC for many invaluable discussions that helped us formulate these ideas.

This web essay uses Fira Sans, Fira Mono and Equity as its title, code and body fonts respectively. It uses CSS stylesheets based on tufte-css. Its favicon is the rocket emoji from the fxemoji collection. It also uses the open-source libraries checked-emitter, codemirror, jquery, ohm, underscore.js, and ua-parser-js.

References

1. John Allen. 1978. Anatomy of LISP. McGraw Hill, Inc., New York, NY, USA.
2. Kayce Basques. Performance Analysis Reference. Retrieved from https://developers.google.com/web/tools/chrome-devtools/evaluate-performance/reference
3. Johannes Bohnet. 2010. Visualization of Execution Traces and its Application to Software Maintenance. PhD Dissertation. Mathematisch-Naturwissenschaftliche Fakultät / Institut für Informatik, Potsdam. Retrieved from https://publishup.uni-potsdam.de/opus4-ubp/frontdoor/index/index/docId/32254
4. Jonathan Edwards. 2004. Example Centric Programming. SIGPLAN Not. 39, 12 (December 2004), 84-91. DOI:https://dx.doi.org/10.1145/1052883.1052894
5. Chris Granger. Light Table. Retrieved from http://lighttable.com/
6. Brendan Gregg. Flame Graphs. Retrieved from http://www.brendangregg.com/flamegraphs.html
7. Philip J. Guo. 2013. Online Python Tutor: Embeddable Web-based Program Visualization for Cs Education. In Proceeding of the 44th ACM Technical Symposium on Computer Science Education (SIGCSE ’13), 579–584. DOI: https://dx.doi.org/10.1145/2445196.2445368
8. Philip J. Guo. Live Programming Mode - Python Tutor. Retrieved from http://pythontutor.com/live.html#mode=edit
9. Jun Kato, Sean McDirmid, and Xiang Cao. 2012. DejaVu: Integrated Support for Developing Interactive Camera-based Programs. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (UIST ’12), 189–186. DOI: https://dx.doi.org/10.1145/2380116.2380142
10. J. B. Kruskal and J. M. Landwehr. 1983. Icicle Plots: Better Displays for Hierarchical Clustering. The American Statistician 37, 2 (1983), 162–168. DOI: https://dx.doi.org/10.2307/2685881
11. Sean McDirmid. 2013. Usable Live Programming. In Proceedings of the 2013 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (Onward! 2013), 53–62. DOI: https://dx.doi.org/10.1145/2509578.2509585
12. Sean McDirmid and Jonathan Edwards. 2014. Programming with Managed Time. In Proceedings of the 2014 ACM International Symposium on New Ideas, New Paradigms, and Reflections on Programming & Software (Onward! 2014), 1–10. DOI: https://dx.doi.org/10.1145/10.1145/2661136.2661145
13. Juha Sorva. 2012. Visual program simulation in introductory programming education. PhD Dissertation. Aalto University. Retrieved from DOI: https://aaltodoc.aalto.fi:443/handle/123456789/3534
14. Bret Victor. 2012. Inventing on Principle. Retrieved from https://vimeo.com/36579366
15. Bret Victor. 2012. Learnable Programming. Retrieved from http://worrydream.com/LearnableProgramming/
16. Bret Victor. 2013. Showreel 2011–2012. Retrieved from https://vimeo.com/62049081#t=1m41s
17. Michelle Q. Wang Baldonado, Allison Woodruff, and Allan Kuchinsky. 2000. Guidelines for Using Multiple Views in Information Visualization. In Proceedings of the Working Conference on Advanced Visual Interfaces (AVI ’00), 110–119. DOI: https://dx.doi.org/10.1145/345513.345271
18. Timeline Profiling with Chrome DevTools. Librato Blog. Retrieved from http://blog.librato.com/posts/chrome-devtools