Clamps Packages

Excursion: Closures

The definition of the watch functions in the NanoKONTROL2 example contains a tricky part: The variable i of the dolist is bound to a symbol n in each iteration using a let expression in its body.

This is necessary: If the let form is omitted and the variable i directly referenced in all 16 watched functions, it wouldn't work as expected: i would be dereferenced at the moment, when the lambda function gets called as an effect of moving a fader and not at the moment, the lambda form got defined in the context of the call to the watch function during the evaluation of the dolist.

After the dolist has finished, the value of i is 8 and that will be the value seen in any of the functions defined by watch. All functions dereference the same i when a fader is moved.

Let's see this in action:

;; WARNING: Faulty implementation of binding watch functions in an
;; iteration:

(with-slots (unwatch nk2-faders) (find-controller :nk2)
  (dotimes (i 8)
    (push (watch
           (lambda () (msg :warn (format nil "Knob ~a turned: ~a" (1+ i)
                                     (get-val (aref nk2-faders i))))))
          unwatch)
    (push (watch
           (lambda () (msg :warn (format nil "Fader ~a moved: ~a" (1+ i)
                                     (get-val (aref nk2-faders (+ i 8)))))))
          unwatch)))
;; => nil
;; Output in the REPL:
;; warn: Knob 1 turned: 0
;; warn: Fader 1 moved: 0
;; warn: Knob 2 turned: 0
;; warn: Fader 2 moved: 0
;; warn: Knob 3 turned: 0
;; warn: Fader 3 moved: 0
;; warn: Knob 4 turned: 0
;; warn: Fader 4 moved: 0
;; warn: Knob 5 turned: 0
;; warn: Fader 5 moved: 0
;; warn: Knob 6 turned: 0
;; warn: Fader 6 moved: 0
;; warn: Knob 7 turned: 0
;; warn: Fader 7 moved: 0
;; warn: Knob 8 turned: 0
;; warn: Fader 8 moved: 0

When initializing the watch functions everything seems to work: On definition of the watch in the dolist iteration, the lambda function gets called once. This moment is called compile-time and at that moment, i has the correct value as can be seen in the REPL output above.

But after the dotimes has finished, the variable i is bound to the value 8, which becomes evident when moving any fader at run-time:

;; Output in the REPL when moving a fader:

clamps>
error: Invalid index 16 for (vector t 16), should be a non-negative integer below 16.
error: Invalid index 16 for (vector t 16), should be a non-negative integer below 16.
error: Invalid index 16 for (vector t 16), should be a non-negative integer below 16.
error: Invalid index 16 for (vector t 16), should be a non-negative integer below 16.
error: Invalid index 16 for (vector t 16), should be a non-negative integer below 16.
clamps>

The error occurs only, when a fader is moved, because their lambda functions use the index (+ 8 i) which is 16 in this faulty implementation and beyond the maximum index 15 of the fader array in the Nanoktl2 instance. When moving a knob, the output in the REPL will state that knob 9 has been turned (which doesn't exist) and the value reported will be the last value received from Fader 1 because that is the Fader at index 8 of the array¹:

;; Output in the REPL when turning a knob:

clamps>
;; warn: Knob 9 turned: 0.0
;; warn: Knob 9 turned: 0.0
;; warn: Knob 9 turned: 0.0
;; warn: Knob 9 turned: 0.0
;; warn: Knob 9 turned: 0.0
clamps>

To avoid this situation, the let expression in the correct example serves the purpose of using a new variable n in each iteration of the dolist to capture the value of i at the compile-time of the function so that dereferencing n at run-time will refer to the correct value.

Capturing variable bindings of functions at compile-time for correct dereferencing at run-time uses so-called closures. It is very important to have a good understanding of this concept when working with clamps (or any dynamic system with functional properties).

Without going into too much detail, following are some examples intended to clarify what is happening above. It is important to study the examples thoroughly until it is completely understood what is happening.

(defparameter *my-fn* nil)
;; => *my-fn*

;; example of a closure: The symbol i is bound outside of the function
;; body and the function dereferences its value, whenever it gets
;; called:

(let ((i 3))
  (setf *my-fn*
        (lambda () i)))
;; => #<function (lambda ())

(funcall *my-fn*) ; => 3

;; This is the same in the context of an iteration: The symbol i is
;; incremented on each iteration of the dotimes until it reaches the
;; value 8, which stops the iteration. In other words: After the
;; iteration has ended, i has the value 8. As the function
;; dereferences the value of i at the time the function gets called,
;; it returns the value 8:

(dotimes (i 8)
  (setf *my-fn* (lambda () i)))
;; => nil

(funcall *my-fn*) ; => 8

;; The same again but showing, that i is never reaching the value 8
;; within the iteration:

(dotimes (i 8)
  (print i)
  (setf *my-fn* (lambda () i)))
;; => nil
;; output in the REPL:
;; 0 
;; 1 
;; 2 
;; 3 
;; 4 
;; 5 
;; 6 
;; 7 

;; As before, when calling *my-fn* it will return the value of i after
;; the iteration has finished:

(funcall *my-fn*) ; => 8

;; Binding the symbol n to the value of i within the dotimes will
;; dereference the values of i during the loop:

(dotimes (i 8)
  (let ((n i))
    (setf *my-fn* (lambda () n))))
;; => nil

;; The let binds a new n with the current value of the outer i in each
;; iteration. When the iteration is done, *my-fn* has been
;; rebound/redefined 8 times to reference the new inner n
;; symbols. After the loop has finished, *my-fn* returns the value of
;; the last binding of n in the last loop iteration.
;;
;; In that iteration i had the value 7 and n was bound to that value,
;; so when n gets dereferenced later, it will evaluate to 7:

(funcall *my-fn*) ; => 7

;; For clarity: The let creates a new and different n on-the-fly on
;; each iteration. This might get clearer when spelling out the
;; 8 iterations of the dotimes:

(block nil
  (let ((i 0))                       ;; start of the dotimes: i is 0
    (let ((n i))                     ;; a new n is bound to 0
      (setf *my-fn* (lambda () n)))
    (incf i)                         ;; i is 1
    (if (= i 8) (return))
    (let ((n i))                     ;; a new n is bound to 1
      (setf *my-fn* (lambda () n)))
    (incf i)                         ;; i is 2
    (if (= i 8) (return))
    (let ((n i))                     ;; a new n is bound to 2
      (setf *my-fn* (lambda () n)))
    (incf i)                         ;; i is 3
    (if (= i 8) (return))
    (let ((n i))                     ;; a new n is bound to 3
      (setf *my-fn* (lambda () n)))
    (incf i)                         ;; i is 4
    (if (= i 8) (return))
    (let ((n i))                     ;; a new n is bound to 4
      (setf *my-fn* (lambda () n)))
    (incf i)                         ;; i is 5
    (if (= i 8) (return))
    (let ((n i))                     ;; a new n is bound to 5
      (setf *my-fn* (lambda () n)))
    (incf i)                         ;; i is 6
    (if (= i 8) (return))
    (let ((n i))                     ;; a new n is bound to 6
      (setf *my-fn* (lambda () n)))
    (incf i)                         ;; i is 7
    (if (= i 8) (return))
    (let ((n i))                     ;; a new n is bound to 7
      (setf *my-fn* (lambda () n)))  ;; last definition of *my-fn* which will get used
    (incf i)                         ;; i is 8
    (if (= i 8) (return))))          ;; end of the dotimes
;; => nil

(funcall *my-fn*) ; => 7

In the correct example, two new watch function bindings are established in each of the 8 iterations. As in the example above, the let binds a new symbol n to the current value of the outer iteration variable i.

It is crucial to understand, that n is not bound to the symbol i, but to its value because the let evaluates i, before binding n to it.

Also it is crucial to understand that the different symbols n in the let forms above are actually different and are only valid and accessible within the body of the let in which they are bound (which is the main idea of let in the first place).

Last not least it should be mentioned that it is not uncommon, to use the same symbol for the outer iteration and the inner let like this:

;; using an array of 8 functions for clarity:

(defparameter *my-fns* #(nil nil nil nil nil nil nil nil))
;; => *my-fns*

(dotimes (n 8)
  (let ((n n))
    (setf (aref *my-fns* n) (lambda () n))))
;; => nil

(funcall (aref *my-fns* 0)) ; => 0
(funcall (aref *my-fns* 1)) ; => 1
(funcall (aref *my-fns* 4)) ; => 4
(funcall (aref *my-fns* 7)) ; => 7

This is exactly the same as before: In the let, a new symbol n gets bound to the value of the outer symbol n in every iteration. In the lexical scope (the body) of the let, the new binding of n takes precedence over the outer n, so referencing n in the aref and in the lambda expression will dereference the new binding established in the let. This is called shadowing, or, in other words, the inner n "shadows" the outer n.

Closures are such a powerful construct that they have become increasingly common in many programming languages and many books have been dedicated to their exploration². But as in real life: With power comes responsibility and it should be mentioned that this power can come at the price of an overwhelming complexity, so it should be handled with care.

Footnotes:

When such an error happens in practice, it is often quite puzzling what is at fault and such behaviour can be very hard to debug. Therefore it is extremely important to be aware of the reasons for such behaviour and to be alert when defining functions referencing variables outside of their scope, expecially in iterations.

For examples see Dan Graham's famous book "On Lisp" or Doug Hoyte's "Let over Lambda".