Kicking the Debugger habit

This post covers my experience of giving up breakpoint and step-through debugging.

My Domain

There are some interesting features of the domain I work in that have led me here. Financial analytics tend to compose many algorithms such as curve fitting, statistics, monte carlo and optimisation. The public API on the other hand is very simple: given this portfolio, return the risks associated.

A bug report is more likely to be 'this risk number looks a little odd' rather than 'an exception was thrown and here is the stack trace'. I can't agree with the idea that you should only unit test your public APIs. Rather, this should be: you need to unit test your public APIs, but if your domain is sufficiently complex, you also need to unit test internal modules. What is key is if a bug report queries an API result how quickly could you investigate and resolve any potential issue?

What's wrong with debugging?

In my field:

What's the alternative?

Since starting to use Expecto, I've been using the command line to run unit tests. Expecto does integrate with Visual Studio and can even do live unit testing in VS Code. I've found using a set of commands I've built up in FAKE to be more flexible and productive e.g.

    test integration
    test all
    test 64 debug --stress 2

Expecto encourages using normal code for organisation, setup & teardown and parameterisation of tests, instead of a limited framework of attributes.

Now apply this concept to debugging. With a debug module in the core of a codebase that is conditional on the debug configuration, code can be annotated with validation and some debug output. The command line records a history of the test results and validation output. Once complete, compiling in release ensures all diagnostic code is removed.

This started out as simple functions to printfn data being sequenced and piped, but expanded into functions to count calls, check for NaNs globally, serialize function inputs and outputs, test convergence of numbers etc. This is normal code and there is huge scope for adding conditional logic.

Conclusion

Kicking the debugger habit has given me a productivity boost. It forces me to think more logically about how I validate and break down a problem.

It also reduces the complexity of the tooling. Finding and fixing bugs feels more like coding and unit testing. I can use a simpler code editor plus the command line.

The result is I now have more confidence that once I've created the initial bug test I will be able to resolve it quickly.

Appendix

I've been asked for some sample code from the debug module. The code below should hopefully start to give an idea of what can be done.

//#if DEBUG
[<AutoOpen>]
module OverflowAndNaNCheck =
    open Checked

    let inline private isNaN v =
        match box v with | :? float as v -> Double.IsNaN v | _ -> false
    let inline (/) a b =
        let c = a/b in if isNaN c then failwithf "NaN found: %A / %A = %A" a b c else c
    let inline (+) a b =
        let c = a+b in if isNaN c then failwithf "NaN found: %A + %A = %A" a b c else c
    let inline (-) a b =
        let c = a-b in if isNaN c then failwithf "NaN found: %A - %A = %A" a b c else c
    let inline (*) a b =
        let c = a*b in if isNaN c then failwithf "NaN found: %A * %A = %A" a b c else c
    let inline ( ** ) a b =
        let c = a**b in if isNaN c then failwithf "NaN found: %A ** %A = %A" a b c else c
    let inline sqrt a =
        let c = sqrt a in if isNaN c then failwithf "NaN found: sqrt %A = %A" a c else c
    let inline log a =
        let c = log a in if isNaN c then failwithf "NaN found: log %A = %A" a c else c
    let inline log10 a =
        let c = log10 a in if isNaN c then failwithf "NaN found: log10 %A = %A" a c else c
    let inline asin a =
        let c = asin a in if isNaN c then failwithf "NaN found: asin %A = %A" a c else c
    let inline acos a =
        let c = acos a in if isNaN c then failwithf "NaN found: acos %A = %A" a c else c
    let inline atan a =
        let c = atan a in if isNaN c then failwithf "NaN found: atan %A = %A" a c else c

module Dbg =
    let private rand = Random()
    let mutable private randN = None
    type atRandom(n:int) =
        do
            randN <- Some n
        interface IDisposable with
            member __.Dispose() = randN <- None
    let write fmt =
        let sb =
            let n = DateTime.Now
            let sb = StringBuilder("DEBUG ")
            sb.Append(n.ToString("dd MMM HH:mm:ss.fffffff")) |> ignore
            sb.Append("> ") |> ignore
            sb
        Printf.kbprintf (fun () ->
            if Option.isNone randN || Option.get randN |> rand.Next = 0 then
                let old = Console.ForegroundColor
                try
                    Console.ForegroundColor <- ConsoleColor.Red
                    sb.ToString() |> Console.WriteLine
                finally
                    Console.ForegroundColor <- old
        ) sb fmt
    let writeIf condition fmt = if condition() then write fmt
    let runIf condition fn = if condition() then fn()
    let pipe fmt = fun a -> write fmt a; a
    let seq desc s =
        let s = Seq.cache s
        Seq.iter (write "%s: %A" desc) s
        s
    let fun1 desc (f:'a->'b) =
        fun a ->
            let b = f a
            write "%s - Input: %A\t\t Output: %A" desc a b; b
    let fun2 desc (f:'a->'b->'c) =
        fun a b ->
            let c = f a b
            write "%s - Input: %A\t\t Output: %A" desc (a,b) c; c
    let fun3 desc (f:'a->'b->'c->'d) =
        fun a b c ->
            let d = f a b c
            write "%s - Input: %A\t\t Output: %A" desc (a,b,c) d; d
    type counter(desc:string) =
        let mutable count = 0
        member __.Count = count
        member __.Increment() = count <- count + 1
        member inline m.Calls fn = fun a -> m.Increment(); fn a
        interface IDisposable with
            member __.Dispose() = write "%s count = %i" desc count
    let descendingChecker desc =
        let mutable last = infinity
        fun x ->
            if x>last then write "%s - should be descending but %A > %A" desc x last
            else last<-x
    let mutable private functionMap = Map.empty
    let addFun (key:string) (fn:unit->unit) = functionMap <- Map.add key fn functionMap
    let runFun (key:string) = Map.find key functionMap ()
//#endif
namespace System
namespace System.Text
Multiple items
type AutoOpenAttribute =
  inherit Attribute
  new : unit -> AutoOpenAttribute
  new : path:string -> AutoOpenAttribute
  member Path : string

--------------------
new : unit -> AutoOpenAttribute
new : path:string -> AutoOpenAttribute
Multiple items
module Checked

from Microsoft.FSharp.Core.Operators

--------------------
module Checked

from Microsoft.FSharp.Core.ExtraTopLevelOperators
val box : value:'T -> obj
Multiple items
val float : value:'T -> float (requires member op_Explicit)

--------------------
type float = Double

--------------------
type float<'Measure> = float
type Double =
  struct
    member CompareTo : value:obj -> int + 1 overload
    member Equals : obj:obj -> bool + 1 overload
    member GetHashCode : unit -> int
    member GetTypeCode : unit -> TypeCode
    member ToString : unit -> string + 3 overloads
    static val MinValue : float
    static val MaxValue : float
    static val Epsilon : float
    static val NegativeInfinity : float
    static val PositiveInfinity : float
    ...
  end
Double.IsNaN(d: float) : bool
val failwithf : format:Printf.StringFormat<'T,'Result> -> 'T
val sqrt : value:'T -> 'U (requires member Sqrt)
val log : value:'T -> 'T (requires member Log)
val log10 : value:'T -> 'T (requires member Log10)
val asin : value:'T -> 'T (requires member Asin)
val acos : value:'T -> 'T (requires member Acos)
val atan : value:'T -> 'T (requires member Atan)
Multiple items
type Random =
  new : unit -> Random + 1 overload
  member Next : unit -> int + 2 overloads
  member NextBytes : buffer:byte[] -> unit
  member NextDouble : unit -> float

--------------------
Random() : Random
Random(Seed: int) : Random
union case Option.None: Option<'T>
Multiple items
val int : value:'T -> int (requires member op_Explicit)

--------------------
type int = int32

--------------------
type int<'Measure> = int
union case Option.Some: Value: 'T -> Option<'T>
type IDisposable =
  member Dispose : unit -> unit
Multiple items
type DateTime =
  struct
    new : ticks:int64 -> DateTime + 10 overloads
    member Add : value:TimeSpan -> DateTime
    member AddDays : value:float -> DateTime
    member AddHours : value:float -> DateTime
    member AddMilliseconds : value:float -> DateTime
    member AddMinutes : value:float -> DateTime
    member AddMonths : months:int -> DateTime
    member AddSeconds : value:float -> DateTime
    member AddTicks : value:int64 -> DateTime
    member AddYears : value:int -> DateTime
    ...
  end

--------------------
DateTime ()
   (+0 other overloads)
DateTime(ticks: int64) : DateTime
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, millisecond: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, millisecond: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
property DateTime.Now: DateTime
Multiple items
type StringBuilder =
  new : unit -> StringBuilder + 5 overloads
  member Append : value:string -> StringBuilder + 19 overloads
  member AppendFormat : format:string * arg0:obj -> StringBuilder + 7 overloads
  member AppendLine : unit -> StringBuilder + 1 overload
  member Capacity : int with get, set
  member Chars : int -> char with get, set
  member Clear : unit -> StringBuilder
  member CopyTo : sourceIndex:int * destination:char[] * destinationIndex:int * count:int -> unit
  member EnsureCapacity : capacity:int -> int
  member Equals : sb:StringBuilder -> bool
  ...

--------------------
StringBuilder() : StringBuilder
StringBuilder(capacity: int) : StringBuilder
StringBuilder(value: string) : StringBuilder
StringBuilder(value: string, capacity: int) : StringBuilder
StringBuilder(capacity: int, maxCapacity: int) : StringBuilder
StringBuilder(value: string, startIndex: int, length: int, capacity: int) : StringBuilder
val ignore : value:'T -> unit
module Printf

from Microsoft.FSharp.Core
val kbprintf : continuation:(unit -> 'Result) -> builder:StringBuilder -> format:Printf.BuilderFormat<'T,'Result> -> 'T
module Option

from Microsoft.FSharp.Core
val isNone : option:'T option -> bool
val get : option:'T option -> 'T
type Console =
  static member BackgroundColor : ConsoleColor with get, set
  static member Beep : unit -> unit + 1 overload
  static member BufferHeight : int with get, set
  static member BufferWidth : int with get, set
  static member CapsLock : bool
  static member Clear : unit -> unit
  static member CursorLeft : int with get, set
  static member CursorSize : int with get, set
  static member CursorTop : int with get, set
  static member CursorVisible : bool with get, set
  ...
property Console.ForegroundColor: ConsoleColor
type ConsoleColor =
  | Black = 0
  | DarkBlue = 1
  | DarkGreen = 2
  | DarkCyan = 3
  | DarkRed = 4
  | DarkMagenta = 5
  | DarkYellow = 6
  | Gray = 7
  | DarkGray = 8
  | Blue = 9
  ...
field ConsoleColor.Red: ConsoleColor = 12
Console.WriteLine() : unit
   (+0 other overloads)
Console.WriteLine(value: string) : unit
   (+0 other overloads)
Console.WriteLine(value: obj) : unit
   (+0 other overloads)
Console.WriteLine(value: uint64) : unit
   (+0 other overloads)
Console.WriteLine(value: int64) : unit
   (+0 other overloads)
Console.WriteLine(value: uint32) : unit
   (+0 other overloads)
Console.WriteLine(value: int) : unit
   (+0 other overloads)
Console.WriteLine(value: float32) : unit
   (+0 other overloads)
Console.WriteLine(value: float) : unit
   (+0 other overloads)
Console.WriteLine(value: decimal) : unit
   (+0 other overloads)
Multiple items
val seq : sequence:seq<'T> -> seq<'T>

--------------------
type seq<'T> = Collections.Generic.IEnumerable<'T>
module Seq

from Microsoft.FSharp.Collections
val cache : source:seq<'T> -> seq<'T>
val iter : action:('T -> unit) -> source:seq<'T> -> unit
Multiple items
val string : value:'T -> string

--------------------
type string = String
val infinity : float
Multiple items
module Map

from Microsoft.FSharp.Collections

--------------------
type Map<'Key,'Value (requires comparison)> =
  interface IReadOnlyDictionary<'Key,'Value>
  interface IReadOnlyCollection<KeyValuePair<'Key,'Value>>
  interface IEnumerable
  interface IComparable
  interface IEnumerable<KeyValuePair<'Key,'Value>>
  interface ICollection<KeyValuePair<'Key,'Value>>
  interface IDictionary<'Key,'Value>
  new : elements:seq<'Key * 'Value> -> Map<'Key,'Value>
  member Add : key:'Key * value:'Value -> Map<'Key,'Value>
  member ContainsKey : key:'Key -> bool
  ...

--------------------
new : elements:seq<'Key * 'Value> -> Map<'Key,'Value>
val empty<'Key,'T (requires comparison)> : Map<'Key,'T> (requires comparison)
type unit = Unit
val add : key:'Key -> value:'T -> table:Map<'Key,'T> -> Map<'Key,'T> (requires comparison)
val find : key:'Key -> table:Map<'Key,'T> -> 'T (requires comparison)