Kicking the Debugger habit

This post covers my experience of giving up breakpoint and step-through debugging.

My Domain

There are some interesting features of the domain I work in that have led me here. Financial analytics tend to compose many algorithms such as curve fitting, statistics, monte carlo and optimisation. The public API on the other hand is very simple: given this portfolio, return the risks associated.

A bug report is more likely to be 'this risk number looks a little odd' rather than 'an exception was thrown and here is the stack trace'. I can't agree with the idea that you should only unit test your public APIs. Rather, this should be: you need to unit test your public APIs, but if your domain is sufficiently complex, you also need to unit test internal modules. What is key is if a bug report queries an API result how quickly could you investigate and resolve any potential issue?

What's wrong with debugging?

In my field:

What's the alternative?

Since starting to use Expecto, I've been using the command line to run unit tests. Expecto does integrate with Visual Studio and can even do live unit testing in VS Code. I've found using a set of commands I've built up in FAKE to be more flexible and productive e.g.

1: 
2: 
3: 
    test integration
    test all
    test 64 debug --stress 2

Expecto encourages using normal code for organisation, setup & teardown and parameterisation of tests, instead of a limited framework of attributes.

Now apply this concept to debugging. With a debug module in the core of a codebase that is conditional on the debug configuration, code can be annotated with validation and some debug output. The command line records a history of the test results and validation output. Once complete, compiling in release ensures all diagnostic code is removed.

This started out as simple functions to printfn data being sequenced and piped, but expanded into functions to count calls, check for NaNs globally, serialize function inputs and outputs, test convergence of numbers etc. This is normal code and there is huge scope for adding conditional logic.

Conclusion

Kicking the debugger habit has given me a productivity boost. It forces me to think more logically about how I validate and break down a problem.

It also reduces the complexity of the tooling. Finding and fixing bugs feels more like coding and unit testing. I can use a simpler code editor plus the command line.

The result is I now have more confidence that once I've created the initial bug test I will be able to resolve it quickly.

Appendix

I've been asked for some sample code from the debug module. The code below should hopefully start to give an idea of what can be done.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
26: 
27: 
28: 
29: 
30: 
31: 
32: 
33: 
34: 
35: 
36: 
37: 
38: 
39: 
40: 
41: 
42: 
43: 
44: 
45: 
46: 
47: 
48: 
49: 
50: 
51: 
52: 
53: 
54: 
55: 
56: 
57: 
58: 
59: 
60: 
61: 
62: 
63: 
64: 
65: 
66: 
67: 
68: 
//#if DEBUG
[<AutoOpen>]
module OverflowAndNaNCheck =
    open Checked

    let inline private isNaN v = match box v with | :? float as v -> Double.IsNaN v | _ -> false
    let inline (/) a b = let c = a/b in if isNaN c then failwithf "NaN found: %A / %A = %A" a b c else c
    let inline (+) a b = let c = a+b in if isNaN c then failwithf "NaN found: %A + %A = %A" a b c else c
    let inline (-) a b = let c = a-b in if isNaN c then failwithf "NaN found: %A - %A = %A" a b c else c
    let inline (*) a b = let c = a*b in if isNaN c then failwithf "NaN found: %A * %A = %A" a b c else c
    let inline ( ** ) a b = let c = a**b in if isNaN c then failwithf "NaN found: %A ** %A = %A" a b c else c
    let inline sqrt a = let c = sqrt a in if isNaN c then failwithf "NaN found: sqrt %A = %A" a c else c
    let inline log a = let c = log a in if isNaN c then failwithf "NaN found: log %A = %A" a c else c
    let inline log10 a = let c = log10 a in if isNaN c then failwithf "NaN found: log10 %A = %A" a c else c
    let inline asin a = let c = asin a in if isNaN c then failwithf "NaN found: asin %A = %A" a c else c
    let inline acos a = let c = acos a in if isNaN c then failwithf "NaN found: acos %A = %A" a c else c
    let inline atan a = let c = atan a in if isNaN c then failwithf "NaN found: atan %A = %A" a c else c

module Dbg =
    let private rand = Random()
    let mutable private randN = None
    type atRandom(n:int) =
        do
            randN <- Some n
        interface IDisposable with
            member __.Dispose() = randN <- None
    let write fmt =
        let sb =
            let n = DateTime.Now
            let sb = StringBuilder("DEBUG ")
            sb.Append(n.ToString("dd MMM HH:mm:ss.fffffff")) |> ignore
            sb.Append("> ") |> ignore
            sb
        Printf.kbprintf (fun () ->
            if Option.isNone randN || Option.get randN |> rand.Next = 0 then
                let old = Console.ForegroundColor
                try
                    Console.ForegroundColor <- ConsoleColor.Red
                    sb.ToString() |> Console.WriteLine
                finally
                    Console.ForegroundColor <- old
        ) sb fmt
    let writeIf condition fmt = if condition() then write fmt
    let runIf condition fn = if condition() then fn()
    let pipe fmt = fun a -> write fmt a; a
    let seq desc s =
        let s = Seq.cache s
        Seq.iter (write "%s: %A" desc) s
        s
    let fun1 desc (f:'a->'b) = fun a -> let b = f a in write "%s - Input: %A\t\t Output: %A" desc a b; b
    let fun2 desc (f:'a->'b->'c) = fun a b -> let c = f a b in write "%s - Input: %A\t\t Output: %A" desc (a,b) c; c
    let fun3 desc (f:'a->'b->'c->'d) = fun a b c -> let d = f a b c in write "%s - Input: %A\t\t Output: %A" desc (a,b,c) d; d
    type counter(desc:string) =
        let mutable count = 0
        member __.Count = count
        member __.Increment() = count <- count + 1
        member inline m.Calls fn = fun a -> m.Increment(); fn a
        interface IDisposable with
            member __.Dispose() = write "%s count = %i" desc count
    let descendingChecker desc =
        let mutable last = infinity
        fun x ->
            if x>last then write "%s - should be descending but %A > %A" desc x last
            else last<-x
    let mutable private functionMap = Map.empty
    let addFun (key:string) (fn:unit->unit) = functionMap <- Map.add key fn functionMap
    let runFun (key:string) = Map.find key functionMap ()
//#endif
namespace System
namespace System.Text
Multiple items
type AutoOpenAttribute =
  inherit Attribute
  new : unit -> AutoOpenAttribute
  new : path:string -> AutoOpenAttribute
  member Path : string

Full name: Microsoft.FSharp.Core.AutoOpenAttribute

--------------------
new : unit -> AutoOpenAttribute
new : path:string -> AutoOpenAttribute
Multiple items
module Checked

from Microsoft.FSharp.Core.ExtraTopLevelOperators

--------------------
module Checked

from Microsoft.FSharp.Core.Operators
val box : value:'T -> obj

Full name: Microsoft.FSharp.Core.Operators.box
Multiple items
val float : value:'T -> float (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.float

--------------------
type float = Double

Full name: Microsoft.FSharp.Core.float

--------------------
type float<'Measure> = float

Full name: Microsoft.FSharp.Core.float<_>
type Double =
  struct
    member CompareTo : value:obj -> int + 1 overload
    member Equals : obj:obj -> bool + 1 overload
    member GetHashCode : unit -> int
    member GetTypeCode : unit -> TypeCode
    member ToString : unit -> string + 3 overloads
    static val MinValue : float
    static val MaxValue : float
    static val Epsilon : float
    static val NegativeInfinity : float
    static val PositiveInfinity : float
    ...
  end

Full name: System.Double
Double.IsNaN(d: float) : bool
val failwithf : format:Printf.StringFormat<'T,'Result> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.failwithf
val sqrt : value:'T -> 'U (requires member Sqrt)

Full name: Microsoft.FSharp.Core.Operators.sqrt
val log : value:'T -> 'T (requires member Log)

Full name: Microsoft.FSharp.Core.Operators.log
val log10 : value:'T -> 'T (requires member Log10)

Full name: Microsoft.FSharp.Core.Operators.log10
val asin : value:'T -> 'T (requires member Asin)

Full name: Microsoft.FSharp.Core.Operators.asin
val acos : value:'T -> 'T (requires member Acos)

Full name: Microsoft.FSharp.Core.Operators.acos
val atan : value:'T -> 'T (requires member Atan)

Full name: Microsoft.FSharp.Core.Operators.atan
Multiple items
type Random =
  new : unit -> Random + 1 overload
  member Next : unit -> int + 2 overloads
  member NextBytes : buffer:byte[] -> unit
  member NextDouble : unit -> float

Full name: System.Random

--------------------
Random() : unit
Random(Seed: int) : unit
union case Option.None: Option<'T>
Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.int

--------------------
type int = int32

Full name: Microsoft.FSharp.Core.int

--------------------
type int<'Measure> = int

Full name: Microsoft.FSharp.Core.int<_>
union case Option.Some: Value: 'T -> Option<'T>
type IDisposable =
  member Dispose : unit -> unit

Full name: System.IDisposable
Multiple items
type DateTime =
  struct
    new : ticks:int64 -> DateTime + 10 overloads
    member Add : value:TimeSpan -> DateTime
    member AddDays : value:float -> DateTime
    member AddHours : value:float -> DateTime
    member AddMilliseconds : value:float -> DateTime
    member AddMinutes : value:float -> DateTime
    member AddMonths : months:int -> DateTime
    member AddSeconds : value:float -> DateTime
    member AddTicks : value:int64 -> DateTime
    member AddYears : value:int -> DateTime
    ...
  end

Full name: System.DateTime

--------------------
DateTime()
   (+0 other overloads)
DateTime(ticks: int64) : unit
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, millisecond: int) : unit
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, millisecond: int, kind: DateTimeKind) : unit
   (+0 other overloads)
property DateTime.Now: DateTime
Multiple items
type StringBuilder =
  new : unit -> StringBuilder + 5 overloads
  member Append : value:string -> StringBuilder + 18 overloads
  member AppendFormat : format:string * arg0:obj -> StringBuilder + 4 overloads
  member AppendLine : unit -> StringBuilder + 1 overload
  member Capacity : int with get, set
  member Chars : int -> char with get, set
  member Clear : unit -> StringBuilder
  member CopyTo : sourceIndex:int * destination:char[] * destinationIndex:int * count:int -> unit
  member EnsureCapacity : capacity:int -> int
  member Equals : sb:StringBuilder -> bool
  ...

Full name: System.Text.StringBuilder

--------------------
StringBuilder() : unit
StringBuilder(capacity: int) : unit
StringBuilder(value: string) : unit
StringBuilder(value: string, capacity: int) : unit
StringBuilder(capacity: int, maxCapacity: int) : unit
StringBuilder(value: string, startIndex: int, length: int, capacity: int) : unit
val ignore : value:'T -> unit

Full name: Microsoft.FSharp.Core.Operators.ignore
module Printf

from Microsoft.FSharp.Core
val kbprintf : continutation:(unit -> 'Result) -> builder:StringBuilder -> format:Printf.BuilderFormat<'T,'Result> -> 'T

Full name: Microsoft.FSharp.Core.Printf.kbprintf
module Option

from Microsoft.FSharp.Core
val isNone : option:'T option -> bool

Full name: Microsoft.FSharp.Core.Option.isNone
val get : option:'T option -> 'T

Full name: Microsoft.FSharp.Core.Option.get
type Console =
  static member BackgroundColor : ConsoleColor with get, set
  static member Beep : unit -> unit + 1 overload
  static member BufferHeight : int with get, set
  static member BufferWidth : int with get, set
  static member CapsLock : bool
  static member Clear : unit -> unit
  static member CursorLeft : int with get, set
  static member CursorSize : int with get, set
  static member CursorTop : int with get, set
  static member CursorVisible : bool with get, set
  ...

Full name: System.Console
property Console.ForegroundColor: ConsoleColor
type ConsoleColor =
  | Black = 0
  | DarkBlue = 1
  | DarkGreen = 2
  | DarkCyan = 3
  | DarkRed = 4
  | DarkMagenta = 5
  | DarkYellow = 6
  | Gray = 7
  | DarkGray = 8
  | Blue = 9
  ...

Full name: System.ConsoleColor
field ConsoleColor.Red = 12
Console.WriteLine() : unit
   (+0 other overloads)
Console.WriteLine(value: string) : unit
   (+0 other overloads)
Console.WriteLine(value: obj) : unit
   (+0 other overloads)
Console.WriteLine(value: uint64) : unit
   (+0 other overloads)
Console.WriteLine(value: int64) : unit
   (+0 other overloads)
Console.WriteLine(value: uint32) : unit
   (+0 other overloads)
Console.WriteLine(value: int) : unit
   (+0 other overloads)
Console.WriteLine(value: float32) : unit
   (+0 other overloads)
Console.WriteLine(value: float) : unit
   (+0 other overloads)
Console.WriteLine(value: decimal) : unit
   (+0 other overloads)
Multiple items
val seq : sequence:seq<'T> -> seq<'T>

Full name: Microsoft.FSharp.Core.Operators.seq

--------------------
type seq<'T> = Collections.Generic.IEnumerable<'T>

Full name: Microsoft.FSharp.Collections.seq<_>
module Seq

from Microsoft.FSharp.Collections
val cache : source:seq<'T> -> seq<'T>

Full name: Microsoft.FSharp.Collections.Seq.cache
val iter : action:('T -> unit) -> source:seq<'T> -> unit

Full name: Microsoft.FSharp.Collections.Seq.iter
Multiple items
val string : value:'T -> string

Full name: Microsoft.FSharp.Core.Operators.string

--------------------
type string = String

Full name: Microsoft.FSharp.Core.string
val infinity : float

Full name: Microsoft.FSharp.Core.Operators.infinity
Multiple items
module Map

from Microsoft.FSharp.Collections

--------------------
type Map<'Key,'Value (requires comparison)> =
  interface IEnumerable
  interface IComparable
  interface IEnumerable<KeyValuePair<'Key,'Value>>
  interface ICollection<KeyValuePair<'Key,'Value>>
  interface IDictionary<'Key,'Value>
  new : elements:seq<'Key * 'Value> -> Map<'Key,'Value>
  member Add : key:'Key * value:'Value -> Map<'Key,'Value>
  member ContainsKey : key:'Key -> bool
  override Equals : obj -> bool
  member Remove : key:'Key -> Map<'Key,'Value>
  ...

Full name: Microsoft.FSharp.Collections.Map<_,_>

--------------------
new : elements:seq<'Key * 'Value> -> Map<'Key,'Value>
val empty<'Key,'T (requires comparison)> : Map<'Key,'T> (requires comparison)

Full name: Microsoft.FSharp.Collections.Map.empty
type unit = Unit

Full name: Microsoft.FSharp.Core.unit
val add : key:'Key -> value:'T -> table:Map<'Key,'T> -> Map<'Key,'T> (requires comparison)

Full name: Microsoft.FSharp.Collections.Map.add
val find : key:'Key -> table:Map<'Key,'T> -> 'T (requires comparison)

Full name: Microsoft.FSharp.Collections.Map.find