F# all the things

Interacting with devices using MQTT

2022-12-16T00:00:00+00:00

This is part of Sergey Thion’s awesome F# advent calendar. Thanks for organizing one again!

Code related to this post is located in the FsAdvent2022 branch in this repository.

In this post we’re going to try to interact with the smart home MQTT API for FutureHome. Fortunately for us, someone has already created a MQTT client library for us, available here. And someone has been so kind as to extend it with observables, instead of having to deal with the utterly useless thing called events. That library can be found here.

FIMP

Is the Futurehome IoT Messaging Protocol. The documentation can be found here. It’s a bit tricky to figure out how it works but we’ll give it a shot anyway. Some examples can be found in their Go implementation here which has been used to figure out what the different request content should be for certain things.

FIMP message format

Messages sent using FIMP are JSON messages containing the following properties (see here for original docs):

Property	Type	Required	Description
corid	String	No	Message correlation id. Used for request - response matching.
ctime	String	Yes	Message creation time, e.g. `"2019-05-31 17:36:31 +0200"`
props	Map<String, String>	Yes	Map of properties.
resp_to	String	No*	Response topic where requester will expect to receive response.
serv	String	Yes	Service name the interface is part of.
src	String	Yes	Source or of the message, should be set only for commands.
tags	List	No	List of tags.
type	String	Yes	Interface type, defines message format.
uid	String	Yes	Unique message identifier.
val	dynamic	Yes	“payload” - type is defined by `val_t`.
val_t	String	Yes	Data format of `val` field. See below.
ver	String	Yes	Version of the message format, default: `"1"`.

Value Types

Since val can be any type, val_t defines what type it is. List of supported val types:

`val_t`	Sample `val`
string	`'Hello world!'`
int	`3`
float	`3.1415`
bool	`true`
null	`null`
str_array	`['hello, 'world']`
int_array	`[0, 1, 1, 2, 3, 5, 8, 13]`
float_array	`[3.14, 2.71]`
int_map	`{"answer": 42}`
str_map	`{"ip": "192.168.1.1"}`
float_map	`{"pi: 3.14"}`
bool_map	`{"normalityRestored": true}`
object*	`{"nested": {"objects": "supported"}}`
base64	`U28gbG9uZywgYW5kIHRoYW5rcyBmb3IgYWxsIHRoZSBmaXNoLg==`

Creating the object type

All the possible types of val_t are pretty simple except the object type. After some digging around here it seems it look something like this:

{
    "cmd": "delete",
    "component": "room",
    "id": 1,
    "param": {},
    "requestId": 7294000000006
}

where param could look like:

"param": {
    "components": [
        "thing",
        "device",
        "room",
        "mode",
        "shortcut"
    ]
}

So let’s start off with the possible commands. We’re going to use Thoth.Json.Net to encode our messages, like this:

[<RequireQualifiedAccess>]
type Cmd =
    | Get
    | Set
    | Delete
    | Edit

module Cmd =
    let encode (value: Cmd) : JsonValue =
        match value with
        | Cmd.Get -> Encode.string "get"
        | Cmd.Set -> Encode.string "set"
        | Cmd.Delete -> Encode.string "delete"
        | Cmd.Edit -> Encode.string "edit"

Next we have components, which can be any of the following:

[<RequireQualifiedAccess>]
type Component =
    | Config
    | Device
    | House
    | Hub
    | Mode
    | Room
    | Service
    | Shortcut
    | Thing

module Component =
    let encode (value: Component) : JsonValue =
        match value with
        | Component.Config -> Encode.string "config"
        | Component.Device -> Encode.string "device"
        | Component.House -> Encode.string "house"
        | Component.Hub -> Encode.string "hub"
        | Component.Mode -> Encode.string "mode"
        | Component.Room -> Encode.string "room"
        | Component.Service -> Encode.string "service"
        | Component.Shortcut -> Encode.string "shortcut"
        | Component.Thing -> Encode.string "thing"

The id can be things like room-id or house mode (home, sleep etc.):

[<RequireQualifiedAccess>]
type ObjectRequestId =
    | DeviceId of int
    | Mode of Mode
    | FireAlarm of Enabled: bool * Supported: bool

module ObjectRequestId =
    let encode (value: ObjectRequestId) : JsonValue =
        match value with
        | ObjectRequestId.DeviceId v -> Encode.int v
        | ObjectRequestId.Mode m -> Mode.encode m
        | ObjectRequestId.FireAlarm (isEnabled, isSupported) ->
            Encode.object [
                "enabled", Encode.bool isEnabled
                "supported", Encode.bool isSupported
            ]

And finally we have the Object type:

type ObjectVal =
    { Cmd: Cmd
      Component: Component option
      Id: ObjectRequestId option
      Param: JsonValue option }

module ObjectVal =
    let create cmd component' id param =
        { Cmd = cmd
          Component = component'
          Id = id
          Param = param }

    let encode (value: ObjectVal) =
        Encode.object [
            "cmd", Cmd.encode value.Cmd

            match value.Component with
            | Some c ->
                "component", Component.encode c
            | None ->
                "component", Encode.nil

            match value.Id with
            | Some id ->
                "id", ObjectRequestId.encode id
            | None ->
                "id", Encode.nil

            match value.Param with
            | Some p ->
                "param", p
            | None ->
                "param", Encode.nil
        ]

The Val type

Now that we have the object type, we can create the val type:

[<RequireQualifiedAccess>]
type Val =
    | String of string
    | Int of int
    | Float of float
    | Bool of bool
    | Null
    | Str_array of string array
    | Int_array of int array
    | Float_array of float array
    | Int_map of Map<string, int>
    | Str_map of Map<string, string>
    | Float_map of Map<string, float>
    | Bool_map of Map<string, bool>
    | Object of ObjectVal
    | Base64 of string

module Val =

    let encode (value: Val) =
        match value with
        | Val.String x -> ["val_t", Encode.string "string"; "val", Encode.string x]
        | Val.Int x -> ["val_t", Encode.string "int"; "val", Encode.int x]
        | Val.Float x -> ["val_t", Encode.string "float"; "val", Encode.float x]
        | Val.Bool x -> ["val_t", Encode.string "bool"; "val", Encode.bool x]
        | Val.Null -> ["val_t", Encode.string "null"; "val", Encode.string "null"]
        | Val.Str_array xs -> ["val_t", Encode.string "string_array"; "val", xs |> Array.map Encode.string |> Encode.array]
        | Val.Int_array xs -> ["val_t", Encode.string "int_array"; "val", xs |> Array.map Encode.int |> Encode.array]
        | Val.Float_array xs -> ["val_t", Encode.string "float_array"; "val", xs |> Array.map Encode.float |> Encode.array]
        | Val.Int_map x -> ["val_t", Encode.string "int_map"; "val", x |> Map.map (fun _ v -> Encode.int v) |> Encode.dict]
        | Val.Str_map x -> ["val_t", Encode.string "str_map"; "val", x |> Map.map (fun _ v -> Encode.string v) |> Encode.dict]
        | Val.Float_map x -> ["val_t", Encode.string "float_map"; "val", x |> Map.map (fun _ v -> Encode.float v) |> Encode.dict]
        | Val.Bool_map x -> ["val_t", Encode.string "bool_map"; "val", x |> Map.map (fun _ v -> Encode.bool v) |> Encode.dict]
        | Val.Object x -> ["val_t", Encode.string "object"; "val", ObjectVal.encode x]
        | Val.Base64 x -> ["val_t", Encode.string "base64"; "val", Encode.string x]

The message type

Finally we can create the FIMP message type that has to be sent as content to the server.

// Modules not included for brevity
type CorId = CorId of CorrelationId: string
type Ctime = Ctime of CreationTime: DateTime
type Props = Props of Properties: Map<string, string>
type RespTo = RespTo of ResponseTopic: string
type Serv = Serv of ServiceName: string
type Src = Src of string
type TagsList = TagsList of string list
type Type = Type of InterfaceType: string
type Uid = Uid of MessageIdentifier: Guid
type Ver = private Ver of string

type Message =
    { CorId: CorId option
      Ctime: Ctime
      Props: Props
      RespTo: RespTo option
      Serv: Serv
      Src: Src
      Tags: TagsList list option
      Type: Type
      Uid: Uid
      Val: Val
      Ver: Ver }

module Message =

    let create ctime props serv src interfaceType uid value =
        { CorId = None
          Ctime = ctime
          Props = props
          RespTo = None
          Serv = serv
          Src = src
          Tags = None
          Type = interfaceType
          Uid = uid
          Val = value
          Ver = Ver.defaultVer }

    let createTimeStamped props serv src interfaceType value =
        create
            (Ctime.create DateTime.Now)
            props
            serv
            src
            interfaceType
            (Uid.newUid())
            value

    let withCorrelationId corId msg =
        { msg with CorId = Some corId }

    let withResponseTopic topic msg =
        { msg with RespTo = Some topic }

    let withTags tags msg =
        { msg with Tags = Some tags }

    let private encodeOpt (f: 'a -> JsonValue) (value: 'a option) =
        value
        |> Option.map f
        |> Option.defaultValue Encode.nil

    let encode (msg: Message) =
        Encode.object [
            "corid", msg.CorId |> encodeOpt CorId.encode
            "ctime", msg.Ctime |> Ctime.encode
            "props", msg.Props |> Props.encode
            "resp_to", msg.RespTo |> encodeOpt RespTo.encode
            "serv", msg.Serv |> Serv.encode
            "src", msg.Src |> Src.encode
            "tags", msg.Tags |> encodeOpt TagsList.encode
            "type", msg.Type |> Type.encode
            "uid", msg.Uid |> Uid.encode
            yield! msg.Val |> Val.encode
            "ver", msg.Ver |> Ver.encode
        ]

Creating the MQTT client

Here we’re using MQTTnet.Extensions.External.RxMQTT.Client to create a client we can interact with using observables.

module MqttClient =
    open MQTTnet.Extensions.External.RxMQTT.Client

    type ClientId = ClientId of string
    type Topic = Topic of string

    type TcpServer =
        { Url: string
          Port: int }

    type Credentials =
        { UserName: string
          Password: string }

    // ClientId -> TcpServer -> Credentials -> Task<IRxMqttClient>
    let create (ClientId clientId) (server: TcpServer) (credentials: Credentials) = task {
        let options =
            ManagedMqttClientOptionsBuilder()
                .WithAutoReconnectDelay(TimeSpan.FromSeconds(5))
                .WithClientOptions(MqttClientOptionsBuilder()
                    .WithProtocolVersion(MqttProtocolVersion.V311)
                    .WithClientId(clientId)
                    .WithTcpServer(server.Url, server.Port)
                    .WithCredentials(credentials.UserName, credentials.Password)
                    .Build())
                .Build();

        let mqttClient = MqttFactory().CreateRxMqttClient()
        do! mqttClient.StartAsync options
        return mqttClient
    }

    // IRxMqttClient -> Topic -> IObservable<MqttApplicationMessageReceivedEventArgs>
    let createSubscription (mqttClient: IRxMqttClient) (Topic topic) =
        mqttClient.Connect(topic)
    
    // Topic -> Message -> MqttApplicationMessage
    let createMessage (Topic topic) (message: Message) =
        MqttApplicationMessageBuilder()
            .WithTopic(topic)
            .WithPayload(message |> Message.encode |> Encode.toString 0)
            .WithQualityOfServiceLevel(MQTTnet.Protocol.MqttQualityOfServiceLevel.ExactlyOnce)
            .WithRetainFlag()
            .Build()

Listing all devices

An example request can be seen here. Let’s try to create the request message:

module Things =

    let requestTopic = RequestTopic.create "pt:j1/mt:cmd/rt:app/rn:vinculum/ad:1"

    // We can select the `ad` for the response
    let responseTopic = ResponseTopic.create "pt:j1/mt:rsp/rt:app/rn:vinculum/ad:things"

    // Val -> Message
    let private createDefaultMessage =
        Message.create
            (Ctime.create DateTime.Now)
            (Props.create List.empty)
            Serv.Vinculum
            src
            (Type.create "cmd.pd7.request") // Not sure what pd7 is, but copied from their example.
            (Uid.newUid())

    // Val -> Message
    let createMessage value =
        value
        |> createDefaultMessage
        |> Message.withResponseTopic responseTopic

    // Component list -> Val
    let private encodeComponents items =
        Encode.object [
            "components",
            items
            |> List.map Component.encode
            |> Encode.list
        ]
        |> Some
        |> ObjectVal.create Cmd.Get None None
        |> Val.Object

    // Val
    let listDevices =
        [
            Component.Device
        ]
        |> encodeComponents

Interacting with the API

The MqttClient we’re using returns responses using observables. If we want to get the list of devices in a request/response fashion, we have to leave the monad by converting the observable to a Task.

let getAsync (mqttClient: IRxMqttClient) requestTopic responseTopic message =
    task {    
        let respObs =
            responseTopic // The response topic we defined above
            |> MqttClient.createSubscription mqttClient
            // We want the first message and the observable to complete
            // so the task below will finish/return.
            |> Observable.first

        // Subscribe/start as task before sending the request
        let response = respObs.ToTask()

        do! message
            |> MqttClient.createMessage requestTopic
            |> mqttClient.PublishAsync

        return! response
    }

/// Gets all available devices
let getAllDevices (mqttClient: IRxMqttClient) = task {
    let! response =
        Things.listDevices
        |> Things.createMessage
        |> getAsync mqttClient Things.requestTopic Things.responseTopic

    return response.ApplicationMessage.Payload.ToUTF8String()
    }

This is an example response for a motion sensor device:

...
{
    "client": {
        "name": "Motion sensor"
    },
    "fimp": {
        "adapter": "zwave-ad",
        "address": "19",
        "group": "ch_0"
    },
    "functionality": null,
    "id": 36,
    "lrn": true,
    "model": "zw_271_2049_4097",
    "param": {
        "alarms": {},
        "batteryLevel": "ok",
        "batteryPercentage": 100,
        "illuminance": 1.0,
        "presence": false,
        "supportedAlarms": {
            "burglar": [
                "inactive",
                "tamper_removed_cover"
            ]
        },
        "temperature": 23.0,
        "timestamp": "2022-11-10 22:20:14 +0100",
        "zwaveConfigParameters": []
    },
    "problem": false,
    "room": 6,
    "services": {
        "alarm_burglar": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:alarm_burglar/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.alarm.get_report",
                "evt.alarm.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true,
                "sup_events": [
                    "inactive",
                    "tamper_removed_cover"
                ]
            }
        },
        "basic": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:basic/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.lvl.get_report",
                "cmd.lvl.set",
                "evt.lvl.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true
            }
        },
        "battery": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:battery/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.lvl.get_report",
                "evt.alarm.report",
                "evt.lvl.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true
            }
        },
        "dev_sys": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:dev_sys/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.config.get_report",
                "cmd.config.set",
                "cmd.group.add_members",
                "cmd.group.delete_members",
                "cmd.group.get_members",
                "cmd.ping.send",
                "evt.config.report",
                "evt.group.members_report",
                "evt.ping.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true
            }
        },
        "sensor_accelx": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:sensor_accelx/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.sensor.get_report",
                "evt.sensor.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true,
                "sup_units": [
                    "m/s2"
                ]
            }
        },
        "sensor_accely": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:sensor_accely/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.sensor.get_report",
                "evt.sensor.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true,
                "sup_units": [
                    "m/s2"
                ]
            }
        },
        "sensor_accelz": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:sensor_accelz/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.sensor.get_report",
                "evt.sensor.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true,
                "sup_units": [
                    "m/s2"
                ]
            }
        },
        "sensor_lumin": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:sensor_lumin/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.sensor.get_report",
                "evt.sensor.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true,
                "sup_units": [
                    "Lux"
                ]
            }
        },
        "sensor_presence": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:sensor_presence/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.presence.get_report",
                "evt.presence.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true
            }
        },
        "sensor_seismicint": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:sensor_seismicint/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.sensor.get_report",
                "evt.sensor.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true,
                "sup_units": [
                    "MERC"
                ]
            }
        },
        "sensor_temp": {
            "addr": "/rt:dev/rn:zw/ad:1/sv:sensor_temp/ad:19_0",
            "enabled": true,
            "intf": [
                "cmd.sensor.get_report",
                "evt.sensor.report"
            ],
            "props": {
                "is_secure": false,
                "is_unsecure": true,
                "sup_units": [
                    "C"
                ],
                "thing_role": "main"
            }
        }
    },
    "supports": [
        "clear",
        "poll"
    ],
    "thing": 13,
    "type": {
        "subtype": "presence",
        "supported": {
            "sensor": [
                "presence"
            ]
        },
        "type": "sensor"
    }
}
...

Interaction between devices

Now that we know which devices we have, their features and topics, we can try to make some interaction between them. We have a Fibaro motion sensor and a Philips Hue Go lamp we can play with. First we need some requests for turning on lights and setting color.

module FsFimp.Commands

open System
open FsFimp.Fimp

let app = "fsfimp"
let src = Src.create app

module Units =
    type [<Measure>] Red
    type [<Measure>] Green
    type [<Measure>] Blue

type LevelSwitch = On | Off

module LevelSwitch =
    let interfaceType = Type.create "cmd.binary.set"
    let service = Serv.OutLevelSwitch

    let createMessage (toggle: LevelSwitch) =
        match toggle with
        | On -> true
        | Off -> false
        |> Val.Bool
        |> Message.createTimeStamped
            Props.empty
            service
            src
            interfaceType

type Red = int<Units.Red>
type Green = int<Units.Green>
type Blue = int<Units.Blue>

module Color =
    let interfaceType = Type.create "cmd.color.set"
    let service = Serv.ColorControl

    let createMessage (red: Red) (green: Green) (blue: Blue) =
        seq {
            "red", int red
            "green", int green
            "blue", int blue
        }
        |> Map.ofSeq
        |> Val.Int_map
        |> Message.createTimeStamped
            Props.empty
            service
            src
            interfaceType

Next we have to listen to events from the motion sensor. We’ll turn on the lights when presence is detected and make it red when the burglar event is triggered (e.g. when shaking the sensor). The necessary topics were found from the getAllDevices response above.

use! mqttClient = MqttClient.create (ClientId "FsFimp") server credentials
use _ =
    mqttClient
        .ConnectingFailedEvent
        .Subscribe(fun msg ->
            printfn "%A, %A" msg.ConnectResult msg.Exception.Message)

let send (msg: MQTTnet.MqttApplicationMessage) : Task<unit> =
    task {
         // This returns Task, but we need Task<unit>, hence the CE.
         // There might be a simpler way to do this though, but I 
         // couldn't figure it out in time.
        return! mqttClient.PublishAsync msg
    }

let turnOnLightOnPresence =
    "pt:j1/mt:evt/rt:dev/rn:zw/ad:1/sv:sensor_presence/ad:19_0"
    |> ResponseTopic.create
    |> MqttClient.createSubscription mqttClient
    |> Observable.map (fun msg ->
        let requestTopic =
            "pt:j1/mt:cmd/rt:dev/rn:hue/ad:1/sv:out_lvl_switch/ad:l14_0"
            |> RequestTopic.create

        LevelSwitch.createMessage LevelSwitch.On
        |> MqttClient.createMessage requestTopic)

let makeLightRedOnBurglar =
    "pt:j1/mt:evt/rt:dev/rn:zw/ad:1/sv:alarm_burglar/ad:19_0"
    |> ResponseTopic.create
    |> MqttClient.createSubscription mqttClient
    |> Observable.map (fun msg ->
        let requestTopic =
            "pt:j1/mt:cmd/rt:dev/rn:hue/ad:1/sv:color_ctrl/ad:l14_0"
            |> RequestTopic.create

        Color.createMessage 255<Units.Red> 0<Units.Green> 0<Units.Blue>
        |> MqttClient.createMessage requestTopic)

// Merge the two streams into one and send the messages when the different
// events are reported by the motion sensor.
use _ =
    Observable.merge turnOnLightOnPresence makeLightRedOnBurglar
    |> Observable.flatmapTask send
    |> Observable.subscribe id

Next steps

We’d want to decode the devices response and capture the different services each device provides. We could do something like this:

type Service =
    | Battery of ResponseTopic
    | BurglarAlarm of ResponseTopic
    | Luminance of ResponseTopic
    | Presence of ResponseTopic
    | Temperature of ResponseTopic

module Service =
    let decode : Decoder<Service list> =
        Decode.object (fun get ->
            [
                get.Optional.At [ "battery"; "addr" ] Decode.string |> Option.map (ResponseTopic.createFromResourceType >> Battery)
                get.Optional.At [ "alarm_burglar"; "addr" ] Decode.string |> Option.map (ResponseTopic.createFromResourceType >> BurglarAlarm)
                get.Optional.At [ "sensor_lumin"; "addr" ] Decode.string |> Option.map (ResponseTopic.createFromResourceType >> Luminance)
                get.Optional.At [ "sensor_presence"; "addr" ] Decode.string |> Option.map (ResponseTopic.createFromResourceType >> Presence)
                get.Optional.At [ "sensor_temp"; "addr" ] Decode.string |> Option.map (ResponseTopic.createFromResourceType >> Temperature)
            ]
            |> List.choose id)

type Device =
    { Id: int
      Room: int option
      Model: string option
      Services: Service list }

module Device =
    let decoder : Decoder<Device> =
        Decode.object (fun get ->
            let id = get.Required.Field "id" Decode.int
            let room = get.Optional.Field "room" Decode.int

            let model = get.Optional.At [ "model" ] Decode.string
            let modelAlias = get.Optional.At [ "modelAlias" ] Decode.string

            let modelName =
                match model, modelAlias with
                | Some model, _ -> Some model
                | _, Some alias -> Some alias
                | _, _ -> None

            let services = get.Required.Field "services" Service.decode

            { Id = id
              Room = room
              Model = modelName
              Services = services })

    let devicesDecoder : Decoder<Device list> =
        Decode.object (fun get ->
            get.Required.At [ "val"; "param"; "device" ] (Decode.list decoder))

    let decodeAll json = Decode.fromString devicesDecoder json

Unfortunately, that’s all I had time for in this post. Next steps would be to try and add interaction with the lights to my smart house dashboard, written using Fable, where I already have weather forecast, netatmo weather station data and today’s electricity prices. But that’s a post for another time.

Exploring your crypto with F#

2021-12-11T00:00:00+00:00

This is part of F# Advent calendar 2021. Go and check out all the other great posts and thank you Sergey Tihon for organizing!

In this post we’re going to create a console application for exploring your crypto assets. The code can be found here.

Exploring the APIs

Let’s start off by exploring the API of a crypto exchange using .NET interactive notebooks in VSCode. If you haven’t already, install the VSCode extension now:

To create a new notebook, press Ctrl + Shift + p and choose the following:

In the first cell we want to include some NuGet packages to work with:

#r "nuget: Plotly.NET, 2.0.0-preview.11"
#r "nuget: Plotly.NET.Interactive, 2.0.0-preview.11"
#r "nuget: FSharp.Data"

With the cell in focus, press Shift + Enter to run it and create a new cell below. Here we we open some namespaces/modules:

open FSharp.Data
open Plotly.NET

Exploring the FIRI exchange’s API

Let’s look at a Norwegian exchange with a pretty simple API, Firi (previously known as MiraiEx). The API documentation can be found here.

There are some public API’s we could easily check using the JSON provider. Put the following into a cell in the notebook:

let [<Literal>] FiriUri = "https://api.miraiex.com/v2/markets"
type FiriMarkets = JsonProvider<FiriUri>
let markets = FiriMarkets.Load FiriUri

markets
|> Array.map (fun m -> (m.Id, m.Low, m.Last, m.High))

It will display something like this:

Next we could create a plot of some historic prices by choosing one of the market Ids loaded above.

let [<Literal>] HistoryUri = "https://api.miraiex.com/v2/markets/ETHNOK/history"
type FiriMarkets = JsonProvider<HistoryUri>
let history = FiriMarkets.Load $"https://api.miraiex.com/v2/markets/{markets.[0].Id}/history"

Chart.Line(
    history |> Array.map (fun h -> h.CreatedAt.UtcDateTime),
    history |> Array.map (fun h -> h.Price),
    ShowMarkers=true)
|> Chart.withTitle $"Price history {markets.[0].Id}"
|> Chart.withXAxisStyle ("Date")
|> Chart.withYAxisStyle ("Price")

That’s all fine for the public API’s, but the interesting ones are the private. For that we have to work a bit harder.

Let’s try to make a typed API

First, let’s define a list of some of their endpoints:

type OrderType =
    | Bid
    | Ask
    member this.AsString =
        match this with
        | Ask -> "Ask"
        | Bid -> "Bid"

type Order =
    { Market: string
      Type: OrderType
      Price: double
      Amount: double }
    static member Create market orderType price amount =
        { Market = market
          Type = orderType
          Price = price
          Amount = amount }

[<RequireQualifiedAccess>]
type Endpoint =
    | ListMarkets
    | ListBalances
    | ListTransactions
    | ListTrades
    | ListDepositHistory
    | ListOrders
    | CreateOrder of Order
    | DeleteAllOrders
with
    member this.Uri =
        match this with
        | ListMarkets -> "markets"
        | ListBalances -> "balances"
        | ListTransactions -> "history/transactions"
        | ListTrades -> "history/trades"
        | ListDepositHistory -> "deposit/history"
        | ListOrders -> "history/orders"
        | CreateOrder _
        | DeleteAllOrders -> "orders"
    member this.HttpMethod =
        match this with
        | ListMarkets
        | ListBalances
        | ListTransactions
        | ListTrades
        | ListDepositHistory
        | ListOrders ->
            HttpMethod.Get
        | CreateOrder _ ->
            HttpMethod.Post
        | DeleteAllOrders ->
            HttpMethod.Delete
    member this.Body =
        match this with
        | ListMarkets
        | ListBalances
        | ListTransactions
        | ListTrades
        | ListDepositHistory
        | ListOrders
        | DeleteAllOrders ->
            None
        | CreateOrder order ->
            Encode.object
                [
                    "market", Encode.string order.Market
                    "type", Encode.string order.Type.AsString
                    "price", Encode.string (string order.Price)
                    "amount", Encode.string (string order.Amount)
                ]
            |> Encode.toString 0
            |> Some

We also need some types to deserialize the responses into, using Thoth.Json:

open Thoth.Json.Net

type Balance =
    { Currency: string
      Balance: float
      Hold: float
      Available: float }
    static member Decoder : Decoder<Balance> =
        Decode.object
            (fun get ->
                { Currency = get.Required.Field "currency" Decode.string
                  Balance = get.Required.Field "balance" Decode.string |> float
                  Hold = get.Required.Field "hold" Decode.string |> float
                  Available = get.Required.Field "available" Decode.string |> float })

type Transaction =
    { Id: Guid
      Amount: decimal
      Currency: string
      Type: string
      Date: DateTimeOffset
      Details: TransactionDetails }
    static member Decoder : Decoder<Transaction> =
        Decode.object
            (fun get ->
                { Id = get.Required.Field "id" Decode.guid
                  Amount = get.Required.Field "amount" Decode.string |> decimal
                  Currency = get.Required.Field "currency" Decode.string
                  Type = get.Required.Field "type" Decode.string
                  Date = get.Required.Field "date" Decode.datetimeOffset
                  Details = get.Required.Field "details" TransactionDetails.Decoder })
and TransactionDetails =
    { MatchId: string option
      DepositId: string option
      DepositAddress: string option
      DepositTxid: string option
      WithdrawId: string option
      WithdrawAddress: string option
      WithdrawTxid: string option }
    static member Decoder : Decoder<TransactionDetails> =
        Decode.object
            (fun get ->
                { MatchId = get.Optional.Field "match_id" Decode.string
                  DepositId = get.Optional.Field "deposit_id" Decode.string
                  DepositAddress = get.Optional.Field "deposit_address" Decode.string
                  DepositTxid = get.Optional.Field "deposit_txid" Decode.string
                  WithdrawId = get.Optional.Field "withdraw_id" Decode.string
                  WithdrawAddress = get.Optional.Field "withdraw_address" Decode.string
                  WithdrawTxid = get.Optional.Field "withdraw_txid" Decode.string })

//...not listing them all here

What about private endpoints?

To access private endpoints we need to create a ClientId and Secret in the Firi settings page and store them in a safe place. Then we have to compute the signature as specified here:

We need a function for computing the hmac hash and return it as lower-case hex:

open System
open System.Text
open System.Security.Cryptography

let computeHash (secret:string) (data:string) =
    use hmacsha256 =
        secret
        |> Encoding.UTF8.GetBytes
        |> fun bs -> new HMACSHA256(bs)

    data
    |> Encoding.UTF8.GetBytes
    |> hmacsha256.ComputeHash
    |> Array.map (fun (x : byte) -> sprintf "%02x" x)
    |> String.concat String.Empty

Now we can use this hashing function to compute the Firi signature:

let computeSignature secret =
    // Oh-noes! Side-effect! Pass it in instead if you want a pure function
    let timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds() |> string
    let validity = "2000"

    // Use Thoth.Json.Net to encode the JSON content
    let body =
        Encode.object
            [
                "timestamp", Encode.string timestamp
                "validity", Encode.string validity
            ]
        |> Encode.toString 0

    let signature = computeHash secret body
    let queryParams = $"?timestamp={timestamp}&validity={validity}"
    (signature, queryParams)

For each message we have to compute this hash and add the query parameters:

type FiriSecrets =
    { ClientId: string
      Secret: string }
    static member Create clientId secret =
        { ClientId = clientId
          Secret = secret }

let private createRequestMessage (secrets:FiriSecrets) (endpoint:Endpoint) =

    let (signature, queryParams) = getSignature secrets.Secret
    let requestUri = $"{endpoint.Uri}{queryParams}"

    let requestMessage = new HttpRequestMessage(endpoint.HttpMethod, requestUri)
    requestMessage.Headers.Add("miraiex-user-clientid", secrets.ClientId);
    requestMessage.Headers.Add("miraiex-user-signature", signature);

    endpoint.Body
    |> Option.iter (fun body ->
        let content = new StringContent(body, Text.Encoding.UTF8, "application/json")
        requestMessage.Content <- content)

    requestMessage

let send (httpClient:HttpClient) (secrets:FiriSecrets) (endpoint:Endpoint) =
    task { // Let's try the new task CE in F# 6
        use requestMessage = createRequestMessage secrets endpoint
        use! response = httpClient.SendAsync(requestMessage)
        let! responseString = response.Content.ReadAsStringAsync()
        return responseString
    }
    |> Async.AwaitTask

And now we can make a type for the API which holds the secrets and can dispose the internal HttpClient:

type FiriApi(secrets:FiriSecrets) =

    let httpClient = new HttpClient(BaseAddress = Uri "https://api.miraiex.com/v2/")
    let sendRequest = send httpClient secrets

    interface System.IDisposable with
        member this.Dispose() =
            httpClient.Dispose()

    member this.GetMarkets () =
        Endpoint.ListMarkets
        |> sendRequest
        |> Async.map (Decode.fromString (Decode.list Market.Decoder))

    member this.GetBalances () =
        Endpoint.ListBalances
        |> sendRequest
        |> Async.map (Decode.fromString (Decode.list Balance.Decoder))

    member this.GetAllTransactions () =
        Endpoint.ListTransactions
        |> sendRequest
        |> Async.map (Decode.fromString (Decode.list Transaction.Decoder))

    member this.GetAllTrades () =
        Endpoint.ListTrades
        |> sendRequest
        |> Async.map (Decode.fromString (Decode.list Trade.Decoder))

    member this.GetDepositHistory () =
        Endpoint.ListDepositHistory
        |> sendRequest
        |> Async.map (Decode.fromString DepositHistory.Decoder)

    member this.GetAllOrders () =
        Endpoint.ListOrders
        |> sendRequest
        |> Async.map (Decode.fromString (Decode.list Order.Decoder))

    member this.CreateOrder (order:Order) =
        order
        |> Endpoint.CreateOrder
        |> sendRequest
        |> Async.map (Decode.fromString CreateOrderResponse.Decoder)

    member this.DeleteAllOrders () =
        Endpoint.DeleteAllOrders
        |> sendRequest
        |> Async.Ignore

What if you’re using multiple crypto exchanges?

For e.g. Coinbase, if not using OAuth, we could reuse that same hashing function for the secrets headers and make something similar to the Firi implementation we did above:

type CoinbaseSecrets =
    { ApiKey : string
      ApiSecret : string }
with
    static member Create (apiKey, apiSecret) =
        { ApiKey = apiKey
          ApiSecret = apiSecret }

let getSignature (secret:string) (timestamp:string) (method:HttpMethod) (url:string) (body:string) =
    let body = timestamp + method.Method + url + body
    computeSignature secret body

let createMessage (secrets:CoinbaseSecrets) (endpoint:Endpoint) =
    let timestamp = DateTimeOffset.UtcNow.ToUnixTimeSeconds() |> string
    let requestUri = $"/v2/{endpoint.Url}"
    let signature =
        endpoint.Body
        |> Option.defaultValue ""
        |> Signature.Coinbase.getSignature secrets.ApiSecret timestamp endpoint.HttpMethod requestUri

    let requestMessage = new HttpRequestMessage(endpoint.HttpMethod, requestUri)
    requestMessage.Headers.Add("CB-VERSION", "2021-12-11") // Some recent date
    requestMessage.Headers.Add("CB-ACCESS-KEY", secrets.ApiKey)
    requestMessage.Headers.Add("CB-ACCESS-SIGN", signature)
    requestMessage.Headers.Add("CB-ACCESS-TIMESTAMP", timestamp)

    endpoint.Body
    |> Option.iter (fun body ->
        let content = new StringContent(body, Text.Encoding.UTF8, "application/json")
        requestMessage.Content <- content)

    requestMessage

The Coinbase responses are a bit more verbose than the Firi ones. For example look at the transactions response. It’s quite big, but we can use Json2FSharp to create the types for us. Just copy the example into the left side of Json2FSharp and it will create the matching F# types for you. You might have to modify them a bit though. Fortunately, there’s a .NET implementation already available here and we can see if we can use that one instead of rolling our own.

Separating the instructions from the side-effects

Now we have two different crypto clients with different ways of getting the data we’re after. We want to make some common instruction set hiding the specific implementations, but how? F# supports interfaces, so why not use that and create one implementation per exchange? Because, MONAAAADS! More specifically a Free Monad with one interpreter per client.

Creating a Free Monad

A Free Monad follows a very specific recipie which can be more or less copy/pasted every time, except for the actual instructions you want it to have. The different instructions are added to a discriminated union type and will look like this:

type MyInstructions<'a>
    | DoSomething of (Args1 * (Out1 -> 'a))
    | DoSomethingElse of (Args2 * (Out2 -> 'a))
    ...

Now let’s make one for showing your total balance and your transaction history. First we start by making some output types for the instructions:

type InstructionBalance =
    { Crypto: string
      Amount: double }
    static member Create crypto amount =
        { Crypto = crypto
          Amount = amount }

type InstructionAmount =
    { Amount: double
      Currency: string }
    static member Create amount currency =
        { Amount = amount
          Currency = currency }

[<RequireQualifiedAccess>]
type TransactionType =
    | Deposit of Amount:InstructionAmount
    | Transaction of Bought:InstructionAmount * Sold:InstructionAmount * Fee:InstructionAmount
    | Withdrawal of Amount:InstructionAmount * Fee:InstructionAmount

type InstructionTransaction =
    { Exchange: string
      TimeStamp: DateTimeOffset
      Type: TransactionType }
    static member Create exchange timeStamp transactionType =
        { Exchange = exchange
          TimeStamp = timeStamp
          Type = transactionType }

type InstructionMarket =
    { MarketPair: string
      Last: decimal
      High: decimal
      Low: decimal }
    static member Create marketPair last high low =
        { MarketPair = marketPair
          Last = last
          High = high
          Low = low }

Then we create the different instructions and the required Free Monad functions:

// The avaiable instructions.
// First item in tuple is `unit`, since there are no arguments.
type CryptoInstruction<'a> =
    | GetBalances of (unit * (InstructionBalance list -> 'a))
    | GetTransactions of (unit * (InstructionTransaction list -> 'a))
    | GetMarkets of unit * (InstructionMarket list -> 'a)

// Free Monad recipie below

// The instructions type has to be a functor, so we add a map function.
module CryptoInstruction =
    let map f = function
        | GetBalances (x, next) -> GetBalances (x, next >> f)
        | GetTransactions (x, next) -> GetTransactions (x, next >> f)
        | GetMarkets (x, next) -> GetMarkets (x, next >> f)

// This will always have this shape.
type CryptoProgram<'a> =
    | Free of CryptoInstruction<CryptoProgram<'a>>
    | Pure of 'a

// And it requires a bind function
module CryptoProgram =
    let rec bind f = function
    | Free x -> x |> CryptoInstruction.map (bind f) |> Free
    | Pure x -> f x

// Computation expression to make it convenient to use
type CryptoBuilder () =
    member this.Bind (x, f) = CryptoProgram.bind f x
    member this.Return x = Pure x
    member this.ReturnFrom x = x
    member this.Zero () = Pure ()

let crypto = CryptoBuilder()

// Convenience functions for creating the instructions
let getBalances () = Free (GetBalances ((), Pure))
let getTransactions () = Free (GetTransactions ((), Pure))
let getMarkets () = Free (GetMarkets ((), Pure))

This lets us create a simple program for getting balances:

let getBalancesProgram = crypto {
    let! markets = getMarkets ()
    let! balances = getBalances ()

    let result = // Combine them somehow...

    return result
}

But the Free Monad follows a recipie you said

True that, and for that we can use FSharpPlus which has Free Monad helper functions. Using this library, all you need is the instructions functor with the map function as a static member (FSharpPlus uses SRTPs) and use Free.liftF for the convenience functions:

open FSharpPlus.Data

type CryptoInstruction<'a> =
    | GetBalances of (unit * (InstructionBalance list -> 'a))
    | GetTransactions of (unit * (InstructionTransaction list -> 'a))
    | GetMarkets of unit * (InstructionMarket list -> 'a)
    static member Map (instruction, f) = // FSharpPlus expects static member 'Map'
        match instruction with
        | GetBalances (x, next) -> GetBalances (x, next >> f)
        | GetTransactions (x, next) -> GetTransactions (x, next >> f)
        | GetMarkets (x, next) -> GetMarkets (x, next >> f)

// Convenience functions for creating the instructions
let getBalances () = GetBalances ((), id) |> Free.liftF
let getTransactions () = GetTransactions ((), id) |> Free.liftF
let getMarkets () = GetMarkets ((), id) |> Free.liftF

And we can replace our crypto computation expression with FSharpPlus’ monad:

open FSharpPlus

let getBalancesProgram = monad {
    let! markets = getMarkets ()
    let! balances = getBalances ()

    let result = // Combine them somehow...

    return result
}

Now for some interpretation

The second part of a Free Monad is the interpreter, which is where the side-effects happen. First, we need functions for getting the API types into the instruction types, e.g. getting the Balance response from the Firi API into the InstructionBalance type:

// Calling the API is Async and Thoth.Json's decoders return Result,
// so we end up with a nested type.

open FsToolkit.ErrorHandling // Has convenient asyncResult CE

// FiriApi -> Async<Result<InstructionBalance list, string>>
let getBalancesApi firiApi =
    asyncResult {
        let! balances = firiApi.GetBalances()
        return
            balances
            |> List.map (fun b ->
                InstructionBalance.Create b.Currency b.Balance)
    }

// FiriApi -> Async<Result<InstructionTransaction list, string>>
let getTransactionsApi firiApi =
    asyncResult {
        let! transactions = firiApi.GetTransactions ()
        let instructionTransactions =
            transactions
            |> // Create InstructionTransactions from the response (see GitHub repo)
        return instructionTransactions
    }

let private getMarketsApi (firiApi:FiriApi) =
    asyncResult {
        let! markets = firiApi.GetMarkets()
        return
            markets
            |> List.map (fun m ->
                InstructionMarket.Create m.Id m.Last m.High m.Low)
    }

then we make the actual interpret function:

// FiriApi -> CryptoInstruction<'a> -> Async<Result<'a, string>>
let interpret firiApi instruction =
    match instruction with
    | GetBalances (_, next) ->
        asyncResult {
            let! balances = getBalancesApi firiApi
            return balances |> next
        }
    | GetTransactions (_, next) ->
        asyncResult {
            let! transactions = getTransactionsApi firiApi
            return transactions |> next
        }
    | GetMarkets (_, next) ->
        asyncResult {
            let! markets = getMarketsApi firiApi
            return markets |> next
        }

To interpret a program, which might contain multiple instructions, we can use FSharpPlus Free.fold and pass in our interpret function:

let balances =
    getBalancesProgram
    |> Free.fold (interpret firiApi) // Doesn't compile, since Async<Result<'a, 'b>> doesn't have static member (>>=)

Free.fold requires the returned type to have two static members, bind (>>=) and return, but we have a nested type. What now?

Option 1: Make a wrapper type

// Has Async.bind, Async.singleton and Result.either
open FsToolkit.ErrorHandling

type AsyncResult<'a, 'b> =
    AsyncResult of Async<Result<'a, 'b>>
with
    static member (>>=) (AsyncResult x, f:'a -> AsyncResult<'c, 'b>) =
        let f' a =
            let (AsyncResult r) = f a
            r

        x
        |> Async.bind (Result.either f' (Error >> Async.singleton))
        |> AsyncResult

    static member Return x = Ok x |> Async.singleton |> AsyncResult

And now we can do this:

let balances =
    getBalancesProgram
    |> Free.fold (interpret firiApi >> AsyncResult) // Wrap nested type
    |> fun (AsyncResult ar) -> ar // Unwrap the final result
    |> Async.RunSynchronously
    |> function
        | Ok bs ->
            //...
        | Error err ->
            //...

Option 2: Monad transformers

Nested monads can be handled by Monad Transformers. Our inner monad is Result so we need a transformer for that. Conveniently FSharpPlus has one called ResultT and we don’t have to make any wrapper types anymore:

let balances =
    getBalancesProgram
    |> Free.fold (interpret firiApi >> ResultT)
    |> ResultT.run
    |> Async.RunSynchronously
    |> function
        | Ok bs ->
            //...
        | Error err ->
            //...

Let’s add an interpreter for Coinbase

We’re going to use an existing .NET client in the interpreter, since someone already did the hard work there. Now, let’s try to get our balances using this client.

First of all, the Coinbase API uses pagination, so we have to be able follow the links to the next pages to extract all the data. The API returns a PagedResponse<‘a> with a link to the next page and the data for the current page. Hence, we’re going to need an async loop, since we’re going to have to call the API multiple times until there are no pages left. To help us with this we’re going to use the FSharp.Control.AsyncSeq library.

open FSharp.Control
open Coinbase
open Coinbase.Models

/// Gets all the following pages for the given initial response
let getPages (client:CoinbaseClient) (initialResponse:PagedResponse<'a>) : AsyncSeq<'a array> =
    initialResponse
    |> AsyncSeq.unfoldAsync (fun x -> async {
        if x.Pagination.NextUri |> isNull then
            return None
        else
            let! nextPage = client.GetNextPageAsync x |> Async.AwaitTask
            return Some (nextPage.Data, nextPage)
    })

Now that we have a generic function to get all pages of a certain response, we can fetch all accounts where the amount is greater than zero quite conveniently using AsyncSeq:

let getBalancesApi (client:CoinbaseClient) : Async<InstructionBalance list> =
    asyncSeq {
        let! response = client.Accounts.ListAccountsAsync() |> Async.AwaitTask
        yield response.Data // Yield the initial response's data
        yield! getPages client response // Yield all the following pages
    }
    |> AsyncSeq.concatSeq
    |> AsyncSeq.filter (fun a -> a.Balance.Amount > 0.0M)
    |> AsyncSeq.map (fun a ->
        InstructionBalance.Create a.Balance.Currency a.Balance.Amount)
    |> AsyncSeq.toListAsync

let getTransactionsApi (client:CoinbaseClient) =
    // ... skipped for brevity (see GitHub repo)

// Could take currency as parameter if we change the instruction to this:
// | GetMarkets of (currency:string) * (InstructionMarket list -> 'a)
//
// and convenience function to this:
//
// let getMarkets (currency:string) = GetMarkets (currency, id) |> Free.liftF
let getMarketsApi (client:CoinbaseClient) =
    async {
        let! response =
            client.Data.GetExchangeRatesAsync("NOK") |> Async.AwaitTask

        return
            response.Data.Rates
            |> Seq.map (fun kvp ->
                let rate = 1.0m / kvp.Value
                InstructionMarket.Create kvp.Key rate rate rate)
            |> Seq.toList
    }

and we can create the interpret function:

// CoinbaseClient -> CryptoInstruction<'a> -> Async<'a>
let interpret coinbaseClient instruction =
    match instruction with
    | GetBalances (_, next) ->
        async {
            let! balances = getBalancesApi coinbaseClient
            return balances |> next
        }
    | GetTransactions (_, next) ->
        async {
            let buys = getTransactionsApi coinbaseClient
            return buys |> next
        }
    | GetMarkets (_, next) ->
        async {
            let! markets = getMarketsApi coinbaseClient
            return markets |> next
        }

let balances =
    getBalancesProgram
    |> Free.fold (interpret coinbaseClient) // No nested type here

We can also make a testing interpreter

Here we add some test data which we could use for testing our instructions:

let [<Literal>] private ExchangeName = "Testing"

let interpret instruction =
    match instruction with
    | GetBalances (_, next) -> async {
        do! Async.Sleep 2000 // Pretend it takes time

        let balances =
            [
                InstructionBalance.Create ExchangeName "BTC" 0.0175m
                InstructionBalance.Create ExchangeName "ETH" 0.03m
                InstructionBalance.Create ExchangeName "ADA" 200.123m
                InstructionBalance.Create ExchangeName "ALU" 123.123m
                InstructionBalance.Create ExchangeName "SHIB" 123456789.0m
            ]
        return balances |> next
        }

    | GetTransactions (_, next) -> async {
        do! Async.Sleep 4000 // Pretend it takes time

        let transactions =
            [
                let startDate = DateTimeOffset(DateTime(2021, 1, 1, 12, 00, 00))
                InstructionTransaction.Create ExchangeName (startDate.AddDays 1) (TransactionType.Deposit (InstructionAmount.Create 1000m "USD"))
                InstructionTransaction.Create ExchangeName (startDate.AddDays 2) (TransactionType.Transaction ((InstructionAmount.Create 0.0348m "BTC"), (InstructionAmount.Create 1000m "USD"), (InstructionAmount.Create 0.5m "USD")))
                InstructionTransaction.Create ExchangeName (startDate.AddDays 3) (TransactionType.Deposit (InstructionAmount.Create 1000m "USD"))
                InstructionTransaction.Create ExchangeName (startDate.AddDays 4) (TransactionType.Transaction ((InstructionAmount.Create 0.03m "ETH"), (InstructionAmount.Create 1000m "USD"), (InstructionAmount.Create 1m "USD")))
                InstructionTransaction.Create ExchangeName (startDate.AddMonths 11) (TransactionType.Transaction ((InstructionAmount.Create 932.4663m "USD"), (InstructionAmount.Create 0.0174m "BTC"), (InstructionAmount.Create 0.5m "USD")))
                InstructionTransaction.Create ExchangeName (startDate.AddMonths 12) (TransactionType.Withdrawal ((InstructionAmount.Create 932.4663m "USD"), (InstructionAmount.Create 1m "USD")))


            ]
        return transactions |> next

    | GetMarkets (_, next) -> async {
        do! Async.Sleep 1000 // Pretend it takes time

        let markets =
            [
                InstructionMarket.Create "BTC" 500000m 500000m 500000m
                InstructionMarket.Create "ETH" 35000m 35000m 35000m
                InstructionMarket.Create "ADA" 8.25m 8.25m 8.25m
                InstructionMarket.Create "ALU" 1.34m 1.34m 1.34m
                InstructionMarket.Create "SHIB" 0.0003m 0.0003m 0.0003m
            ]
        return markets |> next
    }

let run (instructions:Free<CryptoInstruction<'a>, 'a>) =
    instructions
    |> Free.fold interpret

Aggregating the response from multiple interpreters

We have to somehow invoke both Firi and Coinbase when we want to interpret our programs. Say we have the following functions:

module Firi =
    // let interpret ...
    // ...

    let run firiApi (instructions:Free<CryptoInstruction<'a>, 'a>) : Async<Result<'a, string>> =
        instructions
        |> Free.fold (interpret firiApi >> ResultT)
        |> ResultT.run

module Coinbase =
    // let interpret ...
    // ...

    let run coinbaseClient (instructions:Free<CryptoInstruction<'a>, 'a>) : Async<'a> =
        instructions
        |> Free.fold (interpret coinbaseClient)

Then we can run our program for both interpreters and concatenate the result using AsyncSeq like this:

module Program =

    open FSharp.Control
    open FsToolkit.ErrorHandling

    // Runs a program on both interpreteres and returns the concatenated result
    // FiriApi -> CoinbaseClient -> Free<CryptoInstruction<'a list>, 'a list> -> Async<Result<'a list, string list>>
    let run firiApi coinbaseClient program =
        asyncSeq {
            // yield (Testing.run >> Async.map Ok)
            yield Firi.run firiApi
            yield (Coinbase.run coinbaseClient >> Async.map Ok) // Must match return type from Firi.run
        }
        |> AsyncSeq.mapAsyncParallel (fun run -> run program)
        |> AsyncSeq.toListAsync // Async<Result<'a list, string> list>
        |> Async.map (List.sequenceResultA >> Result.map (List.collect id))

let getBalancesProgram = monad {
    let! balances = getBalances ()
    return balances
}

let balances =
    getBalancesProgram
    |> Program.run firiApi coinbaseClient
    |> Async.RunSynchronously

I want to see the results in pretty tables

For this we can create a CLI and use Spectre.Console library to make it awesome.

First we make a function which combines the balances and current values in NOK in a table:

let spinner =
    let spinner = AnsiConsole.Status()
    spinner.Spinner <- Spinner.Known.Dots
    spinner.SpinnerStyle <- Style.Parse("green bold")
    spinner

let showBalances (firiApi:FiriApi) (coinbaseClient:CoinbaseClient) = async {

    let getBalancesProgram = monad {
        let! balances = getBalances ()
        let! markets = getMarkets ()

        let marketLookup =
            ("NOK", 1.0m) ::
            (markets |> List.map (fun m -> (m.MarketPair, m.Last)))
            |> Map.ofList

        let nokPerCoin =
            balances
            |> List.map (fun b ->
                let value =
                    marketLookup
                    |> Map.find b.Crypto

                (b.Crypto, b.Amount * value))
            |> Map.ofList

        return
            balances
            |> List.map (fun b ->
                {|
                    b with
                        Value = nokPerCoin.[b.Crypto]
                        Currency = "NOK"
                |})
        }

    let! balances =
        spinner.StartAsync(
            "[green]Getting balances...[/]",
            fun _ -> run getBalancesProgram |> Async.StartAsTask)
        |> Async.AwaitTask

    match balances with
    | Ok xs ->
        let table =
            Table()
                .AddColumn("[blue]Crypto[/]")
                .AddColumn("[blue]Amount[/]")
                .AddColumn("[blue]Value[/]")
                .AddColumn("[blue]Currency[/]")
                .AddColumn("[blue]Exchange[/]")
        table.Border <- TableBorder.Rounded
        table.Title <- TableTitle("[purple_1]Balances[/]")
        table.Columns.[1].Alignment <- Justify.Right
        table.Columns.[2].Alignment <- Justify.Right

        xs
        |> List.filter (fun t -> t.Amount > 0.0m)
        |> List.iter (fun t ->
            table.AddRow(
                $"[orange1]{t.Crypto}[/]",
                $"[green]%.4f{t.Amount}[/]",
                $"[green]%.2f{t.Value}[/]",
                $"[blue]{t.Currency}[/]",
                $"[deepskyblue1]{t.Exchange}[/]")
            |> ignore)

        let sum = xs |> List.sumBy (fun t -> t.Value)
        table.AddRow("[lightgreen]Sum[/]", "", $"[lightgreen]%.2f{sum}[/]", "[blue]NOK[/]") |> ignore

        AnsiConsole.Write (table)

    | Error err ->
        AnsiConsole.MarkupLine($"[Red]Failed to get balances. Details: {err}[/]")
    }

open Coinbase
open Spectre.Console
open ...

[<RequireQualifiedAccess>]
type Selection =
    | Transactions
    | Balances
    | Quit

[<EntryPoint>]
let main argv =

    // Get from environment var or args
    let firiClientId = ""
    let firiSecret = ""
    let coinbaseApiKey = ""
    let coinbaseSecret = ""

    use firiApi = new FiriApi(FiriSecrets.Create clientId secret)
    use coinbaseClient =
        new CoinbaseClient(new ApiKeyConfig(ApiKey=coinbaseApiKey,
                                            ApiSecret=coinbaseSecret))

    let selectionPrompt =
        let prompt = new SelectionPrompt<Selection>()
        prompt.Title <- "[blue]What do you want to see?[/]"
        prompt.AddChoices(
            [|
                Selection.Transactions
                Selection.Balances
                Selection.Quit
            |]) |> ignore
        prompt

    let rec loop () = async {
        let choice = AnsiConsole.Prompt(selectionPrompt)

        match choice with
        | Selection.Transactions ->
            do! showTransactions firiApi coinbaseClient
            return! loop ()

        | Selection.Balances ->
            do! showBalances firiApi coinbaseClient
            return! loop ()

        | Selection.Quit ->
            ()
        }

    loop ()
    |> Async.RunSynchronously

    0 // return an integer exit code

Now let’s try it out using our testing interpreter:

Profit!

Using the aggregated transaction history from multiple exchanges, we could calculate our gains and losses for, say, tax purposes, but that’s a task for another day.

Creating a Lego Mindstorms DSL in F#

2019-12-11T00:00:00+00:00

This is part of F# Advent calendar 2019. Go and check out all the other great posts and thank you Sergey Tihon for organizing!

We’re going to play around with some Lego Mindstorms. Luckily, there’s already a .NET API for this made by Brian Peek. Sadly, it’s no longer under active development and haven’t been for quite some time. Since I’m on linux, I would like to use .NET Core and new SDK project files, so I made a fork where I patched it a bit and replaced the events with Observables, because why not. It’s using HidSharp to connect to the Mindstorms brick via the USB port. It can be found here.

In this post we’re going try to make a DSL in F# on top of the existing C# code.

Lego Mindstorms

The Lego Mindstorms brick has eight ports, four marked A-D and four marked 1-4. The A-D ports can both send and receive input data. 1-4 can only receive. The C# API defines a Command which can be configured with multiple actions before sending it to the Brick to be executed in order. There’s also an observable which pushes responses from the Brick’s devices, e.g. push sensor button presses or color sensor data. We’re going to try to make a DSL to configure the actions to apply to a Command, but first we’ll define the ports:

type OutputPort =
    | A
    | B
    | C
    | D
    | All

type InputPort =
    | One
    | Two
    | Three
    | Four
    | A
    | B
    | C
    | D

Motor actions

Let’s define what the different motor actions are by picking a subset of the available C# commands and their arguments.

/// Should the break be applied at the end of the action?
type BreakMode =
    | Break
    | Coast

/// A sub-set of motor actions that can be added to a command
type MotorAction =
    | StartMotor
    | StopMotor
    | StepMotorAtPower of {| Power:int; Steps:uint32; Break:BreakMode |}
    | TurnMotorAtPower of {| Power:int |}
    | TurnMotorAtPowerForTime of {| Power:int; Time:uint32; Break:BreakMode |}
    | StepMotorSync of {| Power:int; TurnRatio:int16; Steps:uint32; Break:BreakMode |}

We also need to define the port somewhere. Since we can have different devices connected to the ports, we’ll make a top level BrickActions type.

type BrickAction =
    // Output port is a list, since an action can be applied to multiple motors simultaneously
    | MotorAction of OutputPort list * MotorAction

We need a nice way to define which of these actions to perform. What we’re aiming for is something like this:

let commands = mindstorms {
    Turn (Motor OutputPort.A) With Power 50
    Turn (Motors [ OutputPort.A; OutputPort.B ]) With Power 50
    TurnForTime 1000u (Motors [ OutputPort.A; OutputPort.B ]) With Power 50 Then Break
    Step (Motors [ OutputPort.A; OutputPort.B ]) For 180u Steps With Power 50 Then Coast
    StepSync (Motors [ OutputPort.A; OutputPort.B ]) For 180u Steps With Power 50 And TurnRatio 42s Then Coast
    Start (Motor OutputPort.A)
    Stop (Motor OutputPort.A)
    }

Creating the builder

Our builder’s state will be a list of BrickActions and we’ll start with the required Yield member of the builder. It should just return an empty sequence.

type MindstormsBuilder() =
    member __.Yield(_) = Seq.empty

Let’s look at the 1st line in the commands definition. Here we have some words we have to define, Motor, With and Power. A single command can also combine multiple ports, hence we have to support defining a list of ports as well. Hence the Motor type has to look something like this:

type MotorPorts =
    | Motor of OutputPort
    | Motors of OutputPort list

module MotorPorts =
    let get = function
        | Motor m -> [ m ]
        | Motors ms -> ms

Next is the With and Power keywords. We can simply make a single case union type without any content, since it’s just a word.

type With = With
type Power = Power

Now we can add the first method of the builder:

[<CustomOperation("Turn")>]
member __.Turn(currentState, motor:MotorPorts, _:With, _:Power, power:int) =
    seq {
        yield! currentState
        yield MotorAction (MotorPorts.get motor, TurnMotorAtPower {| Power = power |})
    }

Here _:With and _:Power are just used as placeholders make it look like a proper sentence.

Now if we look at the 3rd line we have this:

TurnForTime 1000u (Motors [ OutputPort.A; OutputPort.B ]) With Power 50 Then Break

The Motors keyword is already defined, but we need Then and Break as well. We can see from the 4th that there is a Coast keyword as well, so we get this:

type Then = Then

type BreakMode =
    | Break
    | Coast

Now we can define the TurnForTime operation like this:

[<CustomOperation("TurnForTime")>]
member __.TurnForTime(currentState, time:uint32, motor:MotorPorts, _:With, _:Power, power:int, _:Then, breakMode:BreakMode) =
    seq {
        yield! currentState
        yield MotorAction (MotorPorts.get motor, TurnMotorAtPowerForTime {| Power = power; Time = time; Break = breakMode |})
    }

The 4th and 5th lines require the Steps And and TurnRatio keywords:

Step (Motors [ OutputPort.A; OutputPort.B ]) For 180u Steps With Power 50 Then NoBreak
StepSync (Motors [ OutputPort.A; OutputPort.B ]) For 180u Steps With Power 50 And TurnRatio 42s Then NoBreak

type And = And
type Steps = Steps
type TurnRatio = TurnRatio

and now we can define the operations like this:

[<CustomOperation("Step")>]
member __.Step(currentState, motor:MotorPorts, _:For, steps:uint32, _:Steps, _:With, _:Power, power:int, _:Then, breakMode:BreakMode) =
    seq {
        yield! currentState
        yield MotorAction (MotorPorts.get motor, StepMotorAtPower {| Power = power; Steps = steps; Break = breakMode |})
    }

[<CustomOperation("StepSync")>]
member __.StepSync(currentState, motor:MotorPorts, _:For, steps:uint32, _:Steps, _:With, _:Power, power:int, _:And, _:TurnRatio, turnRatio:int16, _:Then, breakMode:BreakMode) =
    seq {
        yield! currentState
        yield MotorAction (MotorPorts.get motor, StepMotorSync {| Power= power; TurnRatio = turnRatio; Steps = steps; Break = breakMode |})
    }

And then we have Start and Stop which are a lot simpler:

[<CustomOperation("Start")>]
member __.Start(currentState, motor:MotorPorts) =
    seq {
        yield! currentState
        yield MotorAction (MotorPorts.get motor, StartMotor)
    }

[<CustomOperation("Stop")>]
member __.Stop(currentState, motor:MotorPorts) =
    seq {
        yield! currentState
        yield MotorAction (MotorPorts.get motor, StopMotor)
    }

Last but not least, we have to create an instance of the builder like this:

let mindstorms = MindstormsBuilder()

Creating the command

Now that we have a way to creat a list of BricActions, we need a way to update a command with them. This is done by first creating a command from the Brick and then mutating it by invoking different methods on it. Below we have an update function which translates from BricAction to the correct method on the Command instance.

let updateCommand : Command -> BrickAction -> Command =
    fun command actions ->
    match actions with
    | MotorAction (ports, action) ->
        let ports =
            ports
            |> List.map OutputPort.toEnum // Map to the C# enum flags
            |> List.reduce (|||) // Bitwise OR

        match action with
        | StartMotor -> command.StartMotor(ports)
        | StopMotor -> command.StopMotor(ports, true)
        | TurnMotorAtPower x -> command.TurnMotorAtPower(ports, x.Power)
        | TurnMotorAtPowerForTime x -> command.TurnMotorAtPowerForTime(ports, x.Power, x.Time, x.Break |> BreakMode.asBool)
        | StepMotorAtPower x -> command.StepMotorAtPower(ports, x.Power, x.Steps, x.Break |> BreakMode.asBool)
        | StepMotorSync x -> command.StepMotorSync(ports, x.Power, x.TurnRatio, x.Steps, x.Break |> BreakMode.asBool)

    command

And we can simply fold the list of BricActions over the Command like this:

let createCommand : Brick -> BrickAction seq -> Command =
    fun brick actions ->
    let cmd = brick.CreateCommand(CommandType.DirectReply)
    Seq.fold updateCommand cmd actions

which we then can send to the brick:

let invokeCommand : Brick -> BrickAction seq -> Task<System.Reactive.Unit> =
    fun brick actions ->
    createCommand brick actions
    |> brick.SendCommandAsync

Creating a simple program

Now we’ll try to create a simple program where we turn the engines when the push button is pressed and stop the engines when it’s released.

First we have to connect to the brick:

    let comm = UsbCommunication("MyLegoEV3");
    let responseManager = ResponseManager();
    let brick = Brick(comm, responseManager);

    use connection =
       brick.Connect()
       |> Observable.subscribe ignore

Then we must listen to port changes and create two observables which trigger according to the button state, which is connected to port three:

let (pressed, released) = 
    brick.Ports.[RxMindstorms.InputPort.Three].Changes()
    |> Observable.map (fun struct (p, _) -> p.SIValue)
    |> Observable.distinctUntilChanged
    |> Observable.partition (fun x -> x = 1.0f)

Next we define what happens when you press and release the button:

use __ =
    pressed
    |> Observable.iter (fun x -> printfn "Pressed: %A" x)
    |> Observable.map (fun _ ->
        mindstorms {
            Turn (Motors [ OutputPort.B; OutputPort.C ] ) With Power 100
            Start (Motors [ OutputPort.B; OutputPort.C ])
        }
    )
    |> Observable.flatmapTask (invokeCommand brick)
    |> Observable.subscribe ignore

use __ =
    released
    |> Observable.iter (fun x -> printfn "Released: %A" x)
    |> Observable.map (fun _ ->
        mindstorms {
            Stop (Motors [ OutputPort.B; OutputPort.C ])
        }
    )
    |> Observable.flatmapTask (invokeCommand brick)
    |> Observable.subscribe ignore

Creating programs with state

Here we try to make a program with different states.

Wait for push sensor before moving
When button is pressed, jump to driveForwards state
Then jump to waitForLightSensor and wait for light sensor pass a threshold
If the light sensor hits the threshold, jump to turn() state
Then jump back to driveForwards state

let rec waitUntilPushButton () = async {
    printfn "waitUntilPushButton"
    if brick.Ports.[RxMindstorms.InputPort.Three].SIValue = 1.0f then
        return! driveForwards ()
    else
        do! Async.Sleep 100
        return! waitUntilPushButton()        
    }
and driveForwards () = async {
    printfn "driveForwards"
    do! mindstorms {
            Turn (Motors [ OutputPort.B; OutputPort.C ] ) With Power 100
            Start (Motors [ OutputPort.B; OutputPort.C ] )
        }
        |> invokeCommand brick
        |> Async.AwaitTask
        |> Async.Ignore

    return! waitForLightSensor ()        
    }
and waitForLightSensor () = async {
    printfn "waitForLightSensor"
    if brick.Ports.[RxMindstorms.InputPort.Four].SIValue > 60.0f then
        return! turn ()
    else
        do! Async.Sleep 100
        return! waitForLightSensor ()
    }
and turn () = async {
    printfn "turn"
    do! mindstorms {
           TurnForTime 1000u (Motors [ OutputPort.B ]) With Power 80 Then Coast
           TurnForTime 1000u (Motors [ OutputPort.C ]) With Power -80 Then Coast
        }
        |> invokeCommand brick
        |> Async.AwaitTask
        |> Async.Ignore

    return! driveForwards ()        
    }

waitUntilPushButton ()
|> Async.RunSynchronously

Adding the ‘Run’ method

The computation expressions also have a Run member that can be implemented to change the state before returning it. Say we have the following code:

let snippet : BrickActions seq =
    mindstorms {
        TurnForTime 1000u (Motors [ OutputPort.A; OutputPort.B ]) With Power 50 Then Break
        Start (Motor OutputPort.A)
    }

snippet is of type BrickActions seq.

If we implement Run like this:

member __.Run(actions:BrickActions seq) =

    fun (brick:Brick) ->
        let emptyCommand = brick.CreateCommand(CommandType.DirectReply)

        actions
        |> Seq.fold updateCommand emptyCommand
        |> brick.SendCommandAsync
        |> Async.AwaitTask

snippet would be of type Brick -> Async<unit> instead and we could run them like this:

async {
    do! brick
        |> mindstorms {
            TurnForTime 5000u (Motors [ OutputPort.A; OutputPort.B ]) With Power 50 Then Break
            Start (Motors [ OutputPort.A; OutputPort.B ])
            }

    do! brick
        |> mindstorms {
            TurnForTime 2000u (Motors [ OutputPort.A; OutputPort.B ]) With Power 50 Then Break
            TurnForTime 2000u (Motors [ OutputPort.A; OutputPort.B ]) With Power -50 Then Break
            Stop (Motors [ OutputPort.A; OutputPort.B ])
            }
}
|> Async.RunSynchronously

Conclusion

We’ve seen how to make a DSL in F# using custom computation expressions, which are quite flexible and powerful. This was just a silly example, but it shows we could almost write plain english to configure the commands. It did get a bit more complicated when wiring it all up though.

This is all I had time for unfortunately. Hope you learned something. Thanks for reading!

Make FAKE release on git tag

2019-02-12T00:00:00+00:00

In this post we’re going to see how we can make a FAKE script detect when to do only build and test and when to pack and ship using a SemVer formatted git tag. Then you can have a single build configuration on your CI server and have FAKE take care of which type of build to run.

In this example our release will be a NuGet package, but this could be a docker image or something else. For public NuGet packages, using FAKE’s Release notes helper to define the release version including release notes is an alternative approach.

We’re going to create a small sample project with two classlibs and a test project. The code for the post can be found here.

Tools we’re going to use

FAKE and Paket as global .NET Core tools

Now that both FAKE and Paket have dotnet tools we can just install those to avoid all that bootstrapping that was required before. No need for mono either, unless we’re doing multi-target builds on linux.

$ dotnet tool install -g fake-cli
$ dotnet tool install -g paket

Now you can just write

$ paket restore
$ fake build

in your command line to restore packages and run a FAKE build script.

Creating a sample solution

$ dotnet new classlib -lang F# -o src/MyFsNuget
$ dotnet new classlib -lang C# -o src/MyCsNuget
$ dotnet add src/MyFsNuget/MyFsNuget.fsproj reference src/MyCsNuget/MyCsNuget.csproj
$ dotnet new -i Expecto.Template::*
$ dotnet new expecto -o tests/MyFsNugetTests
$ dotnet add tests/MyFsNugetTests/MyFsNugetTests.fsproj reference src/MyFsNuget/MyFsNuget.fsproj
$ dotnet new sln -n MyNuget
$ dotnet sln add **/**/*.*sproj

Now we’ll convert to Paket by running:

$ paket convert-from-nuget

This will move the packages specified in the Expecto test project into a paket.dependencies file which should look something like this:

source https://www.nuget.org/api/v2
nuget Expecto >= 8.0
nuget FSharp.Core >= 4.0
nuget Microsoft.NET.Test.Sdk >= 15.0
nuget YoloDev.Expecto.TestSdk

It will also add

<Import Project="..\..\.paket\Paket.Restore.targets" />

to all your project files so Paket can hook on to the build.

Adding the FAKE dependencies

FAKE 5 has split every module into a separate NuGet package, so we’re going to need a couple of them:

Fake.Core.Environment: For getting the git tag environment variable
Fake.Core.Semver: For parsing the git tag
Fake.Core.Target: For creating build steps
Fake.DotNet.AssemblyInfoFile: For writing the asssembly info files
Fake.DotNet.Cli: For building, packing and publishing a NuGet package
Fake.IO.FileSystem: For globbing and file system operators

We’re going to add the FAKE modules to a Build group in paket.dependencies so we can easily reference it in the build.fsx file.

(Microsoft.NET.Test.Sdk and YoloDev.Expecto.TestSdk enables running the Expecto tests by invoking dotnet test)

source https://www.nuget.org/api/v2

nuget FSharp.Core

nuget Expecto
nuget Microsoft.NET.Test.Sdk
nuget YoloDev.Expecto.TestSdk

group Build
  source https://www.nuget.org/api/v2

  nuget Fake.Core.Environment
  nuget Fake.Core.SemVer
  nuget Fake.Core.Target
  nuget Fake.DotNet.AssemblyInfoFile
  nuget Fake.DotNet.Cli
  nuget Fake.IO.FileSystem

Now we can start creating our FAKE script. Create a new file called build.fsx and paste the following content into it:

#r "paket: groupref Build //"
#load @".fake/build.fsx/intellisense.fsx"

#if !FAKE
  #r "netstandard"
#endif

This will import the dependencies in the Build group in our paket.dependencies file and enable intellisense. Now we can just open the dependencies we need (after running fake build once to download them):

open Fake.Core
open Fake.DotNet
open Fake.IO
open Fake.IO.Globbing.Operators
open Fake.IO.FileSystemOperators

Detecting the build configuration

To decide if we’re going to release or not we need the following steps:

Trigger a build when tagging
Pass the tag to the build agents
Parse the tag using FAKE

Triggering a build when tagging

In this example we’re going to use TeamCity as CI server. To build a tag we must edit the branch specification and tick Enable to use tags in the branch specification in the VCS root settings:

This will make TeamCity trigger builds when a new tag is pushed, with the tag as the name of the “branch” (e.g. the version numbers seen here):

Passing the tag to the build agents

The next step is to pass that information to the build agents by setting the %teamcity.build.branch% value in an environment variable in Administration -> Root project -> Parameters:

specified here as BRANCH_NAME:

Parsing the tag using FAKE

FAKE has a SemVer helper which can validate and parse SemVer strings. Here we’ll use that to create a simple active pattern which we can use for deciding what to do for a given BRANCH_NAME.

open Fake.Core

let (|Release|CI|) input =
    if SemVer.isValid input then
        let semVer = SemVer.parse input
        Release semVer
    else
        CI

Next we’ll use FAKE’s Environment helper to read our environment variable containing the branch name. If the variable doesn’t exist, say, on your dev environment, we just default to an empty string to run build and test.

let branchName = Environment.environVarOrDefault "BRANCH_NAME" ""

Finally, we’ll use the active pattern together with the branch name to decide what to do. As you can see, both the AssemblyInfo and Pack targets requires the version which is conveniently provided by the active pattern.

let projectOrSln = "src" </> "MyFsNuget" </> "MyFsNuget.fsproj"

...
...

open Fake.Core.TargetOperators

let buildTarget =
    match branchName with
    | Release version ->
        createAssemblyInfoTarget version
        createPackTarget version projectOrSln
        Target.create "Release" ignore

        "Clean"
        ==> "AssemblyInfo"
        ==> "Build"
        ==> "Test"
        ==> "Pack"
        ==> "Push"
        ==> "Release"

    | CI ->
        Target.create "CI" ignore

        "Clean"
        ==> "Build"
        ==> "Test"
        ==> "CI"

Target.runOrDefault buildTarget

The buildTarget value will be either Release or CI depending on which branch it took, so we can just pass that into Target.runOrDefault.

Updating the assembly info

Here we’re passing in the parsed semver info when creating the assembly info target. Assembly info only supports major.minor.patch strings without any -rc.1 or -beta, so we extract what we need from the SemVerInfo and update the assembly info for both F# and C# projects.

let summary = "Silly NuGet package"
let product = "MySillyNuget"
...
...

let createAssemblyInfoTarget (semverInfo : SemVerInfo) =

    let assemblyVersion =
        sprintf "%d.%d.%d" semverInfo.Major semverInfo.Minor semverInfo.Patch

    let toAssemblyInfoAttributes projectName =
        [ AssemblyInfo.Title projectName
          AssemblyInfo.Product product
          AssemblyInfo.Description summary
          AssemblyInfo.Version assemblyVersion
          AssemblyInfo.FileVersion assemblyVersion ]

    // Helper active pattern for project types
    let (|Fsproj|Csproj|) (projFileName:string) =
        match projFileName with
        | f when f.EndsWith("fsproj") -> Fsproj
        | f when f.EndsWith("csproj") -> Csproj
        | _                           -> failwith (sprintf "Project file %s not supported. Unknown project type." projFileName)

    Target.create "AssemblyInfo" (fun _ ->
        let getProjectDetails projectPath =
            let projectName = System.IO.Path.GetFileNameWithoutExtension projectPath
            let directoryName = System.IO.Path.GetDirectoryName projectPath
            let assemblyInfo = projectName |> toAssemblyInfoAttributes
            (projectPath, directoryName, assemblyInfo)

        !! "src/**/*.??proj"
        |> Seq.map getProjectDetails
        |> Seq.iter (fun (projFileName, folderName, attributes) ->
            match projFileName with
            | Fsproj -> AssemblyInfoFile.createFSharp (folderName </> "AssemblyInfo.fs") attributes
            | Csproj -> AssemblyInfoFile.createCSharp ((folderName </> "Properties") </> "AssemblyInfo.cs") attributes)
    )

Since we’re manually creating the assembly info file we have to add the following to our project files:

<GenerateAssemblyInfo>false</GenerateAssemblyInfo>

and for F# projects we also need this:

<Compile Include="AssemblyInfo.fs" />

Packing a NuGet

open Fake.IO.FileSystemOperators

let author = "My Name"
let summary = "Silly NuGet package"

...
...

/// Pass a single project to pack only that one.
/// Pass the .sln to pack all non-test projects.
let createPackTarget (semVerInfo : SemVerInfo) (projectOrSln : string)=
    Target.create "Pack" (fun _ ->

        // MsBuild uses ; and , as properties separator in the cli
        let escapeCommas (input : string) =
            input.Replace(",", "%2C")

        let customParams =
            [ (sprintf "/p:Authors=\"%s\"" author)
              (sprintf "/p:Owners=\"%s\"" author)
              (sprintf "/p:PackageVersion=\"%s\"" (semVerInfo.ToString()))
              (sprintf "/p:Description=\"%s\"" summary |> escapeCommas) ]
            |> String.concat " "

        DotNet.pack (fun p ->
            { p with
                Configuration = DotNet.BuildConfiguration.Release
                Common = DotNet.Options.withCustomParams (Some customParams) p.Common })
            projectOrSln)

If we change the default branch name to, say, 1.2.3-beta.4 like so:

let branchName = Environment.environVarOrDefault "BRANCH_NAME" "1.2.3-beta.4"

and comment out the Push target and then run the build script we get a MyFsProject.nupkg which contains a nuspec with the following content:

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>MyFsNuget</id>
    <version>1.2.3-beta.4</version>
    <authors>My Name</authors>
    <owners>My Name</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>Silly NuGet package</description>
    <dependencies>
      <group targetFramework=".NETStandard2.0">
        <dependency id="MyCsNuget" version="1.2.3-beta.4" exclude="Build,Analyzers" />
        <dependency id="FSharp.Core" version="4.6.1" exclude="Build,Analyzers" />
      </group>
    </dependencies>
  </metadata>
</package>

Here we can see that the referenced C# project is actually added as a NuGet dependency, which means you would have to pack and push that one as well. If you pass a .sln file to dotnet pack it will pack all non-test projects in the solution as mentioned in the createPackTarget function defined above.

However, if you have a multi-project solution where you don’t want to expose all the projects as separate NuGets, you can apply the following workaround, as found in the comments in this github issue, to include the private .dlls in the main NuGet package instead.

Create a file in your repository root called e.g. pack.props with this content:

<Project>
  <PropertyGroup>
    <TargetsForTfmSpecificBuildOutput>$(TargetsForTfmSpecificBuildOutput);CopyProjectReferencesToPackage</TargetsForTfmSpecificBuildOutput>
  </PropertyGroup>
  <Target Name="CopyProjectReferencesToPackage" DependsOnTargets="ResolveReferences">
    <ItemGroup>
      <BuildOutputInPackage Include="@(ReferenceCopyLocalPaths-&gt;WithMetadataValue('ReferenceSourceTarget', 'ProjectReference')-&gt;WithMetadataValue('PrivateAssets', 'All'))" />
    </ItemGroup>
  </Target>
</Project>

Then in your main project file, in our case MyFsProject.fs, add PrivateAssets="All" to the project references you want to include in the main NuGet package and import pack.props:

<?xml version="1.0" encoding="utf-8"?>
<Project Sdk="Microsoft.NET.Sdk">
  <Import Project="../../pack.props"/>
  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
    <GenerateAssemblyInfo>false</GenerateAssemblyInfo>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="AssemblyInfo.fs" />
    <Compile Include="Library.fs" />
  </ItemGroup>
  <ItemGroup>
    <ProjectReference Include="..\MyCsNuget\MyCsNuget.csproj" PrivateAssets="All" />
  </ItemGroup>
  <Import Project="..\..\.paket\Paket.Restore.targets" />
</Project>

Now when we run fake build again, the MyCsProject reference is gone in the .nuspec file and the lib folder in the NuGet package contains the MyCsProject.dll.

Publishing our NuGet

NuGet packages can be pushed using dotnet nuget push which we can invoke using FAKE’s DotNet.exec helper. Here we’re getting the NuGet feed URL and API-KEY from environment variables and then we search all Release folders for .nupkg files to push:

Target.create "Push" (fun _ ->
    let nugetServer = Environment.environVarOrFail "NUGET_WRITE_URL"
    let apiKey = Environment.environVarOrFail "NUGET_WRITE_APIKEY"

    let result =
        !!"**/Release/*.nupkg"
        |> Seq.map (fun nupkg ->
            Trace.trace (sprintf "Publishing nuget package: %s" nupkg)
            (nupkg, DotNet.exec id "nuget" (sprintf "push %s --source %s --api-key %s" nupkg nugetServer apiKey)))
        |> Seq.filter (fun (_, p) -> p.ExitCode <> 0)
        |> List.ofSeq

    match result with
    | [] -> ()
    | failedAssemblies ->
        failedAssemblies
        |> List.map (fun (nuget, proc) -> 
            sprintf "Failed to push NuGet package '%s'. Process finished with exit code %d." nuget proc.ExitCode)
        |> String.concat System.Environment.NewLine
        |> exn
        |> raise)

That’s it! Now you can publish a new NuGet package as simple as git tag 1.2.3 (and push) in any CI environment you want.

Similar resources:

SAFE stack on Ubuntu 18.04 from scratch

2018-10-11T00:00:00+00:00

Now that a new Ubuntu LTS has been available for a while, it’s about time to set it up for F# and this time some SAFE stack development as well.

Requirements

For F#, we need:

Mono (required by Paket and F# interactive FSI/REPL)
F# compiler
dotnet CLI
Visual Studio Code
Ionide VSCode extension
A cool font to make the F# pipe operator |> look like a triangle

For working with Fable we also need:

Yarn
Node

Adding repositories

Since curl is not installed by default, we start with that:

sudo apt-get -y install curl

Then we can add all the required repositories:

# dotnet core sdk
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb

# VSCode
curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main" > /etc/apt/sources.list.d/vscode.list'

# Mono
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
echo "deb https://download.mono-project.com/repo/ubuntu stable-bionic main" | sudo tee /etc/apt/sources.list.d/mono-official-stable.list

# Yarn
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list

# Node
curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -

Installing everything

Due to this issue with Yarn, we first have to remove the cmdtest package before installing Yarn:

sudo apt-get remove cmdtest

Then we can install all the things!

sudo apt-get -y install apt-transport-https
sudo apt-get update
sudo apt-get -y install fsharp dotnet-sdk-2.1 code fonts-firacode yarn nodejs

If you want to keep your dotnet CLI up to date, install the dotnet-sdk-2.1 package, since it installs the newest available dotnet-sdk-2.1.xxx version when running apt-get upgrade. To only install specific versions (which can be installed side-by-side) install the specific dotnet-sdk-2.1.xxx package you want instead.

You can check which versions of the SDK you have installed by running:

$ dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.403
 Commit:    04e15494b6

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  18.04
 OS Platform: Linux
 RID:         ubuntu.18.04-x64
 Base Path:   /usr/share/dotnet/sdk/2.1.403/

Host (useful for support):
  Version: 2.1.5
  Commit:  290303f510

.NET Core SDKs installed:
  2.1.105 [/usr/share/dotnet/sdk]
  2.1.200 [/usr/share/dotnet/sdk]
  2.1.403 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.5 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.7 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

Setting up VSCode

First we want to install some extensions for F# development:

code --install-extension Ionide.ionide-fsharp
code --install-extension Ionide.ionide-Paket
code --install-extension Ionide.ionide-FAKE
code --install-extension ms-vscode.csharp
code --install-extension donjayamanne.githistory
code --install-extension github.vscode-pull-request-github

The ms-vscode.csharp extension is to add debugging support when working in VSCode. donjayamanne.githistory gives you a nice view of the Git log and visual commit graph among other things. github.vscode-pull-request-github gives you a list of PRs straight into VSCode. It is still in preview, but already has a lot of useful stuff it can do.

Next we want to enable the Fira code font and make Ionide use netcore instead of mono. You might want to add this to a file and execute it or you can try to copy/paste into a terminal. Or just open VSCode and press Ctrl+, and paste the settings in there.

folder="$HOME/.config/Code/User"
file="$folder/settings.json"

mkdir -p $folder
cat >> $file <<EOF
{
    "editor.fontLigatures": true,
    "editor.fontFamily": "Fira Code",
    "editor.fontSize": 14,
    "FSharp.fsacRuntime": "netcore"
}
EOF

Setting up Git branch info in terminal

I still like to have the Git branch and status shown in the bash prompt when I’m inside folder with a git repository. The current branch appears inside parentheses like this:

me@mycomputer:~/src/mygitrepo (my-feature-branch)$

The Git source code contains a script which can be used to show this information. To set it up, first fetch the git-prompt.sh script:

curl -fsSL https://raw.githubusercontent.com/git/git/master/contrib/completion/git-prompt.sh -o .git-prompt.sh

Then open ~/.bashrc in your favorite editor and add the following at the bottom:

# Git branch info in terminal prompt
. ~/.git-prompt.sh
export GIT_PS1_SHOWDIRTYSTATE=1
export GIT_PS1_SHOWCOLORHINTS=true
export PROMPT_COMMAND='__git_ps1 "\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]" "\\\$ "'

There are some additional environment variables that can be set which are explained in .git-prompt.sh

Adding some Git aliases for less typing

It can become a bit cumbersome to write the whole Git commands all the time, so adding a few aliases for the most common ones is a good idea. Write the following in a shell:

git config --global alias.co checkout
git config --global alias.cob "checkout -b"
git config --global alias.ci commit
git config --global alias.cm '!git add -A && git commit -m'
git config --global alias.st status
git config --global alias.br branch
git config --global alias.bra "branch -a"

Now you can write things like:

git co instead of git checkout
git st instead of git status
git br instead of git branch
git cm "My change" instead of git add -A && git commit -m "My change"

Start coding

There is a SAFE dojo repository you can clone to get started.

cd ~/src
git clone https://github.com/CompositionalIT/SAFE-Dojo.git
cd SAFE-Dojo
./build.sh run

Now just wait for the build to finish and your browser will open automatically to show the page in all its glory. Now you can go to the instructions to work your way through the dojo.

Resources

Stand-alone scripts with FAKE CLI

2018-06-15T00:00:00+00:00

Is your bash-fu not cutting it? Why not create executable F# script files instead? In this post we will go through how to make executable .fsx scripts and how to include NuGet dependencies. We’ll also be using F# interactive during development.

Prerequisites

Visual Studio Code
Ionide extension
Mono
F# compiler
Paket
FAKE CLI
OS with shebang support

Create an F# script file

To get started with F# FSI REPL, we’re going to create an F# script file with the .fsx extension. Opening this in VSCode+Ionide will give you intellisense, which makes everything much easier. Writing directly into F# interactive is not very intuitive.

Now go ahead and create a new file called Script.fsx somewhere (the name doesn’t really matter). We’re going to try to write some code for navigating a .fsproj XML file to look for e.g. which target frameworks are defined. Maybe we’ll try to edit as well.

Now to start writing a parser for a project file, we need an example. Put the following example into your script file:

let fsprojContents = """<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="Library.fs" />
  </ItemGroup>

</Project>"""

Now you can press Ctrl+Shift+P and choose FSI: Send File. This will open an F# interactive pane at the bottom with something like this:

val fsprojContents : string =
  "<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <Tar"+[143 chars]

When sending things to the FSI, it usually cuts long values like this and writes the number of chars left, e.g. +[143 chars], instead. To view the full value, write the name of the variable in the FSI window followed by ;;.

> fsprojContents;;
val it : string =
  "<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="Library.fs" />
  </ItemGroup>

</Project>"

Let’s code

To parse the XML we’re going to use the .NET XML parser located in System.Xml. To make that available, simply open it at the top of your fsx file like any other source file. Now add something like this to create an XmlDocument from the fsprojContents.

open System.Xml

let fsprojContents = ...

let loadXml (text : string) =
    let doc = XmlDocument()
    doc.LoadXml text
    doc

let xml =
    loadXml fsprojContents

Now you can mark all the text using Ctrl+A and press Alt+Enter to send all selected text to the FSI. Now we have to try to find some XML nodes to see if we can navigate the XML.

let tryGetNode name (node : XmlNode) =
    let xpath = sprintf "%s" name
    match node.SelectSingleNode(xpath) with
    | null -> None
    | n -> Some(n)

let projectNode =
    xml
    |> tryGetNode "Project"

let projectNodeName =
    tryGetProjectNode
    |> Option.map (fun x -> x.Name)

Sending this to the FSI should print:

val projectNodeName : string option = Some "Project"

Allright! Now we’ll just have to add some navigation functions to find the TargetFramework and/or TargetFrameworks nodes. It’s actually supported to have them both at the same time, but then TargetFramework overrides the plural form.

Now we’ll just add some functions for getting those nodes:

let tryGetNodeInPropertyGroup nodeName (node : XmlNode) =
    node
    |> tryGetNode "Project"
    |> Option.bind (tryGetNode "PropertyGroup")
    |> Option.bind (tryGetNode nodeName)

let tryGetTargetFramework (node : XmlNode) =
    node
    |> tryGetNodeInPropertyGroup "TargetFramework"

let tryGetTargetFrameworks (node : XmlNode) =
    node
    |> tryGetNodeInPropertyGroup "TargetFrameworks"

let getNodeValue (node : XmlNode) =
    node.InnerText

Making the script executable

Now we’re going to do something cool. We’re going to make the script executable to make it into an F# bash script. This way we can test the code using the CLI. Add the following shebang at the top of your .fsx and file and the following code to get a hold of the arguments passed to the script:

#!/bin/sh
#if run_with_bin_sh
  exec fsharpi --exec $0 $*
#endif

/// Skip the 4 first arguments, since they are just the script file name etc.
/// [|"/usr/lib/mono/fsharp/fsi.exe"; "--exename:fsharpi"; "--exec"; "./Script.fsx"|]
let argv =
    System.Environment.GetCommandLineArgs().[4..]

// ... all the previous code goes here ...

let targetFramework =
    argv.[0]
    |> System.IO.File.ReadAllText
    |> loadXml
    |> tryGetTargetFramework
    |> Option.map getNodeValue

printfn "TargetFramework: %A" targetFramework

You can now make this script file executable and run it with a path to an .fsproj file.

$ chmod +x Script.fsx
$ ./Script.fsx some/project.fsproj
TargetFramework: netstandard2.0

Changing the target frameworks

To change the target framework we need some code for updating the XML and saving it to file. We also need to take the new target framework as an argument:

let setNodeValue value (node : XmlNode)=
    node.InnerText <- value

let readXmlDocument path =
    path
    |> System.IO.File.ReadAllText
    |> loadXml

let xmlDocument =
    readXmlDocument argv.[0]

xmlDocument
|> tryGetTargetFramework
|> function
    | Some node ->
        setNodeValue argv.[1] node
        xmlDocument.Save argv.[0]
        printfn "'TargetFramework' for project '%s' changed to '%s'" argv.[0] argv.[1]
    | None ->
        printfn "Unable to find 'TargetFramework' tag"

The following command will now change the target framework of project.fsproj to netstandard2.1. Success!

$ ./Script.fsx some/project.fsproj netstandard2.1
'TargetFramework' for project 'some/project.fsproj' changed to 'netstandard2.1'

Adding dependencies

If we want to be more fancy on the input arguments we can use Argu, but how can you do that in a script?

Manually add references to .lib files downloaded by e.g. Paket
With the help from FAKE CLI’s integration with Paket

Manually (with Paket)

If you haven’t already, add the following alias to your ~/.bash_aliases file:

alias paket='mono .paket/paket.exe'

Then run the following commands to download Paket and make it download Argu to a packages folder next to your script.

$ mkdir .paket
$ wget -O .paket/paket.exe https://github.com/fsprojects/Paket/releases/download/5.172.2/paket.bootstrapper.exe
$ paket init
$ paket add Argu

Then below the shebang in your script add the following to reference the library:

#I __SOURCE_DIRECTORY__ // Makes sure relative paths in #r statements starts from here.
#r "packages/Argu/lib/net45/Argu.dll"

open Argu

This, however, is a little brittle, since it requires you to have the packages folder follow your script and all paths to be correct, so now I will show you how to make a stand alone script using FAKE CLI instead.

FAKE CLI

First you have to install FAKE CLI as a global dotnet tool:

$ dotnet tool install fake-cli -g

Now you have fake as a command line tool which has a special Paket integration using some special #r statements. Now we’ll change two things, first the shebang to invoke fake cli instead of fsi and add Argu via the special #r syntax. So replace the top of your script file from

#!/bin/sh
#if run_with_bin_sh
  exec fsharpi --exec $0 $*
#endif

#I __SOURCE_DIRECTORY__
#r "packages/Argu/lib/net45/Argu.dll"

open Argu

let argv =
    Environment.GetCommandLineArgs().[4..]

#!/bin/sh
#if run_with_bin_sh
  exec fake run $0 $*
#endif

#r "paket:
nuget Argu
//"

// Make sure to match the name of your script file, since
// FAKE CLI creates a .fake/{script_name}.fsx folder next to this script
#load "./.fake/Script.fsx/intellisense.fsx"

open Argu

let argv =
    Environment.GetCommandLineArgs().[3..]

Now we can use Argu to parse our command line arguments by adding something like this at the bottom of our script:

type Arguments =
    | [<Mandatory;CustomCommandLine("change")>] ProjectPath of ProjectPath:string
    | [<Mandatory;CustomCommandLine("targetframework")>] TargetFramework of TargetFramework:string
with
    interface IArgParserTemplate with
        member s.Usage =
            match s with
            | ProjectPath _ -> "specify the project file to update"
            | TargetFramework _ -> "specify the target framework to set"

let errorHandler =
        ProcessExiter(colorizer =
            function
            | ErrorCode.HelpText -> None
            | _ -> Some System.ConsoleColor.Red)

let argsParser =
    ArgumentParser.Create<Arguments>(programName = "./Script.fsx", errorHandler = errorHandler)

let args =
    argsParser.Parse argv

if args.GetAllResults() |> List.length <> 2 then
    argsParser.PrintUsage()
    |> printfn "%s"
    1
else
    let projectPath =
        args.GetResult (<@ ProjectPath @>)

    let targetFramework =
        args.GetResult (<@ TargetFramework @>)

    let xmlDocument =
        projectPath
        |> readXmlDocument

    xmlDocument
    |> tryGetTargetFramework
    |> function
        | Some n ->
            setNodeValue targetFramework n
            xmlDocument.Save projectPath
            printfn "'TargetFramework' for project '%s' changed to '%s'" projectPath targetFramework
            0
        | None ->
            printfn "Unable to find 'TargetFramework' tag"
            1

Now we have an argument list that resembles dotnet CLI and has a nice help:

$ ./Script.fsx
ERROR: missing parameter 'change'.
USAGE: ./Script.fsx [--help] change <ProjectPath> targetframework <TargetFramework>

OPTIONS:

    change <ProjectPath>  specify the project file to update
    targetframework <TargetFramework>
                          specify the target framework to set
    --help                display this list of options.

So we try again with proper arguments:

$ ./Script.fsx change some/Project.fsproj targetframework net461
'TargetFramework' for project 'some/project.fsproj' changed to 'net461'

And if you have a bunch of project files you want to update you can leverage bash like this:

$ find . -name "*.fsproj" -type f -exec ./Script.fsx change {} targetframework net461 \;

where {} is the placeholder for the find command’s results and \; means it invokes the script once for every matching file.

Adding more dependencies

To add more dependencies to your script, simply add them in the #r "paket: //" block, then delete the Script.fsx.lock that has been generated and re-run the script. FAKE CLI will then download the new dependencies before running the rest of the script.

#r "paket:
nuget Argu
nuget FSharp.Data
//"

That’s it for now!

Setting up multi-targeting build with Mono

2018-03-07T00:00:00+00:00

In this post I’m going to show how I did my first PR to an open-source project, adding support for multi-target builds using Mono for FSUnit.

Finding an issue

I have always wanted to contribute to an open-source project. I’ve found myself many times looking at the issues list for tools I have used, unfortunately I never managed to actually do it. Then a tweet from Sergey Thion where he asked if anyone would try Don Syme’s suggestion for how to do multi-target builds using dotnet CLI and Mono. I thought why not, I’m coding F# on Linux and find it much simpler to maintain Linux dockers instead of Windows VMs for CI build agents.

Sergey had already created an issue in his project FSUnit, which I said I was willing to try to fix for him. He was very greatful and I was feeling … semi confident :)

Step 1: Forking the project

Since you don’t have write access to someone elses repository, the first thing you’ll have to do is fork the repository to you own account. So, first I logged on to GitHub with my own account, then I moved to the fsprojects/FsUnit repository and pressed the Fork button in the top right corner.

This created a copy of FSUnit to my account, atlemann/FSUnit.

Step 2: Cloning the fork

Next I cloned my copy of FSUnit (Note the path containing my username) to my machine.

Here I’m cloning the repository and opening it in VSCode:

~src$ git clone git@github.com:atlemann/FsUnit.git
Cloning into 'FsUnit'...
remote: Counting objects: 9766, done.
remote: Compressing objects: 100% (57/57), done.
remote: Total 9766 (delta 61), reused 83 (delta 53), pack-reused 9655
Receiving objects: 100% (9766/9766), 72.63 MiB | 4.85 MiB/s, done.
Resolving deltas: 100% (4796/4796), done.
Checking connectivity... done.
~src$ code FsUnit

This is how it looks:

The integrated terminal in VSCode is great. At least when combined with git branch annotations so you always know which branch you’re in.

Step 3: How do I build this thing?

Nothing is as depressing as wanting to contribute to something and not being able to easily compile the thing. Fortunately, Sergey is following one of the most common conventions in the F# world: build using FAKE, handle dependencies using Paket. In the tree we find the usual build.sh/build.cmd files which can be used to build on Linux and Windows respectivey. So here goes build.sh:

Success! However, wasn’t the task to add multi-target build on Mono? Let’s check what it built:

Here we’ll have to check the FsUnit.NUnit.fsproj file to see what’s going on.

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFrameworks>netstandard2.0;net46</TargetFrameworks>
  </PropertyGroup>
  <PropertyGroup Condition=" '$(OS)' != 'Windows_NT' ">
    <TargetFramework>netstandard2.0</TargetFramework>
  </PropertyGroup>
  <PropertyGroup>
    <AssemblyName>FsUnit.NUnit</AssemblyName>
    <EnableDefaultCompileItems>false</EnableDefaultCompileItems>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="AssemblyInfo.fs" />
    <Compile Include="FsUnit.fs" />
    <Compile Include="FsUnitTyped.fs" />
    <Compile Include="GenericAssert.fs" />
    <None Include="paket.references" />
    <None Include="paket.template" />
    <None Include="FsUnitSample.fs.pp">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Include="sample.paket.template" />
  </ItemGroup>
  <ItemGroup Condition=" '$(TargetFramework)' != 'netstandard1.6' ">
    <Reference Include="System" />
  </ItemGroup>
  <Import Project="..\..\.paket\Paket.Restore.targets" />
</Project>

Here we see <PropertyGroup Condition=" '$(OS)' != 'Windows_NT' "> makes sure only netstandard2.0 target is build if not on windows, which explains it. Lets try to remove it and see what happens:

/usr/share/dotnet/sdk/2.1.4/Microsoft.Common.CurrentVersion.targets(1124,5):
error MSB3644: The reference assemblies for framework
".NETFramework,Version=v4.6" were not found. To resolve this, install the SDK
or Targeting Pack for this framework version or retarget your application to
a version of the framework for which you have the SDK or Targeting Pack
installed. Note that assemblies will be resolved from the Global Assembly
Cache (GAC) and will be used in place of reference assemblies. Therefore
your assembly may not be correctly targeted for the framework you intend.
[/home/atle/src/FsUnit/src/FsUnit.NUnit/FsUnit.NUnit.fsproj]
    28 Warning(s)
    1 Error(s)

Time Elapsed 00:00:59.80
Running build failed.

This fails, as expected.

Step 4: Fixing the issue

There are three projects in the solution that have to be fixed:

FsUnit.NUnit
FsUnit.MsTestUnit
FsUnit.Xunit

First things first, now I will create a local branch to apply this fix:

atle@latle:~/src/FsUnit (master)$ checkout -b make.multitarget.build.on.linux
Branch make.multitarget.build.on.linux set up to track remote branch make.multitarget.build.on.linux from origin.
Switched to a new branch 'make.multitarget.build.on.linux'
atle@latle:~/src/FsUnit (make.multitarget.build.on.linux)$

In the issue posted by Sergey he links to a solution proposed by Don Syme, the creator of F#, so he probably knows what he’s talking about. The fix is to add the following content to a file in your repository, called e.g. netfx.props, and import it to the relevant .fsproj files.

<PropertyGroup>
  <!-- When compiling .NET SDK 2.0 projects targeting .NET 4.x on Mono using 'dotnet build' you -->
  <!-- have to teach MSBuild where the Mono copy of the reference asssemblies is -->
  <TargetIsMono Condition="$(TargetFramework.StartsWith('net4')) and '$(OS)' == 'Unix'">true</TargetIsMono>
    
  <!-- Look in the standard install locations -->
  <BaseFrameworkPathOverrideForMono Condition="'$(BaseFrameworkPathOverrideForMono)' == '' AND '$(TargetIsMono)' == 'true' AND EXISTS('/Library/Frameworks/Mono.framework/Versions/Current/lib/mono')">/Library/Frameworks/Mono.framework/Versions/Current/lib/mono</BaseFrameworkPathOverrideForMono>
  <BaseFrameworkPathOverrideForMono Condition="'$(BaseFrameworkPathOverrideForMono)' == '' AND '$(TargetIsMono)' == 'true' AND EXISTS('/usr/lib/mono')">/usr/lib/mono</BaseFrameworkPathOverrideForMono>
  <BaseFrameworkPathOverrideForMono Condition="'$(BaseFrameworkPathOverrideForMono)' == '' AND '$(TargetIsMono)' == 'true' AND EXISTS('/usr/local/lib/mono')">/usr/local/lib/mono</BaseFrameworkPathOverrideForMono>

  <!-- If we found Mono reference assemblies, then use them -->
  <FrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != '' AND '$(TargetFramework)' == 'net45'">$(BaseFrameworkPathOverrideForMono)/4.5-api</FrameworkPathOverride>
  <FrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != '' AND '$(TargetFramework)' == 'net451'">$(BaseFrameworkPathOverrideForMono)/4.5.1-api</FrameworkPathOverride>
  <FrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != '' AND '$(TargetFramework)' == 'net452'">$(BaseFrameworkPathOverrideForMono)/4.5.2-api</FrameworkPathOverride>
  <FrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != '' AND '$(TargetFramework)' == 'net46'">$(BaseFrameworkPathOverrideForMono)/4.6-api</FrameworkPathOverride>
  <FrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != '' AND '$(TargetFramework)' == 'net461'">$(BaseFrameworkPathOverrideForMono)/4.6.1-api</FrameworkPathOverride>
  <FrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != '' AND '$(TargetFramework)' == 'net462'">$(BaseFrameworkPathOverrideForMono)/4.6.2-api</FrameworkPathOverride>
  <FrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != '' AND '$(TargetFramework)' == 'net47'">$(BaseFrameworkPathOverrideForMono)/4.7-api</FrameworkPathOverride>
  <FrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != '' AND '$(TargetFramework)' == 'net471'">$(BaseFrameworkPathOverrideForMono)/4.7.1-api</FrameworkPathOverride>
  <EnableFrameworkPathOverride Condition="'$(BaseFrameworkPathOverrideForMono)' != ''">true</EnableFrameworkPathOverride>

    <!-- Add the Facades directory.  Not sure how else to do this. Necessary at least for .NET 4.5 -->
  <AssemblySearchPaths Condition="'$(BaseFrameworkPathOverrideForMono)' != ''">$(FrameworkPathOverride)/Facades;$(AssemblySearchPaths)</AssemblySearchPaths>
</PropertyGroup>

FsUnit.NUnit

Now I changed the FsUnit.NUnit.fsproj to look like this:

<Project Sdk="Microsoft.NET.Sdk">
  <Import Project="..\..\netfx.props" />
  <PropertyGroup>
    <TargetFrameworks>netstandard2.0;net46</TargetFrameworks>
  </PropertyGroup>
  <PropertyGroup>
    <AssemblyName>FsUnit.NUnit</AssemblyName>
    <EnableDefaultCompileItems>false</EnableDefaultCompileItems>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="AssemblyInfo.fs" />
    <Compile Include="FsUnit.fs" />
    <Compile Include="FsUnitTyped.fs" />
    <Compile Include="GenericAssert.fs" />
    <None Include="paket.references" />
    <None Include="paket.template" />
    <None Include="FsUnitSample.fs.pp">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Include="sample.paket.template" />
  </ItemGroup>
  <Import Project="..\..\.paket\Paket.Restore.targets" />
</Project>

I basically removed that condition for Windows_NT and added an import for the netfx.props in the second line:

<Import Project="..\..\netfx.props" />

Now we’ll try to build this project only, since it’s the only one that has been fixed:

$ dotnet build src/FsUnit.NUnit/FsUnit.NUnit.fsproj
...
Build succeeded
...

Success! For real this time:

FsUnit.MsTestUnit

I added the same fix there, but got the following error when trying to build:

$ dotnet build src/FsUnit.MsTestUnit/FsUnit.MsTest.fsproj
...
...
/home/atle/src/FsUnit/src/FsUnit.MsTestUnit/FsUnit.fs(14,20):
error FS1108: The type 'Exception' is required here and is
unavailable. You must add a reference to assembly 'System.Runtime,
Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a'.
[/home/atle/src/FsUnit/src/FsUnit.MsTestUnit/FsUnit.MsTest.fsproj]

As Don Syme mentions in his comment, he had to add explicit references to some facade assemblies, System.Runtime is one of those, so we’ll add it to FsUnit.MsTest.fsproj and see what happens:

<Project Sdk="Microsoft.NET.Sdk">
  <Import Project="..\..\netfx.props" />
  <PropertyGroup>
    <TargetFrameworks>netstandard2.0;net46</TargetFrameworks>
  </PropertyGroup>
  <PropertyGroup>
    <AssemblyName>FsUnit.MsTest</AssemblyName>
    <EnableDefaultCompileItems>false</EnableDefaultCompileItems>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="AssemblyInfo.fs" />
    <Compile Include="..\FsUnit.Xunit\CustomMatchers.fs">
      <Link>CustomMatchers.fs</Link>
    </Compile>
    <Compile Include="FsUnit.fs" />
    <None Include="paket.references" />
    <None Include="paket.template" />
    <None Include="FsUnitSample.fs.pp">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Include="sample.paket.template" />
  </ItemGroup>
  <ItemGroup>
    <Reference Include="System.Runtime" />
  </ItemGroup>
  <Import Project="..\..\.paket\Paket.Restore.targets" />
</Project>

$ dotnet build src/FsUnit.MsTestUnit/FsUnit.MsTest.fsproj
...
Build succeeded

Success again!

FsUnit.Xunit

Adding netfx.props and building gives a similar error message:

$ dotnet build src/FsUnit.Xunit/FsUnit.Xunit.fsproj
...
...
The type 'Exception' is required here and is unavailable. You must add a reference to assembly 'System.Runtime...'
The type 'Object' is required here and is unavailable. You must add a reference to assembly 'System.Runtime...'

We add the same reference as for MsTestUnit and build again:

$ dotnet build src/FsUnit.Xunit/FsUnit.Xunit.fsproj
...
...
The type 'TypeInfo' is required here and is unavailable. You must add a reference to assembly 'System.Reflection

Now we add the a reference to System.Reflection and build again:

$ dotnet build src/FsUnit.Xunit/FsUnit.Xunit.fsproj
...
...
Build succeeded

The FsUnit.Xunit.fsproj now looks like this:

<Project Sdk="Microsoft.NET.Sdk">
  <Import Project="..\..\netfx.props" />
  <PropertyGroup>
    <TargetFrameworks>netstandard2.0;net46</TargetFrameworks>
  </PropertyGroup>
  <PropertyGroup>
    <AssemblyName>FsUnit.Xunit</AssemblyName>
    <EnableDefaultCompileItems>false</EnableDefaultCompileItems>
  </PropertyGroup>
  <ItemGroup>
    <Compile Include="AssemblyInfo.fs" />
    <Compile Include="CustomMatchers.fs" />
    <Compile Include="FsUnit.fs" />
    <None Include="paket.references" />
    <None Include="paket.template" />
    <None Include="FsUnitSample.fs.pp">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Include="sample.paket.template" />
  </ItemGroup>
  <ItemGroup>
    <Reference Include="System.Runtime" />
    <Reference Include="System.Reflection" />
  </ItemGroup>
  <Import Project="..\..\.paket\Paket.Restore.targets" />
</Project>

Finally we should run build.sh again, since it will build all projects and run all tests, just to make sure everything works.

Finished Target: All

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
Target              Duration
------              --------
Clean               00:00:00.0449729
AssemblyInfo        00:00:00.0276845
InstallDotNetCore   00:00:00.1632733
Build               00:01:02.8727158
CopyBinaries        00:00:00.0349621
NUnit               00:00:11.8522380
xUnit               00:00:12.2566259
RunTests            00:00:00.0001697
All                 00:00:00.0000695
Total:              00:01:27.3429696
---------------------------------------------------------------------
Status:             Ok
---------------------------------------------------------------------

Step 4: Submitting a PR

Now that we have made the changes to resolve the issue, we have to open a pull request to the original repository for the owner to merge. First I’ll commit my changes and push my local branch to my remote repository, also known as origin.

I find VSCode's git integration to be pretty good, so I usually stage files and write commit messages using it. (This shows my second commit, by the way)

Then we need to push it:

$ git push origin make.multitarget.build.on.linux

and then you’ll get a pull request button in your fork:

When pressing the Compare & pull request button, you will be sent to the original repository for opening a PR, where you can write some details on what is done. By adding Fixes: #IssueNr, GitHub will automatically close the issue when the PR is closed, which is convenient.

Now press the Create pull request button and your contribution is given to the owner of the repository.

And that’s it! My first PR to an open source project. And you should now be able to do multi-target builds using Mono for your own projects as well.

Creating F# solutions in VSCode from scratch

2018-02-28T00:00:00+00:00

Now that we have set up our development environment, it’s about time to get started with the coding. In this post we are going to get to know different ways of creating a new F# solution from scratch.

Tools for working with .NET projects:

Forge via Ionide
dotnet CLI
By hand ;)
Paket
FAKE

We’re going to do the following things using those different tools:

Create a console app
Create a class library
Adding a .fs file
Adding project reference from class lib to console app
Building
Running the console app
Adding NuGet packages
Adding test project
Running tests
Debugging

This will be a 2 part tutorial where we will look at how to do all these steps using two different toolsets, first using the combo Ionide, Forge and Paket and after we do the same thing using dotnet CLI and by hand.

Scaffolding using Ionide, Forge and Paket

1 Create a console app

The first thing you’ll have to do is to create a root folder for your new project and open VSCode in that folder:

$ cd ~/src
$ mkdir MyForgeProject
$ code MyForgeProject

Now you’re going to see a very emtpy VSCode instance without anything in it. Now, to create your project, hit Ctrl+Shift+P and start typing new project.

Choose F#: New Project and choose console from the menu that appears.

First, Forge will ask you which folder to put your new project folder in and next the name of your project:

Your project tree will now look like this:

Alternatively use Ionide's F# Project Explorer and press the green plus sign for creating a new project.

The forge template uses Paket for package management and FAKE for build scripting. It also targets .NET Full framework, but we’re going to change that to .NET Core by editing the MyForgeConsoleApp.fsproj file and replacing

<TargetFramework>net461</TargetFramework>

with

<TargetFramework>netcoreapp2.0</TargetFramework>

Paket is also restricted to resolving >= net461 dependencies, but we’ll have to remove that, since we’re now on .NET Core. Open the paket.dependencies file and remove the line:

framework: >= net461

Your paket.dependencies file should now look like this:

source https://www.nuget.org/api/v2

nuget FSharp.Core

group Build
source https://www.nuget.org/api/v2

nuget FAKE

Since the framework restriction is now removed, you have to update the paket.lock file by calling Paket install via the Ctrl+Shift+P menu in VSCode.

2 Create a class library

To create a class library, open the VSCode Command palette (Ctrl+Shift+P) and choose F#: New Project or press the green plus, but this time choose classlib in the menu. Choose src as Project folder and MyForgeClasslib as Project name. Again, it targets net461, so go to src/MyForgeClasslib/MyForgeClasslib.fsproj and replace the full framework target with a netstandard target, since this is a class lib and not a console app:

<TargetFramework>net461</TargetFramework>

with

<TargetFramework>netstandard2.0</TargetFramework>

3 Adding a .fs file

Ionide has a menu for manipulating projects. Simply right-click the project you want to add a file to and select Add file

and choose a name

Since F# cares about the order of the files in your project, Ionide supports moving files up or down by right-clicking the file in the F# project explorer:

Please note that F# script files (.fsx) are not shown in the F# project explorer, but still might influence this command by standing “in the way” if they are included in the .fsproj file. Have the .fsproj file open while doing this and see what happens.

4 Adding a project reference

Now we want to add a project reference to the classlib in our console app. Open the VSCode Command palette (Ctrl+Shift+P) and write project reference and choose F#: Add Project Reference.

This will first ask you which project to edit and next the which project to reference. Choose the console app and classlib respectively.

Alternatively, right click Project References under MyForgeConsoleApp in the F# Project Explorer pane.

and you will be asked which project to reference.

Forge will now update your MyForgeConsoleApp.fsproj with a <ProjectReference>. Now you can go to your MyForgeConsoleApp.fs and replace it with the following contnet:

module MyForgeConsoleApp

open MyForgeClasslib

[<EntryPoint>]
let main argv =
    let x = MyForgeClasslib()

    printfn "%A" x.X
    0 // return an integer exit code

5 Building

There might be some red errors under the MyForgeClasslib entries, which means that we’ll have to build our project to get everything working. Since this template uses FAKE to build, run it by opening the VSCode Command palette (Ctrl+Shift+P) and writing build and selecting FAKE: Build Default or just pressing Ctrl+F5 as you can see here:

This should eventually give you a Build Time Report and Status: Ok. You might still see some red error markers under open MyForgeConsoleApp. If don’t dissapear by themselves after a little while, try opening your VSCode Command palette (Ctrl+Shift+P) and choose Reload window.

6 Running

Ionide can run the project for you. First you have to set your startup project by right clicking the project you want to debug. Set the MyForgeConsoleApp project as startup project and press the round green icon with a play sign inside. This will actually run it in debug mode.

This should open a terminal and show the output of the applicaion.

7 Adding NuGet packages

Now lets add the console app arguments parser Argu to our console app. First, open the project file to add it to by right clicking MyForgeConsoleApp in F# Project Explorer and choose Open project file from the menu. Then open the VSCode Command palette (Ctrl+Shift+P) and write add nuget package and choose Paket: Add NuGet Package (to current project) and write Argu in the box that appears. Tada! You can now add open Argu to the MyForgeConsoleApp.fs file and start parsing arguments.

8 Adding a test project

In the F# world, Expecto is the testing lib to use and Forge has a project template for that. Press the green plus in F# Project Explorer and choose expecto. Put it in tests folder and call it MyForgeExpecto. Change the <TargetFramework> to netcoreapp2.0 in the MyForgeExpecto.fsproj file. If you get any weird issues, try running Paket install again to update your paket.lock file.

9 Running tests

Ionide has buit in support for Expecto, so simply open VSCode Command palette (Ctrl+Shift+P) and write expecto and choose Expecto: Run or press Ctrl+F6. If you open the Tests.fs file you will now see a green and a red chemistry bottle type thing next to the passing and failing test respectively.

10 Debugging

Ionide has build in support for debugging. First you have to set your startup project by right clicking the project you want to debug. Set the MyForgeExpecto project as startup project, add a breakpoint on line 14 in Tests.fs (by clicking right below the red bottle) and press the round green icon with a play sign inside:

If you also press the debug icon in VSCode's sidebar, you should see the following:

Now you should hopefully be able to scaffold your own projects using Ionide and VSCode and start hacking.

Scaffolding using dotnet CLI

1 Creating a console app

Again, create the root folder for your new project and open VSCode.

$ cd ~/src
$ mkdir MyProject
$ code MyProject

Now hit Ctrl+` to open the integrated terminal, which is a great way to keep your CLI and code in the same window. Now check out which templates you have installed by typing:

$ dotnet new
...
Templates                                         Short Name       Language          Tags
----------------------------------------------------------------------------------------------------------------------------
Console Application                               console          [C#], F#, VB      Common/Console
Class library                                     classlib         [C#], F#, VB      Common/Library
SAFE-Stack Web App v0.4.0                         SAFE             F#                F#/Web/Suave/Fable/Elmish/Giraffe/Bulma
Simple Fable App                                  fable            F#                Fable
Unit Test Project                                 mstest           [C#], F#, VB      Test/MSTest
xUnit Test Project                                xunit            [C#], F#, VB      Test/xUnit
ASP.NET Core Empty                                web              [C#], F#          Web/Empty
ASP.NET Core Web App (Model-View-Controller)      mvc              [C#], F#          Web/MVC
ASP.NET Core Web App                              razor            [C#]              Web/MVC/Razor Pages
ASP.NET Core with Angular                         angular          [C#]              Web/MVC/SPA
ASP.NET Core with React.js                        react            [C#]              Web/MVC/SPA
ASP.NET Core with React.js and Redux              reactredux       [C#]              Web/MVC/SPA
ASP.NET Core Web API                              webapi           [C#], F#          Web/WebAPI
global.json file                                  globaljson                         Config
NuGet Config                                      nugetconfig                        Config
Web Config                                        webconfig                          Config
Solution File                                     sln                                Solution
Razor Page                                        page                               Web/ASP.NET
MVC ViewImports                                   viewimports                        Web/ASP.NET
MVC ViewStart                                     viewstart                          Web/ASP.NET
...

We’re going to create an F# Console Application by typing:

$ dotnet new console -lang F# -o src/MyConsoleApp

This will create the following tree of files:

2 Create a class library

Next we’re going to create a library project we can use from our awesome console app.

$ dotnet new classlib -lang F# -o src/MyLibrary
The template "Class library" was created successfully.

and your files tree should now look like this:

3 Adding a .fs file

Adding a new file to a project is as simple as right-clicking the folder you want to add it to and select New File.

Just call it NewFile.fs and we’ll have to manually add it to the .fsproj file above or below the existing file.

  <ItemGroup>
    <Compile Include="Library.fs" />
    <Compile Include="NewFile.fs" />
  </ItemGroup>

Now you’ll have to add some code to NewFile.fs to not break the build. E.g.:

module NewFile

    let add2 x = 2 + x

Now, the VSCode explorer doesn’t take this file ordering into consideration. For that you’ll have to look at Ionide’s F# Project Exploerer.

4 Adding project reference

By hand

With the new MSBuild SDK, it’s actually possible to edit .*proj files by hand. Now add a <ProjectReference> to your MyLibrary.fsproj in your MyConsoleApp.fsproj, which will now look like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
      <OutputType>Exe</OutputType>
      <TargetFramework>netcoreapp2.0</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
      <Compile Include="Program.fs" />
  </ItemGroup>

  <ItemGroup>
      <ProjectReference Include="..\MyLibrary\MyLibrary.fsproj" />
  </ItemGroup>

</Project>

With dotnet CLI

If you’d rather use dotnet CLI, the command works like this:

$ dotnet add <ProjectToAddReferenceTo> reference <ProjectToReference>

So in our case we will run the following from the root directory:

$ dotnet add src/MyConsoleApp/MyConsoleApp.fsproj reference src/MyLibrary/MyLibrary.fsproj
Reference `..\MyLibrary\MyLibrary.fsproj` added to the project.

5 Building

The dotnet CLI build command supports .sln files (we will create one later) or individual .*proj files. To build our console app, type the following:

$ dotnet build src/MyConsoleApp/MyConsoleApp.fsproj

This will create a bunch of files in src/MyConsoleApp/bin/Debug/netcoreapp2.0/. As you can see, .NET Core console apps aren’t by default compiled to an executable, but rather a .dll you have to run using dotnet CLI.

Building as a self-contained executable

.NET Core also supports compiling self-contained executables for a given runtime. Change the <PropertyGroup> of your MyConsoleApp.fsproj file to this:

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <RuntimeIdentifiers>ubuntu.16.04-x64</RuntimeIdentifiers>
  </PropertyGroup>

Now you can build your self-contained app by typing this:

$ dotnet publish -c Release -r ubuntu.16.04-x64

Now take a look in src/MyConsoleApp/bin/Release/netcoreapp2.0/ubuntu.16.04-x64/. There you have a MyConsoleApp executable for Ubuntu!

6 Running

Now that we have added a reference to the class library, we can try to use it. Open Program.fs and Library.fs and change the code to something like this and build it again:

With dotnet CLI

We now have three ways of running the console app using the command line:

dotnet run -p src/MyConsoleApp/MyConsoleApp.fsproj <args>
dotnet src/MyConsoleApp/bin/Debug/netcoreapp2.0/MyConsoleApp.dll <args>
src/MyConsoleApp/bin/Release/netcoreapp2.0/ubuntu.16.04-x64/MyConsoleApp <args>

So for example:

$ dotnet run -p src/MyConsoleApp/MyConsoleApp.fsproj Scott
Hello Scott

With VSCode

Press F5 and you will see the following menu popping up where you should select .NET Core:

This will create a folder called .vscode with a file called launch.json with the following content (and then some).

"version": "0.2.0",
    "configurations": [
        {
            "name": ".NET Core Launch (console)",
            "type": "coreclr",
            "request": "launch",
            "preLaunchTask": "build",
            "program": "${workspaceFolder}/bin/Debug/<insert-target-framework-here>/<insert-project-name-here>.dll",
            "args": ["Scott"],
            "cwd": "${workspaceFolder}",
            "console": "internalConsole",
            "stopAtEntry": false,
            "internalConsoleOptions": "openOnSessionStart"
        },

You’ll have to add /src/MyConsoleApp and edit the <insert-target-framework-here> and <insert-project-name-here> to the target framework in the MyConsoleApp.fsproj file and the name of the console app dll itself. To get the same output as above, add Scott to the args entry :

"program": "${workspaceFolder}/src/MyConsoleApp/bin/Debug/netcoreapp2.0/MyConsoleApp.dll",
"args": ["Scott"],

Press F5 again and you will get the following popup:

Choose Configure Task and VSCode will show you the following:

Just press enter, since there is only one option, and in the following menu, choose .NET Core.

This will create a tasks.json file next to the launch.json file in the .vscode folder, looking like this:

{
    "version": "2.0.0",
    "tasks": [
        {
            "label": "build",
            "command": "dotnet build",
            "type": "shell",
            "group": "build",
            "presentation": {
                "reveal": "silent"
            },
            "problemMatcher": "$msCompile"
        }
    ]
}

If you try to hit F5 again now, the build will fail, since there is no project defined. To fix this, add the MyConsoleApp.fsproj to the command:

"command": "dotnet build src/MyConsoleApp/MyConsoleApp.fsproj",

Now you should be able to press F5 and see the following in the built-in terminal:

If you like buttons, you could also go to the Debug pane in VSCode which now will list all configs in the launch.json file (which you can open by pressing that cogwheel) and press the green play button:

Now you can set breakpoints in your code and debug your (maybe first) .NET Core F# project in VSCode. How cool is that!

7 Adding NuGet reference

Now we’re going to add an arguments parser to our console app from NuGet. The goto arguments parser for F# is Argu.

By hand

Again, with the new MSBuild SDK, adding a NuGet package is as simple as adding a <PackageReference> to Argu in your MyConsoleApp.fsproj, which will now look like this:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>netcoreapp2.0</TargetFramework>
    <RuntimeIdentifiers>ubuntu.16.04-x64</RuntimeIdentifiers>
  </PropertyGroup>

  <ItemGroup>
    <Compile Include="Program.fs" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\MyLibrary\MyLibrary.fsproj" />
  </ItemGroup>

  <ItemGroup>
    <PackageReference Include="Argu" Version="5.0.1" />
  </ItemGroup>

</Project>

Now do a $ dotnet restore src/MyConsoleApp/MyConsoleApp.fsproj. If you are getting a warning like Detected package downgrade: FSharp.Core, add a <PackageReference> to FSharp.Core in the same <ItemGroup> as Argu, matching at least the minumum version required by Argu:

  <ItemGroup>
    <PackageReference Include="FSharp.Core" Version="4.3.2" />
    <PackageReference Include="Argu" Version="5.0.1" />
  </ItemGroup>

With dotnet CLI

dotnet CLI has a command for installing NuGet packages which looks like this:

$ dotnet add <ProjectToAddNuGetTo> package <NuGetPackageId>

So to add Argu to our console app we just type:

$ dotnet add src/MyConsoleApp/MyConsoleApp.fsproj package Argu

To fix the potential Detected package downgrade: FSharp.Core issue, you should in theory just run $ dotnet add src/MyConsoleApp/MyConsoleApp.fsproj package FSharp.Core, however, for me it fails with an error. So you’ll have to add FSharp.Core by hand in the MyConsoleApp.fsproj file as mentioned above or switch to Paket as explained below.

With Paket

Paket is an F# community open source package manager which fixes a lot of the issues NuGet has, e.g. global paket.dependencies and paket.lock files which globally defines which packages and versions are to be used and much more. First, download the paket.bootstrapper.exe and save it as <SolutionFolder>/.paket/paket.exe (yes, you are renaming it. See here for more info).

If you haven’t already added any packages, run the following to get started with Paket and add the Argu package to the MyConsoleApp project:

$ mono .paket/paket.exe init
$ mono .paket/paket.exe add --project src/MyConsoleApp/MyConsoleApp.fsproj Argu

If you’re already using NuGet and want to switch to Paket, run the following command and Paket will figure out which NuGet packages you are using and initialize itself accordingly:

$ mono .paket/paket.exe convert-from-nuget

Paket won’t give you the Detected package downgrade: FSharp.Core issue mentioned above, since it, by default, resolves the higest versions of transient dependencies, unlike NuGet which does the opposite. Paket also automatically hooks into dotnet restore, so you won’t have to do mono .paket/paket.exe restore to restore your packages when using Paket with dotnet CLI.

8 Adding a test project

In F#, Expecto is the goto project for unit-testing. There are two ways to set up a testing project using Expecto. By hand or by dotnet CLI template.

With dotnet CLI

First you need to create a console app, since Expecto is just a library that you can run from console. The second thing you’ll have to do is to add the Expecto NuGet package.

$ dotnet new console -lang F# -o tests/MyTests
$ dotnet add tests/MyTests/MyTests.fsproj package Expecto
$ dotnet restore tests/MyTests/MyTests.fsproj

Replace the contents of Program.fs with the following:

open Expecto

[<EntryPoint>]
let main argv =
    Tests.runTestsInAssembly defaultConfig argv

With dotnet CLI template

As you saw earlier, dotnet new did not show any template for Expecto, however, someone has created this for us. To install it type the following:

$ dotnet new -i Expecto.Template::*
...
...
Templates                                         Short Name       Language          Tags
----------------------------------------------------------------------------------------------------------------------------
Console Application                               console          [C#], F#, VB      Common/Console
Class library                                     classlib         [C#], F#, VB      Common/Library
SAFE-Stack Web App v0.4.0                         SAFE             F#                F#/Web/Suave/Fable/Elmish/Giraffe/Bulma
Simple Fable App                                  fable            F#                Fable
Expecto .net core Template                        expecto          F#                Test
Unit Test Project                                 mstest           [C#], F#, VB      Test/MSTest
xUnit Test Project                                xunit            [C#], F#, VB      Test/xUnit
ASP.NET Core Empty                                web              [C#], F#          Web/Empty
ASP.NET Core Web App (Model-View-Controller)      mvc              [C#], F#          Web/MVC
ASP.NET Core Web App                              razor            [C#]              Web/MVC/Razor Pages
ASP.NET Core with Angular                         angular          [C#]              Web/MVC/SPA
ASP.NET Core with React.js                        react            [C#]              Web/MVC/SPA
ASP.NET Core with React.js and Redux              reactredux       [C#]              Web/MVC/SPA
ASP.NET Core Web API                              webapi           [C#], F#          Web/WebAPI
global.json file                                  globaljson                         Config
NuGet Config                                      nugetconfig                        Config
Web Config                                        webconfig                          Config
Solution File                                     sln                                Solution
Razor Page                                        page                               Web/ASP.NET
MVC ViewImports                                   viewimports                        Web/ASP.NET
MVC ViewStart

Now you’ll se a new template called Expecto .net core Template that we will create by typing:

$ dotnet new expecto -o tests/MyTests

Your tree will now look something like this:

9 Running tests and debugging

Using dotnet CLI

Running the Expecto tests is as simple as running the console app created by the dotnet CLI template, like this:

$ dotnet run -p tests/MyTests/MyTests.fsproj
[21:17:53 INF] EXPECTO? Running tests... <Expecto>
...
...
[21:18:06 INF] EXPECTO! 8 tests run in 00:00:00.2157219 – 2 passed, 1 ignored, 5 failed, 0 errored. ( ರ Ĺ̯ ರೃ ) <Expecto>

Using VSCode

Now you’ll have to configure a build task and launch parameters for the Expecto project by first pressing the cogwheel here:

which opens launch.json. Add the following to the list of configurations:

{
    "name": ".NET Core Launch (tests)",
    "type": "coreclr",
    "request": "launch",
    "preLaunchTask": "build-tests",
    "program": "${workspaceFolder}/tests/MyTests/bin/Debug/netcoreapp2.0/MyTests.dll",
    "args": [""],
    "cwd": "${workspaceFolder}",
    "console": "internalConsole",
    "stopAtEntry": false,
    "internalConsoleOptions": "openOnSessionStart"
},

Here we have changed the name, preLaunchTask and the program entries. Now your list of debug targets should look like this:

Now select the .NET Core Lunch (tests) entry and press the green play button and you will get a popup saying Could not find the preLaunchTask 'build-tests'.

Just select Configure Task again and choose build in the next popup, which will open the tasks.json again. Now add the following task to the list of tasks:

{
    "label": "build-tests",
    "command": "dotnet build tests/MyTests/MyTests.fsproj",
    "type": "shell",
    "group": "test",
    "presentation": {
        "reveal": "silent"
    },
    "problemMatcher": "$msCompile"
}

As you can see, the label matches the preLaunchTask in launch.json. Now try running the .NET Core Lunch (tests) again and Expecto should now start in debug mode.

But where is the solution file?

With .NET Core and dotnet CLI the .sln file is not really necessary anymore, however, it is still supported and it can make the workflow a bit simpler by having dotnet CLI just run the .sln instead of having to point to the specific .*proj files for building. So, to create a solution file we must do the following:

Create the solution file
Add all projects to it

Both can be done using dotnet CLI like this:

$ dotnet new sln -n MyApp
$ dotnet sln add src/MyConsoleApp/MyConsoleApp.fsproj src/MyLibrary/MyLibrary.fsproj tests/MyTests/MyTests.fsproj
Project `src/MyConsoleApp/MyConsoleApp.fsproj` added to the solution.
Project `src/MyLibrary/MyLibrary.fsproj` added to the solution.
Project `tests/MyTests/MyTests.fsproj` added to the solution.

Now to build the application, simply type the following from the root folder of your solution (where the MyApp.sln file is located):

$ dotnet build

Now you can go into tasks.json and replace the two separate build and build-tests tasks with a single default one which looks like this:

{
    "label": "build",
    "command": "dotnet build",
    "type": "shell",
    "group": "build",
    "presentation": {
        "reveal": "silent"
    },
    "problemMatcher": "$msCompile"
}

See that the project path is removed from the command. Now you can open launch.json and change the preLaunchTasks for both configurations to the same build task.

{
    "name": ".NET Core Launch (console)",
    "type": "coreclr",
    "request": "launch",
    "preLaunchTask": "build",
    "program": "${workspaceFolder}/src/MyConsoleApp/bin/Debug/netcoreapp2.0/MyConsoleApp.dll",
    "args": ["Scott"],
    "cwd": "${workspaceFolder}",
    "console": "internalConsole",
    "stopAtEntry": false,
    "internalConsoleOptions": "openOnSessionStart"
},
{
    "name": ".NET Core Launch (tests)",
    "type": "coreclr",
    "request": "launch",
    "preLaunchTask": "build",
    "program": "${workspaceFolder}/tests/MyTests/bin/Debug/netcoreapp2.0/MyTests.dll",
    "args": [""],
    "cwd": "${workspaceFolder}",
    "console": "internalConsole",
    "stopAtEntry": false,
    "internalConsoleOptions": "openOnSessionStart"
}

So the .sln file can actually be quite useful.

This ended up being quite long, but I hope you learned something along the way. So now you should be able to create great F# apps using VSCode on any platform using dotnet CLI and Ionide. See, it even has menus and a Play button!

Now go read this

F# on Ubuntu 16.04 from scratch

2018-01-13T19:55:44+00:00

I am a little late to the party, but here is how you can set up a full development environment for F# on .NET Core.

TL;DR

$ git clone https://github.com/atlemann/installscripts.git
$ cd installscripts
$ ./install-fsharp-devenv.sh

Go hack!

Requirements

Mono (for tooling and multi targeting)
FSharp compiler
.NET Core CLI
Visual Studio Code
Ionide VSCode extension

Installing the FSharp compiler and Mono

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys $ 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
$ echo "deb http://download.mono-project.com/repo/ubuntu xenial main" | sudo tee $ /etc/apt/sources.list.d/mono-official.list
$ sudo apt-get update
$ sudo apt-get install fsharp

Installing .NET Core on the system

$ curl https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor > microsoft.gpg
$ sudo mv microsoft.gpg /etc/apt/trusted.gpg.d/microsoft.gpg
$ sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-xenial-prod xenial main" > /etc/apt/sources.list.d/dotnetdev.list'
$ sudo apt-get update
$ sudo apt-get install dotnet-sdk-2.1.4

Make sure to install a dotnet SDK version > 2.0.0 to support resolving NuGet pre-release packages, which use SemVer v2, e.g. 1.2.3-beta.4.

Now test the CLI

To get started with .NET Core, simply use a dotnet CLI provided template. To list the available templates, simply write

$ dotnet new

and to create an F# console app and start coding, write

$ dotnet new console -lang F# -o ~/src/FSharpConsole
$ cd ~/src/FSharpConsole
$ dotnet run
Hello World from F#!

Now you can check the .fsproj file and see it targeting netcoreapp2.0.

Installing Visual Studio Code

Since we have already added the microsoft gpg key when installing the dotnet sdk, we only have to add the vscode repository to our sources list before installing the package:

$ sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main" > /etc/apt/sources.list.d/vscode.list'
$ sudo apt-get update
$ sudo apt-get install code

Adding the Ionide-fsharp extension

The Ionide-fsharp extension to VSCode gives you a better experience when working with F#. Install it like this:

$ code --install-extension Ionide.Ionide-fsharp

Font ligatures

Since programming in F# uses a lot of multi character symbols, many people like to use fonts with programming ligatures to make those symbols a single symbol instead. E.g. FiraCode.

$ curl -fsSL https://github.com/tonsky/FiraCode/releases/download/1.204/FiraCode_1.204.zip -o firacode.zip
$ unzip firacode.zip -d firacode
$ mkdir -p /usr/local/share/fonts/firacode
$ sudo cp firacode/ttf/*.ttf /usr/local/share/fonts/firacode
$ sudo fc-cache -fv

Now open VSCode and edit the settings.json file by hitting Ctrl+, and paste the following content:

"editor.fontFamily": "Fira Code",
"editor.fontSize": 14,
"editor.fontLigatures": true

Setting up Git branch info in terminal

One helpful trick when using Git is to show the current branch and status in the bash prompt whenever you are inside a folder with a git repository. The Git source code contains a script which can be used to show this information. To set it up, first fetch the git-prompt.sh script:

$ curl -fsSL https://raw.githubusercontent.com/git/git/master/contrib/completion/git-prompt.sh -o .git-prompt.sh

Then open ~/.bashrc in your favorite editor and add the following at the bottom:

# Git branch info in terminal prompt
. ~/.git-prompt.sh
export GIT_PS1_SHOWDIRTYSTATE=1
export GIT_PS1_SHOWCOLORHINTS=true
export PROMPT_COMMAND='__git_ps1 "\[\033[01;32m\]\u@\h\[\033[00m\]:\[\033[01;34m\]\w\[\033[00m\]" "\\\$ "'

There are some additional environment variables that can be set which are explained in .git-prompt.sh

Use .NET Core from a docker

The .NET Core SDK has been coming out in previews before the final release of a new version. To try out a new preview without removing a stable version, .NET Core supports different versions installed side-by-side. However, you now have to do some trickery to choose which version to use. Fanie Reynders has created some nice scripts to do this and Scott Hansleman has elaborated on how to use it.

Another approach is to use docker containers for running our dotnet CLI commands. This way, new previews can be tried out without installing them on your machine. We actually don’t have to install .NET Core on our machine at all. And as an added bonus, you will be making sure your code will work inside a Docker container.

First install docker by running the following commands:

$ sudo apt-get install curl
$ curl -fsSL get.docker.com -o get-docker.sh
$ sh get-docker.sh

Then pull the latest .NET Core image from docker hub. Now we can pull the latest tag to get the most up to date dotnet SDK. We will also pull the v.1 SDK just to show how this will work.

$ docker pull microsoft/dotnet:1-sdk
$ docker pull microsoft/dotnet:latest

Now, since aliases are too old-school, we are going to set up some bash functions to be able to work with the .NET Core dockers from the CLI. Add the following content to ~/.bash_aliases:

dotnet2() {
    docker run --rm -it -v $(pwd):/src -w /src microsoft/dotnet:latest dotnet "$@"
}
dotnet1() {
    docker run --rm -it -v $(pwd):/src -w /src microsoft/dotnet:1-sdk dotnet "$@"
}

Dockers require sudo access unless the current user is a member of the docker group. There are some security implications doing this described here.

To add a user to the docker group (and refreshing the groups without logging out and in again), type

$ sudo usermod -aG docker $USER
$ newgrp docker

Now from the command line we can just write this:

$ dotnet2 --version
2.1.4
$ dotnet1 --version
1.0.4

So, if a 3.0-preview will be released at some point, we could pull this image by its tag and try it out without installing it on our system.