F# for the cloud worker - part 1

This is the first blog post in a series about using the programming language F# for Cloud oriented tasks. I am new to using F# in practice and these series of blog posts are essentially a documentation of my explorations of F# as a language for building various cloud solutions. This starts at the beginning of learning F#.

F# is an open-source, cross-platform functional language. It has been around for about 15 years and since 2010 been a cross-platform language (used to be Windows-only before that). It is a part of .NET Core. It can also be compiled to Javascript (through Fable) and WebAssembly (through Bolero).

It started out as an implementation of the language OCaml for the .NET platform to bring a strongly typed functional language to .NET and has evolved on its own since then.

I am a fan of functional languages and the expressive power they usually bring and I do find that in other languages I use, static type information is a benefit for many solutions. F# being part of .NET there is obviously some usage in Azure in cloud space, but I mainly work with AWS and secondly GCP - so it is interesting to explore how useful it can be. The .NET platform is unknown territory for me though, so this is also a part of the learning effort.

There are some great material for F#, but it seems a lot of it assume familiarity with .NET, C#, maybe Azure and also perhaps working with Windows.

I am writing this from the point of view of a .NET newbie, not really any experience with C# and working on a MacBook Pro. Primary cloud platform target will be AWS and then some GCP - no Azure planned. Hopefully this can be useful for people without a .NET/C#/Azure background.

Before continuing, here is a few good links for material related to F#, well worth looking at for further exploration of F#:

Unfortunately, some of the reference links in the F# documentation at Microsoft are broken.

So let's get started!

Installation

The FSharp website has some links to get you started on various platforms (Linux, MacOS, Windows, FreeBSD), which essentially boils down to:

Install .NET Core SDK - download here

This is the minimum to get started with F# locally on your own computer. To get a feel for the language without installing anything (yet), one can go to Try F#, Repl.it and a few other places.

For a version of .NET Core, pick either 3.1 or possibly the 5.0 preview. The stable version is 3.1, but later we are going to touch on some new features of 5.0 for scripting.

If you have downloaded and installed the .NET Core SDK, then the command line tool dotnet will be available. Open a command-line window and run
```
dotnet --info
```

This will show some information about the .NET Core installation. And that's it for the installation of F# itself.

Running some F# code

F# has a bundled REPL (read eval print loop) in dotnet called F# Interactive, which can be used to type in F# code and/or run F# scripts. The dotnet toolset itself is though mainly focused on building and managing projects and solutions and not just simple scripts. In fact, when googling to look at different "Hello world" implementations in F# a lot of them would not work as simple scripts, but rather require you to build a console application. This introduces you to a slightly larger scope of concepts, which is not really optimal for a quick start in my opinion.

I also want to use F# for some simple scripts to start with and thus do not want all the pieces that are involved with building a complete application.

The REPL

First, let's start with some simple code using the F# Interactive REPL:

Initial steps with the F# REPL

We start the REPL using the command dotnet fsi
First just type in a simple "Hello cloud!" at the prompt (>), by calling the function printfn with the hello cloud string and press Enter.
We get a new prompt (-), which indicates that FSI expects more input. The expression entered is a complete F# expression, but it is necessary to tell FSI that we are done, which is indicated by using two semi-colons (;;).
The expression is evaluated and "Hello cloud!" is printed. Success!
After the result, FSI prints type information about the expression val it : unit = (). For now, skip the details here. You will see a pattern emerging after entering some expressions in FSI.

Worth noting is that the parameter to the printfn function is just separated by space from the function called. Second variation is to provide an additional parameter to printfn to construct the string.

First create a named value, using the let keyword. We use the identifier message to represent the string value "cloud".
F# uses type inference, so in many cases there is no need to declare the type of an identifier - F# knows the type from the context anyway.
Next we call the function printfn again, but in this case with two arguments - a string with a substitution pattern (%s) and the message identifier. The result from execution is our hello cloud string again.
Note that multiple parameters to the printfn function is just separated by space.

The third variation of our "Hello cloud!" is through defining a function of our own to call. This is what any non-trivial development would include anyway.

First use let keyword to create a named function called hello. This is just like in the simple value case, only that we also include a parameter to the function, in this case, the parameter named message.
Although we can technically define the function on a single line, in this case, we continue the definition of the function body in the next line. Note also that we indent the next line; similar to languages like Python F# uses indentation to indicate code blocks.
The function hello is then called with the parameter "cloud" and the result is printed out. Another success!

Script file

So we started with three ways to implement "Hello cloud!" using the F# Interactive REPL. Now the next step is to create a script file to execute it in.

To do that, we can create a file called Hello.fsx with the following content:

// This is our first hello cloud script. This is a comment line.
let hello message =
    printfn "Hello %s!" message

hello "cloud"

This script can then be run using the dotnet command:

dotnet fsi Hello.fsx

We can however do it a bit better and use shebang notation also:

#!/usr/bin/env dotnet fsi
// This is our first hello cloud script. This is a comment line.
let hello message =
    printfn "Hello %s!" message

hello "cloud"

Even though # is not a comment character in F#, FSI understands the shebang notation and this works just fine. Change the Hello.fsx file to be executable and then just run it:

>>> chmod u+x Hello.fsx
>>> ./Hello.fsx
Hello cloud!
>>>

Some notes:

If you search for executing F# script files, you may find references to using the command fsharpi. This seems to be the old way, before FSI became a proper .NET Core component. I found fsharpi on my computer, but that was from a previous installation of .NET Core and fsharpi started F# 4.5 and not F# 4.7, which was bundled with my .NET Core 3.1 installation and which I get when executing dotnet fsi.
There are also references to that you need to specify the --exec option when executing scripts. The documentation says this is to indicate to FSI that it should exit after executing the script. Presumably it would have stayed in the interpreter otherwise. However, it exits properly even without --exec, so I assume it knows nowadays when it is executing as a script and then properly exits afterwards anyway.

Command line arguments

It is all good to be able to execute a simple script, but how can I pass in command line arguments to the script? Turns out that there are different ways, depending on whether you are running the code as a script (through FSI) or as a real application. We focus on the script option now.

The F# Interactive docs specify that one can use fsi.CommandLineArgs in the script.

So one way to process the command line arguments is:

#!/usr/bin/env dotnet fsi
// Now with command line arguments!

let hello messages =
    for message in messages do
        printfn "Hello %s!" message

hello fsi.CommandLineArgs

We have changed the hello function to take a parameter called messages, since it can be multiple arguments on the command line. In fact what fsi.CommandLineArgs return is an array of strings. In order to process each of the messages provided, we iterate over the messages using a for loop - which is one way to iterate over a collection of values.

Note when we execute this, the output will be like this:

❯❯❯ ./Hello.fsx cloud worker
Hello ./Hello.fsx!
Hello cloud!
Hello worker!

So the first argument in the array is the command itself. One way to get rid of that is to make a slice of the original array, skipping the first element (with index 0) and create a new array that starts with the element at index 1 in fsi.CommandLineArgs:

#!/usr/bin/env dotnet fsi
// Now with command line arguments!

let hello messages =
    for message in messages do
        printfn "Hello %s!" message

hello fsi.CommandLineArgs.[1..]

Now we just get the arguments to the script itself.

❯❯❯ ./Hello.fsx cloud worker
Hello cloud!
Hello worker!

Another approach for iteration, which is more common than an explicit loop in many functional languages, is a pipeline approach:

#!/usr/bin/env dotnet fsi
// Now with command-line arguments!

let hello messages =
    messages |> Array.map (printfn "Hello %s!")

hello fsi.CommandLineArgs.[1..]

The output when executed will be the same as the previous version. There are a few concepts that are important here.

First, the |> (forward pipe operator) is an operator that takes the result of the expression on the left side and provides that as a parameter to the function on the right side. The result of the right hand side of |> is a function - expressions in F# can produce function results. A simple concrete example in the REPL:

> printfn "Hello %s!" "cloud"
- ;;
Hello cloud!
val it : unit = ()

> "cloud" |> printfn "Hello %s!"
- ;;
Hello cloud!
val it : unit = ()

>

So essentially the last function parameter to the function on the right side of |> is provided with the result of the expression on the left side of |>.

Second, the expression Array.map (printfn "Hello %s!") produces a function and this expression in itself has multiple functions (Array.map and printfn). Array.map represents a function map in the module Array. A module is a container for types, functions and other modules. Map functions operate on collections of data (in this case an array) and applies a function on each element of that collection. The result is a new collection of these results. A concrete example in the REPL:

> let arr = [| "AWS"; "GCP"; "Azure" |] // Define an array of strings
- ;;
val arr : string [] = [|"AWS"; "GCP"; "Azure"|]

> let hellostr str = "Hello " + str // Define function that concatenates a string with "Hello "
- ;;
val hellostr : str:string -> string

> hellostr "Test"
- ;;
val it : string = "Hello Test"

> Array.map hellostr arr // Apply hellostr function to all strings in array
- ;;
val it : string [] = [|"Hello AWS"; "Hello GCP"; "Hello Azure"|]

>

From this example, we can see that the function parameter in Array.map (printfn "Hello %s!") expression is (printfn "Hello %s!"). This is a function call that produces another function actually. In F# a function call that does not have all expected parameters produce another function which takes the missing parameters as arguments. Another example in the REPL:

> let hellostr = printfn "Hello %s!" // Define function hellostr based on function call to printfn with a missing parameter
- ;;
val hellostr : (string -> unit)

> hellostr "cloud"
- ;;
Hello cloud!
val it : unit = ()

>

This is an example of partial function application. In the expression Array.map (printfn "Hello %s!") we need the parenthesis to make sure that Array.map takes the result of printfn "Hello %s!" as the function to use - if there were no parenthesis, it would just try to use printfn as the function.

Playing around with the pipeline approach, that could be used for stripping away the first element of fsi.CommandLineArgs also:

#!/usr/bin/env dotnet fsi

let hello messages =
    messages |> Array.map (printfn "Hello %s!")

hello (fsi.CommandLineArgs |> Array.skip 1)

So far we have touched a bit on some F# functional concepts and done some simple variations of a Hello cloud-script. Next, it is time to actually connect and use some cloud services.

Connecting to AWS

In order to programmatically connect to AWS, we will need an AWS account and an AWS profile on the locale computer that can access that AWS account. If you do not have an AWS profile configured yet, you can look at this AWS documentation. For this you need the AWS CLI installed. If that is not yet installed, look at this documentation.

Now, before we can communicate with AWS services using F#, we also need the AWS SDK for .NET. If we were building a .NET application, this would be fairly easy to get, since the dotnet command-line tool can add packages, such as AWS SDK for .NET, to a project. This is not as obvious though when using scripts.

I managed to find workarounds for this via some googling, but I think the preferable option to me is the improved support for this that will come with .NET Core 5.0. This is still in preview only though as of today (August 2020), but I think I prefer to document what will be the way to do it than other workarounds though.

Install .NET Core 5.0 preview

This is the same install page as for .NET Core 3.1, just pick the .NET Core 5.0 preview instead. It will be installed in parallel with your 3.1 installation. Run dotnet --info after installation and you will see information for all .NET Core versions you have installed.

Access S3

Let us do a simple script to list the AWS S3 buckets in a configured AWS account - this is at least my typical "hello cloud" type of activity when I want to test some access to an AWS account.

So in this case let us do a script HelloS3.fsx to print names and creation date & time of S3 buckets in the account. In order to figure things out with AWS SDK for .NET if is useful to have a look at the reference documentation.

The AWS SDK for .NET consists of a number of separate packages, more or less one for each service. In our case, we will need two of them - AWSSDK.Core and AWSSDK.S3. These can be retrieved from the official .NET package management service, NuGet. With .NET Core 5.0 it is easy to include these in the script:

#!/usr/bin/env dotnet fsi --langversion:preview

// Get the AWS SDK packages/assemblies needed
#r "nuget: AWSSDK.Core"
#r "nuget: AWSSDK.S3"

The #r directive in FSI can be used to reference the assemblies needed. What is new in .NET Core 5.0 is that if the prefix "nuget:" is added, it will automatically download the package from NuGet, if needed. It seems other package managers can be added as well, but only NuGet is available by default.

AWS SDK calls to various AWS services, regardless of language, in many cases require a client handle - one for each AWS service used. So a starting point is to obtain this client handle for S3, in this case. We also set up our main function to execute here:

#!/usr/bin/env dotnet fsi --langversion:preview

// Get the AWS SDK packages needed
#r "nuget: AWSSDK.Core"
#r "nuget: AWSSDK.S3"

open Amazon.S3

let helloS3 () =
    let client = new AmazonS3Client()
    "dummy"

helloS3()

The helloS3 function does not have any parameters, so we need to use empty parenthesis both in the declaration and call to distinguish it from just a simple value. Leave them out and F# will complain about it.

The line open Amazon.S3 is similar to an import statement in some other languages - it makes the names from a module/namespace available in the current namespace. If open was not used, the namespace/module path would need to be specified completely, e.g. Amazon.S3.AmazonS3Client instead of just AmazonS3Client.
In the SDK, AmazonS3Client is a class. F# supports object programming, but the way programs are typically constructed are not the same as more object-oriented languages. It needs to be and is interoperable with other .NET languages. The expression new AmazonS3Client() is to create a new instance of that class.

In the helloS3 function there is a string "dummy" added. This is because F# does not like functions to end with a let expression, there must be another expression returning a result. So the dummy string is added here - it will be removed later. This is a functional program, but it does not do anything useful - running it will not generate any output - but should not give any error either - assuming that there is an AWS profile configured properly. If the profile you have created is not named "default", you can run the script with setting the AWS_PROFILE environment variable in the call to the script with the AWS profile name:

❯❯❯ AWS_PROFILE=myprofilename ./HelloS3.fsx

Next, we will define a function to get bucket info and return it to us. This will be one string per bucket. We start by adding a dummy function first:

#!/usr/bin/env dotnet fsi --langversion:preview

// Get the AWS SDK packages needed
#r "nuget: AWSSDK.Core"
#r "nuget: AWSSDK.S3"

open Amazon.S3

let getBucketsInfo (s3Client: AmazonS3Client) =
    [| "Bucket 1 info"; "Bucket 2 info" |]

let helloS3 () =
    let client = new AmazonS3Client()
    let bucketsInfo = getBucketsInfo client
    for bucketInfo in bucketsInfo do
        printfn "%s" bucketInfo

helloS3()

Note here that the parameter for the function getBucketsInfo has type information added. We will call methods on the S3 client handle, so F# will need to know the type and since it is statically typed, it will need to know this at compile time. So in this case we need to add type information when we implement the function for real. The main function helloS3 calls the function to get the result and then prints that out.

Running this should print the dummy info:

❯❯❯ AWS_PROFILE=myprofilename ./HelloS3.fsx
Bucket 1 info
Bucket 2 info
❯❯❯

To actually get the bucket information, the SDK has methods to list buckets. There are some difference when using the SDK for different platforms, .NET Core, other .NET, Unity as well as Xamarin. I was not aware of the differences initially and I picked the wrong method calls initially. For .NET Core, only the asynchronous AWS service calls are implemented, i.e. the calls that end with Async in the name. So the method ListBuckets is not available even though it is in the documentation, but ListBucketsAsync is available.

Both the synchronous and the asynchronous will have a result represented by a ListBucketsResponse. In the asynchronous case though, it will not be available directly. The (successful) response will contain a sequence of buckets, represented by the S3Bucket class.

So we can add the needed info to create a string with bucket info from an S3Bucket object through a function:

open Amazon.S3.Model

let getBucketInfo (bucket: S3Bucket) =
    sprintf "Name: %s created at %O" bucket.BucketName bucket.CreationDate

The S3Bucket class is in a different module, so we add an open Amazon.S3.Model to be able to reference that directly. The function sprintf is used to construct a new string based on the parameters provided with the formatting template string.

Now we come to the somewhat tricky part, which I struggled a bit with - doing asynchronous calls with the AWS SDK. I could find a few examples in C#, but none in F#. F# documentation for asynchronous programming helped a bit, but it was not crystal clear for this use case how to do this. I got this to work after some trial and error though in a reasonably compact way and the result was this for the full script:

#!/usr/bin/env dotnet fsi --langversion:preview

// Get the AWS SDK packages needed
#r "nuget: AWSSDK.Core"
#r "nuget: AWSSDK.S3"

open Amazon.S3
open Amazon.S3.Model

let getBucketInfo (bucket: S3Bucket) =
    sprintf "Name: %s created at %O" bucket.BucketName bucket.CreationDate

let listBuckets (s3Client: AmazonS3Client) =
    async {
        let! response = s3Client.ListBucketsAsync() |> Async.AwaitTask
        return response
    }

let helloS3 () =
    let client = new AmazonS3Client()
    let response = (listBuckets client) |> Async.RunSynchronously
    let bucketsInfo = (List.ofSeq response.Buckets) |> List.map getBucketInfo
    for bucketInfo in bucketsInfo do
            printfn "%s" bucketInfo
helloS3()

The asynchronous call is wrapped in what is called a computation expression of type async. A computation expression is a generalized way to describe certain behaviours or workflows. It is not a topic that needs to be understood in depth at this point. There is a lot of material on the topic here, for those who want to jump into this. This particular case of computation expression is a bit similar to promises/futures in other languages, but computation expressions is a more generic construct and there are other use cases as well.

Anyway, the code that should run asynchronously is wrapped in async { }. The keyword let! is used similarly to let, but there is not yet a result when the (asynchronous) function call returns. The return value from ListBucketsAsync is a Task object. This can then be sent to the function Async.AwaitTask which waits for the result produced. This is then returned using the return keyword, which is one way to get data out of computation expression. I had missed that this was needed explicitly initially, so I tried to just state the response itself on the last line, which did not work - no result provided to the caller. After re-reading the docuymentation I did find the return keyword and it worked out fine. The returned data is of type ListBucketResponse - which we do not need to specify explicitly.

In the ListBucketsResponse object there is a list of buckets in the Buckets property. However, this is not the same type of list as the default F# immutable list, so trying functions from the List module did not work. It is instead a System.Collections.Generic.List, which is a template list type for .NET. I struggled a bit to figure out how to work with this type of list and ended up with converting it to an F# List, with the List.ofSeq function. Then I could then use functions in the List module on the result.

Then the very final bit is that the asynchronous code needs to be triggered also and one way seems to be with the Async.RunSynchronously function call.

I am not sure if this is the most suitable approach, but it works for the happy case. Time will tell if there will be other approaches.

From a design perspective the listBuckets function only performs the call to ListBucketsAsync and other logic is done separately. This is to make a clear separation of code that has side effects (reading data from AWS) in our computation logic.

Closing remarks for part 1

Source code

The source code in the blog posts in this series are posted to this Github repository:

github.com/elzcloud/fsharp-for-cloud-worker

Finding F# information

I have listed a few resources that I have found earlier in this blog post. I much enjoyed a lot of the material at F# for fun and profit and the F# language site. The Microsoft documentation for F# is reasonably good, but has some room for improvement I think.

I still do not know much about .NET, so there is a learning curve there also and it would have been more helpful with more examples in F#, rather than almost exclusively in C#. This also goes for the AWS SDK for .NET.

Most of the Youtube videos and introduction posts to the language praise the language and that in many ways is better or more enjoyable to use than C#. I do not know about using C#, but I do find the experience with F# quite enjoyable, despite struggles with finding information or examples sometimes.

Development environment

I have not touched anything on development environment and tools. The main IDEs (integrated development environments) seems to be:

Visual Studio
Visual Studio Code (VS Code)
Jetbrains Rider

I have never used Visual Studio, but that seems to be available for multiple platforms nowadays. VS Code has plugins for F# and that seems to be pretty good as far as I can tell. If you use VS Code for other languages, that maay work well for you. Jetbrains Rider is a commercial IDE from Jetbrains and is part of the Intellij family of IDEs. Personally I am a big fan of their products and use them both at work and for private projects (GoLand, PyCharm, Intellij). So my choice for now is to try it out with Rider on a trial basis for now.

Rider is more oriented towards solutions/projects and not the standalone scripts that I do now and it does not handle the .NET 5.0 NuGet references in the script right now. Other than that, I am pretty happy with it so far.

TTP - Time to print

F# code is compiled and running a script triggers the just-in-time (JIT) compilation of the code. This means there is a bit of a cold start time. This remains to be seen how mmuch of an issue this will be when running plain scripts.

Does it spark joy?

There are numerous videos, blog posts and other material out there that talks about what programming languages to learn in , the N top languages according to some criteria etc. I do not see too often references to in such comparisons if the language or tooling/development workflow sparks joy. In contrast, many introductions to F# talk a bit about the joy of programming with the language.

I think F# looks like a good candidate for a bit of sparkle and I hope this will continue.

Next steps

If you like this, then feel free to continue with part 2 !

Functional Cloud