The Great Json Bake-Off

Read Time: 31 minutes

It is difficult to build much without needing to deal with json at some point. In the world of F# there are several options. Today I want to lay out some of the popular choices and offer some some compare and contrast. These libraries all have their own strengths and weaknesses, and bringing it all into a single place to highlight these is a useful point of reference for decision making and general understanding.

First, a couple disclaimers. What I’m presenting is not an exhaustive list of libraries, but it is what I have found are commonly referenced and used. The benchmarks provided are intended to be representative, but should be taken with a grain of salt (as with all benchmarks). Other systems and data may see slightly different results than what I’m showing here. Most of these libraries have the ability to support custom serialization/deserialization and all kinds of options. The goal is not to dive into a bunch of rabbit holes, but what does typical usage looks like. If there is another library you’d like to see, an error I made, or a performance aspect I missed, hit me up and I’ll make corrections/additions. The goal is to offer an honest representation, and it is important to get this as correct as possible. Second, the libraries I’ll be investigating are below. The first three libraries are F#-specific, the last two are the common json encoders in the .NET ecosystem that everyone knows.

The benchmarks are run on Linux, using .NET 7. Below are the library versions.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# BenchmarkDotNet
dotnet add package BenchmarkDotNet --version 0.13.4

# FSharpLu
dotnet add package Microsoft.FSharpLu.Json --version 0.11.7

# Thoth
dotnet add package Thoth.Json.Net --version 11.0.0

# Fleece (using System.Text.Json)
dotnet add package Fleece --version 0.10.0
dotnet add package Fleece.SystemTextJson --version 0.10.0

# Newtonsoft
dotnet add package Newtonsoft.Json --version 13.0.2

# System.Text.Json (with Discriminated Union support)
dotnet add package System.Text.Json --version 7.0.2
dotnet add package FSharp.SystemTextJson --version 1.1.23

In the spirit of testing common use cases, below are the data types I’ll be serializing/deserializing to/from json. I also include some code samples. This at least helps level-set the type of data being handled.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
// Discriminated Union and Record types
type Email = string

type Industry =
  | Education
  | Manufacturing
  | Finance
  | Retail
  | Other of string

type Stats = int * int * int

type Customer = {
  Id: Guid
  Name: string
  Created: DateTime
  Contacts: Email list
  Industry: Industry
  Stats: Stats
  Notes: string option
}

// int list example
let data :int list = [472; 4; 137; 599; 455; 923; 542; 457; 309; 338]

// string list example
let data: string list = ["abc"; "def"; "ghi"]

// record list example
let data: Customer list = [
  { Customer.Id = Guid.NewGuid()
    Name = "Foo Inc."
    Created = DateTime.UtcNow
    Contacts = [
      Email("user1@test.com")
      Email("user2@test.com") ]
    Industry = Industry.Finance
    Stats = 10, 20, 30
    Notes = Some("This is a note") } 
  { Customer.Id = Guid.NewGuid()
    Name = "Bar Inc."
    Created = DateTime.UtcNow
    Contacts = [
      Email("user1@test.com")
      Email("user2@test.com") ]
    Industry = Industry.Education
    Stats = 40, 50, 60
    Notes = None } ]

// single record example
let data: Customer = {
  Customer.Id = Guid.NewGuid()
  Name = "Foo Inc."
  Created = DateTime.UtcNow
  Contacts = [
    Email("user1@test.com")
    Email("user2@test.com") ]
  Industry = Industry.Finance
  Stats = 10, 20, 30
  Notes = Some("This is a note") }

One of the characteristics that really sticks out when serializing F# objects to json is how the library handles F#-specific types, namely Discriminated Unions (DU) and Tuples. This can be a point of taste, but I prefer the way FSharpLu, Thoth, and Fleece handle DUs. Even for these libraries there are are variations. FSharpLu and Thoth auto-encode without any extra effort, while Fleece requires writing encoders/decoders for anything beyond primitive types. This falls into the balance of more/less power and is a specific library design choice. Fleece is inspired by Aeson. This typically means more work upfront, but offers power regarding data representation. Without putting too fine a point on it, this can be a significant advantage.

In the spirit of investigating variations, I also include multiple Thoth methods. The first is Auto encoding/decoding, which is what most people will use. The second uses pre-cached encoders/decoders by type. Caching requires an extra step prior to usage, but can offer a performance boost. The third is manual encoding/decoding in a similar vein to Fleece. This allows control of the encoding/decoding json, and potentially a performance boost. It is nice to have this as an capability, although the Fleece ergonomics are a bit nicer.

Then there are the traditional .NET json libraries. Newtonsoft.Json auto-encodes out of the box, although the default encoding of some types feels awkward to my F# eyes. System.Text.Json also auto-encodes, except for Discriminated Unions. If you try to use it with a Discriminated Union, you’ll get a compile error. To overcome this, I use FSharp.SystemTextJson to add DU support. I think its great this library exists, but it does highlight a functionality gap for System.Text.Json.

What does all of this mean in practice? Time for some examples of where these libraries differ. Primitive types like int and string aren’t interesting; they all encode the same way, so basically “nothing to see here”. To that end, I’ll focus on the Customer record type specified above. I’ll start where they all encode data in the same manner. For reference, here is where they all match serialization (missing are Industry and Stats fields):

1
2
3
4
5
6
7
8
9
10
{
  "Id": "1d765765-9ad9-46e1-989e-b3e4fa2ff3e6",
  "Name": "Foo Inc.",
  "Created": "2023-03-03T16:49:07.8710678Z",
  "Contacts": [
    "user1@test.com",
    "user2@test.com"
  ],
  "Notes": "This is a note"
}

Using the above as a baseline, I can now dig into the serialization differences. First up is the Industry DU. As a refresher, Industry can be one of several fixed types (e.g. Finance, Manufacturing, etc) or an Other value with a custom string. Below are both encodings, broken down by library. FSharpLu, Thoth, and Fleece are the most intuitive. Fleece is a special case because I chose to encode Other as Other=<value> to show some customization (more on how that happens later), but Fleece could look however makes the most sense to you. The Newtonsoft and System.Text.Json serializations are a bit more verbose. It is a more complex type, so in general this is fine, but the representation does feel a little weird. I think when interacting with external systems this could potentially be an issue, but in the grand scheme of things, this isn’t a really a deal breaker.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
### Industry = Finance

# FSharpLu
  "Industry": "Finance"

# Thoth
  "Industry": "Finance"

# Fleece
  "Industry": "Finance"

# Newtonsoft.Json
  "Industry": {"Case": "Finance"}

# System.Text.Json
  "Industry": {"Case": "Finance"}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
### Industry = Other("Recycling")

# FSharpLu
  "Industry": {
    "Other": "Recycling"
  },

# Thoth
  "Industry": [
    "Other",
    "Recycling" ]

# Fleece
  "Industry": "Other=Recycling"

# Newtonsoft.Json
  "Industry": {
    "Case": "Other",
    "Fields": ["Recycling"]}

# System.Text.Json
  "Industry": {
    "Case": "Other",
    "Fields": ["Recycling"]}

Now on to the Stats tuple. Four of the five encode it as an array. From a data structure perspective this is interesting and mostly pragmatic representation that maps to other languages without too much hassle. Newtonsoft is interesting in that it has the most honest data representation, but feels kind of clunky when encoding. With that said, I prefer this method for the following reason. What I don’t show here is the case when the tuple is mixed types (e.g. int * int * string), in that case the other libraries show [10, 20, "30"], which could be an issue if the decoder expected an array to have a single type.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
### Stats = 10, 20, 30

# FSharpLu
  "Stats": [10, 20, 30]

# Thoth
  "Stats": [10, 20, 30]

# Fleece
  "stats": [10, 20, 30]

# Newtonsoft.Json
  "Stats": {"Item1": 10, "Item2": 20, "Item3": 30}

# System.Text.Json
  "Stats": [10, 20, 30]

Now that I’ve looked at the results, it is time to see how the serialization/deserialize happens. Libraries are more than just the results, they also include the developer ergonomics. In addition to caveats mentioned above, some of these libraries have multiple ways to do things, I’ve just picked what is most convenient for my tests. Now is a good time to mention the above type definitions were abbreviated for readability; its now to bring out all the gruesome details. The most notable blocks are the Fleece codec and Thoth decoder/encoders, both of which are for manual encoding support. I’ve commented where library-supporting code exists. You’ll may also notice static member vs. module code organization; this to to match the respective library conventions, even though within a singular codebase it falls on the side of inconsistency. Frankly, its a lot of extra code to add to a type, but that is the cost of hand encoding/decoding. It may be too much to stomach for some, but sometimes the flexibility is worth the effort. Just to be clear, the Thoth-specific decoder/encoder modules are only required for manual usage, when using Auto options, these are not necessary.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
type Email = string

/// Thoth manual decoder/encoder support
module Email =
  let decoder :Decoder<Email> =
    Decode.object (fun get -> (get.Required.Raw Decode.string))

  let encoder =
    Encode.string

type Industry =
  | Education
  | Manufacturing
  | Finance
  | Retail
  | Other of string
  with 
  /// Fleece json codec (decoder)
  static member jsonDecoder = function
    | JString "Education" -> Decode.Success Education
    | JString "Manufacturing" -> Decode.Success Manufacturing
    | JString "Finance" -> Decode.Success Finance
    | JString "Retail" -> Decode.Success Retail
    | JString x as v -> Decode.Success(Other(x.Replace("Other=", "")))
    | x -> Decode.Fail.strExpected x

  /// Fleece json codec (encoder)
  static member jsonEncoder = function
    | Education -> JString "Education"
    | Manufacturing -> JString "Manufacturing"
    | Finance -> JString "Finance"
    | Retail -> JString "Retail"
    | Other(x) -> JString $"Other={x}"

  /// Fleece json codec
  static member jsonCodec = Industry.jsonDecoder <-> Industry.jsonEncoder

  /// Fleece get_Codec
  static member get_Codec() =
    Industry.jsonCodec

/// Thoth manual decoder/encoder support
module Industry =
  let decoder: Decoder<Industry> =
    Decode.string
    |> Decode.andThen (function
        | "Education" -> Decode.succeed Education
        | "Manufacturing" -> Decode.succeed Manufacturing
        | "Finance" -> Decode.succeed Finance
        | "Retail" -> Decode.succeed Retail
        | string as s -> Decode.succeed (Other(s))
        | invalid -> Decode.fail $"Error: {invalid}")

  let encoder (industry: Industry) =
    let s  =
      match industry with
      | Other(s) -> $"{s}"
      | i -> i.ToString()

    Encode.string s

type Stats = int * int * int

/// Thoth manual decoder/encoder support
module Stats =
  /// Decode array to Stats. json=[#,#,#] to Stats(#,#,#)
  let decoder :Decoder<Stats> =
    Decode.object (fun get ->
      get.Required.Raw (
        Decode.list Decode.int
        |> Decode.andThen (fun data ->
            if data.Length = 3 then
              let (a, b, c) = (data[0], data[1], data[2])
              Decode.succeed (a, b, c)
            else
              Decode.fail "Invalid Stats format. Expected '[ #, #, # ]'.")))

  /// Encode Stats tuple to json array. Stats(#,#,#) to json=[#,#,#]
  let encoder (stats: Stats) =
    let (a, b, c) = stats
    Encode.list [ Encode.int a; Encode.int b; Encode.int c ]

type Customer = {
  Id: Guid
  Name: string
  Created: DateTime
  Contacts: Email list
  Industry: Industry
  Stats: Stats
  Notes: string option
} with
  /// Fleece codec
  static member get_Codec() =
    codec {
      let! id = jreq "Id" (fun x -> Some(x.Id))
      and! name = jreq "Name" (fun x -> Some(x.Name))
      and! created = jreq "Created" (fun x -> Some(x.Created))
      and! contacts = jreq "Contacts" (fun x -> Some(x.Contacts))
      and! industry = jreqWith Industry.jsonCodec "Industry" (fun x -> Some(x.Industry))
      and! stats = jreq "Stats" (fun x -> Some(x.Stats))
      and! notes = jopt "Notes" (fun x -> x.Notes)
      return { 
        Customer.Id = id
        Name = name
        Created = created
        Contacts = contacts
        Industry = industry
        Stats = stats
        Notes = notes }
    }
    |> ofObjCodec

/// Thoth manual decoder/encoder support
module Customer =
  let contactsDecoder :Decoder<Email list> =
    Decode.list Email.decoder

  let contactsEncoder (contacts: Email list) =
    contacts
    |> List.map Email.encoder
    |> Encode.list

  let decoder: Decoder<Customer> =
    Decode.object (fun get ->
      {
        Id = get.Required.Field "Id" Decode.guid
        Name = get.Required.Field "Name" Decode.string
        Created = get.Required.Field "Created" Decode.datetimeUtc
        Contacts = get.Required.Field "Contacts" contactsDecoder
        Industry = get.Required.Field "Industry" Industry.decoder
        Stats = get.Required.Field "Stats" Stats.decoder
        Notes = get.Optional.Field "Notes" Decode.string
      }
    )

  let encoder (customer: Customer) =
    Encode.object [
        "Id", Encode.guid customer.Id
        "Name", Encode.string customer.Name
        "Created", Encode.datetime customer.Created
        "Contacts", contactsEncoder customer.Contacts
        "Industry", Industry.encoder customer.Industry
        "Stats", Stats.encoder customer.Stats
        "Notes", Encode.option Encode.string customer.Notes
    ]

The actual serialization and deserialization for all libraries is simple and straight-forward. Manual encoding when using Thoth takes a bit more effort, but not much. They all support custom options, but this is the bare bones method to get data converted back and forth.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
// FSharpLu
let json = Compact.serialize customer
let customer = Compact.deserialize json

// Thoth
let json = Encode.Auto.toString(2, customer)
let customer = Decode.Auto.fromString<Customer>(json)

// Thoth (Cached)
let thothCachedCustomerEncoder =
  Encode.Auto.generateEncoderCached<Customer>(caseStrategy = PascalCase, extra = Extra.empty)

let thothCachedCustomerDecoder =
  Decode.Auto.generateDecoderCached<Customer>(caseStrategy = PascalCase, extra = Extra.empty)

let json = Encode.toString 2 (thothCachedCustomerEncoder customer)
let customer = Decode.fromString (thothCachedCustomerDecoder) json

// Thoth (Manual)
let json = customer |> Customer.encoder |> Encode.toString 2
let customer = Decode.fromString Customer.decoder json

// Fleece
let json = toJsonText customer
let customer: Customer ParseResult = ofJsonText json

// NewtonSoft.Json
let json = Newtonsoft.Json.JsonConvert.SerializeObject customer
let customer = Newtonsoft.Json.JsonConvert.DeserializeObject<Customer> json

// System.Text.Json
/// Serialization options for System.Text.Json (to support F#)
let jsonOptions = JsonFSharpOptions.Default().ToJsonSerializerOptions()

let json = JsonSerializer.Serialize<Customer>(customer, jsonOptions)
let customer = JsonSerializer.Deserialize<Customer>(json, jsonOptions)

There is one more major piece to discuss, performance. This is where BenchmarkDotNet comes into play. I investigate several different scenarios. I focus mostly on different object types: int list, string list, Customer list, and Customer. An additional variation is list size, for this I run the tests with size: 1, 100, and 1000. The combination of these tests will provide some insight regarding performance (for both speed and memory). For the most part, the benchmark code is straight forward. The only variation to note is that I try to keep the serialized json consistent, but I do need multiple versions of the json strings, since the libraries support slightly different encodings. Beyond that, they are all operating on the same data. Fair warning, the benchmark results are pretty hefty, but it’s the only way to see all the different breakdowns.

### System:
BenchmarkDotNet=v0.13.4, OS=ubuntu 20.04
Intel Core i7-9750H CPU 2.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=7.0.101
[Host] : .NET 7.0.1 (7.0.122.56804), X64 RyuJIT AVX2 DEBUG
DefaultJob : .NET 7.0.1 (7.0.122.56804), X64 RyuJIT AVX2

### Serialization:

Method ListSize Mean Error StdDev Median Gen0 Gen1 Gen2 Allocated
IntListFSharpLu 1 7,043.0 ns 43.89 ns 41.05 ns 7,033.7 ns 0.6 - - 4167 B
IntListThoth 1 1,194.4 ns 7.02 ns 6.22 ns 1,197.0 ns 0.1 - - 1168 B
IntListThothCached 1 406.7 ns 3.48 ns 3.25 ns 405.5 ns 0.1 - - 1039 B
IntListThothManual 1 164.1 ns 0.27 ns 0.24 ns 164.1 ns 0.0 - - 408 B
IntListFleece 1 1,174.0 ns 6.37 ns 5.65 ns 1,172.6 ns 0.9 0.0 - 5832 B
IntListNewtonsoft 1 389.2 ns 0.55 ns 0.43 ns 389.2 ns 0.2 0.0 - 1447 B
IntListTextJson 1 316.4 ns 1.62 ns 1.51 ns 316.2 ns 0.0 - - 136 B
StringListFSharpLu 1 23,558.4 ns 32.47 ns 30.37 ns 23,567.7 ns 2.7 - - 17161 B
StringListThoth 1 17,798.6 ns 34.58 ns 30.65 ns 17,790.3 ns 2.2 - - 14276 B
StringListThothCached 1 17,000.0 ns 28.15 ns 26.34 ns 16,994.6 ns 2.2 - - 14144 B
StringListThothManual 1 16,396.5 ns 12.94 ns 10.81 ns 16,398.5 ns 2.1 - - 13216 B
StringListFleece 1 17,498.7 ns 20.93 ns 16.34 ns 17,500.4 ns 2.9 - - 18841 B
StringListNewtonsoft 1 16,627.6 ns 17.37 ns 14.51 ns 16,625.7 ns 2.2 - - 14320 B
StringListTextJson 1 16,270.4 ns 44.64 ns 41.76 ns 16,262.5 ns 2.0 - - 13032 B
RecordListFSharpLu 1 48,328.4 ns 189.01 ns 176.80 ns 48,287.4 ns 2.4 - - 15483 B
RecordListThoth 1 84,474.2 ns 307.36 ns 287.50 ns 84,523.1 ns 4.7 - - 30290 B
RecordListThothCached 1 44,234.5 ns 150.27 ns 140.56 ns 44,270.8 ns 3.2 0.1 - 20479 B
RecordListThothManual 1 45,829.3 ns 63.85 ns 59.72 ns 45,842.2 ns 3.7 0.1 - 23980 B
RecordListFleece 1 12,916.2 ns 29.86 ns 23.31 ns 12,924.2 ns 3.4 0.1 - 21768 B
RecordListNewtonsoft 1 4,928.5 ns 20.42 ns 18.11 ns 4,922.4 ns 0.7 - - 4871 B
RecordListTextJson 1 4,409.8 ns 12.55 ns 11.12 ns 4,413.0 ns 0.3 - - 2247 B
OneRecordFSharpLu 1 43,885.5 ns 111.43 ns 98.78 ns 43,885.5 ns 2.3 - - 14553 B
OneRecordThoth 1 83,886.9 ns 264.00 ns 246.94 ns 83,845.5 ns 4.6 - - 29321 B
OneRecordThothCached 1 43,521.3 ns 306.90 ns 287.08 ns 43,645.5 ns 3.1 0.1 - 19608 B
OneRecordThothManual 1 48,191.4 ns 94.14 ns 83.45 ns 48,174.8 ns 3.6 0.1 - 23070 B
OneRecordFleece 1 11,719.4 ns 34.92 ns 29.16 ns 11,727.1 ns 3.2 0.1 - 20439 B
OneRecordNewtonsoft 1 4,362.4 ns 38.07 ns 35.61 ns 4,375.7 ns 0.7 - - 4587 B
OneRecordTextJson 1 4,088.9 ns 9.79 ns 8.68 ns 4,085.0 ns 0.3 - - 1963 B
IntListFSharpLu 100 20,188.5 ns 86.07 ns 80.51 ns 20,171.1 ns 3.4 0.1 - 21864 B
IntListThoth 100 14,860.5 ns 42.98 ns 38.10 ns 14,859.4 ns 3.4 0.1 - 21745 B
IntListThothCached 100 13,736.9 ns 44.20 ns 41.35 ns 13,720.9 ns 3.4 0.1 - 21607 B
IntListThothManual 100 9,308.6 ns 15.10 ns 12.61 ns 9,305.5 ns 2.8 0.1 - 17952 B
IntListFleece 100 16,534.7 ns 52.73 ns 46.74 ns 16,519.1 ns 5.8 0.2 - 36727 B
IntListNewtonsoft 100 7,372.3 ns 6.85 ns 6.07 ns 7,371.9 ns 1.3 0.0 - 8741 B
IntListTextJson 100 3,831.8 ns 4.95 ns 4.63 ns 3,832.3 ns 0.7 - - 4469 B
StringListFSharpLu 100 1,649,890.3 ns 2,182.75 ns 2,041.75 ns 1,649,718.0 ns 208.9 13.6 - 1315335 B
StringListThoth 100 1,603,423.3 ns 3,536.47 ns 2,953.12 ns 1,602,939.4 ns 208.9 11.7 - 1313783 B
StringListThothCached 100 1,558,011.5 ns 3,719.41 ns 3,479.14 ns 1,558,608.2 ns 208.9 13.6 - 1313298 B
StringListThothManual 100 1,609,062.4 ns 2,173.69 ns 2,033.27 ns 1,609,041.0 ns 207.0 19.5 - 1299154 B
StringListFleece 100 1,701,875.8 ns 3,307.47 ns 3,093.81 ns 1,701,224.7 ns 212.8 13.6 - 1336981 B
StringListNewtonsoft 100 1,663,057.9 ns 2,827.21 ns 2,506.25 ns 1,662,869.2 ns 207.0 11.7 - 1303146 B
StringListTextJson 100 1,572,979.7 ns 2,086.97 ns 1,850.05 ns 1,573,096.4 ns 205.0 11.7 - 1293890 B
RecordListFSharpLu 100 709,046.5 ns 1,413.84 ns 1,322.50 ns 709,459.4 ns 89.8 27.3 - 568895 B
RecordListThoth 100 4,123,862.4 ns 20,371.95 ns 19,055.93 ns 4,118,491.6 ns 304.6 210.9 - 1914472 B
RecordListThothCached 100 4,068,735.7 ns 18,677.86 ns 17,471.28 ns 4,060,941.0 ns 296.8 195.3 - 1901524 B
RecordListThothManual 100 4,531,003.4 ns 11,349.26 ns 10,616.10 ns 4,534,660.2 ns 351.5 140.6 - 2249710 B
RecordListFleece 100 857,710.4 ns 2,601.90 ns 2,433.82 ns 857,026.0 ns 206.0 0.9 - 1294299 B
RecordListNewtonsoft 100 405,181.7 ns 701.05 ns 621.47 ns 405,120.1 ns 50.7 13.6 - 321528 B
RecordListTextJson 100 362,255.9 ns 631.07 ns 559.42 ns 362,328.9 ns 34.6 8.3 - 220188 B
IntListFSharpLu 1000 133,347.3 ns 1,257.91 ns 1,176.65 ns 133,482.3 ns 28.8 5.3 - 182022 B
IntListThoth 1000 134,059.9 ns 306.36 ns 255.82 ns 134,002.1 ns 31.2 3.4 - 196388 B
IntListThothCached 1000 128,716.7 ns 1,423.84 ns 1,331.86 ns 127,905.5 ns 31.0 2.4 - 196170 B
IntListThothManual 1000 93,161.8 ns 454.78 ns 425.40 ns 93,178.4 ns 27.4 0.1 - 172760 B
IntListFleece 1000 149,448.3 ns 386.25 ns 342.40 ns 149,418.3 ns 44.1 9.7 - 277991 B
IntListNewtonsoft 1000 68,508.4 ns 220.08 ns 195.09 ns 68,464.8 ns 12.2 1.3 - 77127 B
IntListTextJson 1000 34,069.5 ns 107.37 ns 89.66 ns 34,113.2 ns 6.9 0.4 - 43871 B
StringListFSharpLu 1000 17,584,797.3 ns 29,524.67 ns 27,617.39 ns 17,584,905.4 ns 2062.5 125.0 31.2 13116871 B
StringListThoth 1000 16,840,644.9 ns 28,439.32 ns 25,210.72 ns 16,842,766.9 ns 2062.5 125.0 31.2 13110095 B
StringListThothCached 1000 16,990,921.6 ns 51,412.23 ns 45,575.60 ns 16,988,783.6 ns 2062.5 125.0 31.2 13108271 B
StringListThothManual 1000 16,557,071.7 ns 146,791.20 ns 137,308.58 ns 16,601,215.2 ns 2062.5 31.2 - 12984789 B
StringListFleece 1000 17,261,490.7 ns 55,897.70 ns 52,286.74 ns 17,263,192.8 ns 2125.0 93.7 31.2 13385711 B
StringListNewtonsoft 1000 16,650,742.4 ns 50,595.94 ns 44,851.98 ns 16,638,779.1 ns 2062.5 156.2 31.2 13019946 B
StringListTextJson 1000 16,210,066.4 ns 18,023.81 ns 15,977.64 ns 16,215,392.3 ns 2031.2 62.5 - 12938117 B
RecordListFSharpLu 1000 7,418,860.9 ns 16,006.82 ns 14,972.79 ns 7,413,755.7 ns 789.0 757.8 195.3 5522664 B
RecordListThoth 1000 45,465,135.8 ns 385,734.45 ns 360,816.24 ns 45,402,331.0 ns 2818.1 727.2 454.5 18928991 B
RecordListThothCached 1000 48,735,891.0 ns 831,080.08 ns 1,243,921.53 ns 47,969,807.8 ns 2818.1 727.2 454.5 18902199 B
RecordListThothManual 1000 54,555,776.3 ns 125,263.21 ns 117,171.29 ns 54,558,445.9 ns 3600.0 700.0 200.0 22406518 B
RecordListFleece 1000 13,469,301.0 ns 100,449.31 ns 89,045.70 ns 13,458,451.0 ns 1843.7 843.7 578.1 12810464 B
RecordListNewtonsoft 1000 4,360,206.6 ns 30,850.12 ns 28,857.22 ns 4,368,718.1 ns 546.8 375.0 140.6 3208284 B
RecordListTextJson 1000 3,924,525.0 ns 16,541.52 ns 14,663.63 ns 3,920,796.3 ns 406.2 257.8 140.6 2222653 B

### Deserialization:

Method ListSize Mean Error StdDev Gen0 Gen1 Gen2 Allocated
IntListFSharpLu 1 15.6 μs 0.1459 μs 0.1365 μs 1.4954 - - 9.29 KB
IntListThoth 1 17.9 μs 0.0496 μs 0.0440 μs 1.4343 - - 8.88 KB
IntListThothCached 1 16.9 μs 0.1128 μs 0.1055 μs 1.4038 - - 8.74 KB
IntListThothManual 1 10.1 μs 0.0488 μs 0.0457 μs 1.2360 - - 7.66 KB
IntListFleece 1 10.6 μs 0.0179 μs 0.0159 μs 1.0071 - - 6.26 KB
IntListNewtonsoft 1 9.0 μs 0.0284 μs 0.0265 μs 1.0986 - - 6.79 KB
IntListTextJson 1 8.5 μs 0.0106 μs 0.0088 μs 0.6714 - - 4.2 KB
StringListFSharpLu 1 33.7 μs 0.1217 μs 0.1138 μs 3.6011 - - 22.17 KB
StringListThoth 1 35.5 μs 0.0508 μs 0.0450 μs 3.4790 - - 21.56 KB
StringListThothCached 1 32.7 μs 0.0909 μs 0.0850 μs 3.4790 - - 21.42 KB
StringListThothManual 1 30.5 μs 0.0669 μs 0.0593 μs 3.4790 - - 21.43 KB
StringListFleece 1 30.0 μs 0.0498 μs 0.0441 μs 3.1128 - - 19.09 KB
StringListNewtonsoft 1 27.1 μs 0.0321 μs 0.0268 μs 3.1738 - - 19.56 KB
StringListTextJson 1 26.0 μs 0.0560 μs 0.0524 μs 2.7466 - - 16.99 KB
RecordListFSharpLu 1 103.2 μs 0.2298 μs 0.2037 μs 4.8828 - - 30.07 KB
RecordListThoth 1 209.4 μs 0.3070 μs 0.2872 μs 10.9863 0.2441 - 68.71 KB
RecordListThothCached 1 163.9 μs 0.4874 μs 0.4559 μs 9.5215 0.2441 - 58.5 KB
RecordListThothManual 1 72.3 μs 0.1873 μs 0.1752 μs 4.0283 - - 24.73 KB
RecordListFleece 1 24.1 μs 0.0792 μs 0.0741 μs 3.4485 0.0916 - 21.25 KB
RecordListNewtonsoft 1 12.2 μs 0.0365 μs 0.0342 μs 1.5717 0.0153 - 9.65 KB
RecordListTextJson 1 11.9 μs 0.0402 μs 0.0376 μs 0.5035 - - 3.16 KB
OneRecordFSharpLu 1 95.5 μs 0.2836 μs 0.2652 μs 4.5166 - - 28.38 KB
OneRecordThoth 1 182.2 μs 0.5150 μs 0.4817 μs 8.7891 - - 54.73 KB
OneRecordThothCached 1 137.7 μs 0.3629 μs 0.3031 μs 7.0801 - - 44.69 KB
OneRecordThothManual 1 66.1 μs 0.1749 μs 0.1460 μs 3.7842 - - 23.21 KB
OneRecordFleece 1 21.1 μs 0.0523 μs 0.0464 μs 3.1738 0.0916 - 19.46 KB
OneRecordNewtonsoft 1 11.4 μs 0.0185 μs 0.0164 μs 1.4954 0.0153 - 9.21 KB
OneRecordTextJson 1 10.5 μs 0.0313 μs 0.0293 μs 0.4425 - - 2.76 KB
IntListFSharpLu 100 41.6 μs 0.1170 μs 0.1094 μs 5.3711 0.1831 - 33.05 KB
IntListThoth 100 131.1 μs 1.2809 μs 1.1981 μs 10.7422 0.4883 - 66.87 KB
IntListThothCached 100 131.9 μs 1.1325 μs 1.0039 μs 10.7422 0.4883 - 66.72 KB
IntListThothManual 100 54.6 μs 0.0609 μs 0.0570 μs 9.0942 0.4272 - 55.76 KB
IntListFleece 100 53.1 μs 0.1576 μs 0.1474 μs 9.3994 0.3052 - 57.82 KB
IntListNewtonsoft 100 34.5 μs 0.0915 μs 0.0811 μs 4.9438 0.1221 - 30.55 KB
IntListTextJson 100 30.7 μs 0.0543 μs 0.0481 μs 4.2419 0.1221 - 26.05 KB
StringListFSharpLu 100 1,763.7 μs 2.6119 μs 2.4432 μs 210.9375 15.6250 - 1303.58 KB
StringListThoth 100 1,895.9 μs 3.5673 μs 3.1623 μs 216.7969 17.5781 - 1329.64 KB
StringListThothCached 100 1,774.0 μs 6.9559 μs 6.5066 μs 216.7969 17.5781 - 1329.27 KB
StringListThothManual 100 1,761.7 μs 34.6178 μs 35.5499 μs 212.8906 7.8125 - 1308.76 KB
StringListFleece 100 1,773.9 μs 5.0636 μs 4.4888 μs 216.7969 19.5313 - 1333.79 KB
StringListNewtonsoft 100 1,715.8 μs 6.6255 μs 6.1975 μs 210.9375 15.6250 - 1300.96 KB
StringListTextJson 100 1,841.5 μs 2.5186 μs 2.1032 μs 210.9375 15.6250 - 1299.17 KB
RecordListFSharpLu 100 1,506.4 μs 3.7216 μs 3.4812 μs 156.2500 50.7813 - 969.71 KB
RecordListThoth 100 10,375.6 μs 84.5533 μs 79.0912 μs 718.7500 531.2500 15.6250 4452.71 KB
RecordListThothCached 100 10,250.9 μs 34.5145 μs 32.2849 μs 718.7500 562.5000 15.6250 4442.12 KB
RecordListThothManual 100 1,599.8 μs 4.6668 μs 4.1370 μs 189.4531 89.8438 - 1164.58 KB
RecordListFleece 100 498.4 μs 0.6963 μs 0.6173 μs 48.8281 1.9531 - 305.02 KB
RecordListNewtonsoft 100 854.5 μs 3.7517 μs 3.5094 μs 81.0547 22.4609 - 502.68 KB
RecordListTextJson 100 979.8 μs 2.1825 μs 2.0416 μs 48.8281 11.7188 - 304.22 KB
IntListFSharpLu 1000 256.8 μs 0.4628 μs 0.3864 μs 39.5508 6.3477 - 245.56 KB
IntListThoth 1000 1,158.0 μs 1.9641 μs 1.8372 μs 95.7031 37.1094 - 588.79 KB
IntListThothCached 1000 1,159.7 μs 3.3701 μs 2.8142 μs 95.7031 39.0625 - 588.44 KB
IntListThothManual 1000 454.2 μs 0.6121 μs 0.5426 μs 79.1016 1.9531 - 486.03 KB
IntListFleece 1000 443.5 μs 0.5548 μs 0.5189 μs 84.4727 1.9531 - 520.01 KB
IntListNewtonsoft 1000 264.2 μs 1.9403 μs 1.8149 μs 39.5508 9.2773 - 243.05 KB
IntListTextJson 1000 233.1 μs 0.8043 μs 0.7129 μs 35.8887 8.0566 - 221.16 KB
StringListFSharpLu 1000 16,952.5 μs 31.5653 μs 27.9819 μs 2093.7500 31.2500 - 12948.04 KB
StringListThoth 1000 18,092.8 μs 201.8233 μs 188.7856 μs 2156.2500 125.0000 31.2500 13214.55 KB
StringListThothCached 1000 18,802.7 μs 81.7247 μs 76.4453 μs 2156.2500 125.0000 31.2500 13213.3 KB
StringListThothManual 1000 17,291.1 μs 183.9874 μs 172.1019 μs 2093.7500 281.2500 31.2500 13002.46 KB
StringListFleece 1000 18,058.9 μs 51.8915 μs 40.5135 μs 2156.2500 156.2500 31.2500 13305.53 KB
StringListNewtonsoft 1000 17,507.9 μs 66.8627 μs 62.5434 μs 2093.7500 31.2500 - 12945.42 KB
StringListTextJson 1000 16,826.3 μs 37.8847 μs 33.5838 μs 2093.7500 31.2500 - 12950.66 KB
RecordListFSharpLu 1000 16,157.2 μs 68.4580 μs 64.0357 μs 1406.2500 500.0000 250.0000 9433.67 KB
RecordListThoth 1000 106,187.4 μs 633.2908 μs 592.3806 μs 7000.0000 750.0000 500.0000 44255.41 KB
RecordListThothCached 1000 108,517.2 μs 608.9917 μs 539.8552 μs 7200.0000 800.0000 400.0000 44214.61 KB
RecordListThothManual 1000 21,465.3 μs 120.3867 μs 112.6098 μs 1750.0000 1500.0000 718.7500 11445.14 KB
RecordListFleece 1000 5,433.1 μs 98.3105 μs 91.9597 μs 632.8125 500.0000 359.3750 3292.54 KB
RecordListNewtonsoft 1000 9,820.1 μs 18.1347 μs 15.1433 μs 796.8750 781.2500 140.6250 4995.52 KB
RecordListTextJson 1000 10,365.1 μs 26.4786 μs 22.1108 μs 531.2500 515.6250 125.0000 3064.77 KB

There are a couple trends to gleen from the above results. At a high level, F#-specific libraries incur a performance cost. Newtonsoft and System.Text.Json are typically 5x-10x faster than FSharpLu and Thoth, and 2x-3x faster than Fleece. Datatype impacts performance differences, but the trends hold for the most part (with the exception of string lists, where string handling dominates encoding). Library memory usage holds similar trends, just not as pronounced in most cases. Memory usage does show its impact on the larger ListSize (1000), where speed and the GC take a pretty hard hit.

Looking at the results a bit closer, Newtonsoft and System.Text.Json are in the same ballpark, but System.Text.Json nearly always had a slight lead in performance. On average, Thoth is the slowest, although cached encodings offer a small boost. Manual encoding for Thoth certainly makes it more competitive with FSharpLu and Fleece. Speaking of which, FSharpLu and Fleece are steadily in the middle of the pack. Fleece typically has performance several times faster, trending it closer to Newtonsoft and System.Text.Json performance. I suspect this is a result of manual encoding versus reflection, but just a hunch.

What does all this mean? Well, it depends (I hope you weren’t expecting a simple answer). Clearly there is a performance difference between the libraries, sometimes up to an order of magnitude (that is more the exception than the rule). When talking nanoseconds and microseconds, most use cases honestly aren’t going to see a noticeable difference. This is where ergonomics and flexibility need to factor into a decision. Everything has its trade off, and finding what works for your specific case is a process. If straight-up performance matters most, there are some clear leaders. Admittedly, the benchmark results surprised me. I didn’t expect the F#-specific libraries to have such a large gap. I also expected manual encoding to offer a better performance benefit over reflection than it did. As they say, you learn something new every day, and today is no exception. So there you have it, a comparison with some of the major json encoding libraries supported in F#. Seeing all the trade-offs in one place can assist in decisions moving forward and understanding what makes the best library for particular use cases. A better grasp on all the impact of all components in a system will help make any project stronger. As I mentioned in the beginning, if you see any errors or opportunities for improvement, let me know. Until next time, stay calm and keep encoding…

Related Posts: