F# and ML.NET Sentiment Analysis

Read Time: 13 minutes

Today we’ll look at performing sentiment analysis using F# and ML.NET. A new version (v0.9.0) has recently been released, so we use this as an opportunity to play with some new functionality. The goal of today’s post will be to perform sentiment analysis on movie reviews from IMDB.

Note: ML.NET is still evolving, this post was written using Microsoft.ML v0.9.0.

We’ll use .NET Core version 2.2. If you don’t have it installed, head out to the .NET Core Downloads page. Select SDK for your platform. Tangential, but you can also get here by going to dot.net, then navigating to Downloads and .NET Core.

With that out of the way, create a console F# project, then add the ML.NET package.

1
2
3
dotnet new console --language F# --name MLNet-SentimentAnalysis
cd MLNet-SentimentAnalysis
dotnet add package Microsoft.ML --version 0.9.0

Next, it is time to get the data. The source we will use for this post is from UCI. The datafile can be found here. The zip file contains examples for IMDB, Yelp, and Amazon, but we’ll stick with IMDB for this post.

1
2
mkdir data && cd data
curl -O https://archive.ics.uci.edu/ml/machine-learning-databases/00331/sentiment%20labelled%20sentences.zip

Here is a sample of what the data looks like. There is no header row. The tab separated columns represent 1) the review’s text 2) the sentiment where 1 = positive and 0 = negative.

1
2
3
4
Long, whiny and pointless.  	0
But I recommend waiting for their future efforts, let this one go.  	0
Excellent cast, story line, performances.  	1
Totally believable.  	1

Now that we have the data, time to get to the code. First there is some namespace setup.

1
2
3
open System.IO
open Microsoft.ML
open Microsoft.ML.Data

Here are the data types to be used. SentimentData is for loading data, SentimentPrediction is for performing predictions. Here we also get our first taste of 0.9.0. As we’ll see later we can use the SentimentData type for loading. To enable this we will add [<LoadColumn(column position)>] to the members. I have also included Probability. This is not a real column, nor is it needed for training. I have included it because it is a required field when extracting performance metrics. I feel like I shouldn’t need to include it here, but for now it’s the only way I got it to work. The CreateTextReader now accepts a datatype for driving the loading process. Once the data reader is setup, we also perform a train/test split of 70/30, respectively.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
type SentimentData () =
    [<DefaultValue>]
    [<LoadColumn(0)>]
    val mutable public SentimentText :string

    [<DefaultValue>]
    [<LoadColumn(1)>]
    val mutable public Label :bool
    
    // NOTE: Need to add this column to extract metrics
    [<DefaultValue>]
    [<LoadColumn(2)>]
    val mutable public Probability :float32
   
type SentimentPrediction () =
    [<DefaultValue>]
    val mutable public SentimentData :string

    [<DefaultValue>]
    val mutable public PredictedLabel :bool

    [<DefaultValue>]
    val mutable public Score :float32 

[<EntryPoint>]
let main argv =
  let ml = new MLContext()

  let reader = ml.Data.CreateTextReader<SentimentData>(separatorChar = '\t', hasHeader = true)
           
  let dataFile = "./data/imdb_labelled.txt"
  let allData = reader.Read(dataFile);
  let struct (trainData, testData) = ml.Clustering.TrainTestSplit(allData, testFraction = 0.3)

The old way (see below) can still be used, but I find the above newness a nice, more concise, method to load data.

1
2
3
4
5
6
7
8
9
10
11
12
// Pre v0.9.0 way
let reader = 
  ml.Data.CreateTextReader(
    separatorChar = '\t',
    hasHeader = true,
    columns = 
      [|
          Data.TextLoader.Column("SentimentText", Nullable Data.DataKind.Text, 0);
          Data.TextLoader.Column("Label", Nullable Data.DataKind.Bool, 1);
          // NOTE: Need to add this column to extract metrics
          Data.TextLoader.Column("Probability", Nullable Data.DataKind.R4, 2)
      |])

ML.NET also provides methods to perform inspection into the dataset.

1
2
3
4
printfn "### Schema"
allData.Schema
|> Seq.iter(fun x-> printfn "%A" x)
printfn ""

Here is what a simple schema view looks like.

1
2
3
4
### Schema
SentimentText: Text
Label: Bool
Probability: R4

Next we setup the training pipeline. There are other options, like FastTree, but we’ll use FastForest for today’s post. We’ll also take the defaults, but as with previous trainers we’ve looked at, we can provide custom hyperparameters. Once the pipeline is setup, we run Fit to build the model.

1
2
3
4
5
6
7
let pipeline = 
  ml
    .Transforms.Text.FeaturizeText("SentimentText", "Features")
    .Append(ml.BinaryClassification.Trainers.FastForest())
    // Example of custom hyperparameters
    // .Append(mlContext.BinaryClassification.Trainers.FastForest(numTrees = 500, numLeaves = 100, learningRate = 0.0001))
let model = pipeline.Fit(trainData)

Any good machine learning process requires performance evaluation. For that we’ll look at two aspects. First, ML.NET provides evaluators for the trainers. I’ve cherry-picked a couple of the available BinaryClassificationEvaluator metrics. Second, we can perform a preview of the predictions, which allows us to see the sentiment value along with the actual and predicted labels, as well as the score. There are other items in the view as well that I left in to show the extent of the reporting. Then we can run evaluation’s against the train and test sets.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
let displayEvaluation description data = 
  let predictions = model.Transform data

  let metrics = ml.BinaryClassification.Evaluate(predictions)

  printfn ""
  printfn "### %s" description
  printfn "Accuracy          : %0.4f" (metrics.Accuracy)
  printfn "F1                : %0.4f" (metrics.F1Score)
  printfn "Positive Precision: %0.4f" (metrics.PositivePrecision)
  printfn "Positive Recall   : %0.4f" (metrics.PositiveRecall)
  printfn "Negative Precision: %0.4f" (metrics.NegativePrecision)
  printfn "Negative Recall   : %0.4f" (metrics.NegativeRecall)
  printfn ""

  let preview = predictions.Preview()
  preview.RowView
  |> Seq.take 5
  |> Seq.iter(fun row ->
     row.Values
     |> Array.iter (fun kv -> printfn "%s: %A" kv.Key kv.Value)
     printfn "")
  printfn ""

displayEvaluation "Train" trainData
displayEvaluation "Test" testData

As imagined, the metrics are better when run against the training data. The much better view of prediction quality is when run against the testing data. As expected, the model doesn’t perform as well against the test set, there is probably some more work that needs done here. The Preview is also useful when diagnosing more detailed problems, since it shows scores and label predictions. Not related to the results, but the stratification value is used for the train/test split.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
### Train
Accuracy          : 0.9120
F1                : 0.9145
Positive Precision: 0.8987
Positive Recall   : 0.9309
Negative Precision: 0.9267
Negative Recall   : 0.8927

SentimentText: Not sure who was more lost - the flat characters or the audience, nearly half of whom walked out.  
Label: false
StratificationColumn: 0.595641375f
Features: Sparse vector of size 7818, 110 explicit values
PredictedLabel: false
Score: -54.9804649f

SentimentText: Attempting artiness with black & white and clever camera angles, the movie disappointed - became even more ridiculous - as the acting was poor and the plot and lines almost non-existent.  
Label: false
StratificationColumn: 0.58837676f
Features: Sparse vector of size 7818, 188 explicit values
PredictedLabel: false
Score: -13.02876f

SentimentText: Very little music or anything to speak of.  
Label: false
StratificationColumn: 0.753678203f
Features: Sparse vector of size 7818, 52 explicit values
PredictedLabel: false
Score: -5.37574673f

SentimentText: The best scene in the movie was when Gerardo is trying to find a song that keeps running through his head.  
Label: true
StratificationColumn: 0.967485666f
Features: Sparse vector of size 7818, 118 explicit values
PredictedLabel: true
Score: 41.7043114f

SentimentText: The rest of the movie lacks art, charm, meaning... If it's about emptiness, it works I guess because it's empty.  
Label: false
StratificationColumn: 0.929597497f
Features: Sparse vector of size 7818, 119 explicit values
PredictedLabel: false
Score: -15.2312632f


### Test
Accuracy          : 0.6890
F1                : 0.6923
Positive Precision: 0.6471
Positive Recall   : 0.7444
Negative Precision: 0.7385
Negative Recall   : 0.6400

SentimentText: Wasted two hours.  
Label: false
StratificationColumn: 0.171681881f
Features: Sparse vector of size 7818, 21 explicit values
PredictedLabel: true
Score: 4.22176647f

SentimentText: Saw the movie today and thought it was a good effort, good messages for kids.  
Label: true
StratificationColumn: 0.185497403f
Features: Sparse vector of size 7818, 83 explicit values
PredictedLabel: true
Score: 25.0270023f

SentimentText: The movie showed a lot of Florida at it's best, made it look very appealing.  
Label: true
StratificationColumn: 0.250951052f
Features: Sparse vector of size 7818, 86 explicit values
PredictedLabel: true
Score: 18.1396465f

SentimentText: 
Label: false
StratificationColumn: 0.128096819f
Features: Sparse vector of size 7818, 0 explicit values
PredictedLabel: true
Score: 9.73927498f

SentimentText: In other words, the content level of this film is enough to easily fill a dozen other films.  
Label: true
StratificationColumn: 0.229808331f
Features: Sparse vector of size 7818, 90 explicit values
PredictedLabel: true
Score: 20.8655605f

Now that model fitting and some evaluation has been performed, we need to make a prediction function. As with so many things so far, this is simple to do.

1
let predictor = model.CreatePredictionEngine<SentimentData, SentimentPrediction>(ml)

Once the prediction function is in place, we can run predictions and see their underlying scores.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
let tests = 
  [
    "It was cool, cute, and funny."; 
    "It was slow and boring."; 
    "It was the greatest thing I've seen." 
  ]

tests 
|> List.iter (fun x ->
  let input = SentimentData()
  input.SentimentText <- x

  let prediction = predictor.Predict(input)
  printfn ""
  printfn "Text       : %s" x
  printfn "Prediction : %b" (prediction.PredictedLabel)
  printfn "Score      : %0.4f" (prediction.Score)
  )

And here we can see the predictions.

1
2
3
4
5
6
7
8
9
10
11
Text       : It was cool, cute, and funny.
Prediction : true
Score      : 12.2628

Text       : It was slow and boring.
Prediction : false
Score      : -10.0353

Text       : It was the greatest thing I've seen.
Prediction : true
Score      : 68.4614

This is all well and good, but to be useful we need to be able to save a model to a file for later use. Here we have the ability to save and reload a model file.

1
2
3
4
5
6
7
8
9
10
11
12
// Save model to file
let saveModel (ml:MLContext) trainedMode = 
  use fsWrite = new FileStream("test-model.zip", FileMode.Create, FileAccess.Write, FileShare.Write)
  ml.Model.Save(model, fsWrite)

saveModel ml model

// Load model from file
use fsRead = new FileStream("test-model.zip", FileMode.Open, FileAccess.Read, FileShare.Read)
let mlReloaded = MLContext()
let modelReloaded = TransformerChain.LoadFrom(mlReloaded, fsRead)
let predictorReloaded = modelReloaded.CreatePredictionEngine<SentimentData, SentimentPrediction>(mlReloaded)

Once the model file has been reloaded, we can run a sample prediction. We just need to create the prediction function against and away we go.

1
2
3
4
5
6
7
8
let test1 = SentimentData()
test1.SentimentText <- tests.[0]

let predictionReloaded = predictorReloaded.Predict(test1)
printfn ""
printfn "Text                  : %s" tests.[0]
printfn "Prediction (Reloaded) : %b" (predictionReloaded.PredictedLabel)
printfn "Score (Reloaded)      : %0.4f" (predictionReloaded.Score)

Here are the prediction results from a saved model.

1
2
3
Text                  : It was cool, cute, and funny.
Prediction (Reloaded) : true
Score (Reloaded)      : 12.2628

This has been a brief look into sentiment analysis using F# and ML.NET. It has been a pleasure to see the framework progress. It is even more enjoyable performing these types of workloads using F#. Until next time. Thanks.

Related Posts: