Machine Learning Techniques for Modelling Shellfish Harvest Assessments

Mussel farming is a major industry in New Zealand that is currently growing fast. With the expansion of the industry, it has become important to allow the required processes to scale efficiently, which can be aided through the automation of tasks. One of these processes is harvest assessments, which are currently done manually by trained individual workers who generally rely on their domain knowledge to perform the assessments rather than following a fixed standard. A solution to the above issue is to use modern machine learning and computer vision techniques to build a model to automate the process.

such as convolutional neural networks (CNNs), which can be trained to perform tasks such as detecting mussels in images.

We propose using CNNs as the basis of a mobile phone based application that can perform these harvest assessments automatically, using the CNN to perform instance segmentation to extract information about the mussels from an image.

The lack of a mussel dataset labeled for training instance segmentation models requires the development of a new dataset.

This dataset contains a training dataset of 137 images, a validation dataset of 180 images, and a test dataset of 8 images with 6-8 mussels in each image.

The training dataset is also augmented using cropping, rotation, and flipping in order to increase the size of the dataset to 1096 images.

Three convolutional neural network based real-time techniques are investigated, YOLACT, CenterMask, and BlendMask, as well as Mask R-CNN. Mask R-CNN has the highest Mean Average Precision (AP) of 68.16 with a frame rate of 0.10 fps. YOLACT has a final AP at 62.43 with a frame rate of 0.39 fps. CenterMask has the second highest final AP of 63.66 with a frame rate of 0.28 fps. BlendMask has the lowest final AP at 60.89, with a frame rate of 0.35 fps.

By mixing the images in our dataset and redistributing them in order to decrease homogeneity, the AP for each of the algorithms increases, with Mask R-CNN's AP increasing to 91.21 on the original test dataset and 81.44 on the new test dataset, YOLACT's increasing to 84.76 on the original test dataset and 68.51 on the new test dataset, CenterMask's increasing to 92.06 and 80.39, and BlendMask's increasing to 80.27 and 78.65.

To explore the applicability of the CNN methods in the real world, we also investigate deploying the algorithms onto a mobile phone to test the frame rate on our intended platform.

Although it is too challenging to export the real-time algorithms to the Open Neural Network Exchange (ONNX) format, we successfully deploy Mask R-CNN to a mobile phone. The experiments show Mask R-CNN had an inference time of 14.6 seconds, or a frame rate of approximately 0.06 fps.

However, by using the quantisation tools built into ONNX to reduce the model weights to a UInt8 format, the inference time is reduced by 38%, i.e. 9.0 seconds, or 0.11 fps. Though none of the techniques ran at real-time when testing them on the laptop CPU, the inference time for all three real-time methods was below 4 seconds. With the ability to run inferences on multiple mussels at a time, a non-real-time application using these techniques would still decrease the time taken to perform harvest assessments. There is also a significant decrease in inference time by using quantisation on Mask R-CNN.

Although further improvements on deployment of the real-time algorithms need to be made in the future, this is a significant step towards using modern techniques to achieve automated mussel assessments.