TinyML based audio visual keyword detection

Mehmet Tosun; Hamit Erdem

Makale detayları

TinyML based audio visual keyword detection

Yazarlar:

Mehmet Tosun | Hamit Erdem

Yayıncı:

Nigde Omer Halisdemir University

Ana Başlıklar:

Derin Öğrenme | Gömülü Sistemler

Dergi:

Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi Vol.13 Issue 4, pp.1207-1215

Belge Tipi:

Makale

Yayın Tarihi:

10/2024

Belge Dili:

Türkçe

Document ID (Org No.):

2001329706

pISSN:

2564-6605

Anahtar kelimeler:

Özet:

 Keyword detection KWD is one of the areas where machine learning is used. Its purpose is the automatic detection of specific words or objects from audio or image data. As portable artificial intelligence applications become more prevalent, the number of applications in this field is also growing. In particular, hybrid systems the use of audio and video together are being studied to increase the effectiveness of KWD applications. The system aims to combine audio and visual commands detected through two different channels. Extensive work has been done on audiovisual keyword detection in a computer environment, yielding good results. On the other hand, efforts are being made within the scope of TinyML Low-Power Machine Learning to implement deep learning applications on low-capacity processors. In these applications, reducing the parameters of the deep learning model quantization, pruning makes it possible to implement the model on ordinary microcontrollers. In this study, a keyword detection application in the field of TinyML is proposed using audio and visual data. In the training of the proposed hybrid model, the audio and visual models were first trained separately in the Edge Impulse software environment. Developed MobileNetV2 and CNN-based models were loaded onto ESP32-CAM and Arduino Nano BLE development kits and tested. Subsequently, the models were combined using a linear weighted fusion method and tested. In the experimental results, according to the accuracy criterion, the success rate of the audio-based KWD was 85, the success rate of the image-based KWD was 85, while the classification success in the audiovisual hybrid application was around 90.

 Keyword detection KWD is one of the areas where machine learning is used. Its purpose is the automatic detection of specific words or objects from audio or image data. As portable artificial intelligence applications become more prevalent, the number of applications in this field is also growing. In particular, hybrid systems the use of audio and video together are being studied to increase the effectiveness of KWD applications. The system aims to combine audio and visual commands detected through two different channels. Extensive work has been done on audiovisual keyword detection in a computer environment, yielding good results. On the other hand, efforts are being made within the scope of TinyML Low-Power Machine Learning to implement deep learning applications on low-capacity processors. In these applications, reducing the parameters of the deep learning model quantization, pruning makes it possible to implement the model on ordinary microcontrollers. In this study, a keyword detection application in the field of TinyML is proposed using audio and visual data. In the training of the proposed hybrid model, the audio and visual models were first trained separately in the Edge Impulse software environment. Developed MobileNetV2 and CNN-based models were loaded onto ESP32-CAM and Arduino Nano BLE development kits and tested. Subsequently, the models were combined using a linear weighted fusion method and tested. In the experimental results, according to the accuracy criterion, the success rate of the audio-based KWD was 85, the success rate of the image-based KWD was 85, while the classification success in the audiovisual hybrid application was around 90.

 Keyword detection KWD is one of the areas where machine learning is used. Its purpose is the automatic detection of specific words or objects from audio or image data. As portable artificial intelligence applications become more prevalent, the number of applications in this field is also growing. In particular, hybrid systems the use of audio and video together are being studied to increase the effectiveness of KWD applications. The system aims to combine audio and visual commands detected through two different channels. Extensive work has been done on audiovisual keyword detection in a computer environment, yielding good results. On the other hand, efforts are being made within the scope of TinyML Low-Power Machine Learning to implement deep learning applications on low-capacity processors. In these applications, reducing the parameters of the deep learning model quantization, pruning makes it possible to implement the model on ordinary microcontrollers. In this study, a keyword detection application in the field of TinyML is proposed using audio and visual data. In the training of the proposed hybrid model, the audio and visual models were first trained separately in the Edge Impulse software environment. Developed MobileNetV2 and CNN-based models were loaded onto ESP32-CAM and Arduino Nano BLE development kits and tested. Subsequently, the models were combined using a linear weighted fusion method and tested. In the experimental results, according to the accuracy criterion, the success rate of the audio-based KWD was 85, the success rate of the image-based KWD was 85, while the classification success in the audiovisual hybrid application was around 90.

 Anahtar kelime tespiti AKT, makine öğreniminin kullanıldığı alanlardan birisidir. Amacı, ses veya görüntü verisinden belirli kelime veya objenin otomatik tespit edilmesidir. Taşınabilir yapay zekâ uygulamalarının artmasıyla beraber, bu alanda da uygulamalar artmaktadır. Özellikle AKT uygulamalarının etkinliğini artırmak için hibrit sistemler ses ve görüntünün birlikte kullanımı üzerinde çalışma yapılmaktadır. Bu sistem ile birlikle iki farklı kanaldan algılanan ses ve görüntü komutlarının birleştirilmesi amaçlanmaktadır. Bilgisayar PC ortamında görsel işitsel AKT üzerinde birçok çalışma yapılmış ve iyi sonuçlar elde edilmiştir. Diğer taraftan derin öğrenme uygulamalarını düşük kapasiteli işlemciler üzerinde gerçekleştirmek için TinyML Düşük Kapasiteli Makine Öğrenmesi kapsamında çalışmalar yapılmaktadır. Bu uygulamalarda, derin öğrenmeye yönelik geliştirilen modelin parametrelerini azaltarak nicelleştirme, kırpma sıradan mikrodenetleyici üzerinde uygulama imkânı oluşturmaktadır. Bu çalışmada ses ve görüntü verisi kullanılarak, TinyML alanında AKT uygulaması önerilmiştir. Önerilen hibrit modelin eğitiminde öncelikle ses ve görüntü modelleri Edge Impulse yazılım ortamında ayrı ayrı eğitilmiştir. Geliştirilen MobileNetV2 ve CNN tabanlı modeller ESP32-CAM ve Arduino Nano BLE geliştirme kitlerine yüklenerek, denenmiştir. Daha sonra modeller doğrusal ağırlıklı birleştirme metodu ile birleştirilerek denenmiştir. Sistemin başarısı standart ölçütlere göre test edilmiştir. Deneysel sonuçlarda doğruluk ölçütüne göre, sadece ses tabanlı AKT başarısı 85, sadece görüntü tabanlı AKT başarısı 85 olurken, görsel işitsel hibrit uygulamasında sınıflandırma başarısı 90 civarında olmuştur.

Makale detayları

TinyML based audio visual keyword detection

Yazarlar:

Mehmet Tosun | Hamit Erdem

Yayıncı:

Nigde Omer Halisdemir University

Ana Başlıklar:

Derin Öğrenme | Gömülü Sistemler

Dergi:

Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi Vol.13 Issue 4, pp.1207-1215

Belge Tipi:

Makale

Yayın Tarihi:

10/2024

Belge Dili:

Türkçe

Document ID (Org No.):

2001329706

pISSN:

2564-6605

TinyML

Machine Learning

Keyword Spotting

Convolutional Neural Network

Edge ImpulseTinyML

Makine Öğrenimi

Anahtar Kelime Tespiti

Evrişimsel Sinir Ağları

Edge Impulse

Sayı

Makale detayları

Yazarlar: Mehmet Tosun | Hamit Erdem

Yayıncı: Nigde Omer Halisdemir University

Ana Başlıklar: Derin Öğrenme | Gömülü Sistemler

Dergi: Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi Vol.13 Issue 4, pp.1207-1215

Belge Tipi: Makale

Yayın Tarihi: 10/2024

Belge Dili: Türkçe

Document ID (Org No.): 2001329706

pISSN: 2564-6605

Sayı

Yazarlar:

Mehmet Tosun | Hamit Erdem

Yayıncı:

Nigde Omer Halisdemir University

Ana Başlıklar:

Derin Öğrenme | Gömülü Sistemler

Dergi:

Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi Vol.13 Issue 4, pp.1207-1215

Belge Tipi:

Makale

Yayın Tarihi:

10/2024

Belge Dili:

Türkçe

Document ID (Org No.):

2001329706

pISSN:

2564-6605