Heitor R. Guimarães

selected publications

1. Conference

   VIC-KD: Variance-Invariance-Covariance Knowledge Distillation to Make Keyword Spotting More Robust Against Adversarial Attacks

   Heitor R. Guimarães, Arthur Pimentel, Anderson R. Avila, and 1 more author

   In 2024 International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024

   Abs Bib

   Keyword spotting (KWS) refers to the task of identifying a set of predefined words in audio streams. With the advances seen recently with deep neural networks, it has become a popular technology to activate and control small devices, such as voice assistants. Relying on such models for edge devices, however, can be challenging due to hardware constraints. Moreover, as adversarial attacks have increased against voice-based technologies, developing solutions robust to such attacks has become crucial. In this work, we propose VIC-KD, a robust distillation recipe for model compression and adversarial robustness. Using self-supervised speech representations, we show that imposing geometric priors to the latent representations of both Teacher and Student models leads to more robust target models. Experiments on the Google Speech Commands datasets show that the proposed methodology improves upon current state-of-the-art robust distillation methods, such as ARD and RSLAD, by 12% and 8% in robust accuracy, respectively.

   @inproceedings{guimaraes2024vickd,
     author = {Guimar{\~a}es, Heitor R. and Pimentel, Arthur and Avila, Anderson R. and Falk, Tiago H.},
     title = {VIC-KD: Variance-Invariance-Covariance Knowledge Distillation to Make Keyword Spotting More Robust Against Adversarial Attacks},
     year = {2024},
     booktitle = {2024 International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
     pages = {--},
     organization = {IEEE},
   }

2. Conference

   RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness

   Heitor R. Guimarães, Arthur Pimentel, Anderson R. Avila, and 3 more authors

   In 2023 International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2023

   Abs Bib

   Self-supervised speech pre-training enables deep neural network models to capture meaningful and disentangled factors from raw waveform signals. The learned universal speech representations can then be used across numerous downstream tasks. These representations, however, are sensitive to distribution shifts caused by environmental factors, such as noise and/or room reverberation. Their large sizes, in turn, make them unfeasible for edge applications. In this work, we propose a knowledge distillation methodology termed RobustDistiller which compresses universal representations while making them more robust against environmental artifacts via a multi-task learning objective. The proposed layer-wise distillation recipe is evaluated on top of three well-established universal representations, as well as with three downstream tasks. Experimental results show the proposed methodology applied on top of the WavLM Base+ teacher model outperforming all other benchmarks across noise types and levels, as well as reverberation times. Oftentimes, the obtained results with the student model (24M parameters) achieved results inline with those of the teacher model (95M).

   @inproceedings{guimaraes2023robustdistiller,
     author = {Guimar{\~a}es, Heitor R. and Pimentel, Arthur and Avila, Anderson R. and Rezagholizadeh, Mehdi and Chen, Boxing and Falk, Tiago H.},
     title = {RobustDistiller: Compressing Universal Speech Representations for Enhanced Environment Robustness},
     year = {2023},
     booktitle = {2023 International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
     pages = {--},
     organization = {IEEE},
   }

3. Conference

   Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement

   Heitor R. Guimarães, Arthur Pimentel, Anderson R. Avila, and 2 more authors

   In NeurIPS 2022 Efficient Natural Language and Speech Processing Workshop, 2022

   Abs Bib

   Self-supervised speech representation learning aims to extract meaningful factors from the speech signal that can later be used across different downstream tasks, such as speech and/or emotion recognition. Existing models, such as HuBERT, however, can be fairly large thus may not be suitable for edge speech applications. Moreover, realistic applications typically involve speech corrupted by noise and room reverberation, hence models need to provide representations that are robust to such environmental factors. In this study, we build on the so-called DistilHuBERT model, which distils HuBERT to a fraction of its original size, with three modifications, namely: (i) augment the training data with noise and reverberation, while the student model needs to distill the clean representations from the teacher model; (ii) introduce a curriculum learning approach where increasing levels of noise are introduced as the model trains, thus helping with convergence and with the creation of more robust representations; and (iii) introduce a multi-task learning approach where the model also reconstructs the clean waveform jointly with the distillation task, thus also acting as an enhancement step to ensure additional environment robustness to the representation. Experiments on three SUPERB tasks show the advantages of the proposed method not only relative to the original DistilHuBERT, but also to the original HuBERT, thus showing the advantages of the proposed method for “in the wild” edge speech applications.

   @inproceedings{guimaraes2022improving,
     author = {Guimar{\~a}es, Heitor R. and Pimentel, Arthur and Avila, Anderson R. and Rezagholizadeh, Mehdi and Falk, Tiago H.},
     title = {Improving the Robustness of DistilHuBERT to Unseen Noisy Conditions via Data Augmentation, Curriculum Learning, and Multi-Task Enhancement},
     year = {2022},
     booktitle = {NeurIPS 2022 Efficient Natural Language and Speech Processing Workshop},
   }

4. Conference

   A Perceptual Loss Based Complex Neural Beamforming for Ambix 3D Speech Enhancement

   Heitor R. Guimarães, Wesley Beccaro, and Miguel A. Ramirez

   In Proc. L3DAS22: Machine Learning for 3D Audio Signal Processing, 2022

   Abs Bib

   This work proposes a novel approach to B-Format AmbiX 3D speech enhancement based on the short-time Fourier transform (STFT) representation. The model is a Fully Complex Convolutional Network (FC2N) that estimates a mask to be applied to the input features. Then, a final layer is responsible for converting the B-format to a monaural representation in which we apply the inverse STFT (ISTFT) operation. For the optimization process, we use a compounded loss function, applied in the time-domain, based on the short-time objective intelligibility (STOI) metric combined with a perceptual loss on top of the wav2vec 2.0 model. The approach is applied on Task 1 of the L3DAS22 challenge, where our model achieves a score of 0.845 in the metric proposed by the challenge, using a subset of the development set as reference.

   @inproceedings{guimaraes22_l3das,
     author = {Guimarães, Heitor R. and Beccaro, Wesley and Ramirez, Miguel A.},
     title = {{A Perceptual Loss Based Complex Neural Beamforming for Ambix 3D Speech Enhancement}},
     year = {2022},
     booktitle = {Proc. L3DAS22: Machine Learning for 3D Audio Signal Processing},
     pages = {16--20},
     doi = {10.21437/L3DAS.2022-4},
   }

5. Journal

   Monaural speech enhancement through deep wave-U-net

   Heitor R. Guimarães, Hitoshi Nagano, and Diego W. Silva

   Expert Systems with Applications, 2020

   Abs Bib

   In this paper, we present Speech Enhancement through Wave-U-Net (SEWUNet), an end-to-end approach to reduce noise from speech signals. This background context is detrimental to several downstream systems, including automatic speech recognition (ASR) and word spotting, which in turn can negatively impact end-user applications. We show that our proposal does improve signal-to-noise ratio (SNR) and word error rate (WER) compared with existing mechanisms in the literature. In the experiments, network input is a 16 kHz sample rate audio waveform corrupted by an additive noise. Our method is based on the Wave-U-Net architecture with some adaptations to our problem. Four simple enhancements are proposed and tested with ablation studies to prove their validity. In particular, we highlight the weight initialization through an autoencoder before training for the main denoising task, which leads to a more efficient use of training time and a higher performance. Through quantitative metrics, we show that our method is prefered over the classical Wiener filtering and shows a better performance than other state-of-the-art proposals.

   @article{GUIMARAES2020113582,
     title = {Monaural speech enhancement through deep wave-U-net},
     journal = {Expert Systems with Applications},
     volume = {158},
     pages = {113582},
     year = {2020},
     issn = {0957-4174},
     doi = {https://doi.org/10.1016/j.eswa.2020.113582},
     url = {https://www.sciencedirect.com/science/article/pii/S0957417420304061},
     author = {Guimarães, Heitor R. and Nagano, Hitoshi and Silva, Diego W.},
     keywords = {Speech enhancement, Noise reduction, Wave-U-net, Deep learning, Signal to Noise Ratio (SNR), Word Error Rate (WER)},
   }