Enhancement of Bert model with predictive text generation and attention mechanisms for improvements in text autocomplete

ABSTRACT: Transformer-based fine-tuning strategies have shown effectiveness in-low resource and low-data context. Still, the lack of properly established baselines and benchmark datasets makes it difficult to compare different ways of dealing with low-resource settings. This research consists of three contributions. First, two previously unreleased datasets serve as benchmarks for text classification and low-resource multilabel text classification in the Filipino language. Second, the improved BERT models were pre-trained for application in the Filipino context. Third, presentation of a simple degradation text that measures a model’s susceptibility to performance decline as the number of training samples decreases. The deterioration rates of the pre-trained model and the consideration of using this method to compare models designed for low-resource environments are examined. The findings show that BERT models, even when distilled into a smaller model for low-resource contexts, maintain high performance with a minimum fine-tuning and degrade slowly in low-data conditions, making them well-suited to such restriction. There has also been research showing effective transfer from supervised tasks with large datasets, such as natural language inference (Conneau et al., 2022) and machine translation (McCann et al., 2022). Computer vision research has also demonstrated the importance of transfer learning from large pre-trained models, where an effective recipe is to fine-tune models pre-trained with ImageNet (Yosinski et. al., 2019).