The application of Natural Language Processing (NLP) has achieved a high
level of relevance in several areas. In the field of software engineering (SE),
NLP applications are based on the classification of similar texts (e.g.
software requirements), applied in tasks of estimating software effort,
selection of human resources, etc. Classifying software requirements has been a
complex task, considering the informality and complexity inherent in the texts
produced during the software development process. The pre-trained embedding
models are shown as a viable alternative when considering the low volume of
textual data labeled in the area of software engineering, as well as the lack
of quality of these data. Although there is much research around the
application of word embedding in several areas, to date, there is no knowledge
of studies that have explored its application in the creation of a specific
model for the domain of the SE area. Thus, this article presents the proposal
for a contextualized embedding model, called BERT_SE, which allows the
recognition of specific and relevant terms in the context of SE. The assessment
of BERT_SE was performed using the software requirements classification task,
demonstrating that this model has an average improvement rate of 13% concerning
the BERT_base model, made available by the authors of BERT. The code and
pre-trained models are available at this https URL