Connected Papers | Find and explore academic papers
Origin paper
ELECTRAMed: a new pre-trained language representation model for biomedical NLP
SciFive: a text-to-text transformer model for biomedical literature
Transfer Learning for Biomedical Question Answering
Improving Biomedical Pretrained Language Models with Knowledge
Bio-Megatron: Larger Biomedical Domain Language Model
Benchmarking for biomedical natural language processing tasks with a domain specific ALBERT
BioFLAIR: Pretrained Pooled Contextualized Embeddings for Biomedical Sequence Labeling Tasks
On the effectiveness of small, discriminatively pre-trained language representation models for biomedical text mining
BioELECTRA:Pretrained Biomedical text Encoder using Discriminators
A Pre-Training Technique to Localize Medical BERT and to Enhance Biomedical BERT
BioMedBERT: A Pre-trained Biomedical Language Model for QA and IR
Pre-training data selection for biomedical domain adaptation using journal impact metrics
Fine-tuning large neural language models for biomedical natural language processing
BioALBERT: A Simple and Effective Pre-trained Language Model for Biomedical Named Entity Recognition
The Diminishing Returns of Masked Language Models to Science
Inexpensive Domain Adaptation of Pretrained Language Models: Case Studies on Biomedical NER and Covid-19 QA
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
Pretrained Language Models for Biomedical and Clinical Tasks: Understanding and Extending the State-of-the-Art
Improving fine-tuned question answering models for electronic health records
Bioformer: an efficient transformer language model for biomedical text mining
BioVAE: a pre-trained latent variable language model for biomedical text mining
BioM-Transformers: Building Large Biomedical Language Models with BERT, ALBERT and ELECTRA
GPT-3 Powered Information Extraction for Building Robust Knowledge Bases
Trigger Word Detection and Thematic Role Identification via BERT and Multitask Learning
Document-Level Biomedical Relation Extraction Leveraging Pretrained Self-Attention Structure and Entity Replacement: Algorithm and Pretreatment Method Validation Study
Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again
KBioXLM: A Knowledge-anchored Biomedical Multilingual Pretrained Language Model
How to Pre-Train Your Model? Comparison of Different Pre-Training Models for Biomedical Question Answering
An extensive benchmark study on biomedical text generation and mining with ChatGPT
Investigation of BERT Model on Biomedical Relation Extraction Based on Revised Fine-tuning Mechanism
Optimization of Biomedical Language Model with Optuna and a Sentencepiece Tokenization for NER
Self-Supervised Contextual Language Representation of Radiology Reports to Improve the Identification of Communication Urgency
Oversampling effect in pretraining for bidirectional encoder representations from transformers (BERT) to localize medical BERT and enhance biomedical BERT
exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources
Investigation of improving the pre-training and fine-tuning of BERT model for biomedical relation extraction
Transferability of Natural Language Inference to Biomedical Question Answering
Infusing Disease Knowledge into BERT for Health Question Answering, Medical Inference and Disease Name Recognition
Inexpensive Domain Adaptation of Pretrained Language Models: A Case Study on Biomedical Named Entity Recognition
Chemical-protein Interaction Extraction via Gaussian Probability Distribution and External Biomedical Knowledge
Re-train or Train from Scratch? Comparing Pre-training Strategies of BERT in the Medical Domain
Biomedical named entity recognition using BERT in the machine reading comprehension framework
Log in to saveSave
The overwhelming amount of biomedical scientific texts calls for the development of effective language models able to tackle a wide range of biomedical natural language processing (NLP) tasks. The most recent dominant approaches are domain-specific models, initialized with general-domain textual data and then trained on a variety of scientific corpora. However, it has been observed that for specialized domains in which large corpora exist, training a model from scratch with just in-domain knowledge may yield better results. Moreover, the increasing focus on the compute costs for pre-training recently led to the design of more efficient architectures, such as ELECTRA. In this paper, we propose a pre-trained domain-specific language model, called ELECTRAMed, suited for the biomedical field. The novel approach inherits the learning framework of the general-domain ELECTRA architecture, as well as its computational advantages. Experiments performed on benchmark datasets for several biomedical NLP tasks support the usefulness of ELECTRAMed, which sets the novel state-of-the-art result on the BC5CDR corpus for named entity recognition, and provides the best outcome in 2 over the 5 runs of the 7th BioASQ-factoid Challange for the question answering task.