Transfer Learning: What, Why, and How?

Explore the power of Transfer Learning

Photo by Mathew Schwartz on Unsplash

Transfer learning is a technique in the machine learning field where a pre-trained model that learns patterns from a large dataset (usually) is used as a starting point for training on a new, related task. For example, computer vision or NLP model.

This approach uses the knowledge gained from the original model to boost the intended learning process and performance on the new task, even when limited data is available.

In today’s newsletter, we will learn quickly what transfer learning is, why we want to use transfer learning, and how to implement them. Let’s get into it.

---

What is Transfer Learning?

As defined above, transfer learning is a method used for training a previously pre-trained model in a new task. This method allows for bypassing the machine learning exploration's data, computational, and time limitations—which is helpful in many scenarios.

Transfer learning was first coined around the late 90s. At this time, researchers began exploring methods for transferring knowledge from one task to another, such as inductive transfer and instance-based transfer learning.

Then the resurgence of deep learning in the late 2000s played a significant role in popularizing transfer learning. The success of deep learning models, especially in computer vision, led to the development of pre-trained models, such as AlexNet, VGG, and ResNet.

Not only in computer vision, but the field of natural language processing (NLP) also started introducing pre-trained language models. For example, word2vec, GloVe, and later BERT, GPT, and RoBERTa, significantly impacted the adoption of transfer learning in NLP tasks.

It’s an exciting time as the development of machine learning, and AI is pretty much boosted because of the transfer learning method.

So, why are we considering the transfer learning method and its advantages?

Why Transfer Learning?

As we have read above, transfer learning can minimize the effort of machine learning exploration by using the pre-trained model as the starting point.

If we discuss them further, transfer learning is growing in importance in the machine learning field for several reasons:

• Reduced training time

  We can significantly reduce the time required to train a new model using a pre-trained model, as the starting point is already accomplished the learning.

• Lower computational resources

  Transfer learning requires fewer computational resources, as it reuses the learned features from the pre-trained model, minimizing the need for extensive training from scratch.

• Improved performance

  Transfer learning often leads to better performance, particularly in cases where the available data for the new task is limited. The pre-trained model has already learned useful features that can be fine-tuned to the specific requirements of the new task.

• Generalization across domains

  Transfer learning has shown promising results in generalizing knowledge across different domains and tasks, enabling the development of models that can adapt to various real-world scenarios.

• Accessibility

  The growing availability of pre-trained models, especially in areas like computer vision and natural language processing, makes transfer learning more accessible to researchers and practitioners, allowing them to build upon existing knowledge and state-of-the-art models.

How do we do Transfer Learning?

Transfer Learning is available in many popular Python packages, including:

1. TensorFlow (with Keras)

   Easy implementation of transfer learning using pre-trained models available in the tensorflow.keras.applications module.

   Installation: pip install tensorflow

2. PyTorch

   Open-source deep learning library that provides pre-trained models through the torchvision package for computer vision tasks.

   Installation: pip install torch torchvision

3. Hugging Face Transformers

   The Transformers library by Hugging Face offers a wide range of pre-trained models like BERT, GPT-2, RoBERTa, and more.

   Installation: pip install transformers

4. Fast.ai

   High-level deep learning library built on top of PyTorch. It simplifies the transfer learning process with pre-trained models for both computer vision and NLP tasks.

   Installation: pip install fastai

5. Keras-Applications

   Keras-Applications is a standalone package that provides various pre-trained models for Keras. You can use it to perform transfer learning for computer vision tasks.

   Installation: pip install keras-applications

There are still many transfer-learning Python packages, but you can start using the above packages.

We will look deeper at the code part in the next edition of the newsletter, so stay tuned.