Model ini dipublikasikan di HF papers pada 2023-04-14 dan disumbangkan ke Hugging Face Transformers pada 2023-07-18.

DINOv2

DINOv2 adalah model dasar visi yang menggunakan ViT sebagai ekstraktor fitur untuk berbagai tugas turunan seperti klasifikasi gambar dan estimasi kedalaman. Model ini berfokus pada stabilisasi dan percepatan pelatihan melalui teknik seperti perhatian yang lebih cepat dan hemat memori, pengemasan urutan, stochastic depth yang ditingkatkan, Fully Sharded Data Parallel (FSDP), dan distilasi model.

Anda dapat menemukan semua checkpoint DINOv2 asli di bawah koleksi Dinov2.

Klik pada model DINOv2 di bilah sisi kanan untuk melihat lebih banyak contoh cara menerapkan DINOv2 ke berbagai tugas visi.

Contoh di bawah ini menunjukkan cara mendapatkan embedding gambar dengan Pipeline atau kelas AutoModel.

from transformers import pipeline


pipe = pipeline(
    task="image-classification",
    model="facebook/dinov2-small-imagenet1k-1-layer",
    device=0
)

pipe("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg")

import requests
from PIL import Image

from transformers import AutoImageProcessor, AutoModelForImageClassification


url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained("facebook/dinov2-small-imagenet1k-1-layer")
model = AutoModelForImageClassification.from_pretrained(
    "facebook/dinov2-small-imagenet1k-1-layer",
    device_map="auto",
    attn_implementation="sdpa"
)

inputs = processor(images=image, return_tensors="pt").to(model.device)
logits = model(**inputs).logits
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])

Quantization mengurangi beban memori model besar dengan merepresentasikan bobot dalam presisi yang lebih rendah. Lihat ringkasan Quantization untuk backend quantization lain yang tersedia.

Contoh di bawah menggunakan torchao untuk hanya mengkuantisasi bobot ke int4.

# pip install torchao
import requests
from PIL import Image
from torchao.quantization import Int4WeightOnlyConfig

from transformers import AutoImageProcessor, AutoModelForImageClassification, TorchAoConfig


url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
image = Image.open(requests.get(url, stream=True).raw)

processor = AutoImageProcessor.from_pretrained('facebook/dinov2-giant-imagenet1k-1-layer')

quant_config = Int4WeightOnlyConfig(group_size=128)
quantization_config = TorchAoConfig(quant_type=quant_config)

model = AutoModelForImageClassification.from_pretrained(
    'facebook/dinov2-giant-imagenet1k-1-layer',
    device_map="auto",
    quantization_config=quantization_config
)

inputs = processor(images=image, return_tensors="pt").to(model.device)
outputs = model(**inputs)
logits = outputs.logits
predicted_class_idx = logits.argmax(-1).item()
print("Predicted class:", model.config.id2label[predicted_class_idx])

Catatan

• Contoh di bawah ini menunjukkan cara membagi tensor output menjadi:

  • satu embedding untuk seluruh gambar, yang umumnya disebut sebagai token CLS, berguna untuk klasifikasi dan pengambilan (retrieval)
  • seperangkat embedding lokal, satu untuk setiap patch 14x14 dari gambar input, berguna untuk tugas-tugas padat, seperti segmentasi semantik
  Disalin

  from transformers import AutoImageProcessor, AutoModel
  from PIL import Image
  import requests
  
  url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
  image = Image.open(requests.get(url, stream=True).raw)
  print(image.height, image.width)  # [480, 640]
  
  processor = AutoImageProcessor.from_pretrained('facebook/dinov2-base')
  model = AutoModel.from_pretrained('facebook/dinov2-base', device_map="auto")
  patch_size = model.config.patch_size
  
  inputs = processor(images=image, return_tensors="pt").to(model.device)
  print(inputs.pixel_values.shape)  # [1, 3, 224, 224]
  batch_size, rgb, img_height, img_width = inputs.pixel_values.shape
  num_patches_height, num_patches_width = img_height // patch_size, img_width // patch_size
  num_patches_flat = num_patches_height * num_patches_width
  
  outputs = model(**inputs)
  last_hidden_states = outputs[0]
  print(last_hidden_states.shape)  # [1, 1 + 256, 768]
  assert last_hidden_states.shape == (batch_size, 1 + num_patches_flat, model.config.hidden_size)
  
  cls_token = last_hidden_states[:, 0, :]
  patch_features = last_hidden_states[:, 1:, :].unflatten(1, (num_patches_height, num_patches_width))

• Gunakan torch.jit.trace untuk mempercepat inferensi. Namun, hal ini akan menghasilkan beberapa elemen yang tidak cocok. Perbedaan antara model asli dan model yang ditrace adalah 1e-4.

  Disalin

  import torch
  from transformers import AutoImageProcessor, AutoModel
  from PIL import Image
  import requests
  
  url = 'http://images.cocodataset.org/val2017/000000039769.jpg'
  image = Image.open(requests.get(url, stream=True).raw)
  
  processor = AutoImageProcessor.from_pretrained('facebook/dinov2-base')
  model = AutoModel.from_pretrained('facebook/dinov2-base', device_map="auto")
  
  inputs = processor(images=image, return_tensors="pt").to(model.device)
  outputs = model(**inputs)
  last_hidden_states = outputs[0]
  
  # We have to force return_dict=False for tracing
  model.config.return_dict = False
  
  with torch.no_grad():
      traced_model = torch.jit.trace(model, [inputs.pixel_values])
      traced_outputs = traced_model(inputs.pixel_values)
  
  print((last_hidden_states - traced_outputs[0]).abs().max())

Dinov2Config

>>> from transformers import Dinov2Config, Dinov2Model

>>> # Initializing a Dinov2 dinov2-base style configuration
>>> configuration = Dinov2Config()

>>> # Initializing a model (with random weights) from the dinov2-base style configuration
>>> model = Dinov2Model(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

Dinov2Model

Dinov2ForImageClassification

>>> from transformers import AutoImageProcessor, Dinov2ForImageClassification
>>> import torch
>>> from datasets import load_dataset

>>> dataset = load_dataset("huggingface/cats-image")
>>> image = dataset["test"]["image"][0]

>>> image_processor = AutoImageProcessor.from_pretrained("facebook/dinov2-base")
>>> model = Dinov2ForImageClassification.from_pretrained("facebook/dinov2-base")

>>> inputs = image_processor(image, return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> # model predicts one of the 1000 ImageNet classes
>>> predicted_label = logits.argmax(-1).item()
>>> print(model.config.id2label[predicted_label])
...