Using Huggingface Transformer Models in R - r

Question

I am trying to use different Huggingface models in R.
This works by importing the transformers package through reticulate (thank you, https://rpubs.com/eR_ic/transfoRmers)
Models where inputs just require a single string work for me.
Some models require a lists or a vector and I simply don't know where to get the information on how exactly to call the model.
Take this model for example. https://huggingface.co/openai/clip-vit-base-patch32.
From the python example I know it takes a picture and (I assume) a character vector of possible classes.
The Python input is: text=["a photo of a cat", "a photo of a dog"], images=image
library(reticulate)
library(here)
library(tidyverse)
transformers <- reticulate::import("transformers")
image_classification_zero_shot <- transformers$pipeline(task = "zero-shot-image-classification", model = "openai/clip-vit-base-patch32")
image_classification <- transformers$pipeline(task = "image-classification", model = "microsoft/beit-base-patch16-224-pt22k-ft22k")
image_url <- "http://images.cocodataset.org/val2017/000000039769.jpg"
The model just requiring the image works
image_classification(images = image_url)
The model which also requires a character input with the classes does not work.
image_classification_zero_shot(text = c("cats", "dogs"), images = image_url)
image_classification_zero_shot(text = "[cats, dogs]", images = image_url)
> Error in py_call_impl(callable, dots$args, dots$keywords) :
TypeError: object of type 'NoneType' has no len()
View(image_classification_zero_shot) does not yield any information.
How do I get the zero shot model to work?
How do I generally get the information on how to call these models in R? It's a function, shouldn't I be able to find information about its parameters somewhere (in R or on huggingface)?
Thank you very much!

Answer 1

I am experiencing a similar issue with another huggingface transformer called "jonas/sdg_classifier_osdg".
Error in py_call_impl(callable, dots$args, dots$keywords) :
TypeError: linear(): argument 'input' (position 1) must be Tensor, not
BatchEncoding
Solutions:
To write python code within a rmarkdown notebook
Activate repl_python() on the console and then write python code.
The same code generates a tensor when written in python but a character string when written in R.
Further Questions:
How to effectively transform an R string into a tensor that its understood by pythorch? (When I try to load torch together with reticulate, R crashes)