I am trying to write some simple R bindings for a C library. It is tdjson and many languages can interface with it directly.
I compiled the source for the library and got a fully working built (libtdjson.so) and tested it with python.
Here is a reference implementation with python using the exact same library:
from ctypes import *
import json
# load shared library
tdjson_path = "tdlib/lib/libtdjson.so"
tdjson = CDLL(tdjson_path)
_td_execute = tdjson.td_execute
_td_execute.restype = c_char_p
_td_execute.argtypes = [c_char_p]
def td_execute(query):
query = json.dumps(query).encode('utf-8')
result = _td_execute(query)
if result:
result = json.loads(result.decode('utf-8'))
return result
# test TDLib execute method
test_command = {'#type': 'getTextEntities',
'text': '#telegram /test_command https://telegram.org telegram.me',
'#extra': ['5', 7.0, 'a']}
td_execute(test_command)
When I try to interface with the library in R, I do not get any return value from the function calls. I only get a list with one item which contains the original call. Anyone knows how to that?
This is what I've tried in R:
library(jsonlite)
dyn.load("tdlib/lib/libtdjson.so", type = "External")
td_execute <- function(query) {
query <- jsonlite::toJSON(query, auto_unbox = T)
result <- .C("td_execute", charToRaw(query))
return(result)
}
test_command = list("#type"="getTextEntities",
"text"="#telegram /test_command https://telegram.org telegram.me",
"#extra"= c("5", 7.0, 'a'))
t <- td_execute(test_command)
rawToChar(t[[1]])
# t only contains the original JSON string
The only return values from the lists are basically an echo of the function call parameters.
It is not possible in this case. R requires a custom compilation of the source code with R-specific header files.
Related
I'm following this tutorial that codes a sentiment analysis classifier using BERT with the huggingface library and I'm having a very odd behavior. When trying the BERT model with a sample text I get a string instead of the hidden state. This is the code I'm using:
import transformers
from transformers import BertModel, BertTokenizer
print(transformers.__version__)
PRE_TRAINED_MODEL_NAME = 'bert-base-cased'
PATH_OF_CACHE = "/home/mwon/data-mwon/paperChega/src_classificador/data/hugingface"
tokenizer = BertTokenizer.from_pretrained(PRE_TRAINED_MODEL_NAME,cache_dir = PATH_OF_CACHE)
sample_txt = 'When was I last outside? I am stuck at home for 2 weeks.'
encoding_sample = tokenizer.encode_plus(
sample_txt,
max_length=32,
add_special_tokens=True, # Add '[CLS]' and '[SEP]'
return_token_type_ids=False,
padding=True,
truncation = True,
return_attention_mask=True,
return_tensors='pt', # Return PyTorch tensors
)
bert_model = BertModel.from_pretrained(PRE_TRAINED_MODEL_NAME,cache_dir = PATH_OF_CACHE)
last_hidden_state, pooled_output = bert_model(
encoding_sample['input_ids'],
encoding_sample['attention_mask']
)
print([last_hidden_state,pooled_output])
that outputs:
4.0.0
['last_hidden_state', 'pooler_output']
While the answer from Aakash provides a solution to the problem, it does not explain the issue. Since one of the 3.X releases of the transformers library, the models do not return tuples anymore but specific output objects:
o = bert_model(
encoding_sample['input_ids'],
encoding_sample['attention_mask']
)
print(type(o))
print(o.keys())
Output:
transformers.modeling_outputs.BaseModelOutputWithPoolingAndCrossAttentions
odict_keys(['last_hidden_state', 'pooler_output'])
You can return to the previous behavior by adding return_dict=False to get a tuple:
o = bert_model(
encoding_sample['input_ids'],
encoding_sample['attention_mask'],
return_dict=False
)
print(type(o))
Output:
<class 'tuple'>
I do not recommend that, because it is now unambiguous to select a specific part of the output without turning to the documentation as shown in the example below:
o = bert_model(encoding_sample['input_ids'], encoding_sample['attention_mask'], return_dict=False, output_attentions=True, output_hidden_states=True)
print('I am a tuple with {} elements. You do not know what each element presents without checking the documentation'.format(len(o)))
o = bert_model(encoding_sample['input_ids'], encoding_sample['attention_mask'], output_attentions=True, output_hidden_states=True)
print('I am a cool object and you can acces my elements with o.last_hidden_state, o["last_hidden_state"] or even o[0]. My keys are; {} '.format(o.keys()))
Output:
I am a tuple with 4 elements. You do not know what each element presents without checking the documentation
I am a cool object and you can acces my elements with o.last_hidden_state, o["last_hidden_state"] or even o[0]. My keys are; odict_keys(['last_hidden_state', 'pooler_output', 'hidden_states', 'attentions'])
I faced the same issue while learning how to implement Bert. I noticed that using
last_hidden_state, pooled_output = bert_model(encoding_sample['input_ids'], encoding_sample['attention_mask'])
is the issue. Use:
outputs = bert_model(encoding_sample['input_ids'], encoding_sample['attention_mask'])
and extract the last_hidden state using
output[0]
You can refer to the documentation here which tells you what is returned by the BertModel
Let say there is R code for REST API based on using the "plumber" package.
Here is a function from it.
#' Date of sale
#' #get /date_of_sale
#' #param auth_key Auth key
#' #param user_id User ID
#' #example https://company.com/date_of_sale?auth_key=12345&user_id=6789
function(auth_key, user_id) {
# ...
}
Let say there is another R script that uses API request to this server like
api_string <- "https://company.com/date_of_sale?auth_key=12345&user_id=6789"
date_of_sale <- jsonlite::fromJSON(api_string)
Is it possible to get a description of the parameters "auth_key" and "user_id" in the second script to have a full explanation of what means each parameter? For example, get for "auth_key" a string "Auth key"? Or how it will be possible to get access to function "date_of_sale" metadata at all?
Thanks for any ideas?
Using a file plumber.R with content as you provided. Assuming it is in the working directory.
In R
pr_read <- plumber::pr("plumber.R")
spec <- pr_read$getApiSpec()
spec$paths$`/date_of_sale`$get$parameters
spec is an R list with the same structure as an OpenAPI document.
If you do not have access to API plumber file but your API is running somewhere you have access to.
spec <- jsonlite::fromJSON("{api_server}/openapi.json", simplifyDataFrame = FALSE)
spec$paths$`/date_of_sale`$get$parameters
Again this follows the OpenAPI documentation standards.
Since this is an API GET request you can not access the description of the variable unless you explicitly include it in the API response.
I've learned some R script purely for this question, my guess is this is how you've prepared your JSON API response.
You can do something like this in your JSON request.
library(rjson)
auth_key <- "some_key";
user_id <- "340";
x <- list(auth_key = list(
type = typeof(auth_key),
lenght = length(auth_key),
attributes = attributes(auth_key),
string = auth_key
),
user_id = list(
type = typeof(user_id),
lenght = length(user_id),
attributes = attributes(user_id),
string = user_id
),
data = "your_data"
);
#x
json <- toJSON(x, indent=0, method="C" )
fromJSON( json )
You might want to look a these.
https://stat.ethz.ch/R-manual/R-devel/library/base/html/typeof.html
https://ramnathv.github.io/pycon2014-r/learn/structures.html https://rdrr.io/cran/rjson/man/toJSON.html https://www.rdocumentation.org/packages/base/versions/3.6.2/topics/attributes
Currently I am working on several UnitTests and my tests also contain mocking.
In order to "pass" on functions part of my function, I am applying the local_mock() function from the testthat package.
However, there are several functions on which the local_mock function needs to be applied.
They have a different structure, but same number of arguments.
Hence, I created a function as following:
.apply_local_mock = function(Function, File){
local_mock(Function = function(Argument1, Argument2, Argument3) {
result = .readCsv(file.path(targetPath, "Input", paste(File))) %>%
return(result)
})
}
An example of two functions to which local_mock will be applied to:
CalcProfitability <- function(IncomeStatement, BalanceSheet, Months){
result = (12/Months) * (IncomeStatement / BalanceSheet)
return(result)
}
CalcDiffDivByPl <- function(BalanceSheetCY, BalanceSheetPY, IncomeStatement){
result = (BalanceSheetCY - BalanceSheetPY) / IncomeStatement
return(result)
}
So the mocking test code looks as following:
.apply_local_mock(CalcProfitability,"CalcProfitability.csv")
.apply_local_mock(CalcDiffDivByPl ,"CalcDiffDivByPl.csv")
But I am receiving the following error:
Error: Function FunctionName not found in environment testthat.
Can anyone please advise how to include local_mock() function in a new function?
I would like to know how to pass a plot from matlab to be displayed as a webfigure on a servlet page. Note that I'm using the MPS. Hence I'm not packaging the matlab code into java but just using a client proxy to the matlab function.
My eg matlab function:
function varargout = mymagicplot(in,displayPlot)
x = magic(in);
varargout{1} = x;
if (strcmp(displayPlot, 'Plot'))
varargout{2} = {plot(x)};
end
On the servlet side:
interface MatlabMagic {
public Object[] mymagicplot(int num_outargs, int size, String plotOption) throws IOException, MATLABException;
}
Question is how to code the display of the plot as a webfigure on the servlet page?
I tried a workaround by splitting my matlab code into two functions.
First function is used by the client proxy.
function m = mymagic(in)
m = magic(in);
end
Second function is packaged into java classes by the library compiler.
function returnfigure = mygetwebfiguremagicplot(in)
h = figure;
set(h, 'Visible', 'off');
plot(in);
returnfigure = webfigure(h);
close(h);
end
In this way, I can access the mymagic function .ctf in the MPS to return the results to the servlet and plot it as a webfigure using the java classes created from the second function matlab code.
This is just one possible workable solution that I can think of.
I'm currently programming an R-script which uses a java .jar that makes use of the java/lang/Vector class, which in this case uses a class in a method that is not native. In java source code:
public static Vector<ClassName> methodname(String param)
I found nothing in the documentation of rJava on how to handle a template class like vector and what to write when using jcall or any other method.
I'm currently trying to do something like this:
v <- .jnew("java/util/Vector")
b <- .jcall(v, returnSig = "Ljava/util/Vector", method = "methodname",param)
but R obviously throws an exception:
method methodname with signature (Ljava/lang/String;)Ljava/util/Vector not found
How do I work the template class into this command? Or for that matter, how do I create a vector of a certain class in the first place? Is this possible?
rJava does not know java generics, there is no syntax that will create a Vector of a given type. You can only create Vectors of Objects.
Why are you sticking with the old .jcall api when you can use the J system, which lets you use java objects much more nicely:
> v <- new( J("java.util.Vector") )
> v$add( 1:10 )
[1] TRUE
> v$size()
[1] 1
# code completion
> v$
v$add( v$getClass() v$removeElement(
v$addAll( v$hashCode() v$removeElementAt(
v$addElement( v$indexOf( v$retainAll(
v$capacity() v$insertElementAt( v$set(
v$clear() v$isEmpty() v$setElementAt(
v$clone() v$iterator() v$setSize(
v$contains( v$lastElement() v$size()
v$containsAll( v$lastIndexOf( v$subList(
v$copyInto( v$listIterator( v$toArray(
v$elementAt( v$listIterator() v$toArray()
v$elements() v$notify() v$toString()
v$ensureCapacity( v$notifyAll() v$trimToSize()
v$equals( v$remove( v$wait(
v$firstElement() v$removeAll( v$wait()
v$get( v$removeAllElements()