I am attempting to query a large table (30M records) from Salesforce to R using the SalesforceR package, and hope to do so without splitting up into many queries.
The following query (names changed for security/readability) succeeds, if I limit the records:
OrderItemsSF <- sf_query_bulk("SELECT Id,Order,Product,Quantity FROM OrderItems LIMIT 1000000",object_name = "OrderItems", api_type = "Bulk 1.0", verbose = TRUE, max_attempts = 1000)
This query fails ("Error: column name 'result' must not be duplicated") whenever it takes longer than 10 minutes, which fits with SF's documentation saying bulk queries retry after 10 minutes (see link 1 at bottom). I believe the answer is to enable PKChunking to automatically separate my query into smaller batches, but I am having trouble finding a working syntax for this.
I have tried the following:
OrderItemsSF <- sf_query_bulk("SELECT Id,Order,Product,Quantity FROM OrderItems LIMIT 1000000",object_name = "OrderItems", api_type = "Bulk 1.0", verbose = TRUE, max_attempts = 1000, control = sf_control(PKChunkingHeader = list(`Sforce-Enable-PKChunking`= TRUE)))
This results in the error "Error in catch_errors(httr_response) :
ClientInputError: Sforce-Enable-PkChunking doesn't have a valid value. The same error results from using FALSE, and according to the documentation (see link 2 at bottom), FALSE is the default for this parameter, so that doesn't make sense!
Other syntax attempts, like the one below, succeed on queries under 10 minutes and fail over 10 minutes, suggesting that R is ignoring this text and not actually PKChunking:
OrderItemsSF <- sf_query_bulk("SELECT Id,Order,Product,Quantity FROM OrderItems LIMIT 1000000",object_name = "OrderItems", api_type = "Bulk 1.0", verbose = TRUE, max_attempts = 1000, control = list(`Sforce-Enable-PKChunking`= TRUE))
I'm at a loss for what seems like a simple syntax issue, and can't find any specific examples online of PKChunking in a SalesforceR query, despite the documentation saying this can be done. I'd greatly appreciate some guidance here.
Link 1: https://developer.salesforce.com/docs/atlas.en-us.api_asynch.meta/api_asynch/asynch_api_bulk_query_processing.htm
Link 2: https://cran.r-project.org/web/packages/salesforcer/salesforcer.pdf#Rfn.sf.Rul.control
The error listed above was a bug that exists in {salesforcer} v0.1.4 and below. As of v0.2.0, the issue has been resolved and you can execute a bulk query with PKChunking enabled like this:
# install.packages('salesforcer')
library(salesforcer)
# authenticate using username, password, and security token ...
sf_auth(username = "test#gmail.com",
password = "{PASSWORD_HERE}",
security_token = "{SECURITY_TOKEN_HERE}")
# ... or using OAuth 2.0 authentication
sf_auth()
# execute the query
contacts <- sf_query("SELECT Id, Name FROM Contact Limit 10000",
object_name = "Contact",
api_type = "Bulk 1.0",
PKChunkingHeader = list(`Sforce-Enable-PKChunking` = TRUE),
# alternatively you can turn on using specific chunking options like this ...
# PKChunkingHeader = list(`chunkSize` = 500),
interval_seconds = 10,
max_attempts = 200)
contacts
#> # A tibble: 383 x 2
#> Id Name
#> <chr> <chr>
#> 1 0033s000012NdFhAAK John Doe
#> 2 0033s000012NkzwAAC Jane Doe
#> 3 0033s000012NkzxAAC Jane Doe
#> 4 0033s000012NkzyAAC Jane Doe
#> 5 0033s000012NkzzAAC Jane Doe
#> # … with 378 more rows
Related
It looks like things are going wrong on line 9 for me. Here I wish to push a new copy of the TagsTable into a dictionary. I'm aware that once a namedtuple field is recorded, it can not be changed. However, results baffle me as it looks like the values do change - when this code exits all entries of mp3_tags[ any of the three dictionary keys ].date are set to the last date of "1999_03_21"
So, two questions:
Is there a way to get a new TagsTable pushed into the dictionary ?
Why doesnt the code fail and not allow the second (and even third) date to be written to the TagsTable.date field (since it seems to be references to the same namedtuple) ? I thought you could not write a second value ?
from collections import namedtuple
2 TagsTable = namedtuple('TagsTable',['title','date','subtitle','artist','summary','length','duration','pub_date'])
3 mp3files = ['42-001.mp3','42-002.mp3','42-003.mp3']
4 dates = ['1999_01_07', '1999_02_14', '1999_03_21']
5
6 mp3_tags = {}
7
8 for mp3file in mp3files:
9 mp3_tags[mp3file] = TagsTable
10
11 for mp3file,date_string in zip(mp3files,dates):
12 mp3_tags[mp3file].date = date_string
13
14 for mp3file in mp3files:
15 print( mp3_tags[mp3file].date )
looks like this is the fix I was looking for:
from collections import namedtuple
mp3files = ['42-001.mp3','42-002.mp3','42-003.mp3']
dates = ['1999_01_07', '1999_02_14', '1999_03_21']
mp3_tags = {}
for mp3file in mp3files:
mp3_tags[mp3file] = namedtuple('TagsTable',['title','date','subtitle','artist','summary','length','duration','pub_date'])
for mp3file,date_string in zip(mp3files,dates):
mp3_tags[mp3file].date = date_string
for mp3file in mp3files:
print( mp3_tags[mp3file].date )
I am new to MTurk, but have some fluency with R. I am using the MTurkR package the first time, and I am trying to create "micro-batches" that post on MTurk over time. The code I am using can be found below (the XXXX parts are obviously filled with the correct values). I don't get any error messages, the code runs, and posts the HIT both in the Sandbox and in the real correctly. However, the HITs posted do not show up in the Sandbox Requester account, or the real requester account which means - as far as I understand - that I can't evaluate the workers who submit a completion code before they are paid automatically.
Could anyone point out where the error is, and how could I review the HITs?
Thanks.
K
##### Notes:
# 1) Change sandbox from TRUE to FALSE to run live (make sure to test in sandbox first!!)
##### Step 1: Load library, set parameters
#### Load MTurkR library
library(MTurkR)
#### HIT Layout ID
# Layout ID for the choice task
# my_hitlayoutid = "XXXX"
# Layout ID for the choice task in the Sandbox
my_hitlayoutid = "XXXX"
#### Set MTurk credentials
Sys.setenv(
AWS_ACCESS_KEY_ID = "XXXX",
AWS_SECRET_ACCESS_KEY = "XXXX"
)
#### HIT parameters
## Run in sandbox?
sandbox_val <- "FALSE"
## Set the name of your project here (used to retrieve HITs later)
myannotation <- "myannotation"
## Enter other HIT aspects
newhittype <- RegisterHITType(
title = "hope trial",
description = "Description",
reward = "2.5",
duration = seconds(hours = 1),
keywords = "survey, demographics, neighborhoods, employment",
sandbox = sandbox_val
)
##### Step 2: Define functions
## Define a function that will create a HIT using information above
createhit <- function() {
CreateHIT(
hit.type = newhittype$HITTypeId,
assignments = 2,
expiration = seconds(days = 30),
annotation = myannotation,
verbose = TRUE,
sandbox = sandbox_val,
hitlayoutid = my_hitlayoutid
)
}
## Define a function that will expire all running HITs
## This keeps HITs from "piling up" on a slow day
## It ensures that A) HIT appears at the top of the list, B) workers won't accidentally accept HIT twice
# expirehits <- function() {
# ExpireHIT(
# annotation = myannotation,
# sandbox = sandbox_val
# )
#}
##### Step 3: Execute a loop that runs createhit/expirehit functions every hour, and it will log the output to a file
## Define number of times to post the HIT (totalruns)
totalruns <- 2
counter <- 0
## Define log file (change the location as appropriate)
logfile <- file("/Users/kinga.makovi/Dropbox/Bias_Experiment/MTurk/logfile.txt", open="a")
sink(logfile, append=TRUE, type="message")
## Run loop (note: interval is hourly, but can be changed in Sys.sleep)
repeat {
message(Sys.time())
createhit()
Sys.sleep(10)
#expirehits()
counter = counter + 1
if (counter == totalruns){
break
}
}
## To stop the loop before it finishes, click the "STOP" button
## To stop logging, run sink()
You can't see HITs that are created via the API (through MTurkR or otherwise) in the requester website. It's a "feature". You'll have to access the HITs through MTurkR (e.g., SearchHITs() and GetHIT()).
I'm developing a program using Android Studio that runs as expected when the normal APK is run, but if I compile it for Debugging, it crashes before the 1st line of code executes, so setting a brake-point on the 1st line of my code to execute, never trips.
I have already
Completely deleted the program from the test target (nexus 7)
Power offed the tablet and then restarted tablet
Power off/rebooted the development machine
Cleaned the project
Rebuilt the project.
The program still runs as expected when compiled/run without debugging.
Crashes after loading, but before running when compiled/run in debug.
Any ideas on how to try and overcome this problem would be welcome.
Android Studio 2.1.2
compileSdkVersion 21
buildToolsVersion "21.1.2
Copy of exception variable after crash
shadow$klass = {Class#613} "class java.lang.Class"
superClass = {Class#1356} "class java.lang.Exception"
2 = {ArtMethod#3613}
17 = {ArtMethod#3628}
numReferenceStaticFields = 0
verifyErrorClass = null
iFields = {ArtField[2]#3636}
stackState = {Object[22]#3602}
3 = {ArtMethod#3614}
0 = {ArtMethod#3611}
errno = 2
classLoader = null
8 = {ArtMethod#3619}
dexCache = {DexCache#3633}
functionName = "stat"
virtualMethods = {ArtMethod[3]#3639}
shadow$monitor = -1294973706
primitiveType = 0
Exception = {ErrnoException#3590}
16 = {ArtMethod#3627}
value = {char[4]#3607}
ifTable = {Object[2]#3637}
vtable = null
4 = {ArtMethod#3615}
suppressedExceptions = {Collections$EmptyList#3604} size = 0
classSize = 468
functionName = "stat"
6 = {ArtMethod#3617}
errno = 2
hashCode = 3540564
referenceStaticOffsets = 0
1 = {ArtMethod#3612}
13 = {ArtMethod#3624}
detailMessage = null
18 = {ArtMethod#3629}
stackTrace = {StackTraceElement[0]#3603}
sFields = null
dexClassDefIndex = 4
componentType = null
15 = {ArtMethod#3626}
status = 10
10 = {ArtMethod#3621}
shadow$klass = {Class#3428} "class java.lang.String"
shadow$klass = {Class#183} "class android.system.ErrnoException"
19 = {ArtMethod#3630}
12 = {ArtMethod#3623}
5 = {ArtMethod#3616}
referenceInstanceOffsets = -1090519040
cause = {ErrnoException#3590} "android.system.ErrnoException: stat failed: ENOENT (No such file or directory)"
suppressedExceptions = {Collections$EmptyList#3604} size = 0
objectSize = 36
detailMessage = null
shadow$klass = {Class#183} "class android.system.ErrnoException"
dexCacheStrings = {String[23663]#3634}
dexTypeIndex = 6
shadow$monitor = -1665173294
offset = 0
14 = {ArtMethod#3625}
numReferenceInstanceFields = 1
Variables debug info not available
21 = {int[21]#3632}
name = "android.system.ErrnoException"
stackState = {Object[22]#3602}
clinitThreadId = 456
directMethods = {ArtMethod[2]#3635}
7 = {ArtMethod#3618}
20 = {ArtMethod#3631}
9 = {ArtMethod#3620}
11 = {ArtMethod#3622}
count = 4
shadow$monitor = -1353303322
cause = {ErrnoException#3590} "android.system.ErrnoException: stat failed:
ENOENT (No such file or directory)"
stackTrace = {StackTraceElement[0]#3603}
accessFlags = 524305
Well, I solved the problem myself. NOT in a way that I'd like to have solved it, or that in any way explained what caused the problem, but the problem was corrected. - I've been in IT long enough to know you need to do backups, and once I remembered I had them, I pulled the backup of a day before the problem surfaced, restored the entire directory tree when the project resided and PUFF, everything worked. Re-applied any code changes I made in the day after the backup (I keep pretty good paper notes on changes) and away I went.
- BUT -
This doesn't explain why the system was working one day and after a night of sleep, the system wouldn't work the next day. And I still have no idea how a person who received the error message I received would go about debugging/fixing the problem if he/she didn't have a good set of backups.
ALSO – it seems that someone downgraded my question. Maybe it did deserve to be downgraded, but I have no idea why it was downgraded. The question seemed reasonable to me. One day late in the evening the system worked, and the next day it didn't and the information the system offered was posted in the question.
I'd be very happy if anyone could explain to me how a person could extract information from the posted log, to find out what was wrong – or – at least explain why the question was improper enough to warrant a downgrade ?
any reasonable comments would be welcome.
Joe Cullity
I've just started coding ABAP for a few days and I have a task to call the report from transaction SE38 and have
the report's result shown on the screen of the WebDynPro application SE80.
The report take the user input ( e.g: Material Number, Material Type, Plant, Sale Org. ) as a condition for querying, so the WebDynPro application must allow user to key in this parameters.
In some related article they were talking about using SUBMIT rep EXPORTING LIST TO MEMORY and CALL FUNCTION 'LIST_FROM_MEMORY' but so far I really have no idea to implement it.
Any answers will be appreciated. Thanks!
You can export it to PDF. Therefore, when a user clicks on a link, you run the conversion and display the file in the browser window.
To do so, you start by creating a JOB using the following code below:
constants c_name type tbtcjob-jobname value 'YOUR_JOB_NAME'.
data v_number type tbtcjob-jobcount.
data v_print_parameters type pri_params.
call function 'JOB_OPEN'
exporting
jobname = c_name
importing
jobcount = v_number
exceptions
cant_create_job = 1
invalid_job_data = 2
jobname_missing = 3
others = 4.
if sy-subrc = 0.
commit work and wait.
else.
EXIT. "// todo: err handling here
endif.
Then, you need to get the printer parameters in order to submit the report:
call function 'GET_PRINT_PARAMETERS'
exporting
destination = 'LP01'
immediately = space
new_list_id = 'X'
no_dialog = 'X'
user = sy-uname
importing
out_parameters = v_print_parameters
exceptions
archive_info_not_found = 1
invalid_print_params = 2
invalid_archive_params = 3
others = 4.
v_print_parameters-linct = 55.
v_print_parameters-linsz = 1.
v_print_parameters-paart = 'LETTER'.
Now you submit your report using the filters that apply. Do not forget to add the job parameters to it, as the code below shows:
submit your_report_name
to sap-spool
spool parameters v_print_parameters
without spool dynpro
with ...(insert all your filters here)
via job c_name number v_number
and return.
if sy-subrc = 0.
commit work and wait.
else.
EXIT. "// todo: err handling here
endif.
After that, you close the job:
call function 'JOB_CLOSE'
exporting
jobcount = v_number
jobname = c_name
strtimmed = 'X'
exceptions
cant_start_immediate = 1
invalid_startdate = 2
jobname_missing = 3
job_close_failed = 4
job_nosteps = 5
job_notex = 6
lock_failed = 7
others = 8.
if sy-subrc = 0.
commit work and wait.
else.
EXIT. "// todo: err handling here
endif.
Now the job will proceed and you'll need to wait for it to complete. Do it with a loop. Once the job is completed, you can get it's spool output and convert to PDF.
data v_rqident type tsp01-rqident.
data v_job_head type tbtcjob.
data t_job_steplist type tbtcstep occurs 0 with header line.
data t_pdf like tline occurs 0 with header line.
do 200 times.
wait up to 1 seconds.
call function 'BP_JOB_READ'
exporting
job_read_jobcount = v_number
job_read_jobname = c_name
job_read_opcode = '20'
importing
job_read_jobhead = v_job_head
tables
job_read_steplist = t_job_steplist
exceptions
invalid_opcode = 1
job_doesnt_exist = 2
job_doesnt_have_steps = 3
others = 4.
read table t_job_steplist index 1.
if not t_job_steplist-listident is initial.
v_rqident = t_job_steplist-listident.
exit.
else.
clear v_job_head.
clear t_job_steplist.
clear t_job_steplist[].
endif.
enddo.
check not v_rqident is initial.
call function 'CONVERT_ABAPSPOOLJOB_2_PDF'
exporting
src_spoolid = v_rqident
dst_device = 'LP01'
tables
pdf = t_pdf
exceptions
err_no_abap_spooljob = 1
err_no_spooljob = 2
err_no_permission = 3
err_conv_not_possible = 4
err_bad_destdevice = 5
user_cancelled = 6
err_spoolerror = 7
err_temseerror = 8
err_btcjob_open_failed = 9
err_btcjob_submit_failed = 10
err_btcjob_close_failed = 11
others = 12.
If you're going to send it via HTTP, you may need to convert it to BASE64 as well.
field-symbols <xchar> type x.
data v_offset(10) type n.
data v_char type c.
data v_xchar(2) type x.
data v_xstringdata_aux type xstring.
data v_xstringdata type xstring.
data v_base64data type string.
data v_base64data_aux type string.
loop at t_pdf.
do 134 times.
v_offset = sy-index - 1.
v_char = t_pdf+v_offset(1).
assign v_char to <xchar> casting type x.
concatenate v_xstringdata_aux <xchar> into v_xstringdata_aux in byte mode.
enddo.
concatenate v_xstringdata v_xstringdata_aux into v_xstringdata in byte mode.
clear v_xstringdata_aux.
endloop.
call function 'SCMS_BASE64_ENCODE_STR'
exporting
input = v_xstringdata
importing
output = v_base64data.
v_base64data_aux = v_base64data.
while strlen( v_base64data_aux ) gt 255.
clear t_base64data.
t_base64data-data = v_base64data_aux.
v_base64data_aux = v_base64data_aux+255.
append t_base64data.
endwhile.
if not v_base64data_aux is initial.
t_base64data-data = v_base64data_aux.
append t_base64data.
endif.
And you're done!
Hope it helps.
As previous speakers said, you should do extensive training before implementing such stuff in productive environment.
However, calling WebdynPro ABAP within report can be done with the help of WDY_EXECUTE_IN_PLACE function module. You should pass there Webdyn Pro application and necessary parameters.
CALL FUNCTION 'WDY_EXECUTE_IN_PLACE'
EXPORTING
* PROTOCOL =
INTERNALMODE = ' '
* SMARTCLIENT =
APPLICATION = 'Z_MY_WEBDYNPRO'
* CONTAINER_NAME =
PARAMETERS = lt_parameters
SUPPRESS_OUTPUT =
TRY_TO_USE_SAPGUI_THEME = ' '
IMPORTING
OUT_URL = ex_url
.
IF sy-subrc <> 0.
* Implement suitable error handling here
ENDIF.
I am trying to write a generic upsert for a tick Database in R.
The python code would be:
collection.update({'symbol':'somesymbol', 'sha':'SoM3__W3|Re|7__Sh#'},
{'$set':{segment:5},
'$addToSet': {'parent':parent_id}}},
upsert=True)
In R I am using rmongodb and trying to build the BSON Objects
#get the query
mtch_b<-mongo.bson.buffer.create()
mongo.bson.buffer.append(mtch_b, "symbol", "somesymbol")
mongo.bson.buffer.append(mtch_b, "sha", "SoM3__W3|Re|7__Sh#")
mtch<-mongo.bson.from.buffer(mtch_b)
#set the segment
qry_b<-mongo.bson.buffer.create()
mongo.bson.buffer.start.object(qry_b, "$set")
mongo.bson.buffer.append(qry_b, "segment", 5)
mongo.bson.buffer.start.object(qry_b, "$addToSet")
mongo.bson.buffer.append(qry_b, "parent", "Initial")
mongo.bson.buffer.finish.object(qry_b) #end of $addtoSet object
mongo.bson.buffer.finish.object(qry_b) #end of $set object
qry_bsn <-mongo.bson.from.buffer(qry_b)
mongo.update(mongo, "M__test.tmp", mtch, qry_bsn, flags=mongo.update.upsert)
When I run this I get an error:
"The dollar ($) prefixed field '$addToSet' in '$addToSet' is not valid for storage."
looking at the qry_bsn:
qry_bsn
$set : 3
segment : 4
0 : 1 1.000000
1 : 1 2.000000
2 : 1 3.000000
3 : 1 4.000000
$addToSet : 3
parent : 2 Initial
When I remove the $addToSet, append and finish objects of the $addToSet object the query runs fine.
Any help on how to do this would be much appreciated.
I can't find reason for not using mongo.bson.from.list. It make all mongo.bson.buffer.* calls for you. And it is much less chance to produce a bug with bson construction.
query <- mongo.bson.from.list(list("symbol" = "somesymbol", "sha" = "SoM3__W3|Re|7__Sh#"))
upd_obj <- mongo.bson.from.list(list('$set' = list('segment' = 1:4), '$addToSet' = list('parent' = 'PARENT_ID')))
mongo.update(mongo = mongo, ns = "M__test.tmp", criteria = query, objNew = upd_obj, flags=mongo.update.upsert)