I am using ScriptAssert and the script work's fine. Now I would like to move the script to properties file.
I am able to read PONumber.mandatory as part of message. I am unable to read PONumber.mandatory.script as part of script. Any help is appreciated.
ValidationMessages.properties
PONumber.mandatory.script=!(_.OBJSTATE.equals('RELEASED') && (_.PONumber == '' ))
PONumber.mandatory=PONumber value is Mandatory
In Java:
#ScriptAssert(lang="javascript",script="{PONumber.mandatory.script}",alias = "_",message="{PONumber.mandatory}")
Related
I have a JSON file that is loaded by two different Autoloaders.
One uses schema evolution and besides replacing spaces in the json property names, writes the json directly to a delta table, and I can see all the values are there properly.
In the second one I am mapping to a defined schema and only use a subset of properties. So use a lot of withColumn and then a select to narrows to my defined column list.
Autoloader definition:
df = (spark
.readStream
.format('cloudFiles')
.option('cloudFiles.format', 'json')
.option('multiLine', 'true')
.option('cloudFiles.schemaEvolutionMode','rescue')
.option('cloudFiles.includeExistingFiles','true')
.option('cloudFiles.schemaLocation', bronze_schema)
.option('cloudFiles.inferColumnTypes', 'true')
.option('pathGlobFilter','*.json')
.load(upload_path)
.transform(lambda df: remove_spaces_from_columns(df))
.withColumn(...
Writer:
df.writeStream.format('delta') \
.queryName(al_stream_name) \
.outputMode('append') \
.option('checkpointLocation', checkpoint_path) \
.option('mergeSchema', 'true') \
.trigger(once = True) \
.table(bronze_table)
Issue is that some of the source columns are ok load and I get their values, and others are constantly null in the output table.
For example:
.withColumn('vl_rating', col('risk_severity.value')) # works
.withColumn('status', col('status.name')) # always null
...
.select(
'rating',
'status',
...
json is quite simple, these are all string values, they are always populated. The same code works against another simular json file in another autoloader without issue.
I have run out of ideas to fault find on this. My imports are minimal, outside of Autoloader the JSON loads fine.
e.g
%python
import pyspark.sql.functions as psf
jsontest = spark.read.option('inferSchema','true').json('dbfs:....json')
df = jsontest.withColumn('status', psf.col('status.name')).select('status')
display(df)
Results in the values of the status.name property of the json file
Any ideas would be greatly appreciated.
I have found generally what is causing this. Interesting cause!
I am scanning a whole directory of json files, and the schema evolves over time (as expected). But when I clear out the autoloader schema and checkpoint directories and only scan the latest json file it all works correctly.
So what I surmise is that something in schema evolution with the older json files causes Autoloader to get into a state where it will not put certain properties into the stream to the writer.
If anyone has any recommendation on how to implement some data quality analysis in an Autoloader I would be most appreciative if you would share.
I'm newbie to nacl. And I find out there are so many 0 byte files in the directory (nacl_sdk/pepper_38/toolchain/win_*/bin).
When I change the project platform to NaCl64 and compile(hello_nacl_cpp), there comes out an error
(error MSB6006: “D:\nacl_sdk\pepper_38\toolchain\win_x86_newlib\bin\x86_64-nacl-gcc.exe”已退出，代码为 -1)
But I can debug the example "hello_world_gles" with PPAPI platform, so I'm not sure the environment is ok.
Anyone can tell me something? Thanks!
Answer my question.
As #binji says we should use cygtar.py(which is in the dirctory sdk_tools) to extract the file.
Here we go:
Open cygtar.py with your text editor, you will find a class named CygTar who is the real worker.
Move dwon, and insert a snippet of code below Main function.
def MyLogic():
os.chdir('D:\\nacl_sdk\\sdk')
# tar = CygTar('naclports.tar.bz2', 'r', True) #here must use linux file path
tar = CygTar('naclsdk_win.tar.bz2', 'r', True)
tar.Extract()
Then replace sys.exit(Main(sys.argv)) with sys.exit(MyLogic()) at last of file.That all.
Note: If you have learned python, you will know code indent is very important in python, be careful.
And the final code should looks like this:
I am writing a CGI script in Perl with a section of embedded R script which produces a graph. The original data filename is unknown as it has been uploaded by the CGI script and is stored in a Perl variable called $filename.
My question is that I now would like to open that file in R using read.table(). I am using Statistics::R and so I have tried:
my $R = Statistics::R->new();
$R->set('filename',$filename);
my $out1 = $R->run(
q`rm(list=ls())`,
# Fetch data
q`setwd("/var/www/uploads")`,
q`peakdata<-read.table(filename, sep="",col.names=c("mz","intensity","ionsscore","matched","query","index","hit"))`,
q`attach(peakdata)` ...etc
I can get this to work ONLY if I change $filename into something static and known like 'data.txt' before trying to open the file in read.table - is there a way for me to open a file with a variable for a name?
Thank you in advance.
One possible way to do this is by doing a little more work in Perl.
This is untested code, to give you some ideas:
my $filename = 'fileNameIGotFromSomewhere.txt'
my $source_dir = '/var/www/uploads';
my $file = "$source_dir/$fielname";
# make sure we can read it
unless ( -r $file ) {
die 'can read that data file: $!";
}
Then instead of $R->set, you could interpolate the file name into the R program. Where you've used the single-quote operator, use the double-quote operator instead:
So instead of:
q`peakdata<-read.table(filename, sep="",col.names= .... )`
Use:
qq`peakdata<-read.table($filename, sep="",col.names= .... )`
Now this looks like it would be inviting problems similar to SQL/Code Injections, so that's why I put int the logic to insure that the file exists and is readable. You might be able to think other checks to add to safeguard your use of user-supplied info.
I've got AutoIt code as indicated here and I'm encountering issues with DLLOpen() on line 289 of the code i.e it is failing and returning -1.
I've read up here on what DllOpen() should be returning and I cannot figure out why it is failing.
I've also made sure my Dll file is in the same folder as my .exe file.
In addition I've also specified a specific file path to the Dll file.
Any ideas are greatly appreciated.
If it is working in Skype, it should work compiled.
Dont forget to
FileInstall ( "source", "dest" [, flag] )
Also dont forget about
FileChangeDir ( "path" ) ;Changes the current working directory.
I have a bunch of SQL in a file, which creates the tables and so forth. The problem is that the .read command simply returns "can't open xxx" when I try to execute it. I've set the permissions to everybody read/write, but that didn't help. I can cat the file from the command line and see it fine.
This is under Mac OS 10.6.3.
Anybody have any idea here? Thanks!
Here's the source code in shell.c, where the sqlite3 utility executes the .read and tries to read the .sql file:
if( c=='r' && strncmp(azArg[0], "read", n)==0 && nArg==2 ){
FILE *alt = fopen(azArg[1], "rb");
if( alt==0 ){
fprintf(stderr,"can't open \"%s\"\n", azArg[1]);
}else{
process_input(p, alt);
fclose(alt);
}
}else
You can see that the utility will print "can't open xxx" only when the call to fopen fails. Pity they don't print errno for you, but this info should really help you narrow down the problem as not specific to sqlite.
You're either specifying the path wrong (try quoting it), you don't have permission, or the file really doesn't exist (meaning that there's something different in how you're using cat and .read, like the current directory is different).
Watch out for the ; at the end. .read does not like ;'s.