In pig script, I know that we can use passed-in parameters like the following:
%declare CMD `$mycmd $date`
A = load '/data/mydata/$CMD';
B = filter A by $0>'5';
But how can I save the parameters into a file?
Thanks
Related
In OpenMDAO V3.1 I am using an ExternalCodeComp to execute a CFD code. Typically, I would call it as such:
mpirun nodet_mpi --design_run
If the above call is made in the appropriate directory, then it will find the appropriate run file and execute the CFD run. I have tried command args for the ExternalCodeComp;
execute = ['mpirun', 'nodet_mpi', '--design_run']
execute = ['mpirun', 'nodet_mpi --design_run']
execute = ['mpirun nodet_mpi --design_run']
I either get an error such as:
RunTimeError: 255, execvp error on file "nodet_mpi --design_run" (No such file or directory)
Or that the command cannot be found.
Is there any way to setup the execute statement to include commandline args for the flow solver when an input file is not defined?
Thanks in advance!
One detail in your question seems incorrect, you state that you have tried execute = "...". The ExternalCodeComp uses an option called command. I will assume that you are using the correct option in your code.
The most correct form to use is the list with all arguments as single entries in the list:
self.options['command'] = ['mpirun', 'nodet_mpi', '--design_run']
Your error msg seems to indicate that the directory that OpenMDAO is running in is not the same as the directory you would like to execute the CFD code from. The absolute simplest solution would be to make sure that you are in the correct directory via cd in the terminal window before executing your python script.
However, there is likely a reason that your python script is in a different place so there are other options I can suggest:
You can use a combination of os.getcwd() and os.chdir() inside the compute method that you have implemented to make sure you switch into and out of the working directory for the CFD code.
If you would like to, you can modify the entries of the list you've assigned to the self.options['command'] option on the fly within your compute method. You would again be relying on some of the methods in the os module for help. os.path.exists can be used to test if the specific input files you need exist or not, and you can modify the command option accordingly.
For option 2, code would look something like this:
def compute(self, inputs, outputs):
if os.path.exists('some_input.file'):
self.options['command'] = ['mpirun', 'nodet_mpi', '--design_run']
else:
self.options['command'] = ['mpirun', 'nodet_mpi', '--design_run', '--other_options']
# the parent compute function actually runs the external code
super().compute(inputs, outputs)
I tried to execute a store operation in pig hdfs mode but it is throwing me the following error message
org.apache.pig.backend.hadoop.executionenginemapreduceLayer.MapReduceLauncher-Failed.
I checked with the dump operation but it too gave me the same error
start-dfs.sh
start-yarn.sh
pig
A = LOAD '/home/hduser/sample.txt' USING PigStorage(' ') as (ID:int, Name:chararray, .......) ;
B = FOREACH A GENERATE ID, NAME
STORE B INTO '/home/hduser' USING PigStorage(' ');
script for STORE and DUMP operation.
I need help with passing an argument as a variable name in an R script from the terminal. I'll run the script as follows:
R < script.R --args "hello"
And, in the script there should be something like this:
args <- commandArgs(trailingOnly = TRUE)
assign(args[1],24)
save(args[1], file="output.RData")
But, I need to take the argument as the variable name. What I mean is the following: If I run the script with "numbers" argument, the variable name inside the script should be numbers.
assign(args[1], 24)
does the trick. But, inside the save function, args[1] does not work. How can I pass it as a variable name?
Does it work if you try
saveRDS(get(args[1]),file="output.rds")
?
You won't get a text file with the save function. If you want its text version you would need to use `dump". This would "work" despirte the extention. The file ois still an .Rdata file event without the extension:
arg=1
argname="reports"
assign(argname, arg)
reports
#[1] 1
save(reports, file="test.txt")
rm(reports)
rm(argname)
rm(arg)
load("test.txt")
To use dump:
dump('reports', file="test2.txt")
This would appear in that file. It should be parse-able (and readable to humans) R code:
reports <-
1
I am automating instance creation using OpenstackSDK and passing bash script with commands as userdata. But the script does not excute even though the instance is crated. When I do this manually via GUI, the bash scripts executes fine to the newly created instance.
#Reading bash script
with open('elk.sh', 'r') as f:
init_script = f.read()
server = conn.compute.create_server(
name=name,
image_id=IMAGE_ID,
flavor_id=FLAVOUR_ID,
networks=[{"uuid": NETWORK_ID}],
user_data=init_script, # pass script to the instance
key_name=KEY_PAIR
)
Note: Also tried to encode as Base64 file butstill failed with
is not JSON serializable.
Code snippet:
with open(USER_DATA,'r') as file:
f = file.read()
bytes_content = bytes(f,encoding='utf-8')
init_script = base64.b64encode(bytes_content)
Can anyone advice on this, please?
Thanks
Python3 handles string and binary differently. Also, to pass bash/cloud-config file to --user_data via OpenstackSDK, it has to be base46 encoded.
Code snippet:
with open(USER_DATA,'r') as file:
f = encodeutils.safe_encode(file.read().encode('utf-8'))
init_script = base64.b64encode(f).decode('utf-8')
I am writing a CGI script in Perl with a section of embedded R script which produces a graph. The original data filename is unknown as it has been uploaded by the CGI script and is stored in a Perl variable called $filename.
My question is that I now would like to open that file in R using read.table(). I am using Statistics::R and so I have tried:
my $R = Statistics::R->new();
$R->set('filename',$filename);
my $out1 = $R->run(
q`rm(list=ls())`,
# Fetch data
q`setwd("/var/www/uploads")`,
q`peakdata<-read.table(filename, sep="",col.names=c("mz","intensity","ionsscore","matched","query","index","hit"))`,
q`attach(peakdata)` ...etc
I can get this to work ONLY if I change $filename into something static and known like 'data.txt' before trying to open the file in read.table - is there a way for me to open a file with a variable for a name?
Thank you in advance.
One possible way to do this is by doing a little more work in Perl.
This is untested code, to give you some ideas:
my $filename = 'fileNameIGotFromSomewhere.txt'
my $source_dir = '/var/www/uploads';
my $file = "$source_dir/$fielname";
# make sure we can read it
unless ( -r $file ) {
die 'can read that data file: $!";
}
Then instead of $R->set, you could interpolate the file name into the R program. Where you've used the single-quote operator, use the double-quote operator instead:
So instead of:
q`peakdata<-read.table(filename, sep="",col.names= .... )`
Use:
qq`peakdata<-read.table($filename, sep="",col.names= .... )`
Now this looks like it would be inviting problems similar to SQL/Code Injections, so that's why I put int the logic to insure that the file exists and is readable. You might be able to think other checks to add to safeguard your use of user-supplied info.