I want the output for my reducer to be zipped (preferably gzip).
I am successfully able to launch an EMR job using the c# awssdk but do not know how to change the job confiugration for desired result.
I understand i need to set the following property for the emr job
mapred.output.compress: true
how do i do this in c# code?
also how do I make it produce gzip files?
Thanks
String[] zipArgs = new [] { "-jobconf", "mapred.output.compress=true"};
HadoopJarStepConfig config = new StreamingStep()
.WithInputs()
.WithOutput()
.WithMapper()
.WithReducer()
.ToHadoopJarStepConfig()
.WithArgs(zipArgs)
;
you can find more info at
http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-plan-output-compression.html
Related
While using Corda Enterprise 4.2, we have managed to create a
biz-generated-migration....jar which seems to be a pre-requisite to everything, even
when using an H2 database. We were not able to use this .jar file
while executing our tests, even when referencing this brand new .jar
file in our test setup, like this:
val BIZ_COMPONENT_VERSION="1.0.0"
val DEFAULT_MOCK_NETWORK = MockNetwork(
cordappPackages = listOf(
"package.subpkg-infra.cd.contract",
"package.subpkg-infra.contract",
"package.subpkg-infra.flow",
"package.subpkg-infra.cd.flow",
"package.subpkg-infra.cd.pend.flow",
"package.subpkg-infra.schema",
"package.subpkg-cordapp:biz-generated-migration:$BIZ_COMPONENT_VERSION",
"package.subpkg-cordapp:biz-component-base:$BIZ_COMPONENT_VERSION",
"package.subpkg-cordapp:biz-component-core:$BIZ_COMPONENT_VERSION",
"package.subpkg-cordapp:biz-component-interact:$BIZ_COMPONENT_VERSION"
),
notarySpecs = listOf(MockNetworkNotarySpec(DUMMY_NOTARY_NAME)))
Every test fails, complaining about not finding the migration for the
schema.
How can we accomplish to use this generated migration.jar file to
enable our testing ? Or is this approach completely misused ?
Following options have been suggested to fix the issue.
Have you overridden the "migrationResourse"="migration/filename" of MappedSchema class.
make sure to proper XML so this would be not H2 specific.
also could you try to use below network = MockNetwork(MockNetworkParameters(listOf( findCordapp("com.deqode.contracts"), findCordapp("com.deqode.flows"))) ) instead of cordappPackages.
Can you confirm if you are using Windows
I created this script to determine if the Session.getScriptTimeZone() would draw the time zone from the library file rather than AppMaker. Here's the script:
function getFormattedDateString(dt,format){
var format=format||"E MMM dd, yyyy HH:mm";
var dt=dt||new Date();
return Utilities.formatDate(new Date(dt), Session.getScriptTimeZone(), format);
}
I tested it in another script with the following code:
function test(){
Logger.log(AMSLib.getFormattedDateString(new Date()));
}
I went into AppMaker and this dialog:
I've tried the Script ID from here:
I've also tried several deployment ID's from the publish from Manifest dialog and I keep getting the same answer:
I've also tried the Project Key which is used with other apps scripts to load libraries.
I don't know what to try next.
First things first, you need to publish your Apps Script app, after that it'll be assigned Script ID(by the way it can also be found in the published app URL). Once you have Script ID, you can specify it in App Maker and select library version you want to use:
To access library's functions you need to use name specified in the object setting:
// Server side library call
var result = MyLibraryName.doSomeCoolStuff();
App Maker should be smart enough and pickup all library's public functions for autocomplete.
Learn more:
https://developers.google.com/apps-script/guides/libraries
https://developers.google.com/appmaker/scripting/libraries
That's really odd to not have a script key there. You might make a copy of your script, something might be corrupted. You can also get the key from the URL ex: https://script.google.com/a/ignitesynergy.com/d/1oHnk_xl76KagGS4g7O2pC1MM4R3iZR8-7FlmzKXxRDtO1o5nDU2/edit
Remember to File-> Manage Versions and create a version. You also need to set the sharing to public.
I am launching emr cluster using Java API but not able to associate a tag to it. Pl can you help me on this.
Using EMR CLI, it is very easy as below but I have to do this using my Java code
./elastic-mapreduce --create --alive --tag tagKey=stackOverflow
If you need more details on this, pl let me know..
Thanks in advance.
Regards,
Vineet
In earlier versions of EMR Java SDK there wasn't a way to add tags(not a direct one for sure) to add tags, but in newer versions of the Java SDK for EMR, there is a method named addTags(Collection<Tag> tags), using which you can add tags to the resources(EC2) being launched as part of the EMR cluster.
You would use it as follows:
AWSCredentials credentials = new BasicAWSCredentials(accessKey, secretKey);
AmazonElasticMapReduce emr = new AmazonElasticMapReduceClient(credentials);
List<Tag> tags = new ArrayList<Tag>();
Tag stackOverflowTag = new Tag();
stackOverflowTag.setKey("stackOverflow");
tags.add(stackOverflowTag);
AddTagsRequest addTagsRequest = new AddTagsRequest();
addTagsRequest.setTags(tags);
emr.addTags(addTagsRequest);
StepFactory stepFactory = new StepFactory();
// set up the cluster to launch and add steps
RunJobFlowResult result = emr.runJobFlow(request);
Use com.amazonaws.services.elasticmapreduce.model.Tag class in order to create tags, as there are many Tag classes present in the SDK, and for a moment I had imported the wrong one as well.
Read the doc here.
I try to run Openstack-Swift java sdk sample.
I have swift and keystone project to use swift only.
I found this project : https://github.com/woorea/openstack-java-sdk
But, I don't know how to run this project in Window Eclipse.
Should I build all project(nova, etc..) in maven?
Do you know how to run this project or website that post run-way in regular sequence?
#stream
I have not tried Woorea but i know a lot many developers are using Jclouds, the link http://developer.rackspace.com/#home-sdks has well documented guide with example how to use the Java SDK.
Hope it helps.
looks like you can build SWIFT independently (part of woorea peoject)
as it states in the readme file here:
(com.woorea swift-client 3.0.0-SNAPSHOT)
https://github.com/woorea/openstack-java-sdk
the Maven artifact ID should be:
openstack-java-sdk
Here is a nice toturial that can be of hand:
https://github.com/woorea/openstack-java-sdk/wiki/Swift-Tutorial
it has the example for the java api for using SWIFT,
for example, this code snippet (more details in the link):
Properties properties = System.getProperties();
properties.put("verbose", "true");
properties.put("auth.credentials", "passwordCredentials");
properties.put("auth.username", "demo");
properties.put("auth.password", "secret0");
properties.put("auth.tenantName", "demo");
properties.put("identity.endpoint.publicURL","http://192.168.1.43:5000/v2.0");
OpenStackClient openstack = OpenStackClient.authenticate(properties);
AccountResource account = openstack.getStorageEndpoint();
account.container("hellocontainer").put();
account.container("hellocontainer").object("dir1").put();
account.container("hellocontainer").object("test1")
.put(new File("pom.xml"), new SwiftStorageObjectProperties() {{
setContentType("application/xml");
getCustomProperties().putAll(new HashMap<String, String>() {{
put("customkey.1", "customvalue.1");
}});
}});
List<SwiftStorageObject> objects = account.container("hellocontainer").get();
just keep in mind that when using openstack's API you will most likely need to authenticate (get tokens etc..) so that you will need the Keystone lib as well
www.programcreek.com/java-api-examples/index.php?api=com.woorea.openstack.keystone.Keystone
hope this helps.
Im writing a module for drupal, Im trying to create a node from my module, everything is fine , I only have 1 problem with creating an image , The image exist on different server, so I want to grab the page and insert it , I install module http://drupal.org/project/filefield_sources , which has remote option , I search in the module code , I could not find the function that he used for this process, module work very nice from interface , but how i make it do the job from code ? which function should i call and what parameter should i pass .
I'm over Drupal 6.
Hopefully you're using Drupal 7...
The system_retrieve_file() function will download a file from a remote source, copy it from temp to a specified destination and optionally save it to the file_managed table if you want it to be managed.
$managed = TRUE; // Whether or not to create a Drupal file record
$path = system_retrieve_file($url, 'public://my_files/', $managed);
If you want to get the file object immediately after you've done this, the following is the only way I've found so far:
$file = file_load(db_query('SELECT MAX(fid) FROM {file_managed}')->fetchField());
get fid using $path->fid. no need to mysql