Application Crash when attempting to get child from Firebase - firebase

MY application is crashing on line 43:
float value = dataSnapshot.child("temp").getValue(float.class);
When i remove the .child from the addChildEventListener and from the Datasnapshot , the app will display the float value from the database.
My aim is to display both a float value and then add in another line to display the Humidity value my database will hold.
so i will be attempting .child("temp").child("humi") to be displayed on a listview.
Can anybody steer me in the right direction or help me resolve why i cant specify child entries.
ListView listview = (ListView) findViewById(R.id.listview);
adapter = new ArrayAdapter<String>(this, android.R.layout.simple_expandable_list_item_1, list);
listview.setAdapter(adapter);
//INITIALIZE FIREBASE DB
db = FirebaseDatabase.getInstance().getReference();
db.child("temp").limitToLast(1).addChildEventListener(new ChildEventListener() {
#Override
public void onChildAdded(DataSnapshot dataSnapshot, String s) {
float value = dataSnapshot.child("temp").getValue(float.class);
list.add("System Temperture: " + value);
adapter.notifyDataSetChanged();
}
#Override
public void onChildChanged(DataSnapshot dataSnapshot, String s) {
adapter.clear();
float value = dataSnapshot.child("temp").getValue(float.class);
list.add("System Temperture: " + value);
adapter.notifyDataSetChanged();
}
#Override
public void onChildRemoved(DataSnapshot dataSnapshot) {
}
#Override
public void onChildMoved(DataSnapshot dataSnapshot, String s) {
}
#Override
public void onCancelled(DatabaseError databaseError) {
}
});
}
JSON value from firebase
{
"humi" : {
"-KaxKLD00Ib97d1dRbvR" : 17.0
},
"temp" : {
"-KaxKL8G17y4ezfdBMDP" : 21.0
}
}
Logcat
01-20 18:32:18.675 16935-16935/com.m.j.project.josephmay E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.m.j.project.josephmay, PID: 16935
java.lang.NullPointerException: Attempt to invoke virtual method 'float java.lang.Float.floatValue()' on a null object reference
at com.m.j.project.josephmay.MainActivity$1.onChildAdded(MainActivity.java:43)
at com.google.android.gms.internal.zzblz.zza(Unknown Source)
at com.google.android.gms.internal.zzbnz.zzYj(Unknown Source)
at com.google.android.gms.internal.zzboc$1.run(Unknown Source)
at android.os.Handler.handleCallback(Handler.java:739)
at android.os.Handler.dispatchMessage(Handler.java:95)
at android.os.Looper.loop(Looper.java:158)
at android.app.ActivityThread.main(ActivityThread.java:7229)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1230)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1120)
01-20 18:32:26.645 16935-17099/com.m.j.project.josephmay W/DynamiteModule: Local module descriptor class for com.google.firebase.auth not found.
01-20 18:32:26.645 16935-17099/com.m.j.project.josephmay W/ResourcesManager: getTopLevelResources: /data/app/com.google.android.gms-1/base.apk / 1.0 running in com.m.j.project.josephmay rsrc of package com.google.android.gms

I think I was running into a similar issue and this response from Frank helped me: https://groups.google.com/forum/#!topic/firebase-talk/-cdTDFBOmMw.
When you are creating the new record within Firebase, are you doing setValue() for "humi" and "temp" separately or doing setValue() for the entire object? Try switching it to setValue() for the whole object.

Related

RetryingBatchErrorHandler - Offset commit handling

I'm using spring-kafka 2.3.8 and I'm trying to log the recovered records and commit the offsets using RetryingBatchErrorHandler. How would you commit the offset in the recoverer?
public class Customizer implements ContainerCustomizer{
private static ConsumerRecordRecoverer createConsumerRecordRecoverer() {
return (consumerRecord, e) -> {
log.info("Number of attempts exhausted. parition: " consumerRecord.partition() + ", offset: " + consumerRecord.offset());
# need to commit the offset
};
}
#Override
public void configure(AbstractMessageListenerContainer container) {
container.setBatchErrorHandler(new RetryingBatchErrorHandler(new FixedBackOff(5000L, 3L), createConsumerRecordRecoverer()));
}
The container will automatically commit the offsets if the error handler "handles" the exception, unless you set the ackAfterHandle property to false (it is true by default).
EDIT
This works as expected for me:
#SpringBootApplication
public class So69534923Application {
private static final Logger log = LoggerFactory.getLogger(So69534923Application.class);
public static void main(String[] args) {
SpringApplication.run(So69534923Application.class, args);
}
#KafkaListener(id = "so69534923", topics = "so69534923")
void listen(List<String> in) {
System.out.println(in);
throw new RuntimeException("test");
}
#Bean
RetryingBatchErrorHandler eh() {
return new RetryingBatchErrorHandler(new FixedBackOff(1000L, 2), (rec, ex) -> {
this.log.info("Retries exchausted for " + ListenerUtils.recordToString(rec, true));
});
}
#Bean
ApplicationRunner runner(ConcurrentKafkaListenerContainerFactory<?, ?> factory,
KafkaTemplate<String, String> template) {
factory.getContainerProperties().setCommitLogLevel(Level.INFO);
return args -> {
template.send("so69534923", "foo");
template.send("so69534923", "bar");
};
}
}
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.type=batch
so69534923: partitions assigned: [so69534923-0]
[foo, bar]
[foo, bar]
[foo, bar]
Retries exchausted for so69534923-0#2
Retries exchausted for so69534923-0#3
Committing: {so69534923-0=OffsetAndMetadata{offset=4, leaderEpoch=null, metadata=''}}
The log was from the second run.
EDIT2
It does not work with 2.3.x; you should upgrade to a supported version.
https://spring.io/projects/spring-kafka#learn

Extent Reports 3.1.5 not appending after each test run

I am use Extent Reports 3.1.5 and my reports keep getting overwritten on each test run. I have implemented to the bests of my ability the solution found on stack-overflow and various sites and no progress. I have spent 24 hours troubleshooting this issue and need help.
I even went and examined the following sites http://extentreports.com/docs/versions/3/java/#htmlreporter-features
http://extentreports.com/docs/versions/3/java/#htmlreporter-features
and check my code and still could not find the issue. If there is something i am overlooking please help me.
Again my test runs however it only writes the last test ran in my testng.xml file
This is my Base Test Class:
package test;
public class BaseTest {
//-------Page Objects----------
login_Page objBELogin;
poll_Page objCreatePoll;
survey_Page objCreateSurvey;
task_Page objCreateTaskGroup;
discussion_Page objCreateDiscussion;
userprofile_Page objUserProfile;
event_Page objectEvent;
workgroup_Page objectWorkgroup;
workroom_Page objectWorkroom;
//-----------------------------
static WebDriver driver;
static String homePage = "https://automation-ozzie.boardeffect.com/login";
ExtentReports report;
ExtentTest test;//--parent test
#BeforeClass
public void setUp() throws InterruptedException{
//--------Extent Report--------
report = ExtentFactory.getInstance();
//-----------------------------
System.setProperty("webdriver.chrome.driver","C:\\GRID\\chromedriver.exe");
ChromeOptions option = new ChromeOptions();
option.addArguments("disable-infobars");
driver = new ChromeDriver(option);
driver.manage().window().maximize();
driver.get(homePage);
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
}
#BeforeMethod
public void register(Method method) {
String testName = method.getName();
test = report.createTest(testName);
}
#AfterMethod
public void catureStatus(ITestResult result) {
if (result.getStatus()==ITestResult.SUCCESS) {
test.log(Status.PASS,"Test Method named as : "+ result.getName()+" is passed");
}else if(result.getStatus()==ITestResult.FAILURE) {
test.log(Status.PASS,"Test Method named as : "+ result.getName()+" is FAILED");
test.log(Status.FAIL,"Test failure : "+ result.getThrowable());
}
else if(result.getStatus()==ITestResult.SKIP) {
test.log(Status.PASS,"Test Method named as : "+ result.getName()+" is skipped");
}
}
#AfterClass
public void tearDown() throws InterruptedException {
report.flush();
driver.quit();
}
}
public class ExtentFactory {
public static ExtentReports getInstance() {
ExtentHtmlReporter html = new ExtentHtmlReporter("surefire-reports//Extent.html");
html.setAppendExisting(true);
ExtentReports extent = new ExtentReports();
extent.attachReporter(html);
return extent;
}
}
You have to consider adding a unique name to report file for each run.
import java.text.SimpleDateFormat;
import java.util.Date;
public class ExtentFactory {
public static ExtentReports getInstance() {
String out = new SimpleDateFormat("yyyy-MM-dd hh-mm-ss'.html'").format(new Date());
ExtentHtmlReporter html = new ExtentHtmlReporter("surefire-reports//"+out);
ExtentReports extent = new ExtentReports();
extent.attachReporter(html);
return extent;
}
}
You will also need to move the report instance creation to #beforesuite, so that it runs only once for the entire test suite.
#BeforeSuite
public void OneTimesetUp() throws InterruptedException{
//--------Extent Report--------
report = ExtentFactory.getInstance();
//-----------------------------
}

Retrofit Throwing null pointer exception

Showing Null pointer exception** in this code.
** yrWebService.getUserInfo()**
I have tried most of the things but I can't figure it out.
Anyone please help to fix this issue.
The getUserInfo() throwing null pointer exception.
someone help me to solve this. I pasted the interface down there.
public void fetchUserProfile() {
YRWebService.getUserInfo()
.subscribeOn(Schedulers.io())
.observeOn(AndroidSchedulers.mainThread())
.subscribe(new Observer<YRUserResponse>() {
#Override
public void onError(Throwable e) {
Toast.makeText(YarraApplication.this, "Error on server :" +e, Toast.LENGTH_SHORT).show();
}
#Override
public void onComplete() {
}
#Override
public void onSubscribe(Disposable d) {
}
#Override
public void onNext(YRUserResponse yrUserResponse) {
//progressDialog.dismiss();
if (yrUserResponse != null) {
if (yrUserResponse.getUser() != null) {
YRPreferenceManager.setString(PreferenceKeys.USER_PROFILE_RESPONSE,
new Gson().toJson(yrUserResponse));
YRUser yrUser = yrUserResponse.getUser();
YarraApplication.userId = yrUserResponse.getUser().getUser_id();
YarraApplication.access = yrUserResponse.getUser().getGoogle_access_token();
YarraApplication.email = yrUser.getEmail();
YarraApplication.fName = yrUser.getFirst_name();
YarraApplication.lName = yrUser.getLast_name();
YarraApplication.mobile = yrUser.getTelephone();
}else {
// YRUtils.showUnknownErrorDialog(YarraApplication.this);
}
}// else {
// YRUtils.showUnknownErrorDialog(UserProfileActivity.this);
// }
}
});
}
**Interface :::: **
The interface we are using here is this
public interface YRWebService {
#GET("/users")
Observable<YRUserResponse> getUserInfo();
}
public class YRUserResponse {
List<YRError> errors;
YRUser user;
public YRUser getUser() {
return user;
}
public void setUser(YRUser user) {
this.user = user;
}
**retrofit.RetrofitError**:
```
2019-05-06 15:26:49.506 30475-30475/com.rever.app E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.rever.app, PID: 30475
java.lang.RuntimeException: Unable to start activity ComponentInfo{com.rever.app/com.rever.app.UserProfileActivity}: retrofit.RetrofitError
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2814)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892)
at android.app.ActivityThread.-wrap11(Unknown Source:0)
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1613)
at android.os.Handler.dispatchMessage(Handler.java:106)
at android.os.Looper.loop(Looper.java:176)
at android.app.ActivityThread.main(ActivityThread.java:6635)
at java.lang.reflect.Method.invoke(Native Method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:547)
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:823)
Caused by: retrofit.RetrofitError
at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:400)
at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240)
at java.lang.reflect.Proxy.invoke(Proxy.java:913)
at $Proxy0.getUserInfo(Unknown Source)
at com.rever.app.UserProfileActivity.fetchUserProfile(UserProfileActivity.java:220)
at com.rever.app.UserProfileActivity.onCreate(UserProfileActivity.java:132)
at android.app.Activity.performCreate(Activity.java:7084)
at android.app.Activity.performCreate(Activity.java:7075)
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1215)
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2767)
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892) 
at android.app.ActivityThread.-wrap11(Unknown Source:0) 
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1613) 
at android.os.Handler.dispatchMessage(Handler.java:106) 
at android.os.Looper.loop(Looper.java:176) 
at android.app.ActivityThread.main(ActivityThread.java:6635) 
at java.lang.reflect.Method.invoke(Native Method) 
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:547) 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:823) 
Caused by: android.os.NetworkOnMainThreadException
at android.os.StrictMode$AndroidBlockGuardPolicy.onNetwork(StrictMode.java:1450)
at java.net.Inet6AddressImpl.lookupHostByName(Inet6AddressImpl.java:102)
at java.net.Inet6AddressImpl.lookupAllHostAddr(Inet6AddressImpl.java:90)
at java.net.InetAddress.getAllByName(InetAddress.java:787)
at com.squareup.okhttp.internal.Dns$1.getAllByName(Dns.java:29)
at com.squareup.okhttp.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:231)
at com.squareup.okhttp.internal.http.RouteSelector.next(RouteSelector.java:124)
at com.squareup.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:317)
at com.squareup.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:241)
at com.squareup.okhttp.Call.getResponse(Call.java:198)
at com.squareup.okhttp.Call.execute(Call.java:80)
at retrofit.client.OkClient.execute(OkClient.java:53)
at retrofit.RestAdapter$RestHandler.invokeRequest(RestAdapter.java:326)
at retrofit.RestAdapter$RestHandler.invoke(RestAdapter.java:240) 
at java.lang.reflect.Proxy.invoke(Proxy.java:913) 
at $Proxy0.getUserInfo(Unknown Source) 
at com.rever.app.UserProfileActivity.fetchUserProfile(UserProfileActivity.java:220) 
at com.rever.app.UserProfileActivity.onCreate(UserProfileActivity.java:132) 
at android.app.Activity.performCreate(Activity.java:7084) 
at android.app.Activity.performCreate(Activity.java:7075) 
at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1215) 
at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2767) 
at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2892) 
at android.app.ActivityThread.-wrap11(Unknown Source:0) 
at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1613) 
at android.os.Handler.dispatchMessage(Handler.java:106) 
at android.os.Looper.loop(Looper.java:176) 
at android.app.ActivityThread.main(ActivityThread.java:6635) 
at java.lang.reflect.Method.invoke(Native Method) 
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:547) 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:823) 
```

JavaFX Application : cannot find SunPKCS11 class

I am developing a JavaFX application used to sign pdf using eToken Pro. The sign methods run perfectly in a normal Java project. While when run in the JavaFX application, it keeps encountering exceptions like this:
Exception in Application start method
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.javafx.main.Main.launchApp(Main.java:698)
at com.javafx.main.Main.main(Main.java:871)
Caused by: java.lang.RuntimeException: Exception in Application start method
at com.sun.javafx.application.LauncherImpl.launchApplication1(LauncherImpl.java:403)
at com.sun.javafx.application.LauncherImpl.access$000(LauncherImpl.java:47)
at com.sun.javafx.application.LauncherImpl$1.run(LauncherImpl.java:115)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NoClassDefFoundError: sun/security/pkcs11/SunPKCS11
at javafxapplication3.JavaFXApplication3.start(JavaFXApplication3.java:21)
at com.sun.javafx.application.LauncherImpl$5.run(LauncherImpl.java:319)
at com.sun.javafx.application.PlatformImpl$5.run(PlatformImpl.java:216)
at com.sun.javafx.application.PlatformImpl$4$1.run(PlatformImpl.java:179)
at com.sun.javafx.application.PlatformImpl$4$1.run(PlatformImpl.java:176)
at java.security.AccessController.doPrivileged(Native Method)
at com.sun.javafx.application.PlatformImpl$4.run(PlatformImpl.java:176)
at com.sun.glass.ui.InvokeLaterDispatcher$Future.run(InvokeLaterDispatcher.java:76)
at com.sun.glass.ui.win.WinApplication._runLoop(Native Method)
at com.sun.glass.ui.win.WinApplication.access$100(WinApplication.java:17)
at com.sun.glass.ui.win.WinApplication$3$1.run(WinApplication.java:67)
... 1 more
Caused by: java.lang.ClassNotFoundException: sun.security.pkcs11.SunPKCS11
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 12 more
Java Result: 1
the codes I used to sign pdf are as follows:
#Override
public void start(Stage primaryStage) {
Signer signer = new Signer(new File("C:/Users/Adam/Desktop/pdf/hello.pdf"));
signer.signWithToken(true);
}
the codes of class Signer are as follows:
public class Signer {
// define the file to be signed
private final File file;
private static String smartcardDllPath;
private static int level;
private static String reason;
private static String src;
private static String dest;
private static String location;
private static Collection<CrlClient> crlList;
private static OcspClient ocspClient;
private static TSAClient tsaClient;
private static final String DLL = "C:/Windows/System32/eTPKCS11.dll";
public Signer(File theFile) {
location = "HK SAR";
smartcardDllPath = null;
file = theFile;
}
public void signWithToken(boolean certified) {
try {
String config = "name=eToken\nlibrary=" + DLL + "\nslotListIndex=" + getSlotsWithTokens(DLL)[0];
ByteArrayInputStream bais = new ByteArrayInputStream(config.getBytes());
Provider providerPKCS11 = new SunPKCS11(bais);
Security.addProvider(providerPKCS11);
configureParameters(certified);
// create PdfSignatureAppearance
PdfSignatureAppearance appearance = getPdfSigAppearance();
// configure the keystore, alias, private key and certificate chain
char[] pin = "love4Sakura".toCharArray();
KeyStore ks = KeyStore.getInstance("PKCS11");
ks.load(null, pin);
String alias = (String) ks.aliases().nextElement();
PrivateKey pk = (PrivateKey) ks.getKey(alias, null);
Certificate[] chain = ks.getCertificateChain(alias);
printChainInfo(chain);
// configure the CRL, OCSP and TSA
configCrlOcspTsa(chain);
// create the signature
ExternalSignature pks = new PrivateKeySignature(pk, DigestAlgorithms.SHA256, "SunPKCS11-eToken");
ExternalDigest digest = new BouncyCastleDigest();
MakeSignature.signDetached(appearance, digest, pks, chain, crlList, ocspClient,
tsaClient, 0, MakeSignature.CryptoStandard.CMS);
} catch (IOException | DocumentException | GeneralSecurityException ex) {
Logger.getLogger(Signer.class.getName()).log(Level.SEVERE, null, ex);
}
}
private static long[] getSlotsWithTokens(String libraryPath) {
CK_C_INITIALIZE_ARGS initArgs = new CK_C_INITIALIZE_ARGS();
String functionList = "C_GetFunctionList";
initArgs.flags = 0;
PKCS11 tmpPKCS11 = null;
long[] slotList = null;
try {
try {
tmpPKCS11 = PKCS11.getInstance(libraryPath, functionList, initArgs, false);
System.out.println(tmpPKCS11.toString());
} catch (IOException ex) {
try {
throw ex;
} catch (IOException ex1) {
Logger.getLogger(Signer.class.getName()).log(Level.SEVERE, null, ex1);
}
}
} catch (PKCS11Exception e) {
try {
initArgs = null;
tmpPKCS11 = PKCS11.getInstance(libraryPath, functionList, initArgs, true);
} catch (IOException | PKCS11Exception ex) {
}
}
try {
slotList = tmpPKCS11.C_GetSlotList(true);
for (long slot : slotList) {
CK_TOKEN_INFO tokenInfo = tmpPKCS11.C_GetTokenInfo(slot);
System.out.println("slot: " + slot + "\nmanufacturerID: "
+ String.valueOf(tokenInfo.manufacturerID) + "\nmodel: "
+ String.valueOf(tokenInfo.model));
}
} catch (PKCS11Exception ex) {
Logger.getLogger(Signer.class.getName()).log(Level.SEVERE, null, ex);
}
return slotList;
}
Before you guys answer: I want to emphasize that I reference the Signer class in a normal Java Project perfectly. So I donn't think it's a 32-bit or 64-bit problem. Plus, I am using the 32-bit JDK 1.7
This issue can be because unsupported version of java.Do one thing download particular jar file for that and include it on your project it'll work for you.
You can download the jar from this link -
http://www.docjar.com/jar/sunpkcs11.jar
I was also getting the same error -
java.lang.ClassNotFoundException: sun.security.pkcs11.SunPKCS11
Adding the following dependency in pom.xml helped me -
<dependency>
<groupId>sunpkcs11</groupId>
<artifactId>sunpkcs11</artifactId>
<scope>system</scope>
<version>1.0</version>
<systemPath>/Library/Java/JavaVirtualMachines/jdk1.8.0_341.jdk/Contents/Home/jre/lib/ext/sunpkcs11.jar</systemPath>
</dependency>
In <systemPath>...</systemPath>, give the appropriate path of sunpkcs11.jar from your system.

Hadoop: the Mapper didn't read files from multiple input paths

The Mapper didn't manage to read a file from multiple directories. Could anyone help?
I need to read one file in each mapper. I've added multiple input paths and implemented the custom WholeFileInputFormat, WholeFileRecordReader. In the map method, I don't need the input key. I make sure that each map can read a whole file.
Command line: hadoop jar AutoProduce.jar Autoproduce /input_a /input_b /output
I specified two input path----1.input_a; 2.input_b;
Run method snippets:
Job job = new Job(getConf());
job.setInputFormatClass(WholeFileInputFormat.class);
FileInputFormat.setInputPaths(job, new Path(args[0]), new Path(args[1]));
FileOutputFormat.setOutputPath(job, new Path(args[2]));
map method snippets:
public void map(NullWritable key, BytesWritable value, Context context){
FileSplit fileSplit = (FileSplit) context.getInputSplit();
System.out.println("Directory :" + fileSplit.getPath().toString());
......
}
Custom WholeFileInputFormat:
class WholeFileInputFormat extends FileInputFormat<NullWritable, BytesWritable> {
#Override
protected boolean isSplitable(JobContext context, Path file) {
return false;
}
#Override
public RecordReader<NullWritable, BytesWritable> createRecordReader(
InputSplit split, TaskAttemptContext context) throws IOException,
InterruptedException {
WholeFileRecordReader reader = new WholeFileRecordReader();
reader.initialize(split, context);
return reader;
}
}
Custom WholeFileRecordReader:
class WholeFileRecordReader extends RecordReader<NullWritable, BytesWritable> {
private FileSplit fileSplit;
private Configuration conf;
private BytesWritable value = new BytesWritable();
private boolean processed = false;
#Override
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {
this.fileSplit = (FileSplit) split;
this.conf = context.getConfiguration();
}
#Override
public boolean nextKeyValue() throws IOException, InterruptedException {
if (!processed) {
byte[] contents = new byte[(int) fileSplit.getLength()];
Path file = fileSplit.getPath();
FileSystem fs = file.getFileSystem(conf);
FSDataInputStream in = null;
try {
in = fs.open(file);
IOUtils.readFully(in, contents, 0, contents.length);
value.set(contents, 0, contents.length);
} finally {
IOUtils.closeStream(in);
}
processed = true;
return true;
}
return false;
}
#Override
public NullWritable getCurrentKey() throws IOException,InterruptedException {
return NullWritable.get();
}
#Override
public BytesWritable getCurrentValue() throws IOException,InterruptedException {
return value;
}
#Override
public float getProgress() throws IOException {
return processed ? 1.0f : 0.0f;
}
#Override
public void close() throws IOException {
// do nothing
}
}
PROBLEM:
After setting two input paths, all map tasks read files from only one directory..
Thanks in advance.
You'll have to use MultipleInputs instead of FileInputFormat in the driver. So your code should be as:
MultipleInputs.addInputPath(job, new Path(args[0]), <Input_Format_Class_1>);
MultipleInputs.addInputPath(job, new Path(args[1]), <Input_Format_Class_2>);
.
.
.
MultipleInputs.addInputPath(job, new Path(args[N-1]), <Input_Format_Class_N>);
So if you want to use WholeFileInputFormat for the first input path and TextInputFormat for the second input path, you'll have to use it the following way:
MultipleInputs.addInputPath(job, new Path(args[0]), WholeFileInputFormat.class);
MultipleInputs.addInputPath(job, new Path(args[1]), TextInputFormat.class);
Hope this works for you!

Resources