My SSCE is:
public static void main(final String[] args) throws IOException {
try {
SwingUtilities.invokeAndWait(new Runnable() {
#Override
public void run() {
int bitdepth = 10;
GLProfile.initSingleton();
GLProfile glProfile = GLProfile.getDefault();
GLCapabilities glCapabilities = new GLCapabilities( glProfile );
glCapabilities.setBlueBits(bitdepth);
glCapabilities.setGreenBits(bitdepth);
glCapabilities.setRedBits(bitdepth);
glCapabilities.setAlphaBits(2);
glCapabilities.setDoubleBuffered(true);
glCapabilities.setHardwareAccelerated(true);
glCapabilities.setNumSamples(4);
glCapabilities.setBackgroundOpaque(false);
glCapabilities.setSampleBuffers(true);
GraphicsConfiguration gc = getSomeGC();
JFrame jf = new JFrame(gc);
jf.setExtendedState(JFrame.MAXIMIZED_BOTH);
GLCanvas canvas = new GLCanvas(glCapabilities);
canvas.addGLEventListener(new GLEventListener() {
#Override
public void reshape(GLAutoDrawable arg0, int arg1, int arg2, int arg3,
int arg4) {
// TODO Auto-generated method stub
}
#Override
public void init(GLAutoDrawable arg0) {
// TODO Auto-generated method stub
}
#Override
public void dispose(GLAutoDrawable arg0) {
// TODO Auto-generated method stub
}
#Override
public void display(GLAutoDrawable drawable) {
System.out.println("Painting");
BufferedImage image = null;
try {
image = ImageIO.read(new File("img.tiff"));
} catch (IOException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
if(image!=null){
GL2 gl2 = drawable.getGL().getGL2();
//gl2.glClear(GL.GL_COLOR_BUFFER_BIT);
int format = GL.GL_LUMINANCE;
int type = GL.GL_UNSIGNED_SHORT;
DataBufferUShort db = (DataBufferUShort) image.getRaster().getDataBuffer();
short[] shorts = db.getData(0);
Buffer buffer = ShortBuffer.wrap(shorts);
//gl2.glViewport(0, 0, image.getWidth(), image.getHeight());
gl2.glDrawPixels(image.getWidth(), image.getHeight(), format , type, buffer );
}
}
});
JPanel jp = new JPanel();
jp.setLayout(new GridBagLayout());
GridBagConstraints gbc = new GridBagConstraints();
gbc.fill = GridBagConstraints.BOTH;
gbc.gridx=0;
gbc.gridy=0;
gbc.gridwidth=1;
gbc.gridheight=1;
gbc.weightx=1;
gbc.weighty=1;
gbc.anchor= GridBagConstraints.CENTER;
jp.add(canvas,gbc);
JScrollPane jsp = new JScrollPane();
jsp.getViewport().add(jp);
JLayeredPane jlp = new JLayeredPane();
jlp.setLayout(new GridBagLayout());
jlp.add(jsp, gbc);
//jsp.getViewport().add(dsc);
gbc = new GridBagConstraints();
gbc.gridx=0;
gbc.gridy=0;
gbc.gridwidth=1;
gbc.gridheight=1;
gbc.weightx=1;
gbc.weighty=1;
gbc.fill=GridBagConstraints.BOTH;
gbc.anchor= GridBagConstraints.CENTER;
jf.getContentPane().setLayout(new GridBagLayout());
jf.getContentPane().add(jlp,gbc);
jf.setVisible(true);
}
});
} catch (InvocationTargetException e) {
// TODO Auto-generated catch block
e.printStackTrace();
} catch (InterruptedException e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
I get two different displays for GLCanvas over GLJPanel. I trying to get 10 bit display on a monitor capable of 10bit grayscale.
The GLJPanel is in 8 bit while the GLCanvas throws an exception but is I believe displayed correctly at 10 bit.
Exception in thread "AWT-EventQueue-0" javax.media.opengl.GLException: Error making context 0x20000 current on Thread AWT-EventQueue-0, drawableWrite 0x42011b9a, drawableRead 0x42011b9a, werr: 0, WindowsWGLContext [Version 4.2 (Compat profile, arb, ES2 compat, ES3 compat, FBO, hardware) - 4.2.12327 Compatibility Profile Context FireGL 12.104.2.3000 [GL 4.2.0, vendor 12.104.2 (Compatibility Profile Context FireGL 12.104.2.3000)], options 0x1c03, this 0x263ce8f2, handle 0x20000, isShared false, jogamp.opengl.gl4.GL4bcImpl#633cca0,
quirks: [NoDoubleBufferedBitmap],
Drawable: WindowsOnscreenWGLDrawable[Realized true,
Factory jogamp.opengl.windows.wgl.WindowsWGLDrawableFactory#5dba26,
Handle 0x42011b9a,
Since this is a jogl bugreport please post the complete compilable SSCE with all includes into a bugreport at https://jogamp.org/bugzilla/
don't forget to attach the source code and the img.tiff
You are technically reporting two bugs thus you should file two bugs:
bug one:
We need to know more about your use-case and what kind of differences you see in the two images when using a GLCanvas and GLJPanel.
bug two:
We need a reproducer for the GLException: Error making context 0x20000 current on Thread
For more details read the JOGL Bugreports & Testing FAQ:
http://jogamp.org/wiki/index.php/Jogl_FAQ#Bugreports_.26_Testing
Related
I have the following POST servlet that adds new node under certain resource with parameters(name and last nam) from the request:
#Component(
service = Servlet.class,
property = {
"sling.servlet.paths=/bin/createuser",
"sling.servlet.methods=" + HttpConstants.METHOD_POST
})
public class CreateNodeServlet extends SlingAllMethodsServlet {
/**
* Logger
*/
private static final Logger log = LoggerFactory.getLogger(CreateNodeServlet.class);
#Override
protected void doPost(final SlingHttpServletRequest req, final SlingHttpServletResponse resp) throws IOException {
log.info("Inside CreateNodeServlet");
ResourceResolver resourceResolver = req.getResourceResolver();
final Resource resource = resourceResolver.getResource("/content/test/us/en");
String name = req.getParameter("name");
String lastname = req.getParameter("lastname");
log.info("name :{}",name);
log.info("lastname :{}",lastname);
Node node = resource.adaptTo(Node.class);
try {
log.info("Node {}", node.getName() );
Node newNode = node.addNode(name+lastname, "nt:unstructured");
newNode.setProperty("name", name);
newNode.setProperty("lastname", lastname);
resourceResolver.commit();
} catch (RepositoryException e) {
e.printStackTrace();
} catch (PersistenceException e) {
e.printStackTrace();
}
resp.setStatus(200);
resp.getWriter().write("Simple Post Test");
}
}
I tried creating unit test for this I got this so far:
#ExtendWith(AemContextExtension.class)
class CreateNodeServletTest {
private final AemContext context = new AemContext();
private CreateNodeServlet createNodeServlet = new CreateNodeServlet();
#Test
void doPost() throws IOException, JSONException {
context.currentPage(context.pageManager().getPage("/bin/createuser"));
context.currentResource(context.resourceResolver().getResource("/bin/createuser"));
context.requestPathInfo().setResourcePath("/bin/createuser");
MockSlingHttpServletRequest request = context.request();
MockSlingHttpServletResponse response = context.response();
createNodeServlet.doPost(request, response);
JSONArray output = new JSONArray(context.response().getOutputAsString());
assertEquals("Simple Post Test", output);
}
}
however this is not working I am getting null pointer on this line
Node node = resource.adaptTo(Node.class);
can some one help what I am missing and some tips will be of great help as I am new to AEM, and there is not much resources about unit testing sling servlets ?
I think you need to register JCR_MOCK as resource resolver type
new AemContext(ResourceResolverType.JCR_MOCK);
I am a newbie to Windows Application Driver and my project demands automating the desktop application, so I decided to use winappdriver as it is similar to selenium, on which I am pretty confident about using.
speaking of the issue,
Just wondering if there is a way to achieve explicit wait and implicit wait using winappdriver. Following is the code i used as part of my test cases, the test fails with an exception (NoSuchElementException), however, if I put a static wait in place instead of explicit wait, it works as expected.
//Driver Setup
public class OscBase {
public static WindowsDriver<WebElement> applicaitonSession, driver = null;
public static WindowsDriver<RemoteWebElement> desktopSession = null;
public static DesiredCapabilities capabilities, cap1, cap2;
public static ProcessBuilder pBuilder;
public static Process p;
public void startDriver() {
try {
pBuilder = new ProcessBuilder("C:\\Program Files (x86)\\Windows Application Driver\\WinAppDriver.exe");
pBuilder.inheritIO();
p = pBuilder.start();
}
catch (IOException e) {
e.printStackTrace();
}
}
public void stopDriver() {
p.destroy();
}
public void createDesktopSession() throws MalformedURLException {
cap1 = new DesiredCapabilities();
cap1.setCapability("app", "Root");
desktopSession = new WindowsDriver<RemoteWebElement>(new URL("http://localhost:4723"), cap1);
}
public void openApplication() throws InterruptedException, MalformedURLException {
if (driver == null) {
try {
capabilities = new DesiredCapabilities();
capabilities.setCapability("app",
"Appnamewithlocation");
applicaitonSession = new WindowsDriver<WebElement>(new URL("http://localhost:4723"),
capabilities);
} catch (Exception e) {
System.out.println("Application opened!!!");
} finally {
createDesktopSession();
}
Thread.sleep(8000L);
String handle = desktopSession.findElementByAccessibilityId("InstallerView5")
.getAttribute("NativeWindowHandle");
System.out.println(handle);
int inthandle = Integer.parseInt(handle);
String hexHandle = Integer.toHexString(inthandle);
//System.out.println(hexHandle);
cap2 = new DesiredCapabilities();
cap2.setCapability("appTopLevelWindow", hexHandle);
driver = new WindowsDriver<WebElement>(new URL("http://localhost:4723"), cap2);
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);
}
}
public boolean isDisplayed_SafeLoginNoBtn() {
wait = new WebDriverWait(driver, 40);
return wait.until(ExpectedConditions.visibilityOf(safeLoginNoBtn())).isDisplayed();
}
#Test
public void osc_Get_Data() throws InterruptedException, IOException {
//Thread.sleep(20000);
// Boolean value=oscLogin.safeLoginNoBtn().isDisplayed();
try {
Boolean value = oscLogin.isDisplayed_SafeLoginNoBtn();
System.out.println("IS displayed========>"+value);
if (value) {
oscLogin.click_safeLogin();
}
} catch (Exception e) {
System.out.println("Safe Login!!!!");
}
Of course yes, the WebDriverWait class will work. Here's an example
WebDriverWait waitForMe = new WebDriverWait();
WebDriverWait waitForMe = new WebDriverWait(session, new TimeSpan.Fromseconds(10));
var txtLocation = session.FindElementByName("Enter a location");
waitForMe.Until(pred => txtLocation.Displayed);
I've created a detailed course about UI Automation using WinAppDriver and C# .Net. I'll be publishing it in a few days. Do let me know if you're interested :)
I am new to netty and I am trying to design a solution as below for transfer of file from Server to Client over TCP:
1. Zero copy based file transfer in case of non-ssl based transfer (Using default region of the file)
2. ChunkedFile transfer in case of SSL based transfer.
The Client - Server file transfer works in this way:
1. The client sends the location of the file to be transfered
2. Based on the location (sent by the client) the server transfers the file to the client
The file content could be anything (String /image /pdf etc) and any size.
Now, I get this TooLongFrameException: at the Server side, though the server is just decoding the path received from the client, for running the code mentioned below (Server/Client).
io.netty.handler.codec.TooLongFrameException: Adjusted frame length exceeds 65536: 215542494061 - discarded
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.fail(LengthFieldBasedFrameDecoder.java:522)
at io.netty.handler.codec.LengthFieldBasedFrameDecoder.failIfNecessary(LengthFieldBasedFrameDecoder.java:500)
Now, My question is:
Am I wrong with the order of Encoders and Decoders and its configuration? If so, what is the correct way to configure it to receive a file from the server?
I went through few related StackOverflow posts SO Q1,SO Q2 , SO Q3, SO Q4. I got to know about the LengthFieldBasedDecoder, but I didn't get to know how to configure its corresponding LengthFieldPrepender at the Server (Encoding side). Is it even required at all?
Please point me into the right direction.
FileClient:
public final class FileClient {
static final boolean SSL = System.getProperty("ssl") != null;
static final int PORT = Integer.parseInt(System.getProperty("port", SSL ? "8992" : "8023"));
static final String HOST = System.getProperty("host", "127.0.0.1");
public static void main(String[] args) throws Exception {
// Configure SSL.
final SslContext sslCtx;
if (SSL) {
SelfSignedCertificate ssc = new SelfSignedCertificate();
sslCtx = SslContextBuilder.forServer(ssc.certificate(), ssc.privateKey()).build();
} else {
sslCtx = null;
}
// Configure the client
EventLoopGroup group = new NioEventLoopGroup();
try {
Bootstrap b = new Bootstrap();
b.group(group)
.channel(NioSocketChannel.class)
.option(ChannelOption.SO_KEEPALIVE, true)
.handler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
if (sslCtx != null) {
pipeline.addLast(sslCtx.newHandler(ch.alloc(), HOST, PORT));
}
pipeline.addLast("frameDecoder", new LengthFieldBasedFrameDecoder(64*1024, 0, 8));
pipeline.addLast("frameEncoder", new LengthFieldPrepender(4));
pipeline.addLast(new ObjectDecoder(ClassResolvers.cacheDisabled(null)));
pipeline.addLast(new ObjectEncoder());
pipeline.addLast( new FileClientHandler()); }
});
// Start the server.
ChannelFuture f = b.connect(HOST,PORT).sync();
// Wait until the server socket is closed.
f.channel().closeFuture().sync();
} finally {
// Shut down all event loops to terminate all threads.
group.shutdownGracefully();
}
}
}
FileClientHandler:
public class FileClientHandler extends ChannelInboundHandlerAdapter{
#Override
public void channelActive(ChannelHandlerContext ctx) {
String filePath = "/Users/Home/Documents/Data.pdf";
ctx.writeAndFlush(Unpooled.wrappedBuffer(filePath.getBytes()));
}
#Override
public void channelRead(ChannelHandlerContext ctx, Object msg) throws Exception {
System.out.println("File Client Handler Read method...");
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
}
}
FileServer:
/**
* Server that accept the path of a file and echo back its content.
*/
public final class FileServer {
static final boolean SSL = System.getProperty("ssl") != null;
static final int PORT = Integer.parseInt(System.getProperty("port", SSL ? "8992" : "8023"));
public static void main(String[] args) throws Exception {
// Configure SSL.
final SslContext sslCtx;
if (SSL) {
SelfSignedCertificate ssc = new SelfSignedCertificate();
sslCtx = SslContextBuilder.forServer(ssc.certificate(), ssc.privateKey()).build();
} else {
sslCtx = null;
}
// Configure the server.
EventLoopGroup bossGroup = new NioEventLoopGroup(1);
EventLoopGroup workerGroup = new NioEventLoopGroup();
try {
ServerBootstrap b = new ServerBootstrap();
b.group(bossGroup, workerGroup).channel(NioServerSocketChannel.class)
.option(ChannelOption.SO_KEEPALIVE, true).handler(new LoggingHandler(LogLevel.INFO))
.childHandler(new ChannelInitializer<SocketChannel>() {
#Override
public void initChannel(SocketChannel ch) throws Exception {
ChannelPipeline pipeline = ch.pipeline();
if (sslCtx != null) {
pipeline.addLast(sslCtx.newHandler(ch.alloc()));
}
pipeline.addLast("frameDecoder",new LengthFieldBasedFrameDecoder(64*1024, 0, 8));
pipeline.addLast("frameEncoder", new LengthFieldPrepender(4));
pipeline.addLast(new ObjectDecoder(ClassResolvers.cacheDisabled(null)));
pipeline.addLast(new ObjectEncoder());
pipeline.addLast(new ChunkedWriteHandler());
pipeline.addLast(new FileServerHandler());
}
});
// Start the server.
ChannelFuture f = b.bind(PORT).sync();
// Wait until the server socket is closed.
f.channel().closeFuture().sync();
} finally {
bossGroup.shutdownGracefully();
workerGroup.shutdownGracefully();
}
}
}
FileServerHandler:
public class FileServerHandler extends ChannelInboundHandlerAdapter {
#Override
public void channelRead(ChannelHandlerContext ctx, Object obj) throws Exception {
RandomAccessFile raf = null;
long length = -1;
try {
ByteBuf buff = (ByteBuf)obj;
byte[] bytes = new byte[buff.readableBytes()];
buff.readBytes(bytes);
String msg = new String(bytes);
raf = new RandomAccessFile(msg, "r");
length = raf.length();
} catch (Exception e) {
ctx.writeAndFlush("ERR: " + e.getClass().getSimpleName() + ": " + e.getMessage() + '\n');
return;
} finally {
if (length < 0 && raf != null) {
raf.close();
}
}
if (ctx.pipeline().get(SslHandler.class) == null) {
// SSL not enabled - can use zero-copy file transfer.
ctx.writeAndFlush(new DefaultFileRegion(raf.getChannel(), 0, length));
} else {
// SSL enabled - cannot use zero-copy file transfer.
ctx.writeAndFlush(new ChunkedFile(raf));
}
}
#Override
public void exceptionCaught(ChannelHandlerContext ctx, Throwable cause) {
cause.printStackTrace();
System.out.println("Exception server.....");
}
}
I referred Netty In Action and code samples from here
There are multiple things wrong with your server/client. First thing the SSL, for the client you don't need to initialize a SslContext for a server instead you would do something like this:
sslCtx = SslContextBuilder.forClient().trustManager(InsecureTrustManagerFactory.INSTANCE).build();
On the server side of things you use a SelfSignedCertificate which in itself isn't wrong but would like to remind you that it should only be used for debugging purposes and not in production. In addition you use the ChannelOption.SO_KEEPALIVE which isn't recommended since the keepalive interval is OS-dependent. Furthermore you added Object En-/Decoder to your pipeline which in your case don't do anything useful so you can remove them.
Also you configured your LengthFieldBasedFrameDecoder wrong due to an incomplete and wrong parameter list. In the netty docs you need the version of the constructor which defines the lengthFieldLength and initialBytesToStrip. Besides the not stripping the length field you also defined the wrong lengthFieldLength which should be the same as your LengthFieldPrepender's lengthFieldLength which is 4 bytes. In conlusion you could use the constructor like this:
new LengthFieldBasedFrameDecoder(64 * 1024, 0, 4, 0, 4)
In both your handler you don't specify a Charset when en-/decoding your String which could lead to problems because if no ´Charset´ is defined the systems default will be used which could vary. You could do something like this:
//to encode the String
string.getBytes(StandardCharsets.UTF_8);
//to decode the String
new String(bytes, StandardCharsets.UTF_8);
Additionally you tried to use the DefaultFileRegion if no SslHandler was added to the pipeline which would have been fine if you didn't added the LengthFieldHandler since they would need a memory copy of the byte[] to send to added the length field. Moreover I would recommend using the ChunkedNioFile instead of the ChunkedFile because it's nonblocking which is always a good thing. You would do this like that:
new ChunkedNioFile(randomAccessFile.getChannel())
One final thing on how to decode a ChunkedFile as it's split in chunks you can simply assamble them tougether with a simple OutputStream. Here's an old file handler of mine:
public class FileTransferHandler extends SimpleChannelInboundHandler<ByteBuf> {
private final Path path;
private final int size;
private final int hash;
private OutputStream outputStream;
private int writtenBytes = 0;
private byte[] buffer = new byte[0];
protected FileTransferHandler(Path path, int size, int hash) {
this.path = path;
this.size = size;
this.hash = hash;
}
#Override
protected void channelRead0(ChannelHandlerContext ctx, ByteBuf byteBuf) throws Exception {
if(this.outputStream == null) {
Files.createDirectories(this.path.getParent());
if(Files.exists(this.path))
Files.delete(this.path);
this.outputStream = Files.newOutputStream(this.path, StandardOpenOption.CREATE, StandardOpenOption.APPEND);
}
int size = byteBuf.readableBytes();
if(size > this.buffer.length)
this.buffer = new byte[size];
byteBuf.readBytes(this.buffer, 0, size);
this.outputStream.write(this.buffer, 0, size);
this.writtenBytes += size;
if(this.writtenBytes == this.size && MurMur3.hash(this.path) != this.hash) {
System.err.println("Received file has wrong hash");
return;
}
}
#Override
public void channelInactive(ChannelHandlerContext ctx) throws Exception {
if(this.outputStream != null)
this.outputStream.close();
}
}
So I'm trying to make a billing system in which I want to print a receipt.I was able to do it with some code that I found online,but the font size is too big to print in the 58mm wide paper.I'm not able to adjust the font size.Any kind of help with this issue will be highly appreciated.Thank You.
Here is The Code :
public class PrinterService implements Printable {
public List<String> getPrinters(){
DocFlavor flavor = DocFlavor.BYTE_ARRAY.AUTOSENSE;
PrintRequestAttributeSet pras = new HashPrintRequestAttributeSet();
PrintService printServices[] = PrintServiceLookup.lookupPrintServices(
flavor, pras);
List<String> printerList = new ArrayList<String>();
for(PrintService printerService: printServices){
printerList.add( printerService.getName());
}
return printerList;
}
#Override
public int print(Graphics g, PageFormat pf, int page)
throws PrinterException {
if (page > 0) { /* We have only one page, and 'page' is zero-based */
return NO_SUCH_PAGE;
}
/*
* User (0,0) is typically outside the imageable area, so we must
* translate by the X and Y values in the PageFormat to avoid clipping
*/
Graphics2D g2d = (Graphics2D) g;
g2d.translate(pf.getImageableX(), pf.getImageableY());
/* Now we perform our rendering */
g.setFont(new Font("Roman", 0, 8));
g.drawString("Hello world !", 0, 10);
return PAGE_EXISTS;
}
public void printString(String printerName, String text) {
// find the printService of name printerName
DocFlavor flavor = DocFlavor.BYTE_ARRAY.AUTOSENSE;
PrintRequestAttributeSet pras = new HashPrintRequestAttributeSet();
PrintService printService[] = PrintServiceLookup.lookupPrintServices(
flavor, pras);
PrintService service = findPrintService(printerName, printService);
DocPrintJob job = service.createPrintJob();
try {
byte[] bytes;
// important for umlaut chars
bytes = text.getBytes("CP437");
Doc doc = new SimpleDoc(bytes, flavor, null);
job.print(doc, null);
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
public void printBytes(String printerName, byte[] bytes) {
DocFlavor flavor = DocFlavor.BYTE_ARRAY.AUTOSENSE;
PrintRequestAttributeSet pras = new HashPrintRequestAttributeSet();
PrintService printService[] = PrintServiceLookup.lookupPrintServices(
flavor, pras);
PrintService service = findPrintService(printerName, printService);
DocPrintJob job = service.createPrintJob();
try {
Doc doc = new SimpleDoc(bytes, flavor, null);
job.print(doc, null);
} catch (Exception e) {
e.printStackTrace();
}
}
private PrintService findPrintService(String printerName,
PrintService[] services) {
for (PrintService service : services) {
if (service.getName().equalsIgnoreCase(printerName)) {
return service;
}
}
return null;
}
}
#FXML
public void printit(ActionEvent actionEvent)
{
PrinterService printerService = new PrinterService();
System.out.println(printerService.getPrinters());
//print some stuff
printerService.printString("POS-58-Series", area.getText());
}
The Mapper didn't manage to read a file from multiple directories. Could anyone help?
I need to read one file in each mapper. I've added multiple input paths and implemented the custom WholeFileInputFormat, WholeFileRecordReader. In the map method, I don't need the input key. I make sure that each map can read a whole file.
Command line: hadoop jar AutoProduce.jar Autoproduce /input_a /input_b /output
I specified two input path----1.input_a; 2.input_b;
Run method snippets:
Job job = new Job(getConf());
job.setInputFormatClass(WholeFileInputFormat.class);
FileInputFormat.setInputPaths(job, new Path(args[0]), new Path(args[1]));
FileOutputFormat.setOutputPath(job, new Path(args[2]));
map method snippets:
public void map(NullWritable key, BytesWritable value, Context context){
FileSplit fileSplit = (FileSplit) context.getInputSplit();
System.out.println("Directory :" + fileSplit.getPath().toString());
......
}
Custom WholeFileInputFormat:
class WholeFileInputFormat extends FileInputFormat<NullWritable, BytesWritable> {
#Override
protected boolean isSplitable(JobContext context, Path file) {
return false;
}
#Override
public RecordReader<NullWritable, BytesWritable> createRecordReader(
InputSplit split, TaskAttemptContext context) throws IOException,
InterruptedException {
WholeFileRecordReader reader = new WholeFileRecordReader();
reader.initialize(split, context);
return reader;
}
}
Custom WholeFileRecordReader:
class WholeFileRecordReader extends RecordReader<NullWritable, BytesWritable> {
private FileSplit fileSplit;
private Configuration conf;
private BytesWritable value = new BytesWritable();
private boolean processed = false;
#Override
public void initialize(InputSplit split, TaskAttemptContext context)
throws IOException, InterruptedException {
this.fileSplit = (FileSplit) split;
this.conf = context.getConfiguration();
}
#Override
public boolean nextKeyValue() throws IOException, InterruptedException {
if (!processed) {
byte[] contents = new byte[(int) fileSplit.getLength()];
Path file = fileSplit.getPath();
FileSystem fs = file.getFileSystem(conf);
FSDataInputStream in = null;
try {
in = fs.open(file);
IOUtils.readFully(in, contents, 0, contents.length);
value.set(contents, 0, contents.length);
} finally {
IOUtils.closeStream(in);
}
processed = true;
return true;
}
return false;
}
#Override
public NullWritable getCurrentKey() throws IOException,InterruptedException {
return NullWritable.get();
}
#Override
public BytesWritable getCurrentValue() throws IOException,InterruptedException {
return value;
}
#Override
public float getProgress() throws IOException {
return processed ? 1.0f : 0.0f;
}
#Override
public void close() throws IOException {
// do nothing
}
}
PROBLEM:
After setting two input paths, all map tasks read files from only one directory..
Thanks in advance.
You'll have to use MultipleInputs instead of FileInputFormat in the driver. So your code should be as:
MultipleInputs.addInputPath(job, new Path(args[0]), <Input_Format_Class_1>);
MultipleInputs.addInputPath(job, new Path(args[1]), <Input_Format_Class_2>);
.
.
.
MultipleInputs.addInputPath(job, new Path(args[N-1]), <Input_Format_Class_N>);
So if you want to use WholeFileInputFormat for the first input path and TextInputFormat for the second input path, you'll have to use it the following way:
MultipleInputs.addInputPath(job, new Path(args[0]), WholeFileInputFormat.class);
MultipleInputs.addInputPath(job, new Path(args[1]), TextInputFormat.class);
Hope this works for you!