posix_fallocate() failed: Operation not permitted while opening .realm file - realm

I get the below error when i try to open and download .realm file in /tmp directory of serverless framework.
{"errorType":"Runtime.UnhandledPromiseRejection","errorMessage":"Error: posix_fallocate() failed: Operation not permitted" }
Below is the code:
let realm = new Realm({path: '/tmp/custom.realm', schema: [schema1, schema2]});
realm.write(() => {
console.log('completed==');
});

EDIT: this might soon be finally fixed in Realm-Core: see issue 4957.
In case you'll run into this problem elsewhere, here's a workaround.
This caused by AWS Lambda not supporting the fallocate and fallocate64 system calls. Instead of returning the correct error code in this case, which would be EINVAL for not supported on this file system, Amazon has blocked the system call so that it returns EPERM. Realm-Core has code that handles EINVAL return value correctly but will be bewildered by the unexpected EPERM returned from the system call.
The solution is to add a small shared library as a layer to the lambda: compile the following C file on Linux machine or inside lambda-ci Docker image:
#include <errno.h>
#include <fcntl.h>
int posix_fallocate(int __fd, off_t __offset, off_t __len) {
return EINVAL;
}
int posix_fallocate64(int __fd, off_t __offset, off_t __len) {
return EINVAL;
}
Now, compile this to a shared object with something like
gcc -shared fix.c -o fix.so
Then add it to a root of a ZIP file:
zip layer.zip fix.so
Create a new lambda layer from this zip
Add the lambda layer to your lambda function
Finally make the shared object be loaded by configuring the environment value LD_PRELOAD with value /opt/fix.so to your Lambda.
Enjoy.

Related

Accessing an external variable from a C library

I am currently learning C and am trying to understand the possibilities of dynamic libraries.
My current question is, if I have a simple "Hello World" application in C called "ProgA", and this program dynamically loads a shared library with some example code called "LibB", can LibB access a global variable in ProgA, which was declared as external?
Given is the following example code for demonstration of the problem:
file header.h
#ifndef TEST_H
#define TEST_H
typedef struct test_import_s {
int some_field;
} test_import_t;
extern test_import_t newtestimport;
#endif
file prog_a.c
#include <stdio.h>
#include <windows.h>
#include "header.h"
test_import_t newtestimport = {
.some_field = 42
};
int main()
{
HINSTANCE hinstLib;
typedef void (*FunctionPointer)();
newtestimport.some_field = 42;
hinstLib = LoadLibrary("lib_b.dll");
if (hinstLib != NULL)
{
FunctionPointer initialize_lib_b;
initialize_lib_b = (FunctionPointer)GetProcAddress(hinstLib, "initialize_lib_b");
if (initialize_lib_b != NULL)
{
initialize_lib_b();
}
FreeLibrary(hinstLib);
}
return 0;
}
file lib_b.c
#include <stdio.h>
#include "header.h"
test_import_t *timp;
void initialize_lib_b() {
timp = &newtestimport;
int some_field = timp->some_field;
printf("Result from function: %d\n", some_field);
}
file CMakeLists.txt
cmake_minimum_required(VERSION 3.24)
project(dynamic-library-2 C)
set(CMAKE_C_STANDARD 23)
add_library(lib_b SHARED lib_b.c)
set_target_properties(lib_b PROPERTIES PREFIX "" OUTPUT_NAME "lib_b")
add_executable(prog_a prog_a.c)
target_link_libraries(prog_a lib_b)
In the above example, the headerfile header.h defines the struct test_import_t and an external variable newtestimport using this struct. In the C file of the main program prog_a.c one property of this struct is assigned the value 42. It then dynamically loads the library lib_b.c using the Windows API and executes a function in it. The function then should access the variable newtestimport of the main program and print out the value of the variable (42).
This example does not work. The compiler throws the following error:
====================[ Build | prog_a | Debug ]==================================
C:\Users\user1\AppData\Local\JetBrains\Toolbox\apps\CLion\ch-0\223.8617.54\bin\cmake\win\x64\bin\cmake.exe --build C:\Users\user1\projects\learning-c\cmake-build-debug --target prog_a -j 9
[1/2] Linking C shared library dynamic-library-2\lib_b.dll
FAILED: dynamic-library-2/lib_b.dll dynamic-library-2/liblib_b.dll.a
cmd.exe /C "cd . && C:\Users\user1\AppData\Local\JetBrains\Toolbox\apps\CLion\ch-0\223.8617.54\bin\mingw\bin\gcc.exe -fPIC -g -Wl,--export-all-symbols -shared -o dynamic-library-2\lib_b.dll -Wl,--out-implib,dynamic-library-2\liblib_b.dll.a -Wl,--major-image-version,0,--minor-image-version,0 dynamic-library-2/CMakeFiles/lib_b.dir/lib_b.c.obj -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomdlg32 -ladvapi32 && cd ."
C:\Users\user1\AppData\Local\JetBrains\Toolbox\apps\CLion\ch-0\223.8617.54\bin\mingw\bin/ld.exe: dynamic-library-2/CMakeFiles/lib_b.dir/lib_b.c.obj:lib_b.c:(.rdata$.refptr.newtestimport[.refptr.newtestimport]+0x0): undefined reference to `newtestimport'
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
How can the example be fixed to accomplish the described goal?
Windows DLLs are self-contained, and can not have undefined references similar to newtestimport, unless these references are satisfied by another DLL.
How can the example be fixed to accomplish the described goal?
The best fix is to pass the address of newtestimport into the function that needs it (initialize_lib_b() here).
If for some reason you can't do that, your next best option is to define the newtestimport as a dllexport variable in another DLL, e.g. lib_c.dll.
Then both the main executable and lib_b.dll would be linked against lib_c.lib, and would both use that variable from lib_c.dll.
P.S. Global variables are a "code smell" and a significant source of bugs. You should avoid them whenever possible, and in your example there doesn't seem to be any good reason to use them.

path not being detected by Nextflow

i'm new to nf-core/nextflow and needless to say the documentation does not reflect what might be actually implemented. But i'm defining the basic pipeline below:
nextflow.enable.dsl=2
process RUNBLAST{
input:
val thr
path query
path db
path output
output:
path output
script:
"""
blastn -query ${query} -db ${db} -out ${output} -num_threads ${thr}
"""
}
workflow{
//println "I want to BLAST $params.query to $params.dbDir/$params.dbName using $params.threads CPUs and output it to $params.outdir"
RUNBLAST(params.threads,params.query,params.dbDir, params.output)
}
Then i'm executing the pipeline with
nextflow run main.nf --query test2.fa --dbDir blast/blastDB
Then i get the following error:
N E X T F L O W ~ version 22.10.6
Launching `main.nf` [dreamy_hugle] DSL2 - revision: c388cf8f31
Error executing process > 'RUNBLAST'
Error executing process > 'RUNBLAST'
Caused by:
Not a valid path value: 'test2.fa'
Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run
I know test2.fa exists in the current directory:
(nfcore) MN:nf-core-basicblast jraygozagaray$ ls
CHANGELOG.md conf other.nf
CITATIONS.md docs pyproject.toml
CODE_OF_CONDUCT.md lib subworkflows
LICENSE main.nf test.fa
README.md modules test2.fa
assets modules.json work
bin nextflow.config workflows
blast nextflow_schema.json
I also tried with "file" instead of path but that is deprecated and raises other kind of errors.
It'll be helpful to know how to fix this to get myself started with the pipeline building process.
Shouldn't nextflow copy the file to the execution path?
Thanks
You get the above error because params.query is not actually a path value. It's probably just a simple String or GString. The solution is to instead supply a file object, for example:
workflow {
query = file(params.query)
BLAST( query, ... )
}
Note that a value channel is implicitly created by a process when it is invoked with a simple value, like the above file object. If you need to be able to BLAST multiple query files, you'll instead need a queue channel, which can be created using the fromPath factory method, for example:
params.query = "${baseDir}/data/*.fa"
params.db = "${baseDir}/blastdb/nt"
params.outdir = './results'
db_name = file(params.db).name
db_path = file(params.db).parent
process BLAST {
publishDir(
path: "{params.outdir}/blast",
mode: 'copy',
)
input:
tuple val(query_id), path(query)
path db
output:
tuple val(query_id), path("${query_id}.out")
"""
blastn \\
-num_threads ${task.cpus} \\
-query "${query}" \\
-db "${db}/${db_name}" \\
-out "${query_id}.out"
"""
}
workflow{
Channel
.fromPath( params.query )
.map { file -> tuple(file.baseName, file) }
.set { query_ch }
BLAST( query_ch, db_path )
}
Note that the usual way to specify the number of threads/cpus is using cpus directive, which can be configured using a process selector in your nextflow.config. For example:
process {
withName: BLAST {
cpus = 4
}
}

QBS: Explicitly setting qbs.profiles inside Products causing build to fail

My use-case is this:
I have a static library which I want to be available for some profiles (e.g. "gcc", "arm-gcc", "mips-gcc").
I also have an application which links to this library, but this applications should only build using a specific profile (e.g. "arm-gcc").
For this I am modifying the app-and-lib QBS example.
The lib.qbs file:
import qbs 1.0
Product {
qbs.profiles: ["gcc", "arm-gcc", "mips-gcc"] //I added only this line
type: "staticlibrary"
name: "mylib"
files: [
"lib.cpp",
"lib.h",
]
Depends { name: 'cpp' }
cpp.defines: ['CRUCIAL_DEFINE']
Export {
Depends { name: "cpp" }
cpp.includePaths: [product.sourceDirectory]
}
}
The app.qbs file:
import qbs 1.0
Product {
qbs.profiles: ["arm-gcc"] //I added only this line
type: "application"
consoleApplication: true
files : [ "main.cpp" ]
Depends { name: "cpp" }
Depends { name: "mylib" }
}
The app build fails. Qbs wrongly tries to link to the "gcc" version of the library instead of the "arm-gcc" version, as you can see in the log:
Build graph does not yet exist for configuration 'default'. Starting from scratch.
Resolving project for configuration default
Setting up build graph for configuration default
Building for configuration default
compiling lib.cpp [mylib {"profile":"gcc"}]
compiling lib.cpp [mylib {"profile":"arm-gcc"}]
compiling lib.cpp [mylib {"profile":"mips-gcc"}]
compiling main.cpp [app]
creating libmylib.a [mylib {"profile":"gcc"}]
creating libmylib.a [mylib {"profile":"mips-gcc"}]
creating libmylib.a [mylib {"profile":"arm-gcc"}]
linking app [app]
ERROR: /usr/bin/arm-linux-gnueabihf-g++ -o /home/user/programs/qbs/usr/local/share/qbs/examples/app-and-lib/default/app.7d104347/app /home/user/programs/qbs/usr/local/share/qbs/examples/app-and-lib/default/app.7d104347/3a52ce780950d4d9/main.cpp.o /home/user/programs/qbs/usr/local/share/qbs/examples/app-and-lib/default/mylib.eyJwcm9maWxlIjoiZ2NjIn0-.792f47ec/libmylib.a
ERROR: /home/user/programs/qbs/usr/local/share/qbs/examples/app-and-lib/default/mylib.eyJwcm9maWxlIjoiZ2NjIn0-.792f47ec/libmylib.a: error adding symbols: File format not recognized
collect2: error: ld returned 1 exit status
ERROR: Process failed with exit code 1.
The following products could not be built for configuration default:
app
The build fails only when selecting one profile in the app.qbs file, and this profile should not be the first profile in the qbs.profiles line in the lib.qbs file.
When selecting two or more profiles - the build succeeds.
My analysis:
I think this problem is related to multiplexing:
The lib.qbs contains more than one profile. This turns on multiplexing when building the library, which, in turn, adds additional 'multiplexConfigurationId' to the build-directory name (moduleloader.cpp).
The app.lib contains only one profile, so multiplexing is not turned on and the build-directory name does not get the extra string.
The problem can be solved by changing the code (moduleloader.cpp) so that multiplexing is turned even if there is only one profile i.e. with the following patch:
--- moduleloader.cpp 2018-10-24 16:17:43.633527397 +0300
+++ moduleloader.cpp.new 2018-10-24 16:18:27.541370544 +0300
## -872,7 +872,7 ##
= callWithTemporaryBaseModule<const MultiplexInfo>(dummyContext,
extractMultiplexInfoFromProduct);
- if (multiplexInfo.table.size() > 1)
+ if (multiplexInfo.table.size() > 0)
productItem->setProperty(StringConstants::multiplexedProperty(), VariantValue::trueValue());
VariantValuePtr productNameValue = VariantValue::create(productName);
## -891,7 +891,7 ##
const QString multiplexConfigurationId = multiplexInfo.toIdString(row);
const VariantValuePtr multiplexConfigurationIdValue
= VariantValue::create(multiplexConfigurationId);
- if (multiplexInfo.table.size() > 1 || aggregator) {
+ if (multiplexInfo.table.size() > 0 || aggregator) {
multiplexConfigurationIdValues.push_back(multiplexConfigurationIdValue);
item->setProperty(StringConstants::multiplexConfigurationIdProperty(),
multiplexConfigurationIdValue);
This worked for my use case. I don't know if it make sense in a broader view.
Finally, the questions:
Does it all make sense?
Is this a normal behavior?
Is this use-case simply not supported?
Is there a better solution?
Thanks in advance.
Yes, the default behavior with multiplexing is that the a non-multiplexed product depends on all variants of the dependency. In general, there is no way for a user to change that behavior, but there should be.
However, luckily for you, profiles are special:
Depends { name: "mylib"; profiles: "arm-gcc" }
This should fix your problem.

Compilation issue with sitmo prng, c++11, and armadillo

I'm trying to compile the sitmo prng under C++11 within an R package. The problematic code has been packaged and is available here. The objective of this R package is to make available the sitmo header file so that other packages are able to use the LinkTo field within description. As an added bonus, the package is scheduled to ship with an Armadillo + OpenMP example. There is one other package, mvnfast, that uses sitmo, but only under c++98 and boost headers.
I believe that the error which I am receiving is specific to OS X and clang. I haven't been able to replicate it on Windows via win-build. With that being said, the error is:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/random:3641:44: error: non-type template argument is not a constant expression
const size_t __logR = __log2<uint64_t, _URNG::max() - _URNG::min() + uint64_t(1)>::value;
The error has only popped up on the Rcpp dev list. The resolution in this case was to compile under C++98 and use boost.
The above error is followed by the following notes:
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/random:3773:18: note: in instantiation of function template specialization 'std::__1::generate_canonical<double, 53, sitmo::prng_engine>' requested here
* _VSTD::generate_canonical<_RealType, numeric_limits<_RealType>::digits>(__g)
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/random:3737:17: note: in instantiation of function template specialization 'std::__1::uniform_real_distribution<double>::operator()<sitmo::prng_engine>' requested here
{return (*this)(__g, __p_);}
^
sitmo_test.cpp:77:26: note: in instantiation of function template specialization 'std::__1::uniform_real_distribution<double>::operator()<sitmo::prng_engine>' requested here
double u = distunif(engine);
^
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/random:3641:44: note: non-constexpr function 'max' cannot be used in a constant expression
const size_t __logR = __log2<uint64_t, _URNG::max() - _URNG::min() + uint64_t(1)>::value;
^
../inst/include/prng_engine.hpp:100:23: note: declared here
static result_type (max)() { return 0xFFFFFFFF; }
The version of clang being used is:
Apple LLVM version 7.0.2 (clang-700.1.81)
Target: x86_64-apple-darwin15.3.0
Thread model: posix
Looking into the code, there is a bug in the sitmo prng_engine.h. min() and max() were declared as
static result_type (min)() { return 0; }
static result_type (max)() { return 0xFFFFFFFF; }
If you take a look at, say, standard LCG max from here, you could see that it is declared constexpr, ditto for min.
As soon as you make those methods constexpr in the sitmo header file, I believe you could use them in template expression.
UPDATE
I've looked into GCC 5 headers, methods indeed are declared constexpr

Cross platform way of testing whether a file is a directory

Currently I have some code like (condensed and removed a bunch of error checking):
dp = readdir(dir);
if (dp->d_type == DT_DIR) {
}
This works swimmingly on my Linux machine. However on another machine (looks like SunOS, sparc):
SunOS HOST 5.10 Generic_127127-11 sun4u sparc SUNW,Ultra-5_10
I get the following error at compile time:
error: structure has no member named `d_type'
error: `DT_DIR' undeclared (first use in this function)
I thought the dirent.h header was crossplatform (for POSIX machines). Any suggestions.
Ref http://www.nexenta.org/os/Porting_Codefixes:
The struct dirent definition in solaris does not contain the d_type field. You would need to make the changes as follows
if (de->d_type == DT_DIR)
{
return 0;
}
changes to
struct stat s; /*include sys/stat.h if necessary */
..
..
stat(de->d_name, &s);
if (s.st_mode & S_IFDIR)
{
return 0;
}
Since stat is also POSIX standard it should be more cross-platform. But you may want to use if ((s.st_mode & S_IFMT) == S_IFDIR) to follow the standard.

Resources