I often create some DLLs with Haskell that I load in R, and this works very well.
But I have some code dealing with the xlsx library, I can compile it to a DLL without issue, but when I load the DLL in R, this totally crashes the R session. However this occurs on Windows only, there's no issue on Linux.
I managed to find a minimal example and there's something weird. This is my minimal example:
{-# LANGUAGE ForeignFunctionInterface #-}
{-# LANGUAGE OverloadedStrings #-}
module TestDLL where
import Codec.Xlsx
import Control.Lens
import qualified Data.ByteString.Lazy as L
import Foreign
import Foreign.C
import Foreign.C.String (peekCString, newCString)
test :: IO ()
test = do
bs <- L.readFile "report.xlsx"
let value = toXlsx bs ^? ixSheet "List1" .
ixCell (3,2) . cellValue . _Just
putStrLn $ "Cell B3 contains " ++ show value
... some elementary functions here ...
If I compile this code to a DLL, loading this DLL in R crashes the R session on Windows. There's no such issue if I remove the test function. However the test function is not even exported (with foreign export) and it is not called by the other functions, isn't it weird ? If I don't export this function and if I don't use it, why the DLL deals with this function ?
And more importantly, why the R session crashes when I load the DLL, and how to fix that ?
Edit
I have a more minimal example now. This works:
test :: IO Xlsx
test = do
bs <- L.readFile "report.xlsx"
return $ toXlsx bs
And this crashes:
test :: IO (Maybe Worksheet)
test = do
bs <- L.readFile "report.xlsx"
return $ toXlsx bs ^? ixSheet "List1"
It looks like Windows has a problem with ^?.
Edit 2
No crash with this equivalent code:
test :: IO (Maybe Worksheet)
test = do
bs <- L.readFile "report.xlsx"
let xlsx = toXlsx bs
let sheets = _xlSheets xlsx
let mapping = DM.fromList sheets
return $ DM.lookup "List1" mapping
Windows has a problem with ^? ixSheet. Now let me try on my real example...
I don't have a solution (edit: I have one, see below) but I can say this is due to the limit of number of exported symbols.
When I compile the code
test :: IO (Maybe Worksheet)
test = do
bs <- L.readFile "report.xlsx"
let xlsx = toXlsx bs
let sheets = _xlSheets xlsx
let mapping = DM.fromList sheets
return $ DM.lookup "List1" mapping
and I inspect the DLL with DependencyWalker, I see there are 48318 exported symbols. That's acceptable.
But for the other code:
test :: IO (Maybe Worksheet)
test = do
bs <- L.readFile "report.xlsx"
return $ toXlsx bs ^? ixSheet "List1"
the generated DLL reaches the maximal number of exported symbols: there are 65535=2^16-1 exported symbols. This DLL is "truncated".
Edit: A possible solution !
A possible solution consists in using a def file. In a file MyDef.def, list the functions you want to export, e.g. funexport and HsStart, like this:
EXPORTS
funexport
HsStart
and add MyDef.def at the end of the command line you use to compile:
ghc -shared foo.hs StartEnd.c -o foo.dll MyDef.def
I have just tested this solution and it works. However this is the first time I test it, so I would not guarantee yet. I'm also surprised that ghc does not automatically do that.
Related
it is appearing in some big modules like matplotlib. For example expression :
import importlib
obj = importlib.import_module('matplotlib')
obj_entries = obj.__dict__
Between runs len of obj_entries can vary. From 108 to 157 (expected) entries. Especially pyplot can be ignored like some another submodules.
it can work stable during manual debug mode with len computing statement after dict extraction. But in auto it dont work well.
such error occures:
RuntimeError: dictionary changed size during iteration
python-BaseException
using clear python 3.10 on windows. Version swap change nothing at all
during some attempts some interesting features was found.
use of repr is helpfull before dict initiation.
But if module transported between classes like variable more likely lazy-import happening? For now there is evidence that not all names showing when command line interpriter doing opposite - returning what expected. So this junk of code help bypass this bechavior...
Note: using pkgutil.iter_modules(some_path) to observe modules im internal for pkgutil ModuleInfo form.
import pkgutil, importlib
module_info : pkgutil.ModuleInfo
name = module_info.name
founder = module_info.module_finder
spec = founder.find_spec(name)
module_obj = importlib.util.module_from_spec(spec)
loader = module_obj.__loader__
loader.exec_module(module_obj)
still unfamilliar with interior of import mechanics so it will be helpfull to recive some links to more detail explanation (spot on)
I'm wondering if it's possible to do the following with hdf5/cxi file:
have one external h5 file that stores numpy array with 4D dimension
have another cxi file in which it is highle required to add group which will contain an array of external links to h5 file
Expect in cxi to get somethink like that:
Group1/subgroup {num, ExternalLink to h5 file}
where num is a length of required array of links.
I try to do:
import h5py as h5
import numpy as np
h5_file = sys.argv[1]
h5path = sys.argv[2]
cxi_file = sys.argv[3]
cxi_path = sys.argv[4]
num = sys.argv[5]
link = h5.ExternalLink(h5_file, h5path)
l = np.array([link] * num)
with h5.File(cxi_file, 'a') as f:
dset = f.create_dataset(cxi_path, (num,))
for i in range(num):
dset[i] = l[i]
But it didn't work. I also tried dset = f.create_dataset(path_to_new_mask,data=l) and made list of this file with length = num, but all this steps failed.
I'll be very greatful if someone can help.
kitsune_breeze, I reviewed the Q&A and the comments. There are several areas that need to be clarified. Let's start with external links versus object or region references.
As I understand you want to create a dataset (aka an array) of external links (with each link referencing a different HDF5 file).
The answer from Mahsa Hassankashi on 19-April describes how to create a dataset of dtype=h5py.ref_dtype or dtype=h5py.regionref_dtype. The first is an object reference, and the second is a region reference. They are not the same as external links! Also, the example code requires h5py 2.10.0 and you are using h5py 2.9.0.. (FYI, there is a solution to this in 2.9.0 if you choose to use object or region references.)
Here's the bad news: based on my tests, you can't create a dataset (or np array) of HDF5 external links. Here are the steps to see why:
In [1]: import h5py
In [2]: h5fw = h5py.File('SO_61290760.h5',mode='w')
# create an external link object
In [3]: link_obj = h5py.ExternalLink('file1.h5','/')
In [4]: type(link_obj)
Out[4]: h5py._hl.group.ExternalLink
In [5]: link_dtype = type(link_obj)
In [6]: h5fw.create_dataset("MyRefs", (10,), dtype=link_dtype)
Traceback (most recent call last):
...
TypeError: Object dtype dtype('O') has no native HDF5 equivalent
Reading the h5py documentation, it appears object and region references are also dtype('O') datatypes, and required additional metadata to implement them. There is no mention that this was done for External Links. As a result, I don't think you can create an array of External Links (because there isn't a dtype to support them).
That said, you can still create External Links from 1 HDF5 file to multiple HDF5 files. I have a simple example here (look under Method 1: Create External Links).
How can I combine multiple .h5 file?
If you decide to use Object or Region References, you need to use a different dtype specification in h5py 2.9.0.
Object Reference:
2.10.0 use: h5py.ref_dtype
2.9.0 use: h5py.special_dtype(ref=h5py.Reference)
Region Reference:
2.10.0 use: h5py.regionref_dtype
2.9.0 use: h5py.special_dtype(ref=h5py.RegionReference)
Code below demonstrates the behavor in 2.9.0:
In [9]: type(h5py.ref_dtype)
Traceback (most recent call last):
...
AttributeError: module 'h5py' has no attribute 'ref_dtype'
In [10]: type(h5py.special_dtype(ref=h5py.Reference))
Out[10]: numpy.dtype
In [11]: type(h5py.regionref_dtype)
Traceback (most recent call last):
...
AttributeError: module 'h5py' has no attribute 'regionref_dtype'
In [12]: type(h5py.special_dtype(ref=h5py.RegionReference))
Out[12]: numpy.dtype
In [13]: dset = h5fw.create_dataset("MyRefs", (10,), dtype=h5py.special_dtype(ref=h5py.Reference))
In [14]: dset.dtype
Out[14]: dtype('O')
Try it
myfile = h5py.File('foo.hdf5','w')
myfile['ext link'] = h5py.ExternalLink("otherfile.hdf5", "/path/to/resource")
dset = f.create_dataset("MyRefs", (100,), dtype=h5py.ref_dtype)
Or:
dset = f.create_dataset("ref", (2,), dtype=h5py.regionref_dtype)
http://docs.h5py.org/en/stable/refs.html#storing-references-in-a-dataset
http://docs.h5py.org/en/latest/high/group.html#external-links
I am using Cleanide for Clean3.0 programming language.
What I am trying to do is to implement a function that receive name of a directory in my system, and return a list of all the files in that directory.
I don't know if the defintion of such function needs to be like File -> [string] or maybe something else, even that directory is a file maybe this is not the developers of Clean meant...
Thank a lot!
This functionality is not available in the StdEnv environment, but there are two libraries that can help with this:
The Directory library contains a module Directory which has a function getDirectoryContents :: !Path !*env -> (!(!DirError, [DirEntry]), !*env) | FileSystem env.
The Platform library contains a module System.Directory which has a function readDirectory :: !FilePath !*w -> (!MaybeOSError [FilePath], !*w).
In both cases the first argument is a path to the directory and the second argument is the *World, which is the typical way of Clean to perform impure operations (see chapter 9 of the language report).
Code examples
With Directory:
import Directory
Start w
# (dir,w) = getDirectoryContents (RelativePath []) w
= dir
With Platform:
import System.Directory
Start w
# (dir,w) = readDirectory "." w
= dir
I'm still a noob with F#, and I don't understand all the syntax and logic for loading and using packages.
For example, i would like to use (Blue Mountain's) RProvider.
http://bluemountaincapital.github.io/FSharpRProvider/index.html
Using VS2015, in my current solution, I've installed the package with the PM console and Install-Package RProvider
I modified a bit the RProvider.fsx because I've got newer versions of R.NET Community
#nowarn "211"
// Standard NuGet or Paket location
#I "."
#I "lib/net40"
// Standard NuGet locations for R.NET
#I "../R.NET.Community.1.6.4/lib/net40"
#I "../R.NET.Community.FSharp.0.1.9/lib/net40"
// Standard Paket locations for R.NET
#I "../R.NET.Community/lib/net40"
#I "../R.NET.Community.FSharp.1.6.4/lib/net40"
// Try various folders that people might like
#I "bin"
#I "../bin"
#I "../../bin"
#I "lib"
#I "../packages"
// Reference RProvider and RDotNet
#r "RDotNet.dll"
#r "RDotNet.FSharp.dll"
#r "RProvider.dll"
#r "RProvider.Runtime.dll"
open RProvider
do fsi.AddPrinter(fun (synexpr:RDotNet.SymbolicExpression) -> synexpr.Print())
Now my questions are
1) how to load a package (RProvider) from F# interactive ?
well actually i managed to do it this way
For example the RProvider.fsx file is in the path
C:\Users\Fagui\Documents\GitHub\Learning Fsharp\Algo Stanford\packages\RProvider.1.1.15\RProvider.fsx
what i did is
#I #"C:\Users\Fagui\Documents\GitHub\Learning Fsharp\Algo Stanford";;
#load "packages\RProvider.1.1.15\RProvider.fsx";;
and it works :-)
but can I avoid writing the whole path ?
2) In VS2015 if I want to include it in a solution...
in the solution explorer i have included the RProvider.fsx file (below AssemblyInfo.fs, App.config and packages.config come after, is this right ?)
and last the program itself Rtype.fs
I'm trying to reproduce the example from
http://bluemountaincapital.github.io/FSharpRProvider/Statistics-QuickStart.html
open System
open *RDotNet* // the namespace or module 'RDotNet' is not defined
open *RProvider*
open *RProvider*.graphics
open RProvider.stats
// let x = System.Environment.CurrentDirectory
// val x : string
printfn "hello world"
Console.ReadKey() |> ignore
// Random number generator
let rng = Random()
let rand () = rng.NextDouble()
// Generate fake X1 and X2
let X1s = [ for i in 0 .. 9 -> 10. * rand () ]
let X2s = [ for i in 0 .. 9 -> 5. * rand () ]
// Build Ys, following the "true" model
let Ys = [ for i in 0 .. 9 -> 5. + 3. * X1s.[i] - 2. * X2s.[i] + rand () ]
let dataset =
namedParams [
"Y", box Ys;
"X1", box X1s;
"X2", box X2s; ]
|> R.data_frame
let result = R.lm(formula = "Y~X1+X2", data = dataset)
let coefficients = result.AsList().["coefficients"].AsNumeric()
let residuals = result.AsList().["residuals"].AsNumeric()
let summary = R.summary(result)
*summary.AsList().["r.squared"].AsNumeric()
R.plot result*
//this expression should have type 'unit' but has type 'NumericVector'...
I'm getting some warnings/errors by Intellisense although the compiler managed a build.
When executing the exe, it looks like the windows screen is busy, i manage to see some graphs, but they look like they have got nothing to do with what Rtype.fs is saying...
thanks for helping !
EDIT
First of all, I would not recommend using a different version of R.NET than the one that RProvider installs automatically as a dependency. The loading is a bit fragile and it might break things.
1) Regarding the path, you should be able to pass relative path to #load, so just dropping the #I from your script should do the trick.
2) When referencing a dependency from a project (rather than from a script file), you need to add a dependency to the project references. In Visual Studio, this is done by right click on the "References" in your project and using "Add Reference". For type providers, you also need to click "Enable" when the reference is loaded.
I would like to construct a variable for RobotFramework within
the variable file but using the Linux environmental variable.
Could you advise on the syntax please ??
My current attempts with this:
vif_vlan = "110"
path_scripts = '%{MY_DIR}/my_path/scripts'
remote_path = "/home/mcast/mgen"
end up in not expanding the env variable %{MY_DIR} ...
Tx
Environment variables are in the environ dictionary of the os module:
import os
path_scripts = os.path.join(os.environ['MY_DIR']', 'my_path', 'scripts')
Your syntax is almost correct - Not for Python, but for Robot Framework.
In your .robot files, you can retrieve environment variables as follows:
** Variables **
| ${MY_PATH_TO_SCRIPTS} | %{MY_DIR}/my_path/scripts