Kotlin classes cannot be found when reflecting filepaths to source directories or jars - reflection

===Update: Using org.reflections:reflections:0.9.11
Looking to use the following line to pull a list of class names from Kotlin source...
Reflections.getSubTypesOf(Any::class.java)
However I receive a message that Kotlin class files aren't being seen when I run the following script...
val classLoader = URLClassLoader(this.getDirectoryUrls(), null)
println("retrieved class loader")
val config = getConfig(classLoader)
println("retrieved source config")
val reflections = Reflections(config)
println("retrieved reflections")
// For 3 paths: Reflections took 3 ms to scan 3 urls, producing 0 keys and 0 values
=== Update: The 3 urls added by "getDirectoryUrls()" are directories containing kotlin class source files.
Below is my config... ideas?
private fun getConfig(classLoader: ClassLoader): ConfigurationBuilder {
val config = ConfigurationBuilder().setUrls(ClasspathHelper.forClassLoader(classLoader))
// .setScanners(SubTypesScanner(false), ResourcesScanner())
if (!packagePath.isNullOrBlank()){
System.out.println("looking in package [$packagePath]")
config.filterInputsBy(FilterBuilder().include(FilterBuilder.prefix(packagePath)))
}
config.addClassLoader(classLoader)
config.setScanners(SubTypesScanner(), TypeAnnotationsScanner())
return config
}

Setting SubTypesScanner(false) seems to be required to get any types with getSubTypesOf(Any::class.java) (that parameter itself stands for excludeObjectClass). Looking at the bytecode of Kotlin classes you immediately see, that they are actually looking the same as Java classes. There is no Any-superclass there. Note that Kotlins Any is actually also in other means very similar to Javas Object (but not the same, check also the following answer to 'does Any == Object'). So, we need to include the Object-class when scanning for subtypes of Any (i.e. excludeObjectClass=false).
Another problem could be the setup of your URL array. I just used the following to setup the reflections util:
val reflections = Reflections(ConfigurationBuilder()
.addUrls(ClasspathHelper.forPackage("my.test.package"))
.setScanners(TypeAnnotationsScanner(), SubTypesScanner(false)))
which will resolve all matching subtypes and will return subtypes also for Any.
reflections.getSubTypesOf(MyCustomSuperType::class.java).forEach(::println)
reflections.getSubTypesOf(Any::class.java).forEach(::println)
Analysing further: you mention "Kotlin class source files"... if that means you are pointing to the directory containing the .kt-files, then that is probably your problem. Try to use the directory which contains the .class-files instead. Moreover, ensure that the classes are on the classpath.
Maybe you know already, maybe not? Note also that if you have a (classes) directory, say /sample/directory, which is on the classpath and which contains a package, say org.example (which corresponds to the folder structure org/example or full path /sample/directory/org/example) then you must ensure that you add an URL similar to the following:
YourClass::class.java.classLoader.getResource("")
and not:
YourClass::class.java.classLoader.getResource("org.example")
// nor:
YourClass::class.java.classLoader.getResource("org/example")
You basically require the "base" directory (in the example /sample/directory or from the view of the classloader just "")) where to lookup the packages and not the package itself. If you would supply one of the latter URLs, only classes that are in the default package (within /sample/directory/org/example) would actually be found, which however is a rather uncommon setup.

Related

Sqlalchemy sqlite url relative to home or environment variable

A relative sqlalchemy path to a sqlite database can be written as:
sqlite:///folder/db_file.db
And an absolute one as:
sqlite:////home/user/folder/db_file.db
Is it possible to write a path relative to home? Like this:
sqlite:///~/folder/db_file.db
Or even better, can the path contain environment variables?
sqlite:////${MY_FOLDER}/db_file.db
This is the context of an alembic.ini file. So if the previous objectives are not possible directly, may I be able to cheat using variable substitution?
[alembic]
script_location = db_versions
sqlalchemy.url = sqlite:///%(MY_FOLDER)s.db
...
I have gone around this issue by modifying the values in the config object just after env.py imports it:
# this is the Alembic Config object, which provides
# access to the values within the .ini file in use.
config = context.config
# import my custom configuration
from my_app import MY_DB_URI
# overwrite the desired value
config.set_main_option("sqlalchemy.url", MY_DB_URI)
Now config.get_main_option("sqlalchemy.url") returns the MY_DB_URI you wanted.
As others have pointed out, one key is 3 slashes for relative, 4 for absolute.
But it took for me than just that...
Had trouble with just a string, I had to do this:
db_dir = "../../database/db.sqlite"
print(f'os.path.abspath(db_dir): {str(os.path.abspath(db_dir))}')
SQLALCHEMY_DATABASE_URI = "sqlite:///" + os.path.abspath(db_dir) # works
# SQLALCHEMY_DATABASE_URI = "sqlite:///" + db_dir # fails
From the alembic documentation (emphasis mine):
sqlalchemy.url - A URL to connect to the database via SQLAlchemy. This configuration value is only used if the env.py file calls upon them; in the “generic” template, the call to config.get_main_option("sqlalchemy.url") in the run_migrations_offline() function and the call to engine_from_config(prefix="sqlalchemy.") in the run_migrations_online() function are where this key is referenced. If the SQLAlchemy URL should come from some other source, such as from environment variables or a global registry, or if the migration environment makes use of multiple database URLs, the developer is encouraged to alter the env.py file to use whatever methods are appropriate in order to acquire the database URL or URLs.
So for this case, the sqlalchemy url format can be circunvented and generated by python itself.

spring boot/spring web app embedded version number

What are the strategies to embed a unique version number in a Spring application?
I've got an app using Spring Boot and Spring Web.
Its matured enough that I want to version it and see it displayed on screen at run time.
I believe what you are looking for is generating this version number during build time (Usually by build tools like Ant, Maven or Gradle) as part of their build task chain.
I believe a quite common approach is to either put the version number into the Manifest.mf of the produced JAR and then read it, or create a file that is part of the produced JAR that can be read by your application.
Another solution would be just using Spring Boot's banner customization options described here: http://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-spring-application.html#boot-features-banner
However, this will only allow you to change spring-boot banner.
I also believe that Spring Boot exposes product version that is set in Manifest.MF of your application. To achieve this you will need to make sure Implementation-Version attribute of the manifest is set.
Custom solution for access anywhere in the code
Lets assume you would like to have a version.properties file in your src/main/resources that contains your version information. It will contain placeholders instead of actual values so that these placeholders can be expanded during build time.
version=${prodVersion}
build=${prodBuild}
timestamp=${buildTimestamp}
Now that you have a file like this you need to fill it with actual data. I use Gradle so there I would make sure that processResources task which is automatically running for builds is expanding resources. Something like this should do the trick in the build.gradle file for Git-based code:
import org.codehaus.groovy.runtime.*
import org.eclipse.jgit.api.*
def getGitBranchCommit() {
try {
def git = Git.open(project.file(project.getRootProject().getProjectDir()));
def repo = git.getRepository();
def id = repo.resolve(repo.getFullBranch());
return id.abbreviate(7).name()
} catch (IOException ex) {
return "UNKNOWN"
}
}
processResources {
filesMatching("**/version.properties") {
expand (
"prodVersion": version,
"prodBuild": getGitBranchCommit(),
"buildTimestamp": DateGroovyMethods.format(new Date(), 'yyyy-MM-dd HH:mm')
)
}
}
processResources.outputs.upToDateWhen{ false }
In the code about the following is happening:
We defined a function that can take a build number out of the VCS
(in this case Git). The commit hash is limited to 7 characters.
We configure the processResources task to process
version.properties file and fill it with our variables.
prodVersion is taken from Gradle project version. It's usually set
as version in gradle.properties file (part of the general build
setup).
As a last step we ensure that it's always updated (Gradle
has some mechanics to detect if files ened to be processed
Considering you are on SVN, you will need to have a getSvnBranchCommit() method instead. You could for instance use SVNKit or similar for this.
The last thing that is missing now is reading of the file for use in your application.
This could be achieved by simply reading a classpath resource and parsing it into java.util.Properties. You could take it one step further and for instance create accessor methods specifically for each field, e.g getVersion(), getBuild(), etc.
Hope this helps a bit (even though may not be 100% applicable straight off)
Maven can be used to track the version number, e.g.:
<!-- pom.xml -->
<version>2.0.3</version>
Spring Boot can refer to the version, and expose it via REST using Actuator:
# application.properties
endpoints.info.enabled=true
info.app.version=#project.version#
Then use Ajax to render the version in the browser, for example using Polymer iron-ajax:
<!-- about-page.html -->
<iron-ajax auto url="/info" last-response="{{info}}"></iron-ajax>
Application version is: [[info.app.version]]
This will then show in the browser as:
Application version is: 2.0.3
I'm sure you've probably figured something out since this is an older question, but here's what I just did and it looks good. (Getting it into the banner requires you to duplicate a lot).
I'd recommend switching to git (it's a great SVN client too), and then using this in your build.gradle:
// https://github.com/n0mer/gradle-git-properties
plugins {
id "com.gorylenko.gradle-git-properties" version "1.4.17"
}
// http://docs.spring.io/spring-boot/docs/current/reference/html/deployment-install.html
springBoot {
buildInfo() // create META-INF/build-info.properties
}
bootRun.dependsOn = [assemble]
And this in your SpringBoot application:
#Resource
GitProperties props;
#Resource
BuildProperties props2;
Or this way to expose those properties into the standard spring environment:
#SpringBootApplication
#PropertySources({
#PropertySource("classpath:git.properties"),
#PropertySource("classpath:META-INF/build-info.properties")
})
public class MySpringBootApplication {
and then referencing the individual properties as needed.
#Value("${git.branch}")
String gitBranch;
#Value("${build.time}")
String buildTime;

JDO + datanucleus API enhancement

I want to load dynamically XML meta-data files that are not in the classpath (the XML meta-data files will be generated after launching my app).
I thought I could call the method
MetaDataManager.registerFile(java.lang.String fileURLString, FileMetaData filemd, ClassLoaderResolver clr)
Then, I tried the API enhancement, so I added the following lines:
JDOEnhancer enhancer = JDOHelper.getEnhancer();
enhancer.setVerbose(true);
enhancer.addClasses(ClassToPersist.class.getName()).enhance();
getClass().getClassLoader().loadClass(ClassToPersist.class.getName());
The following jars are in the classpath: datanucleus-api-jdo.jar, datanucleus-connectionpool.jar, datanucleus-core.jar datanucleus-rdbms.jar, jdo-api.jar, asm.jar.
But when I launch my app, I get this exception:
Caused by: mypackage.MyException:
org.datanucleus.api.jdo.exceptions.ClassNotPersistenceCapableException: The class "mypackage.ClassToPersist" is not persistable. This means that it either hasnt been enhanced, or that the enhanced version of the file is not in the CLASSPATH (or is hidden by an unenhanced version), or the Meta-Data/annotations for the class are not found.
Do you please have any idea how to fix this ?
PS: I also noticed that the method enhance returns 0 which indicates that the class has not been enhanced (we could exclude the other options)
Thanks
I think I found an easy way to fix this.
At runtime, I created a jar that contains the updated metadata file as META-INF/package.jdo. Then I added this jar to the classpath as described here
Using this hacks, I didn't need to re-enhace my class since it is enhanced after compilation.
(But for information, I had to call JDOEnhancer.addFiles() to enhance my class.)
So your call to loadClass has loaded the unenhanced class (since it would have been loaded in order to perform the enhancement presumably), and you didn't follow the tutorial that is provided at http://www.datanucleus.org/documentation/development/dynamic_class_metadata_enhance_runtime.html

What is job.get() and job.getBoolean() in mapreduce

I am working on pdf document clustering over hadoop so I am learning mapreduce by reading some examples on internet.In wordcount examples have lines
job.get("map.input.file")
job.getboolean()
What is function of these functions?what is exactly map.input.file where is it to set? or is it just a name given to input folder?
Please post answer if anyone know.
For code see the following link
wordcount 2.0 example=http://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html
These are job configurations. i.e. set of configurations which are passed on to each mapper and reducer. Now, these configurations consist of well defined mapreduce/hadoop related configurations as well as user-defined configurations.
In your case, map.input.file is a pre-defined configuration and yes it is set to a comma separated list of all the paths you have set as input path.
While wordcount.skip.patterns is a custom configuration which is set as per user's input, and you may see this configuration to be set in run() as follows:
conf.setBoolean("wordcount.skip.patterns", true);
As for when to use get and when to use getBoolean, it should be self-explanatory, as whenever you want to set a value of type boolean you will use getBoolean and setBoolean to get and set the specific config value respectively. Similarly you have specific methods for other data types as well. If it is string then you may use get().

Flex - error 1046 - some .as files don't get importet

I received a Flex project and when trying to compile it i get a few 1046 errors that say the Type was not found or was not a compile-time constant MyClass
however - the respective files are listed on the top of the file in an import clause like this:
import com.folder1.folder2.folder3.MyClass;
and if i check the folder structure, MyClass.as is there.
however, if i type this same line (import com.folder1.folder2.folder3.MyClass;) and check at each . what the autocompletion suggests, I see only a subset of the as classes that are actually there on the harddisk.
What determines which classes and folders are suggested by the autocompletion function? I don't get any compile error on the corresponding import statements that import MyClass
//edit:
screenshot 1 shows the file in which the error occurs that tries to import the class in question (Updater)
http://neo.cycovery.com/flex_problem.gif
screenshot 2 shows the file Updater.as
http://neo.cycovery.com/flex_problem2.gif
the censored part of the path matches in both cases (folder structure and package statement in Updater.as)
screenshot 3 shows where the error actually happens:
http://neo.cycovery.com/flex_problem3.gif
interestingly, the variable declaration
private var _updater:Updater = new Updater();
further up in the file does not give an error
This project is set up wrong. Its obvious your application can not find the classes.
Move your "com" folder and all of the contents into your "src" folder.
Or perhaps include the files in your source path?
right click on the project name->properties->flex Build Path->add folder
the import is based on the 'package' declaration within the file itself (at the top of the file). If the file's package declaration does not match the actual folder structure, you will get problems.
Check the classes you can't see in the autocompletion list. Maybe those classes' package name doesn't match the actual structure.
Rob
Check your actionscript source paths. Any chance that the folders you are seeing (events and objects) are in there explicitly, and the others are not? Normally, you have all your source inside a folder like src that is in the source path, so that the compiler can find anything anywhere inside it. But you can just as easily make your source paths too specific and just see a few things...

Resources