I am using Pax Exam to perform integration tests to my OSGi application. I have a configuration factory in which I specify the Karaf feature of my application to be installed in the test container and then modify some of a proerty of a .cfg file installed as part of my feature.
public class TestConfigurationFactory implements ConfigurationFactory {
#Override
public Option[] createConfiguration() {
return options(
karafDistributionConfiguration()
.frameworkUrl(
maven().groupId("org.apache.karaf")
.artifactId("apache-karaf")
.version("3.0.1").type("tar.gz"))
.unpackDirectory(new File("target/exam"))
.useDeployFolder(false),
keepRuntimeFolder(),
// Karaf (own) features.
KarafDistributionOption.features(
maven().groupId("org.apache.karaf.features")
.artifactId("standard").classifier("features")
.version("3.0.1").type("xml"), "scr"),
// CXF features.
KarafDistributionOption.features(maven()
.groupId("org.apache.cxf.karaf")
.artifactId("apache-cxf").version("2.7.9")
.classifier("features").type("xml")),
// Application features.
KarafDistributionOption.features(
maven().groupId("com.me.project")
.artifactId("my-karaf-features")
.version("1.0.0-SNAPSHOT")
.classifier("features").type("xml"), "my-feature"),
KarafDistributionOption.editConfigurationFilePut(
"etc/com.me.test.cfg", "key", "value"));
}
}
The property I specify in editConfigurationFilePut is modified correctly, however the rest of the .cfg file's properties are deleted. If I use the editConfigurationFilePut method to edit one of Karaf's configuration files it works as expected (just adds the new property without modifying the existing ones) so I am thinking that perhaps the problem is that Pax Exam attempts to modify the configuration before the .cfg file is installed by my feature and therefore creates a new file to put the property in. If this is the case is there some way to synchronise this process so that the .cfg file is edited only after the feature is properly installed?
There are a two different reasons for this.
1) The feature does get installed after the configfile has been "edited"
2) The feature only contains a config section and not a configfile section
I'd guess reason one is the most likely cause of this since it needs a running Karaf to install a feature through Pax Exam. So to work around reason one, replace the config with a config file present in your test project.
For reason two, make sure the feature actually does reference a config instead of a configuration admin config, or add your config to the configuration of the config-admin service. You can achieve this by injecting the ConfigAdmin service in your unit test and add your properties to the configuration pid.
EDIT:
Combine both solutions
Since because of 1) it takes longer for the config-file to be actually available, let config-admin service do the rest.
Make sure your test does retrieve the config-admin service either by injecting it or by waiting for it's availability.
Now within a #Before method make sure you wait till your config is complete and change it from there on. This way you don't need to duplicate the config files.
Related
Just trying hands with Spring Cloud Contract. While running the generated test on provider side and when the application context is instantiated, it is unable to read config values from application.yml. When i move the test from generated build folder to src/test/java then issue is not seen anymore.
Which implies since build folder is outside of project src/.. structure, it can't read the config.
How can i fix it?
How do you access the value from application.yml
Let's suppose in application.yml you have the following content:
example:
baseUri: https://jsonplaceholder.typicode.com
You can simply access it in your test using:
#Value("${example.baseUri}")
String exampleBaseUri;
Additionally, if you want a profile just for tests you can create a file application-test.yml where you add properties. To access it values from this file you need to add before your test class:
#ActiveProfiles("test")
I am looking for a way to handle this challenge: we are a geographically dispersed dev team using ASP.NET Web API and Angular to build a web app.
The thing that causes the grief is the fact that not all team members use the same database setup for their dev work. Yes, I know - I can use web.config transforms to set the proper connection strings for test, staging and production (and I'm already doing this) - but this is not what I'm talking about.
Due to reasons beyond our control at this time, we have
some developers working on a local SQL Server instance using server=(local);database=OurDB as their connection string
other developers using a central developer SQL Server in their location, using something like server=someserver.mycorp.com;database=OurDB
and a few exotic cases with yet other settings
Now every time someone commits a change to the Git repo, and happens to also change something in the web.config, his connection string is committed to the repo. So when I then go pull that latest commit, my settings to my local DB server are overwritten by this other guy's settings.
I am looking for a way to handle this - I was hoping I might be able to
hook into the Git pull process and automagically update the web.config connection string to my local needs whenever I pull something
somehow reference a connection string (or external config file) based on e.g. my currently logged in user's name or something like that
But I can't seem to find any way of doing this. I was wondering if I need to build a VS extension to handle this - any starters for that? Has anyone done something like this before and could share his code? (or has it up on Github)
The web.config configuration system used in ASP.NET is not flexible enough to support the more advanced scenario you have described. So, why use it? You could store the configuration in files within the repository, one per developer. Or they could be stored outside the repository or otherwise ignored.
The real trick is that most older applications don't have a single root that retrieve the configuration, so you have to refactor your application to utilize a flexible configuration system. For your staging/production environments you probably still want to use the config in web.config. The following code can give you a basic idea of one way to structure it:
public class MyApplicationConfiguration
{
public string MainConnectionString { get; set; }
}
public class ConfigurationRetriever
{
public MyApplicationConfiguration GetConfiguration()
{
// You might look for the absence or presence of an environment variable to determine this
bool isLocalDevelopment = IsApplicationLocalDevelopment();
var config = new MyApplicationConfiguration();
if(isLocalDevelopment)
{
config.MainConnectionString = Environment.GetEnvironmentVariable("MyApplication_MainConnectionString");
//or get it from a JSON file or XML file or config database
}
else
{
config.MainConnectionString = ConfigurationManager.ConnectionStrings["MainConnectionString"].ConnectionString;
}
}
}
Rather than rolling your own config building logic, you might refactor your application to leverage Microsoft.Extensions.Configuration. It's not just for .NET Core. It's for .NET Standard. So you can use it even in your legacy ASP.NET applications. For reading the web.config, you could probably use Microsoft.Extensions.Configuration.Xml. Or you can write your own adapter that pulls values out of ConfigurationManager. I did a basic test, and this worked as expected.
Given a feature with a <configfile> and a <bundle>, how can I ensure that the file is deployed before the bundle? What I'm seeing is that my bundle gets started first and the file deployed second (even if <configfile> is the first tag).
I guess this is could be the <bundle> being considered a pre-requisite so it makes sense to start that before processing the rest of the <feature>?
Actually if you are doing it right, your bundles shouldn't care if the configuration is available or not. The service should either not be started or with a default setting in case no "external" configuration has been set. In case of a new configuration the service will be restarted with the new configuration.
I have an ASP.NET web application written in C# 4.0. The application references a class library that comes with its own configuration file. At runtime, the class library uses similar to the following code to load this specific configuration:
var exeConfigPath = this.GetType().Assembly.Location;
var config = ConfigurationManager.OpenExeConfiguration(exeConfigPath);
This is done because the library has to load its bundled configuration rather than the application configuration. The application configuration should not be concerned of the library's settings and should not be able to alter them.
Now, there are a few other things that need to be done for this concept to work. I have to set the library's configuration file build operation as Content in the properties window and the Copy to be Copy Always or Copy If Newer. So far so good - the file gets automatically both into the class library's bin directory, and the web applications's bin directory, and is correctly renamed from App.config to CustomLibrary.dll.config (as supposed, the library's dll is CustomLibrary.dll).
Now I am facing two issues.
1) When I publish the web application to a filesystem location (mapped in IIS), the CustomLibrary.dll.config appears back as App.config in the bin folder of the published app. OK - I will rename it in the class library project to match the expected convention - and problem solved.
2) Even when published, the IIS compiles the application again and stores it in the ASP.NET Temporary Files. There is a fancy directory structure with a folder dedicated for each assembly referenced. The folder corresponding to the CustomLibrary.dll does not contain the config file in it. Since this.GetType().Assembly.Location will return the path to the temp folder, the application fails to load the configuration and crashes as it should.
I need to preserve the pattern of having the configuration in the class library, and be able to make it work in the web application. When manually copying the .config to the temp folder, the app works, but see, I really hate manual copying to randomly-named folders.
Is there a way to either prevent IIS from using the temp folders, or to make it copy along the config files? I believe the problem I am facing is configuration-related rather than conceptual since the application works as expected when the config file is in place. I'd prefer not to mess with using hard-coded physical paths to the config file either.
Edit:
To make it clearer, I will point out what and why I want to achieve. The idea is that the library and the web project will be developed as separate products - there will be no user or application specific information in the configuration of the library, so it will not change for different use scenarios. It is also rather specific to the class library functionality rather than the end application. It makes sense for me to keep the library's configuration information bundled within it (similar to Java, where a spring context xml file, or a properties file, get bundled with the jar of the library). I'd like to avoid having to copy the configuration in each app/web config of the consumer application. There will be cases where the consumer application is developed by third parties, and I do not want to rely on them doing their configuration right for my stuff to work. Again, the only issue here is not having the config file copied to the right place.
If those are static, internal settings that nobody should see or change, wouldn't you be better off having a file with the configuration included within the class library as an embedded resource? Either that or just a static class with the settings.
That way you'd be certain that nobody alters it, which in your scenario seems to be a plus.
I have come along a way to work arround the described issue, still not a very pleasant one to my requirements.
The solution is to take advantage of the application configuration (web.config in web apps, or app.config) which is always available. I have added as settings the absolute paths to the config file for each library. So I ended up with:
<!--
THIS IS IN THE WEB.CONFIG FILE
-->
<appSettings>
<add key ="ClassLibrary_ConfigPath"
value ="{My Publish Output Folder}\ClassLibrary.dll.config"/>
</appSettings>
and the class library now uses the following code to load its configuration:
Configuration config = null;
try
{
var exeConfigPath = this.GetType().Assembly.Location;
config = ConfigurationManager.OpenExeConfiguration(exeConfigPath);
}
catch (Exception e)
{
if (!IsConfigurationNotFoundError(e))
{
// IsConfigurationNotFoundError logic skipped for brevity
var exeConfigPath =
ConfigurationManager.AppSettings["ClassLibrary_ConfigPath"];
if (exeConfigPath != null)
{
config = ConfigurationManager.OpenExeConfiguration(exeConfigPath);
}
}
else
{
throw;
}
}
While this works, I will wait for a better solution if possible. Still, I do not have to copy the entire ClassLibrary.dll.config into the web.config file, but now I must manage filesystem locations and be aware of app-setting names. What I really want is the consumer app of the ClassLibrary.dll not to deal with its configuration in any way. If it were a desktop app, I have this covered, as Visual Studio copies the ClassLibary.dll.config appropriately. I hope there is a way to make it work smoothly for web apps.
The short answer is: you can't. You have to merge both configuration sections and place all settings in the main configuration file of your application. In case of the web application it would be the web.config. Read this
I've seen many manifestations of ways to use the user class path as precedent to the hadoop one. Often times this is done if an m/r job needs a specific version of a library that hadoop coincidentally already uses an older version of (for example jackson's json parser or commons http , etc.)
In any case : I've seen :
mapreduce.task.classpath.user.precedence
mapreduce.task.classpath.first
mapreduce.job.user.classpath.first
Which one of these parameters is the right one to set in my job configuration, in order to force mappers and reducers to have a class path which puts my user defined hadoop_classpath jars BEFORE the hadoop default dependency jars ?
By the way, this is related to this question :
Dynamodb requestHandler acception which I recently have found is due to a jar conflict.
So, assuming you're using 0.20.203, this is handled in the TaskRunner.java code as follows:
The property you're looking for is on line 94 - mapreduce.user.classpath.first
Line 214 is where the call is made to build the list of classpaths, which delegates to a method called getClassPaths(..)
getClassPaths() is defined on line 524, and you should be able to see that the configuration property is used to decide on whether your job + dist cache libraries, or the hadoop libraries go on the classpath first
For other versions of hadoop, you're best to check the TaskRunner.java class to confirm the name of the config property after all this is a "semi hidden config":
static final String MAPREDUCE_USER_CLASSPATH_FIRST =
"mapreduce.user.classpath.first"; //a semi-hidden config
As in the latest Hadoop version (2.2+), you should set:
conf.setBoolean(MRJobConfig.MAPREDUCE_JOB_USER_CLASSPATH_FIRST, true);
These settings work for referencing classes of external jars only in your mapper or reducer tasks. If, however, you are using these in, for example a customized InputFormat, it will fail to load the class. A way to make sure this also works everywhere (in MR2) is exporting this setting when submitting your job:
export HADOOP_USER_CLASSPATH_FIRST=true
I had the same issue and the parameter that worked for me on Hadoop Version 0.20.2-cdhu03 is "mapreduce.task.classpath.user.precedence"
This setting is tested not work on CDH3U3, following answer is from Cloudera team:
// JobConf job = new JobConf(getConf(), MyJob.class);
// job.setUserClassesTakesPrecedence(true);
http://archive.cloudera.com/cdh/3/hadoop/api/org/apache/hadoop/mapred/JobConf.html#setUserClassesTakesPrecedence%28boolean%29
In the MapR distribution, the property is "mapreduce.task.classpath.user.precedence"
http://www.mapr.com/doc/display/MapR/mapred-site.xml
<property>
<name>mapreduce.task.classpath.user.precedence</name>
<value>true</value>
<description>Set to true if user wants to set different classpath. (AVRO) </description>
</property>
jobConf.setUserClassesTakesPrecedence(true);