Manually create spring cloud stream bindings based on dynamic configuration - spring-kafka

I have a requirement where one or more spring cloud stream kafka-streams bindings need to be created based on dynamic configuration. By dynamic config I mean stream bindings (input-output) will be specified run-time. Either via external property file or from database.
E.g We need to create multiple stream processors where input output topic pairs and relevant configs are provided. Then code should loop through this config create and start those bindings.
In spring cloud stream we write this in a java file
#StreamListener(StreamBindings.INPUT)
#SendTo(StreamBindings.OUTPUT)
public KStream<String,String> process(KStream<String,String> inputStream) {
return inputStream
.map( ... )
.selectKey( ... )
.mapValues( ... );
}
Where StreamBindings is like
public interface StreamBindings {
String INPUT = "input-topic";
String OUTPUT = "output-topic";
#Input(INPUT)
KStream<String,String> inputStream();
#Input(OUTPUT)
KStream<String,String> outputStream();
}
Now I want a piece of code to create this in run-time based on info I specified above.
Can this be done and how? And can we specify body of process function as an argument, like some kind of message handler?

Related

CamelCase property names with NJsonSchema C# CodeGeneration

does anybody know a way to configure NJsonSchema to use CamelCase property naming durching code generation? I've a JSON schema with property names like message_id which lead to C# property name 'Message_id' where i.e. 'MessageId' whould be a more C#-like way.
With an attribute like '[JsonProperty("message_id"]' it would be no problem to specified the connection between the different names.
So, you asked about code generation. I was having trouble with the schema it generated not matching what was getting sent to my Angular app. So, while this isn't exactly what you were looking for, perhaps it helps you find an answer (maybe?).
To generate the schema with the camel case property names, I'm setting the Default Property Name Handling to CamelCase, but this is using the deprecated call to set these settings directly. There should be some way to use the SerializerSettings directly, but I wasn't quite able to make that work. This isn't production code for me, so it will do.
internal class SchemaFileBuilder<T>
{
public static void CreateSchemaFile()
{
CreateSchemaFile(typeof(T).Name);
}
public static void CreateSchemaFile(string fileName)
{
JsonSchemaGeneratorSettings settings = new JsonSchemaGeneratorSettings();
settings.DefaultPropertyNameHandling = PropertyNameHandling.CamelCase;
var schema = NJsonSchema.JsonSchema.FromType<T>(settings);
var json = schema.ToJson();
Directory.CreateDirectory("Schemas");
File.WriteAllText($"Schemas\\{fileName}.schema.json", json);
}
}
I set this up as a generic function so I could pass multiple schemas in to either createSchemaFile functions. Here's are some example calls which would generate a Person.schema.json file and a Persons.schema.json file:
SchemaFileBuilder<Person>.CreateSchemaFile();
SchemaFileBuilder<Dictionary<string, Person>>.CreateSchemaFile("Persons");

Modifying swagger annotations at runtime

I have a Swagger API documentation. Some of the methods I want to hide dynamically, depending on user permissions for reading/ accessing the API.
Since I couldn't come up with a Swagger functionality on this, I have implemented a ContainerRequestFilter on my service, which is executed when getting the openapi.json document for the Swagger UI. This filter is scanning all methods containing the #Operation annotation and changing the field hidden from true to false for users with the correct permission.
import io.swagger.v3.oas.annotations.Operation;
#Provider
#Priority(Priorities.USER)
public class MyFilter implements ContainerRequestFilter{
public void filter(ContainerRequestContext crc) throws IOException{
Reflections reflections = new Reflections(new ConfigurationBuilder()
.setUrls(ClasspathHelper.forPackage("my.package"))
.setScanners(new MethodAnnotationsScanner()));
// get all Operation methods
Set<Method> resources = reflections.getMethodsAnnotatedWith(Operation.class);
for(Method method : resources) {
method.setAccessible(true);
Operation operation = method.getAnnotation(Operation.class);
InvocationHandler h = Proxy.getInvocationHandler(operation);
Field hField = h.getClass().getDeclaredField("memberValues");
hField.setAccessible(true);
Map memberValues = (Map) hField.get(h);
for(Object key : memberValues.keySet()) {
System.out.println("memberValue: "+key+", "+memberValues.get(key));
}
memberValues.put("hidden", false);
System.out.println("operation is hidden?: " +operation.hidden());
}
}
}
The code seems to work insofar as the hidden field value is changed from true to false inside this filter class. However, the openapi.json document returned from the same service request does not reflect those changes.
Why are the changes on the modified annotations not consistent within the executed request?
This approach does not seem to be viable. Reflection changes the annotations globally, so other requests would read the same changes annotations which is not what I wanted here.
I ended up going with the filtering of annoations as described here.

Alfresco CMIS: How to translate a ChangeEvent object id to a Document object id?

Context
In CMIS, a ChangeEvent tells me that an object has been deleted on the remote server, and that the objectId of this object is for instance workspace://SpacesStore/3aca9034-3f53-4946-a0d9-ebcf054912a2;1.0 (real example of ChangeEvent.Properties[cmis:objectId]).
I locally kept the various identifiers of all objects, but none of the identifiers match exactly the identifier in the ChangeEvent:
Document.Properties[cmis:objectId] is
3aca9034-3f53-4946-a0d9-ebcf054912a2;1.0
Document.Properties[alfcmis:nodeRef] is
workspace://SpacesStore/3aca9034-3f53-4946-a0d9-ebcf054912a2.
Question
How should I translate from ChangeEvent.Properties[cmis:objectId] to Document.Properties[cmis:objectId]?
(eg. from workspace://SpacesStore/3aca9034-3f53-4946-a0d9-ebcf054912a2;1.0 to 3aca9034-3f53-4946-a0d9-ebcf054912a2;1.0
I could just remove the first 24 characters, but I guess it would only work for Alfresco, and not with other CMIS servers.
Environment
Alfresco Community 4.2.e accessed by DotCMIS via
http://server/alfresco/api/-default-/public/cmis/versions/1.0/atom
Need to assign from DotCMIS from Alfresco apache chemeistry support please go through below code for access object id from file name using c#..
IFolder selectedFolder1 = session1.GetObject(ObjectID) as IFolder;
foreach (ICmisObject cmisObject in selectedFolder1.GetChildren())
{
if (cmisObject.GetType() != typeof(DotCMIS.Client.Impl.Folder))
{
if (cmisObject.Name.Contains(Uid))
{
return false;
}
}
}

returning a JSON formatted file via WCF

We've seen a number of posts relating to JSON data returns via WCF, but they all cover the aspect of converting object to JSON and then returning that object converted to JSON via the magic of attributes.
We've got a number of preformatted JSON files that we want to return via an WCF service. Essentially all we need to do is read the files in (or a cached copy of of the file) and then return the data as a string . I think ... It seems wasteful to read in the JSON file, serialize it to an object then deserialize back to JSON.. Any help on this?
When using the WebHttpBinding, this is as simple as creating a WebGet annotated method with a Stream return type:
[WebGet]
public Stream GetFile(Int32 someId)
{
//your logic to lookup or create the file here.
//Open the file (a MemoryStream would be acceptible as well if you were creating the data on the fly
Stream stream = File.OpenRead(yourFilePath);
//register an event to clean up the temporary file (if necessary)
OperationContext.Current.OperationCompleted += (s, e) =>
{
File.Delete(yourFilePath);
};
return stream;
}

Referencing a port in an orchestration via a string variable

I am attempting to develop a generic BizTalk application for configuring dynamic ports. I have an orchestration that pulls back all the configuration settings for each port and I want to loop through these settings and configure the ports. The settings are held in MSSQL and, for instance, two of the properties are PortName and Address. So from within the orchestration I would like to reference the port by the string variable PortName. So is there some way to get a collection of all the ports in an orchestration or reference a port via a string variable i.e. Port['MyPortName'](Microsoft.XLANGs.BaseTypes.Address) = "file://c:\test\out\%MessageId%.xml" Thanks
In order to dynamically configure Dynamic Logical Send Ports from within an orchestration, one has to store the settings into a persistent datastore (e.g. a database or configuration file) and implement a way to assign those properties dynamically at runtime.
But first, we need to understand what is happening when configurating a Dynamic Send Port.
How to Configure a Dynamic Logical Send Port
Configuring the properties of a dynamic logical send port from within an orchestration involves two steps:
First, the TransportType and target Address properties must be specified on the Send Port. This is usually done in an Expression Shape with code similar to this:
DynamicSendPort(Microsoft.XLANGs.BaseTypes.TransportType) = "FILE";
DynamicSendPort(Microsoft.XLANGs.BaseTypes.Address) = "C:\Temp\Folder\%SourceFileName%";
Second, any additional transport properties must be specified on the context of the outgoing message itself. Virtually all BizTalk adapters have additional properties that are used for the communication between the Messaging Engine and the XLANG/s Orchestration Engine. For instance, the ReceivedFileName context property is used to dynamically set a specific name for when the FILE adapter will save the outgoing message at its target location. This is best performed inside an Assignment Shape, as part of constructing the outgoing message:
OutgoingMessage(FILE.ReceiveFileName) = "HardCodedFileName.xml"
You'll notice that most configuration properties must be specified on the context of the outgoing messages, specifying a namespace prefix (e.g. FILE), a property name (e.g. ReceiveFileName) and, obviously, the value that gets assigned to the corresponding property.
In fact, all the context properties are classes that live Inside the well-known Microsoft.BizTalk.GlobalPropertySchemas.dll assembly. This is confirmed by looking up this assembly in Visual Studio's object explorer.
Even though most context properties that are necessary to configure Dynamic Logical Send Ports live Inside this specific assembly, not all of them do. For instance, the MSMQ BizTalk adapter uses a separate assembly to store its context properties. Obviously, third-party or custom adapters come with additionnal assemblies as well.
Therefore, in order to setup a context property on a Dynamic Send Port using a flexible approach like the one describe below, four pieces of information are necessary:
The fully qualified name of the assembly containing the context property classes.
The namespace prefix.
The property name.
The property value.
Storing Port Settings in a Persistent Medium
The following .XSD schema illustrate one possible structure for serializing port settings.
Once serialized, the specified context properties can then be stored in a SQL database or a configuration file very easily. For instance, here are the settings used as an example in this post:
A Flexible Approach to Configuring Dynamic Logical Send Ports
With a simple helper Library, setting up the dynamic port configuration is very easy. First, you have to retrieve the serialized settings from the persistent medium. This can easily be achieved using the WCF-SQL Adapter and a simple stored procedure.
Once retrieved, those properties can then be deserialized into a strongly-typed C# object graph. For this, first create a C# representation of the ContextProperties schema shown above, using the following command-line utility:
xsd.exe /classes /language:cs /namespace:Helper.Schemas .\ContextProperties.xsd
This generates a partial class that can be improved with the following method:
namespace Helper.Schemas
{
public partial class ContextProperties
{
public static ContextProperties Deserialize(string text)
{
using (MemoryStream stream = new MemoryStream())
{
byte[] buffer = Encoding.UTF8.GetBytes(text);
stream.Write(buffer, 0, buffer.Length);
stream.Seek(0, SeekOrigin.Begin);
return (ContextProperties)
Deserialize(
stream
, typeof(ContextProperties));
}
}
public static Object Deserialize(Stream stream, Type type)
{
XmlSerializer xmlSerializer = new XmlSerializer(type);
return xmlSerializer.Deserialize(stream);
}
}
}
Second, applying this configuration involves creating an XLANG/s message from code and setting up the context properties dynamically using reflection, based upon the description of the context property classes specified in the deserialized ContextProperties object graph.
For this, I use a technique borrowed from Paolo Salvatori's series of articles regarding dynamic transformations, which consists in creating a custom BTXMessage-derived class, used internally by the BizTalk XLANG/s engine.
namespace Helper.Schemas
{
using Microsoft.BizTalk.XLANGs.BTXEngine; // Found in Microsoft.XLANGs.BizTalk.Engine
using Microsoft.XLANGs.Core; // Found in Microsoft.XLANGs.Engine
[Serializable]
public sealed class CustomBTXMessage : BTXMessage
{
public CustomBTXMessage(string messageName, Context context)
: base(messageName, context)
{
context.RefMessage(this);
}
public void SetContextProperty(string assembly, string ns, string name, object value)
{
if (String.IsNullOrEmpty(ns))
ns = "Microsoft.XLANGs.BaseTypes";
if (String.IsNullOrEmpty(assembly))
assembly = "Microsoft.BizTalk.GlobalPropertySchemas";
StringBuilder assemblyQualifiedName = new StringBuilder();
assemblyQualifiedName.AppendFormat("{0}.{1}, {2}", ns, name, assembly);
Type type = Type.GetType(assemblyQualifiedName.ToString(), true, true);
SetContextProperty(type, value);
}
internal void SetContextProperty(string property, object value)
{
int index = property.IndexOf('.');
if (index != -1)
SetContextProperty(String.Empty, property.Substring(0, index), property.Substring(index + 1), value);
else
SetContextProperty(String.Empty, String.Empty, property, value);
}
}
}
Now, the last piece of the puzzle is how to make use of this custom class from within an Orchestration. This is easily done in an Assignment Shape using the following helper code:
namespace Helper.Schemas
{
using Microsoft.XLANGs.BaseTypes;
using Microsoft.XLANGs.Core; // Found in Microsoft.XLANGs.Engine
public static class Message
{
public static XLANGMessage SetContext(XLANGMessage message, ContextProperties properties)
{
try
{
// create a new XLANGMessage
CustomBTXMessage customBTXMessage = new CustomBTXMessage(message.Name, Service.RootService.XlangStore.OwningContext);
// add parts of the original message to it
for (int index = 0; index < message.Count; index++)
customBTXMessage.AddPart(message[index]);
// set the specified context properties
foreach (ContextPropertiesContextProperty property in properties.ContextProperty)
customBTXMessage.SetContextProperty(property.assembly, property.#namespace, property.name, property.Value);
return customBTXMessage.GetMessageWrapperForUserCode();
}
finally
{
message.Dispose();
}
}
}
}
You can use this static method inside your Assignment Shape like the code shown hereafter, where OutboundMessage represents the message which you want to set the context:
OutboundMessage = Helper.Schemas.Message.SetContext(OutboundMessage, contextProperties);
In the first place you shouldn't attempt to do configuration changes like this using an Orchestration. Technically it's feasible to do what you are attempting to do, but as a practice you shouldn't mix up your business process with administration.
The best way to do such things will be by either writing some normal scripts or PowerShell.
To answer you question, you can get the data you want from BtsOrchestration class in ExplorerOM
http://msdn.microsoft.com/en-us/library/microsoft.biztalk.explorerom.btsorchestration_members(v=bts.20)

Resources