I'm trying to implement a workflow where files from multiple SFTP sources get handled by the same integration flow.
Now, it works out alright in the beginning:
I can list the files in the SFTP source, using a different SFTP session factory for each SFTP source (SFTP_DEDUPED_FILES_0, SFTP_DEDUPED_FILES_1).
I can forward the message with the file name to a single channel (SFTP_FILES_CHANNEL).
I can process the information about the file (FILE_PROCESSING_FLOW).
Now, here's the catch: I want to delete the source file after processing.
For this, I need the right session factory again.
I already stored the SessionFactory in a message header - I just can't seem to extract it in order to create an outbound gateway for the delete call.
How can I implement deleteTheFile()?
#Bean(SFTP_DEDUPED_FILES_0)
public IntegrationFlow getDedupedFilesFromSftp(List<SftpConnectionInfo> connectionInfos) {
return createSftpGetFlow(connectionInfos.get(0));
}
#Bean(SFTP_DEDUPED_FILES_1)
public IntegrationFlow getDedupedFilesFromSftp1(List<SftpConnectionInfo> connectionInfos) {
return createSftpGetFlow(connectionInfos.get(1));
}
private IntegrationFlow createSftpGetFlow(SftpConnectionInfo connectionInfo) {
return IntegrationFlows
.from(PROVIDE_POLL_FREQUENCY)
.handle(listRemoteFiles(connectionInfo.sessionFactory(), connectionInfo.dedupingKeyPrefix(), connectionInfo.sftpRemotePath()))
.split()
.enrichHeaders(addFilePathMessageHeader(connectionInfo.sftpRemotePath()))
.enrichHeaders(addSessionFactoryMessageHeader(connectionInfo.sessionFactory()))
.channel(SFTP_FILES_CHANNEL)
.get();
}
#Bean(SFTP_FILES_CHANNEL)
public SubscribableChannel sftpFilesChannel() {
return new PublishSubscribeChannel();
}
#Bean(FILE_PROCESSING_FLOW)
public IntegrationFlow sftpGetFlow() {
return IntegrationFlows
.from(SFTP_FILES_CHANNEL)
.log(logTheFilePath())
.log(logTheMessageHeaders())
.handle(deleteTheFile())
.get();
}
private SftpOutboundGatewaySpec listRemoteFiles(SessionFactory<ChannelSftp.LsEntry> sessionFactory, String dedupingKeyPrefix, String sftpRemotePath) {
return Sftp.outboundGateway(sessionFactory, LS, asExpression(sftpRemotePath))
.options(RECURSIVE, NAME_ONLY)
.filter(onlyFilesWeHaveNotSeenYet(dedupingKeyPrefix))
.filter(onlyFiles());
}
What implementations of deleteTheFile() I tried:
This implementation seems to do something, but I don't want to create a new gateway every time, maybe there is a better solution.
private MessageHandler deleteTheFile() {
return message -> {
SessionFactory<ChannelSftp.LsEntry> sessionFactory = (SessionFactory<ChannelSftp.LsEntry>) message.getHeaders().get(CUSTOM_HEADER_SESSION_FACTORY);
SftpOutboundGatewaySpec gatewaySpec = Sftp.outboundGateway(sessionFactory, RM, "headers['" + CUSTOM_HEADER_REMOTE_FILE + "']");
gatewaySpec.get().handleMessage(message);
};
}
Plus, it's not a proper bean, so I guess that's why it fails and throws an exception:
java.lang.RuntimeException: No beanFactory
at org.springframework.integration.expression.ExpressionUtils.createStandardEvaluationContext(ExpressionUtils.java:90) ~[spring-integration-core-5.5.10.jar:5.5.10]
at org.springframework.integration.util.AbstractExpressionEvaluator.getEvaluationContext(AbstractExpressionEvaluator.java:111) ~[spring-integration-core-5.5.10.jar:5.5.10]
at org.springframework.integration.util.AbstractExpressionEvaluator.getEvaluationContext(AbstractExpressionEvaluator.java:97) ~[spring-integration-core-5.5.10.jar:5.5.10]
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:169) ~[spring-integration-core-5.5.10.jar:5.5.10]
at org.springframework.integration.util.AbstractExpressionEvaluator.evaluateExpression(AbstractExpressionEvaluator.java:127) ~[spring-integration-core-5.5.10.jar:5.5.10]
at org.springframework.integration.handler.ExpressionEvaluatingMessageProcessor.processMessage(ExpressionEvaluatingMessageProcessor.java:109) ~[spring-integration-core-5.5.10.jar:5.5.10]
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.obtainRemoteFilePath(AbstractRemoteFileOutboundGateway.java:760) ~[spring-integration-file-5.5.10.jar:5.5.10]
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.doRm(AbstractRemoteFileOutboundGateway.java:709) ~[spring-integration-file-5.5.10.jar:5.5.10]
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.handleRequestMessage(AbstractRemoteFileOutboundGateway.java:592) ~[spring-integration-file-5.5.10.jar:5.5.10]
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:136) ~[spring-integration-core-5.5.10.jar:5.5.10]
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:56) ~[spring-integration-core-5.5.10.jar:5.5.10]
at com.example.sftp.incoming.SftpMergedIncomingRecursiveConfiguration.lambda$deleteTheFile$4(SftpMergedIncomingRecursiveConfiguration.java:136) ~[classes/:na]
Related
Im trying to use olingo with Flutter on Android. I set up my channel and I can call the library but I keep getting this message:
E/AndroidRuntime(28391): FATAL EXCEPTION: main
E/AndroidRuntime(28391): Process: com.example.odata, PID: 28391
E/AndroidRuntime(28391): org.apache.olingo.client.api.http.HttpClientException: android.os.NetworkOnMainThreadException
E/AndroidRuntime(28391): at org.apache.olingo.client.core.communication.request.AbstractODataRequest.doExecute(AbstractODataRequest.java:312)
So it looks like it is running on the main thread - which is a no go as this would block. I tried the looper to ask Java to run on the UI Thread:
public void onMethodCall(MethodCall call, Result result) {
// Note: this method is invoked on the main thread.
Log.i("test", "using " + call.method);
String serviceUrl = "http://services.odata.org/OData/OData.svc/";
new Handler(Looper.getMainLooper()).post(new Runnable() {
#Override
public void run() {
if (call.method.equals("getMetaData")) {
String metadata;
final Edm edm = ODataClientFactory.getClient().getRetrieveRequestFactory().getMetadataRequest(serviceUrl).execute().getBody();
metadata = edm.toString();
if (metadata != "") {
result.success(metadata);
} else {
result.error("UNAVAILABLE", "Metadata cannot read.", null);
}
} else {
result.notImplemented();
}
}
});
But Im still getting the same error.
So how exactly can I deal with external JAR Library which are doing blocking operations ? To my understanding an external call is a Future anyway so it will not block my Flutter thread anyway - but Android Java does not think so ...
This is my method call in flutter
Future<void> _getMetaData() async {
String metadata;
try {
final String result = await platform.invokeMethod('getMetaData');
metadata = result;
} on PlatformException catch (e) {
metadata = e.message;
}
setState(() {
_metadata = metadata;
});
}
Thanks for the answer, this is the solution for anyone that may be interested:
public void onMethodCall(MethodCall call, Result result) {
if (call.method.equals("getMetaData")) {
class MetadataLoader extends AsyncTask<String , Integer, String> {
#Override
protected String doInBackground(String... urls) {
// call your Java library method here, including blocking methods
return your_return_value;
}
protected void onPostExecute(String _result) {
// your_return_value is now passed in _result
result.success(_result);
}
}
new MetadataLoader().execute(); // Start the Async
}
On the flutter side,
Future<void> _getMetaData() async {
String metadata;
try {
final String result = await platform.invokeMethod('getMetaData');
// do something with the result
// the Flutter thread will stop at the await and resume when the Java
// will call result.success
}
}
You will need to create a new Java thread or Worker. (Note that the "main" thread and the "UI" thread are the same thing - so by posting to the main looper you've ended up in the same place - trying to do network i/o on the main thread.)
Yes, the Flutter engine is running in different threads, but you still need to leave the main native thread unblocked as it is responsible for detecting user input, etc.
Also note that when your blocking activity completes - on its non-main thread - it will likely want to deliver the response to Dart. To do this it will need to use part of your code above - to post the results back to the main thread, which can then invoke method channel operations.
You'll probably want to use your method channel bi-directionally. From flutter to native to request an operation (returning, say, a sequence number), and from native to flutter to deliver the results (quoting the sequence number so that the result can be tied back to the request).
I'm implementing an enricher pattern (https://www.enterpriseintegrationpatterns.com/patterns/messaging/DataEnricher.html) using a command/consumer queue where the consumer is the enricher and publishes the enriched message to a separate endpoint (SQL database in this case). The consumer is running as a HostedService which implements cancellation token.
Because I'm consuming commands from one transport and publishing events to another there is a possibility that the transport I'm publishing to is down while the one I'm consuming from is up. In that case I'd like to log an error and stop my Hosted service. However, I cannot see how that would work since whatever calls the Handle method already handles exceptions, and I cannot access my cancellation token. Does anyone have any ideas?
This is a draft of what I want to do.
public async Task Handle(EditedEventData message)
{
var enricher = _enricherFactory.GetEnricher(message);
object #event = await enricher.EnrichAsync(message);
var transformers = _transformerFactory.GetTransformers(message);
var messages = new List<object>();
foreach (var transformer in transformers)
{
messages.AddRange(transformer.Transform(#event, message));
}
foreach (var item in messages)
{
try
{
await _bus.Publish(item);
}
catch (Exception ex)
{
_logger.LogCritical("Publishing event message {#item} failed with error {ex}", item, ex);
//how do I exit from here?
}
}
}
If I were you, I would come up with some kind of application service, e.g. IApplicationControlService, which you can configure to be injected into your handlers using whichever IoC container you're using.
It could look somewhat like this:
public interface IApplicationControlService
{
void RequestApplicationShutdown();
}
and then your code could simply
public class YourHandler : IHandleMessages<EditedEventData>
{
readonly IApplicationControlService applicationControlService;
public YourHandler(IApplicationControlService applicationControlService)
{
this.applicationControlService = applicationControlService;
}
public async Task Handle(EditedEventData message)
{
// (...)
foreach (var item in messages)
{
try
{
await _bus.Publish(item);
}
catch (Exception ex)
{
_logger.LogCritical("Publishing event message {#item} failed with error {ex}", item, ex);
applicationControlService.RequestApplicationShutdown();
}
}
}
}
to request the application be stopped, when an error occurs.
An implementation of IApplicationControlService could then be something like
public class BruteForceApplicationControlService : IApplicationControlService
{
public void RequestApplicationShutdown()
{
Environment.FailFast("you should probably not do THIS 😉");
}
}
or something more gentle 😁 – the point is, that you will be able to provide a way to request your application to shut down "from the outside", most likely from the place where your application is assembled (i.e. the "composition root")
I have a very basic http-POST triggered api which creates a TelemetryClient. I needed to provide a custom property in this telemetry for each individual request, so I implemented a TelemtryProcessor.
However, when subsequent POST requests are handled and a new TelemetryClient is created that seems to interfere with the first request. I end up seeing maybe a dozen or so entries in App Insights containing the first customPropertyId, and close to 500 for the second, when in reality the number should be split evenly. It seems as though the creation of the 2nd TelemetryClient somehow interferes with the first.
Basic code is below, if anyone has any insight (no pun intended) as to why this might occur, I would greatly appreciate it.
ApiController which handles the POST request:
public class TestApiController : ApiController
{
public HttpResponseMessage Post([FromBody]RequestInput request)
{
try
{
Task.Run(() => ProcessRequest(request));
return Request.CreateResponse(HttpStatusCode.OK);
}
catch (Exception)
{
return Request.CreateErrorResponse(HttpStatusCode.InternalServerError, Constants.GenericErrorMessage);
}
}
private async void ProcessRequest(RequestInput request)
{
string customPropertyId = request.customPropertyId;
//trace handler creates the TelemetryClient for custom property
CustomTelemetryProcessor handler = new CustomTelemetryProcessor(customPropertyId);
//etc.....
}
}
CustomTelemetryProcessor which creates the TelemetryClient:
public class CustomTelemetryProcessor
{
private readonly string _customPropertyId;
private readonly TelemetryClient _telemetryClient;
public CustomTelemetryProcessor(string customPropertyId)
{
_customPropertyId = customPropertyId;
var builder = TelemetryConfiguration.Active.TelemetryProcessorChainBuilder;
builder.Use((next) => new TelemetryProcessor(next, _customPropertyId));
builder.Build();
_telemetryClient = new TelemetryClient();
}
}
TelemetryProcessor:
public class TelemetryProcessor : ITelemetryProcessor
{
private string CustomPropertyId { get; }
private ITelemetryProcessor Next { get; set; }
// Link processors to each other in a chain.
public TelemetryProcessor(ITelemetryProcessor next, string customPropertyId)
{
CustomPropertyId = customPropertyId;
Next = next;
}
public void Process(ITelemetry item)
{
if (!item.Context.Properties.ContainsKey("CustomPropertyId"))
{
item.Context.Properties.Add("CustomPropertyId", CustomPropertyId);
}
else
{
item.Context.Properties["CustomPropertyId"] = CustomPropertyId;
}
Next.Process(item);
}
}
It's better to avoid creating Telemetry Client per each request, isntead re-use single static Telemetry Client instance. Telemetry Processors and/or Telemetry Initializers should also typically be registered only once for the telemetry pipeline and not for every request. TelemetryConfiguration.Active is static and by adding new Processor with each request the queue of processor only grows.
The appropriate setup would be to add Telemetry Initializer (Telemetry Processors are typically used for filtering and Initializers for data enrichment) once into the telemetry pipeline, e.g. though adding an entry to ApplicationInsights.config file (if present) or via code on TelemetryConfiguration.Active somewhere in global.asax, e.g. Application_Start:
TelemetryConfiguration.Active.TelemetryInitializers.Add(new MyTelemetryInitializer());
Initializers are executed in the same context/thread where Track..(..) was called / telemetry was created, so they will have access to the thread local storage and or local objects to read parameters/values from.
We are using spring and spring-security-3.2. Recently We are adding annotations #PreAuthorize to RestAPIs(earlier it was URL based).
#PreAuthorize("hasPermission('salesorder','ViewSalesOrder')")
#RequestMapping(value = "/restapi/salesorders/", method = RequestMethod.GET)
public ModelAndView getSalesOrders(){}
We already have Global exception handler which annotated with - #ControllerAdvice and custom PermissionEvaluator in place, everything works fine except the error message.
Lets say some user is accessing API At moment without having 'ViewSalesOrder' permission then spring by default throws the exception 'Access is denied',but didn't tell which permission is missing (Its our requirement to mention which permission is missing).
Is it possible to throw an exception which also include the permission name, so final error message should be look like "Access is denied, you need ViewSalesOrder permission"(here permission name should be from #PreAuthorize annotation)?
Please note that we have 100 such restAPI in place so generic solution will be highly appreciated.
There is no pretty way of achieving what you expect since PermissionEvaluator interface doesn't let you pass the missing permission along with the result of the evaluation.
In addition, AccessDecisionManager decides on the final authorization with respect to the votes of the AccessDecisionVoter instances, one of which is PreInvocationAuthorizationAdviceVoter which votes with respect to the evaluation of #PreAuthorize value.
Long story short, PreInvocationAuthorizationAdviceVoter votes against the request (giving the request –1 point) when your custom PermissionEvaluator returns false to hasPermission call. As you see there is no way to propagate the cause of the failure in this flow.
On the other hand, you may try some workarounds to achieve what you want. One way can be to throw an exception within your custom PermissionEvaluator when permission check fails. You can use this exception to propagate the missing permission to your global exception handler. There, you can pass the missing permission to your message descriptors as a parameter. Beware that this will halt execution process of AccessDecisionManager which means successive voters will not be executed (defaults are RoleVoter and AuthenticatedVoter). You should be careful if you choose to go down this path.
Another safer but clumsier way can be to implement a custom AccessDeniedHandler and customize the error message before responding with 403. AccessDeniedHandler provides you current HttpServletRequest which can be used to retrieve the request URI. However, bad news in this case is, you need a URI to permission mapping in order to locate the missing permission.
I have implemented the second possible solution mentioned by Mert Z. My solution works only for #PreAuthorize annotations used in the API layer (e.g. with #RequestMapping). I have registered a custom AccessDeniedHandler bean in which I get the value of the #PreAuthorize annotation of the forbidden API method and fills it into error message.
public class CustomAccessDeniedHandler implements AccessDeniedHandler {
private DispatcherServlet dispatcherServlet;
public void handle(HttpServletRequest request, HttpServletResponse response,
AccessDeniedException accessDeniedException) throws IOException,
ServletException {
if (!response.isCommitted()) {
List<HandlerMapping> handlerMappings = dispatcherServlet.getHandlerMappings();
if (handlerMappings != null) {
HandlerExecutionChain handler = null;
for (HandlerMapping handlerMapping : handlerMappings) {
try {
handler = handlerMapping.getHandler(request);
} catch (Exception e) {}
if (handler != null)
break;
}
if (handler != null && handler.getHandler() instanceof HandlerMethod) {
HandlerMethod method = (HandlerMethod) handler.getHandler();
PreAuthorize methodAnnotation = method.getMethodAnnotation(PreAuthorize.class);
if (methodAnnotation != null) {
response.sendError(HttpStatus.FORBIDDEN.value(),
"Authorization condition not met: " + methodAnnotation.value());
return;
}
}
}
response.sendError(HttpStatus.FORBIDDEN.value(),
HttpStatus.FORBIDDEN.getReasonPhrase());
}
}
#Inject
public void setDispatcherServlet(DispatcherServlet dispatcherServlet) {
this.dispatcherServlet = dispatcherServlet;
}
}
The handler is registered in WebSecurityConfigurerAdapter:
#EnableGlobalMethodSecurity(jsr250Enabled = true, prePostEnabled = true)
#EnableWebSecurity
public abstract class BaseSecurityInitializer extends WebSecurityConfigurerAdapter {
#Override
protected void configure(HttpSecurity http) throws Exception {
...
http.exceptionHandling().accessDeniedHandler(accessDeniedHandler());
...
}
#Bean
public AccessDeniedHandler accessDeniedHandler() {
return new CustomAccessDeniedHandler();
}
}
Beware that if there is also a global resource exception handler with #ControllerAdvice the CustomAccessDeniedHandler won't be executed. I solved this by rethrowing the exception in the global handler (as advised here https://github.com/spring-projects/spring-security/issues/6908):
#ControllerAdvice
public class ResourceExceptionHandler {
#ExceptionHandler(AccessDeniedException.class)
public ResponseEntity accessDeniedException(AccessDeniedException e) throws AccessDeniedException {
log.info(e.toString());
throw e;
}
}
You can throw an org.springframework.security.access.AccessDeniedException from a method that was called inside an EL-Expression:
#PreAuthorize("#myBean.myMethod(#myRequestParameter)")
Ideally, the #PreAuthorize annotation should be supporting String message(); in addition to the SpEl value. But, for whatever reason, it does not. Most of the suggestions here seem unnecessarily cumbersome and elaborate. As #lathspell has suggested, the simplest way to provide your own error message - along with any custom access validation logic - would be to add a simple method that performs the check and throws the AccessDeniedException in case the check fails, and then reference that method in the SpEl expression. Here's an example:
#RestController
#RequiredArgsConstructor // if you use lombok
public class OrderController {
private final OrderService orderService;
...
#GetMapping(value = "/salesorders", produces = MediaType.APPLICATION_JSON_VALUE)
#PreAuthorize("#orderController.hasPermissionToSeeOrders(#someArgOfThisMethod)")
public Page<OrderDto> getSalesOrders(
// someArgOfThisMethod here, perhaps HttpRequest, #PathVariable, #RequestParam, etc.
int pageIndex, int pageSize, String sortBy, String sortOrder) {
Pageable pageRequest = PageRequest.of(pageIndex, pageSize, Sort.Direction.fromString(sortOrder), sortBy);
return ordersService.retrieveSalesOrders(..., pageRequest);
}
public static Boolean hasPermissionToSeeOrders(SomeArgOfTheTargetMethod argToEvaluate) {
//check eligibility to perform the operation based on some data from the incoming objects (argToEvaluate)
if (condition fails) {
throw new AccessDeniedException("Your message");
}
return true;
}
I'm starting a web application that contains the following projects:
Booking.Web
Booking.Services
Booking.DataObjects
Booking.Data
I'm using the repository pattern in my data project only. All services will be the same, no matter what happens. However, if a customer wants to use Access, it will use a different data repository than if the customer wants to use SQL Server.
I have StructureMap, and want to be able to do the following:
Web project is unaffected. It's a web forms application that will only know about the services project and the dataobjects project.
When a service is called, it will use StructureMap (by looking up the bootstrapper.cs file) to see which data repository to use.
An example of a services class is the error logging class:
public class ErrorLog : IErrorLog
{
ILogging logger;
public ErrorLog()
{
}
public ErrorLog(ILogging logger)
{
this.logger = logger;
}
public void AddToLog(string errorMessage)
{
try
{
AddToDatabaseLog(errorMessage);
}
catch (Exception ex)
{
AddToFileLog(ex.Message);
}
finally
{
AddToFileLog(errorMessage);
}
}
private void AddToDatabaseLog(string errorMessage)
{
ErrorObject error =
new ErrorObject
{
ErrorDateTime = DateTime.Now,
ErrorMessage = errorMessage
};
logger.Insert(error);
}
private void AddToFileLog(string errorMessage)
{
// TODO: Take this value from the web.config instead of hard coding it
TextWriter writer = new StreamWriter(#"E:\Work\Booking\Booking\Booking.Web\Logs\ErrorLog.txt", true);
writer.WriteLine(DateTime.Now.ToString() + " ---------- " + errorMessage);
writer.Close();
}
}
I want to be able to call this service from my web project, without defining which repository to use for the data access. My boostrapper.cs file in the services project is defined as:
public class Bootstrapper
{
public static void ConfigureStructureMap()
{
ObjectFactory.Initialize(x =>
{
x.AddRegistry(new ServiceRegistry());
}
);
}
public class ServiceRegistry : Registry
{
protected override void configure()
{
ForRequestedType<IErrorLog>().TheDefaultIsConcreteType<Booking.Services.Logging.ErrorLog>();
ForRequestedType<ILogging>().TheDefaultIsConcreteType<SqlServerLoggingProvider>();
}
}
}
What else do I need to get this to work? When I defined a test, the ILogger object was null.
Perhaps some details on how you are calling this code from a test would be useful.
My understanding is that you need to ensure that the ConfigureStructureMap call has been made early in the applications life (e.g. in the Global.asax in a web project).
After that you would be calling for instances of IErrorLog using something like:
IErrorLog log = StructureMap.ObjectFactory.GetNamedInstance<IErrorLog>();