I'am using the inbound-channel-adapter from Spring Integration to retrieve files over sftp from a remote server. Everything works fine.
But I have an additional requirement: after a file is received on the local side, that file needs to moved to a "send" directory on the remote server.
The "SFTP Outbound Gateway" has the appropriate method for that move action, but my problem is when to call it.
Situation: 10 files on remote server, 0 on local server
When I start my application it will receive all 10 files from the remote server and write them to my local file system. Perfect.
Situation: 1 file on remote server, 10 on local server
In this situation the remote file is received, but for every file on the local file system the receive method of the QueueChannel is also called.
Example log from one file: (file1.zip)
18:12:52.118 [task-scheduler-1] INFO o.s.i.file.FileReadingMessageSource - Created message: [[Payload File content=C:\Downloads\sftpTest\file1.zip][Headers=...]
18:12:52.119 [task-scheduler-1] DEBUG o.s.i.e.SourcePollingChannelAdapter - Poll resulted in Message: [Payload File content=C:\Downloads\sftpTest\file1.zip][Headers=...]
18:12:52.119 [task-scheduler-1] DEBUG o.s.integration.channel.QueueChannel - preSend on channel 'fromChannel', message: [Payload File content=C:\Downloads\sftpTest\file1.zip][...]
18:12:52.119 [task-scheduler-1] DEBUG o.s.integration.channel.QueueChannel - postSend (sent=true) on channel 'fromChannel', message: [Payload File content=C:\Downloads\sftpTest\file1.zip][Headers=...]
18:12:52.119 [main] DEBUG o.s.integration.channel.QueueChannel - postReceive on channel 'fromChannel', message: [Payload File content=C:\Downloads\sftpTest\file1.zip][Headers=......]
So even when the file it not physicaly retreived from the remote server, the channel.receive() method will still receive a message with that file as payload.
This confuses me, because I can't determine from the message if the file was already on the local file system or was just retrieved from the remote server.
I experimented using a custom org.springframework.messaging.support.ChannelInterceptorAdapter, FileFilter, ServiceActivator, but the problem still remains.
My application will process high volumes, so sending the received file to the required directory on the remote server is not an option. And also simply trying to move the file remote for every message that is locally received, is not an option since it will clutter the logfiles with exceptions of not being able to move the file. This way in case of an real error situation the problem will not be detected.
One solution might be a hook in the method copyFileToLocalDirectory of the org.springframework.integration.file.remote.synchronizer.AbstractInboundFileSynchronizer.
There is a check performed if the remote file should be deleted and that loop is only called for the files that were actually transferred from the remote server. My attempts to override this method and add my move behaviour did not succeed, since Spring has already instantiated the classes
that will handle this.
So what is the best way to achieve this? I know the problem will probably be located between my keyboard and my chair, but I've run out of options and any help is highly appreciated.
Thanks a lot,
Frank
You would probably be better off using MGET and an outbound gateway to retrieve the files instead of using the inbound adapter which, as you say, is two-stage - synchronize, and emit message(s) for file(s) in the local dir (unless you use a persistent file list filter, in which case you'll only see "new" files).
Related
I am building an integration that should fetch a zip-file from SFTP and save it as XML on disk. I have got everything working except the unzip-part...
The integration looks like this.
Receive location polls the SFTP, and in the receive port "Inbound maps" I have provided a mapping schema (converting the txt-file in the zip file to XML).
Receive location: polls the SFTP, custom receive pipeline.
Custom pipeline: In Disassemble first the unzip-component from BizTalk Utility pack ( BizTalk Server 2020 – 20 days, 20 posts: Unzip File Pipeline Component for BizTalk Server 2020) and then the flat file disassembler. The document-schema here matches the source-schema in the mapping.
Send port: Save file on disk.
The problem is that when I run the integration, the file is picked up from the SFTP and the file is indeed unzipped but the send port saves the content from the original file to the send-location. It is as if the mapping is never executed.
But if I remove the unzip-component in the custom pipeline and manually unzip the file and upload the content (.txt-file) to the SFTP everything works fine.
As per Sandro Pereira's instruction no configuration is needed, so I don't really understand why it is not working.
Receive Pipelines - Disassemble stage (Microsoft.com)
If this stage contains more than one component, only the first component that recognizes the message format is run. If none of the components within the stage recognize the message format, the message processing fails.
So you can't have both in the disassemble stage of the pipeline and have them both execute. Move the Unzip to the Decode stage of the pipeline.
I'm have local access to a shared folder with credentials.
net use Z: \\server\folder /u:user /persistent:yes passwd
The files in this folder are to be used by other processes.
The problem is that when the machine is restarted, the connection is lost, meaning that I can see that the Network Device Z is there, but it does not automatically connect to it, so that the processes have no longer access to the folder.
I have tried several solutions with no avail:
Creating a Windows scheduled task to run a script with 'net use Z:...',
a. with my_user I get this error in Event Viewer: "The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID"
b. with System, I get no errors but the connection is not created.
I've tried to log stdout and stderr to debug the script, but it does not even
create the log file. (The used script is shown below)
#echo off
'> map.log 2>&1(
net use Z: \server\folder /u:user /persistent:yes passwd
)
Crating a Symlink was another solution I had found, but it is not suitable for my case,
because it does not allow credentials. It seems that the resource has to be either
unprotected or already available locally.
Creating a Startup Script from Local Group Policy Editor, I get the same error in
Event Viewer
Running 'net use Z:...' via a Python script installed ad Windows Service,
import subprocess
proc=subprocess.Popen(r'net use Z: \server\folder /u:user pass', shell=True)
out, err = proc.communicate()
print("out:%s, err:%s"%(out,err))
If I run the python script manually, it works, while running it as a service creates a
disconnected drive.
Can anyone give me an advice on how to proceed?
Thanks
When I load my docker shiny app domain name in the browser, it crashes (greys out) and I get this "ERROR: [_parse_http_data] invalid HTTP method".
I have developed an web application that consists of a shiny app (has a login feature connected to an RMySQL database), a website and a mariadb database. I put them together in a docker-compose file and tested it on my local computer and it works fine. I then proceeded to deploy them in a Kubernetes cluster in GCE and that was also successful. I used cloudflare to install a ssl certificate for the shiny app domain (i.e. trnddaapp.com). Now when I load the shiny app domain in the browser it appends the https and loads the app successfully but after about a minute it crashes (greys out). I loaded the shiny app external ip with http and this doesn’t crash.
The closest solution I have come to is https://github.com/rstudio/shiny-server/issues/392 but there doesn't seem to be any other solution to my problem. I would be grateful if anyone help me resolve this problem.
This is the error message I get when I check with kubectl log [app pod name], I get this error:
ERROR: [_parse_http_data] invalid HTTP method
ERROR: [_parse_http_data] invalid HTTP method
ERROR: [_parse_http_data] invalid HTTP method
I expect the app not to crash when the shiny app domain (trnddaapp.com) is appended with the https.
Let's start with the analysis of the error message, it says:
[_parse_http_data]
So we know that your app is receiving something, but it doesn't understand what it is (it may be a malformed HTTP/1.0 or HTTP/1.1 or even binary data) then we have an
invalid HTTP method
Now we are sure it is not a HTTP/1.X call but a stream of (non recognized) data.
We now know is not the instance since it "deploys" and "delivers" the service, but something inside that is just breaking.
There are a few things that may be happening, since it runs in your local machine (where I am assuming it has access to more resources, especially memory) it may be an issue of resource allocation and that once ran in a container, it could be possible that it empties its allocated amount of resources and breaks (perhaps a library that is called in real time that uses a chunk of memory?) but we won't be sure unless we can debug it inside a container, so could it be possible for you to add a debug library that records your requests to see if it parses all of those and at some point in time it stops and why? I know a person from R-Studio created a httpuv that logs every request this can be done as in:
devtools::install_github('rstudio/httpuv#wch-print-req')
And after that, maybe share the output and see why the application is behaving like that and killing its own service.
I really thank you in advance, hopefully with those logs we may be able to shed more light into this matter.
Thanks once again!
-JP
I have a pipeline that is working for a file receive location. This pipeline accepts a csv file and maps it to an XML.
I'm now trying to setup a new email receive location using the same port, pipeline and pipeline settings.
Shouldn't biztalk ignore everything but the attachment if I set the body part index equal to 2? And then it should place the attachment in the pipeline just like with the file location, then the pipeline would output an XML file.
Error:
Microsoft.XLANGs.Core.PersistenceException: Exception occurred when persisting state to the database. ---> Microsoft.BizTalk.XLANGs.BTXEngine.PersistenceItemException: A batch item failed persistence Item-ID 72fbeba9-6bfe-48e0-a0e6-ca5bbd191aa1 OperationType MAIO_CommitBatch Status -1061151998 ErrorInfo The published message could not be routed because no subscribers were found. . ---> Microsoft.BizTalk.XLANGs.BTXEngine.PublishMessageException: Failed to publish (send) a message in the batch. This is usually because there is no one expecting to receive this message. The error was The published message could not be routed because no subscribers were found. with status -1061151998
Pop3 properties:
PipelineConfigurations:
I tried stopping the orchestration and the send port and test both locations.
Testing the email location I got the same error with an email with a .csv attachment.
Testing the file location the data didn't reach the database, but the csv was processed because I could see it in the information logs.
This leads me to the conclusion that the problem is related with the mime decoding and whatever my pipeline is outputing from the email body parts.
Also after researching for a while all solutions seem to point to the necessity of having a pipeline exclusively for email since I need to say which part of the multibody part to decode. I was hoping there was a solution that would allow me to reuse the pipeline I use for the file location.
As #Dijkgraaf mentioned:
It is not failing in the pipeline. It is failing due to there being no Orchestration or Send Port that is expecting the message the Receive Port has published to message box.
That means that the receive worked, that the message has passed the pipeline, and is published in the Messagebox, but there is no matching subscription.
Check for routing failures in the BizTalk management console to find out why. It may be that the message type is not waht you expected, or that one more published properties are not set correctly.
See the suspended message and check which body part is the CSV file. Your config says its it should be the 3rd message BodyPart = 2.
I want to pull something from the server (no delete), parse the file in the pipeline component, process the file, if everything goes successfully, I want the adapter delete the file.
i am thinking to enlist the parsing into the pipeline context, this way, I am picturing if the file cannot be parsed, the file will not get to the message box, therefore it will be deemed as a failed transaction, question, will the adapter participate in this transaction? in other words, my goal is to instruct the adapter to delete the file from the server ONLY when the pipeline processed successfully (transaction is commited), the file is left untouched on the server if the pipeline failed (transaction is rolled back, no message is commited to msg box)
Is this achievable? thanks in advance
I think a little experiment is in order. BizTalk, as part of it's nature, will not delete anything until it has been peristed to the message box. That being said, persitence might happen before PipeLine execution. So, the receive adapter receives the file, perists it to the message box and deletes the file. The message might subsequently fail in the pipeline. If this is the case, then the message is a bad format and it will have to be subsequently resubmitted by the sender. If you want to keep this message, you'll have to pick it up with Failed Message Routing. You can then write it to a directory and implement a resubmit pattern. Or, you can pick up the file via Failed Message Routing and put it back on the FTP server (this is sort of a compensation step).
On the otherhand, if the pipeline fails and the message isn't deleted fromt he server... you're fine.