I am trying to save copies of the EDI files I am receiving from an ftp server. I have tried using a send port to a file, but the parties will not resolve and it is using Fallback settings so I cannot set up proper validation, specifically allow leading and trailing zeros. Is there a better way to save a copy of the received EDI file.
Adding a Archiving Custom Pipeline component to the Receive Pipeline works for most scenarios.
The solution of Dijkgraaf (archiving pipeline component) makes a lot of sense, but just to provide you with other options:
The EDI files come in from somewhere. If this is a FILE or an FTP server for example, you could consider to script the archival of the file, by scripting the file move from the incoming folder to a folder where BizTalk is listening to. At that point it is very easy to make an extra copy of the original file before BizTalk is actually even seeing this. This can be done by a Scheduled Task or by BizTalk itself (PassThru pipelines).
Typically, it's not a bad design to separate your incoming folder from the folder where BizTalk will pick it up. This mostly depends on your own preferences. Do consider however that your BizTalk Server is often monitored and logged and your external scripts/scheduled tasks are not.
In that case I always tend to choose for the BizTalk 'PassThru' solution, since if this fails, you will be notified via the event log.
Before doing any custom work, take some to see if any of the built it BizTalk features can satisfy your requirement. There are two features specifically that address your situation:
EDI and AS2 Status Reporting - Stores incoming and outgoing EDI transactions and interchanges with their delivery status (997). The UI is built into BizTalk Administrator. Details here: EDI and AS2 Status Reporting
BizTalk Tracking: Can store any incoming or outgoing message. The search and retrieve UI is also part of BizTalk Administrator. Sample of how to use Tracking to archive messages: BizTalk Archiving, The BizTalk Way
Finally, you can consider a fully out-of-band solution by saving copies of the files before they even get to your BizTalk app.
Related
I have a problem where I receive files from a third party via a website. These files come in order from the third-party and sit in a folder. Because of security constraints, I am not able to poll the directory directly via SMB but instead I have to fetch the files every 1 minute using SFTP. This creates a problem because the files that were delivered to me in-order are now all together in my receive location's folder. I need to deliver these files in the order they came to my send port.
I have thought of creating a separate program that would open the files and then copy them in order (based a segment called SequenceId in the XML) to a folder within the BizTalk server that is monitored by the receive location which would ensure the files are delivered in order. I would prefer not to introduce another failure point (the program) but I am not sure how I can do this with pure BizTalk.
You can do this with pure BizTalk (search for BizTalk resequencing), however you end up with a complex solution including a singleton Orchestrations and it is usually easier to use a database table.
First interface picks it up, and just insert the data into a table. Either as flat table if your message structure is flat, or as a one field with XML data and another field that contains your sequence number extracted from the payload.
Your second interface polls a stored procedure that looks if the next in sequence is available to be processed.
You will have to have considerations about what to do if a particular message for a sequence never arrives, do you send out an alert, or process the messages you do have after a preset delay.
I am sharing the link of a file (e.g. pdf), which is stored in my server. Is it possible to track whenever some user is downloading the file? I don't have access to the script of the other page but I thought I could track the incoming requests to my server. Would that be computationally expensive? Any hints towards which direction to look?
You can use the measurement protocol, a language agnostic description of a http tracking request to the Google Analytics tracking server.
The problem in your case is that you do not have a script between the click and the download to send the tracking request. One possible workaround would be to use the server access log, provided you have some control over the server.
For example the Apache web server can user piped logs, e.g. instead if being written directly to a file the log entry is passed to a script or program. I'm reasonably sure that other servers have something similar.
You could pipe the logs to a script that evaluates if the log entry points at the URL of your pdf file, and if so breaks down the info into individual data fields and sends them via a programming language of your choice to the GA tracking server.
If you cannot control the server to that level you'd need to place a script with the same name and location as the original file on the server, map the pdf extension to a script interpreter of your choice (in apache via addType, which with many hosts can be done via a htaccess file) and have the script sending the tracking request before delivering the original file.
Both solutions require a modicum of programming practice (the latter much less than the former). Piping logs might be expensive, depending on the number of requests to your server (you might create an extra log file for downloadable files, though). An intermediary script would not be an expensive operation.
I have an orchestration which polls data from a database (which is actually used by an ERP, so i am not able to manipulate data in this database), Once the polling port finds matching data it executes the orchestration and sends data to a third party web service.
The logic used in this orchestration is complicated and often prone to change, and so it's important to cover it with proper set of tests. I am thinking about this for a while and even thought of using 3 different components so that,
First part (can be only 2 ports) reads the data from the database and put into a folder
Second one (current orchestration) uses a file port to read data and dumped by the first component and it dumps the resultant file to another folder
Third component reads the file dumped by the second component and send it to the web service
However I have few concerns,
Is this a frowned upon practice, when it comes to the BizTalk? Or is it a normal way to do things?
The performance - would it be significant slower compared to the current solution?
We are currently using the one of the server to run the tests / do the build using BTDF and Jenkins. Is there a way to disable the components 1 and 3, run the tests and re-enable them once build is completed so that it can function normally?
You can avoid the overhead of writing to and reading from files by using the built-in functionality of the MessageBox. The first place to start is here: https://msdn.microsoft.com/en-us/library/aa949234.aspx
There is an excellent Biztalk sample which shows how you can use this approach to modularise your functionality into a set of orchestrations which independently read from and write to the MessageBox. It's referenced at the bottom of the previous page and is called "Direct Binding to the MessageBox Database in Orchestrations".
I'd recommend against this approach. You'd be better off making the three orchestrations direct bound to the MessageBox and subscribe to the messages published by the previous orchestration. You could also create send ports that subscribe to these messages, or just use the management console to debug the messages.
You can also write unit tests for your various tasks. If you're doing some work in a .NET helper library, you can have a plain old unit tests project. You might also want to look into the BizUnit framework (https://bizunit.codeplex.com/) - it takes a little doing to get used to but it's a great resource for writing BizTalk unit tests.
We have a solution that receives HIPAA X12 files using the Multiple schema, then splits them and sends them on to the orchestration. Occasionally the client will send a file with a structural error somewhere in the middle, and the message will be suspended, but only after it's already send a few of the sub-documents on to the orchestration. This is annoying...
What I'd like to do is validate the entire message and suspend it before disassembling it into multiple messages... I know I could do this if I switched to the Single schema, or set Preserve Interchange, but then I have to deal with the splitting somewhere else.
Suggestions? Thanks.
Check the Them->You tab of the Agreement under Local Host Settings.
The default option for Inbound batch processing option is to "suspend Transaction Sets (ST/SE) on error."
Change to "suspend Interchange on Error" and test to see if that satisfies your requirement.
That should work unless the Trading Partner sends files with multiple Interchanges.
I have a receive location in my BizTalk 2010 project, and sometimes that receive location will receive an empty file. The receive pipeline is PassThruReceive. We then have a Send Port that has a filter for that Receive Port Name. So all we are doing is moving the file from the receive location to the send location.
The issue that I'm running into is that in the event that we get an empty file in the receive location, my client wants the file to still be moved to the send port. I know that out of the box, the FILE adapter discards empty files and writes an event to the Event Log stating that it was deleted.
I have followed articles that show a custom FILE adapter accomplishing this task. I have had some success with this custom adapter. The file is picked up. Received by BizTalk and the Send Port successfully sends the file. However, even with this solution, I'm running into an issue in the receive side where the file is locked and cannot be deleted. I have followed various articles on this subject, and I get the same issue every time where the file is locked and can not be deleted.
My question is. Even though batchMessage.Message.BodyPart.Data.Close(); is being called, the stream is still locked. Is there any way for me to find where else BizTalk may be locking the file? Is there any other way of handling this?
One of the articles that I followed is located here: http://biztalkwithshashikant.blogspot.com/2011/04/processing-empty-files-in-biztalk.html
It seems to me that you are running into issues when running your custom FILE adapter multi-server. I bet you are running more than 1 server in the BizTalk Group?
I haven't done this myself, but I heard that getting an adapter to run smoothly multi-server is one of the hardest things to do in BizTalk. The trick is to find a way to be able to share the load between multiple instances of the same BizTalk host.
Do you still have the same problem when running the instance only on 1 server instead of 2?
in the custom pipeline component code method:
IBaseMessage Execute(IPipelineContext pContext, IBaseMessage pInMsg)
You should return pInMsg, not set to null, and the .BodyPart stream positioned at the end. If pInMsg is null BizTalk will discard the message silently. You don't need to close it but you do need to move it to the end to let BizTalk know you read and processed it all.
A way around this is to use the FTP adapter instead to pick up the files, the FTP adapter does not discard empty files.
It is quite possible that to system creating the files, it is still a file location, but also accessible via FTP.