Before I tackle this solution, I wanted to run it by the community to get feedback.
Questions:
Is my approach feasible? i.e. can it even be done this way?
Is it the right/most efficient solution?
If it isn’t the right solution, what would be a better approach?
Problems:
Need to send mass emails through the application.
The shared hosted server only permits a maximum of 500 emails to be sent per hour before getting labeled a spammer
Server timeout while sending batch emails
Proposed Solution:
Upon task submittal (i.e. the user provides all necessary email information using a form and frontend template, selects the target audience, etc..), the action will then:
Determines how many records (from a stored db of contacts) the email will be sent to
If the number of records in #1 above is more than 400:
Assign a batch number to all these records in the DB.
Run a CRON job that:
Every hour, selects 400 records in batch “X” and sends the saved email template until there are no more records with batch “X”. Each time a batch of 400 is sent, it’s batch number is erased (so it won’t be selected again the following hour).
If there is an unfinished CRON JOB scheduled ahead of it (i.e. currently running), it will be placed in a queue.
Other clarification:
To send these emails I simply iterate over the SWIFT mailer using the following code:
foreach($list as $record)
{
mailers::sendMemberSpam($record, $emailParamsArray);
// where the above simply contains: sfContext::getInstance()->getMailer()->send($message);
}
*where $list is the list of records with a batch_number of “X”.
I’m not sure this is the most efficient of solutions, because it seems to be bogging down the server, and will eventually time out if the list or email is long.
So, I’m just looking for opinions at this point... thanks in advance.
Related
I have a script that works as a daemon.
This script every so often is supposed to retrieve the order data and then processes the data.
In a situation where the script is running and retrieves the data of a given order and the order has a status of, for example, "on-hold" and then I change its status to "processing" the script still sees the status "on-hold" when I retrieve the data via wc_get_order because it uses an internal wp cache that is not refreshed.
So how do I retrieve the most current order data from the database.
I searched in the source code if there is perhaps a parameter to force the retrieval of data from the database but did not find it.
After hours of searching I made it.
wp_using_ext_object_cache( false );
wp_cache_flush();
wp_cache_init();
Using this 3 lines of code clear cache.
Problem Description:
1.There is Biztalk Application receiving formatted/zipped data file containing > 2 million data records.
2.Created pipeline component that process file and 'de batching' these 2 million records of data into smaller slice-messages with ~2000 records each .
3.Slice-messages are being sent to SQL port and processed by stored procedure.Slice-messages contains filename and batch id.
Questions:
A.What would be the best way to know that all slice-messages received and processing of whole file completed on SQL side ?
B.Is there any way in biztalk port to say "do not send message of type B, until all messages of Type A have been send" (messages priority)?
Here are possible solutions I've tried :
S1.Add specific 'end of file' tags to end of last slice-message saying that file is being processed and stored procedure will receive this part of message mark file as completed.
But because messages are being delivered asynchronously last message can be received on sql earlier that other messages and I will have false-competed event.
So this solution is only possible for "Ordered delivery ports" - but this type of ports have poor performance because sending only one message at a time.
S2.Transfer total records count into every slice-message and run count() sql statement after every slice-message received.
Because table where data is stored is super huge, even running count with filename as parameter takes time.
I'm wondering if there is better solution to know that all messages are being received ?
Have your pipeline component emit a "batch" message that contains the count of the records in the batch and some unique identifier that can link it back to the slice-messages records.
Have both the stored procedure that processes the slice-message and the batch message check to see if the batch total (if it exists yet for the slice-message process) matches the processed total, if they match, then you've finished processing them all.
Here's how I would approach this.
Load the 2MM records into a SQL Server table or tables by SSIS.
Drain the table at whatever rate give you an acceptable performance profile.
Delete records as they are processed (completed).
When no more records for "FILE001.txt" exist, SQL Server will return a flag
saying "FILE001.txt complete".
Do further processing.
When the staging table is empty, the Polling SP can either return nothing (the Adapter will silently ignore the response) or return a flag that says "nothing to do" and you handle that yourself.
Within my BizTalk 2010 solution I have the following orchestration that’s is started by the receipt of a courier update message. It them makes a couple of call to AX's WCF AIF via two solicit-response ports, a Find request and an Update request.
For this application we are meeting audit requirements through use of the tracking database to store the message body. We are able to link to this from references provided in BAM when we use the TPE. The result for the customer is nice, they get a web portal from which they can view BAM data of message timings etc. but they can also click a link to pull up a copy of the message payloads from the tracking db. Although this works well and makes use of out-of-box functionality for payload storage it has led to relatively complex jobs for the archiving of the tracking db (but that's another story!).
My problem relates to continuation. I have created the following Tracking Profile:
I have associated the first of the orchestration's two solicit response ports with the continuation RcvToOdx based on the interchange Id and this works, I get the following single record written to the Completed activity table:
So, in this case we can assume that an entry was first written on receipt in the inbound message, with the TimeReceivedIntoBts column populated by the physical file receive port. The FindRequestToAx column was then populated by the physical WCF send port. Because this was bound to the RcvToOdx continuation Id and used the same interchange Id and the previously mentioned file receive message, the update was made to the same activity. Notification of the resulting response was also updated to the same activity record - into the FindResponseFromAx column.
My problem is that I would also like BAM to record a timestamp for the subsequent UpdateRequestToAx. Because this request will have the same interchange Id as the previous messages I would expect to be able to solve this problem by simply binding the AxUpdate send port (both send and receive parts of it) to the same continuation id, as seen in the following screen grab:
I also map the UpdateRequestToAx milestone to the physical Ax_TrackAndTraceUpdate_SendPort (Send) and the OrchestrationCompleted milestone to Ax_TrackAndTraceUpdate_SendPort (Receive)
Unfortunately, when I try this I get the following result:
Two problems can be seen from the above db screen grab:
1. Date for the update send port was inserted into a new activity record
2. The record was never completed
I was surprised by this because I'd thought since they update port was enlisted to use the same continuation, and the single InterchangeId was being used by all ports for the continuation Id then all the data milestones would be applied to a single activity.
In looking for a solution to this problem I came across the following post on Stack Overflow suggesting that the continuation must be closed from the BAM API: BAM Continuation issue with TPE. So, I tried this by calling the following method from an expression shape in my orchestration:
public static void EndBAMContinuation(string continuationId)
{
OrchestrationEventStream.EndActivity(CARRIER_ORDER_ACTIVITY_NAME, continuationId);
}
I can be sure the method was called ok because I wrapped the call with a log entry from the CAT framework which I could see in debug view:
I checked the RcvToOdx{867… continuation Id against the non-closed activity and confirmed they matched:
This would suggest that perhaps the request to end the continuation is being processed before the milestone of the received message from the UpdateAx call?
When I query the Relationsips tables I get the following results:
Could anyone please advise why the UpdateToAx activity is not being completed?
I realise that I may be able to solve the problem using only the BAM API but I really want to exhaust any possibility of the TPE being fit for purpose first since the TPE is widely used in other BizTalk solutions of the organisation.
To solve this problem I created a 2nd continuation in the TPE.
"RcvToOdx" continuation for the Find and "OdxToUpdate" continuation for the update - source is InterchangeId on the initial receive port - UPS_TrackAndTrace (same as for other "RcvToOdx" continuation), dest Id is the InterchangeId mapped to update send port.
I have a BizTalk receive port monitoring an FTP location. I expect a file to arrive at least once per day in that location and for BizTalk to pick it up and kick off an orchestration. This part is working fine.
However, sometimes the sender fails to send a message during a day, in which case I want an email to sent to notify the users that something is amiss.
I could solve this outside of BizTalk, by creating a daily job that looks in our database for processed files and makes sure there is at least one in any given day. However, I'd prefer to solve this "in line" with the BizTalk solution that is already in place, and not deploy a separate, unrelated job which will increase maintenance headaches.
Is there any functionality in BizTalk that would allow me to send a notification if a receive port doesn't receive something in a given timeframe?
Short answer: Not really.
The logic you want to implement would require a customised version of the FTP Adapter. Depends on how comfortable you are rolling up your sleeves and getting into the Adapter SDK.
If you wanted to keep your solution "Purely BizTalk", you could set up a secondary Orchestration using a SQL Receive Location tied to a stored procedure. This stored procedure executes regularly and looks for records in your "Processed File" table received in the past (business) day. If none are found, it fabricates a record and returns it via the SQL Receive Location. This would be your trigger to send the email notification.
One solution, not elegant though, is to have a secondary FILE receive location, with a schedule window, outside your cutoff time.
Failure scenario:
In this FILE receive location, you have an intelligent/dummy message conforming to the same schema as FTP receive. The intelligent part is to have one of the fields in the message telling us when was the last time we received the file from FTP. The rest of the content is dummy.
Within your orchestration, you check where you received your file from. If its the secondary receive location (using the context property BTS.ReceiveLocationName), you check the date field of this dummy/intelligent message and if it is in past 24 hours ( or similar logic) send an email notifying you did not receive the file from the upstream FTP process and also save a copy of the dummy message (received) back to the secondary FILE receive location unchanged.
Success Scenario:
Apart from normal processing, you save a copy of the dummy/intelligent message to the secondary FILE receive location, with the datetime field reflecting when you processed the file you received from FTP receive location.
Initialising:
You start with a dummy/intelligent message in the secondary FILE receive location with the datetime field value well in the past ( assuming we never received the file from FTP) or with yesterday's date ( assuming we received a file successfully from FTP the day before.)
Overview:
Your orchestration has two trigger points.
When you receive a file via FTP
A scheduled FILE receive location, triggered after the cut-off time.
I want to send a string from a website to a local machine.
My local machine is connected into a network through a router.
Thanks You
email Id: manish.m.meshram#gmail.com
Well, that largely depends on what the receiving computer needs to do with that string.
If you only need to notify the user of this, I would suggest the easier way is to go with the net send command.
Since you are wking in ASP.net, you can use the Process and ProcessStartInfo class to launch a command like
net send YourPC "String to send"
If you need to do something more sophisticated with the string message, you could for example print it in some sort of log file and then read it from the destination machine.
If you can give more information on your needs, we'll be probably able to help you better.
Luca
I suggest you poll the webapp for messages.
For instance, let the webapp have an URL that simply returns the timestamp of the most recent message, at http://thesite.com/messages/MostRecentTimetamp.aspx
The page should return the timestamp only, in an format you can parse, for instance:
2009-08-29 14:00:00
Then, on another URL, http://thesite.com/messages/FromLastHour.aspx display the list of messages for the last N hours (or other suitable time period). This page could return one message per line, with the message timestamp at the start of the line.
For instance:
2009-08-29 13:58:20 A message
2009-08-29 13:59:30 Here's a message
2009-08-29 14:00:00 Another message
On your local machine, create a program that as often as needed reads and parses http://thesite.com/messages/MostRecentTimetamp.aspx. If the program detects that the timestamp has changed, read http://thesite.com/messages/FromLastHour.aspx and process the new messages.
Adjust the timing according to your needs.
Or even better, have an URL: http://thesite.com/messages/MoreRecentThan.aspx?timestamp=2009-08-29 13:50:00.
That returns messages that are newer than the timestamp passed in. The program on your local machine should then pass the timestamp of the most recent message it has handled.
Of course, your web site has to keep track of outgoing messages in some sort of queue. You could use a database table for this. The web app can delete old messages from this table periodically.
If you want to get fancy, you could implement this as a SOAP web service. Or you could let the URLs return the data formatted as JSON.