I know client side _underscore.js can be used to throttle click rates, but how do you throttle calls server side? I thought of using the same pattern but unfortunately _throttle doesn't seem to allow for differentiating between Meteor.userId()'s.
Meteor.methods({
doSomething: function(arg1, arg2){
// how can you throttle this without affecting ALL users
}
);
Here's a package I've roughed up - but not yet submitted to Atmosphere (waiting until I familiarize myself with tinytest and write up unit tests for it).
https://github.com/zeroasterisk/Meteor-Throttle
Feel free to play with it, extend, fix and contribute (pull requests encouraged)
The concept is quite simple, and it only runs (should only be run) on the server.
You would first need to come up with a unique key for what you want to throttle...
eg: Meteor.userId() + 'my-function-name' + 'whatever'
This system uses a new Collection 'throttle' and some helper methods to:
check, set, and purge records. There is also a helper checkThenSet
method which is actually the most common pattern, check if we can do something,
and the set a record that we did.
Usage
(Use Case) If your app is sending emails, you wouldn't want to send the same email over
and over again, even if a user triggered it.
// on server
if (!Throttle.checkThenSet(key, allowedCount, expireInSec)) {
throw new Meteor.Error(500, 'You may only send ' + allowedCount + ' emails at a time, wait a while and try again');
}
....
On Throttle Methods
checkThenSet(key, allowedCount, expireInSec) checks a key, if passes it then sets the key for future checks
check(key, allowedCount) checks a key, if less than allowedCount of the (unexpired) records exist, it passes
set(key, expireInSec) sets a record for key, and it will expire after expireInSec seconds, eg: 60 = 1 min in the future
purge() expires all records which are no longer within timeframe (automatically called on every check)
Methods (call-able)
throttle(key, allowedCount, expireInSec) --> Throttle.checkThenSet()
throttle-check(key, allowedCount) --> Throttle.check()
throttle-set(key, expireInSec) --> Throttle.set()
there is not built in support for this currently in meteor, but its on the roadmap https://trello.com/c/SYcbkS3q/18-dos-hardening-rate-limiting
in theory you could use some of the options here Throttling method calls to M requests in N seconds but you would have to roll your own solution
Related
I am trying to create something of an application bot. I need the bot to be triggered in a generic channel and then continue the application process in a private DM channel with the applicant.
My issue is this : The bot can have only one on_message function defined. I find it extremely complicated (and inefficient) to check everytime if the on_message was triggered by a message from a DM channel vs the generic channel. Also, makes it difficult to keep track of an applicants answers. I want to check if the following is possible : Have the bot respond to messages from the generic channel as usual. If it receives an application prompt, start a new subprocess (or bot?) that handles the DMs with the applicant separately.
Is the above possible? if not, is there an alternative to handling this in a better way ?
#client.event
async def on_message(message):
if message.author == client.user:
return
if message.channel.type==discord.ChannelType.private:
await dm_channel.send("Whats your age?") ## Question 2
elif message.channel.type == discord.ChannelType.text:
if message.content.startswith('$h'):
member = message.author
if "apply" in message.content:
await startApply(member)
else:
await message.channel.send('Hello!')
# await message.reply('Hello!', mention_author=True)
async def startApply(member):
dm_channel = await member.create_dm()
await dm_channel.send("Whats your name?") ## Question 1
I have the above code as of now. I want the startApply function to trigger a new bot/subprocess to handle the DMs with an applicant.
Option 1
Comparatively speaking, a single if check like that is not too much overhead, but there are a few different solutions. First, you could try your hand at slash commands. This is library built as an extension for the discord.py library for slash commands. You could make one that only works in DM's, and then have it run from there with continuous slash commands.
Option 2
Use a webhook to start up a new bot. This is most likely more complicated, as youll have to get a domain or find some sort of free service to catch webhooks. You could use a webhook like this though to 'wake up' a bot and have it chat with the user in dm's.
Option 3 (Recommended)
Create functions that handle the text depending on the channel, and keep that if - elif in there. As i said, one if isn't that bad. If you had functions that are called in your code that handled everything, it actually should be fairly easy to deal with:
#client.event
async def on_message(message):
if message.author == client.user:
return
if message.channel.type==discord.ChannelType.private:
respondToPrivate(message)
elif message.channel.type == discord.ChannelType.text:
repondToText(message)
In terms of keeping track of the data, if this is a smaller personal project, MySQL is great and easy to learn. You can have each function store whatever data needed to the database so that you can have it stored to be looked at / safe in case of bot crash & then it will also be out of memory.
On 2 occasions in the past month, we have managed to hit our daily limit on asynchronous apex executions. Salesforce temporarily increased our limit to 425000 but it will be scaled down to 250000 in a week's time. Once we reach the limit, a lot of the SF functions will fail and this has tremendously impacted both internal staff and external customers.
So to prevent this from happening in the future, we need to create some kind of alert in Salesforce to monitor our daily asynchronous apex method executions. Our maximum daily limit is 250000. The alert will need to create a P3 helpdesk ticket and notify couple of users say USER A and USER B once it reaches 70% threshold.
Kindly advise what is possible to achieve the same
Thanks & Regards,
Harjeet
There's a promising Limits method but it doesn't seem to work currently ("reserved for future use"): System.debug(Limits.getAsyncCalls() + ' / ' + Limits.getLimitAsyncCalls());
There's an idea you can upvote: https://success.salesforce.com/ideaView?id=0873A0000003VIFQA2 ;)
You could query SELECT COUNT() FROM AsyncApexJob WHERE ... but that sounds like a bad idea ;)
I think your best course of action is to use SF REST API. There's a "limits" resource you can fetch. You could do it from SF itself (bad idea because if you'd schedule it to run every hour then well, of course it will contribute to the limit consumption too ;)) or from some external app that'd connect to your SF...
https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/dome_limits.htm
You can quickly try it out for example in workbench.developerforce.com before you decide you do want to deep dive into coding it.
Of course if you have control over your batch jobs, queuable, schedulable & #future calls you could implement some rough counter of executions in a helper object for example... won't help you much if most of the jobs are coming from managed packages though...
Got 1 more idea but it's pretty hardcore - you should be able to make a REST API call from javascript. so you could create a simple VF page (even without any apex controller), put JS callout on it, have it check every 5 mins and do something if threshold is hit... But that means IT person would have to have this page open all the time (perhaps as a home page component)... Messy :)
I was having the exact same issue so I created a simple JsForce script in NodeJS to monitor the call to the /limits endpoint.
You can connect a Free Monitoring service like UpTimerobot.com or PingDom.com and get an email when you find the Word "Warning" >50% or "Error" > 80%.
async function getSfLimits() {
try {
//Let's login into salesforce
const login = await conn.login(SF_USERNAME, SF_PASSWORD+SF_SECURITY_TOKEN);
//Call the API
const sfLimits = await conn.requestGet('/services/data/v51.0/limits');
return sfLimits;
} catch(err) {
console.log(err);
}
}
https://github.com/carlosdevia/salesforcelimits
UPDATE: Google has recently updated their error message with an additional error code possibility: "timeout-or-duplicate".
This new error code seems to cover 99% of our previously mentioned mysterious
cases.
We are still left wondering why we get that many validation requests that are either timeouts or duplicates. Determinining this with certainty is likely to be impossible, but now I am just hoping that someone else has experienced something like it.
Disclaimer: I cross posted this to Google Groups, so apologies for spamming the ether for the ones of you who frequent both sites.
I am currently working on a page as part of a ASP.Net MVC application with a form that uses reCAPTCHA validation. The page currently has many daily users.
In my server side validation** of a reCAPTCHA response, for a while now, I have seen the case of the reCAPTCHA response having its success property set to false, but with an accompanying empty error code array.
Most of the requests pass validation, but some keep exhibiting this pattern.
So after doing some research online, I explored the two possible scenarios I could think of:
The validation has timed out and is no longer valid.
The user has already been validated using the response value, so they are rejected the second time.
After collecting data for a while, I have found that all cases of "Success: false, error codes: []" have either had the validation be rather old (ranging from 5 minutes to 10 days(!)), or it has been a case of a re-used response value, or sometimes a combination of the two.
Even after implementing client side prevention of double-clicking my submit-form button, a lot of double submits still seem to get through to the server side Google reCAPTCHA validation logic.
My data tells me that 1.6% (28) of all requests (1760) have failed with at least one of the above scenarios being true ("timeout" or "double submission").
Meanwhile, not a single request of the 1760 has failed where the error code array was not empty.
I just have a hard time imagining a practical use case where a ChallengeTimeStamp gets issued, and then after 10 days validation is attempted, server side.
My question is:
What could be the reason for a non-negligible percentage of all Google reCAPTCHA server side validation attempts to be either very old or a case of double submission?
**By "server side validation" I mean logic that looks like this:
public bool IsVerifiedUser(string captchaResponse, string endUserIp)
{
string apiUrl = ConfigurationManager.AppSettings["Google_Captcha_API"];
string secret = ConfigurationManager.AppSettings["Google_Captcha_SecretKey"];
using (var client = new HttpClient())
{
var parameters = new Dictionary<string, string>
{
{ "secret", secret },
{ "response", captchaResponse },
{ "remoteip", endUserIp },
};
var content = new FormUrlEncodedContent(parameters);
var response = client.PostAsync(apiUrl, content).Result;
var responseContent = response.Content.ReadAsStringAsync().Result;
GoogleCaptchaResponse googleCaptchaResponse = JsonConvert.DeserializeObject<GoogleCaptchaResponse>(responseContent);
if (googleCaptchaResponse.Success)
{
_dal.LogGoogleRecaptchaResponse(endUserIp, captchaResponse);
return true;
}
else
{
//Actual code ommitted
//Try to determine the cause of failure
//Look at googleCaptchaResponse.ErrorCodes array (this has been empty in all of the 28 cases of "success: false")
//Measure time between googleCaptchaResponse.ChallengeTimeStamp (which is UTC) and DateTime.UtcNow
//Check reCAPTCHAresponse against local database of previously used reCAPTCHAresponses to detect cases of double submission
return false;
}
}
}
Thank you in advance to anyone who has a clue and can perhaps shed some light on the subject.
You will get timeout-or-duplicate problem if your captcha is validated twice.
Save logs in a file in append mode and check if you are validating a Captcha twice.
Here is an example
$verifyResponse = file_get_contents('https://www.google.com/recaptcha/api/siteverify?secret='.$secret.'&response='.$_POST['g-recaptcha-response'])
file_put_contents( "logfile", $verifyResponse, FILE_APPEND );
Now read the content of logfile created above and check if captcha is verified twice
This is an interesting question, but it's going to be impossible to answer with any sort of certainly. I can give an educated guess about what's occurring.
As far as the old submissions go, that could simply be users leaving the page open in the browser and coming back later to finally submit. You can handle this scenario in a few different ways:
Set a meta refresh for the page, such that it will update itself after a defined period of time, and hopefully either get a new ReCAPTCHA validation code or at least prompt the user to verify the CAPTCHA again. However, this is less than ideal as it increases requests to your server and will blow out any work the user has done on the form. It's also very brute-force: it will simply refresh after a certain amount of time, regardless of whether the user is currently actively using the page or not.
Use a JavaScript timer to notify the user about the page timing out and then refresh. This is like #1, but with much more finesse. You can pop a warning dialog telling the user that they've left the page sitting too long and it will soon need to be refreshed, giving them time to finish up if they're actively using it. You can also check for user activity via events like onmousemove. If the user's not moving the mouse, it's very likely they aren't on the page.
Handle it server-side, by catching this scenario. I actually prefer this method the most as it's the most fluid, and honestly the easiest to achieve. When you get back success: false with no error codes, simply send the user back to the page, as if they had made a validation error in the form. Provide a message telling them that their CAPTCHA validation expired and they need to verify again. Then, all they have to do is verify and resubmit.
The double-submit issue is a perennial one that plagues all web developers. User behavior studies have shown that the vast majority occur because users have been trained to double-click icons, and as a result, think they need to double-click submit buttons as well. Some of it is impatience if something doesn't happen immediately on click. Regardless, the best thing you can do is implement JavaScript that disables the button on click, preventing a second click.
Current Project:
ASP.NET 4.5.2
MVC 5
PayPal API
I am using this example to build myself a PayPal transaction (and yes, my code is virtually identical), as I do not know of any other method that will return the three values in the title.
My main problem is that, the example I am utilizing is much more concise and compact than the one I used for a much older Web Forms application, and as such, I am unsure as to where or even how to grab the three values I need.
My initial thought was to do so right after the ACK, and indeed I was able to obtain the CorrelationId as well as the TimeStamp, but because this was prior to the user being carted off to PayPal’s site (sandbox in this case -- see the return new PayPalRedirect contained within the if), the TransactionId was blank. And in this example, PayPal explicitly redirects the user to a Success page without returning to the Action that sent the user to PayPal in the first place, and I am not seeing any GET values in the URL at all aside from the Token and the PayerId, much less ones that could provide me with the TransactionId.
Suggestions?
I have also looked at the following examples:
For ASP.NET Core, was unsure how to adapt to my current project particularly due to appsettings.json, but it looked quite well done. I really liked how the values were rolled up in lists.
For MVC 4, but I couldn’t find where ACK was being used to determine success or successwithwarning so I couldn’t hook into that.
I have also found the PayPal content to be like trying to drink from a fire hose at full blast -- not only was the content was hopelessly outdated (Web Forms code, FTW!) but there was also so many different examples it would have taken me days to determine which one was most appropriate to use.
Any assistance would be greatly appreciated.
Edit: my initial attempt at modifying the linked code has this portion:
values = Submit(values);
var ack = values["ACK"].ToLower();
if(ack == "success" || ack == "successwithwarning") {
using(_db = new ApplicationDbContext()) {
var updateOrder = await _db.Orders.FirstOrDefaultAsync(x => x.OrderId == order.OrderId);
if(updateOrder != null) {
updateOrder.OrderProcessed = false;
updateOrder.PayPalCorrelationId = values["CORRELATIONID"];
updateOrder.PayPalTransactionId = values["TRANSACTIONID"];
updateOrder.PayPalTimeStamp = values["TIMESTAMP"];
updateOrder.IPAddress = HttpContext.Current.Request.UserHostAddress;
_db.Entry(updateOrder).State = EntityState.Modified;
await _db.SaveChangesAsync();
}
}
return new PayPalRedirect {
Token = values["TOKEN"],
Url = $"https://{PayPalSettings.CgiDomain}/cgi-bin/webscr?cmd=_express-checkout&token={values["TOKEN"]}"
};
}
Everything within and including the using() is my added content. As I mentioned, the CorrelationId and the TimeStamp come through just fine, but I have yet to successfully obtain the TransactionId.
Edit 2:
More problems -- the transactions that are “successful” through the sandbox site (the ReturnUrl is getting called) aren’t reflecting properly on my Facilitator and Buyer accounts, even when I do payments straight from the buyer’s PayPal account (not using the Credit Card). I know I am supposed to see transactions in the Buyer’s account, either through the overall Dev account (Accounts -> Profile -> balance or Accounts -> Notifications) or through the Buyer’s account in the sandbox front end. And yet -- multiple transactions returning me to the ReturnUrl path, and yet no transactions in either.
Edit 3:
Okay, this is really, really weird. I have gone over all settings with a fine-toothed comb, and intentionally introduced errors to see where things should crap out. It turns out that the entire process goes swimmingly - except nothing shows up in my notifications and no amounts get moved between my different accounts (Facilitator and Buyer). It’s like all my transactions are going into /dev/null, yet the process is successful.
Edit 4: A hint!
In the sandbox, where Buyer accepts the transaction, there is a small note, “You will be able to review the transaction before completing it” or something like that -- suggesting that an additional page is not coming up and that the user is being uncerimoniously dumped back to the success page. Why the success page? No clue. But it’s happening.
It sounds like you are only doing the first part of the process.
Express Checkout consists of 3 API calls:
SetExpressCheckout
GetExpressCheckoutDetails
DoExpressCheckoutPayment
SEC generates a token, and then you redirect to PayPal where the user signs in and reviews the transactions before agreeing to pay.
They are then sent to the ReturnURL included in your SEC request, and this is where you'll call GECD in order to obtain all the buyer details that are now available since they signed in.
Using that data you can complete the final DECP request, which is what finalizes the procedure. No money is actually processed until this final call is completed successfully.
I've got a simple CSV file with 40,000 rows which I'm processing browser-side with papa-parse.
I'm trying to insert them one-by-one into a collection using the techniques in Discover Meteor and other 101 posts I find when Googling.
40000 insert browser-side pretty quickly - but when I check mongo server side it's only got 387 records.
Eventually (usually after 20 seconds or so) it starts to insert server-side.
But if I close or interrupt the browser, the already-inserted records disappear obviously.
How do I force inserts to go server-side, or at least monitor so I know when to notify the user of success?
I tried Tracker.flush() no difference.
I'd go server-side inserts in a Meteor.method, but all the server-side CSV libraries are more complex to operate than client-side (I'm a beginner to pretty much everything programming :)
Thanks!
This is the main part of my code (inside client folder):
Template.hello.events({
"submit form": function (event) {
event.preventDefault();
var reader = new FileReader();
reader.onload = function (event) {
var csv = Papa.parse(this.result, {header: true});
var count = 0;
_.forEach(csv.data, function (csvPerson) {
count++;
Person.insert(csvPerson);
console.log('Inserting: ' + count + ' -> ' + csvPerson.FirstName);
});
};
reader.readAsText(event.target[0].files[0]);
}
});
The last few lines of console output:
Inserting: 39997 -> Joan
Inserting: 39998 -> Sydnee
Inserting: 39999 -> Yael
Inserting: 40000 -> Kirk
The last few lines of CSV (random generated data):
Jescie,Ayala,27/10/82,"P.O. Box 289, 5336 Tristique Road",Dandenong,7903,VI,mus.Proin#gravida.co.uk
Joan,Petersen,01/09/61,299-1763 Aliquam Rd.,Sydney,1637,NS,sollicitudin#Donectempor.ca
Sydnee,Oliver,30/07/13,Ap #648-5619 Aliquam Av.,Albury,1084,NS,Nam#rutrumlorem.ca
Yael,Barton,30/12/66,521 Auctor. Rd.,South Perth,2343,WA,non.cursus.non#etcommodo.co.uk
Kirk,Camacho,25/09/08,"Ap #454-7701 A, Road",Stirling,3121,WA,dictum.eu#morbitristiquesenectus.com
The hello template is a simple form obviously, just file select and submit.
Client code is under client directory.
Person defined in a file in application root.
CSV parsed as strings for now, to avoid complexity.
The records inserted look fine, retrieve by name, whatever.
Person.find().count() browser-side in console results in 40000.
Happy to send the file, which is only 1.5MB and it's random data - not sensitive.
I think call() should work as follows:
On client side
Meteor.call("insertMethod",csvPerson);
And method on server side
insertMethod: function(csvPerson){
Person.insert(csvPerson);
}
In Meteor, on some scenarios, if you don't pass a callback the operation will sync.
If you run the code Person.insert(csvPerson); on the server, the operation will be sync not async. Depending on what you want to do, you might have serious problems in the future. On the client, it won't be sync but async.
Since node.js is an event-based server, a single sync operation can halt the entire system. You've to be really about your sync operations.
For importing data, the best option is to do at server-side inside Meteor.startup(function(){ //import code goes here}).
The solution propose by Sindis works but it slow and if the browser closes (for some reason), you're not keeping a track of the already inserted records. If you use Meteor.call("insertMethod",csvPerson);, this operation will be sync on the client.
The best option on your beginner scenario (not optimal) is to:
1- While (You have record to insert)
2- Call Meteor.call without a callback
3- Count all the inserted fields in the Collection
4- Save this value to localStorage
5- Go back to step 1
This works assuming that the order of insertion is the same on every insert attempt. If you browser fails, you can always get the value from localStorage and skip that number of records.