We have created an application to send out bulk emails using AWS SES. We are able to send out the emails and track the metrics like Opens, Clicks etc using AWS SNS successfully. The only problem we have is that in the "Opens" object that SNS is sending, it is always returning the same value "Mozilla/5.0 (Windows NT 5.1; rv:11.0) Gecko Firefox/11.0 (via ggpht.com GoogleImageProxy)". What we are looking at is to determine where the email is opened like Mobile/Tab/Desktop and in which browser. Even when the email is opened in Chrome, it is returning as Mozilla. Any help/suggestion in this regard is highly appreciated.
Additional Info: I figured out that the userAgent is being correctly returned in "clicks" object. But not in the "Open" object. Not sure why. We would like to track the same information when the email is opened also as not all the recipients click on a link.
There isn't actually a way to determine that a message has been opened.¹ Detecting "opens" relies on detection of the viewer fetching an image embedded in the message when the mail is "opened."
At the bottom of each message, we insert a 1 pixel by 1 pixel transparent GIF image. Each email includes a unique link to this image file; when the image is opened, we can tell exactly which message was opened and by whom.
When the viewer is Gmail, the user's browser doesn't fetch this image.
https://aws.amazon.com/blogs/messaging-and-targeting/open-and-click-tracking-have-arrived/
When a message is opened in gmail, the user's browser doesn't fetch the image directly, it fetches it from the google image proxy, and the image proxy fetches it from SES and generates the tracking event. Hence, (via ggpht.com GoogleImageProxy).
This isn't something that you have control over, as the sender.
The proxy can identify itself by saying whatever it likes in the User-Agent field -- there is no reason to believe that the entire user-agent string isn't being created by the proxy. Google searching the topic seems to confirm that this is how the proxy always appears. Mozilla/5.0 is a generic user agent string, that does not mean anything more than "I am some kind of web browser, or want the server to believe that I am."
¹there isn't actually a way... well, technically, there is, but thanks to the widespread profusion of spam, this standard is almost never applied to Internet mail. As noted in RFC-8098, "The presence of a Disposition-Notification-To header field in a message is merely a request for an MDN. The recipients' user agents are always free to silently ignore such a request." This is almost always what happens... nothing.
Related
When hosting a website using HTTP Authentication, if the client has not authenticated, the server will send a 401 Unauthorized response, including a WWW-Authenticate header. An optional directive in this header is realm:
A string describing a protected area. A realm allows a server to partition up the areas it protects (if supported by a scheme that allows such partitioning), and informs users about which paricular username/password are required.
(emphasis mine)
Let's say www.example.com requires authorization and has configured a realm value of Test Area. Back in the day, most web browsers would display a login dialog when receiving such a response, and say something along the lines of "Authorization required. The site at www.example.com says 'Test Area'."
But (at least recent versions of) Chrome, Firefox, and Edge, now all just display a generic message, without including the realm value. Interestingly enough, IE still displays the realm value (screenshots below, all Windows 10 64bit):
Chrome (v93.0.4577.82)
Firefox (v92.0)
Edge (v93.0.961.52)
IE (v11.1411.18362.0)
I'm pretty sure Firefox and Edge used to display it. Chrome may have a while back but seems like it was the first to stop. Since all modern browsers aren't displaying it, I assume there's some reason why..?? I've search all over the internet, and can't figure it out. I have a use-case where it would be helpful to users to have the realm displayed, as it would make it clearer which particular credentials they need to use. I know that you can't force the browser to display it, but it's just annoying. However if there is a valid reason for it not being shown I'll accept that.
The reason is that this could be abused for phishing attacks, by putting some misleading message into the realm. The login dialog for http authentication is part of the trusted browser UI, and giving the server the opportunity to modify that UI - even by just displaying text - is a security risk.
I'm trying to fetch data from a website (https://gesetze.berlin.de/bsbe/search). Using Mozilla, I've taken a look at the network analysis. Usually, I'm just messing around with the parameters of the POST-Request to see how I might influence the response of the server. But when I simply re-send the request (making no changes at all), I'm getting HTTP-response 500. The server answer states as message: security_notAuthenticated.
Can anyone explain that behaviour? The request is done by the same PC, the same browser in the same session, and there is no login function on that website. Pictures shown below.
Picture 1 - Code 200
Picture 2 - Code 500
The response security_notAuthenticated indicates, that your way of repeating the request omits some authentication-related information.
When I repeat the request, using Mozilla Firefox's "Resend" or "Edit and resend" function, the Cookie header is not sent with the request. Although it occurs in the editable header list when using "Edit and resend" it's missing when watching the actual sent request. I'm not sure whether this is a feature or a bug.
When using Firefox's "Use as Fetch in Console" function, the header will automatically be included and you still have the ability to change the headers and the body. The fetch API is a web standard and some introductory material about fetch can be found on MDN.
If you want to do custom requests, in the browser, fetch is a good option.
In other environments and languages you usually use some HTTP client (just search the web for "...your language... http request" or similar, you will find something).
I've encountered a weird situation, after registration we're sending an email with a verification link, pretty standard stuff, but somehow clicking on the link seems to make the request twice, looking at the logs, the first time it comes from my IP and the second request comes from some Google IP: 66.102.8.60 (doing a reverse lookup shows google-proxy-66-102-8-60.google.com).
Any idea what's going on and how to prevent this?
The server is running Nginx and the site is Ruby on Rails if that helps.
I do not know the root cause but my best guess is same as Tripleee wrote above - most probably google is scanning urls. This happens in all browsers (well at least in Chrome and Firefox), but only under following circumstances:
the url is clicked from gmail (if you copy paste it to browser tab, the second request is not issued)
the url is clicked for the first time... Subsequent clicks from the same email do not trigger second request
I know it is probably not the answer you expected, but after giving it some thought I figured that operation like this should be handled on server side. In my case I am tracking information about confirmation urls anyways, so the first time the request comes to my backend I am deleting it and proceeding with confirmation normally. Since the confirmation entry is missing in the database for the second request it returns immediately with status 404, 422 or something whatever suits you.
Hope that helps anyone who gets here looking for an answer to this problem ;)
I want to change the message that pops up during implementation of Basic Auth.The current default message is:
Server requires a username and password.
Something that would be more accurate for me is :
Server requires an Email and Password.
My problem is that i can't find or don't know where this message is set and if it can be changed. Most of the questions online are about Basic Auth implemention but this is not my problem -- i can implement it very fine. I just need a more accurate response for the user.
Here is how i force an authentication window using echo:
c.Response().Header().Set(echo.HeaderWWWAuthenticate, `Basic realm="Your Email is your Username"`)
return echo.ErrUnauthorized
NB: Only Firefox shows the realm message. Both Chrome and Opera do not.
This is not related to Go but actually to browser behaviour when receiving that header.
It seems Chrome/Chromium has a known issue with this related to the feature not considered secure by the development team, so I don't think you'd be able to fix it on your side unless you resort to some other authentication mechanism.
See here for more details:
https://bugs.chromium.org/p/chromium/issues/detail?id=544244#c32
Thanks for the responses but they were not satisfactory. I had to do some reading on this topic.
The correct answer is that the login prompt/dialog is a response built into the user-agent/browser and cannot be changed by the server. This also explains why some browsers show realm while others don't.
According to Wikipedia Basic access authentication all the server does is:
When the server wants the user agent to authenticate itself towards the server, it must respond appropriately to unauthenticated requests.
Unauthenticated requests should return a response whose header contains a HTTP 401 Unauthorized status[4] and a WWW-Authenticate field.[5]
The WWW-Authenticate field for basic authentication (used most often) is constructed as following:
WWW-Authenticate: Basic realm="User Visible Realm"
Is there any sort of data dump or data set with information from Web Server logs?
The information that I am mainly looking for are:
a) what type of request is it (POST or GET or HTTP or something else)
b) What type of data is being transferred (image, audio, video or text)
c) what is the size of the data that is being transferred
Information such as IP address, URL can be anonymous.
Are you using Firefox? If so, you can use the included Web Console tool to view all the HTTP request body being sent from your browser to the server and the response bodies, along with things like the method (GET, POST, etc.). This would be the same thing that a web server would be logging (except the IP address of the client is always you, obviously). You should be able to copy all the data and paste it to a file if you want a data dump.
To use the web console, click the orange Firefox button and then Web Developer > Web Console. Or if you're using an older version or have the Firefox button disabled, it's under the tools menu.
Edit: To get the most out of it, you'll want to right click on the console and select Log Request and Response Bodies. This will get you more information than just the headers.