I have been using the amazon advertising api to fetch my client's the seller central data. I've observed that there is always a deviation in metrics data when comparing the API response with the amazon seller central interface.
Note: The API response has less data when compared to the interface.
Please let me know how to address this issue, or the reason for the data latency.
Related
Consider a user who is using a service (say an app backend) and routing their connection through an intermediary proxy and/or vpn. Specifically let’s assume the user is in Shanghai-China, the proxy is in the Dallas-Texas and the backend is on AWS. In theory, compared to a user who actually lives in Dallas-Texas (on the same network) the Shanghai-China user will have additional latency in sending/receiving events due to the Asia<-> USA trip.
Questions:
Are there known/published methodologies for seeing this additional latency and thereby identifying imposters from far away? The simplest I can think of is grouping by isp providers and then looking for outliers in latency.
Are there additional ways to honeypot such users? I’m not a network export but I think various sorts of media (eg video streaming) get different treatment on these networks so I’m wondering if it is possible to send additional event data to honeypot more precise latency anomalies.
Assumptions:
We can assume that we have plenty of user data, from each network provider. We also have streams of event data that includes client and server side timestamps for sending and receiving data.
I’m strictly interested in identifying users who are very far away from the IP source. I am NOT interested in methodologies that strictly try to classify an IP as a VPN (eg what Maxmind does in the above link).
Since you have stated you are not interested in the VPN or Relay identification aspect of it, but only latency detection for "far away" users, I will offer some ideas:
Run your own HTTP(S) measurement server that all clients must run an HTTP ping against. HTTP round-trips time in milliseconds will be acceptable for broad-strokes classification of a user's "distance" from your server - assuming that the initial TCP handshake has been completed by the client, all intermediaries, and your measurement server ("pre-warmed" connection).
Use an IP geolocation API. These will give you the country code (almost always accurate), and approximate latitude, longitude (broadly accurate) that you may use to calculate the distance from your server. This of course assumes that the public IP of the client is visible to you, and not completely obfuscated by the intermediaries.
I'm trying to calculate the price of network data transfer in and out from an AWS WP website.
Everything is behind Cloudfront. EC2/RDS returns dynamic resources and few statics, S3 returns only static resources. The Application Loadbalancer is there just for autoscaling purpose.
Even if everything seems simple the experience taught that the devil is in the detail.
So, at the end of my little journey (reading blogs and docs) I would like to share the result of my search and understand what the community thinks of.
Here is the architecture, all created within the same region/availability zone (let's say Europe/Ireland):
At time of writing, the network data transfer charge is:
the traffic out from Cloudfront (first 10 TB $0.15/GB per month, etc.)
the traffic in and out from the Application load balancer (processed bytes: 1 GB per hour for EC2 instance costs ~7.00$/GB)
For the rest, within the same region is free of charge and Cloudfront does not charge the incoming data.
For example: within the same region, there should be no charge between an EC2 and an RDS DB Instance.
Do anyone knows if I'm missing something? There are subtle costs that I have to oversee?
Your question is very well described. Thanks for the little graph you drew to help clarify the overall architecture. After reading your question, here are the things that I want to point out.
The link to the CloudFront data transfer price is very outdated. That blog post was written by Jeff Barr in 2010. The latest CloudFront pricing page is linked here.
The data transfer from CloudFront out to the origin S3 is not free. This is listed in "Regional Data Transfer Out to Origin (per GB)" section. In your region, it's $0.02 per GB. Same thing applies to the data from CloudFront to ALB.
You said "within the same region, there should be no charge between an EC2 and an RDS DB Instance". This is not accurate. Only the data transfer between RDS and EC2 Instances in the same Availability Zone is free. [ref]
Also be aware that S3 has request and object retrieval fees. It will still apply in your architecture.
In addition, here is a nice graph made by the folks in lastweekinaws which visually listed all the AWS data transfer costs.
Source: https://www.lastweekinaws.com/blog/understanding-data-transfer-in-aws/
I have multiple applications making use of Application Insights for Production Data. I'm trying to use the City telemetry field to map our current users. This data appears to be tracked very inconsistently and in most cases (> 75%) is just unavailable.
I understand some customers will be using VPNs which could affect the results, but not to the extent I'm seeing.
Here is the info from the Azure FAQ:
How are City, Country and other geo location data calculated? We look
up the IP address (IPv4 or IPv6) of the web client using GeoLite2.
Browser telemetry: We collect the sender's IP address.
Server telemetry: The Application Insights module collects the client IP
address. It is not collected if X-Forwarded-For is set.
You can configure the ClientIpHeaderTelemetryInitializer to take the IP
address from a different header. In some systems, for example, it is
moved by a proxy, load balancer, or CDN to X-Originating-IP.
Does anyone know how to improve geolocating user cities for App Insights?
IP Geolocation is not 100% accurate and you need to live with it. City accuracy is quite low because the information is guessed from multiple data that change frequently. One way to improve accuracy is to use a service that aggregates data from multiple sources and does it continuously, multiple times a day.
A second manner to enhance the results is to filter based on whether the IP is associated with a proxy by using threat data.
For both purposes, I recommend looking at Ipregistry, a service I work for:
https://api.ipregistry.co/?key=tryout
It would be great if MSFT could provide an example of manually setting the location in Browser telemetry. I understand privacy concerns, but our use-case is for internal enterprise apps used by our field service teams. Since Browsers can access the Geolocation APIs, it's probably straightforward to add that info. It's just a matter of knowing the right way to do it so it's picked up consistently.
I never got the right pricing policy from Google. It's a little confusing for me. I'm currently testing google compute engine to try to understand how it all works.
In a simple example when using Cloud Laucher Wordpress there is a sustainable forecast of $4,49, Using a machine Instance of the VM: 1 shared vCPU + 0.6 GB of memory (f1-micro) and standard 10G disk.
In less than 10 days of testing where I am the only user, where the instance became connected throughout the period and my use was very little. I began tracking billing details.
Look at the numbers:
Generic Micro instance with burstable CPU, no scratch disk 4.627 Minutes $0,62
Storage PD Capacity 1,92 GB-month $0,08
And my big surprise
Network Internet Egress from Americas to Americas 12,82 GB $1,54
I am aware that this value is very small, this is very clear
But imagine an example 100 people making use in the same period
Network Internet Egress from Americas to Americas Would jump $ 154,00
Is my reasoning correct?
Is there a way to lower this value?
Another doubt.
Which has the lowest cost Google compute engine or Google app engine?
Don't buy web server server in cloud platform unless you know the pricing strategy inside out.
Yes, GCS and other cloud platform charge a hefty sum for Egress/outgoing traffics if you are not careful, e.g. if you get DDoS on your website, you will be doomed with a huge bills. As shown in the table, GCS is charging $0,12/GB egress
# Europe using "," as decimal separator
12,82GB x $0,12/GB = $1,5384 ~ $1,54
If you expect more traffics, then you should look into Google cloud CDN service offer, which charge a lower egress price. However, you need to be careful when using those CDN, they will charge ingress traffics (so only allow traffics between your storage repo with the CDN).
It is a good idea to setup alarm/event alert services to warn you about abnormal traffics.
Since you are in the cloud, you should compare prices of CDN services
(Update)
Google App Engine(GAE) is just a Platform as a Services, which Google will give you daily free resources, i.e. 1GB egress per day. $0,12/GB price still apply if you use above the limit. In addition, you are limited to the offering given, there is not web services GAE at the moment.
I am doing the network traffic analysis for a live camera device from a Chinese company which allows to view the real time video from ios6 app.
172.17.200.2 is the IP of the ios device.And the vendor of the camera claims that they dont store the video on their server.
However, the network traffic analysis shows that the app will upload exactly the same amount of data to different servers after it receives from one server.
Does it mean that they are storing the video data to their servers?