Anyone know how to send multiple chained commands via SabreCommandLLSRQ HostCommand?
I'm trying to avoid the network back and forward (gets slow when you need 50 commands to get one full page)
Ex:
FQYYZBKK15JAN-AC + all the necessary move downs, or get the fare rules for all the 99 or so fares there.
Sounds like you want to scrape all available Fares + the attributes (penalties, baggage etc). I cant clarify because of my reputation.
If so you might be better of with Vayant or some caching company of Fares as it is way cheaper (you are producing tons of looks but no books and might brake the limit (look/book) that has been agreed upon with sabre.
If you want to stay with sabre you should use: FareLLSRQ also called Air Fare by City Pairs + OTA_AirRulesLLSRQ
Related
There are two Swiss (.ch) websites, let's call them A and B. A is owned by me and B by a customer.
Because of legal data protection issues B is hosted in Switzerland and not allowed to store any user information abroad. Which means that software like Google Analytics is not available on B. A is a Swiss website but hosted in a (European) cloud.
Now we would like to find out how many common users we both have over the duration of 30 days. In short:
numberOfUsersA ∩ numberOfUsersB
For the sake of simplicity: Instead of users we are perfectly happy to measure common browsers.
What would you suggest is the simplest way to solve this problem?
First off all, best regards from Zurich/Zug :) Swiss people are everywhere...
I don't think you're correct that it's not legal to collect data in Switzerland at all (also abroad). As I'm working in the financial industry I know this topic very well and we also had to do a lot research to use GA at all.
It's always the question what and how you collect data. What you can't do - beside you got in upfront the permission of the user - is storing personal identifiable information. That's anyway not allowed by GA - you can't import/save in custom dimension/metrics for example email addresses.
Please check https://support.google.com/adsense/answer/6156630?hl=en as general basic information about this topic.
If you save the IP addresses via IP anonymization, you shouldn't run into problems if you're declaring this in your data-privacy statements. Take this approach: https://support.google.com/analytics/answer/2763052?hl=en
I'm not a lawyer and also not want to give you legal advises, but ours told us that's fine. If you are real paranoid about sending data to the USA - like we have to be - you can exclude your tracking from very sensitive forms.
To go back to your basic question, if you want to find this out via Google Analytics, your key is "cross domain tracking". Check https://support.google.com/analytics/answer/1034342?hl=en for more information in this direction.
The only work-around I have in my mind beside this, is if you start collecting browser-fingerprints yourself and then connect both collections over the finger prints together (that's not save, as your visitors will use more than one device/configuration). I personally would go for the IP anonimization, exclude very sensitive forms and ensure that your data-privacy declaration contains all necessary parts for and offer an opt-out option then you should be on the safe side.
All the best and TGIF :)
google has an API for downloading search suggestions:
https://www.google.com/support/enterprise/static/gsa/docs/admin/70/gsa_doc_set/xml_reference/query_suggestion.html
unfortunately, as far as i can tell, these results are specific to your location. for an analysis, i would like to be able to define the city/location that google thinks it is making the suggestion to. here's what happens when i scrape from dar es salaam, tanzania:
http://suggestqueries.google.com/complete/search?client=firefox&q=insurance
["insurance",["insurance","insurance companies in tanzania","insurance group of tanzania","insurance principles","insurance act","insurance policy","insurance act tanzania","insurance act 2009","insurance definition","insurance industry in tanzania"]]
i understand that a vpn would partially solve this issue, but only by giving me a different location and not lots of locations. is there a reasonable way to replicate this sort of thing quickly and easily from, say, the 100 largest cities in the united states?
confirmation that results differ within the usa-
thanks!
Google will use your IP and your location history (if turned on) to determine your location.
To be able to go around it, you can spoof your IP while logged off your google account (but I don't know if google will consider it a trial of hacking no matter what your intentions are).
Another way is to use Tor browser (even though it is not it's original purpose). You can configure tor to exit from a certain country using the Exitnode parameter in the torrc config file
As found in the docs:
ExitNodes node,node,…
A list of identity fingerprints, country codes, and address patterns of nodes to use as exit node
But if you want a fast way to do it, I don't think that's possible since google wants to know the real location of the users and have put a lot of effort into making such tricks fail.
The hl param for interface language changes the search results, but I can't tell if it's actually changing the location. For example:
http://suggestqueries.google.com/complete/search?client=chrome&q=why&hl=FR
Here's an example with 5 different values of hl:
http://jsbin.com/tusacufaza/edit?js,output
I think the question has been answered here before,but i could not find the desired topic.I am a newbie in web scraping.I have to develop a script that will take all the google search result for a specific name.Then it will grab the related data against that name and if there is found more than one,the data will be grouped according to their names.
All I know is that,google has some kind of restriction on scraping.They provide a custom search api.I still did not use that api,but hoping to get all the resulted links corresponding to a query from that api. But, could not understand what will be the ideal process to do the scraping of the information from that links.Any tutorial link or suggestion is very much appreciated.
You should have provided a bit more what you have been doing, it does not sound like you even tried to solve it yourself.
Anyway, if you are still on it:
You can scrape Google through two ways, one is allowed one is not allowed.
a) Use their API, you can get around 2k results a day.
You can up it to around 3k a day for 2000 USD/year. You can up it more by getting in contact with them directly.
You will not be able to get accurate ranking positions from this method, if you only need a lower number of requests and are mainly interested in getting some websites according to a keyword that's the choice.
Starting point would be here: https://code.google.com/apis/console/
b) You can scrape the real search results
That's the only way to get the true ranking positions, for SEO purposes or to track website positions. Also it allows to get a large amount of results, if done right.
You can Google for code, the most advanced free (PHP) code I know is at http://scraping.compunect.com
However, there are other projects and code snippets.
You can start off at 300-500 requests per day and this can be multiplied by multiple IPs. Look at the linked article if you want to go that route, it explains it in more details and is quite accurate.
That said, if you choose route b) you break Googles terms, so either do not accept them or make sure you are not detected. If Google detects you, your script will be banned by IP/captcha. Not getting detected should be a priority.
What's the best way to track RSS subscribers reliably without using Feedburner? Some of the obvious approaches like tracking by IP or by the number of hits have some fata flaws. IP addresses can change with each request or multiple users can use the same IP. Also, feed readers can request a feed multiple times per day or even hour. Both problems make it really hard to get reliable stats on unique subscribers.
I've read articles by both Leo Notenboom and Tim Bray on the topic, but none of their suggestions seems to really solve how to track subscribers in an accurate and reliable way. Leo suggests having a unique ID generated programatically to be appended to the RSS feed URL for each time the referring page is loaded. Tim advocates having RSS readers generate a unique hashtag and also has suggestions ranging from tracking the referrers to using cookies. A unique URL would be reliable, but it has two flaws: It's not a user-friendly URL and it creates duplicate content for SEO. Are there any other reliable methods of tracking RSS subscribers? How does Feedburner estimate subscribers?
There isn't really a standard way to do this. Subscriber counting is always unreliable but you can get good estimates with it.
Here's how Google does it (source):
Subscribers counts are calculated by matching IP address and feed reader
combinations, then using our detailed understanding of the multitude of
readers, aggregators, and bots on the market to make additional inferences.
Of course part of this is easy for Google, as they can first calculate how many Google Reader users are subscribed to the feed in question. After that they use IP address matching also, and that's what you should use too.
You could calculate individual IP addresses (i.e. unique) from the web-servers logs, but that would count 10 people as 1 if they all use the same address. That's why you should inspect the HTTP-headers which are sent by the client, more specifically header fields HTTP_X_FORWARDED_FOR and HTTP_VIA. You could use the HTTP_VIA address as the "main" address, and then calculate how many unique HTTP_X_FORWARDED_FOR addresses are subscribed to the feed. If the subscriber doesn't have these proxy-added fields, then it's counted as a unique IP address. These should be handled in the code that generates the feed. You could also add a GeoIP lookup for the IP's and store everything to a database. This would allow you to see which country has the most subscribers to your feed.
This has it's problems too. All proxies don't use these fields and it doesn't fix the problem of calculating subscribers behind NAT gateways. It is however a good estimate. Besides, you are probably more interested in the order of magnitude rather than the exact count of subscribers, aren't you? If the counter says that you have 5989 subscribers you probably have more subscribers as the counter gives you the lower bound.
Standard and Reliable are not exactly word in RSS dictionary :-) Got to remember that the thing doesn't even have standard XSD after how many years ? If by tracking you mean the "count" there are a few things you can do and the tactics depend on the purpose i.e. are demonstrating a big number or small number? It is a marketing thing so you have to define your goals :-)
You may have to classify IP numbers for a start - to have the basic collection of big / corporate / umbrella IP numbers. For them, you can use referrer as a reasonable filtering criteria and count everything else as unique unless proven otherwise. Vast majority of IP numbers remain stable for about 2 days but again it always good to use basic referrer logic as a filter for people who just keep "clicking" so to speak.
Then you need a decent list of aggregators and a classification on how they process URLs and if they obscure end readers completely then you need either published or inferred averages - it's always fair game to use equitable distribution of an average count. Using cookies may help to collect aggregator IPs and differentiate between automated agents and individuals.
One very important thing is to keep in mind that you can't use just one method and expect it to be a silver bullet - you need to use these 3-4 aspects at the same time plus basic statistical reasoning.
You could query your web server logs for traffic to your RSS feed, perhaps filter it by IP to get the number of uniques.
The problem is, that would rely on folks checking the feed daily. The frequency of hits to your RSS feed by one individual could vary day to do and the number could be lower.
If you configure your RSS feed to require some kind of authentication, you can do user-based metrics instead of ip-based metrics. Although this would be a technically-correct solution, getting people to opt into an authenticated blog in anything other than an Intranet scenario is a stretch.
I have a client which is shipping via UPS, and therefore cannot deliver to Post Office boxes. I would like to be able to validate customer address fields in order to prevent them from entering addresses which include a PO box. It would be best if this were implemented as a regex so that I could use a client-side regex validation control (ASP.NET).
I realize there's probably no way to get a 100% detection rate, I'm just looking for something that will work most of the time.
UPS also has tools that you can integrate to do this... that way you can verify an address exactly as to whether or not they will ship, what the cost would be, schedules, etc. I suggest visiting the UPS IT Solutions page for more information.
This should get you started. Test to see if the Address field matches this regex.
"^P\.?\s?O\.?\sB[Oo][Xx]."
Translation to English: That's a P at the beginning of the line, followed by an optional period and space, followed by an O, followed by an optional period, followed by a space, followed by "Box", followed by anything else.
You might be better off putting a disclaimer on the page warning that you can not ship to post office boxes, opposed to validating the input.
More than likely if you do create a regex that catches most of the P.O. Box scenarios, there's a good chance it'll also catch things you weren't intending (i.e. a customer with a street name containing the letters 'p' 'o' and 'box')
Unfortunately, UPS's online software allows P.O. Boxes to go through, but will choke on them once they're in the shipping channel.
In our case, our cart abandonment rate went up when we tried to gracefully prevent P.O. Boxes. We found it much more cost effective to leave it alone, accept the sale, bring it to the attention of customer service, and let them resolve it.
Of course, if you get a high incidence of P.O Boxes, this may not be the case for you.
I'd start with a regex ala Lizard (but use the "ignore case" flag :)), test on historical data, then iterate as you see what invalid inclusions and exclusions you see in testing.
Most shipping providers (for example FedEx) will validate the shipping address. For example, with FedEx web services, there is a call to validate a shipping address and get the estimated cost. This not only ensures that the address is not a PO Box, but also makes sure that the rest of the address is valid.
Regarding the OP's comment to Jason Coco's answer:
Since you're in a position to add regex validation to the shipping address, I assume that you have control of the application (i.e., you have the source and can modify it). If that's the case, then you should have the ability to, on reciept of the submitted data, check whether it is to be shipped via USPS, FedEx, or UPS and submit a request to the appropriate shipper-specific address validator, gaining all the benefits suggested in Jason's answer.
By making it shipper-specific, this would also allow you to avoid implementing one-size-fits-all rules, such as "no PO boxes because UPS doesn't deliver to them", even though the user can select non-UPS shippers who do deliver to PO boxes.
What if it doesn't start with "PO Box.." or "P.O. Box" ?
Example:
John Schmidt |
Silver Valley PO Box 3901 |
Whereswaldoville, SI. 78946
I used an onblur event for the address field to use a javascript function, indexOf, to recognize the input.toUpperCase "PO BOX" || "P.O" that is >= 0.
If either of these two searches are not found, the return is -1, otherwise, it will return the string's start position which will always be 0 or more.
This will ensure that lazy typing, 'po box,' 'p.o box,' and as well as 'p.o. box' will be recognized. I suppose you could add 'po. box' as well.
Anyway, the condition triggers an unobtrusive message to show that 'We can't ship to a PO Box address." It's a feature to not see it if it doesn't apply to you. Otherwise, for users who don't have js or css enabled, they'll just see the message. The only fail on this graceful degradation is if a user has css, but not js enabled (where they just won't see the message at all). I only came up with the solution today, but if I think of a better way, I'll come back to post it here.