Known Ethereum Addresses - web-scraping

Does anyone know where I can find a list of all known ethereum or other cryptocurrency addresses?Currently building a scraper that searches an address on etherscan (https://etherscan.io/ ) and returns info. Anyone know of a better way to do this?

this was the most useful https://etherscan.io/labelcloud; however, is not completely consistent. As in there are wallets that are classified as X which do not appear under the label X. e.g. Yobit.net does not appear on the exchange label but is classified as an exchange if you look for it manually.

Related

searching address (geocode.search.hereapi.com) weird behavior

I'm using https://geocode.search.hereapi.com/v1/geocode?q to search for address in USA.
When I give q=765 Rive I get lot of address but the moment when I give 765 River My results are very restricted and I get only very less responses and which does not include lot of addresses I got in the previous q search.
Please check now. When I trying with the postman .It is giving the same number count in the response.
https://geocode.search.hereapi.com/v1/geocode?q=765 Rive&apikey={your_api_key}
https://geocode.search.hereapi.com/v1/geocode?q=765 River&apikey={your_api_key}
And Please follow below document, which explains how we can formulate a qualified query.
https://developer.here.com/documentation/geocoding-search-api/dev_guide/topics-api/code-geocode-qualified.html

Geolocation of BGP Automous Systems

Hi friends I've been looking around for the past few days on a way to find the geolocation of the BGP AS's, preferably through the use of some API. I've been using the ripestat API for the majority of my work on this, but it comes up inconclusive on some of the AS's, for example AS 10000. RIPE tells me the location is in JP. Which is sort of fine, I just would like to narrow it down more to like a city / postal code / etc if possible. Is there another API suited for this? or is it just a manual task of fixing all the information once gathered.
Alternatively, if it is possible to grab the IP address of the actual AS itself, and not the range, that would likely work as well.
IP Geolocation isn't nearly accurate enough to pinpoint an IP to a specific City/ZIP code. In many cases, IPs from the same block will be used across a large area in an ISP's control, so it's not possible to be very accurate. Autonomous Systems don't really have "an IP", as there's no one specific location of them.
If you're looking for the locations where they peer to other providers, you might want to check out PeeringDB.

Finding the number of common users between two websites

There are two Swiss (.ch) websites, let's call them A and B. A is owned by me and B by a customer.
Because of legal data protection issues B is hosted in Switzerland and not allowed to store any user information abroad. Which means that software like Google Analytics is not available on B. A is a Swiss website but hosted in a (European) cloud.
Now we would like to find out how many common users we both have over the duration of 30 days. In short:
numberOfUsersA ∩ numberOfUsersB
For the sake of simplicity: Instead of users we are perfectly happy to measure common browsers.
What would you suggest is the simplest way to solve this problem?
First off all, best regards from Zurich/Zug :) Swiss people are everywhere...
I don't think you're correct that it's not legal to collect data in Switzerland at all (also abroad). As I'm working in the financial industry I know this topic very well and we also had to do a lot research to use GA at all.
It's always the question what and how you collect data. What you can't do - beside you got in upfront the permission of the user - is storing personal identifiable information. That's anyway not allowed by GA - you can't import/save in custom dimension/metrics for example email addresses.
Please check https://support.google.com/adsense/answer/6156630?hl=en as general basic information about this topic.
If you save the IP addresses via IP anonymization, you shouldn't run into problems if you're declaring this in your data-privacy statements. Take this approach: https://support.google.com/analytics/answer/2763052?hl=en
I'm not a lawyer and also not want to give you legal advises, but ours told us that's fine. If you are real paranoid about sending data to the USA - like we have to be - you can exclude your tracking from very sensitive forms.
To go back to your basic question, if you want to find this out via Google Analytics, your key is "cross domain tracking". Check https://support.google.com/analytics/answer/1034342?hl=en for more information in this direction.
The only work-around I have in my mind beside this, is if you start collecting browser-fingerprints yourself and then connect both collections over the finger prints together (that's not save, as your visitors will use more than one device/configuration). I personally would go for the IP anonimization, exclude very sensitive forms and ensure that your data-privacy declaration contains all necessary parts for and offer an opt-out option then you should be on the safe side.
All the best and TGIF :)

how to spoof location so google autocomplete API will provide local results, ideally with R

google has an API for downloading search suggestions:
https://www.google.com/support/enterprise/static/gsa/docs/admin/70/gsa_doc_set/xml_reference/query_suggestion.html
unfortunately, as far as i can tell, these results are specific to your location. for an analysis, i would like to be able to define the city/location that google thinks it is making the suggestion to. here's what happens when i scrape from dar es salaam, tanzania:
http://suggestqueries.google.com/complete/search?client=firefox&q=insurance
["insurance",["insurance","insurance companies in tanzania","insurance group of tanzania","insurance principles","insurance act","insurance policy","insurance act tanzania","insurance act 2009","insurance definition","insurance industry in tanzania"]]
i understand that a vpn would partially solve this issue, but only by giving me a different location and not lots of locations. is there a reasonable way to replicate this sort of thing quickly and easily from, say, the 100 largest cities in the united states?
confirmation that results differ within the usa-
thanks!
Google will use your IP and your location history (if turned on) to determine your location.
To be able to go around it, you can spoof your IP while logged off your google account (but I don't know if google will consider it a trial of hacking no matter what your intentions are).
Another way is to use Tor browser (even though it is not it's original purpose). You can configure tor to exit from a certain country using the Exitnode parameter in the torrc config file
As found in the docs:
ExitNodes node,node,…
A list of identity fingerprints, country codes, and address patterns of nodes to use as exit node
But if you want a fast way to do it, I don't think that's possible since google wants to know the real location of the users and have put a lot of effort into making such tricks fail.
The hl param for interface language changes the search results, but I can't tell if it's actually changing the location. For example:
http://suggestqueries.google.com/complete/search?client=chrome&q=why&hl=FR
Here's an example with 5 different values of hl:
http://jsbin.com/tusacufaza/edit?js,output

How to allow only IP address from a town to view a server

I am trying to create a specific webpage, and I want to only allow people in my town to view it, so that later on I can expand on it and let surrounding towns view it. First, is this even possible, and second, if so, what language would be used to do this?
An IP address does not provide that kind of information but there are services out there that can "geolocate" an address based on information they've collected. I'm not sure how accurate it will be, though, so may not be able to identitify a specific town.

Resources