For an university project (Big Data lecture), I’d like to analyze auctions on eBay. I wasn’t able to find reliable information so far whether it’s possible to get all current auctions on eBay via their API or not. I only need the auction title and the current price and I am aware that this is a huge load of data, but I’m just curios.
I don't think it's possible, in part because of the huge amount of data, and perhaps also because I don't think eBay wants people downloading data en masse like that. Doing so might allow people to do data mining and market research from a vantage point that is too publicly revealing for them.
If you're willing to settle for a large segment of data, look into eBay's Large Merchant Services and their LMS API.
For your research project, you should be able to make sense of an even smaller subset of data by just pulling from eBay's Finding API in a few automated large chunks.
Running NopCommerce 3.40
I think i read somewhere that NopCommerce's Shopping cart is PCI compliant. It also looks like NopCommerce's shopping cart stores Credit card data.
1.) So what does this PCI process do?
2.) Is it just a fashion of encryption of data as it passes to the database?
3.) Is it my fault if the hosted database say from GoDaddy gets hacked for a bunch of credit cards?
I don't know why NopCommerce is storing Credit Cards. I think I have to figure out how to disable that part. I mean if I don't then I need to purchase a PCI Certificate?
EDIT :
I did some more reading. It looks like if I use Stripe or another third party gateway no Credit card information gets stored. I think only Manual Payments credit card information gets stored. Is this right?
1) PCI-DSS stands for Payment Card Industry Data Security Standard. The standard guides you so you know what are you doing with the cards data and that it is secure enough, given your transaction volume, data collected and stored, etc.
2) It is not. Basically, it says that if the card data (CC number, CVV code) ever pass near your system (in a logical sense), then you are in the PCI scope, and have to comply or the acquirers (banks) won't talk to you.
3) I think you'd be liable, look into GoDaddy disclaimers.
I doubt there's anything like 'PCI compliant shopping cart'. This is how you process the payment, as in, can your javascript access the CC number input box, can your router see the unencrypted CC number, etc.
In your case, I assume a small e-commerce, I'd look to outsource the payment page. Be it Stripe, PayPal, Skrill, Neteller, whatever makes you never see the data except for the income amount.
I am creating a Recurring payment profile using Direct payment(credit card details entered during registration are not stored in database). I want to update/change the bill amount for every bill cycle(because billing is done per user per month).
I am using UpdateRecurringPaymentsProfileDetails API method to change the amount, but the method needs credit card details again(for each time it is called). I am getting the credit card details using GetRecurringPaymentsProfileDetails, but this method just returns the last 4 digits of credit card(and does not return CVV2, but this is not major issue here).
Is there any other option to get the full credit card details(at least credit card number) using Paypal APIs or should I have to store credit card numbers in my database? Any help would be highly appreciated.
It's unlikely that you'll be able to get the full credit card number from PayPal; that's why people use PayPal.
Storing credit card information in your database is a huge liability, and should be avoided if at all possible. Once you start storing that information, your company is responsible for keeping that information secure (and that includes developers). Use another third party component if you must, but don't take on such an enormous liability.
I don't know what company you work for, but that's not a decision that a developer should be making on their own. Entire companies have been shut down for mishandling credit card information. The law around that stuff is extremely murky, and even the smallest mistake can result in massive fines.
The company I work for has just purchased 4 32" LCD screens to be mounted at the front of the office for demonstration purposes. Whilst we are not demonstrating (most of the time), the screens are to be used as development information screens for the whole team.
What information would people recommend displaying to be most useful to the team? Our focus is on hosted business web-apps but I am interested in what other teams doing other types of development find useful too. Pointers on how to gather the displayed information would be useful also.
Information about your continuous integration status.
Major Development Milestones that have been hit in the last week
Releases within the last month (including a short description why this release is awesome)
Use it as motivational board. The achievements of software development are seldom communicated well enough.
Since you're hosting apps for your customers, server and network status information would probably be useful.
Heck, why not create a "chat room" for the dev team to discuss issues and post a streaming version of that as well?
Schedule information, Scrum notes from that morning, a gantt chart...the possibilities abound.
Outstanding bugcount, sorted by priority and severity. You can likely get this from your bugtracking tool programmatically.
Depending on your process management
system, possibly a list of feature
requests and the percentage complete
on each of them. Again, you can probably get this programmatically from your process management / time tracking tool.
Time spent in the current development
cycle, and time remaining. Again, this should be available from your process / management / time tracking tool. You may want to use this data with your bugcounts as well to give a bugs / day fix rate.
If you're a public company with a
profit-sharing plan (i.e. stock or
options), the current price of the
stock (this can be surprisingly
strongly motivating). You can get stock data from several sources online programmatically (although a small delay may be injected unless you're paying for the service).
The movie 'Office Space'
Weather radar from intellicast.com
Latest Checkin.
Number of checkins per day
Number of customers that use software
Metrics on Bugs found/fixed and the ratio.
One screen could be an aggregated RSS feed of development topics pulled from sites such as Stack Overflow (or even Coding Horror). Not sure what your goal for these screens is, but I could see it useful to me if you had a feed with topics specific to your development team headlined. If I were there, I'd glimpse them, maybe catch an interesting thread, and go learn something. Funnel a bunch of keywords and tags through a Yahoo Pipe and dump it to the screen.
That's if they are more "informal and informational."
I think most popular pages from your webapp(s) would be a fun/interesting thing to show on a big monitor up front.
Another would be a live feed of your error reporting.
We have one monitor showing all meetings for the day, with start-end, subject, and room. I find this helpful, not only for my orientation, but also to see what other people do at our company.
xkcd, bunny, dilbert and savage chickens :-)
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 7 years ago.
Improve this question
I'm trying to determine why an enterprise wouldn't want to use Google Analytics.
Here are the main reasons I've seen mentioned:
Inability to track clients that have Javascript disabled.
Lack of ownership of the statistics - Google owns the data.
Most of the web clients with Javascript disabled will probably be bots/spiders. This data is interesting, but probably not very useful.
As for the ownership issue, this is a bit paranoid IMO.
What am I missing here? When is Google Analytics not good enough?
Here are my findings from additional research:
Google Analytics is limited to 5 million page views per month - source
If a web site generates more than 5 million pageviews per month it will need linked to an active AdWords account to avoid interruption of service.
Lack of / slow technical support
All Google support is handled through email and response times can take a week or more. Commercial analytics products often have much faster & personalized support.
Inability to track files (PDF's, Images, etc.)
GA relies on Javascript and files lack the ability to execute Javascript. The workaround to this problem is to tag the link, but this won't track requests that go directly to the file.
Limited ability to customize
This is a selling point that I see pushed by commercial analytics tools (WebTrends). However it's never explained what customizations are denied by GA but allowed by WebTrends.
The Google Analytics EULA does not allow you to track individual users by identifying them. So if you wanted to add a custom variable for username to track how many times each user logs in, then you would be in a gray zone if not outright violating the EULA.
I use Google Analytics on about 10 sites right now and it's a great tool. In addition to all the analytics stats, you can tie it in with AdSense and it becomes a marketing/revenue tool and not just "wow look at all these cool user stats". If there was a way to track by user ID in certain circumstances (e.g. if user's agreed to it, or if they work for the company that owns the site) then I would have no issues.
Besides, it's free and all you have to do is add JavaScript to the files, so give it a try and see what you think after a few months.
One reason that was, surprisingly, not posted:
timing / speed of reaction
It takes at least 4 hours (up to 24) for GA to update your data.
This is ok for me personally in most of the cases, but when reacting fast is crucial (news sites, one-off events, etc.) you may want to employ some other solution (Mint comes to mind, but it's not the only one out there of course).
Thought I'd add my two pence worth to this thread, as this a topic close to my heart and one I've debated with colleagues for years. We've used webtrends in house for as long as i can remember, back to version 4 of the log analyzer (how different things were back then!). Since Google Analytics came along, we've started to come under increasing pressure from certain parts of our business to switch, as 'it does everything we need form an analytics tool'
Well, true in many senses it does, especially these days. But I championed the integration of our CRM and web analytics tools back in 2006, and as our business isn't e-commerce (the 'conversion' happens offline, sometimes months after the visitor acquisition) we need to integrate in this way to get a true picture of campaign effectiveness, and notion of ROI.
All of this means, we need access to the raw data, need to be able to join visitor records on sessionID etc, without this access we'd be screwed. I'd love it if we could roll without it, but the current requirements mean we can't, so this alone is a HUGE reason why Google analytics is not good enough.
Over and out
For tracking desktop software or creating a whitelabel solution there are better solutions.
For white label an integration based analytics, i use MixPanel. For Desktop Software, i use Deskmetrics
Google Analytics does not work well with mobile phones. While the iPhone and the Palm may be supported, many of the existing handsets do not support the javascript that Google uses.
If you're based in the UK, then theoretically you could be breaking the Data Protection Act by using Analytics.
If information about your users (like which web pages they're looking at) goes "outside the European Economic Area" and onto Google's servers in the US, then you're breaking the DPA.
Pretty obscure, but you did ask :)
Piwik avoids the problem because you host it on your own servers.
Lack of ownership of the statistics - Google owns the data.
... As for the ownership issue, this is a
bit paranoid IMO.
One problem with it is that we can't even access the raw data. We had a use case this week where we wanted a visitor map for an executive presentation. We needed to get more flexible with how the visitor map is displayed (wanted to view the map in Google Earth plug-in). In GA, you can't. You take what they give you. You can see a map of how many visits came from each city, but you can't export a data file of cities and number of visits, to run the data through other tools. So, paranoia aside, there are significant limitations on what you can accomplish with GA.
However this is not a problem if you use Urchin, the self-hosted version of GA: you can export the data and do what you want with it. (And the exported data is richer than the web server log's, as it includes some analysis already.)
Since Piwik is open source, and pluggable, I imagine you could enhance the visitor map plug-in any way you wanted to. And export whatever data you want.
Whether this limitation affects you depends on your needs, obviously.
Update: I've now looked at the GA Data Export API, and it turns out that things you cannot do through the UI (as you can with Urchin), you can do with this API. It does look like you can export the visit data I was talking about, via a feed (although there are daily traffic caps on those requests). So sprinkle salt heavily on what I wrote above.
A couple more points that I've come across:
GA doesn't let you dig beyond full-day statistics; I would often like the ability to investigate whether a traffic dip the previous day was caused by the design update I did at 1pm or the soccer match on TV at 8pm.
GA doesn't offer a workaround for traffic spikes caused by DDoS attacks, Slashdotting etc. When I'm looking at a GA visitor graph of 2009, all I can see is the 2-million-pageview-spike on October 16th, pushing the entire rest of the year down flat against the horizontal axis of the graph. To get a meaningful graph, GA should offer the ability to trim or exclude outlying data points, or the ability to limit/bracket the graph window itself
GA doesn't have an event monitoring client (think Reinvigorate's Snoop tool)
While GA is very user-friendly, I've found it's not as granular as some of the other stats programs (or maybe I'm not looking in the right places). Before the marketing monkeys I work with began pushing GA, we were very satisfied with AWStats. The sheer scope of the data helped us on several occasions hone sites to better suit their audience. While GA is very shiny and laid out well, I personally still prefer the raw numbers like I used to get through AWStats.
Slow data processing speed - Can be as low as 15-30 mins for page views, but may be up to 48 for eCommerce
EULA is limiting in some cases
You won't own or have any control of the data. Google's engineers might use it (anonymously) for testing
Anything more complex requires customization - Downloads and such care of no issue, but there are limits
Cross domain tracking by linker is faulty at best
Visit based - Proper tools are based on Visitor level, GA works on Visit based reporting mostly
Limited number of custom vars used at one time (5)
No tech support, if you're realistic
Usually when there is a downtime notice, it's already gone
API limitations (4 dimensions and 10 metrics at one time, not all can be used together in addition to that)
I have many more, but at the end of the day it is a good tool for it's price.
From the non-technical point, I think the most important is that some enterprise has the high level data security policy. All of the data should be controlled and managed by themselves.
If you use the Google analytics,the data is stored in google's server. For some special enterprise, like insurance, financial company. The policy should be followed.
I would NOT go with server logs. In fact I have them disabled on my server. Why you ask me?
For the simple reason that everytime you hit my server that stupid logging program makes an entry in the physical log file on my HDD. So if my server gets 100,000 hits in a day that's 100,000 time a HDD write operation happens.
You think that's cool? Well it's not. It's slowing your server down, specially if the log file is huge.
Why would someone even consider doing that to their server? Specially when we're working so hard to minify javascript, css and make image files 2 KB smaller!
Please do yourself a favor don't log directly on your server.
At least Google Analytics logs it on Google's server so my server's healthier.
I wouldn't use it for any of my sites, because you're forcing the user to accept your proprietary JavaScript code in their browser, which is bad. Also, giving your data is Google is a really bad idea.
See Piwiki for something you can run yourself as in free software, eliminating both of the problems.