Creating random clicks and moments to simulate human behavior- Scrapy - web-scraping

I'm looking for some help in improving my spider so it won't repeat obvious patterns of grabbing data making the activity look like bots. So basically random mouse movements, clicks, and maybe some scrolling. Every article I've read suggests this, but I've yet to see any real code implementations. So if anyone has ever incorporated this practice I'd really appreciate some pointers.
So far I've been looking into pyautogui and Bézier curves with no luck.

Related

What's the best method to implement multiplayer on a Unity Billiard game?

I'm making an online billiard game. I've finished all the mechanics for single player, online account system, online inventory system etc. Everything's fine but I've gotten to the hardest part now, the multiplayer. I tried syncing the position of each ball every frame but the movement wasn't smooth at all, the balls would move back and forth and it looked "bad" in general. Does anyone have any solution for this ? How do other billiard games like the one in Miniclip do it, I'm honestly stuck here and frustrated as it took me a while to learn Photon networking then to find out it's not that good at handling the physics synchronization.
Would uNet be a better choice here ?
I appreciate any help you give me. Thank you!
This is done with PUN already: https://www.assetstore.unity3d.com/en/#!/content/15802
You can try to play with synchronization settings or implement custom OnPhotonSerializeView (see DemoSynchronization in PUN package). Make sure that physic simulation disabled on synchronized clients. See DemoBoxes for physics simulation sample.
Or, if balls can move along lines only, do not send all positions every frame. Send positions and velocities only when balls colliding and do simple velocity simulation between. This can work even with more comprehensive physics but general rule is the same: synchronize it at key points. Of course this is not as simple as automatic synchronization.
Also note that classic billiard is turnbased game and you do not have all the complexity of players interaction. In worst case you can 'record' simulation on current player client and 'playback' it on others.

Grid exploration robots with no visions and completely blind

I am doing a research on how grid explorer robots move.
most articles are about robots with no vision but with sensors to see state of all surrounding cells, but my robot has really no vision and only can sense a cell after it could or failed to explore it(imagine the sensors are 4 pushed buttons that will be pushed when it hits the obstacles.)
I was hoping someone could lead me to some clues of how to find papers about this kind of robots. I am really having a hard time figuring out how to find related information.
If you have a priori knowledge of the environment, but imperfect ability to sense your location in the environment (and probably imperfect actions) then what you're describing may be a partially observable markov decision process (POMDP.) Your four-button sensor fits nicely into that idea.
If you don't even have prior knowledge of the environment, then you need to augment the POMDP notion with elements of exploration or machine learning.
Note that POMDPs are geared toward scenarios with "rewards", and that while POMDPs are pretty well understood, efficient solutions are still a topic of research. As are ML/POMDP hybrids.
Environmental exploration and traversal with just push-button sensors for obstacle awareness, and without priori knowledge sounds to me like a Simultaneous localization and mapping (SLAM) problem.
If I recall correctly, Roombas use this, so there should be a fair bit of study done on it in that regard.

Scientific Algorithms that can produce imagery, pseudocode perhaps?

I have a client who is based in the field of mathematics. We are developing, amongst other things, a website. I like to create a mock-up of a drawing tool that can produce some imagery in the background based on some scientific algorithms. The intention being that the client, later, may create their own. (They use emacs for everything, great client.)
I'm looking for an answer about where or what to search. Not code specific, pseudocode even, as we can adapt and have not yet settled on a platform.
I'm afraid my mathematics stops at the power of two and some trigonometry. Appreciated if they're are any mathematics related students/academics how could enlighten me? What to search for will be accepted?
Edit: To summarise/clarify, I want to draw pretty pictures (the design perspective). I want them to have some context (i.e. not just for the sake of pretty images but have some explanation available). In essence I would to create a rendering engine which they can draw/code the images and we set the style parameters: line, colour, etc... But to pursue this option I want to experiment myself.
Edit: great responses thanks. The aim is to make something along the lines of http://hascanvas.com/ if anyone is interested.
Thanks
Ross
Mandelbrot set, Julia sets, random graphs, Lorenz attractor.
Maybe minimising energy functions on a sphere.
I'm quite sure that I don't fully understand what you are after, so to provoke you and others into clarifying, I suggest you grab a copy of Mathematica and of Web Mathematica and knock your clients out with that.
Mandelbulb.
Fractals with pseudo code.
You can have a look at these links:
https://mathshistory.st-andrews.ac.uk/Curves/
https://www.nctm.org/classroomresources/
https://planetmath.org/famouscurves

Convert graph in to data points using Mechanical Turk?

I looked around but did not see anyone using Mechanical Turk for this. I've heard of the service, but never used it before. I need to take the following graph and digitize it so I get a list of data points for each line (noting that there are two Y-axes, and thus depends on which line we are talking about). This is pretty time consuming for me, and I saw other posts on StackOverflow about digitizing software doing a poor job at this. Would Mechanical Turk be well suited to my task?
Here is the graph for reference: http://www.yourpicturehost.com/dyno_hbspeed.jpg
Depends how many of these you have. Mechanical turk could work quite well, but you'd have to check the accuracy carefully (eg by re-plotting the graphs, and comparing them yourself).
If you have a lot, though - you should be able to design an image processing algorithm to pick up the data.

How To: Pattern Recognition

I'm interested in learning more about pattern recognition. I know that's somewhat of a broad field, so I'll list some specific types of problems I would like to learn to deal with:
Finding patterns in a seemingly random set of bytes.
Recognizing known shapes (such as circles and squares) in images.
Noticing movement patterns given a stream of positions (Vector3)
This is a new area of experimentation for me personally, and to be honest, I simply don't know where to start :-) I'm obviously not looking for the answers to be provided to me on a silver platter, but some search terms and/or online resources where I can start to acquaint myself with the concepts of the above problem domains would be awesome.
Thanks!
ps: For extra credit, if said resources provide code examples/discussion in C# would be grand :-) but doesn't need to be
Hidden Markov Models are a great place to look, as well as Artificial Neural Networks.
Edit: You could take a look at NeuronDotNet, it's open source and you could poke around the code.
Edit 2: You can also take a look at ITK, it's also open source and implements a lot of these types of algorithms.
Edit 3: Here's a pretty good intro to neural nets. It covers a lot of the basics and includes source code (albeit in C++). He implemented an unsupervised learning algorithm, I think you may be looking for a supervised backpropagation algorithm to train your network.
Edit 4: Another good intro, avoids really heavy math, but provides references to a lot of that detail at the bottom, if you want to dig into it. Includes pseudo-code, good diagrams, and a lengthy description of backpropagation.
This is kind of like saying "I'd like to learn more about electronics.. anyone tell me where to start?" Pattern Recognition is a whole field - there are hundreds, if not thousands of books out there, and any university has at least several (probably 10 or more) courses at the grad level on this. There are numerous journals dedicated to this as well, that have been publishing for decades ... conferences ..
You might start with the wikipedia.
http://en.wikipedia.org/wiki/Pattern_recognition
This is kind of an old question, but it's relevant so I figured I'd post it here :-) Stanford began offering an online Machine Learning class here - http://www.ml-class.org
OpenCV has some functions for pattern recognition in images.
You might want to look at this :http://opencv.willowgarage.com/documentation/pattern_recognition.html. (broken link: closest thing in the new doc is http://opencv.willowgarage.com/documentation/cpp/ml__machine_learning.html, although it is no longer what I'd call helpful documentation for a beginner - see other answers)
However, I also recommend starting with Matlab because openCV is not intuitive to use.
Lot of useful links on this page on computer vision related pattern recognition. Some of the links seem to be broken now but you may find it useful.
I am not an expert on this, but reading about Hidden Markov Models is a good way to start.
Beware false patterns! For any decently large data set you will find subsets that appear to have pattern, even if it is a data set of coin flips. No good process for pattern recognition should be without statistical techniques to assess confidence that the detected patterns are real. When possible, run your algorithms on random data to see what patterns they detect. These experiments will give you a baseline for the strength of a pattern that can be found in random (a.k.a "null") data. This kind of technique can help you assess the "false discovery rate" for your findings.
learning pattern-recoginition is easier in matlab..
there are several examples and there are functions to use.
it is good for the understanding concepts and experiments...
I would recommend starting with some MATLAB toolbox. MATLAB is an especially convenient place to start playing around with stuff like this due to its interactive console. A nice toolbox I personally used and really liked is PRTools (http://prtools.org); they have an implementation of pretty much every pattern recognition tool and also some other machine learning tools (Neural Networks, etc.). But the nice thing about MATLAB is that there are many other toolboxes as well you can try out (there is even a proprietary toolbox from Mathworks)
Whenever you feel comfortable enough with the different tools (and found out which classifier is perfomring best for you problem), you can start thinking about implementing the machine learning in a different application.

Resources