Basic scraper that collects the description from gallery website - web-scraping

The link: https://vsco.co/chrysc/gallery
Is it possible to create a scraper goes to that link and clicks on each picture, writes the description into a single textfile?
So for example:
the textfile would look like:
Circle Loop #vsco #kusama
#vsco #kusama ✨
#vsco White has eyes?
etc...
I was thinking "BeaustufilSoup4" or "Selenium" could work? Would this be a difficult script and I should just do this process manually?

This script will print all descriptions to the screen and to the file:
import re
import json
import requests
url = 'https://vsco.co/chrysc/gallery'
html_data = requests.get(url).text
data = json.loads(re.search(r'window\.__PRELOADED_STATE__ = (\{.*\})', html_data).group(1))
# uncomment this to see all data:
# print(json.dumps(data, indent=4))
# print description to the screen:
for img in data['entities']['images'].values():
print(img['description'])
# print description to the file:
with open('output.txt', 'w') as f_out:
for img in data['entities']['images'].values():
print(img['description'], file=f_out)
Prints:
Circle Loop #vsco #kusama
#vsco #kusama ✨
#vsco White has eyes?
missing nights like these #vsco
#vsco :)
#vsco
#vsco Water//Alberta
who she is, is a result of who she was
#vsco #chrystene peace out
#vsco #chrystene DO NOT DISTURB
#vsco #chrystene Beautiful Morning
#vsco #chrystene Paradise
#vsco Wandering Fairy Cat ✨💖
#vsco close your eyes so you’re less embarrassed ???
And writes the same contents to the file named output.txt.

Related

tkinter how do i save changes made to the buttons

thx a lot for replying my questions for making my previous code simpler.. this is the result... now for the next phase is save changes made to the buttons.. im still learning though looking for sources anything can help :)
from tkinter import*
import tkinter as tk
import tkinter.simpledialog
def onChange(i):
btn_list[i].config(text='Updating...',bg='red')
btn_list[i].grid(in_=root,row=rw[i],column=2)
ans=tk.simpledialog.askfloat('Updating....', 'What is the current price?')
if ans:
btn_list[i].config(text='RM{:,.2f}'.format(ans))
btn_list[i].config(bg='yellow')
root=Tk()
Title=['Item','Unit','Price']
Item=['Kopi O','Teh O','Teh Tarik']
Unit= '1 cup'
Price=[1,0.9,1.2]
cl=[0,1,2]
rw=[1,2,3]
btn_list=[]
for i in range(3):
btnT1=tk.Button(root,text=Title[i],width=10,bg='light green')
btnT1.grid(in_=root,row=0,column=cl[i])
for x in range(3):
btnT2=tk.Button(root,text=Item[x],width=10)
btnT2.grid(in_=root,row=rw[x],column=0)
for y in range(3):
btnT3=tk.Button(root,text=Unit,width=10)
btnT3.grid(in_=root,row=rw[y],column=1)
for z in range(3):
btnT4=tk.Button(root,text=('RM {:,.2f}'.format(Price[z])),bg='yellow',width=10,\
command=lambda i=z:onChange(i))
btnT4.grid(in_=root,row=rw[z],column=2)
btn_list.append(btnT4)
root.mainloop()
If you want changes to be saved when your program exits and restarts, you'll have to do all of the work yourself. You'll need to write a function that gathers all the data you want to save, another function to write that data to a file or database, a third to be able to read the data from the file or database, and a forth to update the UI with the saved values.

Python Interpreting things from document

So, I am essentially just dreaming up ideas right now.
I was wondering if it was possible to make a python program that can read a document, take a line from the document, make an if/else statement with it (Like if the text on that line is equal to Hello, than say hello back), and then continue onto the next line. I have already kind of done this in a shell fashion but I want to see if it is possible to have python read the line of a document, interpret it, display something, and move on to the next line of the document.
(I am prepared for this post to get tons of -1's for not knowing how to program a lot of python, and probably just not being clear enough. So before you -1, just add a comment saying what you need me to be clear about.)
The version of python of my choice would be 2.5.
Since you don't know any Python, try this:
with open("file.txt") as f:
for line in f:
if line.strip() == "Hello":
print "Hello back"
or without the exception-safe clause:
for line in open("file.txt"):
if line.strip() == "Hello":
print "Hello back"
the strip() removes the ending newline \n from the line
That is actually a very simple task in Python:
file = open("file.txt") # open the file
while True:
word = file.readline() # read a line from the file
print word # print it to the console
if word == "": # if out of words...
file.close() # ...close the file
break # and break from while loop and exit program

Can you unprint a line in R?

I am writing up some data processing stuff and I wanted to have a concise progress status printing a fraction that updates over time on a single line in the console.
To get this done I wanted to have something like this
print(Initiating data processing...)
for(sample in 1:length(data)){
print(paste(sample,length(data),sep="/"))
process(data[[sample]])
#Unprint the bottom line in the console ... !!! ... !!!.. ?
}
Keeping the screen clean and what not. I don't quite know how to do it. I know that there is a R text progress bar but for utilities sake I'm looking for a little more control.
Thanks!
I think your best bet is to do exactly what the R text progress bar does, which is "\r to return to the left margin", as seen in the help file. You'll have to use cat instead of print because print ends with a newline.
cat("Initiating data processing...\n")
for(sample in 1:length(data)){
cat(sample, length(data), sep="/")
process(data[[sample]])
cat("\r")
}
cat("\n")

Cat and Paste for variables UNIX

I have been reading tutorials and I need to use cat and paste functions (for making a kind of array, table) the thing is that all tutorials use these commands on files, and reading files from the hard drive are making my task very very slow, so I wanted to know how to use them with variables, I tried and got erroneous results, so maybe you can help me with the sintaxis using that.
basically I want to make a table in a variable like this:
00001 Tacos
00023 pizza
00076 burger
00103 chopsuey
00167 burrito
01034 Tamales
And I'm getting every element after executing a program and taking specific data from the output, so I'm getting:
00001
Tacos
00023
pizza
....
You dont have to do the program, just wanted to be sure that cat and paste are the ones to use here and their syntax, if they aren't I accept any suggestion.
Sorry, I maybe did not explain myself, sorry about that, I have a and b, both variables, a is 00001 and b y is Tacos, then i want them to merge together and store them in a variable, then do the same again but to put them in a new line. Sorry for the misundertanding.
at the end i want a variable with this in it:
00001 Tacos
00023 pizza
00076 burger
00103 chopsuey
00167 burrito
01034 Tamales
Thanks!
If you have a tool only working on files as input (e. g. diff or paste), you can use the <(…) notation to create a fake file whose contents is created by a shell command:
cat <(echo "hello world")
This will print hello world. The fake file lacks some of the abilities of on-disk files; it cannot be seeked for instance. So programs which want to seek a specific position in the file, for instance to read the file twice, will fail on this. But for your case, it should suffice and you can use stuff like this:
paste <(echo "$a") <(echo "$b")
For your case more concrete:
cat input.txt | {
x=''
y=''
while read a
do
read b
x=$(echo "$x"; echo "$a")
y=$(echo "$y"; echo "$b")
done
paste <(echo "$x") <(echo "$y")
}
(I'm assuming the input to be this here:)
00001
Tacos
00023
pizza
00076
burger
00103
chopsuey
00167
burrito
01034
Tamales

What are the dark corners of Vim your mom never told you about? [closed]

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 11 years ago.
There are a plethora of questions where people talk about common tricks, notably "Vim+ctags tips and tricks".
However, I don't refer to commonly used shortcuts that someone new to Vim would find cool. I am talking about a seasoned Unix user (be they a developer, administrator, both, etc.), who thinks they know something 99% of us never heard or dreamed about. Something that not only makes their work easier, but also is COOL and hackish. After all, Vim resides in the most dark-corner-rich OS in the world, thus it should have intricacies that only a few privileged know about and want to share with us.
Might not be one that 99% of Vim users don't know about, but it's something I use daily and that any Linux+Vim poweruser must know.
Basic command, yet extremely useful.
:w !sudo tee %
I often forget to sudo before editing a file I don't have write permissions on. When I come to save that file and get a permission error, I just issue that vim command in order to save the file without the need to save it to a temp file and then copy it back again.
You obviously have to be on a system with sudo installed and have sudo rights.
Something I just discovered recently that I thought was very cool:
:earlier 15m
Reverts the document back to how it was 15 minutes ago. Can take various arguments for the amount of time you want to roll back, and is dependent on undolevels. Can be reversed with the opposite command :later
:! [command] executes an external command while you're in Vim.
But add a dot after the colon, :.! [command], and it'll dump the output of the command into your current window. That's : . !
For example:
:.! ls
I use this a lot for things like adding the current date into a document I'm typing:
:.! date
Not exactly obscure, but there are several "delete in" commands which are extremely useful, like..
diw to delete the current word
di( to delete within the current parens
di" to delete the text between the quotes
Others can be found on :help text-objects
de Delete everything till the end of the word by pressing . at your heart's desire.
ci(xyz[Esc] -- This is a weird one. Here, the 'i' does not mean insert mode. Instead it means inside the parenthesis. So this sequence cuts the text inside parenthesis you're standing in and replaces it with "xyz". It also works inside square and figure brackets -- just do ci[ or ci{ correspondingly. Naturally, you can do di (if you just want to delete all text without typing anything. You can also do a instead of i if you want to delete the parentheses as well and not just text inside them.
ci" - cuts the text in current quotes
ciw - cuts the current word. This works just like the previous one except that ( is replaced with w.
C - cut the rest of the line and switch to insert mode.
ZZ -- save and close current file (WAY faster than Ctrl-F4 to close the current tab!)
ddp - move current line one row down
xp -- move current character one position to the right
U - uppercase, so viwU upercases the word
~ - switches case, so viw~ will reverse casing of entire word
Ctrl+u / Ctrl+d scroll the page half-a-screen up or down. This seems to be more useful than the usual full-screen paging as it makes it easier to see how the two screens relate. For those who still want to scroll entire screen at a time there's Ctrl+f for Forward and Ctrl+b for Backward. Ctrl+Y and Ctrl+E scroll down or up one line at a time.
Crazy but very useful command is zz -- it scrolls the screen to make this line appear in the middle. This is excellent for putting the piece of code you're working on in the center of your attention. Sibling commands -- zt and zb -- make this line the top or the bottom one on the sreen which is not quite as useful.
% finds and jumps to the matching parenthesis.
de -- delete from cursor to the end of the word (you can also do dE to delete until the next space)
bde -- delete the current word, from left to right delimiter
df[space] -- delete up until and including the next space
dt. -- delete until next dot
dd -- delete this entire line
ye (or yE) -- yanks text from here to the end of the word
ce - cuts through the end of the word
bye -- copies current word (makes me wonder what "hi" does!)
yy -- copies the current line
cc -- cuts the current line, you can also do S instead. There's also lower cap s which cuts current character and switches to insert mode.
viwy or viwc. Yank or change current word. Hit w multiple times to keep selecting each subsequent word, use b to move backwards
vi{ - select all text in figure brackets. va{ - select all text including {}s
vi(p - highlight everything inside the ()s and replace with the pasted text
b and e move the cursor word-by-word, similarly to how Ctrl+Arrows normally do. The definition of word is a little different though, as several consecutive delmiters are treated as one word. If you start at the middle of a word, pressing b will always get you to the beginning of the current word, and each consecutive b will jump to the beginning of the next word. Similarly, and easy to remember, e gets the cursor to the end of the current, and each subsequent, word.
similar to b/e, capital B and E move the cursor word-by-word using only whitespaces as delimiters.
capital D (take a deep breath) Deletes the rest of the line to the right of the cursor, same as Shift+End/Del in normal editors (notice 2 keypresses -- Shift+D -- instead of 3)
One that I rarely find in most Vim tutorials, but it's INCREDIBLY useful (at least to me), is the
g; and g,
to move (forward, backward) through the changelist.
Let me show how I use it. Sometimes I need to copy and paste a piece of code or string, say a hex color code in a CSS file, so I search, jump (not caring where the match is), copy it and then jump back (g;) to where I was editing the code to finally paste it. No need to create marks. Simpler.
Just my 2cents.
:%!xxd
Make vim into a hex editor.
:%!xxd -r
Revert.
Warning: If you don't edit with binary (-b), you might damage the file. – Josh Lee in the comments.
gv
Reselects last visual selection.
Sometimes a setting in your .vimrc will get overridden by a plugin or autocommand. To debug this a useful trick is to use the :verbose command in conjunction with :set. For example, to figure out where cindent got set/unset:
:verbose set cindent?
This will output something like:
cindent
Last set from /usr/share/vim/vim71/indent/c.vim
This also works with maps and highlights. (Thanks joeytwiddle for pointing this out.) For example:
:verbose nmap U
n U <C-R>
Last set from ~/.vimrc
:verbose highlight Normal
Normal xxx guifg=#dddddd guibg=#111111 font=Inconsolata Medium 14
Last set from ~/src/vim-holodark/colors/holodark.vim
:%TOhtml
Creates an html rendering of the current file.
Not sure if this counts as dark-corner-ish at all, but I've only just learnt it...
:g/match/y A
will yank (copy) all lines containing "match" into the "a/#a register. (The capitalization as A makes vim append yankings instead of replacing the previous register contents.) I used it a lot recently when making Internet Explorer stylesheets.
Want to look at your :command history?
q:
Then browse, edit and finally to execute the command.
Ever make similar changes to two files and switch back and forth between them? (Say, source and header files?)
:set hidden
:map <TAB> :e#<CR>
Then tab back and forth between those files.
Vim will open a URL, for example
vim http://stackoverflow.com/
Nice when you need to pull up the source of a page for reference.
Macros can call other macros, and can also call itself.
eg:
qq0dwj#qq#q
...will delete the first word from every line until the end of the file.
This is quite a simple example but it demonstrates a very powerful feature of vim
Assuming you have Perl and/or Ruby support compiled in, :rubydo and :perldo will run a Ruby or Perl one-liner on every line in a range (defaults to entire buffer), with $_ bound to the text of the current line (minus the newline). Manipulating $_ will change the text of that line.
You can use this to do certain things that are easy to do in a scripting language but not so obvious using Vim builtins. For example to reverse the order of the words in a line:
:perldo $_ = join ' ', reverse split
To insert a random string of 8 characters (A-Z) at the end of every line:
:rubydo $_ += ' ' + (1..8).collect{('A'..'Z').to_a[rand 26]}.join
You are limited to acting on one line at a time and you can't add newlines.
^O and ^I
Go to older/newer position.
When you are moving through the file (by searching, moving commands etc.) vim rember these "jumps", so you can repeat these jumps backward (^O - O for old) and forward (^I - just next to I on keyboard). I find it very useful when writing code and performing a lot of searches.
gi
Go to position where Insert mode was stopped last.
I find myself often editing and then searching for something. To return to editing place press gi.
gf
put cursor on file name (e.g. include header file), press gf and the file is opened
gF
similar to gf but recognizes format "[file name]:[line number]". Pressing gF will open [file name] and set cursor to [line number].
^P and ^N
Auto complete text while editing (^P - previous match and ^N next match)
^X^L
While editing completes to the same line (useful for programming).
You write code and then you recall that you have the same code somewhere in file. Just press ^X^L and the full line completed
^X^F
Complete file names.
You write "/etc/pass" Hmm. You forgot the file name. Just press ^X^F and the filename is completed
^Z or :sh
Move temporary to the shell. If you need a quick bashing:
press ^Z (to put vi in background) to return to original shell and press fg to return to vim back
press :sh to go to sub shell and press ^D/exit to return to vi back
Typing == will correct the indentation of the current line based on the line above.
Actually, you can do one = sign followed by any movement command. ={movement}
For example, you can use the % movement which moves between matching braces. Position the cursor on the { in the following code:
if (thisA == that) {
//not indented
if (some == other) {
x = y;
}
}
And press =% to instantly get this:
if (thisA == that) {
//not indented
if (some == other) {
x = y;
}
}
Alternately, you could do =a{ within the code block, rather than positioning yourself right on the { character.
" insert range ip's
"
" ( O O )
" =======oOO=(_)==OOo======
:for i in range(1,255) | .put='10.0.0.'.i | endfor
This is a nice trick to reopen the current file with a different encoding:
:e ++enc=cp1250 %:p
Useful when you have to work with legacy encodings. The supported encodings are listed in a table under encoding-values (see help encoding-values). Similar thing also works for ++ff, so that you can reopen file with Windows/Unix line ends if you get it wrong for the first time (see help ff).
imap jj <esc>
Let's see some pretty little IDE editor do column transposition.
:%s/\(.*\)^I\(.*\)/\2^I\1/
Explanation
\( and \) is how to remember stuff in regex-land. And \1, \2 etc is how to retrieve the remembered stuff.
>>> \(.*\)^I\(.*\)
Remember everything followed by ^I (tab) followed by everything.
>>> \2^I\1
Replace the above stuff with "2nd stuff you remembered" followed by "1st stuff you remembered" - essentially doing a transpose.
Not exactly a dark secret, but I like to put the following mapping into my .vimrc file, so I can hit "-" (minus) anytime to open the file explorer to show files adjacent to the one I just edit. In the file explorer, I can hit another "-" to move up one directory, providing seamless browsing of a complex directory structures (like the ones used by the MVC frameworks nowadays):
map - :Explore<cr>
These may be also useful for somebody. I like to scroll the screen and advance the cursor at the same time:
map <c-j> j<c-e>
map <c-k> k<c-y>
Tab navigation - I love tabs and I need to move easily between them:
map <c-l> :tabnext<enter>
map <c-h> :tabprevious<enter>
Only on Mac OS X: Safari-like tab navigation:
map <S-D-Right> :tabnext<cr>
map <S-D-Left> :tabprevious<cr>
Often, I like changing current directories while editing - so I have to specify paths less.
cd %:h
I like to use 'sudo bash', and my sysadmin hates this. He locked down 'sudo' so it could only be used with a handful of commands (ls, chmod, chown, vi, etc), but I was able to use vim to get a root shell anyway:
bash$ sudo vi +'silent !bash' +q
Password: ******
root#
I often use many windows when I work on a project and sometimes I need to resize them. Here's what I use:
map + <C-W>+
map - <C-W>-
These mappings allow to increase and decrease the size of the current window. It's quite simple but it's fast.
:r! <command>
pastes the output of an external command into the buffer.
Do some math and get the result directly in the text:
:r! echo $((3 + 5 + 8))
Get the list of files to compile when writing a Makefile:
:r! ls *.c
Don't look up that fact you read on wikipedia, have it directly pasted into the document you are writing:
:r! lynx -dump http://en.wikipedia.org/wiki/Whatever
Not an obscure feature, but very useful and time saving.
If you want to save a session of your open buffers, tabs, markers and other settings, you can issue the following:
mksession session.vim
You can open your session using:
vim -S session.vim
Map F5 to quickly ROT13 your buffer:
map <F5> ggg?G``
You can use it as a boss key :).
I use vim for just about any text editing I do, so I often times use copy and paste. The problem is that vim by default will often times distort imported text via paste. The way to stop this is to use
:set paste
before pasting in your data. This will keep it from messing up.
Note that you will have to issue :set nopaste to recover auto-indentation. Alternative ways of pasting pre-formatted text are the clipboard registers (* and +), and :r!cat (you will have to end the pasted fragment with ^D).
It is also sometimes helpful to turn on a high contrast color scheme. This can be done with
:color blue
I've noticed that it does not work on all the versions of vim I use but it does on most.
I just found this one today via NSFAQ:
Comment blocks of code.
Enter Blockwise Visual mode by hitting CTRL-V.
Mark the block you wish to comment.
Hit I (capital I) and enter your comment string at the beginning of the line. (// for C++)
Hit ESC and all lines selected will have // prepended to the front of the line.

Resources