Twint Twitter Scraper - date / time function not working (--since) - web-scraping

Having the following issue trying to scrap data with following code with TWINT:
twint --search "ElonMusk -filter:replies" --since "2022-01-01" --links exclude --csv -o EM1.csv --lang "en"
Scraping never goes further back than 2022-11-22 for some reason.
I've tried to insert other time dates but nothing works. I also tried to add a time stamp and add --until function too, but nothing worked.

Related

Finding time between two patterns

I am new to unix and can someone help me with following query?
I have a deployment log file and there I wanted to search for two pattern (eg: started , completed ) and after getting the out put I just want to calculate and print the time taken between the started step and completed step
I tried a lot I’m able to print the the out put but unable to print the time taken between those steps
Thanks in advance,
Mahi
Below is the sample deployment log and at the starting line i am searching for "DeploymentStart" and, at the end of the line "DeploymentCompleted".
After printing this out put i want to calculate the time taken between deploymentStart time and DeployemntCompleted time.
us>InterimDeploymentStart1523623301227ETA0submittedCount3suppressedCount0listRejectedCount0<
/Pair>batchSplitRejectedCount0emailRendererRejectedCount0
Apr13 12:41:49,492 INFO com.epsilon.deploymentStatusNotifier.util.DeploymentStatusNotificationConverter taskScheduler-2 - JobMessage createdcom.epsilon.common.jobs.model.JobMessage#530c458e
Apr13 12:41:49,488 INFO com.epsilon.deploymentStatusNotifier.util.DeploymentStatusNotificationConverter taskScheduler-2 - JobMessage createdcom.epsilon.comm
JobMessage>de841c92-6682-4696-883c-3087a9194531HarmonyPipelinecompletedDeploymentCompleted1523623301227ETA0

Remove date from filename UNIX

I am working in UNIX and trying to write the following commands. I am receiving a source file daily whose filename is in the format :
ONSITE_EXTR_ONSITE_EXTR_20170707.
Since I am receiving a file daily, the file name would change based on the current date, so ONSITE_EXTR_ONSITE_EXTR_20170708, ONSITE_EXTR_ONSITE_EXTR_20170709 etc. I need to strip the date out of the filename and rename it to ONSITE_EXTR_ONSITE_EXTR. After I have finished whatever data reading and processing I need to do, I need to change the file name back to ONSITE_EXTR_ONSITE_EXTR_20170707 for example. So since the file is being delivered daily, I cant hard code the date in whatever commands I write. Any help would be greatly appreciated
Depending on your toolchain, this may be as simple as running:
$ mv ONSITE_EXTR_ONSITE_EXTR_$(date +%Y%m%d) ONSITE_EXTR_ONSITE_EXTR
... before running the rest of your script, assuming you're using a Bash-like shell.
Having said that, you can just drop in ONSITE_EXTR_ONSITE_EXTR_$(date +%Y%m%d) into your script when trying to access your file instead.
This is all assuming the script's run the same day and in the same time zone as the file is downloaded.
If you were using bash and you had the file name in a variable, you could do:
IN="ONSITE_EXTR_ONSITE_EXTR_20170707"
echo ${IN:0:23}
to give ONSITE_EXTR_ONSITE_EXTR
Googling gives all sorts of guides here...

In Notepad++, how can I insert the current date and time?

On Windows Notepad.exe, I simply press F5 and it inserts the date and time like this:
11:05 AM 1/14/2015
Can I add similarly simple functionality to Notepad++?
If your Notepad++ shipped with TextFX, you can find this in TextFX > TextFX Insert > Date & Time - short format, which will insert a string in the exact same format. If you don't see a TextFX menu, install it via the plugin manager and it will appear.
You can then bind it to a keyboard shortcut in Settings > Shortcut Mapper... > Plugin Commands. Look for I:Date & Time - short format in the mappings.
Step 1 : Install plugin TextFX
Step 2 : Insert Date & time
Try "Python Script" plugin
I prefer to use the Python Script plugin as documented here: https://ardamis.com/2014/03/09/adding-an-insert-datestamp-or-timestamp-macro-to-notepad-plus-plus/
because it gives me total control over how I want to the datetime stamp formatting to look, and it also allows me to create macro scripts for inserting other types of values that I want to compute.
Install "Python Script" MANUALLY
Please note that you must download the Python Script plugin from http://npppythonscript.sourceforge.net/download.shtml because downloading it from the Plugin Manager in Notepad++ doesn't always work. (See this thread for details.)
Write "Time.py" script
Then you can write a simple script like this:
import time
# import sys
timeStr = time.strftime( '%Y-%m-%d' + ' # ' + '%I:%M %p' )
# sys.stdout.write(timeStr)
editor.addText( timeStr )
You can change the format string as you wish. This allows you to have total control over the text output. (See Python 2's time.strftime documentation for details.)
Then save the script to this filename:
"%AppData%\Notepad++\plugins\Config\PythonScript\scripts\Time.py"
Create "menu item" inside "Python Script"
Navigate here: Menu bar -> Plugins -> Python Script -> Configuration like this:
Then you can add the script like this:
Assign hotkey
Then, when you restart Notepad++, you can assign the script to a shortcut like this by going to Menu bar -> Settings -> Shortcut Mapper -> Plugin Commands -> Time:
Further reading
More documentation on the Python Script plugin is available here: PythonScript plugin documentation for Notepad++
This features has been added in Notepad++ v8.1.4.
Menu > Edit > Insert >
Date Time (short)
Date Time (long)
That answers this question without the need for a plugin.
Can you find a way to map it to a shortcut? Currently, the shortcut mapper does not have an entry to map it to. When I try to record and playback a macro for it, I get gibberish on the screen (looks like a bug).
It looks like the TextFX plugin proposed here is not available for 64bit any more.
The alternative as of today is to install the Notepad++ Plugin Demo plugin, which provides Date Time - short format and Date Time - long format. Map to shortcuts as desired. In my German locale, the results are, respectively:
12:10 01.07.2020
12:10 Mittwoch, 1. Juli 2020
With the TextFx add on there's an option to insert date and time. I guess you can assign a keyboard shortcut to it.
A simple method through LuaScript (https://github.com/notepad-plus-plus/notepad-plus-plus/issues/497 and https://www.lua.org/pil/22.1.html): install LuaScript plugin first, then in LuaScript Edit Startup Script past the following:
npp.AddShortcut("Insert Current Date/Time", "Ctrl+Shift+D", function()
editor:ReplaceSel(os.date("%I:%M %p %m/%d/%Y"))
end)
After restart npp, the Ctrl-Shift-D will do as F5 in notepad.
Starting with Notepad++ 8.1.5, you can insert a custom date/time format using Edit/Insert/Date Time (customized). You can customize it in Settings/Preferences/Multi-Instance & Date.
In one of the recent N++ updates, my previous shortcut of CTRL + SHIFT + D, stopped working. I then rechecked and had to remap the custom date shortcut to ALT+ SHIFT + D. This is how it looks:
Make sure your customized date match your expectations:
A more generic answer using "Autohotkey":
https://autohotkey.com/board/topic/21387-how-to-insert-current-date-into-a-hotkey/
Examples:
;//will replace ddd keystroke with yyyy-MM-dd formatted date
:R*?:ddd:
FormatTime, CurrentDateTime,, yyyy-MM-dd
SendInput %CurrentDateTime%
return
;//will replace ttt keystroke with yyyy-MM-dd HH:mm formatted date/time
:R*?:ttt::
FormatTime, CurrentDateTime,, yyyy-MM-dd HH:mm
SendInput %CurrentDateTime%
return
Here's an easy and flexible way to insert a date/time stamp in any format you like.
Like many other developers, I used the FingerText plugin which allows you to insert customizable snippets of code or text by typing a "trigger" word and pressing the tab key.
One of my snippets is to insert a date/time stamp that I use in code comments:
I just type stamptab and it inserts a date like this: 2020-07-31 # 11:45
Here is the FingerText code I use to generate the stamp, which you can modify for any datetime format you choose:
$[![(key)DATE:yyyy-MM-dd]!] # $[![(key)TIME:HH:mm]!] $[0[]0][>END<]
You can download FingerText through the Plugins Admin dialog.
The quickest way that comes to my mind is to open Windows Notepad, press F5, copy/paste. No need to install anything...

Making a self destructive code in R

I was making a package in R and would like it to make it as a trial version for a period of 30 days .
Well my question is how to make a code self destructive depends on number of days ??
I had played with time and date package for a while where i came to know ,
Sys.Date() could give todays date , so i get forard with something below
today=Sys.Date()
a=today
b=a+1
if(a==today)
{
print(paste("today is sunday"))
if(b==today){
print(paste("today is monday"))
}
I know it is stupid work whatever i had done , my sole idea was to fix the 1st use of package as starting day ,and every day it will increment till 30 days ,when it will reach the limit it will automatically destroy using
file.remove () <- through which I can remove some file ........
May be I am clear with my ideas .
Sorry for the novice question .
Add this condition to the license. ("30 days for free, after that you'll have to pay".) and expect users to comply with this.
There is really nothing else you can do.
Well, actually you can. For example, on the first occasion your code is run, save the current date to a file in a certain location (say, "~/.datetocheck"). Then every time your code is run, check for the existence of this file, and if it exists, compare the dates. If more than 30 days have passed, give an error message:
stop("Time is over! You have to pay!")
The problem is that nothing prevents the user from simply deleting this file.

Trying get the price of products with RCurl

Im scrapping the price of some products from a website . In Python I used the urllib2 without problems, but when I tried using RCurl in R I couldn't donwload the source code.
I have to paste the source code with the product code, then I catch the price. The path of a product is: http://www.americanas.com.br/produto/code_of_product.
Actually, I can't download the source code of a product with RCurl. When I try for example getURL('http://www.americanas.com.br/produto/111467594') it returns "".
I tried using getURL('.../produtos/111467594') and I could download the source, but in this way I'm unable to get the price. :(
Anyone know how could I get the price of the products?
Thanks.
Ps.: Sorry for my bad english. :)
welcome to StackOverflow.
It's hard to say for me why it doesn't work, could you include a verbose=TRUE in the getURL? Also, I notice there's different prices on the webpage you linked. You want all or just the first? How about this to get the "Por price":
library("stringr")
productwebpage<-readLines("http://www.americanas.com.br/produto/111467594")
pricerow<-productwebpage[grep("p class=\"sale price\"",productwebpage)]
price<-str_extract_all(pricerow,"\\(?[0-9,.]+\\)?")[[1]]
You could also substitute the grep("p class=\"sale price\"",productwebpage) to either grep("<p><span class=\"regular price\">",productwebpage) (to get the "de price" / old price) or grep("<span class=\"p-v interest\">",productwebpage) (which will give you the "sem jouros" price / per month payment). For the last example you will get the months first and the payment after so it will be:
> price
[1] "12" "83,25"
This should hopefully work for other products as well (just tried 5 and seemed to work for all of them).

Resources