I have been using readLines() to scrape information from a website in an R tutorial. I now wish to extract data from my own website (specifically the awstats data) however the domain is password protected.
Is there a way that I can pass the url for the specific awstats data I require with a username and password.
the format of the url is:
http://domain.name:port/awstats.pl?month=02&year=2011&config=domain.name&lang=en&framename=mainright&output=alldomains
Thanks.
If it is indeed a http basic access authentication, the documentation on connections provides some help:
URLs
Note that https:// connections are
only supported if --internet2 or
setInternet2(TRUE) was used (to make
use of Internet Explorer internals),
and then only if the certificate is
considered to be valid. With that
option only, the http://user:pass#site
notation for sites requiring
authentication is also accepted.
So your URL string should look like this:
http://username:password#domain.name:port/awstats.pl?month=02&year=2011&config=domain.name&lang=en&framename=mainright&output=alldomains
This might be Windows-only though.
Hope this helps!
You can embed the username and password in the url like :
http://userid:passw#domain.name:port/...
This you can try to use with readLines(). If that doesn't work, you can always try a workaround using url() to open the connection :
zz <- url("http://userid:passw#domain.name:port/...")
readLines(zz)
close(zz)
You can also download the file and save it somewhere using download.file()
download.file("theurl","/path/to/file/filename",method="wget")
This saves the file on the local path that is specified.
EDIT :
as csgillespie said, you shouldn't include your username and password in the script. If you run scripts with source() or interactively, you could add eg :
user <- readline("Give the username : ")
passw <- readline("Give the password : ")
Url <- paste("http://",user,":",passw,"#domain.name...")
readLines(Url,...)
When running from the commandline, you could pass the arguments after --args and access them using commandArgs (see ?commandArgs)
If you have access to the box, you could always just read the awstats log files. If you can ssh into the box, then you could easily sync the latest file using rsync.
The slight snag with using
http://username:password#domain...
is that you are putting your password in an R script - best to avoid this. Of course you can secure it the script, but it only takes one slip. For example,
Someone asks you a similar question and you publish your script
The url http://username:password#domain... will(?) now show up on your server logs
...
Formatting the url as http://username:password#domain... for use with download.file didn't work for me, but R.utils provides the function downloadFile that works perfectly:
require(R.utils)
downloadFile(myurl, myfile, username = "myusername", password ="mypassword")
See #joris-meys answer for a way to avoid including your username and password in plain text in your script.
EDIT Except it looks like downloadFile just reformats the URL to http://username:password#domain...? Hmm...
Related
I have made an script in R that goes through my mail in Outlook in search of determined emails. The thing is that everytime that the script tries to access my emails a window pop-ups and ask for permission. The script works as intended but the proccess is slowed down as I have to be there to click "Allow" everytime.
I have read that it is possible to allow programs to enter my Outlook in Trusct Center options but the Windows version is too old and can not fix it this way. Also, as this is a shared server and I have not Administrator access I can not make big changes.
I have read other similar questions but nothing in R, like this[Suppress Outlook pop-up allow access or this[How to supress the outlook pop up message while sending mails in PB8.
This is my script:
library(RDCOMClient)
library(stringr)
#Folder in OutLook where to look for the emails
folderName = "Specific folder"
OutApp <- COMCreate("Outlook.Application")
outlookNameSpace = OutApp$GetNameSpace("MAPI")
folder <- outlookNameSpace$Folders(1)$Folders(folderName)
folder$Name(1)
emails <- folder$Items
for (i in 1:emails()$Count()){
subject <- emails(i)$Subject() #Get the subject of the email
#If the subject of the email has an specific string print the body of the message
if (grepl("Subject to look for", subject)== TRUE){
body <- emails(i)$Body() # Here asks for permission
print(body)
}
}
Is there any way to embeb a piece of code to avoid this pop-up or to automatically "click" on Allow?
It looks like you are faced with a standard security prompt in Outlook. There are possible routes where you can go to:
Use a low-level code on which Outlook is based on - Extended MAPI or just any other third-party wrappers around this API such as Redemption.
Use third-party components designed for turning off such security triggers in Outlook - Security Manager for Microsoft Outlook.
Set up a group policy to avoid such triggers.
Set up a valid antivirus software on the system.
I have to get rid of the clear text password in ActiveMQ server in the jetty-realm.properties file as part of the security requirements of our project.
I followed the procedure for encrypting passwords found here: https://activemq.apache.org/encrypted-passwords.html.
In Jetty-real.properties I replaced plain password with ${activemq.password}
But I could not login into the web console (http://localhost:8161/admin)
Are there additional config parameters I need to change? Any help would be greatly appreciated.
For ecrypt password we can use jetty utility. This can be downloaded from http://download.eclipse.org/jetty/.
use:
java -cp lib/jetty-util-$JETTY_VERSION.jar org.eclipse.jetty.util.security.Password admin admin
admin
OBF:1u2a1toa1w8v1tok1u30
MD5:21232f297a57a5a743894a0e4a801fc3
CRYPT:adpexzg3FUZAk
Add in bus\conf\jetty.xml\jetty-realm.properties
admin: MD5:21232f297a57a5a743894a0e4a801fc3, admin
A very important note: the sequence of data specified by you should be as follows:
org.eclipse.jetty.util.security.Password [username] password
Also note that in some shells (like default on Ubuntu 20.04) the dollar sign inside the password will effectively truncate your password to the part preceding the dollar sign.
(I meant to put a comment to the answer by #dorin.canepa, but clicked on a wrong button and now it's an answer, oh well)
How can I change the admin password for a Glassfish Domain using a password file? I know the conventional method of manually typing the password upon prompt.
However I want to change the admin password using a script where in I do not have to manually type the password.
This is possible, but you will need 2 password files if you want to script this fully in the easiest way.
Create a temporary file (tmpfile in my example) which will hold the current password (blank by default) and the desired new password:
AS_ADMIN_PASSWORD=
AS_ADMIN_NEWPASSWORD=myNewPassword
Now create a password (pwdfile in my example) file which will contain the changed admin password:
AS_ADMIN_PASSWORD=myNewPassword
You can then use the files to change the password using the commands below, making sure to use tmpfile when changing the password, then pwdfile afterwards
$PAYARA_PATH/bin/asadmin start-domain
$PAYARA_PATH/bin/asadmin --user $ADMIN_USER --passwordfile=/opt/tmpfile change-admin-password
$PAYARA_PATH/bin/asadmin --user $ADMIN_USER --passwordfile=/opt/pwdfile enable-secure-admin
$PAYARA_PATH/bin/asadmin restart-domain
This example was adapted from the way the Payara Server dockerfile works
For anyone still interested in manually setting the admin account password:
I tried to generate the contents of the "admin-keyfile" located in "glassfish/domains/{ACTIVE_DOMAIN_NAME}/config/admin-keyfile" based on the current implementation of the Payara Repo. This file (as the data source for the FileRealm) is used to authenticate the admin user when accessing the admin interface under port 4848.
Each line of this text file represents an account and is structured as
USERNAME;PASSWORD;GROUPS
The field "PASSWORD" is prefixed with a hash algorithm keyword (wrapped in curly braces, e.g. "SSHA" or "SSHA256") followed by a BASE64 encoded hash of the concatenated salted hash and the salt value itself (some random bytes):
{SSHA}BASE64(SHA(password,salt),salt)
Long story short: If you want to generate user accounts manually you could for example use the following Python script:
import hashlib
from base64 import b64encode
from secrets import token_bytes
from getpass import getpass
username = 'admin' # input('Username: ')
plainTextPassword = getpass()
randomSalt = token_bytes(8)
passwordHash = hashlib.sha256()
passwordHash.update(plainTextPassword.encode('utf-8'))
passwordHash.update(randomSalt)
passwordDigest = passwordHash.digest()
# cryptic range reflects the strange implementation... feel free to change it to "range(98)"
# https://github.com/payara/Payara/blob/6488cbdc90fd0f6c42de6a42affcd09f697be715/nucleus/common/common-util/src/main/java/org/glassfish/security/common/SSHA.java#L108
for run in range(2, 101):
passwordHash = hashlib.sha256()
passwordHash.update(passwordDigest)
passwordDigest = passwordHash.digest()
saltedHashAndSalt = b64encode(passwordDigest + randomSalt).decode('utf-8')
result = '{0};{{SSHA256}}{1};asadmin'.format(username, saltedHashAndSalt)
print(result)
Insert the console output into the "admin-keyfile" and (re)start your server.
As far as I know, it is impossible to change it via a file as a parameter for security reasons.
You can consider an alternative solution (pipe) but the confirmation of the password is always necessary. https://docs.oracle.com/cd/E19798-01/821-1758/change-admin-password-1/index.html
I am trying to use setup_twitter_oauth authentication (from TwitteR package) to later on run StreamR search.
There is a related question in Stackoverflow about the same subject -
Retrieving cached oauth token with packages httr, twitteR, and streamR
However my problem that I don't see .httr-oauth file in my working directory
The code I use to establish connection with Twitter is below
setup_twitter_oauth(consumer_key, consumer_secret, access_token=NULL, access_secret=NULL)
(I use proper keys from my registered Twitter App)
It works I am getting [1] Using direct authentication message after execute it.
However there is no file called .httr-oaut in my work directory. Other hidden files are visible.
getOption("httr_oauth_cache") returns TRUE
What am I doing wrong and how can I get this file and use it in the future to establish connection with Twitter using StreamR, like for example in the command below
filterStream( file="tweets_rstats.json", track="rstats",
locations=c(-74,40,-73,41), timeout=600, oauth=my_oauth)
As I understood I need to pass it as oauth, so need in some way to save the token in my_oauth variable
Any help is appreciated
I can't figure out how to change the webdav password. I've done some searching, found many resources of how to add a new user to webdav, but nothing about changing password. Anyone know?
Passwords are stored in webdav.htpasswd file.
If you open it, you will see an entry like this
wampp:XAMPP with WebDAV:bc7f2b670e7e965e307feb30492e642e
That's the entry for user wampp: in realm XAMPP with WebDAV: with password which is encrypted.
To change password, you should use htdigest.exe.
See manual. It is used to create and update user authentication files. You should find in the bin directory of xampp installation.
To do that, do :
htdigest.exe "pathto/webdav.htpasswd" "XAMPP with WebDAV" yourusername
This will be returned:
Adding user yourusername in realm "XAMPP with WebDAV"
You will then be asked for the password for yourusername
New password: yourpassword
Re-type new password: yourpassword
Reference: http://www.apachefriends.org/f/viewtopic.php?f=16&t=38897
Replace the content of the file C:\Program Files\xampp\security\htpasswd.webdav with your username, a colon and the password. Note: Everyone who can see this file can see the password!
Assuming you're talking about doing a password change at the server from a client, I don't believe WebDAV supports such a transaction.
You'd use something like a shell logon or a Web-based admin page to do this.
This always struck me as odd for a lot of network services, for example FTP or email. It seems to be common though.