I am having issues with the Dexador library (and same issue with the Drakma library) when attempting to webscrape. They work fine with HTTP requests, but I receive an error when working with HTTPS requests. Here is an example of the basic get request that I sent:
(defvar *url* "https://www.amazon.com/")
(defvar *request* (dex:get *url*))
Then I receive this backtrace:
No OpenSSL version number could be determined, both SSLeay and OpenSSL_version_num failed.
So i'm assuming something may be wrong with the OpenSSL library, but i'm not 100% sure what's going on. I achieved the request successfully on my windows machine from the command line, and OpenSSL works fine on my Windows 10 machine, but not in Common Lisp.
One solution that I saw was to set the keyword argument insecure to true:
(defvar *request* (dex:get *url* :insecure t))
But this does not work and a receive the same error. When I attempted to scrape an http website the request was successful:
(defvar *request* (dex:get "http://paulgraham.com"))
I am wondering if anyone else has had this problem and if there are any solutions that anyone has found. I do not know if this is an issue with Windows 10, a dependency library, or something else.
This works fine from the command line with:
curl -v "https://www.amazon.com"
The issue here is that my version of OpenSSL was too new. CL+SSL is a dependency of Dexador, and CL+SSL wants the 1.0.1 version, and it wants the 32-bit version of OpenSSL specifically. After playing around with various versions, that seemed to fix the problem.
Related
While using SBCL normally I want to try CCL for some testing and installed it via homebrew on my computer. That worked fine but I fail to use quicklisp with CCL. If I try to load quicklisp's setup.lisp I get the following error message:
➜ ~ ccl64
Clozure Common Lisp Version 1.11.6 DarwinX8664
For more information about CCL, please see http://ccl.clozure.com.
CCL is free software. It is distributed under the terms of the Apache
Licence, Version 2.0.
? (require 'asdf)
ASDF
("uiop" "UIOP" "asdf" "ASDF")
? (load "~/quicklisp/setup.lisp")
> Error: There is no package named "ASDF/SYSTEM-REGISTRY" .
> While executing: CCL::%FASL-NVPACKAGE, in process listener(1).
> Type :GO to continue, :POP to abort, :R for a list of available restarts.
> If continued: Retry finding package with name "ASDF/SYSTEM-REGISTRY".
> Type :? for other options.
1 > :POP
I couldn't find a simple solution searching the web. CCL comes with ASDF as (require 'ASDF) is working. In quicklisp's documentation I couldn't find anything about extra efforts to use it with two lisp implementations in parallel.
If I check ASDF's version as suggested here I get "3.1.5" as installed version, which should be quite recent.
Is there anything obvious I am missing?
I think I found it: I had some old data in my ~/.cache directory from another attempt to install ccl. After deleting it, I can load quicklisp's setup.lisp without any error.
Please advice If I should delete my question or leave it here to prevent others from repeating my errors.
Please note that various Lisps have some startup files. Some of them are by default:
SBCL: ~/.sbclrc
CLISP: ~/.clisprc.lisp
Clozure: ~/.ccl-init.lisp
From the documentation of CCL:
By default, Clozure CL will look for a file named ccl-init.lisp in your home directory, and load it upon startup. On Unix systems, it will also look for .ccl-init.lisp.
CCL uses an ordinary lisp file called ccl-init.lisp which on unix systems is usually put on your home path. So you can add the following lines in that file (on my Ubuntu machine it's path is /home/me/.ccl-init.lisp) which instructs CCL to load quicklisp upon startup:
#-quicklisp
(let ((quicklisp-init (merge-pathnames "quicklisp/setup.lisp"
(user-homedir-pathname))))
(when (probe-file quicklisp-init)
(load quicklisp-init)))
This is enough for enabling your CCL to communicate with quicklisp.
When I download files (Windows 7) using
tryCatch(download.file(paste0(url_bv,arq), paste0(dir_bv,arq))
,error=function(cond) message(paste('erro:',arq,'não encontrado'))
,warning=function(cond) message(paste('warning:',arq,'não encontrado')))
I can´t delete files, getting the msg in the question title.
If I just do
download.file(paste0(url_bv,arq), paste0(dir_bv,arq))
there is no problem.
How can I release the folder/file in R?
I think this is likely a bug in R (windows version). I can faithfully reproduce it in R-3.2.5 (win10_64). I suggest you file a bug report (read R FAQ 9.2 for clear direction).
I tried it with several variations:
different url schemes ("http://" and "https://")
different download.file(..., method=...) options ("wininet", "internal", and "auto" all behaved in this manner; "libcurl" did not, but it merely downloaded the 404.html without telling me there was a problem)
triggers whenever warning=... or condition=... clauses are given to tryCatch and something weird happens in expr
does not trigger with error=... or finally=... arguments
32bit and 64bit variants
not in RStudio
as R --vanilla
All attempted on ubuntu-14.04 (R-3.2.3-64bit) as well, it would not trigger.
Today's (1 Mar 2016) OpenSSL release has caused the following error when running Plone/Zope
.buildout/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/persistent/cPersistence.so: undefined symbol: SSLv2_method
It's hard to see what's going on since it's a binary file. I also tried updating to ZODB3 3.11.0 which yields the following traceback
.buildout/eggs/ZConfig-2.9.0-py2.7.egg/ZConfig/loader.py", line 217, in schemaComponentSource
package=package)
ZConfig.SchemaResourceError: could not load package ZServer:
.buildout/eggs/zope.security-3.7.4-py2.7-linux-x86_64.egg/zope/security/_proxy.so: undefined symbol: SSLv2_method
Package name: 'ZServer'
File name: 'component.xml'
Package path: None
Is there any workaround for this other than reverting OpenSSL?
zope security is a compiled egg, like all the ones ending with -py2.7-linux-x86_64.egg.
As the traceback says, it cannot find anymore a symbol.
Probably you have to recompile it with the new openssl-dev.
I would try (on a development server first):
backup your compiled egg (mkdir eggs-backup && mv `eggs/zope.security-3.7.4-py2.7-linux-x86_64.egg eggs-backup/)
rerun buildout
This will recompile your missing egg.
Hopefully it works and hopefully it is the only one linked to that library.
Anyway, dependending on the way you patched openssl you may have a lot of other issues (I am thinking about Python, urllib*, curl, wget, ...)
OpenSSL 1.0.2g by default doesn't build with SSLv2 (because of the recent DROWN attack). You may need to manually build it without OPENSSL_NO_SSL2 flag.
(but in fact you shouldn't do this if you're doing some server-related stuff, there is a serious security reason because of which it was disabled, see https://drownattack.com)
I was able to resolve this by upgrading python to 2.7.10+, and then upgrading Pillow and lxml.
After going through a lot of sites about best terminal for system admins I was trying to install on Cygwin. Unfortunately, I did not find any good site with instruction about how to do it.
Is anyone done this before? Please help me with steps and packages that I need to install.
Also is there any terminals I can try (like Cygwin)?
I don't know since what version, but now you can install terminator just from the Cygwin installer. And runs great
Regards
If you don't want to use cygwinports, you can actually install all of terminator's dependencies from the cygwin installer, except for terminator itself.
In the cygwin installer select and install:
python-dbus
python-gobject
python-gtk2.0
python-vte (under GNOME tab for some reason)
GConf2
Then, pull down the latest terminator tarball from https://launchpad.net/terminator/+download and extract it somewhere. In a administrator terminal just run python setup.py install and as long as you have a running X server just running terminator will work perfectly.
The sources about how to install terminator are a bit obscures. What I did, and may help you, was this: (although I'm still having segmentation faults errors)
Update your Cygwin to the latest
Open a Cygwin terminal and run: (if you have the x86 version use that. The idea of this step is to use Cygwin Ports)
cygstart -- /your/cygwin/path/setup-x86_64.exe -K http://cygwinports.org/ports.gpg
In the section Choose A Download Site:
Add "http://downloads.sourceforge.net/cygwin-ports"
Add " ftp://ftp.cygwinports.org/pub/cygwinports"
Select another mirror close to you
Check that you have a total of three URLs selected
It may show you warning about not loading the .ini configuration but ignore them (Note: I looked for different port URLs but the official ones threw me errors and I could not pass this step, that's why I used alternatives URLs)
First, you need to install the packages for the X Window:
http://x.cygwin.com/docs/ug/setup.html
Basically they are:
xorg-server (required, the Cygwin/X X Server)
xinit (required, scripts for starting the X server: xinit, startx, startwin (and a shortcut on the Start Menu to run it), startxdmcp.bat )
xorg-docs (optional, man pages)
Also search and select the terminator package
It takes quite a while before it finishes.
Go to Start->All Programs->Cygwin-X->X Win Server (windows tool bar)
A xterm window should open. Type:
terminator
You should know have terminator with Cygwin.
Note: After I run terminator I get this error:
/usr/lib/python2.7/site-packages/terminatorlib/terminator.py:87: Warning: Attempt to add property GnomeProgram::sm-connect after class was initialised
self.gnome_program = gnome.init(APP_NAME, APP_VERSION)
/usr/lib/python2.7/site-packages/terminatorlib/terminator.py:87: Warning: Attempt to add property GnomeProgram::show-crash-dialog after class was initialised
self.gnome_program = gnome.init(APP_NAME, APP_VERSION)
/usr/lib/python2.7/site-packages/terminatorlib/terminator.py:87: Warning: Attempt to add property GnomeProgram::display after class was initialised
self.gnome_program = gnome.init(APP_NAME, APP_VERSION)
/usr/lib/python2.7/site-packages/terminatorlib/terminator.py:87: Warning: Attempt to add property GnomeProgram::default-icon after class was initialised
self.gnome_program = gnome.init(APP_NAME, APP_VERSION)
Warning: python-keybinder is not installed. This means the hide_window shortcut will be unavailable
Unable to bind hide_window key, another instance/window has it.
Segmentation fault (core dumped)
I've looking to fix this issue but sadly I couldn't find anything. If you use Cygwin x86 your outcome can be different.
Hope this can help you.
My Problem
I compiled OpenSSL into QT to enable OpenSSL support. Everything appeared to go correctly in the compile.
However, when I try to use the official HTTP example application that can be found here, everytime I try to download an https page, it will signal two QSslError, each with contents NoError.
The types of QSslErrors, including NoError, are documented here, poorly. There is no explanation on why they even included an error type called NoError, or what it means.
Bizarrely, the NoError error code seems to be true, as it downloads the remote https document perfectly even while signaling the error.
Does anyone have any idea what this means and what could possibly be causing it?
Optional Background Reading
Here is the relevant part of the code from the example app (this is connected to the network connection's sslErrors signal by the constructor):
void HttpWindow::sslErrors(QNetworkReply*,const QList<QSslError> &errors)
{
QString errorString;
foreach (const QSslError &error, errors) {
if (!errorString.isEmpty())
errorString += ", ";
errorString += error.errorString();
}
if (QMessageBox::warning(this, tr("HTTP"),
tr("One or more SSL errors has occurred: %1").arg(errorString),
QMessageBox::Ignore | QMessageBox::Abort) == QMessageBox::Ignore) {
reply->ignoreSslErrors();
}
}
I have tried the old version of this example, and it produced the same result.
I have tried OpenSSL 1.0.0a and 0.9.8o. I have tried tried compiling OpenSSL myself, I have tried using pre-compiled versions of OpenSSL from the net. All produce the same result.
If this were my first time using QT with SSL, I would almost think this is the intended result (even though their example application is popping up error warning message windows), if not for the fact that last time I played with QT, using what would now be an old version of QT with an old version of SSL, I distinctly remember everything working fine with no error windows.
My system is running Windows 7 x64.
The only path where a QSslError can be constructed with a NoError code is during conversion from OpenSSL error codes to QSslError::SslError values, when the error code is X509_V_OK. There is an interesting note about this error code in the OpenSSL docs:
If no peer certificate was presented, the returned result code is X509_V_OK. This is because no verification error occurred, it does however not indicate success.
Can you check with Wireshark or something similar if the certificate is being transmitted?
I get 4 errors, 3 times over.
The 4 (expected) errors are:
1.The host name did not match any of the valid hosts for this certificate
2.The issuer certificate of a locally looked up certificate could not be found
3.The root CA certificate is not trusted for this purpose
4.No certificates could be verified
I suspect your "NoError" refers to the last (#4)...
The reason I got 3 repeats appears to be because there are 3 threads running - each fires the same sslErrors signal.
I suspect your 2 repeats were due to 2 threads running in the WebView widget.