How can I configure logging to file requests's get or post?
my_config = {'verbose': sys.stderr}
requests.get('http://httpbin.org/headers', config=my_config)
What should I use in verbose?
Have you tried simply opening a file?
>>> import sys
>>> type(sys.stderr)
file
>>> f = open('test.log', 'w')
>>> type(f)
f
So the example above will look like this:
my_config = { 'verbose': open('/path/to/file', 'w') }
requests.get('http://httpbin.org/headers', config = my_config)
HTH
Related
I have a directory contains a bunch of txt files:
dir/train/[train1.txt, train2.txt, train3.txt]
I'm able to read a single file, if I define following in a config.yaml
file_name: ${paths.data_dir}/train/train1.txt
So I get the str and I used np.loadtxt(self.hparams.file_name)
I tried
file_name: ${paths.data_dir}/train/*
So I have List[str], I then loop over file_name
dat = []
for file in self.hparams.file_name:
dat.append(np.loadtxt(file))
but it didn't work out.
You could define an OmegaConf custom resolver for this:
# my_app.py
import pathlib
from pathlib import Path
from typing import List
from omegaconf import OmegaConf
yaml_data = """
paths:
data_dir: dir
file_names: ${pathlib_glob:${paths.data_dir}, 'train/*'}
"""
def pathlib_glob(data_dir: str, glob_pattern: str) -> List[str]:
"""Use Pathlib glob to get a list of filenames"""
data_dir_path = pathlib.Path(data_dir)
file_paths: List[Path] = [p for p in data_dir_path.glob(glob_pattern)]
filenames: List[str] = [str(p) for p in file_paths]
return filenames
OmegaConf.register_new_resolver("pathlib_glob", pathlib_glob)
cfg = OmegaConf.create(yaml_data)
assert cfg.file_names == ['dir/train/train3.txt', 'dir/train/train2.txt', 'dir/train/train1.txt']
Now, at the command line:
mkdir -p dir/train
touch dir/train/train1.txt
touch dir/train/train2.txt
touch dir/train/train3.txt
python my_app.py # the assertion passes
sample Url:
site.com/category/%D9%81%D8%AA%D9%88%DA%AF%D8%B1%D8%A7%D9%81%DB%8C_%D8%B1%DB%8C%D9%86%D9%88%D9%BE%D9%84%D8%A7%D8%B3%D8%AA%DB%8C/
url config:
url(r'^category/(?P<page_slug>.*)/$', views.category, name='category'),
passenger config:
import imp
import os
import sys
sys.path.insert(0, os.path.dirname(__file__))
wsgi = imp.load_source('wsgi', 'photography/wsgi.py')
application = wsgi.application
wsgi config:
"""
WSGI config for photography project.
It exposes the WSGI callable as a module-level variable named ``application``.
For more information on this file, see
https://docs.djangoproject.com/en/1.10/howto/deployment/wsgi/
"""
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "photography.settings")
#os.environ["DJANGO_SETTINGS_MODULE"] = "photography.settings"
application = get_wsgi_application()
but response 404 not found in url!
in problem for all url persian slug.
when change config wsgi to unquote:
from urllib.parse import unquote
def application(environ, start_fn):
environ['PATH_INFO'] = unquote(environ['PATH_INFO'])
app = get_wsgi_application()
print(environ)
return app(environ, start_fn)
change url into:
site.com/category/%C3%99%C2%81%C3%98%C2%AA%C3%99%C2%88%C3%9A%C2%AF%C3%98%C2%B1%C3%98%C2%A7%C3%99/tag/%D9%81%D8%AA%D9%88%DA%AF%D8%B1%D8%A7%D9%81%DB%8C-%D9%BE%D8%B2%D8%B4%DA%A9%DB%8C/
But there is an open problem !
I applied all the changes I found with the search, but there is still a problem!
Output one of the changes in the wsgi:
App 3585081 output: set_script_prefix(get_script_name(environ))
App 3585081 output: File "/home/sepandteb/virtualenv/sepandteb/3.5/lib/python3.5/site-packages/django/core/handlers/wsgi.py", line 210, in get_script_name
App 3585081 output: return script_name.decode(UTF_8)
App 3585081 output: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc3 in position 27: unexpected end of data
use encode('utf-8') for Persian character and set # -*- coding: utf-8 -*- on top of file.
for example when you save data on DB or read on DB use this function.
i.e :
slug.encode('utf-8')
You can use "uri_to_iri" in the view, for example:
from django.shortcuts import get_object_or_404
from django.utils.encoding import uri_to_iri
def blog_detail(request, slug):
post= get_object_or_404(Post, slug=uri_to_iri(slug))
I'm creating a fork of my Plone site (which has not been forked for a long time). This site has a special catalog object for user profiles (a special Archetypes-based object type) which is called portal_user_catalog:
$ bin/instance debug
>>> portal = app.Plone
>>> print [d for d in portal.objectMap() if d['meta_type'] == 'Plone Catalog Tool']
[{'meta_type': 'Plone Catalog Tool', 'id': 'portal_catalog'},
{'meta_type': 'Plone Catalog Tool', 'id': 'portal_user_catalog'}]
This looks reasonable because the user profiles don't have most of the indexes of the "normal" objects, but have a small set of own indexes.
Since I found no way how to create this object from scratch, I exported it from the old site (as portal_user_catalog.zexp) and imported it in the new site. This seemed to work, but I can't add objects to the imported catalog, not even by explicitly calling the catalog_object method. Instead, the user profiles are added to the standard portal_catalog.
Now I found a module in my product which seems to serve the purpose (Products/myproduct/exportimport/catalog.py):
"""Catalog tool setup handlers.
$Id: catalog.py 77004 2007-06-24 08:57:54Z yuppie $
"""
from Products.GenericSetup.utils import exportObjects
from Products.GenericSetup.utils import importObjects
from Products.CMFCore.utils import getToolByName
from zope.component import queryMultiAdapter
from Products.GenericSetup.interfaces import IBody
def importCatalogTool(context):
"""Import catalog tool.
"""
site = context.getSite()
obj = getToolByName(site, 'portal_user_catalog')
parent_path=''
if obj and not obj():
importer = queryMultiAdapter((obj, context), IBody)
path = '%s%s' % (parent_path, obj.getId().replace(' ', '_'))
__traceback_info__ = path
print [importer]
if importer:
print importer.name
if importer.name:
path = '%s%s' % (parent_path, 'usercatalog')
print path
filename = '%s%s' % (path, importer.suffix)
print filename
body = context.readDataFile(filename)
if body is not None:
importer.filename = filename # for error reporting
importer.body = body
if getattr(obj, 'objectValues', False):
for sub in obj.objectValues():
importObjects(sub, path+'/', context)
def exportCatalogTool(context):
"""Export catalog tool.
"""
site = context.getSite()
obj = getToolByName(site, 'portal_user_catalog', None)
if tool is None:
logger = context.getLogger('catalog')
logger.info('Nothing to export.')
return
parent_path=''
exporter = queryMultiAdapter((obj, context), IBody)
path = '%s%s' % (parent_path, obj.getId().replace(' ', '_'))
if exporter:
if exporter.name:
path = '%s%s' % (parent_path, 'usercatalog')
filename = '%s%s' % (path, exporter.suffix)
body = exporter.body
if body is not None:
context.writeDataFile(filename, body, exporter.mime_type)
if getattr(obj, 'objectValues', False):
for sub in obj.objectValues():
exportObjects(sub, path+'/', context)
I tried to use it, but I have no idea how it is supposed to be done;
I can't call it TTW (should I try to publish the methods?!).
I tried it in a debug session:
$ bin/instance debug
>>> portal = app.Plone
>>> from Products.myproduct.exportimport.catalog import exportCatalogTool
>>> exportCatalogTool(portal)
Traceback (most recent call last):
File "<console>", line 1, in <module>
File ".../Products/myproduct/exportimport/catalog.py", line 58, in exportCatalogTool
site = context.getSite()
AttributeError: getSite
So, if this is the way to go, it looks like I need a "real" context.
Update: To get this context, I tried an External Method:
# -*- coding: utf-8 -*-
from Products.myproduct.exportimport.catalog import exportCatalogTool
from pdb import set_trace
def p(dt, dd):
print '%-16s%s' % (dt+':', dd)
def main(self):
"""
Export the portal_user_catalog
"""
g = globals()
print '#' * 79
for a in ('__package__', '__module__'):
if a in g:
p(a, g[a])
p('self', self)
set_trace()
exportCatalogTool(self)
However, wenn I called it, I got the same <PloneSite at /Plone> object as the argument to the main function, which didn't have the getSite attribute. Perhaps my site doesn't call such External Methods correctly?
Or would I need to mention this module somehow in my configure.zcml, but how? I searched my directory tree (especially below Products/myproduct/profiles) for exportimport, the module name, and several other strings, but I couldn't find anything; perhaps there has been an integration once but was broken ...
So how do I make this portal_user_catalog work?
Thank you!
Update: Another debug session suggests the source of the problem to be some transaction matter:
>>> portal = app.Plone
>>> puc = portal.portal_user_catalog
>>> puc._catalog()
[]
>>> profiles_folder = portal.some_folder_with_profiles
>>> for o in profiles_folder.objectValues():
... puc.catalog_object(o)
...
>>> puc._catalog()
[<Products.ZCatalog.Catalog.mybrains object at 0x69ff8d8>, ...]
This population of the portal_user_catalog doesn't persist; after termination of the debug session and starting fg, the brains are gone.
It looks like the problem was indeed related with transactions.
I had
import transaction
...
class Browser(BrowserView):
...
def processNewUser(self):
....
transaction.commit()
before, but apparently this was not good enough (and/or perhaps not done correctly).
Now I start the transaction explicitly with transaction.begin(), save intermediate results with transaction.savepoint(), abort the transaction explicitly with transaction.abort() in case of errors (try / except), and have exactly one transaction.commit() at the end, in the case of success. Everything seems to work.
Of course, Plone still doesn't take this non-standard catalog into account; when I "clear and rebuild" it, it is empty afterwards. But for my application it works well enough.
I have a setup such that an nginx server passes control off to uWsgi, which launches a pylons app using the following in my xml configuration file:
<ini-paste>...</ini-paste>
Everything is working nicely, and I was able to set it to debug mode using the following in the associated ini file, like:
debug = true
Except debug mode only prints out errors, and doesn't reload the code everytime a file has been touched. If I was running directly through paste, I could use the --reload option, but going through uWsgi complicates things.
Does anybody know of a way to tell uWsgi to tell paste to set the --reload option, or to do this directly in the paste .ini file?
I used something like the following code to solve this, the monitorFiles(...) method is called on application initialization, and it monitors the files, sending the TERM signal when it sees a change.
I'd still much prefer a solution using paster's --reload argument, as I imagine this solution has bugs:
import os
import time
import signal
from deepthought.system import deployment
from multiprocessing.process import Process
def monitorFiles():
if deployment.getDeployment().dev and not FileMonitor.isRunning:
monitor = FileMonitor(os.getpid())
try: monitor.start()
except: print "Something went wrong..."
class FileMonitor(Process):
isRunning = False
def __init__(self, masterPid):
self.updates = {}
self.rootDir = deployment.rootDir() + "/src/python"
self.skip = len(self.rootDir)
self.masterPid = masterPid
FileMonitor.isRunning = True
Process.__init__(self)
def run(self):
while True:
self._loop()
time.sleep(5)
def _loop(self):
for root, _, files in os.walk(self.rootDir):
for file in files:
if file.endswith(".py"):
self._monitorFile(root, file)
def _monitorFile(self, root, file):
mtime = os.path.getmtime("%s/%s" % (root, file))
moduleName = "%s/%s" % (root[self.skip+1:], file[:-3])
moduleName = moduleName.replace("/",".")
if not moduleName in self.updates:
self.updates[moduleName] = mtime
elif self.updates[moduleName] < mtime:
print "Change detected in %s" % moduleName
self._restartWorker()
self.updates[moduleName] = mtime
def _restartWorker(self):
os.kill(self.masterPid, signal.SIGTERM)
Use the signal framework in 0.9.7 tree
http://projects.unbit.it/uwsgi/wiki/SignalFramework
An example of auto-reloading:
import uwsgi
uwsgi.register_signal(1, "", uwsgi.reload)
uwsgi.add_file_monitor(1, 'myfile.py')
def application(env, start_response):
...
Is there a way to make https calls with the Network.Browser package.
I'm not seeing it in the documentation on Hackage.
If there isn't a way to do it with browse is there another way to fetch https pages?
My current test code is
import Network.HTTP
import Network.URI (parseURI)
import Network.HTTP.Proxy
import Data.Maybe (fromJust)
import Control.Applicative ((<$>))
import Network.Browser
retrieveUrl :: String -> IO String
retrieveUrl url = do
rsp <- browse $ request (Request (fromJust uri) POST [] "Body")
return $ snd (rspBody <$> rsp)
where uri = parseURI url
I've been running nc -l -p 8000 and watching the output.
I see that it doesn't encrypt it when I do retrieveUrl https://localhost:8000
Also when I try a real https site I get:
Network.Browser.request: Error raised ErrorClosed
*** Exception: user error (Network.Browser.request: Error raised ErrorClosed)
Edit: Network.Curl solution (For doing a SOAP call)
import Network.Curl (curlGetString)
import Network.Curl.Opts
soapHeader s = CurlHttpHeaders ["Content-Type: text/xml", "SOAPAction: " ++ s]
proxy = CurlProxy "proxy.foo.org"
envelope = "myRequestEnvelope.xml"
headers = readFile envelope >>= (\x -> return [ soapHeader "myAction"
, proxy
, CurlPost True
, CurlPostFields [x]])
main = headers >>= curlGetString "https://service.endpoint"
An alternative and perhaps more "haskelly" solution as Travis Brown put it with http-conduit:
To just fetch https pages:
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = simpleHttp "https://www.noisebridge.net/wiki/Noisebridge" >>= L.putStr
The below shows how to pass urlencode parameters.
{-# LANGUAGE OverloadedStrings #-}
import Network.HTTP.Conduit
import qualified Data.ByteString.Lazy as L
main = do
initReq <- parseUrl "https://www.googleapis.com/urlshortener/v1/url"
let req' = initReq { secure = True } -- Turn on https
let req = (flip urlEncodedBody) req' $
[ ("longUrl", "http://www.google.com/")
-- ,
]
response <- withManager $ httpLbs req
L.putStr $ responseBody response
You can also set the method, content-type, and request body manually. The api is the same as in http-enumerator a good example is: https://stackoverflow.com/a/5614946
I've wondered about this in the past and have always ended up just using the libcurl bindings. It would be nice to have a more Haskelly solution, but Network.Curl is very convenient.
If all you want to do is fetch a page, Network.HTTP.Wget is the most simple way. Exhibit a:
import Network.HTTP.Wget
main = putStrLn =<< wget "https://www.google.com" [] []