APIGEETool DeployNodeApp - apigee

We are trying to use the apigeetool deploynodeapp utility to deploy a node.js app to apigee.
We think we are following the instructions properly, but are getting a "broken pipe" error every time we try.
The command we are running is...
apigeetool deploynodeapp -n sampleService -d . -m server.js -o ourOrgNameHere -e test -b sample -u ourUserName -p ourPassword
Obviously, ourOrgNameHere, ourUserName, and ourPassword are not the values we really used.
When we run that from the command line, there is a 20 second pause, followed by the output below...
Traceback (most recent call last):
File "/usr/local/bin/apigeetool", line 24, in <module>
deploynodeapp.run()
File "/Library/Python/2.7/site-packages/ApigeePlatformTools/deploynodeapp.py", line 180, in run
revision = deploytools.importBundle(Organization, Name, tf.getvalue())
File "/Library/Python/2.7/site-packages/ApigeePlatformTools/deploytools.py", line 115, in importBundle
resp = httptools.httpCall('POST', uri, hdrs, data)
File "/Library/Python/2.7/site-packages/ApigeePlatformTools/httptools.py", line 24, in httpCall
conn.request(verb, uri, body, hdrs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/httplib.py", line 805, in send
self.sock.sendall(data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 229, in sendall
v = self.send(data[count:])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 198, in send
v = self._sslobj.write(data)
Any help would be greatly appreciated.

You don't have to tell Apigee about the Node module dependencies. The apigeetool utility is supposed to (by default) resolve your Node.js dependencies automatically after your project is uploaded to Apigee Edge (as if running npm install on Edge itself).
Some users have found that if this fails for some reason, it works if you use the --upload-modules option like this:
apigeetool deploynodeapp --upload-modules
The upload-modules flag tells the tool to zip and upload all the modules from your system (rather than trying to update them on Edge). As long as your Node.js works locally, it should work when you deploy it. Try using --upload-modules to see if that works for you.

Related

Using environment variable in jupyter_notebook_config.py

I want to use an environment variable as the location for the c.NotebookApp.notebook_dir parameter in jupyter_notebook_config.py but I have added
c.NotebookApp.notebook_dir = $JUPYTER_NOTEBOOKS
I have also created the variable
>export JUPYTER_NOTEBOOKS=/home/jupyter
but when I run Jupyter notebook I get the below error message and I can tell its not pulling the value of the variable and is trying to use the variable as a literal. Has anyone used environment variables successfully in jupyter notebook config?
[E 20:45:04.513 NotebookApp] Exception while loading config file /root/.jupyter/jupyter_notebook_config.py
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/traitlets/config/application.py", line 562, in _load_config_files
config = loader.load_config()
File "/usr/local/lib/python3.6/site-packages/traitlets/config/loader.py", line 457, in load_config
self._read_file_as_dict()
File "/usr/local/lib/python3.6/site-packages/traitlets/config/loader.py", line 489, in _read_file_as_dict
py3compat.execfile(conf_filename, namespace)
File "/usr/local/lib/python3.6/site-packages/ipython_genutils/py3compat.py", line 198, in execfile
exec(compiler(f.read(), fname, 'exec'), glob, loc)
File "/root/.jupyter/jupyter_notebook_config.py", line 766
c.NotebookApp.notebook_dir = $JUPYTER_NOTEBOOKS
^
SyntaxError: unexpected character after line continuation character
[I 20:45:04.523 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
Traceback (most recent call last):
File "/usr/local/bin/jupyter-notebook", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/usr/local/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "</usr/local/lib/python3.6/site-packages/decorator.py:decorator-gen-7>", line 2, in initialize
File "/usr/local/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/notebook/notebookapp.py", line 1628, in initialize
self.init_webapp()
File "/usr/local/lib/python3.6/site-packages/notebook/notebookapp.py", line 1407, in init_webapp
self.http_server.listen(port, self.ip)
File "/usr/local/lib/python3.6/site-packages/tornado/tcpserver.py", line 151, in listen
sockets = bind_sockets(port, address=address)
File "/usr/local/lib/python3.6/site-packages/tornado/netutil.py", line 174, in bind_sockets
sock.bind(sockaddr)
OSError: [Errno 99] Cannot assign requested address
It isn't clear that you are using environment variables properly. I glanced at the docs and it looks like %env JUPYTER_NOTEBOOKS is the right way to do it. I checked this on my local system and it works (for a different env variable). Give that a try - good luck! :-) . Below are the docs.
https://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-env
Thanks jimf your link helped. I was able to get things working because I was NOT using environment variables correctly. Instead of modifying the .py file I added the parameter to my command of starting jupyter notebook like this:
--NotebookApp.notebook_dir=$JUPYTER_NOTEBOOKS
I was then able to change the variable value and change where jupyter was looking for notebooks.
You can make use of environment variables in the jupyter_notebook_config.py file, but note that the file is a Python file, so (as you already figured out) bash syntax won't work. Instead you can use Python syntax. You can actually write any Python code you like in this file, so this would be valid syntax:
import os
c.NotebookApp.notebook_dir=os.getenv('JUPYTER_NOTEBOOKS')
This would allow you to do even more complex things if you needed.

Building Plone.4-3.14 gives "need more than 0 values to unpack"

When running buildout with the Plone-4.3.14-version-configs, the errors below are thrown.
Getting distribution for 'feedparser==5.0.1'.
error: Not a recognized archive type: /tmp/tmpuOPdYIget_dist/feedparser-5.0.1.tar.bz2
An error occurred when trying to install /tmp/tmpuOPdYIget_dist/feedparser-5.0.1.tar.bz2. Look above this message for any errors that were output by easy_install.
While:
Installing instance.
Getting distribution for 'feedparser==5.0.1'.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/home/ida/.virtenv/lib/python2.7/site-packages/zc/buildout/buildout.py", line 2127, in main
getattr(buildout, command)(args)
File "/home/ida/.virtenv/lib/python2.7/site-packages/zc/buildout/buildout.py", line 797, in install
installed_files = self[part]._call(recipe.install)
File "/home/ida/.virtenv/lib/python2.7/site-packages/zc/buildout/buildout.py", line 1557, in _call
return f()
File "/home/ida/.buildout/eggs/plone.recipe.zope2instance-4.2.22-py2.7.egg/plone/recipe/zope2instance/__init__.py", line 114, in install
installed.extend(self.install_scripts())
File "/home/ida/.buildout/eggs/plone.recipe.zope2instance-4.2.22-py2.7.egg/plone/recipe/zope2instance/__init__.py", line 618, in install_scripts
requirements, ws = self.egg.working_set(['plone.recipe.zope2instance'])
File "/home/ida/.buildout/eggs/zc.recipe.egg-1.3.2-py2.7.egg/zc/recipe/egg/egg.py", line 101, in working_set
**kw)
File "/home/ida/.virtenv/lib/python2.7/site-packages/zc/buildout/easy_install.py", line 924, in install
return installer.install(specs, working_set)
File "/home/ida/.virtenv/lib/python2.7/site-packages/zc/buildout/easy_install.py", line 726, in install
for dist in self._get_dist(req, ws):
File "/home/ida/.virtenv/lib/python2.7/site-packages/zc/buildout/easy_install.py", line 570, in _get_dist
dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
File "/home/ida/.virtenv/lib/python2.7/site-packages/zc/buildout/easy_install.py", line 1704, in _move_to_eggs_dir_and_compile
[tmp_loc] = glob.glob(os.path.join(tmp_dest, '*'))
ValueError: need more than 0 values to unpack
Took a moment to figure out why, because the explanation does not lie in the last error-message, but the preceding one:
Not a recognized archive type, which hints that easy_install cannot handle bunzip-files.
It means that the required sys-package bzip-devel was not present when installing Python.
So one must either install that package and install Python again, or in this case also commenting out the pin for feedparser lets the build run without errors.
Afterwards noticed that feedparser is not in the eggs-cache-dir, as it used to be in another build with same versions-configs. At a glance cannot tell the difference why, but the errors are resolved.

Plone 4.3 AssertionError Running Buildout

We recently got a new Linux machine that got re-imaged from our older machine where our current plone installation resides. I am trying to run buildout on the new machine but I am getting this AssertionError on our plonetheme src product.
Installing 'buildout.dumppickedversions', 'buildout.sanitycheck'.
We have the distribution that satisfies 'buildout.dumppickedversions==0.5'.
Picked: buildout.sanitycheck = 1.0b1
Develop: '/var/db/zope/plone43/zeocluster/src/products.okctypes'
in: '/var/db/zope/plone43/zeocluster/src/products.okctypes'
/tmp/tmpVPeu_l -q develop -mxN -d /var/db/zope/plone43/zeocluster/develop-eggs/tmp4eer1vbuild
Develop: '/var/db/zope/plone43/zeocluster/src/plonetheme.ourtheme'
in: '/var/db/zope/plone43/zeocluster/src/plonetheme.ourtheme'
/tmp/tmppWGq8Z -q develop -mxN -d /var/db/zope/plone43/zeocluster/develop-eggs/tmpUl2ukSbuild
No local packages or download links found for PasteScript
Traceback (most recent call last):
File "/tmp/tmppWGq8Z", line 11, in <module>
execfile('/var/db/zope/plone43/zeocluster/src/plonetheme.ourtheme/setup.py')
File "/var/db/zope/plone43/zeocluster/src/plonetheme.ourtheme/setup.py", line 37, in <module>
paster_plugins=["ZopeSkel"],
File "/var/db/zope/plone43/Python-2.7/lib/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
File "/var/db/zope/plone43/buildout-cache/eggs/distribute-0.6.28-py2.7.egg/setuptools/dist.py", line 221, in __init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/var/db/zope/plone43/buildout-cache/eggs/distribute-0.6.28-py2.7.egg/setuptools/dist.py", line 245, in fetch_build_eggs
parse_requirements(requires), installer=self.fetch_build_egg
File "/var/db/zope/plone43/buildout-cache/eggs/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 580, in resolve
dist = best[req.key] = env.best_match(req, self, installer)
File "/var/db/zope/plone43/buildout-cache/eggs/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 825, in best_match
return self.obtain(req, installer) # try and download/install
File "/var/db/zope/plone43/buildout-cache/eggs/distribute-0.6.28-py2.7.egg/pkg_resources.py", line 837, in obtain
return installer(requirement)
File "/var/db/zope/plone43/buildout-cache/eggs/distribute-0.6.28-py2.7.egg/setuptools/dist.py", line 294, in fetch_build_egg
return cmd.easy_install(req)
File "/var/db/zope/plone43/buildout-cache/eggs/distribute-0.6.28-py2.7.egg/setuptools/command/easy_install.py", line 592, in easy_install
raise DistutilsError(msg)
distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('PasteScript')
While:
Installing.
Processing develop directory '/var/db/zope/plone43/zeocluster/src/plonetheme.ourtheme'.
An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
File "/var/db/zope/plone43/buildout-cache/eggs/zc.buildout-1.7.1-py2.7.egg/zc/buildout/buildout.py", line 1866, in main
getattr(buildout, command)(args)
File "/var/db/zope/plone43/buildout-cache/eggs/zc.buildout-1.7.1-py2.7.egg/zc/buildout/buildout.py", line 487, in install
installed_develop_eggs = self._develop()
File "/var/db/zope/plone43/buildout-cache/eggs/zc.buildout-1.7.1-py2.7.egg/zc/buildout/buildout.py", line 727, in _develop
zc.buildout.easy_install.develop(setup, dest)
File "/var/db/zope/plone43/buildout-cache/eggs/zc.buildout-1.7.1-py2.7.egg/zc/buildout/easy_install.py", line 1184, in develop
*args) == 0
AssertionError
*************** PICKED VERSIONS ****************
[versions]
*************** /PICKED VERSIONS ***************
Our Product's init.py script contains the following:
# See http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages
try:
__import__('pkg_resources').declare_namespace(__name__)
except ImportError:
from pkgutil import extend_path
__path__ = extend_path(__path__, __name__)
I've seen a similar error reported here but the answer provided does not apply to our case. We have other products in src but this is the only one causing problems.
Can someone please help us with this error?
Check the setup.py file in your theme package. You'll likely find:
# The next two lines may be deleted after you no longer need
# addcontent support from paster and before you distribute
# your package.
setup_requires=["PasteScript"],
paster_plugins = ["ZopeSkel"],
Remove those lines. They are included in the generated template for your package so that you may use Zopeskel local commands to add new functionality. You don't need it after development, and it's often a source of problems later.
The alternative solution, as mentioned in the comments, is to make sure you have the right egg in your local buildout cache. But why have old development packages sitting around on a live site?

Any other ways to install heat resource plug-in?

According to the directions of Openstack Official Heat/Plugins wiki https://wiki.openstack.org/wiki/Heat/Plugins, we only need "To install a plugin, copy the Python modules to one of the configured plugin directories. Note that heat-engine must be restarted after this in order to load the new plugins.". But I hit the following error messages after I restart heat-engine..
[root#cs14 heat]# heat-engine
2015-05-04 06:02:09.774 20839 WARNING heat.common.config [-] HT-A65A0DF The "instance_user" option in heat.conf is deprecated and will be removed in the Juno release.
[05/04/2015 06:02:10 EDT]heatCRITICAL : ImportError: No module named my_heat_plugin.client
Traceback (most recent call last):
File "/usr/bin/heat-engine", line 67, in <module>
srv = engine.EngineService(cfg.CONF.host, rpc_api.ENGINE_TOPIC)
File "/usr/lib/python2.6/site-packages/heat/engine/service.py", line 288, in __init__
resources.initialise()
File "/usr/lib/python2.6/site-packages/heat/engine/resources/__init__.py", line 44, in initialise
_load_global_environment(global_env)
File "/usr/lib/python2.6/site-packages/heat/engine/resources/__init__.py", line 49, in _load_global_environment
_load_global_resources(env)
File "/usr/lib/python2.6/site-packages/heat/engine/resources/__init__.py", line 54, in _load_global_resources
manager = plugin_manager.PluginManager(__name__)
File "/usr/lib/python2.6/site-packages/heat/engine/plugin_manager.py", line 58, in __init__
self.modules = list(modules())
File "/usr/lib/python2.6/site-packages/heat/common/plugin_loader.py", line 91, in load_modules
module = _import_module(importer, module_name, package)
File "/usr/lib/python2.6/site-packages/heat/common/plugin_loader.py", line 72, in _import_module
module = loader.load_module(module_name)
File "/usr/lib64/python2.6/pkgutil.py", line 238, in load_module
mod = imp.load_module(fullname, self.file, self.filename, self.etc)
File "/usr/lib/heat/abc_heat_plugin/resources/abc/abc_server.py", line 24, in <module>
from abc_heat_plugin.client import constants as const
ImportError: No module named abc_heat_plugin.client
To solve this problem, I've figured out two ways which are workable.
Method 1. copy abc_heat_plugin to /usr/lib/python2.6/site-packages and restart heat-engine
Method 2. use .pth file.
1) Create a file /usr/lib/python2.6/site-packages/.pth with the following three lines (no need to do this if it exists)
/usr/lib/heat
/usr/lib/heat/abc_heat_plugin
/usr/lib/heat/abc_heat_plugin/client
2) copy plugin "my_heat_plugin" to /usr/lib/heat
3) restart heat service
BUT both ways have to do more than the official guide, so I wonder whether I missed anything important. Any suggestions? Thanks.
(BTW, my_heat_plugin is working well.)
I think the basic idea is that you drop the resource in /usr/lib/heat, but any modules or packages that are used by the resource must be installed the usual way. You sort of did that manually by moving the files to site-packages.
Take a look at any of the plugins in heat/contrib for example. There are a few that have dependencies such as clients to other OpenStack services, and in all cases the assumption is made that those clients were installed separately.
So if you can, just create a setup.py for your client package, and install that separately from the heat plugin.

wapiti crashes my ASP.NET project. Why? How do i fix it?

Heres one scan of Wapiti. I notice when i had images uploaded (users can upload) i get a crash before Launching module crlf. So just using a fresh instance of my site i ran this and got the result below.
My questions are
1. How do i fix the crashes
2. How might i find out what is causing the crash. I used -v 2 to figure out the url and log them in my app. In both cases i dont see any issues and the project crashes outside of my code
3. How so i solve the unicode warning below?
Wapiti-2.2.1 (wapiti.sourceforge.net)
..............................
Notice
========
This scan has been saved in the file C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src/s
cans/localhost:17357.xml
You can use it to perform attacks without scanning again the web site with the "
-k" parameter
[*] Loading modules :
mod_crlf, mod_exec, mod_file, mod_sql, mod_xss, mod_backup, mod_htaccess
, mod_blindsql, mod_permanentxss, mod_nikto
[+] Launching module crlf
[+] Launching module exec
[+] Launching module file
[+] Launching module sql
C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src\attack\mod_sql.py:185: UnicodeWarning:
Unicode equal comparison failed to convert both arguments to Unicode - interpre
ting them as being unequal
if (page, tmp) not in self.attackedPOST:
[+] Launching module xss
Traceback (most recent call last):
File "wapiti.py", line 449, in <module>
wap.attack()
File "wapiti.py", line 266, in attack
x.attack(self.urls, self.forms)
File "C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src\attack\attack.py", line 121, i
n attack
self.attackGET(page, dictio, headers)
File "C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src\attack\mod_xss.py", line 71, i
n attackGET
self.findXSS(page, {}, "", code, "", payloads, headers["link_encoding"])
File "C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src\attack\mod_xss.py", line 306,
in findXSS
dat = self.HTTP.send(url).getPage()
File "C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src\net\HTTP.py", line 94, in send
info, data = self.h.request(target, headers = _headers)
File "C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src\net\httplib2\__init__.py", lin
e 1084, in request
(response, content) = self._request(conn, authority, uri, request_uri, metho
d, body, headers, redirections, cachekey)
File "C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src\net\httplib2\__init__.py", lin
e 888, in _request
(response, content) = self._conn_request(conn, request_uri, method, body, he
aders)
File "C:\unzipped\wapiti-2.2.1\wapiti-2.2.1\src\net\httplib2\__init__.py", lin
e 853, in _conn_request
response = conn.getresponse()
File "C:\dev\bin\Python26\lib\httplib.py", line 974, in getresponse
response.begin()
File "C:\dev\bin\Python26\lib\httplib.py", line 391, in begin
version, status, reason = self._read_status()
File "C:\dev\bin\Python26\lib\httplib.py", line 349, in _read_status
line = self.fp.readline()
File "C:\dev\bin\Python26\lib\socket.py", line 397, in readline
data = recv(1)
socket.error: [Errno 10054] An existing connection was forcibly closed by the re
mote host
Wapiti can crash applications because it uses a lot of your application. Wapiti stack traced when doing an XSS test, and I don't think an xss test can crash an application. However, by submitting a lot of 1 type of request, then this could cause a DoS condition. You need to track down the last request that Wapiti made. Wapiti has a verbose mode, I think its -v and it will print out every request it makes. Once you have the file that is crashing you should review it manually.
Wapiti's blind sql injection attack module uses mysql's benchmark() function which WILL DoS your mysql server, I recommend turning this one off if you are have trouble scanning your entire site.

Resources