Pyexcel, loading a file to create a book in memory - python-3.4

This is solved; thanks to #vmontco's solution: I was missing MEDIA_URL, now it works perfectly.
----------original question below-----------
I welcome suggestions from every angle; I am fairly new to Django and Python. I'm sure I am missing something simple.
Using a Model Form, with a FileField, I upload and save an Excel file to a folder structure under MEDIA_ROOT. This works.
I want to read that same file later to perform operations using Pyexcel. This is where I am stuck. I am attempting to upload the file using the FileField stored in the DB.
This is where I have problems, and I am not sure if am misunderstanding MEDIA_ROOT, or some other aspect of Django.
When I pass the pk to the 2nd view, I then instantiate an object based on the Model. It has the FileField 'docfile', which I am trying to use to access the file to do some operations using Pyexcel,
here is the FileField declaration from models.py:
docfile = models.FileField(
verbose_name="Choose file to upload:",
upload_to='Excel_CSV_Assets/%Y/%m/%d')
EDIT: If I hard-code the pth to the file like this, everything works, including operations afterwards:
thedocfile='site_static/site/original_assets/Excel_CSV_Assets/2016/04/23/Animals_oglc4DV.xlsx'
book=pyexcel.get_book(file_name=thedocfile)
:END OF EDIT
Here is the code from the 2nd view, where I attempt to read the file into memory, and make a 'book' class object using Pyexcel. I am stuck here:
asset = Excel_CSV_Asset.objects.get(id=assetid)
book=pyexcel.get_book(file_name=asset.docfile)
Here is my error description:
Here is the info right at where my code breaks:
Although it says "Wrong filename", I can see the file is in the folder:
I'm able to open the file by double-clicking; the file is not corrupted.
EDIT:
If I cast the 'asset.docfile' to str, like so:
asset = Excel_CSV_Asset.objects.get(id=assetid)
book=pyexcel.get_book(file_name=str(asset.docfile))
I get a different error:
[Errno 2] No such file or directory: 'Excel_CSV_Assets/2016/04/23/Animals_oglc4DV.xlsx'
...but this is the correct directory, located beneath the MEDIA_ROOT file structure.
Here is settings.py MEDIA_ROOT:
MEDIA_ROOT = 'site_static/site/original_assets/'
Here is urls.py:
urlpatterns = [
url(r'^admin/', include(admin.site.urls)),
url(r'^e/', include('excel_to_mongo.urls')),
] + static(settings.STATIC_URL, document_root=settings.STATIC_ROOT)
Here is the url.py of that app:
url(r'^efactory/(?P<assetid>\d+)/$', 'display_sheet_column_choices', {}),

I think your problem is that you don't fully understand the media files management with Django.
What are media files?
Media files are all the files that are user-uploaded (at running time).
You must not mistake them with Static files that are assets needed by your project to work and that you add at development time (CSS, background picture and JS files for instance).
You shouldn't mix them because they are managed differently by the server and that it could lead to security problems (cf. the warning here):
Static files management :
You put your static files as a part of the code either in one static subdirectory from the installed django applications, either in one of the locations you added to STATICFILES_DIRS.
Static files have to be gathered before starting the server by calling ./manage.py collectstatic, this command will collect (copy) the static files into the a directory (STATIC_ROOT's value).
You then have to set STATIC_URL to choose with wich url you should serve your static files. An usual choice would be /static/. To access the static file you should then try to reach /static/path/to/static/file/in/static_root/dir.
Media files management :
Your media files are added at running time. They are stored in the MEDIA_ROOT location that has to be an absolute path. Hence the fact I suggested you to join the BASE_DIR value (an absolute path) and the subdir you would choose with something like :
MEDIA_ROOT = os.path.join(BASE_DIR, "/media/subdir")
You then have to set an URL for your media files, by using the MEDIA_URL variable. To access your media files, the urls will start with the value you choose :
MEDIA_URL = '/media/'
Then, add this to your urls.py file :
if settings.DEBUG:
urlpatterns = urlpatterns + static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)
With the current example, your mymediafile.txt will be located at /path/to/your/project/media/subdir/path/in/media/root/mymediafile.txt and served at http://127.0.0.1:8000/media/path/in/media/root/mymediafile.txt.
But this is suitable only for a development use as told here. And this would work only for DEBUG == TRUE
For a production use, you should consider deploying your media files with your http server (apache for instance).
Conclusion :
Take the time to understand this. Because I suspect you don't really understood what you did and this lack of understanding could lead to future bugs and errors.

Related

Vaadin Flow 14, Jetty embedded and static files

I'm trying to create app based on Jetty 9.4.20 (embedded) and Vaadin Flow 14.0.12.
It based on very nice project vaadin14-embedded-jetty.
I want to package app with one main-jar and all dependency libs must be in folder 'libs' near main-jar.
I remove maven-assembly-plugin, instead use maven-dependency-plugin and maven-jar-plugin. In maven-dependency-plugin i add section <execution>get-dependencies</execution> where i unpack directories META-INF/resources/,META-INF/services/ from Vaadin Flow libs to the result JAR.
In this case app work fine. But if i comment section <execution>get-dependencies</execution> then result package didn't contain that directories and app didn't work.
It just cannot give some static files from Vaadin Flow libs.
This error occurs only if i launch packaged app with ...
$ java -jar vaadin14-embedded-jetty-1.0-SNAPSHOT.jar
... but from Intellij Idea it launch correctly.
There was an opinion that is Jetty staring with wrong ClassLoader and cannot maintain requests to static files in Jar-libs.
The META-INF/services/ files MUST be maintained from the Jetty libs.
That's important for Jetty to use java.util.ServiceLoader.
If you are merging contents of JAR files into a single JAR file, that's called a "uber jar".
There are many techniques to do this, but if you are using maven-assembly-plugin or maven-dependency-plugin to build this "uber jar" then you will not be merging critical files that have the same name across multiple JAR files.
Consider using maven-shade-plugin and it's associated Resource Transformers to properly merge these files.
http://maven.apache.org/plugins/maven-shade-plugin/
http://maven.apache.org/plugins/maven-shade-plugin/examples/resource-transformers.html
The ServicesResourceTransformer is the one that merges META-INF/services/ files, use it.
As for static content, that works fine, but you have to setup your Base Resource properly.
Looking at your source, you do the following ...
final URI webRootUri = ManualJetty.class.getResource("/webapp/").toURI();
final WebAppContext context = new WebAppContext();
context.setBaseResource(Resource.newResource(webRootUri));
That won't work reliably in 100% of cases (as you have noticed when running in the IDE vs command line).
The Class.getResource(String) is only reliable if you lookup a file (not a directory).
Consider that the Jetty Project Embedded Cookbook recipes have techniques for this.
See:
WebAppContextFromClasspath.java
ResourceHandlerFromClasspath.java
DefaultServletFileServer.java
DefaultServletMultipleBases.java
XmlEnhancedServer.java
MultipartMimeUploadExample.java
Example:
// Figure out what path to serve content from
ClassLoader cl = ManualJetty.class.getClassLoader();
// We look for a file, as ClassLoader.getResource() is not
// designed to look for directories (we resolve the directory later)
URL f = cl.getResource("webapp/index.html");
if (f == null)
{
throw new RuntimeException("Unable to find resource directory");
}
// Resolve file to directory
URI webRootUri = f.toURI().resolve("./").normalize();
System.err.println("WebRoot is " + webRootUri);
WebAppContext context = new WebAppContext();
context.setBaseResource(Resource.newResource(webRootUri));

Django: Image referenced in CSS showing locally but not on Heroku

I'm working in Django, deploying via Heroku, trying to use WhiteNoise for static assets.
I have a CSS file which refers to an image to use as a background:
body { background: white url("images/nyc.jpg") left top; }
When I load the page locally, the background image shows up perfectly.
However, when I deploy to Heroku (using git push from my local machine), the background image does NOT show up on the page. I get this error message when I do the git push to Heroku:
Traceback (most recent call last):
whitenoise.django.MissingFileError: The file 'multiblog/images/nyc.jpg' could not be found with <whitenoise.django.GzipManifestStaticFilesStorage object at 0x7fab5c4fb210>.
The CSS file 'multiblog/style.css' references a file which could not be found:
Please check the URL references in this CSS file, particularly any
relative paths which might be pointing to the wrong location.
Here are the relevant lines from my project level settings.py (happy to share more from it if needed):
# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/1.7/howto/static-files/
BASE_DIR2 = os.path.dirname(os.path.abspath(__file__))
STATIC_ROOT = 'staticfiles'
STATIC_URL = '/static/'
STATICFILES_DIRS = [
os.path.join(BASE_DIR2, 'static'),
]
# Simplified static file serving.
# https://warehouse.python.org/project/whitenoise/
from django.core.wsgi import get_wsgi_application
from whitenoise.django import DjangoWhiteNoise
application = get_wsgi_application()
application = DjangoWhiteNoise(application)
STATICFILES_STORAGE = 'whitenoise.django.GzipManifestStaticFilesStorage'
And here's a picture of the file structure.
In the screenshot you provided, the filename is nyc.JPG, while your code references nyc.jpg. Filenames are case-sensitive :)
Just run
python manage.py collectstatic
see more : collectstatic , Django and Static Assets

Setting Jetty resourcebase to static file embedded in the same jar file

I am trying to access static resource (eg. first.html) packed inside the same .jar file (testJetty.jar), which also has a class which starts the jetty (v.8) server (MainTest.java). I am unable to set the resource base correctly.
The structure of my jar file (testJetty.jar):
testJetty.jar
first.html
MainTest.java
==
Works fine on local machine, but when I wrap it in jar file and then run it, it doesn't work, giving "404: File not found" error.
I tried to set the resourcebase with the following values, all of which failed:
a) Tried setting it to .
resource_handler.setResourceBase("."); // Results in directory containing the jar file, D:\Work\eclipseworkspace\testJettyResult
b) Tried getting it from getResource
ClassLoader loader = this.getClass().getClassLoader();
File indexLoc = new File(loader.getResource("first.html").getFile());
String htmlLoc = indexLoc.getAbsolutePath();
resource_handler.setResourceBase(htmloc); // Results in D:\Work\eclipseworkspace\testJettyResult\file:\D:\Work\eclipseworkspace\testJettyResult\testJetty1.jar!\first.html
c) Tried getting the webdir
String webDir = this.getClass().getProtectionDomain()
.getCodeSource().getLocation().toExternalForm();
resource_handler.setResourceBase(webdir); // Results in D:/Work/eclipseworkspace/testJettyResult/testJetty1.jar
None of these 3 approaches worked.
Any help or alternative would be appreciated
Thanks
abbas
The solutions provided in this thread work but I think some clarity to the solution could be useful.
If you are building a fat jar and use the ProtectionDomain way you may hit some issues because you are loading the whole jar!
class.getProtectionDomain().getCodeSource().getLocation().toExternalForm();
So the better solution is the other provided solution
contextHandler.setResourceBase(
YourClass.class
.getClassLoader()
.getResource("WEB-INF")
.toExternalForm());
The problem here is if you are building a fat jar you are not really dumping your webapp resources into WEB-INF but are probably going into the root of the jar, so a simple workaround is to create a folder XXX and use the second approach as follows:
contextHandler.setResourceBase(
YourClass.class
.getClassLoader()
.getResource("XXX")
.toExternalForm());
Or change your build tool to export the webapp files into that given directory. Maybe Maven does this on a Jar for you but gradle does not.
Not unusually, I found a solution to my problem. The 3rd approach mentioned by Stephen in Embedded Jetty : how to use a .war that is included in the .jar from which Jetty starts? worked!
So, I changed from Resource_handler to WebAppContext, where WebAppContext is pointing to the same jar (testJetty.jar) and it worked!
String webDir = MainTest.class.getProtectionDomain()
.getCodeSource().getLocation().toExternalForm(); ; // Results in D:/Work/eclipseworkspace/testJettyResult/testJetty.jar
WebAppContext webappContext = new WebAppContext(webDir, "/");
It looks like ClassLoader.getResource does not understand an empty string or . or / as an argument. In my jar file I had to move all stuf to WEB-INF(any other wrapping dir will do). So the code looks like
contextHandler.setResourceBase(EmbeddedJetty.class.getClassLoader().getResource("WEB-INF").toExternalForm());
so the context looks like this then:
ContextHandler:744 - Started o.e.j.w.WebAppContext#48b3806{/,jar:file:/Users/xxx/projects/dropbox/ui/target/ui-1.0-SNAPSHOT.jar!/WEB-INF,AVAILABLE}

Meteor reading CSV files to populate database don't work after deploy

So I have a bunch of data that I want to load into database from CSV. I've hacked together a solution that works in local development, but when I deploy to meteor.com, it no longer works.
I'm loading the csv file in the folder /server/data/:
function readData(name){
var fs = __meteor_bootstrap__.require('fs');
var path = __meteor_bootstrap__.require('path');
var base = path.resolve('.');
var data = fs.readFileSync(path.join(base, '/server/data/', name));
return CSVToArray(data);
}
After I deploy to meteor.com, i got:
INFO Error: ENOENT, no such file or directory '/meteor/containers/98eb1286-120b-ee84-8e98-ce673fa2eab7/public/data/categories.csv'
at Object.openSync (fs.js:240:18)
at Object.readFileSync (fs.js:128:15)
at readData (app/server/models.js:10:16)
at app/server/categories.js:6:7
at /meteor/containers/98eb1286-120b-ee84-8e98-ce673fa2eab7/bundle/server/server.js:132:63
at Array.forEach (native)
at Function.<anonymous> (/meteor/containers/98eb1286-120b-ee84-8e98-ce673fa2eab7/bundle/server/underscore.js:76:11)
at /meteor/containers/98eb1286-120b-ee84-8e98-ce673fa2eab7/bundle/server/server.js:132:7
Any idea how I can get meteor to see the csv file after deployment?
I realize this question is old, but it still ranks high on certain keyword searches. So, if you're using Meteor 0.6.5+, you can use the new Assets API.
The issue is that meteor only bundles files that it knows about (ie. JS/CSS/HTML/+more depending on which packages you use) up when it deploys.
Try putting the file you need in the public directory (this directory is exempt from the above rule).
Thanks to SamuelDavis and Tom Coleman's tips. I ended up figuring out what the problem is. Turns out the bundled app is no longer formated as client, public, and server. I ended up debugging it by running meteor bundle to create a tarball. extract the tarball and took a look inside to find where the data folder is. Tom was also right that the data folder needed to be in the public folder in order to get bundled in.
It appears that the base directory is not in the same location that contains the file '/server/data/xxx.csv'.
Before you try anything else, log the base path after calling "var base = path.resolve('.'). If that value is what you expected, log the files that appear in that directory. Again if the files are what you expected, navigate into the /server folder and print out those directories and so forth.
This should pinpoint you to which folder and/or directory is missing and should indicate where you should place the CSV file in future.

Custom stylesheets in Django (1.3) admin

I've got several custom stylesheets that override default admin styles. They live in myproject/static/admin/css-extended. I'm overriding several of Django's default admin templates with templates that live in 'myproject/templates/admin'.
In the templates my stylesheet references are: {{ STATIC_URL }}/admin/css-extended/[stylesheet].css.
However, I can't get the custom stylesheets to pull through.
I've got the following url-related settings in settings.py:
MEDIA_ROOT = os.path.join(PROJECT_DIR, 'media')
MEDIA_URL = '/media/'
STATIC_ROOT = os.path.join(PROJECT_DIR, 'static')
STATIC_URL = '/static/'
ADMIN_MEDIA_PREFIX = '/static/admin/'
I've got the following template context processors listed:
'django.core.context_processors.media',
'django.core.context_processors.static',
Can anyone help please?
Thanks
UPDATE:
I suspect the issue has something to do with the fact that my ADMIN_MEDIA_PREFIX is the same as the first part of the css-extended url ('/static/admin'). I guess Django is trying to find a directory called 'css-extended' in the core admin rather than in the project itself. But how do I get round this?
Are you having this problem on your local dev environment (with runserver)? If so, do you have the following in your urls.py?
from django.contrib.staticfiles.urls import staticfiles_urlpatterns
# ... the rest of your URLconf goes here ...
urlpatterns += staticfiles_urlpatterns()
Read the Django Docs for more.
Ok, I now think I know what's going on here. Hopefully this will clarify:
STATIC_ROOT is only used by the "collectstatic" management command, to figure out where to dump the static files it collects.
STATIC_URL is used by the dev server to define the URL at which the static files will be served.
STATICFILES_DIRS, which you haven't set, is used by both the dev server and the "collectstatic" management command to identify the locations of the static files to serve. In the case of the dev server, the files are served directly in place. In the case of the management command, the files are gathered and copied into STATIC_ROOT.
[Note: there's a convention here -- if you have /static subdirectories in your apps (not your project), they'll be picked up along with anything explicitly defined in STATICFILES_DIRS.]
You just need to add the following to settings.py:
STATICFILES_DIRS = {
'/absolute/path/to/myproject/static/',
}

Resources