So, here is my situation. I have a JavaScript application where I'm appending the hashes to the filenames, as is the standard for Webpack output. This way the content can be safely cached by the browser, with the fresh load controlled by the changing file hash.
My problem is I have a situation where I need other applications to access mine, and they won't be able to be updated every time the hash changes. So I need a request like this:
https://my-domain.com/assets/js/app.js
to be redirected to
https://my-domain.com/assets/js/app.ab12cd34.js
My application currently uses nginx to serve up the pages, but nginx is static. I don't know how to configure it to dynamically identify the hashed file name and return it.
The app is being deployed to a Pivotal CloudFoundry environment. PCF supports evaluating dynamic Ruby code in an nginx.conf file, so that seemed like an easy way around this. Unfortunately, my company requires that the nginx.conf go through a special parser to enforce security headers. This parser only knows nginx syntax, and mangles any Ruby code there.
So, that leaves me with Webpack. I started investigating ways for Webpack to modify files during the build process, and I discovered the transform() function in the copy-webpack-plugin. It has the ability to modify the files exactly how I need. What is still a challenge, though, is getting the hash filename.
So, I'm hoping there's some way to gain access to what the hash filename will be in this plugin, so that I can inject it into the nginx.conf.
Alternatively, if someone knows another way to get around my core problem, I'm all ears.
You can use the webpack-manifest-plugin to create a manifest file with a filename -> chunkname/bundlename mapping.
This manifest file can then be consumed by any piece of software that needs it.
Total newbie question but what is the best practice when it comes to using SSH with Git? I'm working on a WordPress project. In the root I have gulp and other dev files/folders like SASS and Scripts that I don't need on the server and in the same project I have my WordPress folder that contains a theme and a few custom plugins. As you can imagine when the theme or any of the plugins are ready to be deployed I don't want to pull everything in my repository on the server. So far as a newbie I've always just pull and pushed the entire repository and used FTP to upload what I need to the server, so how is this done with SSH and Git and is there a better way to have my setup?
EDIT: To make my question a little bit more clear let me give you an example of what I think my issue is. In my main project folder, I have a SASS folder next to my WordPress folder. All I really need to deploy to the server is the WordPress folder. My build process that happens on my dev machine combines all of the SASS files into a single CSS that is then placed into the WordPress folder. I need the SASS folder to be tracked by Git so that any other developer can pull them and continue developing so I can't have git ignore it. However none of those SASS files need to be on the server for WordPress to work either. I just simply need to deploy the WordPress folder and everything that's in it.
I understand the idea of creating a bare repository on the server and using post-receive hook to point the git folder sitting outside your web root to point to where the web root is. But that's basically how GIT and SSH work and that's not answering my concern.
Not with Git
Git is not designed to pull specific files or directories only. It's a directed acyclic graph with binary blobs as objects and sometimes multiple objects get compacted into a single larger object.
Due to Git design, your specific request is not possible.
Alternatives
post-receive hook
If your website only contains simple static files then it's okay to push to a git repository over SSH. In reality, it's unlikely your repository will be large as long as you don't have non-text files.
Take for example the following setup.
/var/lib/www - apache web dir which is the cloned copy of www.git
/var/lib/www.git - a bare git repository.
/var/lib/www.git/hooks/post-recieve - A server side git hook. It can be a shell script that pulls the www repository when this repository is updated.
Sample post-recieve hook script:
#!/bin/bash
cd /home/sam/sandbox/git-hooks/www
unset GIT_DIR
git fetch origin master
git reset --hard origin/master
Zip up build in a tar.gz
At the end of your build you can zip up your files in a tar.gz. This file should be hosted somewhere (perhaps GitHub releases if you're using GitHub). Some enterprises use on premise artifact hosting like Nexus or Artifactory.
The idea being: you have a tested artifact that has a specific sha256sum. The artifact you test is the exact same artifact which eventually goes to production.
Diving into more detail such as continuous integration, continuous delivery, and the software development life cycle might be out of scope for your question.
No best practice.
Git is for source control, not for deployment. There is no best practice for using git this way because git is not a deployment tool. You also don't need git history on your server. In fact, you don't need git at all unless you insist on using it for deployment. You are welcome to use it this way but it's not ideal because of exactly the kind of problem you're asking about.
What is the best practice?
There are a number of tools you could use to handle your deployments. Most of the tools generally let you set up a series of steps that let you deploy the code you want into the environment you want. You could go with simple tools such as Phing or Deployer in the PHP world, or something more sophisticated like Puppet or Chef if you have more complex needs. You could just write your own bash scripts if what you need is really very simple. I recommend Phing or Deployer given the info you've provided. https://deployer.org/ https://www.phing.info/
You'll just configure whichever tool you want to ssh into your target box and copy over only the files you want into the directory you want on the server, in whatever way you would like to do that. Usually, you have the script copy files into a temp dir, tarball them up, ssh them over and untar them. After that, you'll usually do some additional work on the server to move files around, change symlinks, whatever else you might need to do.
What about compiled SASS, ES6 js files, or modern static stuff?
All you need to do is add steps to the handle the static files and where you want them to go. Include the generated static files in your tarball when you push stuff up, and put them in the right directories in the server once you untar it.
When you configured your SASS compiler, and whatever other pre-compiled static code you may have - you configured it to create a destination file. That is, the file(s) of actual CSS and JS that they generate. That's all you need to bring along - and if you have the destination directory set to be inside your wordpress theme, you may not even have to pay all that much special attention to it's handling. You may need to move them somewhere else once they are on the server but that all depends on the specific setup in your server, which I think is outside the scope of this question.
Additional Notes:
You didn't ask about this but I thought it was worth mentioning, that you shouldn't be sending the entire wordpress repository every time you update. Just like you don't need the uncompiled SASS code, you also don't need to be repackaging core WordPress. You don't even need to be commiting core WordPress, its a dependency and you don't need to be changing it.
All that should be getting committed by you is your theme and plugin code, and the uncompiled static files. Compiled static files and external dependencies like the WordPress core don't belong in your git history. For deployment purposes, WordPress should already be installed. The stuff in your tarballs should just be plugins and themes, and additional static files if they aren't already in there for some reason.
TLDR;
Don't use git for this. Use a tool like Phing or Deployer. Build your static files into your theme, and create phing/deployer scripts that tarball up only the code you want, SSH's it over to your server, and untars it into the directories you want. If you have some special location on the server for your static files, just make sure to add steps in your script for that.
So, based on your question and comment, there are three computers involved. There is a web server (when you say "server", I take it as a Web server in this scenario, or the server computer that runs a Web server program). There is another server where your git repo is hosted. And, there is your dev workstation. Is this correct?
It seems like, you have a cloned git repo on your Web server. Your current practice/workflow appears to be (1) (based on your expression "SSH'ed into my server") you log into the web server via SSH (just like Telnet) from your workstation (SSH is just a protocol, which can be used for different purposes). (2) you pull from your repo on hosted service (e.g., github), and (3) deploy it to your "www" directory on the same server. Is this correct?
(I can think of an alternative scenario based on your use of the word "FTP", etc., but let's focus on the above scenario, for now.)
Now, your question is, whenever you "pull" (on your Web server), you feel like you are pulling everything from your repo on your hosted service. And, is there a better way? Am I understanding your question correctly?
If so, as another commenter suggested, git (and, any version control system, in general) is very good at fetching "deltas" only. If you are worried about "fetching everything" every time you pull (the step (2) above), then your worry is unfounded.
Now, the question is, why do you have a git repo on your Web server, if that is indeed the case? This is a pretty legit setup and I've done this before (e.g., on EC2). But, as a best practice, people generally don't do that on "production" servers. It's because you have to "build" your web app, and you really don't want to do that on production servers.
The next question is, what do you exactly do in Step (3)? The build process (whatever process you use) typically generates an "output" which can be directly deployed to the web server. (The convention is the output is generally a single folder, "public", "www", "dist", or whatever, or a single file (e.g., tar.gz, zip, jar, war), etc.) Regardless of whether you build the deployable output on your dev workstation (or, a build machine) or on your Web server, you don't generally do "deltas" in this context. Even if you've only changed a single file (say, a CSS file), you generally build the whole output again (instead of, say, just replacing the changed CSS file only). When you use FTP to upload files, etc., you can selectively upload certain files and/or directories, etc., but as a general practice, we don't do that. We always build the complete output from scratch and deploy it to the Web server. (This is mainly to reduce the potential deployment errors and increase the reliability.)
So, to answer your question, (A) If you are pulling git repo on your Web server, you should really change that practice, and move the build process to your dev computer or a dedicated build machine. (BTW, services like github, gitlab, TFS, ... provide the build service for you.) (B) If you are currently selectively FTP'ing your web app files to your Web server, then you should really consider adopting some kind of formal build, and deployment, process moving forward.
After your SASS build process is done use scp or rsync to move the files to the prod server:
scp -r /[local wordpress dir]/wp-content/themes/your-theme/ username#your.prod.server.com:/path/to/dir/wordpress/wp-content/themes/
scp -r /[local wordpress dir]/wp-content/plugins/* username#your.prod.server.com:/path/to/dir/wordpress/wp-content/plugins/
I am working in a project and using git ssh with bitbucket following is the process i am using it may work for you also if not please correct me :
Step 1 ->I have setup git and create repo in bit-bucket.
Step 2 ->And setup project with my local and linked with my repo.
Step 3 ->connect my server using ssh.
Step 4 ->Work in my local and commit and push all changes in my git repo.
Step 5 ->Run git pull on ssh so all changes deployed in my server.
I am using above process and i love this process.i have used .gitignore file that is not required for push on my repo.
Thanks
I am trying to use nginx for serving static contents(images/css etc.)
I need to span up multiple instances of nginx to support as per the incoming load.
So i am looking for Mongo+gridfs solution to store the static files- since it provides replication and sharding.
I see i can serve contents from gridfs using either of these these modules.
Direct nginx module -
https://github.com/mdirolf/nginx-gridfs
Using Lua scripting language
https://github.com/bigplum/lua-resty-mongol
Question is - can i create UploadImage api in nginx itself to store files in gridfs when user calls a POST method passing the file.
It looks to me that it is possible using lua resty module but not sure. Any idea?
You can use the lua-resty-upload module to handle user uploads, and then pass the data over to lua-resty-mongol for writing to Mongo.
For large files you may be able to write the chunks directly as they are read to avoid buffering all of the data in memory, there's a good example on the page using a file.
I have used the upload resty module along with lua mongol module.. and it works well..
Now i got a suggestion from people around to see if we can use java. instead of lua to do db connections primarily to store retrive static file contents.
I see there is a Java module as well that can be used to do the job, or can use php or python as well in nginx.
Q is What would be the difference in using any of these languages- Lua vs Java vs PHP. and what factors should i need to consider while picking up a language.. Performance, solution usage, packaging, etc. Point of view
Background
I develop a web application that lives on an embedded device. In order to make dev times sane, frontend development is done using apache serving static documents, with PHP proxying out to the embedded device for specifically configured dynamic resources. This requires that we keep various server-simulation scripts hanging around in source control, and it requires updating those scripts whenever we add a new dynamic resource.
Problem
I'd like to invert the logic: if the requested document is available in the static documents directory, serve it; otherwise, proxy the request to the embedded device.
Optimally, I want a software package that will do this for me (for Windows or buildable on cygwin). I can deal with forcing apache to do it with PHP, but I'm unsure how to configure it to make it happen. I've looked at squid and privoxy, but neither of them seem to do what I want.
Any ideas? I'd rather not have to roll my own.
Now, Varnish is available in cygwin, see:
Installation instructions: http://varnish-cache.org/trac/wiki/VarnishOnCygwinWindows
I think what you want is varnish.
Now that I've looked at varnish, I understand that what I actually want is a special case of a reverse proxy, and that squid can be configured to do what I need. (With the added bonus of having it available as a cygwin package.)
As part of my overall development practices review I'm looking at how best to streamline and automate our ASP.net web development practices.
At the moment, our process goes something like this:
Designer builds frontend as static HTML/CSS on a network share. This gets tweaked until signed off. (e.g. http://myserver/acmesite_design)
Once signed off, developer takes over and copies over frontend HTML/CSS to a new directory on the same server (e.g. http://myserver/acmesite_development)
Multiple developers work on local copy until project is complete.
Developer publishes code to an external publicly accessible server for a client to review/signoff.
Edits made locally based on feedback.
Republish to external server.
Signoff
Developer publishes to live public server
What goes wrong? Lots of things!
Version Control — this is obviously a must and is being introduced
Configuration errors — many many times, there are environment specific paths and variables (such as DB names, image upload directories, web server paths etc. etc.) which incorrectly get copied from local to staging to live etc. etc. with very embarrassing results.
I'm pretty confident I've got no.1 under control. What about configuration management? Does anyone have any advice as to how best to manage an applications structure within asp.net apps to minimize these kinds of problems?
I found that using SVN, NAnt and NUnit with Cruise Control.net solves a lot of the issues you describe. I think it works well for small groups and it's all free. Just need to learn how to use them.
CruiseControl.net helps you put together builds and continuous integration.
Use NAnt or MSBuild to do different environment builds (DEV, TEST, PROD, etc).
http://confluence.public.thoughtworks.org/display/CCNET/Welcome+to+CruiseControl.NET
You got the most important part right. Use version control. Subversion is a good choice.
I usually store configuration along with the site; i.e. when coding a PHP-based site I have a file named config.php-dist. If you want the site to work at all you'll have to copy + edit in all the required parameters (this avoids storing passwords in version control). The -dist file should have reasonable defaults.
Upload directories should be relative if possible; actually all directories should be relative. I'm not experienced in ASP.net, but if it's anything like PHP the current directory is always the directory of the file being requested. If you channel all requests through a single file (i.e. index.asp), then this can even be found programmatically. Or you could find it programmatically by using the equivalent of dirname(____FILE____) in your configuration file.
I also recommend installing IIS (or whatever webserver you are using) on all development workstations (including the designers). Makes life easier as noone can step on each others toes. What one has to do is simply add test hosts to the hosts file (\windows\system32\drivers\etc\hosts iirc) in addition to adding a site to the local IIS. This plays well with version control (checkout, add site to IIS and hosts-file, edit edit edit commit).
One thing that really helps is making sure you keep your paths relative where you can and centralise them where you can't, so when I've been working with ASP.Net I have tended to use web.config to store any configuration and path related data that can't be found programmatically. It is quite possible to find information like your current application path programmatically through the Request object - it's worth looking in some detail over what the environment makes available to you.
One way to make sure you don't end up on something that is dependent on the path name is having a continuous integration server executing your test suite against your application. Each time this happens you create a random filepath. As soon as someone introduces a dependency on the filepath it will fail.