This question focuses on finding a perfect solution for an asynchronous image resizing process guided by a cron job / job queue.
Images are getting uploaded and saved in the original size on the server. The resize process picks them up, resizes and saves them in two new sizes. For this job I use Imagemagick with the following setup [nice -n 19 convert -density 500 "' . $fullpath . '" -resample ' . $dpi . ' "' . $destFolder . '/%d.jpg"']. "nice" will not prevent a command from using 100% CPU. What it does do is say that if any other process needs CPU, it will get a higher priority than the "nice" processes. As you see this is just some improvement :( because I am sure if users will upload pictures simulataneously the server will definitely crash.
My intention is to process all images through the server and not locally via Flash or Java. So if I can't get more server CPU & RAM, I think I need to implement an asynchronous resize job queue through a cron job which gets offered a limited CPU & Memory usage. While an image is processed, the user gets a thumbnail with the info, "your image will be soon available", to stop him from uploading the same image again. That's my basic idea at the moment.
I work with a PHP framework on a Debian Linux server for this web application. Any help, guiding tips or information resources for such a cron job/job queue/implementation ideas are highly appreciated!!!
Many thanks in advance!
Cheers
Thommy
I don't know about the ready made solutions but if you can make a PHP file that can select and upload images from the directory/database - you can setup a worker at www.iron.io and get this task done!
May be implement a REST API to interact remotely is also a good idea. Good luck!
Related
I have an ASP.NET website, and I am using Quartz to do some background work. The Quartz job is set to trigger every 5 minutes.
Will Quartz:
Run in the background continiously after the last user has left the
website?
Stop at some point of time, and continue when a new user enters the website?
If the answer is #2, is there a way to have the job run all the time?
At some point, after some time of inactivity, your IIS hosted application will fall in some king of 'sleep mode'. From this moment, your trigger won't launch jobs anymore, until your app get reactivated by a user trying to access to it.
Here are some solution available to you :
Run your job in a separated windows service. Need some refactoring job but clearly the safest solution
Hangfire is a another .Net background job library. Here's a link they provide to their users to fix this problem, by Making ASP.NET application always running. Note that you need to have full control of your hosting environment to achieve this.
You could set some service as Pingdom to to prevent your site from faling into this 'sleep mode'. However, I would not recommand this solution.
Note that the time needed before your app fall into 'sleep mode' can change from an environment to another. If in your case It need more than 5 minutes (the time between each of your jobs), this process could be enough to keep your web site up. In fact, it would act as the solution #3 (use an external service to poke your environment at a frequent rate)
When starting the Purge Tool from the Content Manager server, nothing happens except I just get a 100% CPU process which kills my server instantly, any suggestions?
It's the first time I've executed this tool so I'm not sure what's supposed to happen; does it connect to the DB immediately? - the 190 GB DB I'm lumbered with might not be helping it if that's the case.
Thanks in advance.
I have also seen this in past, you may try this with publication by publication or with limited number of folders/structure group.
Yes purge tool use the database immediately.
You should get a dialog box - It can take a while to load, but in the past I have seen this to be a permissions problem. Try right-clicking on the program, and using "Run as Administrator" if you are on a Windows 2008 Server
The Purge tool starts out by trying to "consume" your blueprint structure. It does this in a pretty cumbersome way however, it will get there in the end. Just leave it to run, it will get there.
The Purge tool doesn't connect to the db directly. It uses the Tridion API. (Of course, Tridion then goes to the database.)
On starting, the Purge tool will load all your publications, which it then displays for you. If you have a lot of publications, and a slow system, this can take a while.
If there is a permissions problem, I'd expect it to be one of Tridion permissions, so ensure that the account you are running as it a Tridion administrator.
I am thinking of designing a webbased frontend to a dataanalysis package. This is what I have in mind:
Data-analysis jobs are submitted to a queue.
People can login to the frontend and choose to run a job on the local machine or add a new job to the queue.
Details:
A person logs into the web-application and can either choose to run an exisiting job in the work queue on the local machine or assign a new job to the work queue
If a job is selected to be run on the local machine, the webpage queries the package installed on the local machine : "This is the job I have for you. How long would it take for you to finish this?"
2.1 The package would return a value (based on heuristics) to the webpage
2.2 The webpage can, depending on the value returned, choose to
allocate the job to the current
machine
2.3 If the job is allocated to the current machine, the backend package
goes on its way all the while
keeping the webpage updated with the
progress it is making.
A job can take from a few seconds to
a few hours to finish.
2.4 Any time the webpage can tell the package "You are taking too long -
send me what you have" or perhaps "I have something more important for you -
send me what you have".
The package can send the data it has by connecting to the webapp backend
directly at this point.
This needs to run on Windows and Linux - so I will be using the npapi plugin architecture.
The webpage would do most of the interfacing via Javascript.
My questions are:
How do I make the C++ npapi plugin return a value (a json structure) to the webpage javascript and then have a function in the webpage execute without the webpage asking the plugin to do it (to achieve 2.1). That is - the plugin initiates the interaction instead of the usual "webpage asks the plugin to do something"
Is there a better way to implement 2.3 than setInterval?
How do I make the C++ npapi plugin return a value (a json structure) to the webpage javascript and then have a function in the webpage execute without the webpage asking the plugin to do it (to achieve 2.1). That is - the plugin initiates the interaction instead of the usual "webpage asks the plugin to do something"
Use this example.
Is there a better way to implement 2.3 than setInterval?
setInterval works well enough.
I am about to implement the schduled kind of task, not sure whether should implement in window sevice or window schduler.
The use case is, there will be one executable deployed on the mechine, which is attached to the scanner. For every five minutes, the exe will be reading the scanned files from the specified folder and upload the files to the server.
What would be the best solution for this use case.
Thanks
Use a scheduled task. A windows service would have to be specially written, and this is perfectly suited for a simple job which runs at 5 minute intervals.
You'll find some good comparisons here:
windows service vs scheduled task
Personally, I would use the Windows Service because it is easier to troubleshoot, locate the logs and restart if necessary.
I have deployed an application written in ASP.NET 2.0 into production and it's experiencing some latency issues. Pages are taking about 4-5 seconds to load. GridView refreshing are taking around the same time to load.
The app runs fine on the develpment box. I did the following investigation on the server
Checked the available memory ... 80% used.
Cheched the processor ... 1%
Checked disk IO from perfmon, less than 15%
The server config is
Windows Server 2003 Sp2
Dual 2.0 GZH
2GB RAM
Running SQL Server 2005 and IIS only
Is there anything else I can troubleshoot? I also checked the event log for errors, it's clean.
EDITED ~ The only difference I just picked up is on the DEV box I am using IE7 and the clients are using IE6 - Could this be an issue?
UPDATE ~ I updated all clients to IE8 and noticed a 30% increase in the performance. I finally found out I left my debug=true in the web.config file. Setting that to flase got the app back to the stable performance... I still can't believe I did that.
First thing I would do is enable tracing. (see: https://web.archive.org/web/20210324184141/http://www.4guysfromrolla.com/webtech/081501-1.shtml)
then add tracing points to your page generation code to give you an idea of how long each part of the page build takes:
System.Diagnostics.Trace.Write(
"Starting Page init",
"TraceCheck");
//Init page
System.Diagnostics.Trace.Write(
"End Page init",
"TraceCheck");
System.Diagnostics.Trace.Write(
"Starting Data Fetch",
"TraceCheck");
//Get Data
System.Diagnostics.Trace.Write(
"End Data Fetch",
"TraceCheck");
etc
this way you can see exactly how long each stage is taking and then target that area.
Double check that you application is not running in debug mode. In your web.config file check that the debug attribute under system.web\compilation is set to false.
Besides making the application run slower and using more system memory you will also experience slow page loading since noting is cached when in debug mode.
Also check your page size. A developer friend of mine once loaded an entire table into viewstate. A 12 megabyte page will slip by when developing on your local machine, but becomes immediately noticeable in production.
Are you running against the same SQL Server as in your tests or a different one?
In order to find out where the time's coming from you could add some trace statements to your page load, and then load the page with tracing turned on. That might help point to the problem.
Also, what are the specs of your development box? The same?
Depending on the version of visual studio you have, Team Developer has a Performance Wizard you might want to investigate.
Also, if you use IE 8, it has a Profiler which will let you see how long the site takes to load in the browser itself. One of the first things to determine is whether the time is client side or server side.
If client side, start looking at what javascript you have and optimize / get rid of it.
If server side, you need to look at all of the performance counters (perfmon). For example, we had an app that crawled on the production servers due to a tremendous amount of JIT going on.
You also need to look at the communication between the web and database server. How long are queries taking? Are the boxes thrashing the disk drives? etc.