Can't create or update records in a loop, using ActiveStorage - ruby-on-rails-6

Ruby 2.6.3
Rails 6.0.3
I don't why, when I try to create or update a record in a loop, adding an attachment or attachments, just only one or two recoreds can be updated. I also tested this behaviour with only one attachment and only for creating. But the behaviour is the same.
In the code below, I'm trying to reload attachments on aws, using active storage. But only 1 record is updated.
namespace :uploads do
task migrate_to_active_storage: :environment do
Upload.where.not(csv_file_name: nil).find_each do |upload|
%w[csv log].map do |attachment_type|
attach_url = [
'https://',
ENV['AWS_BUCKET'],
".s3-#{ENV['AWS_REGION']}.amazonaws.com/",
upload.business.normalized_name,
'/upload/',
upload.id.to_s,
"/#{attachment_type}/",
upload.csv_file_name
].join
upload.send(attachment_type).attach(
io: open(attach_url),
filename: upload["#{attachment_type}_file_name"],
content_type: upload["#{attachment_type}_content_type"]
)
end
end
ap 'DONe WITH SUCCESS' # never appears
end
end
I've managed to fix this, using threds. But I don't know why it doesn't work in the code above.
This solution works fine.
namespace :uploads do
task migrate_to_active_storage: :environment do
uploads = Thread.new { Upload.ids }.value
uploads.each do |id|
Thread.new do
sleep 2
upload = Upload.find id
upload.update csv: get_attachment(upload, 'csv'),
log: get_attachment(upload, 'log')
end.join
end
end
end
def get_attachment(upload, attachment_type)
attachment_url = [
'https://',
ENV['AWS_BUCKET'],
".s3-#{ENV['AWS_REGION']}.amazonaws.com/",
upload.business.normalized_name,
'/upload/',
upload.id.to_s,
"/#{attachment_type}/",
upload["#{attachment_type}_file_name"]
].join
{
io: open(attachment_url),
filename: "#{upload.id}_#{upload["#{attachment_type}_file_name"]}",
content_type: upload["#{attachment_type}_content_type"]
}
end
And what is really a mystic. When I try to call a differen model, for instance, like User. And even if it does nothing, the code above won't work. Only 1 record will'be updated.

Related

How can a function get the module name of the caller?

Say I have two lua modules, like this.
Filename: my/lua/app/caller.lua
Module name: my.lua.app.caller
local reflect = require('my.lib.reflect')
-- should return "my.lua.app.caller"
reflect.get_caller_module_name()
Filename: my/lib/reflect.lua
Module name: my.lib.reflect
M = {}
function M.get_caller_module_name()
-- magic here?
return module_name
end
return M
Now running my.lua.app.caller should allow it to get its own module name, my.lua.app.caller. Is this doable in Lua?
(I realize that I can get the module name by inserting local module_name, _ = ... as the very first line of the caller's module, but I want to be able to do this without this without needing any special modifications to the caller's module.)

Virtual Filesystem for PHPUnit tests in Laravel 5.4

i'm having a bit of a problem with my PHPUnit integration tests, i have a method which handles a form upload for a video file as well as a preview image for that video.
public function store($request)
{
/** #var Video $resource */
$resource = new $this->model;
// Create a new Content before creating the related Photo
$contentRepo = new ContentRepository();
$content = $contentRepo->store($request);
if($content->isValid()) {
$resource->content_id = $content->id;
$directory = 'frontend/videos/assets/'.date("Y").'/'.date('m').'/'.time();
\File::makeDirectory($directory, 0755, true);
$request->video->move($directory.'/', $request->video->getClientOriginalName());
$resource->video = '/'.$directory.'/'.$request->video->getClientOriginalName();
$request->preview_image->move($directory.'/', $request->preview_image->getClientOriginalName());
$resource->preview_image = '/'.$directory.'/'.$request->preview_image->getClientOriginalName();
$resource->highlighted = intval($request->input('highlighted') == 'on');
$resource->save();
return $resource;
}
else {
return $content;
}
}
The important part to keep is the $request->video->move() call which i probably need to replace in order to use Virtual Filesystem.
and then the test
public function testVideoUpload(){
File::put(__DIR__.'/frontend/videos/assets/image.mp4', 'test');
$file = new UploadedFile(__DIR__.'/frontend/videos/assets/image.mp4', 'foofile.mp4', 'video/mp4', 100023, null, $test=true);
File::put(__DIR__.'/frontend/images/assets/image.jpg', 'test');
$preview = new UploadedFile(__DIR__.'/frontend/images/assets/image.jpg', 'foofile.jpg', 'image/jpeg', 100023, null, $test=true);
$this->post('/admin/videos', [
'title' => 'My Video #12',
'description' => 'This is a description',
'actors' => [$this->actor->id, $this->actor2->id],
'scenes' => [$this->scene->id, $this->scene2->id],
'payment_methods' => [$this->paymentMethod->id],
'video' => $file,
'preview_image' => $preview
])->seeInDatabase('contents', [
'title' => 'My Video #12',
'description' => 'This is a description'
]);
}
As you can see, i need to create a dummy file in some local directory and then use that in the HTTP request to the form's endpoint, then after that, that file would be moved and i need to delete the created folder and the new moved file... it's an authentic mess.
As such i want to use Virtual Filesystem instead, but i have no idea how to set it up in this particular case, i've already downloaded a package and set it up, but the questions are, first, which package have you used/recommend and how would you tweak the class and the test to support the Virtual Filesystem? Would i need to switch over to using the Storage facade instead of the $request->video->move() call? If so how would that be done exactly?
Thank you in advance for your help
I couldn't figure out the VFS system, however i do have somewhat of an alternative that's still kinda messy but gets the job done.
Basically i set up two methods on my PHPUnit base class to setup and teardown the temp folders i need on any test that requires them, because i'm using Database Transactions the files get deleted on every test run and i need to create new dummy files every time i run the test.
So i have two methods setupTempDirectories and teardownTempDirectories which i will call at the beginning and at the end of each test that requires those temporary directories.
I put my temp files in the Storage directory because sometimes i run my tests individually through PHPStorm and the __DIR__ command gets messed up and points to different directories when i do that, i also tried __FILE__ with the same result, so i just resorted to using Laravel's storage_path instead and that works fine.
Then that leaves the problem of my concrete class which tries to move files around and create directories in the public folder for them... so in order to fix that i changed the code to use the Storage facade, then i Mock the Storage facade in my tests
So in my concrete class
$directory = 'frontend/videos/assets/'.date("Y").'/'.date('m').'/'.time();
Storage::makeDirectory($directory, 0755, true);
Storage::move($request->video, $directory . '/' . $request->video->getClientOriginalName());
$resource->video = '/'.$directory.'/'.$request->video->getClientOriginalName();
Storage::move($request->preview_image, $directory . '/' . $request->preview_image->getClientOriginalName());
$resource->preview_image = '/'.$directory.'/'.$request->preview_image->getClientOriginalName();
And then in my test i mock both the makeDirectory and the move methods like such
// Override the Storage facade with a Mock version so that we don't actually try to move files around...
Storage::shouldReceive('makeDirectory')->once()->andReturn(true);
Storage::shouldReceive('move')->twice()->andReturn(true);
That makes my tests work and does not actually leave files behind after it's done...i hope someone has a better solution but for the time being this is what i came up with.
I was actually trying to use VFS but it never worked out... i keep getting errors that the original file in the storage directory is not found even though it's right there...
I'm not even sure the Storage facade was using VFS in the background to begin with even though it should...

Meteor Package: Add Custom Options

I've created a Meteor smart package, and would like to add user generated custom options to the API.
However, I'm having issues due to Meteor's automatic load ordering.
SocialButtons.config({
facebook: false
});
This runs a config block that adds defaults.
SocialButtons.config = function (options) {
... add to options if valid ...
};
Which in turn grabs a set of defaults:
var defaults = {
facebook: true,
twitter: true
}
Which are mixed into the settings.
var settings = _.extend(defaults, options);
...(program starts, uses settings)...
The problem is that everything must run in the proper order.
Create SocialButtons object
Run the optional SocialButtons.config()
Create settings & run the program
How can I control the load order in Meteor without knowing where a user might place the optional configuration?
Step 2 will be in a different folder/file, but must run sandwiched between steps 1 & 3.
You can't really control load order right now so it's not guaranteed but placing files at /libs are loaded first but in your case it's doesn't really matter it might be something else here is a very simple package you can view the source on how I setup default options and allow to replace those easily https://github.com/voidale/meteor-bootstrap-alerts
Figured this out.
Put your package into a /lib directory.
Include a setup function that sets the settings when called, and loads the data
Return the data from the startup function
In this case:
SocialButtons.get = function () {
return initButtons();
}
function initButtons() { ... settings, startup, return final value ... }

How to list files in folder

How can I list all files inside a folder with Meteor.I have FS collection and cfs:filesystem installed on my app. I didn't find it in the doc.
Another way of doing this is by adding the shelljs npm module.
To add npm modules see: https://github.com/meteorhacks/npm
Then you just need to do something like:
var shell = Meteor.npmRequire('shelljs');
var list = shell.ls('/yourfolder');
Shelljs docs:
https://github.com/arturadib/shelljs
The short answer is that FS.Collection creates a Mongo collection that you can treat like any other, i.e., you can list entries using find().
The long answer...
Using cfs:filesystem, you can create a mongo database that mirrors a given folder on the server, like so:
// in lib/files.js
files = new FS.Collection("my_files", {
stores: [new FS.Store.FileSystem("my_files", {"~/test"})] // creates a ~/test folder at the home directory of your server and will put files there on insert
});
You can then access this collection on the client to upload files to the server to the ~test/ directory:
files.insert(new File(['Test file contents'], 'my_test_file'));
And then you can list the files on the server like so:
files.find(); // returns [ { createdByTransform: true,
_id: 't6NoXZZdx6hmJDEQh',
original:
{ name: 'my_test_file',
updatedAt: (Date)
size: (N),
type: '' },
uploadedAt: (Date),
copies: { my_files: [Object] },
collectionName: 'my_files'
}
The copies object appears to contain the actual names of the files created, e.g.,
files.findOne().copies
{
"my_files" : {
"name" : "testy1",
"type" : "",
"size" : 6,
"key" : "my_files-t6NoXZZdx6hmJDEQh-my_test_file", // This is the name of the file on the server at ~/test/
"updatedAt" : ISODate("2015-03-29T16:53:33Z"),
"createdAt" : ISODate("2015-03-29T16:53:33Z")
}
}
The problem with this approach is that it only tracks the changes made through the Collection; if you add something manually to the ~/test directory, it won't get mirrored into the Collection. For instance, if on the server I run something like...
mkfile 1k ~/test/my_files-aaaaaaaaaa-manually-created
Then I look for it in the collection, it won't be there:
files.findOne({"original.name": {$regex: ".*manually.*"}}) // returns undefined
If you just want a straightforward list of files on the server, you might consider just running an ls. From https://gentlenode.com/journal/meteor-14-execute-a-unix-command/33 you can execute any arbitrary UNIX command using Node's child_process.exec(). You can access the app root directory with process.env.PWD (from this question). So in the end if you wanted to list all the files in your public directory, for instance, you might do something like this:
exec = Npm.require('child_process').exec;
console.log("This is the root dir:");
console.log(process.env.PWD); // running from localhost returns: /Users/me/meteor_apps/test
child = exec('ls -la ' + process.env.PWD + '/public', function(error, stdout, stderr) {
// Fill in this callback with whatever you actually want to do with the information
console.log('stdout: ' + stdout);
console.log('stderr: ' + stderr);
if(error !== null) {
console.log('exec error: ' + error);
}
});
This will have to run on the server, so if you want the information on the client, you'll have to put it in a method. This is also pretty insecure, depending on how you structure it, so you'd want to think about how to stop people from listing all the files and folders on your server, or worse -- running arbitrary execs.
Which method you choose probably depends on what you're really trying to accomplish.

Marionette js itemview not defined: then on browser refresh it is defined and all works well - race condition?

Yeah it's just the initial browser load or two after a cache clear. Subsequent refreshes clear the problem up.
I'm thinking the item views just aren't fully constructed in time to be used in the collection views on the first load. But then they are on a refresh? Don't know.
There must be something about the code sequence or loading or the load time itself. Not sure. I'm loading via require.js.
Have two collections - users and messages. Each renders in its own collection view. Each works, just not the first time or two the browser loads.
The first time you load after clearing browser cache the console reports, for instance:
"Uncaught ReferenceError: MessageItemView is not defined"
A simple browser refresh clears it up. Same goes for the user collection. It's collection view says it doesn't know anything about its item view. But a simple browser refresh and all is well.
My views (item and collection) are in separate files. Is that the problem? For instance, here is my message collection view in its own file:
messagelistview.js
var MessageListView = Marionette.CollectionView.extend({
itemView: MessageItemView,
el: $("#messages")
});
And the message item view is in a separate file:
messageview.js
var MessageItemView = Marionette.ItemView.extend({
tagName: "div",
template: Handlebars.compile(
'<div>{{fromUserName}}:</div>' +
'<div>{{message}}</div>' +
)
});
Then in my main module file, which references each of those files, the collection view is constructed and displayed:
main.js
//Define a model
MessageModel = Backbone.Model.extend();
//Make an instance of MessageItemView - code in separate file, messagelistview.js
MessageView = new MessageItemView();
//Define a message collection
var MessageCollection = Backbone.Collection.extend({
model: MessageModel
});
//Make an instance of MessageCollection
var collMessages = new MessageCollection();
//Make an instance of a MessageListView - code in separate file, messagelistview.js
var messageListView = new MessageListView({
collection: collMessages
});
App.messageListRegion.show(messageListView);
Do I just have things sequenced wrong? I'm thinking it's some kind of race condition only because over 3G to an iPad the item views are always undefined. They never seem to get constructed in time. PC on a hard wired connection does see success after a browser refresh or two. It's either the load times or the difference in browsers maybe? Chrome IE and Firefox on a PC all seem to exhibit the success on refresh behavior. Safari on iPad fails always.
PER COMMENT BELOW, HERE IS MY REQIRE BLOCK:
in file application.js
require.config({
paths: {
jquery: '../../jquery-1.10.1.min',
'jqueryui': '../../jquery-ui-1.10.3.min',
'jqueryuilayout': '../../jquery.layout.min-1.30.79',
underscore: '../../underscore',
backbone: '../../backbone',
marionette: '../../backbone.marionette',
handlebars: '../../handlebars',
"signalr": "../../jquery.signalR-1.1.3",
"signalr.hubs": "/xyvidpro/signalr/hubs?",
"debug": '../../debug',
"themeswitchertool": '../../themeswitchertool'
},
shim: {
'jqueryui': {
deps: ['jquery']
},
'jqueryuilayout': {
deps: ['jquery', 'jqueryui']
},
underscore: {
exports: '_'
},
backbone: {
deps: ["underscore", "jquery"],
exports: "Backbone"
},
marionette: {
deps: ["backbone"],
exports: "Marionette"
},
"signalr": {
deps: ["jquery"],
exports: "SignalR"
},
"signalr.hubs": {
deps: ["signalr"],
exports: "SignalRHubs"
},
"debug": {
deps: ["jquery"]
},
"themeswitchertool": {
deps: ["jquery"]
}
}
});
require(["marionette", "jqueryui", "jqueryuilayout", "handlebars", "signalr.hubs", "debug", "themeswitchertool"], function (Marionette) {
window.App = new Marionette.Application();
//...more code
})
Finally, inside the module that uses creates the collection views in question, the list of external file dependencies is as follows:
var dependencies = [
"modules/chat/views/userview",
"modules/chat/views/userlistview",
"modules/chat/views/messageview",
"modules/chat/views/messagelistview"
];
Clearly the itemViews are listed before collectionViews. This seems correct to me. Not sure what accounts for the collectionViews needing itemViews before they are defined. And why is all ok after a browser refresh?
The sequence in which you load files is most likely wrong: you need to load the item view before the collection view.
Try putting all of your code in the same file in the proper order, and see if it works.
The free preview to my book on Marionette can also guide you to displaying a collection view.
Edit based on calirification:
The dependencies listed for the module are NOT loaded linearly. That is precisely what RequireJS was designed to avoid. Instead the way to get the files loaded properly (i.e. in the correct order), is by defining a "chain" of dependencies that RequireJS will compute and load.
What you need to do is define (e.g.) your userlistview to depend on userview. In this way, they will get loaded in the proper order by RequireJS. You can see an example of a RequireJS app here (from by book on RequireJS and Marionette). Take a look at how each module definition decalre which modules it depends on (and that RequireJS therefore needs to load before). Once again, listing the modules sequentially within a dependecy array does NOT make them get loaded in that sequence, you really need to use the dependency chain mechanism.

Resources