I have a DAG that downloads a file from Cloud Storage and saves it to the following path: /home/airflow/gcs/data/FILENAME.txt
This file then appears in the Cloud Composer storage bucket under /data.
However, when I originally wrote the DAG I didn't specify the download location to be: /home/airflow/gcs/data/ and simply had it downloading the file in place. I would like to go delete those files but I don't know where to find them.
Where do downloaded files in Cloud Composer reside when you don't specify the folder?
It looks like you don't need to worry about cleanup from when you first wrote the DAG - if you're using the gcs_download_operator, then according to its source code, if you did not specify a value for the filename parameter, the downloaded file won't be stored on the local file system.
Related
I did't find here an answer for that, so thought anyone can help:
I'm receiving a csv file from a Get request.
I want to upload it to S3 (and then continue the pipeline..)
I'm using Airflow on the managed AMAA platform.
Since when uploading to S3, the script required a file path for the csv file.
how can I pass a file path when it's on the AMAA platform? is it even stored anywhere?
do I need a middle man to store it in between?
I would like to sync all of the files in my Google Cloud Storage bucket with the exported files in my Firebase Storage Emulator.
I downloaded all of my cloud files using gsutil to my local machine.
I used BeyondCompare to move all of the new files to the '../storage_export/blobs/ directory.
How do I update/create the JSON metadata in '../storage_export/metadata' to reflect these new files and make them available when I run the emulator and import them in?
Edit:
The gsutil docs mention the following:
when you download data from the cloud, it ends up in a file with no associated metadata, unless you have some way to keep or re-create that metadata.
How would one "keep" or "re-create" that metadata during a gsutil cp download?
You can use gsutil or the SDK to get each object's metadata and then write it down to a JSON file however, there's currently no native way to import Google Cloud Storage data in the Storage Emulator. But as I stated in my answer to this post, you can study how the emulator register the object by uploading sample files within the emulator and then running the export, you will see that the emulator will require 1 object and 1 JSON file that contains it's metadata.
Lastly, you can add the option --export-on-exit when starting the emulator, Downloaded all data from the real Firebase project, uploaded everything with the Emulator, then kill the emulator; as stated in this post.
Note: This is not a documented feature! Firebase doesn't expose the concept of download tokens in its public SDKs or APIs, so manipulating tokens this way feels a bit "hacky". For your further reference, check this post.
I'm running containerized Alfresco in Docker (pom shows alfresco-core.version 7.21). According to the official documentation, the files should be stored as a .bin file in \alf_data\contentstore, but when I go into the alfresco container, alf_data is an empty directory. Even when I search the whole container for .bin files I find nothing related to my files.
Can anyone tell me how I can find my files?
Thanks!
Look in your Docker Compose file and see if an external volume as been defined. It is likely, as any content stored directly in the container would be ephemeral. Using a volume allows content to be written to the host file system.
Just in case you were tempted, though, you shouldn't be doing anything with those files directly. The Alfresco content store uses a hashed directory structure and renames all files using a GUID and an extension of "bin".
You should check your repository.properties/alfresco-global.properties files and look for the configured location. Note, only files - as in Word, PDF, etc - will be stored on disk and metadata goes into the database.
https://hub.alfresco.com/t5/alfresco-content-services-forum/changing-the-location-of-contentstore-dir/td-p/215540
I have a meteor project that runs a python script once a day to query an API and save the results to a JSON file in private. The meteor server watches that JSON file for changes. When the file changes, the server reads and parses the file and updates a collection accordingly.
The problem is that the assets in private are loaded at meteor startup and copied into a different asset folder as read-only, and thus the script can't make changes to the file.
I could maybe change the permissions on that asset destination folder but that seems hacky. I don't think assets in private are intended to be dynamic, anyways. Anybody know how I can accomplish this flow?
Meteor server kicks off python script once per day
Script queries API, saves results to JSON file on server
Meteor server reads JSON file and updates collection
The simplest solution may be for the Python script to write its JSON to a Mongo database. Then Meteor can automatically subscribe to any changes made to that collection.
Here is what I ended up doing:
Instead of having the meteor app kick off the python script daily, I just put the python script on the host's file system (outside of the meteor app bundle) and created a cron job on the host to run it daily. The JSON output file was saved to the host's file system.
Then, in my meteor app, I created a file watch on the output JSON that triggered a file read.
I was running into some issues with Meteor Up (mup), so see my other answered question here.
I'm creating an app where users need to work with large databases. Rather than having the users download the data and populate an SQLite database on the client (which would take a long time), I'm hoping I can use downloadable, pre-populated databases.
I found cordova-sqlite-ext, which allows working with pre-populated databases, but SQLite files must be located in the www folder for this to work. Is it actually possible to download files to this folder in Ionic/Cordova (on non-rooted devices)?
It's always good practise to store your files in app's directory. Check this comment of mine and see if it helps you:
https://github.com/driftyco/ionic-native/issues/881#issuecomment-270832521
I had a requirement of downloading a zip file(with sqlite file in it), unzip the file, store the file in app's directory and query the DB. I was able to achieve it using plugins and works quite well.