I have a couple of large .xlsb files in 2FA-protected SharePoint. They refresh periodically, and I'd like to automate the process of pulling them across to a local directory. I can do this in PowerBI already by polling the folder list, filtering to the folder/files that I want, importing them and using an R script to write that to an .rds (it doesn't need to be .rds - any compressed format would do). Here's the code:
let
#"~ Query ~"="",
//Address for the SP folder
SPAddress="https://....sharepoint.com/sites/...",
//Poll the content
Source15 = SharePoint.Files(SPAddress, [ApiVersion=15]),
//... some code to filter the content list down to the 2 .xlsb files I'm interested in - they're listed as nested 'binary' items under column 'Content' within table 'xlsbList'
//R export within an arbitrary 'add column' instruction
ExportRDS = Table.AddColumn(xlsbList, "Export", each R.Execute(
"saveRDS(dataset, file = ""C:/Users/current.user/Desktop/XLSBs/" & [Label] & ".rds"")",[dataset=Excel.Workbook([Content])[Data]{0}]))
However, the files are so large that my login times out before the refresh can complete. I've tried using R's file.copy command instead of saveRDS, to pick up the files as binaries (so PowerBI never has to import them):
R.Execute("file.copy(dataset, ""C:/Users/current.user/Desktop/XLSBs/""),[dataset=[Content]])
with dataset=[Content] instead of dataset=Excel.Workbook([Content])[Data]{0} (which gives me a different error, but in any event would result in the same runtime issues as before) but it tells me The Parameter 'dataset' isn't a Table. Is there a way to reference what PowerBI sees as binary objects, from within nested R (or Python) code so that I can copy them to a local directory without PowerBI importing them as data?
Unfortunately I don't have permissions to set the SharePoint site up for direct access from R/Python, or I'd leave PowerBI out entirely.
Thanks in advance for your help
Related
I am currently trying to read pdf files stored in my google cloud storage. So far I have figured out how to read one file at a time from my google cloud storage, but I want to be able to loop through multiple files in my google cloud storage without manually reading them one by one. How can I do this? I have attached my code below.
To iterate all files in your bucket move your code for downloading and parsing in the for loop. Also I changed the for loop to for blob in blob_list[1:]: since GCS always prints the top folder in the first element and you do not want to parse that since it will result to and error. My folder structure used for testing is "gs://my-bucket/output/file.json....file_n.json".
Output when looping through the folder (for blob in blob_list:):
Output files:
output/
output/copy_1_output-1-to-1.json
output/copy_2_output-1-to-1.json
output/output-1-to-1.json
Output when skipping the first element (for blob in blob_list[1:]:):
Output files:
output/copy_1_output-1-to-1.json
output/copy_2_output-1-to-1.json
output/output-1-to-1.json
Loop through files. Skip the first element:
blob_list = list(bucket.list_blobs(prefix=prefix))
print('Output files:')
for blob in blob_list[1:]:
json_string = blob.download_as_string()
response = json.loads(json_string)
first_page_response = response['responses'][0]
annotation = first_page_response['fullTextAnnotation']
print('Full text:\n')
print(annotation['text'])
print('END OF FILE')
print('##########################')
NOTE: If you have a different folder structure versus the test made, just adjust the index in the for loop.
Whenever I make a custom item, I am unable to drag it onto a hotbar. When I try to pick it up the icon turns to a question mark and will not stick to a hotkey.
For example, I made an exact copy of the Murloc Costume (id 33079) at id 50017 (which was a free slot on my DB). The original I can put on a hotbar. The custom one I cannot.
Here's a gif of the issue
Answer: if it's not in the client side DBC's, it will not work properly.
From ReynoldsCahoon on the AC Discord:
I just modified a DBC recently. I had to use a tool (I used Ladik's MPQ Editor) to extract the specific DBC I wanted to modify, and then I used WDBX to open the DBC and manipulate it. In WDBX you can output it in a variety of formats (like CSV or SQL) so you can modify the values any way you like, and then reimport (via CSV or SQL) the values back in.
I loosely followed the guide here: https://model-changing.net/tutorials/article/23-41-creating-your-first-mpq-patch/
I exported a DBC containing all of the Character Titles in the game. Erased all of them, and then imported a bunch of new values of my own choosing. Instead of importing that back into the MPQ I got it from, I created a new MPQ called patch-4.mpq that I placed in my client WotLK/Data/ directory, and then on the server, I placed the DBC file into the worldserver/data/dbc/ directory, replacing the original DBC.
I think this can optionally be overwritten in the database, by using the associated _dbc table to override values from the dbc files (someone correct me if this part is wrong).
How can I enable tab completion in Julia with a function I wrote that loads a file outside of the working directory? Tab completion is super nice, but it only seems to work for files in my working directory. e.g. I store all of my files in a directory foo and I wrote a function to conveniently read these files.
const WHEREMUHFILES = "foo/"
function read_foo_file(filename::String)
readlines(WHEREMUHFILES * filename)
end
However, when I try read_foo_file("ba and hit Tab, tab completion doesn't work, in that it doesn't search the directory WHEREMUHFILES for bar.txt. Is there a way to enable this?
In reality, I have many different types of files organized in different directories, and the read_foo_file populates complex data structures after reading in the files, so a simple work around such as "put your files in your working directory!" is not what I'm looking for.
Im new using Sql Developer data Modeler, I made a Model using this tool but I dont Know how to re-open cause SqlDeveloer has created XML files, The app has created a folder that contains two folders: map and rel, map contains two files:
I dont Know what file i have to open from sqldeveloper, to reload the designed model that is a relational model, i dont Know how to reload all table.
I opened every xml file but the model is not loading the tables, what i have to do?
Solution,
I was created New relational model Design,
Looked up for main project File, I copied the main file from the damaged project and change next lines:
<relationalModel class="oracle.dbtools.crest.model.design.relational.RelationalDesign" name="PreciosModel" id="CF6F32EB-1ADF-A4D4-42E5-1871F2A3D75A" mainViewID="1A7856BA-66B4-D0E8-BF00-0204C3F41BBB">
<createdBy>Raul Martinez</createdBy>
<createdTime>2016-02-15 23:01:24 UTC</createdTime>
<ownerDesignName>Diseño2</ownerDesignName>
<shouldBeOpen>false</shouldBeOpen>
<selectedRDBMSSite>32076570-2523-435C-2E92-BF29817DFF70</selectedRDBMSSite>
</relationalModel>
LastOwnerName:preciosModel
NewOwnerNameDiseño2
Again I solved it by my self.
How can I upload multiple files to a folder in xquery ?
Are you trying to iterate through a directory on the filesystem and insert those files at some location in the db, like this?
for $e in xdmp:filesystem-directory('c:\my-files\')/dir:entry
let $file := xdmp:filesystem-file($e/dir:pathname)
let $dest-uri := concat('/dest-path/',$e/dir:filename)
where ($e/dir:type eq 'file')
return xdmp:document-insert($dest-uri,$file)
You should take a look at Information Studio. It provides a UI for loading and transforming content. One of the built-in “collectors” allows you to point at a file system directory and recursively load its contents. Information Studio automatically handles the directory traversal and breaking the load up into multiple transactions, among many other conveniences.