I have a Power Bi Dashboard I've made that pulls its Data from a Redcap Database using an API. It looks like this, with mostly Text in the various columns:
What I'd love to do is make it so that thee fields circled in red were real files that could be clicked and downloaded. I know that the API allows me to pull files from it. I've used R with code like this (that individually mentions what record and field I want):
library(REDCapR)
redcap_download_file_oneshot( redcap_uri="https://redcap.company.org/redcap/api/", token="################", record="1", field='full_protocol_attachment_t_v2', event = "", repeat_instrument = NULL, repeat_instance = NULL, verbose = TRUE, config_options = NULL, overwrite = TRUE )
To individually download files one at a time. The problem is twofold:
If I were to use R, I have no idea how to automate that snippet of code for every row I may pull from the database (and if there are new rows)
My understanding of PowerBi is that if I do use R, it makes refreshing the data harder when the report is published online. Right now given all the data just comes from an api directly into PowerBi, I don't have to setup any fancy permissions or gateways to have automated refreshes.
So my question is: is there a way to do this directly within PowerBi? Like a calculated column or something that would pull a particular records file based on what row it was in?
The only thing you can do in native PBI is have a URL which when clicked will open the destination for you. Can you create a full url for the file download?
Related
I'm using BigQuery integrated with Firebase and all the datasets are in the same Project. My analytics dataset is in useast-4 but for some reason my firebase_imported_segments dataset region is just marked as US
I'd like to move data from the analytics dataset into a table in the firebase_imported_segments.
At first, I tried a simple INSERT query but I get the error firebase_imported_segments was not found in location us-east4
So then I tried building a SELECT statement and exporting the rows using "Save Results > Big Query Table" but that gives a similar error that the destination dataset is not found. Oddly enough, if I create a table in firebase_imported_segments and try to save the results using that table name, I get a "Table already exists" error. So it's not that it can't find the firebase_imported_segments dataset, it just won't create a new table in that dataset.
How can I get around this? I saw some BQ documentation that moving data between regions is possible but I didn't a simple walkthrough of how it's accomplished. I'm also confused by why firebase would put some data in one specific region (useast-4) and then other data in a multi-region (US) if they aren't compatible.
You can move datasets using "Copy" in the BigQuery UI then delete the old dataset. See Copy dataset documentation.
Option 1: Use the Copy button.
Go to the BigQuery page in the Cloud console.
In the Explorer panel, expand your project and select a dataset.
Expand the More Actions option (triple dot button) and click Open.
Click Copy. In the Copy dataset dialog that appears, do the following:
a. In the Dataset field, either create a new dataset or select an
existing dataset ID from the list.
Dataset names within a project must be unique. The project and dataset
can be in different regions, but not all regions are supported for
cross-region dataset copying.
b. In the Location field, the location of the source dataset is
displayed.
c. Optional: To overwrite both data and schema of the destination tables
with the source tables, select the Overwrite destination tables
checkbox.
d. To copy the dataset, click Copy.
To avoid additional storage costs, consider deleting the old dataset.
Option 2: Use the BigQuery Data Transfer Service.
Enable the BigQuery Data Transfer Service.
Create a transfer for your
data source.
I tested this and can confirm that it works. I created a dataset in us-east4 named analytics_us_regional and has a table named east_4_table and copied it to a dataset located in US.
Copy us-east4 to US dataset:
When copy is initiated a data transfer job is created:
Copied to US:
With regards to the data in firebase located in us-east4 based from the firebase export to BQ. When the export is enabled the first time, the user will define the location of the tables. It might be possible thatus-east4 region was selected initially.
Don't know if it will work in your case, but I had a dataset in europe-west1 and I want to copy it to EU region, I have done these two ways and it both worked:
First way:
1- Click on the dataset you want to copy and click on "COPY".
2- On the copy menu on the dataset destination click on "CREATE NEW DATA SET" and select the destination region you want that dataset to be. Click on CREATE DATA SET.
3 - On the "Copy data set" menu click on COPY.
4 - You will get an error "Cannot create a transfer in REGION_EUROPE_WEST_1 when destination dataset is located in JURISDICTION_EU" but a dataset with no tables will be created on your destination Region.
5 - Now if you try to copy the source dataset by clicking on COPY and selecting the dataset created in set 4, it will work now.
Second way: (best way)
1 - Open a New Query sheet Click on MORE- >Query settings-> Advanced options, uncheck the "Automatic location selection" and select the destination region or Multi-region you want (in my case EU).
2- On this query sheet run "CREATE SCHEMA your_new_dataset_name" -> this will create the dataset "your_new_dataset_name" in the destination region selected in point 1.
3 - Click on the dataset you want to copy and click on "COPY".
4 - On the copy menu on the Data Set destination select the dataset created in point 2, and click on COPY.
Both ways under the wood utilize the BigQuery Data Transfer Service but you don't need to access the service directly.
In fact, both ways do exactly the same thing which is creating a destination empty dataset in the correct region you want to copy yours, once you have that the Copy function will work correctly.
Is there a way to extract a table from a web page in Automation Anywhere after taking certain steps using web recorder. The table does not appear directly, it appears after clicking few controls after launching the URL.
The table that I want to extract is coming after loggin in to the website and filtering using a control for search criteria.
I used web recorder to login and putting the desired search criteria in a text field and I want to extract the table now. When I use web recorder, it launches the URL again and takes me back to the login page which I dont want. I want the bot to stay on the page. Pls help.
Also, what is the significance of session name of an extracted table?
If you clicked on Advanced View, you will find at Step 5 : to run this command using an existing IE window. Try to write the URL of the page with the table and not the one of the login page.
The extracted table is to be used using variable $Table Column(Index)$ with index being the column number or column name
you can export directly using object cloning and in the selection criteria export to csv file. But we need to click on html inner text also in search criteria
An old question, but my experience has been the Extract Data/Table commands are rather poor. Not only do they only work in IE, you cannot call them as commands, they have to be called via a web recording.
Instead, I've found it much more useful to object clone the initial element, grab the DOMXPath, and variablize that. Then throw it into a loop while command and set the condition on finding at least one element (of the elements for the table you are trying to build). You can grab all sorts of useful info in the object clone command and then right that to a variable/table.
For example
//div[#id='updatable-standings']/div[1]/div[1]/div[2]/div[1]/table[1]/tbody[1]/tr[3]/td[2]/div[1]/span[2]
//div[#id='updatable-standings']/div[1]/div[1]/div[2]/div[1]/table[1]/tbody[1]/tr[4]/td[2]/div[1]/span[2]
I can create a incremental variable for {tr[3]} and call it $vTeamLoop$ and change my DOMXPath value in the Object Clone to be
//div[#id='updatable-standings']/div[1]/div[1]/div[2]/div[1]/table[1]/tbody[1]/tr[$vTeamLoop$]/td[2]/div[1]/span[2]
Ultimately, it is more steps than the Data/Table Extract command, but it is far less limited in scope.
Hope that helps.
enter code here
I have managed to create a data model, and in doing so import data variables and values from my googlespreadsheets (by clicking on 'import data from sheet'). My table is of the form:
table = {"age": {15,22}, "name": {"ted", "sally"}, "surname":{"anderson","medina"}}
I would like to have a table that uploads this on a page, and then I can click on each row which would then open up another page and I could edit the contents
I open a new page and drag a table in and link it to my data model. However, it only shows the variable titles and not the actual data. How can I get the data to automatically upload into the table?
I read alot of the tutorials but they all assume I want to start with only headers, and then manually input the data to create a data entry table. However, my table updates automatically in googlesheets, so I would like to import it to appmaker, and then have the ability to click on each row and add notes/edits. Any help would be greatly appreciated!! thanks
At this time App Maker doesn't support spreadsheets as data backend. You'll need first import all data to App Maker's models and then play with it in deployed application. You can find all pieces of the puzzle in Vendor Ratings template:
https://developers.google.com/appmaker/templates/vendor-rating/
Your actual data won't come up in your editor view, only when you preview it.
It seems to me that you could make this using an embedded Google sheet in your page.
Ex:
-drag an html block on the page
-set the content to something like:
<iframe src="[link to your google sheet]" height="1380" width="1100"></iframe>
*get the embed link from 'Publish to the Web'
Hope that helps, it might not be what you're looking for.
I have a requirement to load the csv into DB using oracle apex or pl/sql code, but the problem is they are asking to load the csv file which will not come with same number of columns and column names .
I should create table & upload data dynamically based on the file name and data that i'm uploading.
For every file i need to create a new table dynamically and insert data that are present in csv file.
For Example:
File1:
col1 col2 col3 col4 (NOTE: If i upload File 1, Table should be created dynamically based on the file name and table should contain same column name and data same as column headers of csv file . )
file 2:
col1 col2 col3 col4 col 5
file 3:
col4 col2 col1 col3
Depending on the columns and file name i need to create table for every file upload.
Can we load like this or not?
If yes, Please help me on this.
Regards,
Sachin.
((Where's the PL/SQL code in this solution!!??! Bear with me... the
answer is buried in here somewhere... I introduced some considerations
and assumptions you will need to think about before going into the
task. In the end, you'll find that Oracle APEX actually has a
built-in solution that satisfies exactly what you've specified... with
some caveats.))
If you are working within the Oracle APEX platform, you will have some advantages. APEX Version 4.2 and higher has a new page element called "Data Loading". The disadvantage however is that the definition of the upload target is fixed and not dynamic. You will need to know how your table is structured prior to loading the data.
One approach to overcome this is to build a generic, two-column table as your target, which will serve for all uploads. Column 1 will be your file-name and column two will be a single clob data type, which will contain the entire data file's contents including the header row. The "Data Loading" element will give the user the opportunity to verify and select this mapping convention in a couple of clicks.
At this point, it's mostly PL/SQL backend work doing the heavy lifting to parse and transform the data uploaded. As far as the dynamic table creation, I have noticed that the Oracle package, DBMS_SQL allows the execution of DDL SQL commands, which could be the route to making custom tables.
Alex Poole's comment is important as well, you will need to make some blanket assumption about the data type or have a provision to give more clues about what kind of data is contained. Assuming you can rely on a sample of existing data values is not good... what if all the values in your upload are null? I recommend perhaps a second column in the data input with a clue about the type of data for each column... just like the intended header names, maybe: AAAAA = for a five character column, # = for a numeric, MM/DD/YYYY = for a date with a specific masking.
The easier route:
You will need to allow your end-user access to a developer-role account on a workspace of your APEX server. It is not as scary as you think. With careful instruction and some simple precautions, I have been able to make this work with even the most non-technical of users. The reason for this is that there is a more powerful upload tool found under the following menu item:
SQL Workshop --> Utilities --> Data Workshop
There is a choice under "Data Load" --> "Spreadsheet Data"
The data load tool will automatically do the following:
Accept a CSV formatted file through a browse function on your client machine
Upload the file and parse the first record for the column layout (names)
Allow the user to create a new table from the uploaded file, or to map to an existing one.
For new tables, each column data type can be declared and also a specific numeric/date mask if additional conversion from the uploaded data is necessary.
Delimiter type, optional enclosures (like double quotes), decimal conventions and currency types can also be declared prior to parsing the uploaded file.
Once the user has identified all these mappings and settings, the table is created with the uploaded data. Any errors in record upload are reported immediately afterwards with detailed feedback on the failed records.
A security consideration to note:
You probably do not want to give end users access to your APEX server's backend... but you CAN create a new workspace... just for your end users... create a new database schema for receiving their uploads, maybe with some careful resource controls. Developer is the minimum role needed... but even if the end users see the other stuff there won't be access to anything important from an isolated workspace.
I have implemented the isolated workspace approach on a 4.0/4.1 release APEX platform a few years back, and it worked nicely. Our end user had control over the staging and quality checking of her data inputs (from excel spreadsheet/csv exports collected from a combination of sources). I suppose it may have been even better to cut her out of the picture entirely and focused on automating the export-review-upload process between our database and her other sources. In this case, the volume of data involved was not great enough (100's to 1000's of records) and the need for manual review and edit of the exported data was very important prior to pushing it into the database... so the human element was still important in this case - it is something you'll want to think about now.
Running the Databus server from the command line I have successfully uploaded timeseries data via curl, and am able to query the same data with the api. I'm unable to view any of the data in the table in the UI. After selecting "My Databus" -> Tables is says "You do not belong to any groups that have tables yet. Add some groups, then tables!!!". Navigating to the Database and selecting the table -> chart no data comes back there either.
I have noticed that the query it issues is from a recent time range, while the data I loaded is for an earlier time period. Is there a default way to show the most recent data available in a table?
Is your table type relational or stream? If relational, what is the primary key?
If time series, this url will give you the last 10 values because of the parameter 10 and reverse=true.
http://[yourhost]/api/firstvaluesV1/10/rawdataV1/[yourtablename]?reverse=true
If relational table, you can retreive all values like so
http://[yourhost]/api/getdataV1/select+c+from+[yourtablename]+as+c
replace either urls [yourhost] and [yourtablename] values.
We do not use the tables page much. It is better to click in the specific database as in My Databus -> Databases and then click on the database that has your table. We are about to add a view data link in there showing most recent 1000 values or something like that. There is already a view chart which shows most recent 2 hours(again, we want to change that to most recent 1000 data points instead as well).