Can we call azure data lake analytics database's table value functions from c# code behind in U-SQL script? Any operation that can be done on Azure data lake analytics database would be fine.
Thanks
Related
I have a requirement wherein I need to fetch historical data from a third party system which is exposed as a web service and train the model on that data.
I am able to achieve the above requirement by using "Execute Python Script" node and invoking the web service using python.
The main problem arises when I need to fetch data from the third party system every time the Azure ML web service is invoked, since the data in the third party system keeps on changing hence my Azure ML model should be trained for new data always.
I have gone through the link (https://learn.microsoft.com/en-us/azure/machine-learning/machine-learning-retrain-a-classic-web-service) but I am not sure how we can do this for my requirement as for me the new historical data set should be obtained every time the Azure ML web service is invoked.
Please suggest.
Thanks.
I recommend that you:
look into the new Azure Machine Learning Service. Azure ML Studio (classic) is quite limited in what you can do, and
consider creating a historical training set stored in Azure blob storage for the purposes of training, so that you only need to fetch from the 3rd party system when you have a trained model and would like to score the new records. To do so, check out this high-level guidance on how to use Azure Data Factory to create datasets for Azure Machine Learning
tl;dr: I want to reference an external data source from a Kusto query in Application Insights.
My application is writing logs to Application Insights, and we're querying it using Kusto in the Azure portal. To give an example of what I'm trying to do:
We're currently looking at these logs to find an action that triggers when a visitor viewed a blog post on our site. This is working well on a per blog-post level, but now we want to group this data by the category these blog posts are in, or by the tags they have, but that's not information I have within the logs.
The information we log contains unique info about that blog post (unique url, our internal id, etc) that I could use to look up this information in another data source (e.g. our SQL DB where this relation is stored), but I have no idea if/how this is possible. So that's the question, is this possible? Can I query a SQL DB, or get data in JSON via a URL or something?
Alternative solutions would be to move the reporting elsewhere (e.g. PowerBI) and just use AI as a data source, or to actually log all the category/tag info, but I really don't want to go down that route.
Kusto supports accessing external data (blobs, Azure SQL, Cosmos DB), however
Application Insights / Azure Monitor and other multi-tenant services are blocking this functionality due to security and resource governance concerns.
You could try setting-up your own Azure Data Explorer (Kusto) cluster, where this functionality will be available, and then access your Application Insights data using cross-cluster query, or by exporting the data from Application Insights and hooking up EventGrid ingestion into your Kusto cluster.
Relevant links:
Kusto supporting external data:
https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/schema-entities/externaltables
Querying data inside Application Insights:
https://learn.microsoft.com/en-us/azure/data-explorer/query-monitor-data
Continuous export data from Application Insights:
https://learn.microsoft.com/en-us/azure/azure-monitor/app/export-telemetry
Data ingestion into Kusto from EventGrid:
https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-event-grid
I have 2 documents in a partition of document db. How can I combine these 2 documents and prepare one document which I can send as a Data Source for Azure Search Service.
I found a way by stored procedure we can combine 2 documents. Now I am facing a difficult scenario. Can we use a stored procedure in the query part of Azure search Data Source?
I am looking for a solution to read all the documents by partition
key.. is it possible by stored procedure.
Of course,it is possible to call the stored procedure by partition key. In fact, the partition key is necessary. You could refer to my previous case for more details:Azure Cosmos DB asking for partition key for store procedure.
Based on my researching, stored procedure can't be invoked in azure search data source.You could only configure sql database,cosmos db, blob storage ,table storage etc. So, I suggest you using Azure Function Http Trigger to call your stored procedure with sdk then save the combined document into new collection or save it as json file into blob storage corresponding to your Azure Search index.
I have linked Bigquery to my app through Firebase and it has worked well for more than 3 months. However, couple days ago, a random dataset called "analytics_156951304" was created and then the app data was streamed into this dataset instead of the original dataset that Bigquery created for me when I first linked my app to Bigquery, "my_app_name_IOS". And the table schema was changed too. I checked Stackdriver logging and it said an account called "firebase-measurement#system.gserviceaccount.com" created this job at midnight of my local time. However, I have no clue what happened and how to get my streaming data back to my original dataset, "my_app_name_IOS". Thank you for all the answers!!!
Firebase recently scheduled a huge schema migration - your previous queries won't work anymore.
(AFAIK this was communicated via email)
There's a script that helps migrating your previous datasets to the new schema:
https://support.google.com/analytics/answer/7029846?visit_id=1-636661955728880760-637850888&rd=1#migrationscript
(tho the script won't help with modifying the existing queries)
From the email I got:
Currently, the data from each app in a Firebase project are exported to a distinct BigQuery dataset in the corresponding BigQuery project. Going forward, the data from each app in a Firebase project will be exported to a single dataset in that BigQuery project. Additionally, the data in this new dataset will follow a new schema.
I have tried finding the answer to this but have only found anecdotal references. Does Google encrypt the data that is stored on BigQuery? If so, what encryption mechanisms are used?
All new data being added to BigQuery tables are encrypted using strong, industry-standard encryption methods. All old data will be encrypted over time, but currently with no specific timeline. If you'd like more detail on security across the Google Cloud Platform you might want to check out this blog post:
http://googlecloudplatform.blogspot.com/2014/08/googles-cloud-is-secure-but-you-dont.html
BigQuery is a part of Google's Cloud Platform Offering. As part of utilizing BigQuery you need to first load the data into it.
You can load Data 2 ways.
Load jobs support two data sources:
Objects in Google Cloud Storage
Data sent with the job or streaming insert
Data sourced from Google Cloud Storage is always encrypted as per the link below. Streaming data into BigQuery probably depends on how you stream it (I'm haven't found any absolute data on this point)
Loading data into BigQuery
Description of Compute Engine Disks
Yes, data stored in BigQuery is encrypted.
Google Cloud Platform encrypts data stored in BigQuery, without any action required from the user, using one or more encryption mechanisms. Data stored in Google Cloud Platform is encrypted at the storage level using either AES256 or AES128.
There are more details in this whitepaper:
https://cloud.google.com/security/encryption-at-rest/