I thinking of putting a logical layer on top of all raw files that comes in to data lake store.
I would like to have a view that combines all files that are the same "type" but are divided into date folders. I was thinking of doing this with a view and a dynamic folder path.
The problem that I have is that the files are avro and json file and for this I need assemblies. Is there a way that I can refrence the the asseblies in the views?
Or is it possible to do in an other way? suc as using table value functions etc?
The query expression inside the USQL View at the moment do not allow any User Defined Objects and you cannot REFERENCE any assemblies within a USQL View definition.
You may be better with parametrized view (i.e. Table value functions) - you don't have to necessarily have a parameter. TVF provides great flexibility, for example if you want get just a Month or year worth of data - you could use USQL filesets and pass in parameters.
Related
I have a requirement to consume a csv "dataset" consisting of 3 flat files - a control file, a headers file, and a line file - which together define a nested data structure.
The control file items have a field called ControlID, which can be used in the headers file to identify those header records which "belong" to that control item.
The header records have a field called HeaderID, which can be used in the lines file to identify those line records which "belong" to a given header record.
I'd like to consume all three files and then map them into some kind of nested schema structure. My question is how would I do this? Can I do it in a pipeline component?
I would look at two options. Both involve correlation all three files to an Orchestration using a Parallel Convoy.
Use a Multi-input Map to join the files. You should be able to use the HeaderID as filter using the Equal Function to match the lines to their header.
Use a SQL Stored Procedure to group the data as described here: BizTalk: Sorting and Grouping Flat File Data In SQL Instead of XSL
I have a table in another database I would like to scaffold via ASP.NET Dynamic Data and incorporate into my existing Entity Model - is there anyway to do this? (eg using a view or other mechanism or customize the view, edit or insert operations via ad-hoc SQL or stored procedures?)
I don't want to replicate the entire DynamicData sub-folder structure and create another entity model for just one table
I was able to solve this by manually creating an entity in the SSDL and CSDL sections of the .edmx file by using a DefiningQuery and then defining the EntitySets for my entity class
I also added insert / update / delete Function elements to the SSDL with inline SQL using the CommandText property
At this point I had enough to let the Designer map the CRUD methods to these inline SQL functions I defined
It's a little tricky but it works and the general approach opens up many possibilities I had not thought about
I have the following scenario:
My website db has a system table called "Companies", which includes an id field, companyName field, and companyImageUrl field.
How do I set up an umbraco document type for adding entries to this table ?
Maybe I shouldn't use a custom table at all ?
Thanks.
As far as I know, Umbraco doesn't support what you want to do out of the box (mapping a document type to a table that isn't part of the umbraco core).
One approach that might work is to create an action handler that syncs a Company doc type to your table when creating a node of that type.
It's a bit of a hack though. I've found that I've very rarely needed to create custom tables. What exactly are you trying to do with it? My guess is that you don't really need it and would be better off working with a doc type instead. Umbraco provides a variety of ways to get and act upon doc types from within custom C# code (check out the umbraco.NodeFactory namespace). You'll also get the added benefit of being able to easily interact with these nodes from XSLT/Razor.
I have a stored procedure that search a view using full text.
I'm trying to map the results to an existing Entity (Sales), where I mapped the column MivType to SaleType (as it makes more sense, and I want to keep db names away from my web site). The stored procedure is mapped to a Function Import, and I've defined its ReturnType to Sales.
This work well as long as the entity has the same property names as fields names.
Here's my problem: when I change the property's name, I get the following error after running the imported function:
The data reader is incompatible with the specified 'Model.Sale'. A member of the type, 'SaleType', does not have a corresponding column in the data reader with the same name.
I can fix this if I change the property 'SaleType' to 'MivType' on the entity, but why should I do that? Isn't that what the mapping is for?
This means I have to use the exact same names on the stored procedure and the entity, so in effect, the mapping is ignored (I have names like YzrName, MivYaad, etc, and I don't like it).
Is there a simple way around this? I don't want to use the db names on my application, and prefer not to change the stored procedure...
(I should mention I'm a beginner with the EF, so this can be a rookie mistake)
Thanks.
Well the entity designer doesnt work very well. I generally try to do everything in the XML. In the XML there are 3 parts. The Storage (a representation of the SQL Database). The Conceptual (a represention of your .Net Objects. and the Conceptual to Storage Mapping
It sounds like the error is in your Conceptual to Storage Mapping. You can keep the property name SalesType on the conceptual side but the mapping must map the the correct names on both the conceptual and storage side.
Refer to MSDN here are some articles
http://msdn.microsoft.com/en-us/library/cc716731.aspx
Often I need to combine data from multiple tables and display the result in a GridView control.
I can write a Linq query inline in the Page_load event, return an anonymous type that combines all the fields that I need, and databind the result to the GridView control.
Problem: I use 'helper methods' as described by Scott Guthrie on his blog. Such a helper method cannot return an anonymous type. The query would have to be inline for this approach.
I can write a database view that returns the data that I need, and write a helper method with a query against this (new and known) type that it returns.
Problem: I will need a lot of views in my database schema, and I will introduce a lot of redundant aspects of my data. I also lose some of the advantage of using Linq - removing all business logic from the database.
I would like to take an approach that lets me keep the Linq queries in helper methods, yet allows me to access all the attributes that I need on the grid in their respective databinding expressions. Can this be done?
I asked the wrong question, as I frequently do. What prompted me to look into anonymous types was an apparent limitation of the GridView - my inability to use a databinding expression in an <asp:BoundField> (the DataField parameter only accepts column names of the table that the Linq query pulls in).
Turns out that in a TemplateField it is possible to use Eval and access members of the Linq data item, and Linq takes care of the query for me.
In other words, I can keep the query in my helper method, have it return a primary database table type (e.g. Account), and I bind the Accounts to the GridView.
In the databinding expressions I can access data members of the Account objects that reside in other tables, without having to explicitly pull them in in the query. Perfect.
I don't know if there is a viable way to achieve this using anonymous types. But I have a suggestion that will work in WinForms, but I am not sure about ASP.NET.
What you need is a type with properties where neither the number of properties, nor the types and names of the properties are known at compile time. One way to create such a thing is ICustomTypeDescriptor.
You have to create a type implementing this interface with an private backing store of objects backing the properties returned by the query for one row from the query. Then you implement GetProperties() to return one PropertyDescriptor per column and PropertyDescriptor.GetValue() and PropertyDescriptor.SetValue() to access the backing array.
By implementing PropertyDescriptor.Name you will get the correct column name; this will probably require another backing store storing the property names. And there is a lot more to implement, but in the end your new type will behave almost like a normal type - and now the if - if the control you are binding to knows about and uses ICustomTypeDescriptor.
UPDATE
I just found an bit of text stating that ASP.NET data binding knows and uses ICustomTypeDescriptor.
Scott's earlier post in the series talks about shaping the result set before inserting into a grid:
Part 3 - Querying our Database
Scroll down to "Shaping our Query Results".