I am new to the data warehouse and currently working on this project.
Is there any way to insert new data transactional to existing cube? With tools or with MDX query maybe?
MDX is usually just a read-only language.
With an OLAP cube you have two options to change the data:
UPDATE/INSERT to the underlying SQL data mart yourself, and then rebuild the cube
Use something called WRITEBACK where you set numbers directly in the cube, and it decides how to save these back to the data mart (which is tricky if you set a number at the top level, and it has to decide how to split that value up between all the members down to the bottom level)
Usually there is an ETL (Extract, Transform, Load) tool like Pentaho (open source) or Informatica which populates a data warehouse. The data warehouse itself may use a proper database and a product like Mondrian is used to hold data in cubes.Jasper Server, for example has mondrian packaged with it. Data from transactional system is populated in the data warehouse, then the cube is 'refreshed'. There may be other possible approaches.
Related
I'm working with Progress 11.6 appBuilder and procedure editor (and Data Dictionary).
Regularly we are doing modifications at the customer's database, there are two types of modifications:
Modifications of the structure: those are done, using interactive GUI of the data dictionary.
Modifications of the data: those are done, using the procedure editor
An example of a data modification in the procedure typically looks like this:
FOR EACH Table1 WHERE Table1.Field1 = <value>:
CREATE Table2.
Table2.Field1 = <value>.
Table2.Field2 = <some-other-value>.
END.
This is completely in contradiction with one of the basics of software delivery quantity, repeatability: there is no way to return to the previous situation!
Therefore I'm looking for ways to do this in an (automatable) repeatable way, hence my questions:
What can we use instead of the interactive GUI of data dictionary (without undo feature) in order to perform/undo database structure modifications?
What can we do in order to undo database data modifications? (Is there something like a Oracle redo log or a Oracle archive log in Progress?)
In case you say "What are you talking about? You can do "Undo transaction" in the data dictionary.", I mean the following:
I perform a transaction using the data dictionary, I leave the data dictionary and the day later the customer complains. When I open the data dictionary at that moment, the "Undo transaction" feature is disabled.
At a high level you should be creating "df files" (DDL scripts) and applying those to the customer database rather than manually making changes. There are many ways to create those files and you can automate the entire process with the appropriate tooling.
One of the most common ways to create a df file is to create whatever new schema you need in your development database and then use the "create an incremental df" facility in the data dictionary tool. This tool compares the development database schema to the target schema and builds a "df file" (DDL script) of the differences. You could connect directly to the target db for this process or you could have an empty skeleton db that you use for this.
How to create an incremental df file
(If you then reverse the comparison you can also create a reversing df file to undo the changes.)
Most df files consist of additions - new tables, new fields, new indexes. These can all be added online and that can all be completely scripted. And, of course, the individual df files and all of the supporting scripts can (and should) be stored in a repository (like git or whatever).
As for the data change scripts... there's no reason that those programs cannot be written as actual programs and saved in a repository. You can enclose the whole update in a transaction and UNDO it if that is appropriate. For what it is worth, I personally do not think that is a very good idea. Especially when large amounts of data are involved you really don't want to be creating monstrous multi-gigabyte undo logs. You're better off with a second "reversing transaction" script that will roll things back piecemeal. A side benefit is that you can still use that if you decide to back out the change a day or three afterwards.
The really gory details are going to depend on your development process and the customers change management process and the tooling available. It kind of sounds like there is not much process or tooling at either end of this relationship so you probably have a lot of adventures ahead of you!
I was looking for the best way to capture historical data in HANA for master data tables without the VALID_TO and VALID_FROM fields.
From my understanding, we have 2 options here.
Create a custom history table and run a stored procedure that populates this history table from the original table. Here we compromise with the real-time reporting capability on top of this table.
Enable the History table flag in SLT for this table so that SLT creates this as a history table which solves this problem.
Option 2 looks like a clear winner to me but I would like your thoughts on this as well.
Let me know.
Thanks,
Shyam
You asked for thoughts...
I would not use history tables for modeling time dependent master data. That's not the way history tables work. Think of them as system versioned temporal tables using commit IDs for the validity range. There are several posts on this topic in the SAP community.
Most applications I know need application time validity ranges instead (or sometimes both). Therefore I would rather model the time dependency explicitly using valid from / valid to. This gives you the opportunity e.g. to model temporal joins in CalcViews or query the data using "standard" SQL. The different ETL tools like EIM SDI or BODS have also options for populating such time dependent tables using special transformations like "table comparison" or "history preserving". Just search the web for "slowly changing dimensions" for the concepts.
In the future maybe temporal tables as defined in SQL 2011 could be an option as well, but I do not know when those will be available in HANA.
I have created a connection to the database in a SQL developer. Now in there I can see lots of Table having different dependencies and constraints applied. now its very confusing and time consuming to see the details of each table manually. I want them in a way(GUI) so that I can easily Identify that particular table is master one and all the dependencies of all other tables. does it provide any kind of tool ? or is there any other Method ?
You can generate an ER diagram for the objects and their relations.
File -> Data Modeler -> Import -> Data Dictionary
Choose the database and schema which contains the objects.
Choose the objects
I am writing an ASP.NET application with a SQLServer database in which I have to calculate rates for members of my application. These calculations affect more than 10 database tables.
I believe I have two options :
In the data access layer, fetch the data for a member from the first table in the database and return it to the business layer. Once there, perform some calculation on the fetched data to generate new data to be saved in a second table. Finally, call back into the data access layer to save the result. Afterwards, repeat this whole process, pulling information from the second table to calculate results to be saved in a third table, and keep doing this for all necessary tables.
Use a stored procedure to encapsulate calculating and saving the new rates for a member in the correct tables within a database transaction.
Which option do you think is best and why? If I use the first option, how should I perform all of the inserts and updates in one ADO.NET transaction from the business logic layer? I am currently barred from using an ADO.NET transaction outside of the data access layer.
IMO it depends on how much priority needs to be given for performance and modular design.
If this was a trading application where I would have to calculate the prices instantaneously from different tables, I would use a stored procedure
Better performance for certain amount of load but when it gets too much queries, then a distributed database becomes essential
One disadvantage is that if you want to move to a different database in the future (in most cases people don't), then you have to port the stored proc to the new ones
The other option I would have is to keep most of the values in a distributed cache (like memcache) and do the calculations in a business layer.
The values will be pre-populated into the cache and the cache will be refreshed as and when there are changes.
This gives flexibility in terms of removing the database dependency.
But it seems to me that your operations are quite heavily DB dependent in functionality and suggest a stored procedure route. If you think the second option of cache is possible, it is worth a try.
I need to store the rdf schema in one table and rdf instance data in another table in ORACLE.
How I can do this?
How to configure the joseki-config.ttl to work for multiple models? Some example will help me to understand the solution.
Is there any possibility to create a single model for multiple tables?
Please let me know.
You need to use SDB with Joseki and Oracle. Then you can have a persistent datasets (a collection of models). There is an example of an SDB configuration in the Joseki download in joseki-config-sdb.ttl.
SDB controls the database table layout. A model is stored in the default graph name or in the named graphs table. There is no control for other layouts without changing the code of SDB.
Note that TDB, a custom database layer for Jena, scales better and is faster than using a relational database over JDBC. Fuseki is the new version of Joseki.
The Jena user mailing list is jena-users#incubator.apache.org.