I am using NDBUbit to load data from XML file.Right now,I am manually giving GUID for each record(our primary key for all tables is unique-identifier) in the xml file.But,I wonder if there is a better way to do this?
For all test data in XML files for support of NDbUnit where your PK values are of type GUID, take note that all that's required is for the data type to be a GUID -- its doesn't actually have to be a 'random' GUID.
For such test data, so that you can properly reference other PKs in FKs (for example), I always recommend 'deterministic human-created GUIDs' be used in your test data. They both satisfy the req'ment that the value be a GUID and make working with them easier when you are crafting your test data.
As an example, see the following:
<?xml version="1.0" encoding="utf-8" ?>
<UserDS xmlns="http://tempuri.org/UserDS.xsd">
<Role>
<ID>11111111-1111-1111-1111-111111111111</ID>
<Name>Admin</Name>
<Description>Serves as an administrator</Description>
</Role>
<Role>
<ID>22222222-2222-2222-2222-222222222222</ID>
<Name>User</Name>
<Description>User with limited security</Description>
</Role>
<dbo.User>
<ID>22222222-2222-2222-2222-222222222222</ID>
<FirstName>John</FirstName>
<LastName>Williams</LastName>
<Age>30</Age>
<SupervisorID>11111111-1111-1111-1111-111111111111</SupervisorID>
</dbo.User>
<dbo.User>
<ID>11111111-1111-1111-1111-111111111111</ID>
<FirstName>Hammad</FirstName>
<LastName>Awan</LastName>
<Age>29</Age>
</dbo.User>
<UserRole>
<UserID>11111111-1111-1111-1111-111111111111</UserID>
<RoleID>11111111-1111-1111-1111-111111111111</RoleID>
</UserRole>
<UserRole>
<UserID>22222222-2222-2222-2222-222222222222</UserID>
<RoleID>11111111-1111-1111-1111-111111111111</RoleID>
</UserRole>
<UserRole>
<UserID>11111111-1111-1111-1111-111111111111</UserID>
<RoleID>22222222-2222-2222-2222-222222222222</RoleID>
</UserRole>
</UserDS>
In this case, to manage this test data all you need to 'know' about GUIDs is that they are 32chars long in the format 8chars-4chars-4chars-4chars-12chars. There's NOTHING that keeps all the characters from being the SAME if you are crafting your test data 'by hand' like this. This approach eliminates any need to generate "actual GUIDs" for your test data and permits you to easily refer to these by simple repeat-representations of GUIDs as shown in the sample above.
Note that in this kind of a scenario YOU are taking 100% of the responsibility for the 'uniqeness' of the GUIDs that you enter into your test data so this test data can only safely be used in isolation from other GUID-based data rows. This (generally) shouldn't be any issue as the whole point of NDbUnit is to load and manage data such as this in isolation.
If you don't want to edit your test data by hand in an XML file like this, we are close to an alpha-release of a tool called 'NDbUnit DataSet Editor' that provides a GUI for editing test data and also contains a 'Generate and insert GUID' toolbar button for just such situations. However, do note that the GUIDs generated in such a case will be 'real' GUIDs meaning that copy-and-paste would be the only reasonable way to reference one as a FK of another record elsewhere in the DataSet.
Hope this helps.
Related
Context: I'm following this guide: https://learn.microsoft.com/en-us/azure/kusto/api/netfx/kusto-ingest-client-examples
I'm using IngestFromStorageAsync - I see that the results have an IngestionSourceId (a GUID) - but I don't know what to do with this. (this is not the extent id)
I was assuming that you could use this ID to remove all the records that were imported...
Does anyone know how to undo an ingestion?
Currently, I'm using .show cluster extents to show the extent ids, then I call .drop extent [id]. Is this the right way to undo an ingestion?
"undo"ing an ingestion is essentially dropping the data that was ingested.
dropping data can be done at the resolution of extents (data shards), and extents can get merged with one another at any given moment (e.g. straight after data was ingested).
if you know there's a chance you'll want to drop the data you've just ingested (and you can't fix the ingestion pipeline that leads to those "erroneous"(?) ingestions), one direction you could follow would be to use extent tags, to be able to identify the extents that were created as part of your ingestion, then drop them.
more information can be found here: https://learn.microsoft.com/en-us/azure/kusto/management/extents-overview.
if you do choose to use tags for this purpose (and can't avoid the situations where you need to "undo" your ingestions), please make sure you read the "performance notes" in that doc.
Excerpt from documentation link:
'ingest-by:' extent tags
Tags that start with an ingest-by: prefix can be used to ensure that data
is only ingested once. You can issue an ingestIfNotExists property command that prevents the data from being ingested if there already exists an extent with this specific ingest-by: tag.
The values for both tags and ingestIfNotExists are arrays of strings,
serialized as JSON.
The following example ingests data only once. The 2nd and 3rd commands do nothing:
.ingest ... with (tags = '["ingest-by:2016-02-17"]')
.ingest ... with (ingestIfNotExists = '["2016-02-17"]')
.ingest ... with (ingestIfNotExists = '["2016-02-17"]', tags = '["ingest-by:2016-02-17"]')
[!NOTE]
Generally, an ingest command is likely to include
both an ingest-by: tag and an ingestIfNotExists property,
set to the same value, as shown in the 3rd command above.
[!WARNING]
Overusing ingest-by tags isn't recommended.
If the pipeline feeding Kusto is known to have data duplications, we recommend that you solve these duplications as much as possible, before ingesting the data into Kusto.
Attempting to set a unique ingest-by tag for each ingestion call might result with severe impact on performance.
If such tags aren't required for some period of time after the data is ingested, we recommend that you drop extent tags.
To drop the tags automatically, you can set an extent tags retention policy.
I'm trying to get a better understanding of what Rows and Rowsets are used for in PeopleCode? I've read through PeopleBooks and still don't feel like I have a good understanding. I'm looking to get more understanding of these as it pertains to Application Engine programs. Perhaps walking through an example may help. Here are some specific questions I have:
I understand that Rowsets, Row, Record, and Field are used to access component buffer data, but is this still the case for stand alone Application Engine programs run via Process Scheduler?
What would be the need or advantage to using these as opposed to using SQL objects/functions (CreateSQL, SQLExec, etc...)? I often see in AE programs where the CreateRowset object is instantiated and uses a .Fill method with a SQL WHERE Clause and I don't quite understand why a SQL was not used instead.
I've seen in PeopleBooks that a Row object in a component scroll is a row, how does a component scroll relate to the row? I've seen references to rows having different scroll levels, is this just a way of grouping and nesting related data?
After you have instantiated the CreateRowset object, what are typical uses of it in the program afterwards? How would you perform logic (If, Then, Else, etc..) on data retrieved by the rowset, or use it to update data?
I appreciate any insight you can share.
You can still use Rowsets, Rows, Records and fields in stand alone Application Engines. Application Engines do not have component buffer data as they are not running within the context of a component. Therefore to use these items you need to populate them using built-in methods like .fill() on a rowset, or .selectByKey() on a record.
The advantage of using rowsets over SQL is that it makes the CRUD easier. There are built-in methods for selecting, updating, inserting and deleting. Additionally you don't have to worry about making a large number of variables if there were multiple fields like you would with a SQL object. Another advantage is when you do the fill, the data is read into memory, where if you looped through the SQL, the SQL cursor would be open longer. The rowset, row, record and field objects also have a lot of other useful methods such as allowing you to executeEdits (validation) or copy from one rowset\row\record to another.
This question is a bit less clear to me but I'll try and explain. If you have a Page, it would have a level 0 row. It then could have multiple Level 1 rowsets. Under each of those it could have a level 2 rowsets.
Level0
/ \
Level1 Level1
/ \ / \
Level2 Level2 Level2 Level2
If one of your level1 rows had 3 rows, then you would find 3 rows in the Rowset associated with that level1. Not sure I explained this to answer what you need, please clarify if I can provide more info
Typically after I create a rowset, I would loop through it. Access the record on each row, do some processing with it. In the example below, I look through all locked accounts and prefix their description with LOCKED and then updated the database.
.
Local boolean &updateResult;
local integer &i;
local record &lockedAccount;
Local rowset &lockedAccounts;
&lockedAccounts = CreateRowset(RECORD.PSOPRDEFN);
&lockedAccounts.fill("WHERE acctlock = 1");
for &i = 1 to &lockedAccounts.ActiveRowCount
&lockedAccount = &lockedAccounts(&i).PSOPRDEFN;
if left(&lockedAccount.OPRDEFNDESCR.value,6) <> "LOCKED" then
&lockedAccount.OPRDEFNDESCR.value = "LOCKED " | &lockedAccount.OPRDEFNDESCR.value;
&updateResult = &lockedAccount.update();
if not &updateResult then
/* Error handle failed update */
end-if;
end-if;
End-for;
I have a requirement to load the csv into DB using oracle apex or pl/sql code, but the problem is they are asking to load the csv file which will not come with same number of columns and column names .
I should create table & upload data dynamically based on the file name and data that i'm uploading.
For every file i need to create a new table dynamically and insert data that are present in csv file.
For Example:
File1:
col1 col2 col3 col4 (NOTE: If i upload File 1, Table should be created dynamically based on the file name and table should contain same column name and data same as column headers of csv file . )
file 2:
col1 col2 col3 col4 col 5
file 3:
col4 col2 col1 col3
Depending on the columns and file name i need to create table for every file upload.
Can we load like this or not?
If yes, Please help me on this.
Regards,
Sachin.
((Where's the PL/SQL code in this solution!!??! Bear with me... the
answer is buried in here somewhere... I introduced some considerations
and assumptions you will need to think about before going into the
task. In the end, you'll find that Oracle APEX actually has a
built-in solution that satisfies exactly what you've specified... with
some caveats.))
If you are working within the Oracle APEX platform, you will have some advantages. APEX Version 4.2 and higher has a new page element called "Data Loading". The disadvantage however is that the definition of the upload target is fixed and not dynamic. You will need to know how your table is structured prior to loading the data.
One approach to overcome this is to build a generic, two-column table as your target, which will serve for all uploads. Column 1 will be your file-name and column two will be a single clob data type, which will contain the entire data file's contents including the header row. The "Data Loading" element will give the user the opportunity to verify and select this mapping convention in a couple of clicks.
At this point, it's mostly PL/SQL backend work doing the heavy lifting to parse and transform the data uploaded. As far as the dynamic table creation, I have noticed that the Oracle package, DBMS_SQL allows the execution of DDL SQL commands, which could be the route to making custom tables.
Alex Poole's comment is important as well, you will need to make some blanket assumption about the data type or have a provision to give more clues about what kind of data is contained. Assuming you can rely on a sample of existing data values is not good... what if all the values in your upload are null? I recommend perhaps a second column in the data input with a clue about the type of data for each column... just like the intended header names, maybe: AAAAA = for a five character column, # = for a numeric, MM/DD/YYYY = for a date with a specific masking.
The easier route:
You will need to allow your end-user access to a developer-role account on a workspace of your APEX server. It is not as scary as you think. With careful instruction and some simple precautions, I have been able to make this work with even the most non-technical of users. The reason for this is that there is a more powerful upload tool found under the following menu item:
SQL Workshop --> Utilities --> Data Workshop
There is a choice under "Data Load" --> "Spreadsheet Data"
The data load tool will automatically do the following:
Accept a CSV formatted file through a browse function on your client machine
Upload the file and parse the first record for the column layout (names)
Allow the user to create a new table from the uploaded file, or to map to an existing one.
For new tables, each column data type can be declared and also a specific numeric/date mask if additional conversion from the uploaded data is necessary.
Delimiter type, optional enclosures (like double quotes), decimal conventions and currency types can also be declared prior to parsing the uploaded file.
Once the user has identified all these mappings and settings, the table is created with the uploaded data. Any errors in record upload are reported immediately afterwards with detailed feedback on the failed records.
A security consideration to note:
You probably do not want to give end users access to your APEX server's backend... but you CAN create a new workspace... just for your end users... create a new database schema for receiving their uploads, maybe with some careful resource controls. Developer is the minimum role needed... but even if the end users see the other stuff there won't be access to anything important from an isolated workspace.
I have implemented the isolated workspace approach on a 4.0/4.1 release APEX platform a few years back, and it worked nicely. Our end user had control over the staging and quality checking of her data inputs (from excel spreadsheet/csv exports collected from a combination of sources). I suppose it may have been even better to cut her out of the picture entirely and focused on automating the export-review-upload process between our database and her other sources. In this case, the volume of data involved was not great enough (100's to 1000's of records) and the need for manual review and edit of the exported data was very important prior to pushing it into the database... so the human element was still important in this case - it is something you'll want to think about now.
I'm trying to build a Dynamic Data website to act as a simple admin interface to a MySQL database.
This isn't my area of expertise - but it all looks pretty straightforward when following the microsoft website. However for some reason when using the Entity Data Model to the project, the edmx model is created, but doesn't read one of the tables correctly.
In the MySQL database I have a large table containing 9 columns. The first column is the primary key and is an INT. The remaining columns are of type Decimal with a length of 6,2.
The edmx file shows the INT column - but fails to read any of the Decimal columns. If I change these to VARCHAR columns - everything is read perfectly?
The Dynamic Data Field Templates contains a control for decimals and surely the entity framework must be able to recognize columns of DECIMAL type - so I must be doing something wrong - I just can't work out what!
Any help much appreciated!
Cheers
Darren
Ok -I discovered the answer myself!
The issue was due to using unsigned decimal values in the table structure.
Dynamic Data / Entity frame work does not recognise the decimal data type if the column is marked as unsigned. When I changed this, the entity framework was able to read the DB structure correctly!
Hope this helps anyone else who comes across the same issue!
Cheers
Darren
Dynamic Data / Entity frame work does not recognize the decimal data type if the column is marked as unsigned. When I changed this, the entity framework was able to read the DB structure correctly!
I got it with:
Property(x => x.NameColumn).HasColumnName(#"NameColumn").HasColumnType("decimal").IsRequired().HasPrecision(9, 4)
I'm looking to create a table for user preferences and can't figure the best way to do it. The way that the ASP.NET does it by default seems extremely awkward, and would like to avoid that. Currently, I'm using one row per user, where I have a different column for each user preference (not normalized, I know).
So, the other idea that I had come up with was to split the Preferences themselves up into their own table, and then have a row PER preference PER user in a user preferences table; however, this would mean each preference would need to be the exact same datatype, which also doesn't sound too appealing to me.
So, my question is: What is the best/most logical way to design a database to hold user preference values?
Some of the ideas that I try to avoid in database work, is data duplication and unnecessary complication. You also want to avoid "insert, update, and deletion anomalies". Having said that, storing user preferences in one table with each row = one user and the columns, the different preferences that are available, makes sense.
Now if you can see these preferences being used in any other form or fashion in your database, like multiple objects (not just users) using the same preferences, then you'll want to go down your second route and reference the preferences with FK/PK pairs.
As for what you've described I see no reason why the first route won't work.
I usually do this:
Users table (user_id, .... etc.)
.
Options table (option_id, data_type, ... etc.)
(list of things that can be set by user)
.
Preferences table (user_id, option_id, setting)
I use the new SQLVARIANT data type for the setting field so it can be different data types and record the data type of the option as part of the option definition in the Options table for casting it back to the right type when queried.
If you store all your user preferences in a single row of a User table you will have a maintenance nightmare!
Use one row per preference, per user and store the preference value as a varchar (length 255 say, or some value large enough to meet your requirements). You will have to convert values in/out of this column obviously.
The only situation where this won't work easily is if you want to store some large binary data as a User preference, but I have not found that to be a common requirement.
Real quick, one method:
User(UserID, UserName, ...)
PreferenceDataType(PreferenceDataTypeID, PreferenceDataTypeName)
PreferenceDataValue(PreferenceDataValueID, PreferenceDataTypeID, IntValue, VarcharValue, BitValue, ...)
Preference(PreferenceID, PreferenceDataTypeID, PreferenceName, ...)
UserHasPreference(UserID, PreferenceID, PreferenceDataValueID)