I'm just trying to import data to database (Sql) from a dataset, but its a bit slower when I try to import 70000 rows. Am I doing something wrong or missing?
Could please give me some advice how can I do it better?
Here is my asp.net code:
ArtiDB entity = new ArtiDB();
int grid = 50;
foreach (string item_kisiler in kisiler)
{
if (item_kisiler == "")
continue;
if (Tools.isNumber(item_kisiler) == false)
continue;
else
{
string gsm1 = item_kisiler;
if (gsm1.Length > 10)
gsm1 = gsm1.Substring(1, 10);
entity.veriaktar(gsm1, gg, grid);
}
}
This is my store prosedure:
alter proc veriaktar
(
#gsm1 nvarchar(50)=null,
#userid uniqueidentifier,
#grupid int = 0
)
as
begin
Declare #AltMusID int
if not exists (select * from tbl_AltMusteriler with (updlock, rowlock, holdlock) where Gsm1=#gsm1 and UserId=#userid)
begin
insert into tbl_AltMusteriler (Gsm1,UserId)
values (#gsm1,#userid)
Set #AltMusID = scope_identity()
end
else
begin
Set #AltMusID = (select AltMusteriID from tbl_AltMusteriler with (updlock, rowlock, holdlock) where Gsm1=#gsm1 and UserId=#userid)
end
if (#grupid != 0)
begin
if not exists (select * from tbl_KisiGrup with (updlock, rowlock, holdlock) where GrupID=#grupid and AltMusteriID=#AltMusID)
begin
insert into tbl_KisiGrup values(#grupid,#AltMusID)
end
end
end
go
The server is designed to work with sets. You're requiring it to deal with one row at a time, and with each row three times. Stop doing that and things will get better.
First go back to your VB docs and look for a way to do one INSERT for all 70,000 rows. If you can use the Bulk Copy (bcp) feature of SQL Server, you should be able to insert the whole set in 10-20 seconds.
The read-test-update paradigm might work here, but it's error-prone and forces the server to work much harder than necessary. If some of the 70,000 are new and others are updates, bulk them into a temporary table and use MERGE to apply it to tbl_AltMusteriler.
Second, uniqueidentifier isn't a good sign. It looks like tbl_AltMusteriler is used to generate a surrogate key. Why wouldn't a simple integer do? It would be faster to generate (with IDENTITY), easier to read, faster to query, and have better PK properties generally. (Also, make sure both the natural key and the surrogate are declared to be unique. What would it mean if two rows have the same values for gsm1 and userid, differing only by AltMusteriID?)
In short, find a way to insert all rows at once, so that your interaction with the DBMS is limited to one or at most two calls.
Related
I need to merge some randomly uppercased data that has been collected in an SQLite table key_val, such that key is always lowercase and no vals are lost. There is a unique compound index on key,val.
The initial data looks like this:
key|val
abc|1
abc|5
aBc|1
aBc|5
aBc|3
aBc|2
AbC|1
abC|3
The result after the merge would be
key|val
abc|1
abc|2
abc|3
abc|5
In my programmer brain, I would
for each `key` with upper case letters;
if a lower cased `key` is found with the same value
then delete `key`
else update `key` to lower case
Re implementing the loop has a sub query for each row found with upper case letters, to check if the val already exists as a lower case key
If it does, I can delete the cased key.
From there I can UPDATE key = lower(key) as the "duplicates" have been removed.
The first cut of the programming method of finding the dupes is:
SELECT * FROM key_val as parent
WHERE parent.key != lower(parent.key)
AND 0 < (
SELECT count(s.val) FROM key_val as s
WHERE s.key = lower(parent.key) AND s.val = parent.val
)
ORDER BY parent.key DESC;
I'm assuming there's a better way to do this in SQLite? The ON CONFLICT functionality seems to me like it should be able to handle the dupe deletion on UPDATE but I'm not seeing it.
First delete all the duplicates:
DELETE FROM key_val AS k1
WHERE EXISTS (
SELECT 1
FROM key_val AS k2
WHERE LOWER(k2.key) = LOWER(k1.key) AND k2.val = k1.val AND k2.rowid < k1.rowid
);
by keeping only 1 combination of key and val with the min rowid.
It is not important if you kept the key with all lower chars or not, because the 2nd step is to update the table:
UPDATE key_val
SET key = LOWER(key);
See the demo.
Honestly it might just be easier to create a new table and then insert into it. As it seems you really just want a distinct select here, use:
INSERT INTO kev_val_new ("key", val)
SELECT DISTINCT LOWER("key"), val
FROM key_val;
Once you have populated the new table, you may drop the old one, and then rename the new one to the previous name:
DROP TABLE key_val;
ALTER TABLE key_val_new RENAME TO key_val;
I agree with #Tim that it would be easire to re-create table using simple select distict lower().. statement, but that's not always easy if table has dependant objects (indexes, triggers, views). In this case this can be done as sequence of two steps:
insert lowered keys which are not still there:
insert into t
select distinct lower(tr.key) as key, tr.val
from t as tr
left join t as ts on ts.key = lower(tr.key) and ts.val = tr.val
where ts.key is null;
now when we have all lowered keys - remove other keys:
delete from t where key <> lower(key);
See fiddle: http://sqlfiddle.com/#!5/84db50/11
However this method assumes that key is always populated (otherwise it would be a strange key)
If vals can be null then "ts.val = tr.val" should be replaced with more complex stuff like ifnull(ts.val, -1) = ifnull(tr.val, -1) where -1 is some unused value (can be different). If we can't assume any unused value like -1 then it should be more complex check for null / not null cases.
I have set my column to int not null default 1... but whenever I save my record, it sets default value for that record to be 0.
I am not setting it anywhere. I don't know where I am making a mistake.
I have debugged my code , and when I am passing new entity object it is setting default value for not null to 0 .May be it is something with LINQ, But I don't know how to handle it.I don't want to explicitly assign value.
Thanks!
For sql-server, you can use SQL Server Profiler to catch all the scripts you run into the DB.
This may show you some details
Try running this query, replacing the 'myTable' and 'myColumn' values with your actual TABLE and COLUMN names, and see what's returned:
SELECT
OBJECT_NAME(C.object_id) AS [Table Name]
,C.Name AS [Column Name]
,DC.Name AS [Constraint Name]
,DC.Type_Desc AS [Constraint Type]
,DC.Definition AS [Default Value]
FROM sys.default_constraints DC
INNER JOIN sys.Columns C
ON DC.parent_column_id = C.column_id
AND DC.parent_object_id = C.object_id
WHERE OBJECT_NAME(DC.parent_object_id) = 'myTable'
AND COL_NAME(DC.parent_object_id,DC.parent_column_id) = 'myColumn'
;
Should return something like this:
[Table Name] [Column Name] [Constraint Name] [Constraint Type] [Default Value]
-------------------------------------------------------------------------------------------
myTable myColumn DF_myTable_myColumn DEFAULT_CONSTRAINT ('0')
If the [Default Value] returned is indeed (1), then it means that you have set the constraint properly and something else is at play here. It might be a trigger, or some other automated DML that you've forgotten/didn't know about, or something else entirely.
I am not the world's biggest fan of using a TRIGGER, but in a case like this, it could be handy. I find that one of the best uses for a TRIGGER is debugging little stuff like this - because it lets you see what values are being passed into a table without having to scroll through mountains of profiler data. You could try something like this (again, switching out the myTable and myColumn values with your actual table and column names):
CREATE TABLE Default_Check
(
Action_Time DATETIME NOT NULL DEFAULT GETDATE()
,Inserted_Value INT
);
CREATE TRIGGER Checking_Default ON myTable
AFTER INSERT, UPDATE
AS
BEGIN
INSERT INTO Default_Check (Inserted_Value)
SELECT I.myColumn
FROM Inserted I
;
END
;
This trigger would simply list the date/time of an update/insert done against your table, as well as the inserted value. After creating this, you could run a single INSERT statement, then check:
SELECT * FROM Default_Check;
If you see one row, only one action (insert/update) was done against the table. If you see two, something you don't expect is happening - you can check to see what. You will also see here when the 0 was inserted/updated.
When you're done, just make sure you DROP the trigger:
DROP TRIGGER Checking_Default;
You'll want to DROP the table, too, once it's become irrelevant:
DROP TABLE Default_Check;
If all of this still didn't help you, let me know.
In VB use
Property VariableName As Integer? = Nothing
And
In C# use
int? value = 0;
if (value == 0)
{
value = null;
}
Please check My Example:
create table emp ( ids int null, [DOJ] datetime NOT null)
ALTER TABLE [dbo].[Emp] ADD CONSTRAINT DF_Emp_DOJ DEFAULT (GETDATE()) FOR [DOJ]
1--Not working for Default Values
insert into emp
select '1',''
2 ---working for Default Values
insert into emp(ids) Values(13)
select * From emp
I've got a function that I'd like use to delete a row in my database. This is the only way I've used the DELETE statement to remove a row before but I want the 1 to be replaced by a variable called recID so that the value of recID is the row ID number which is deleted. So if recID = 6, I want the function to delete the row with ID = 6. I hope that makes sense.
'DELETE FROM MyRecords WHERE ID=1';
The notation I've been using is the following, if it helps or makes any difference.
db.transaction(function(transaction) {
transaction.executeSql( //DELETE STATEMENT HERE );
});
executeSql supports arguments (check definition).
Use it like:
db.transaction(function(transaction) {
transaction.executeSql("DELETE FROM MyRecords WHERE ID=?", [recId]);
});
If you're certain that your variable, recID, will only ever contain numbers, you can just use:
transaction.executeSql("DELETE FROM MyRecords WHERE ID=" + recID);
If recID comes from outside your application (user input) however, it either needs to be sanitized, or use a prepared statement and use the database API to set the parameter after the statement has been prepared. Otherwise you open yourself up to SQL injection attacks.
I don't know the details of your SQLite wrapper, or what version of SQLite it wraps, but creating a prepared statement using the SQLite3 C API would go something like this:
// sqlite3* db = ...
sqlite3_stmt* stmt;
sqlite3_prepare_v2(db, "DELETE FROM MyRecords WHERE ID=?", -1, &stmt, 0);
sqlite3_bind_int(stmt, 1, recID);
sqlite3_step();
// ...
sqlite3_finalize(stmt);
This simple example excludes all the error checking you'd want to do in a real application, but since you're using a wrapper that has different syntax anyway, you'd have to figure out how it wraps these functions anyway.
This is my table where i want my PNRNo to be generated as 'PNRRES001' for the first entry, and consecutive entries with 'PNRRES002','PNRRES002' so on.
So while creating table only i called that column to function which will generate the PNR no, User just has to enter the CustomerNo from the front end, and data wit PNR & Customer No will updated to the PNRDetails table.
CREATE TABLE PNRDetails(PNRNo AS (DBO.FuncIncPNR()) ,customerNo INT
--FUNCTION TO GENERATE THE PNR NUMBER
ALTER FUNCTION dbo.FuncIncPNR()
RETURNS VARCHAR(20)
AS
BEGIN
DECLARE #RR VARCHAR(20) SET #RR='PNRRESA001'
--here i have checked if no value is there then return the first value as 'PNRRESA001'
IF((SELECT COUNT(*)FROM PNRDetails)=0)
BEGIN
RETURN #RR
END
ELSE
-- if any value is there then take the last value and add 1 to it and update to the table
BEGIN
DECLARE #pnr VARCHAR(20),#S1 VARCHAR(20),#S2 INT
DECLARE PNRCursor CURSOR Static
FOR SELECT PNRNo FROM PNRDetails
OPEN PNRCursor
FETCH LAST FROM PNRNo INTO #pnr
SET #S1=SUBSTRING(#pnr,1,7)
SET #S2=RIGHT(#PNR,3)
SET #S2=#S2+1;
SET #pnr=#S1+#S2;
END
RETURN #pnr
END
--Here am inserting only customerNo as 5 and the PNR should be generated by my function
INSERT INTO PNRDetails VALUES(5)
--it shows 1 row updated :)
SELECT * FROM PNRDetails
-- but when i run select command it shows
--Maximum stored procedure, function, trigger, or view nesting level exceeded (limit 32). :(
U can run this.And pls do help if u find anything that could help me. any help will be appreciated...
Waiting for your kind response...
You could try to use a computed column and an identity column instead.
create table PNRDetails
(
ID int identity,
PNRNo as 'PNRRES'+right(1000+ID, 3),
customerNo int
)
I would suggest just using an IDENTITY instead as your id, let SQL Server handle the assignment of each id number with all it's built-in guards for concurrency, and leave the formatting up to the UI....or, create a computed column that defines the formatted version of the ID if you really do need it in the DB.
The risk you run with your intended approach is:
poor performance
concurrency issues - if loats of ids are being generate around the same time
If you are happy to change the table structure. Following will do the job.
CREATE TABLE [dbo].[PNRDetails](
[autoId] [int] IDENTITY(1,1) NOT NULL,
[prnNo] AS ('PNRRES'+right('000'+CONVERT([varchar](3),[dbo].[GetRowCount]([autoId]),(0)),(3))),
[customerNo] [int] NOT NULL,
CONSTRAINT [PK_Table1] PRIMARY KEY CLUSTERED
(
[autoId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
EDIT: to address identity issue for your requirement pls create following function and pass the [autoId] in as above (edited) in the computed column.
CREATE FUNCTION dbo.GetRowCount
(
#autoId INT
)
RETURNS INT
AS
BEGIN
DECLARE #RESULTS AS INT
SELECT #RESULTS = COUNT(autoId) FROM PNRDetails WHERE PNRDetails.autoId<#autoId
RETURN #RESULTS + 1
END
GO
--INSERT
INSERT INTO PNRDetails (customerNo) VALUES(5)
1) You can use an identity column in your database (INTEGER)
PROS: easy/No gaps in between generated ids
CONS: You have to select the inserted id & return via procedure/query
if you were to show it to end user
2) Define a database sequence
PROS: easy to implement/Can be stored/shown to user before the form is
even saved
CONS: Gaps in between if the certain id is once generated & not used
3). Select max(id) from column + 1
PROS: Useful where only single user inserts in a table
CONS: disastrous if you were in an environment where multiple users
were inserting in the same tablle (mismatched max ids)
4) Use a database trigger to autoincrement the column
PROS:automated
CONS: hard to debug (you have to make sure it don't breaks for some
reason otherwise insert fails)
Change the way your trigger works. Something like this
CREATE FUNCTION dbo.fn_FuncIncPNR(#ID int)
RETURNS varchar(20)
BEGIN
Declare #Retval varchar(20),
#No varchar(4)
Select #No = convert(varchar(4), #ID)
while Len(#No) < 4
Select #No = '0' + #No
Select #Retval = 'PNRRESA' + #No
RETURN #Retval
END
You will notice there is a parameter field
Change your table create to this
CREATE TABLE PNRDetails(PNRNo AS (dbo.fn_ShowPNRNo(wID)), wID int IDENTITY(1,1) NOT NULL, customerNo INT)
That should solve your problem
I've just thought about best way to store comments in database with appropriate numbers according to the article.
The idea is to store comments with composite primary key (commentId, articleId) where commentId is generated according to the given articleId. The system of generating should has same principle as IDENTITY generated columns in SQL Server, because if someone delete the comment, the number will be never used again. I guess there is not any functionality in Microsoft SQL Server to do that with composite PK, so I am asking about some replacement for this solution.
First thought was to use transaction to get MAX(commentId) + 1, but I am looking for something more abstract (maybe INSTEAD OF trigger), something that could be used for example in LINQ with no knowledge of the background, just insert to the appropriate table all required values (so no commentId) and save it.
I would use an autogenerated identity column for the commentId and have it be the primary key alone. I'd create an index on the articleId for look ups. I would also have createdDate column that is autopopulated with the current date on insertion -- mark it as db generated and readonly in LINQ so it doesn't require or try to insert/update the value. To get a numbering -- if showing them by date isn't enough -- I'd order by createdDate inversed and assign a numeric value in the select using Row_Number() or a numbering on the client side.
I would use an identity column as the key for the comments, why do you need a numbering for the comments stored in the database?
Thank you for responses, I wanted something with numbered comments because of referencing in the text of comments. I did not want to make reaction by names, sometimes one person reacts more times, so with this system, I will know to which one the person is replying.
So today I made up this INSTEAD OF INSERT trigger:
CREATE TRIGGER InsertComments ON Comments
INSTEAD OF INSERT
AS
DECLARE #Inserted TABLE
(
ArticleId INT NOT NULL,
UserId INT NOT NULL,
CommentDate DATETIME NOT NULL,
Content NVARCHAR(1000) NOT NULL,
RowNumber INT NOT NULL
)
INSERT INTO #Inserted
SELECT ArticleId, UserId, CommentDate, Content, ROW_NUMBER() OVER (ORDER BY CommentDate) AS RowNumber
FROM INSERTED
DECLARE #NumberOfRows INT = (SELECT COUNT(*) FROM #Inserted)
DECLARE #i INT = 1
WHILE (#i <= #NumberOfRows)
BEGIN
SET TRANSACTION ISOLATION LEVEL SERIALIZABLE
BEGIN TRAN
DECLARE #CommentId INT = (SELECT ISNULL(MAX(CommentId), 0)
FROM Comments WHERE ArticleId = (SELECT ArticleId
FROM #Inserted WHERE RowNumber = #i)) + 1
INSERT INTO Comments(CommentId, ArticleId, UserId, CommentDate, Content)
SELECT #CommentId, ArticleId, UserId, CommentDate, Content
FROM #Inserted WHERE RowNumber = #i
COMMIT
SET #i = #i + 1
END
I know this is not the perfect solution, but it works exactly how I needed. If any of you has some comments, I'll be happy to read them.