We use SSDT to deploy our database changes. We have a script that recreates the index every week. Our script looks like this:
declare #cmd varchar(max)
set #cmd = '
CREATE NONCLUSTERED INDEX [iAudit-ModifiedDateTime] ON [dbo].[Audit]
(
[ModifiedDateTime] ASC
)
WHERE ModifiedDateTime > ''###''
WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, SORT_IN_TEMPDB = OFF, DROP_EXISTING = ON, ONLINE = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON, FILLFACTOR = 75) ON [PRIMARY]
'
set #cmd = replace(#cmd, '###', convert(varchar(8), dateadd(day, -3, getdate()), 112))
exec (#cmd)
Unfortunately when we run SSDT to update the database it changes the index to the definition in the project, or drops it when it is not included. Is there some way I can get around this?
The reason we need the filtered index is to add the latest records from an Audit table with 100's of millions of rows, into a data warehouse.
There are some options, in order of complexity:
Don't include the index definition in the project and disable the "Drop indexes not in source" option. In Visual Studio this is found in the Advanced options dialog of the Publish dialog. When using SqlPackage.exe to publish, you can use the parameter /p:DropIndexesNotInSource=false
Don't include the index definition in the project and put the index creation script into a post-deployment script. This will ensure that the index is always recreated after schema updates are deployed.
Use a community-authored deployment contributor to filter out modifications to this index. See https://the.agilesql.club/Blogs/Ed-Elliott/HOWTO-Filter-Dacpac-Deployments
Author a deployment contributor to filter out modifications to this index. See https://github.com/Microsoft/DACExtensions/
Related
I am trying to change the type of a column from VARCHAR to INT.
During the deployment, the database project will stop the deployment due to the "data loss" error:
RAISERROR (N'Rows were detected. The schema update is terminating because data loss might occur.', 16, 127)
I know the data is convertible and if I run a manual ALTER TABLE script it will be fine. However, I cannot integrate that properly with this scenario to avoid the error during the deployment.
What is your solution to resolve my problem?
Is there a method
to override this behaviour in a database project and for this
particular case, use a custom script?
One way in such scenario is using PreDeployment script and deploy twice.
Change data type column in table definition as usual
Add in Predeploy script:
-- this script has to be idempotent, and removed after some time
IF EXISTS (SELECT 1
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'table_name'
AND TABLE_SCHEMA = 'schema_name'
AND COLUMN_NAME = 'column_name
AND DATA_TYPE != 'INT'
)
BEGIN
ALTER TABLE schema_name.table_name ALTER COLUMN Column_name INT NULL/NOT NULL;
END
First publish will change the data type during PreDeploy, and deploy will fail with Potential Data loss error.
Second publish will omit the part of PreDeploy(if condition), and schema compare does not detect any changes, meaning it has been changed.
Next step should be removing the manual part from PreDeployment script.
I want to create a SQL Server database project where a new database is created. Here is the structure of my project :
SQL Server Database Project Structure
Basically, I want 4 tables, so I have added 4 .sql scripts for them.
CREATE TABLE [dbo].[MovieActors]
(
[Id] INT NOT NULL PRIMARY KEY,
[ActorID] INT NOT NULL,
[MovieID] INT NOT NULL
)
CREATE TABLE [dbo].[Actors]
(
[Id] INT NOT NULL PRIMARY KEY,
[Name] NVARCHAR(50) NOT NULL,
[Bio] NVARCHAR(MAX) NULL,
[Sex] NCHAR(10) NOT NULL,
[DOB] DATE NULL
)
CREATE TABLE [dbo].[Movies]
(
[Id] INT NOT NULL PRIMARY KEY,
[Name] NVARCHAR(50) NOT NULL,
[Year of Release] SMALLINT NULL,
[PLOT] NVARCHAR(MAX) NULL,
[Poster] NVARCHAR(MAX) NULL,
[ProducerId] int FOREIGN KEY REFERENCES Producers(Id)
)
CREATE TABLE [dbo].[Producers]
(
[Id] INT NOT NULL PRIMARY KEY,
[Name] NVARCHAR(50) NOT NULL,
[Bio] NVARCHAR(MAX) NULL,
[Sex] NCHAR(10) NULL,
[DOB] DATE NULL
)
And I have also added pre-deployment (for deleting existing data) and post-deployment scripts (for seeding initial data).
Pre-Deployment:
DROP TABLE [dbo].MovieActors
DROP TABLE [dbo].Movies
DROP TABLE dbo.Actors
DROP TABLE dbo.Producers
Post-Deployment:
INSERT INTO dbo.Actors (Id, Name, Sex)
VALUES (1, 'Actor1', 'MALE')
INSERT INTO dbo.Producers (Id, Name, Sex)
VALUES (1, 'Producer1', 'MALE')
INSERT INTO dbo.Movies (Id, Name, ProducerId)
VALUES (1, 'Movie1', 1)
INSERT INTO dbo.MovieActors (Id, MovieID, ActorID)
VALUES (1, 1, 1)
But everytime I am building the project it does not get deployed - only showing build succeeded
Moreover, where is the database creation script.
When I right clicked on publish and chose option to generate script , only pre and post deployment scripts were in the main script. -
Right Click Project -> Publish option
This is the script that gets generated -
/*
Deployment script for MoviesDatabase
This code was generated by a tool.
Changes to this file may cause incorrect behavior and will be lost if
the code is regenerated.
*/
GO
SET ANSI_NULLS, ANSI_PADDING, ANSI_WARNINGS, ARITHABORT, CONCAT_NULL_YIELDS_NULL, QUOTED_IDENTIFIER ON;
SET NUMERIC_ROUNDABORT OFF;
GO
:setvar DatabaseName "MoviesDatabase"
:setvar DefaultFilePrefix "MoviesDatabase"
:setvar DefaultDataPath "C:\Users\viiye\AppData\Local\Microsoft\VisualStudio\SSDT"
:setvar DefaultLogPath "C:\Users\viiye\AppData\Local\Microsoft\VisualStudio\SSDT"
GO
:on error exit
GO
/*
Detect SQLCMD mode and disable script execution if SQLCMD mode is not supported.
To re-enable the script after enabling SQLCMD mode, execute the following:
SET NOEXEC OFF;
*/
:setvar __IsSqlCmdEnabled "True"
GO
IF N'$(__IsSqlCmdEnabled)' NOT LIKE N'True'
BEGIN
PRINT N'SQLCMD mode must be enabled to successfully execute this script.';
SET NOEXEC ON;
END
GO
USE [$(DatabaseName)];
GO
/*
Pre-Deployment Script Template
--------------------------------------------------------------------------------------
This file contains SQL statements that will be executed before the build script.
Use SQLCMD syntax to include a file in the pre-deployment script.
Example: :r .\myfile.sql
Use SQLCMD syntax to reference a variable in the pre-deployment script.
Example: :setvar TableName MyTable
SELECT * FROM [$(TableName)]
--------------------------------------------------------------------------------------
*/
Drop table [dbo].MovieActors
Drop table [dbo].Movies
Drop table dbo.Actors
Drop table dbo.Producers
GO
GO
/*
Post-Deployment Script Template
--------------------------------------------------------------------------------------
This file contains SQL statements that will be appended to the build script.
Use SQLCMD syntax to include a file in the post-deployment script.
Example: :r .\myfile.sql
Use SQLCMD syntax to reference a variable in the post-deployment script.
Example: :setvar TableName MyTable
SELECT * FROM [$(TableName)]
--------------------------------------------------------------------------------------
*/
INSERT INTO dbo.Actors (Id,Name,Sex)
values (1,'Actor1','MALE')
INSERT INTO dbo.Producers (Id,Name,Sex)
values (1,'Producer1','MALE')
INSERT INTO dbo.Movies (Id,Name,ProducerId)
values (1,'Movie1',1)
Insert into dbo.MovieActors (Id,MovieID,ActorID)
values (1,1,1)
GO
GO
PRINT N'Update complete.';
GO
As can be seen , the create tables script are not there in this script which is used for publish .Why? and is this the correct way to go about requirement where I need to create tables from scratch?
This is are the simplified steps how the project is published:
Project is built and DACPAC file is the output of this step
DACPAC is compared with the destination database and the publish script is generated
Publish script is executed against the destination database
The problem in your case is that at the time when the 2nd step is executed the tables existed at the destination database, so their creation was not included to the publish script. Pre deployment script is executed at the step 3!
Basically what you need to do is not to drop tables. Just TRUNCATE them in pre script and populate in the post. Or simply use MERGE statements in the post script. You can use generate-sql-merge procedure that would generate MERGE statement for needed table.
This looks like a huge pile, but it's actually a very focused question. This looks bigger than it is because I am providing context and what I have been able to work out so far.
Let me start with the question in more precise terms: "Using an ASP.NET webpage, how can I: (a) "attach" a saved PDF file to a database table using a Formview, and (b) allow the user to view that saved PDF file when the database table row is selected in a Gridview?"
While I can easily store the path and filename of the PDF file, if the PDF file is renamed, moved, or deleted, the database record now has a broken link. My client has requested that I "attach" the actual PDF file to the database record to prevent broken links.
That answers the why: because my client requested it. Now it's a matter of figuring out how.
Here is what I have done up to now in my research:
I learned how to enable Filestream for a SQL Server 2012 database.
I created a table where one of the columns is a varbinary(max).
(Table definition language shown below in "Code Block #1".)
Using available online examples, I was able to test and verify a working T-SQL script -- however, I have not yet succeeded in making this a stored procedure because I do not know how to make the filename a variable in an "Openrowset" statement . (Script shown below in "Code Block #2".)
Where I'm drawing the big blank is the ASP.NET side of the equatiion. Here is the system I hope to set up. I'm not as restricted in terms of the details so long as they work along these lines.
User uses the Formview (connected to the database via SqlDataSource) to type in the values entered on the paper form, and finally "attach" the saved PDF file to the "Scanned_PDF_File" field.
A gridview immediately refreshes, showing the results from the "Scanned_PDFs" table, allowing the user to select a row and view the saved PDF file.
Is this approach possible? Any directions of further research would be greatly appreciated. Thank you for your help.
Code Block #1: Here is the definition of the SQL database table.
USE [IncidentReport_v3]
GO
/****** Object: Table [dbo].[Scanned_PDFs] Script Date: 1/13/2015 11:56:58 ******/
SET ANSI_NULLS ON
GO
SET QUOTED_IDENTIFIER ON
GO
SET ANSI_PADDING ON
GO
CREATE TABLE [dbo].[Scanned_PDFs](
[ID] [int] IDENTITY(1,1) NOT NULL,
[DateEntered] [date] NOT NULL,
[Scanned_PDF_File] [varbinary](max) NOT NULL,
CONSTRAINT [PK_Scanned_PDFs] PRIMARY KEY CLUSTERED ([ID] ASC) WITH (
PAD_INDEX = OFF,
STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON
) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO
SET ANSI_PADDING OFF
GO
Code Block #2: This is the T-Sql Script I used to test the ability to insert a row w/a PDF file. It works great as a proof of concept if I hand-type the PDF file name and path, but I will need to make that filename a variable that the user supplies. I envision using this as a stored procedure -- or perhaps I could use this code on the client side? Not sure yet.
USE IncidentReport_v3;
GO
DECLARE #pdf AS VARBINARY(max)
SELECT #pdf = cast(bulkcolumn AS VARBINARY(max))
FROM openrowset(BULK '\\wales\e$\test\test.pdf', SINGLE_BLOB) AS x
INSERT INTO dbo.Scanned_PDFs (
DateEntered,
Scanned_PDF_File
)
SELECT cast('1/12/2015' AS DATE),
#pdf;
GO
You'll want to convert the PDF to a byte array before getting to the data layer. The [Scanned_PDF_File] gets set to the result. You can parse the file name or take it from some other value.
This link here, might give you everything you need.
This question already has answers here:
Identity increment is jumping in SQL Server database
(6 answers)
Closed 7 years ago.
I have a strange scenario in which the auto identity int column in my SQL Server 2012 database is not incrementing properly.
Say I have a table which uses an int auto identity as a primary key it is sporadically skipping increments, for example:
1,
2,
3,
4,
5,
1004,
1005
This is happening on a random number of tables at very random times, can not replicate it to find any trends.
How is this happening?
Is there a way to make it stop?
This is all perfectly normal. Microsoft added sequences in SQL Server 2012, finally, i might add and changed the way identity keys are generated. Have a look here for some explanation.
If you want to have the old behaviour, you can:
use trace flag 272 - this will cause a log record to be generated for each generated identity value. The performance of identity generation may be impacted by turning on this trace flag.
use a sequence generator with the NO CACHE setting (http://msdn.microsoft.com/en-us/library/ff878091.aspx)
Got the same problem, found the following bug report in SQL Server 2012
If still relevant see conditions that cause the issue - there are some workarounds there as well (didn't try though).
Failover or Restart Results in Reseed of Identity
While trace flag 272 may work for many, it definitely won't work for hosted Sql Server Express installations. So, I created an identity table, and use this through an INSTEAD OF trigger. I'm hoping this helps someone else, and/or gives others an opportunity to improve my solution. The last line allows returning the last identity column added. Since I typically use this to add a single row, this works to return the identity of a single inserted row.
The identity table:
CREATE TABLE [dbo].[tblsysIdentities](
[intTableId] [int] NOT NULL,
[intIdentityLast] [int] NOT NULL,
[strTable] [varchar](100) NOT NULL,
[tsConcurrency] [timestamp] NULL,
CONSTRAINT [PK_tblsysIdentities] PRIMARY KEY CLUSTERED
(
[intTableId] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY]
and the insert trigger:
-- INSERT --
IF OBJECT_ID ('dbo.trgtblsysTrackerMessagesIdentity', 'TR') IS NOT NULL
DROP TRIGGER dbo.trgtblsysTrackerMessagesIdentity;
GO
CREATE TRIGGER trgtblsysTrackerMessagesIdentity
ON dbo.tblsysTrackerMessages
INSTEAD OF INSERT AS
BEGIN
DECLARE #intTrackerMessageId INT
DECLARE #intRowCount INT
SET #intRowCount = (SELECT COUNT(*) FROM INSERTED)
SET #intTrackerMessageId = (SELECT intIdentityLast FROM tblsysIdentities WHERE intTableId=1)
UPDATE tblsysIdentities SET intIdentityLast = #intTrackerMessageId + #intRowCount WHERE intTableId=1
INSERT INTO tblsysTrackerMessages(
[intTrackerMessageId],
[intTrackerId],
[strMessage],
[intTrackerMessageTypeId],
[datCreated],
[strCreatedBy])
SELECT #intTrackerMessageId + ROW_NUMBER() OVER (ORDER BY [datCreated]) AS [intTrackerMessageId],
[intTrackerId],
[strMessage],
[intTrackerMessageTypeId],
[datCreated],
[strCreatedBy] FROM INSERTED;
SELECT TOP 1 #intTrackerMessageId + #intRowCount FROM INSERTED;
END
I have a project which generates snapshots of a database, converts it to XML and then stores the XML inside a separate database. Unfortunately, these snapshots are becoming huge files, and are now about 10 megabytes each. Fortunately, I only have to store them for about a month before they can be discarded again but still, a month of snapshots turn out to become real bad for it's performance...I think there is a way to improve performance a lot. No, not by storing the XML in a separate folder somewhere, because I don't have write access to any location on that server. The XML must stay within the database. But somehow, the field [Content] might be optimized somehow so things will speed up...I won't need any full-text search options on this field. I will never do any searching based on this field. So perhaps by disabling this field for search instructions or whatever?The table has no references to other tables, but the structure is fixed. I cannot rename things, or change the field types. So I wonder if optimizations is still possible.Well, is it?
The structure, as generated by SQL Server:
CREATE TABLE [dbo].[Snapshots](
[Identity] [int] IDENTITY(1,1) NOT NULL,
[Header] [varchar](64) NOT NULL,
[Machine] [varchar](64) NOT NULL,
[User] [varchar](64) NOT NULL,
[Timestamp] [datetime] NOT NULL,
[Comment] [text] NOT NULL,
[Content] [text] NOT NULL,
CONSTRAINT [PK_SnapshotLog]
PRIMARY KEY CLUSTERED ([Identity] ASC)
WITH (PAD_INDEX = OFF,
STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON,
FILLFACTOR = 90) ON [PRIMARY],
CONSTRAINT [IX_SnapshotLog_Header]
UNIQUE NONCLUSTERED ([Header] ASC)
WITH (PAD_INDEX = OFF,
STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON,
FILLFACTOR = 90)
ON [PRIMARY],
CONSTRAINT [IX_SnapshotLog_Timestamp]
UNIQUE NONCLUSTERED ([Timestamp] ASC)
WITH (PAD_INDEX = OFF,
STATISTICS_NORECOMPUTE = OFF,
IGNORE_DUP_KEY = OFF,
ALLOW_ROW_LOCKS = ON,
ALLOW_PAGE_LOCKS = ON,
FILLFACTOR = 90)
ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
Performance isn't just slow when selecting data from this table but also when selecting or inserting data in one of the other tables in this database! When I delete all records from this table, the whole system is fast. When I start adding snapshots, performance starts to decrease. After about 30 snapshots, performance becomes bad and the risk of connection timeouts increase.Maybe the problem isn't in the database itself, although it's still slow when used through the management tool. (Fast when Snapshots is empty.) I mainly use ASP.NET 3.5 and the Entity Framework to connect to this database and then read the multiple tables. Maybe some performance can be gained here, although that wouldn't explain why the database is also slow from the management tools and when used through other applications with a direct connection...
The table is in PRIMARY filegroup. Could you move this table to a different filegroup or even that is constrained? If you can, you should move it to a different filegroup with its own physical file. That should help a lot. Check out how create new filegroup and move the object to a new file group.
Given your constraints you could try zipping the XML before inserting into the DB as binary. This should significantly reduce the storage cost of this data.
You mention this is bad for performance, how often are you reading from this snapshot table? If this is just stored it should only effect performance when writing. If you are often reading this are you sure the performance issue is with the datastoreage not the parsing of 10MB of XML?
The whole system became a lot faster when I replaced the TEXT datatype with the NVARCHAR(MAX) datatype. HLGEM pointed out to me that the TEXT datatype is outdated, thus troublesome. It's still a question if the datatype of these columns could be replaced this easy with the more modern datatype, though. (Translated: I need to test if the code will work with the altered datatype...)
So, if i would alter the datatype from TEXT to NVARCHAR(MAX), is there anything that would break because of this? Problems that I can expect?
Right now, this seems to solve the problem but I need to do some lobbying before I'm allowed to make this change. So I need to be real sure it won't cause any (unexpected) problems.