I am trying to copy data from a cosmosdb container to an Azure SQL database table using Azure Data Factory.
Some of my columns in cosmosdb are not mandatory and might not be defined. The issue is that for every of these columns, I get the following error when running the copy activity :
Data type of column 'MyProperty' can't be inferred from 1st row of data, please specify its data type in mappings of copy activity or structure of DataSet.
However I checked in the mapping tab and the types of these properties are correctly infered to string, and they are well nullable in my SQL stored procedure table type.
I also have the same problem for optional decimal properties where the errors says that the value can't be parsed to Int64, though the infered type in the mapping tab is set to number and not integer...
Here is the mapping I currently have :
And the stored procedure with the table type
CREATE TYPE [dbo].[MyTableType] AS TABLE
(
[Id] varchar(256) NOT NULL PRIMARY KEY,
[SupplierId] varchar(256) NOT NULL,
[SupplierClientId] varchar(256) NULL,
[BuyerId] varchar(256) null
)
CREATE PROCEDURE [dbo].[UpsertItems]
#itemsTable MyTableType readonly
AS
BEGIN
MERGE MyTable AS target
USING #itemsTable AS source
ON (target.Id = source.Id)
WHEN MATCHED THEN
UPDATE SET
SupplierId = source.SupplierId,
SupplierClientId = source.[SupplierClientId],
BuyerId = source.[BuyerId]
WHEN NOT MATCHED THEN
INSERT (Id, SupplierId, SupplierClientId, BuyerId)
VALUES (
source.Id,
source.SupplierId,
source.[SupplierClientId],
source.[BuyerId]);
END
I can't find a way to force the datatype of this property either in the dataset directly of in the mapping tab of the copy activity. How can I fix the issue ?
You can specify the data types if you are creating a new table while copy. But if you are adding or copying to an already existing table, the data type is inferred automatically.
While, auto creating table in copying
If this does not help, please share sample source data and some snips
Related
Is there any way I can find labels which are not used in D365 FO (labels which dont have references)?
The cross references are stored in database DYNAMICSXREFDB. You can use a sql query to generate a list of labels that have no references.
This query uses two tables in the database:
Names holds an entry for each object in the application that can be referenced.
The Path field of the table holds the name of the object (e.g. /Labels/#FormRunConfiguration:ViewDefaultLabel is the path of the ViewDefaultLabel in the FormRunConfiguration label file.
Field Id is used to reference a record in this table in other tables.
References holds the actual references that connect the objects.
Field SourceId contains the Id of the Names record of the object that references another object identified by field TargetId.
The actual query could look like this:
SELECT LabelObjects.Path AS UnusedLabel
FROM [dbo].[Names] AS LabelObjects
WHERE LabelObjects.Path LIKE '/Labels/%'
AND NOT EXISTS
(SELECT *
FROM [dbo].[References] AS LabelReferences
WHERE LabelReferences.TargetId = LabelObjects.Id)
Make sure to compile the application to update the cross reference data. Otherwise the query might give you wrong results. When I run this query on a version 10.0.3 PU27 environment, it returns one standard label as a result.
I am trying to create a table in SQLite with blob type column (Content):
create table [Files]
(
Id int identity not null
constraint PK_File_Id primary key,
MimeType nvarchar (400) not null,
Content varbinary (max) null
constraint DF_File_Content default (0x),
);
However the following is not being accepted:
Content varbinary (max) null
constraint DF_File_Content default (0x),
Why?
"Max" is the name of a standard SQLite3 function, so is not available as part of a type name.
See the syntax reference for the CREATE TABLE statement and data types. A type name can include numbers in parentheses (which are ignored), but not the word “MAX”.
It looks like you're trying to use MS SQL Server syntax, and there are several errors in your code:
As mentioned above, (max) is not accepted as part of a type name. Since value lengths are unconstrained by default, simply omit it.
varbinary gives the column “numeric affinity”. While such a column can store a blob, you'll probably want to declare it as blob instead.
0x is not a valid blob literal. The correct way to write an empty blob is x''.
identity is called autoincrement. And in order to use it, the type name must be integer rather than int. The not null is redundant: If you try to insert a null value into such a column, you'll get the auto-incremented ROWID instead.
Note: If you simply need Id to have unique values at any given time and don't care if previously-deleted values get re-used, then you can simply declare the column as integer primary key, and inserting null will fill in the column with an unused integer. The autoincrement keyword prevents the re-use of ROWIDs over the lifetime of the database, more closely matching the semantics of MS SQL's identity keyword. See the discussion at the link above.
While it's syntactically legal to declare a name for a column constraint, it's not necessary.
Try this:
create table [Files]
(
Id integer primary key autoincrement,
MimeType nvarchar (400) not null,
Content blob null default (x'')
);
Note that this does not enforce a length limit on a MimeType column. If you need to, add the constraint check (length(MimeType) <= 400).
I imported a SQLite DB into MS-Access using ODBC and it was successful except for one column. One of the columns in SQLite DB was of 'Integer' datatype but it contained a string since datatypes are flexible in SQLite. But in the imported MS-Access DB, the column corresponding to that particular datatype is blank. I assume that the Access DB was expecting a integer datatype and not accepting a string. The following is the table structure:
CREATE TABLE test
(
num INTEGER,
name TEXT,
script INTEGER NOT NULL
);
I have problem with the column 'script'. I am not allowed to change the datatype here so is there any solution that I can workout in Access?
You can create a SQLite VIEW named "forimport" to CAST the troublesome column as TEXT
CREATE VIEW forimport AS SELECT num, name, CAST(script AS TEXT) AS script FROM test;
and then import the view (instead of the table) from SQLite into Access
I 'm working on an ASP.NET Dynamic Data Entities Web Application.
When trying to insert a new entry in one of the tables, with some value in the column_name field, I get the following message :
Cannot insert the value NULL into column 'column_name', table 'DATABASE.dbo.table_name'; column does not allow nulls. INSERT fails.
The column properties are :
Entity Key : True
Nullable : False
Type : String
I believe Dynamic Data is trying to send null value to entity framework for some reason, but I don't know which.
Do you know why Dynamic Data is behaving that way ?
Or have you any idea how to debug the insert process ?
Thanks
I found where the problem come from. A table from the database the model is based contains a trigger. For some reason the model makes an association between the two tables involved in the trigger.
Within Teradata, when executing an ALTER TABLE command to modify the data type for an existing column from VARCHAR(10) to CHAR(10), I receive a 3558 error indicating that the specified attribute can not be altered. Is there an alternate method of coding this to achieve the desired objective or does the column need to be dropped and re-created in order to change the data type?
You can't modify the data type when the internal storage changes and this is the case for VARCHAR <-> CHAR.
Instead of ADD CHAR -> UPDATE CHAR from VARCHAR (needs a huge Transient Journal) -> DROP VARCHAR you better create a new table -> INSERT/SELECT (no TJ) -> DROP/RENAME.
Edit: As Rob Paller suggested, using MERGE INTO instead of INSERT SELECT will avoid spooling the source table.