I have an ASP.NET app that uses a SQL Server database. I now need to pull data from Sybase ASE into that SQL Server database for my app to consume, and I'm not having any success with my ideas.
Has anyone done this? Any ideas/suggestions/tips?
You can configure a linked server from SQL Server to Sybase. It should be fairly vanilla using the Sybase provider on the MS side.
Okay, I've finally (through lame trial and error) found out how to link my Sybase ASE (12.5) server to my SQL Server (2008) which will allow the integration I want. Here's roughly how I did it:
Logged in to Sybase ASE OLE DB Configuration Manager (this is like the Sybase version of Windows' ODBC Data Sources) and added an OLE DB data source. I believe you must be an admin on the PC to do this.
In SQL Server 2008 Management Studio, went to Server Objects > Linked Servers. Right click and select "New Linked Server".
In the Linked Server Properties, I set the following properties:
General:
--Linked server: the name of your linked server as you want it to appear in your linked server list
--Provider: Select Sybase ASE OLE DB Provider from the dropdown list.
--Product name: The exact name of the OLD DB data source you just created in Sybase ASE OLE DB Configuration Manager.
--Data source: Same as Product name.
--Provider string: I left this blank
--Location: I left this blank
--Catalog: The default database (master or whatever) to log on to.
Security:
--You need to map a valid SQL Server logon to a valid Sybase logon. I did not use impersonation (which does a credentials pass-thru).
--I chose my connection Be made without using a security context.
Server Options:
--All the defaults worked for me.
Throughout, the standard SQL Server help worked fairly well as a guide. Though not always true, F1 was my friend here.
I can now do distributed queries, DTS or SSIS packages, and use SSRS. This takes a lot of the suck out of Sybase ASE.
Of course the above can be done via the command line using sp_linkserver, but the GUI is more comfortable for a lowly dev like me.
Use Management Studio or Enterprise Manager to import the data using the data importation wizard. That should be it, just make sure you pick the right data provider in the wizard and you should be good to go.
If you want this to be a live feed create a small windows service to manage the exchange of information. It should be relatively simple to do, just a little bit of leg work on your end. If you are adverse to that there are plenty of off the shelf solutions that can do this for you.
The question is a little vague on specifics:
Is this a one time conversion or part of a repeated process.
Is the source machine "reachable" from your destination machine (can you connect the two or do you need to read in files)
With most conversions there are two parts:
Physically getting data from the source into the destination.
Mapping data from the source to the destination tables.
It is hard to make any recommendations without more info. What would be fine for a one time conversion would not work if you need to read in data all day every day. Also, if the source database can not be connected to and you have to pass files, they methods change.
Related
We need to connect to on premise Teradata from Azure Databricks .
Is that possible at all ?
If yes please let me know how .
I was looking for this information as well and I recently was able to access our Teradata instance from Databricks. Here is how I was able to do it.
Step 1. Check your cloud connectivity.
%sh nc -vz 'jdbcHostname' 'jdbcPort'
- 'jdbcHostName' is your Teradata server.
- 'jdbcPort' is your Teradata server listening port. By default, Teradata listens to the TCP port 1025
Also check out Databrick’s best practice on connecting to another infrastructure.
Step 2. Install Teradata JDBC driver.
Teradata Downloads page provides JDBC drivers by version and archive type. You can also check the Teradata JDBC Driver Supported Platforms page to make sure you pick the right version of the driver.
Databricks offers multiple ways to install a JDBC library JAR for databases whose drivers are not available in Databricks. Please refer to the Databricks Libraries to learn more and pick the one that is right for you.
Once installed, you should see it listed in the Cluster details page under the Libraries tab.
Terajdbc4.jar dbfs:/workspace/libs/terajdbc4.jar
Step 3. Connect to Teradata from Databricks.
You can define some variables to let us programmatically create these connections. Since my instance required LDAP, I added LOGMECH=LDAP in the URL. Without LOGMECH=LDAP it returns “username or password invalid” error message.
(Replace the text in italic to the values in your environment)
driver = “com.teradata.jdbc.TeraDriver”
url = “jdbc:teradata://Teradata_database_server/Database=Teradata_database_name,LOGMECH=LDAP”
table = “Teradata_schema.Teradata_tablename_or_viewname”
user = “your_username”
password = “your_password”
Now that the connection variables are specified, you can create a DataFrame. You can also explicitly set this to a particular schema if you have one already. Please refer to Spark SQL Guide for more information.
Now, let’s create a DataFrame in Python.
My_remote_table = spark.read.format(“jdbc”)\
.option(“driver”, driver)\
.option(“url”, url)\
.option(“dbtable”, table)\
.option(“user”, user)\
.option(“password”, password)\
.load()
Now that the DataFrame is created, it can be queried. For instance, you can select some particular columns to select and display within Databricks.
display(My_remote_table.select(“EXAMPLE_COLUMN”))
Step 4. Create a temporary view or a permanent table.
My_remote_table.createOrReplaceTempView(“YOUR_TEMP_VIEW_NAME”)
or
My_remote_table.write.format(“parquet”).saveAsTable(“MY_PERMANENT_TABLE_NAME”)
Step 3 and 4 can also be combined if the intention is to simply create a table in Databricks from Teradata. Check out the Databricks documentation SQL Databases Using JDBC for other options.
Here is a link to the write-up I published on this topic.
Accessing Teradata from Databricks for Rapid Experimentation in Data Science and Analytics Projects
If you create a virtual network that can connect to on prem then you can deploy your databricks instance into that vnet. See https://docs.azuredatabricks.net/administration-guide/cloud-configurations/azure/vnet-inject.html.
I assume that there is a spark connector for terradata. I haven't used it myself but I'm sure one exists.
You can't. If you run Azure Databricks, all the data needs to be stored in Azure. But you can call the data using REST API from Teradata and then save data in Azure.
How to Synchronize local database(datas) value to Server(Hosting) database values in SQL server 2008 R2? ex:from our client having a pc wen they are enterting entry it will insert kay ah...but in our concern keeping backup from server Hosting i mean IBM server...suppose client
connecting internet connection means clicking single button event want to transfer OUR own server database also..can u got it
Is there such any option there? Please let me know.
Thank you in advance!
I suggest that use Redgate Data compare tools in order to synchronize data from one database to another database.
You can also use some query such as following query in order to determine deferent record in two database and synch them
USE Datbase1
INSERT INTO Schema1.Table1 (columns)
SELECT t1.Columns
FROM Datbase2.Schema2.Table2 t1
LEFT JOIN Schema1.Table1 t2 ON t1.keyColumn = t2.Keycolumn
WHERE t2.keycolumn IS NULL
The RedGate will be costly for you. Use can use Open Source Database compare and sync tool as
Open DBDiff
I have used it for my Live Databases compare with Local Database. Still today I have not found any issue.
Hope It will help you.
Open DBDiff is an open source database schema comparison tool for SQL Server 2005/2008.
It reports differences between two database schemas and provides a synchronization script to upgrade a database from one to the other.
Apparently, using AttachDbFilename and user instance in your connection string is a bad way to connect to a DB. I'm using SQL server express on my local machine and it all seems to work fine. But what's the proper way to connect to SQL server then?
Thanks for your explanation.
Using User Instance means that SQL Server is creating a special copy of that database file for use by your program. If you have two different programs using that same connection string, they get two entirely different copies of the database. This leads to a lot of confusion, as people will test updating data with their program, then connect to a different copy of their database in Management Studio, and complain that their update isn't working. This sends them through a flawed series of wild goose chase steps trying to troubleshoot the wrong problem.
This article goes into more depth about how to use this feature, but heed the very first note: the User Instance feature has been deprecated. In SQL Server 2012, the preferred alternatives are (in this order, IMHO):
Create or attach your database to a real instance of SQL Server. Your connection string will then just need to specify the instance name, the database name, and credentials. There will be no mixup as Management Studio, Visual Studio and your program(s) will all be connecting to a single copy of the database.
Use a container for local development. Here's a great starter video by Anna Hoffman and Anthony Nocentino, and I have some other resources here, here, and here. If you're on an M1 Mac, you won't be able to use a full-blown SQL Server instance, but you can use Azure SQL Edge if you can get by with most SQL Server functionality (the omissions are enumerated here).
Use SqlLocalDb for local development. I believe I pointed you to this article yesterday: "Getting Started with SQL Server 2012 Express LocalDB."
Use SQL Server Compact. I like this option the least because the functionality and syntax is not the same - so it's not necessarily going to provide you with all the functionality you're ultimately going to want to deploy. Compact Edition is also deprecated, so there's that.
Of course if you are using a version < SQL Server 2012, SqlLocalDb is not an option - so you should be creating a real database and using that consistently. I only mention the Compact option for completeness - I think that can be almost as bad an idea as using AttachDbFileName.
EDIT: I've blogged about this here:
Bad Habits : Using AttachDBFileName
In case someone had the problem.
When attaching the database with a connection string containing AttachDBFile
with SQLEXPRESS, I noticed this connection was exclusive to the ASP.NET application that was using the database. The connection did block the access to all other processes on the file level when made with System.Data.SqlClient as provider.
In order to assure the connection to be shareable with other processes
instead use DataBase to specify the database name in your connection string
Example or connection string :
Data Source=.\SQLEXPRESS;DataBase=PlaCliGen;User ID=XXX;password=ZZZ; Connect Timeout=30
,where PlaCliGen is the name (or logical name) by which SQLEXPRESS server knows the database.
By connecting to the data base with AttachDBFile giving the path to the .mdf file
(namely : replacing DataBase = PlacliGen by AttachDBFile = c:\vs\placligen\app_data\placligen.mdf) the File was connected exclusively and no other process could connect to the database.
I kind of new to SQL Server, I always used access db for my sites.
I created a SQL Server on my local computer and now I want to take this db and transfer it to the server. In access all I had to do is, take the mdb file and put it on the server and change the connection string. How can I transfer the SQL Server db to the server?
Is there any file to put on the server ?
Also the connection string isn't a folder but a local computer like this:
Data Source=my-PC;Initial Catalog=storeSQL1;User ID='my-PC\com';Password='';Trusted_Connection=YES;
Who can provide me this connection string for the server (the hosting company) ?
The easiest way would probably be to create a backup of the database on your local machine, then restore that backup on the new server.
Roadmap is:
Do simple backup-restore to move user databases to target server.
Create script on source server, that can recover permissions and login-users pairing
Restore the CLR and TRUSTWORTHY security for databases, that using unsafe assemblies, simpliest way is (in proper DB):
exec sp_changedbowner 'sa' --sa just for example
ALTER DATABASE dbname SET TRUSTWORTHY ON
Enjoy
Depending on your version of SQL Server here is a good article that outlines all the ways to move a SQL Server Database.
http://blogs.msdn.com/b/sreekarm/archive/2009/09/11/move-a-database-from-one-server-to-another-server-in-sql-server-2008.aspx
As for getting the connection string yes the hosting company would provide you with that. Where is the database hosted, you could check their knowledge base articles or if it's an in house data base I'm sure a dba could provide you with that information. It won't change much from what you have but it will change.
I'm not sure what tools your using, but to start you need to do a dump or backup of your current database on your machine. After you do that then you can do and import which should create all the tables and import any data you have.
After the data exists on the server then as far as the connection string, you just need to say the Data Source is the server ip address or host name and change your User ID and Pass to match that server.
If you need more details on any part of this process, post what tools your using and what your environment looks like and I would be more than happy to assist you.
In my opinion the best way to do that is to detach the db from one server(pc), copy the files to the second one and then attach them on the second server/pc.
To detach:
USE master;
GO
EXEC sp_detach_db #dbname = N'AdventureWorks2008R2';
GO
To attach:
USE master;
GO
CREATE DATABASE MyAdventureWorks
ON (FILENAME = 'C:\MySQLServer\AdventureWorks2008R2_Data.mdf'),
(FILENAME = 'C:\MySQLServer\AdventureWorks2008R2_Log.ldf')
FOR ATTACH;
GO
I have a Sql Server 2008 Express database file that's currently being used by an ASP.NET application, and I'm not sure how to query the database without taking the website down.
I'm unable to copy the database files (.mdf and .ldf files) to another directory, since they're in use by the web server. Also, if I attach the databases to an instance of the sql server (using the 'Create Database [DB name] on (filename = '[DB filename.mdf]') for attach;' command at the sqlcmd prompt), then the application pool user becomes unable to access the database (i.e. the webpages start producing http 500 errors. I think this might have to do with the username for the application pool becoming somehow divorced from the login credentials in the sql server database).
Any suggestions? I realize this is probably a newbie question, since it seems like a rather fundamental task. However, due to my inexperience, I really don't know what the answer is, and I'm pretty stumped at this point, since I've tried a couple of different things.
Thanks!
Andrew
if I attach the databases to an instance of the sql server (using the 'Create Database [DB name] on (filename = '[DB filename.mdf]') for attach;' command at the sqlcmd prompt),
Don't do this to a live database - it's attempting to be setup an MDF to be written to by two different databases...
Use Backup/Restore
As you've found, Attach/ReAttach requires the database to be offline - use the Backup/Restore functionality:
MSDN: Using SSMS to Backup the Database
MSDN: Using SSMs to Restore the Backup
Be aware that the backup/restore doesn't maintain logins (& jobs if you have any associated with the database) - you'll have to recreate & sync if using an account other than those with uber access.
Maybe Linked Server would work?
Another alternative would be to setup another SQL Server Express/etc instance on a different box, and use the Linked Server functionality to create a connection to the live/prod data. Use a different account than the one used for the ASP application...