I am using pyodbc, ODBC with freetds to access a remote MSSQL database. I recently migrated from Centos + Python 2.4 to Ubuntu + Python 2.7 and the two are behaving differently. My freetds.conf is:
[global]
# TDS protocol version
tds version = 4.2
# If you get out-of-memory errors, it may mean that your client
# is trying to allocate a huge buffer for a TEXT field.
# Try setting 'text size' to a more reasonable limit
text size = 64512
[server]
host = server
port = 1333
tds version = 8.0
client charset = UTF-8
My Python code is (simplified):
import pyodbc
msConn = pyodbc.connect('DSN=%s;UID=%s;PWD=%s' % (self.dsn, self.user, self.passwd))
msCur = msConn.cursor()
msQuery='''select ...'''
msCur.execute(msQuery % startUpdateStamp)
for row in msRows:
print rows
When I execute the code above on my Centos server (with python 2.4), any special characters are encoded in utf-8. However when executing the same code with same configuration on the Ubuntu server (python 2.7), the special characters are in unicode. It is as if the configuration client charset = UTF-8 is being completely ignored.
However I've tried setting client charset to other values, in return the server does not allow the connection. Which means the client charset setting is getting through. I have also tried changing the tds version = 8.0 to no avail.
I've also looked here, but the answer does not explain why given the same database, two systems give two encodings.
How can I have the Ubuntu return UTF-8 as well?
I'm not proficient at MSSQL or ODBC. Any help is greatly appreciated.
Related
Sorry if this is a newbie question. I have been searching and can't seem to figure this out.
I am trying to move a PHP webapp from Plesk, to RunCloud. Both are hosted on a Digital Ocean server.
I exported the database from Plesk, and imported it into RunCloud. After updating the config, it only partially works. Some of the tasks read data, a few seem to write, but a majority just break it.
The strange thing is, if I point the config file to the Plesk server, the app works great using it as a remote database.
Here is from the PHPMyAdmin. The only thing that looks fishy is UNIX, vs TCP. But I cannot find a way to change RunCloud to UNIX.
I would copy the errors, but they are across the board. Any ideas? Thank you so much!
PHPMyAdmin Plesk Server:
Server: Localhost via UNIX socket
Server type: MariaDB
Server connection: SSL is not being used Documentation
Server version: 10.1.48-MariaDB-0ubuntu0.18.04.1 - Ubuntu 18.04
Protocol version: 10
User: prod#localhost
Server charset: UTF-8 Unicode (utf8mb4)
From RunCloud Server:
Server: 127.0.0.1 via TCP/IP
Server type: MariaDB
Server connection: SSL is not being used Documentation
Server version: 10.4.21-MariaDB-1:10.4.21+maria~focal-log - mariadb.org binary distribution
Protocol version: 10
User: prod#localhost
Server charset: UTF-8 Unicode (utf8)
As a newbie, I learned that 10.1 and 10.4 were rather different. I then went through each error to see if I could find the change on the MariaDB update guide (which is great). Somehow, I came across this post: error code 1292 incorrect date value mysql. It was the zero date. I just added the line in my.cnf that it suggested. Fixed!!!
I observe a strange situation in Windows 10 with XAMPP Control Panel v3.2.4
Terminal window is launched and connected to Mysql DB.
Note: prior in terminal window issued command 'chcp 65001' to support UTF8 encoding.
Now when I attempt to update some table with value which is in Cyrillic then MySQL complains about not closed quote symbol. If I replace Cyrillic input to English then command is accepted.
MariaDB [youtube]> update episodes set name='Катя' where id=11;
'>
If I attempt to insert a new record into DB same situation happens
MariaDB [youtube]> insert into episodes (youtube_id,series_id,season,episode,name) values (12345678904,1,0,1,'Катя');
'>
If double quotes are used situation is the same
MariaDB [youtube]> insert into episodes (youtube_id,series_id,season,episode,title) values (12345678904,1,0,1,"Катя");
">
What a magic touch required to make it work through terminal window?
Update:
John suggested to look into configuration file of MariaDB for UTF8 settings.
The settings was changed to the following and the problem still persists
# The MySQL server
default-character-set=utf8mb4
[mysqld]
init-connect=\'SET NAMES utf8\'
character_set_server=utf8
collation_server=utf8_unicode_ci
skip-character-set-client-handshake
character_sets-dir="C:/bin/XAMPP/App/xampp/mysql/share/charsets"
Initially settings was
# The MySQL server
default-character-set=utf8mb4
[mysqld]
character-set-server=utf8mb4
collation-server=utf8mb4_general_ci
Server status report
MariaDB [youtube]> \s
--------------
mysql Ver 15.1 Distrib 10.4.10-MariaDB, for Win64 (AMD64), source revision c24ec3cece6d8bf70dac7519b6fd397c464f7a82
Connection id: 17
Current database: youtube
Current user: root#localhost
SSL: Not in use
Using delimiter: ;
Server: MariaDB
Server version: 10.4.10-MariaDB mariadb.org binary distribution
Protocol version: 10
Connection: localhost via TCP/IP
Server characterset: utf8
Db characterset: utf8mb4
Client characterset: utf8mb4
Conn. characterset: utf8mb4
TCP port: 3306
Uptime: 11 min 12 sec
Threads: 7 Questions: 59 Slow queries: 0 Opens: 22 Flush tables: 1 Open tables: 16 Queries per second avg: 0.087
--------------
MariaDB Documentation has reference to an option --default-character-set=name.
An attempt to use --default-character-set=utf8mb4 on command line had no effect on behavior of insert/update record in terminal client.
mysql -u root -p --default-character-set=utf8mb4 youtube
....
MariaDB [youtube]> update episodes set title='Катя' where id=11;
'>
I highly recommend getting a copy of the freeware program HeidiSQL. It's not perfect and even crashes occasionally though compared to everything else I've worked with? Oh boy, totally worth my time.
Secondly you want to make sure that you're using the following:
Database Character Set: utf8mb4
Database Collation: utf8mb4_unicode_520_ci
These have the greatest UTF-8 support from what I've read from Stackoverflow member Rick James, via his website. He's a database superstar here on SO so if you ever hire him dump buckets of money on his face. His site has a comparison chart. In fact it's been a while and 520 might have been superseded since I last checked.
To set/change the Database Character Set you will need to change the my.cnf configuration file for MariaDB, I recommend using Notepad++ for code editing. This should make any newly created databases use the correct encoding however you may have to go through and manually update the character sets and collations for databases, tables and table columns so do not forget to be thorough!
[client]
default-character-set = utf8mb4
[mysqld]
character-set-server = utf8mb4
collation-server = utf8mb4_unicode_520_ci
Once you do that your queries with Russian/Greek/etc should work normally as they do with English. Keyword: should.
Since I only speak two languages (English and bad English) I encode all characters past a certain Unicode numeric point just to be certain. It'll take up a bit more space however there are sometimes characters added for languages to Unicode after the majority of the language has been defined thus potentially fragmenting language support. If you're interested comment and I'll go find that code for you.
I'm at a moderate level of comprehension and I'm no Rick James though I have about two or three dozen translation pages (use the search feature and search for 'translation') on the site in my profile if you want to see the output. After I did these things I stopped having the data get corrupted. I hope this helps!
I've been trying to establish a connection to an MS Access database I have on my local hard drive using FreeTDS and UnixODBC. My ultimate goal is to open the connection in R via RODBC and implement some SQL scripts developed for this specific database to extract data. I've followed advice from this page (How do I install RODBC on Mac OS X Yosemite with unixodbc and freetds?), but am still having trouble.
When I implement isql in terminal I get the following error message.
[S1000][unixODBC][FreeTDS][SQL Server]Unable to connect to data source
[01000][unixODBC][FreeTDS][SQL Server]Unknown host machine name.
[ISQL]ERROR: Could not SQLConnect
I'm assuming my error is in how I've identified the host in my various setup files, which are as follows.
freetds.conf
[global]
; tds version = 8.0
; dump file = /tmp/freetds.log
; debug flags = 0xffff
; timeout = 10
; connect timeout = 10
text size = 64512
[my_db]
# insert the actual host below
host = My_computer_name.local
port = 1433
tds version = 8.0
odbc.ini
[my_db]
Driver = MSSQL
Servername = My_computer_name.local
Port = 1433
Database = /filepath_to_db/my_db.mdb
TDS_Version = 8.0
odicinst.ini
[MSSQL]
Description = Microsoft SQL Server driver
Driver = /usr/local/Cellar/freetds/1.00.39/lib/libtdsodbc.so
Setup = /usr/local/Cellar/freetds/1.00.39/lib/libtdsodbc.so
FreeTDS is for connecting to Microsoft SQL Server and Sybase databases. It is not designed to work with Microsoft Access databases.
I have a python script running on python 2.7 in CentOS 2.6 that connects to a Sql erver database:
pyodbc.connect("DRIVER=FreeTDS;SERVER=someServer;DATABASE=someDb;UID=myUser;PWD=superSecret;CHARSET=UTF8;TDS_Version=7.2")
That call will fail with the following: pyodbc.Error: ('08001', '[08001] [unixODBC][FreeTDS][SQL Server]Unable to connect to data source (0) (SQLDriverConnect)')
The freetds trace will say login.c:436:invalid port number
Adding PORT=1433; will cause the connection to succeed even though that is the default port number, and I added the following to my freetds.conf:
[global]
# TDS protocol version
tds version = 7.0
port = 1433
How do I make FreeTDS try port 1433 as the default port so I don't have to set it in the query string?
Instead of this, as you noted:
pyodbc.connect("DRIVER=FreeTDS;SERVER=someServer;DATABASE=someDb;UID=myUser;PWD=superSecret;CHARSET=UTF8;TDS_Version=7.2")
...you could put these values in your connection string, like this:
pyodbc.connect("DRIVER={FreeTDS};SERVER=someServer;PORT=1433;DATABASE=someDb;UID=myUser;PWD=superSecret;TDS_Version=7.2;CHARSET=UTF8")
For your freetds.conf configuration:
[global]
client charset = UTF-8
tds version = 7.2
[someServer]
host = someServer
port = 1433
tds version = 7.2
Both ways have always worked for me.
On a side note, I've started using this driver, made by Microsoft specifically for RedHat and CentOS (although I don't know if it will work on a version as old as yours):
https://msdn.microsoft.com/en-us/library/hh568451%28v=sql.110%29.aspx
Best of luck!
Can it be done: Reading data stored in an MS Access (.accdb) database, from within Shiny apps running on Ubuntu Shiny server?
We have no knowledge of SQL Server Express. We have our data organized in simple MS Access databases, and want to deploy our Shiny apps (who visualize this data) on an Ubuntu Shiny server.
It all works on our local Windows machines, but how to make it also work with an Ubuntu Shiny server?
I understand that with our minimal knowledge of database systems, it is not straightforward to go porting our databases to SQL Server Express.
Thanks in advance for your expertise!
I had a bit of a job setting this up myself. I had to take info from several sources to get all the required packages – the following is a list of good info sources:
http://guywyant.info/log/206/connecting-to-ms-sql-server-from-ubuntu/
http://driftharmony.wordpress.com/2008/08/15/connecting-ubuntu-804-to-microsoft-sql-server/
https://code.google.com/p/django-pyodbc/wiki/FreeTDS
FreeTDS working, but ODBC cannot connect
The 3 files were ultimately configured thus:
Detail of file: /etc/odbc.ini
[NameThis]
Driver = FreeTDS
TDS_Version=8.0
Servername = YourServer
Port = 1433
Database = testing
Trace = No
Detail of file: /etc/odbcinst.ini
[FreeTDS]
Description = FreeTDS
Driver = /usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Detail of file: /etc/freetds/freetds.conf
# $Id: freetds.conf,v 1.12 2007/12/25 06:02:36 jklowden Exp $
# This file is installed by FreeTDS if no file by the same name is found in the installation directory.
# For information about the layout of this file and its settings, see the freetds.conf manpage "man freetds.conf".
# Global settings are overridden by those in a database server specific section
[global]
# TDS protocol version
; tds version = 4.2
# Whether to write a TDSDUMP file for diagnostic purposes
# (setting this to /tmp is insecure on a multi-user system)
; dump file = /tmp/freetds.log
; debug flags = 0xffff
# Command and connection timeouts
; timeout = 10
; connect timeout = 10
# If you get out-of-memory errors, it may mean that your client
# is trying to allocate a huge buffer for a TEXT field. Try setting 'text size' to a more reasonable limit
text size = 64512
# Test Kx
[NameThis]
host = YOUR IP
port = 1433
tds version = 7.2