I am trying to store binary data in a sqlite database using the Twisted adbapi. However, when I run a query to store the data, I get an error:
sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.
After googling a bit, I found the answer for a normal sqlite connection:
con = sqlite3.connect(...)
con.text_factory = str
However, I can't find an equivalent setting to use with a twisted adbapi sqlite connection:
dbpool = adbapi.ConnectionPool("sqlite3", "data.db", check_same_thread=False)
I would appreciate any help!
I figured it out. In order to make changes to the connection after it opens, you have to use the cp_openfun parameter for the ConnectionPool. The following code worked:
def set_text_factory(conn):
conn.text_factory = str
dbpool = adbapi.ConnectionPool("sqlite3", "data.db", check_same_thread=False,
cp_openfun=set_text_factory)
Related
it is appearing in some big modules like matplotlib. For example expression :
import importlib
obj = importlib.import_module('matplotlib')
obj_entries = obj.__dict__
Between runs len of obj_entries can vary. From 108 to 157 (expected) entries. Especially pyplot can be ignored like some another submodules.
it can work stable during manual debug mode with len computing statement after dict extraction. But in auto it dont work well.
such error occures:
RuntimeError: dictionary changed size during iteration
python-BaseException
using clear python 3.10 on windows. Version swap change nothing at all
during some attempts some interesting features was found.
use of repr is helpfull before dict initiation.
But if module transported between classes like variable more likely lazy-import happening? For now there is evidence that not all names showing when command line interpriter doing opposite - returning what expected. So this junk of code help bypass this bechavior...
Note: using pkgutil.iter_modules(some_path) to observe modules im internal for pkgutil ModuleInfo form.
import pkgutil, importlib
module_info : pkgutil.ModuleInfo
name = module_info.name
founder = module_info.module_finder
spec = founder.find_spec(name)
module_obj = importlib.util.module_from_spec(spec)
loader = module_obj.__loader__
loader.exec_module(module_obj)
still unfamilliar with interior of import mechanics so it will be helpfull to recive some links to more detail explanation (spot on)
With Free Pascal 3.0.4, this test program correctly writes ÄÖÜ
program FPCTest;
uses IdURI;
begin
WriteLn(TIdURI.URLDecode('%C3%84%C3%96%C3%9C'));
ReadLn;
end.
However if the unit LazUTF8 (as described here) is used, it writes ???
program FPCTest;
uses IdURI, LazUTF8;
begin
WriteLn(TIdURI.URLDecode('%C3%84%C3%96%C3%9C'));
ReadLn;
end.
How can I fix this decoding error for programs which use LazUTF8?
When the String type is an alias for AnsiString 1, much of Indy's functionality exposes extra parameters/properties to let users control which ANSI encodings are used when AnsiString values are passed around in operations that perform AnsiString<->byte conversions.
1: Delphi pre-2009, and FreePascal/Lazarus when {$ModeSwitch UnicodeStrings} and {$Mode DelphiUnicode} are not used (FYI, Indy 11 will use them!).
In most cases, Indy's default byte encoding is ASCII (because many of the Internet protocols that Indy implements originally supported only ASCII - individual Indy components upgrade themselves to UTF as appropriate per protocol), though some things use the OS default codepage/charset instead.
Indy's default byte encoding can be changed at runtime by setting the global GIdDefaultTextEncoding variable in the IdGlobal unit, eg:
GIdDefaultTextEncoding := encUTF8;
But, in this particular situation, TIdURI.URLEncode() does not use GIdDefaultTextEncoding, but it does have an optional ADestEncoding parameter that you can use to specify a specific byte encoding for the returned AnsiString (in addition to an optional AByteEncoding parameter to specify the byte encoding of the parsed url octets - UTF-8 by default), eg:
TIdURI.URLDecode('%C3%84%C3%96%C3%9C'
{$IFNDEF FPC_UNICODESTRINGS}, IndyTextEncoding_UTF8, IndyTextEncoding_UTF8{$ENDIF}
)
The above will parse the url-encoded octets as UTF-8, and then return that data as-is in a UTF-8 encoded AnsiString.
If you do not specify an output encoding for ADestEncoding, URLDecode() defaults to the OS default. If you want it to use GIdDefaultTextEncoding instead, specify IndyTextEncoding_Default in the ADestEncoding parameter:
TIdURI.URLDecode('%C3%84%C3%96%C3%9C'
{$IFNDEF FPC_UNICODESTRINGS}, IndyTextEncoding_UTF8, IndyTextEncoding_Default{$ENDIF}
)
Another option would be to use the IndyTextEncoding(CodePage) function for ADestEncoding, passing it FreePascal's DefaultSystemCodePage variable, which the LazUtils package sets to CP_UTF8 2:
TIdURI.URLDecode('%C3%84%C3%96%C3%9C'
{$IFNDEF FPC_UNICODESTRINGS}, IndyTextEncoding_UTF8, IndyTextEncoding(DefaultSystemCodePage){$ENDIF}
)
2: I have opened a ticket in Indy's issue tracker to add support for DefaultSystemCodePage when compiling for FreePascal/Lazarus.
With this change in TIdURI.URLDecode lines 386ff LazUTF8 can be used:
{$IFDEF FPC}
Result := string(AByteEncoding.GetString(LBytes));
{$ELSE}
{$IFDEF STRING_IS_ANSI}
EnsureEncoding(ADestEncoding, encOSDefault);
CheckByteEncoding(LBytes, AByteEncoding, ADestEncoding);
SetString(Result, PAnsiChar(LBytes), Length(LBytes));
{$ELSE}
Result := AByteEncoding.GetString(LBytes);
{$ENDIF}
{$ENDIF}
Notes
This change assumes that the LazUTF8 unit is used always, and the Indy source code change needs to be applied every time when a new version is used.
Also I found no way to fix the TIdURI.URLDecode in a way which works with and without LazUTF8.
Can someone point me out how to properly search using imaplib in python. The email server is Microsoft Exchange - seems to have problems but I would want a solution from the python/imaplib side.
https://github.com/barbushin/php-imap/issues/128
I so far use:
import imaplib
M = imaplib.IMAP4_SSL(host_name, port_name)
M.login(u, p)
M.select()
s_str = 'hello'
M.search(s_str)
And I get the following error:
>>> M.search(s_str)
('NO', [b'[BADCHARSET (US-ASCII)] The specified charset is not supported.'])
search takes two or more parameters, an encoding, and the search specifications. You can pass None as the encoding, to not specify one. hello is not a valid charset.
You also need to specify what you are searching: IMAP has a complex search language detailed in RFC3501§6.4.4; and imaplib does not provide a high level interface for it.
So, with both of those in mind, you need to do something like:
search(None, 'BODY', '"HELLO"')
or
search(None, 'FROM', '"HELLO"')
I am having troubles manipulating raw data. I am trying to change around a
resp_cookie in my ISAKMP header and when I do a sniff on the packet it is all in raw data format under Raw Load='\x00\x43\x01........... ' with about 3 lines like that. When I do a Wireshark capture I see the information I want to change but I cant seem to find a way to convert and change that raw data to find and replace the information I am looking for. Also, I can see the information I need when I do a hexdump(), but I can't store that in a variable. when I type i = hexdump(pkt) it spits out the hexdump but doesn't store the hexdump in i.
So this post is a little old, but I've come across it a dozen or so times trying to find the answer to a similar problem I'm having. I doubt OP has need for an answer anymore, but if anyone else is looking to do something similar...here you go!
I found the following code snippet somewhere in the deep, dark depths of google and it worked for my situation.
Hexdump(), show() and other methods of Scapy just output the packet to the terminal/console; they don't actually return a string or any other sort of object. So you need a way to intercept that data that it intends to write and put it in a variable to be manipulated.
NOTE: THIS IS PYTHON 3.X and SCAPY 3K
import io
import scapy
#generic scapy sniff
sniff(iface=interface,prn=parsePacket, filter=filter)
With the above sniff method, you're going to want to do the following.
def parsePacket(packet):
outputPacket = ''
#setup
qsave = sys.stdout
q = io.StringIO()
#CAPTURES OUTPUT
sys.stdout = q
#Text you're capturing
packet.show()
#restore original stdout
sys.stdout = qsave
#release output
sout = q.getvalue()
#Add to string (format if need be)
outputPacket += sout + '\n'
#Close IOStream
q.close()
#return your packet
return outputPacket
The string you return (outputPacket) can now be manipulated how you want.
Swap out .show() with whatever function you see fit.
P.S. Forgive me if this is a little rough from a Pythonic point of view...not a python dev by any stretch.
I am trying to encrypt a string using ColdFusion encrypt() with a 3rd party provided key like this:
encrypteded = encrypt('theString', 'FD52250E230D1CDFD5C2DF0D57E3E0FEFD52250E230D1CDF', 'DESEDE/CBC/NoPadding', 'BASE64', ToBase64('0'));
I get:
"The key specified is not a valid key for this encryption: Wrong key algorithm, expected DESede."
What do I have to do to this key in terms of encoding/decoding to get it into the right format?
Generally, when using provided keys from other languages, you have to do a little gymnastics on it to get it into Base64.
Try this for the key argument:
ToBase64(BinaryDecode('FD52250E230D1CDFD5C2DF0D57E3E0FEFD52250E230D1CDF','hex'))
But, to make this work for me, the input string needed to be a multiple of 8 bytes (because you're specifying NoPadding), and the IV needed to also be a multiple of 8 bytes.
So, this ended up working for me - not sure if you'll be able to decrypt it on the other end, tho, if the IV they're specifying is really what you've got listed there.
encrypteded = encrypt('theStrin', ToBase64(BinaryDecode('FD52250E230D1CDFD5C2DF0D57E3E0FEFD52250E230D1CDF','hex')), 'DESEDE/CBC/NoPadding', 'BASE64', ToBase64('0000'));
No IV also worked as well (with different output, obviously):
encrypteded = encrypt('theStrin', ToBase64(BinaryDecode('FD52250E230D1CDFD5C2DF0D57E3E0FEFD52250E230D1CDF','hex')), 'DESEDE/CBC/NoPadding', 'BASE64');
If you've been given a Hex IV, then you can use it as such:
encrypteded = encrypt('theStrin', ToBase64(BinaryDecode('FD52250E230D1CDFD5C2DF0D57E3E0FEFD52250E230D1CDF','hex')), 'DESEDE/CBC/NoPadding', 'BASE64', BinaryDecode("7fe8585328e9ac7b","hex"));
Hopefully this is enough info to get you on your way!
The only thing that seems off to me is the algorithm value you're using. Maybe try this?
encrypteded = encrypt('theString', 'FD52250E230D1CDFD5C2DF0D57E3E0FEFD52250E230D1CDF', 'DESEDE', 'BASE64', ToBase64('0'));
I don't know if the /CBC/NoPadding settings will be what you want, but I don't think they will be allowed in the algorithm argument.