split one line into multiple line using number of bytes - unix

Question

I want to split single line into multiple line of 8 bytes each. And I am using the fold command and since this file contains the special characters the fold command does not work and it breaks in the middle of multibyte character.
File Content
あいbbえおかcc髙①こさし㈱㈱ちつて髙aabbc
Command Used
fold -b8 dummy_file.dat
Appreciate any help on this.

Answer 1

The problem here is that your text contains multi-bytes characters that will be broken by the fold command if we split them on 2 lines.
echo "あいbbえおかcc髙①こさし㈱㈱ちつて髙aabbc" | fold -b8
あいbb
えお��
�cc髙��
�こさ�
��㈱㈱
ちつ��
�髙aabb
c
If you want to have 8 characters per line you can use the following sed command:
echo "あいbbえおかcc髙①こさし㈱㈱ちつて髙aabbc" | sed 's/.\{8\}/&\n/g'
あいbbえおかc
c髙①こさし㈱㈱
ちつて髙aabb
c
that add a breakline after each occurrence of 8 characters.
If you do not want to display 8 characters but want to constraint each line to be at most 8 bytes without breaking the text content then you can use the python script:
import sys
def utf8len(s):
return len(s.encode('utf-8'))
entry = unicode(sys.stdin.read(),'utf-8')
tmp = ''
for c in entry:
if utf8len(tmp)+utf8len(c) > 8:
print tmp
tmp = c
elif utf8len(tmp)+utf8len(c) == 8:
print tmp,c
tmp = ''
else:
tmp += c
if tmp:
print tmp
output:
echo -n "あいbbえおかcc髙①こさし㈱㈱ちつて髙aabbc" | python max8bytes.py
あいb b
えお
かcc 髙
①こ
さし
㈱㈱
ちつ
て髙a a
bbc
Explanations:
You define a function that will count how many bytes you have per char.
You read char by char stdin and you avoid to have more than 8 bytes on the same line. If you do not want to have less than you can add some spaces char at the end of each line.