python - Is there a faster way to clean out control characters in a file? -

previously, had been cleaning out data using code snippet below

import unicodedata, re, io  all_chars = (unichr(i) in xrange(0x110000)) control_chars = ''.join(c c in all_chars if unicodedata.category(c)[0] == 'c') cc_re = re.compile('[%s]' % re.escape(control_chars)) def rm_control_chars(s): # see http://www.unicode.org/reports/tr44/#general_category_values     return cc_re.sub('', s)  cleanfile = [] io.open('filename.txt', 'r', encoding='utf8') fin:     line in fin:         line =rm_control_chars(line)         cleanfile.append(line) 

there newline characters in file want keep.

the following records time taken cc_re.sub('', s) substitute first few lines (1st column time taken , 2nd column len(s)):

0.275146961212 251 0.672796010971 614 0.178567171097 163 0.200030088425 180 0.236430883408 215 0.343492984772 313 0.317672967911 290 0.160616159439 142 0.0732028484344 65 0.533437013626 468 0.260229110718 236 0.231380939484 204 0.197766065598 181 0.283867120743 258 0.229172945023 208 

as @ashwinichaudhary suggested, using s.translate(dict.fromkeys(control_chars)) , same time taken log outputs:

0.464188098907 252 0.366552114487 615 0.407374858856 164 0.322507858276 181 0.35142993927 216 0.319973945618 314 0.324357032776 291 0.371646165848 143 0.354818105698 66 0.351796150208 469 0.388131856918 237 0.374715805054 205 0.363368988037 182 0.425950050354 259 0.382766962051 209 

but code slow 1gb of text. there other way clean out controlled characters?

found solution working character charater, bench marked using 100k file:

import unicodedata, re, io time import time  # generate randomly file test script  string import lowercase random import random  all_chars = (unichr(i) in xrange(0x110000)) control_chars = [c c in all_chars if unicodedata.category(c)[0] == 'c'] chars = (list(u'%s' % lowercase) * 115117) + control_chars  fnam = 'filename.txt'  out=io.open(fnam, 'w')  line in range(1000000):     out.write(u''.join(chars[int(random()*len(chars))] _ in range(600)) + u'\n') out.close()   # version proposed alvas all_chars = (unichr(i) in xrange(0x110000)) control_chars = ''.join(c c in all_chars if unicodedata.category(c)[0] == 'c') cc_re = re.compile('[%s]' % re.escape(control_chars)) def rm_control_chars(s):     return cc_re.sub('', s)  t0 = time() cleanfile = [] io.open(fnam, 'r', encoding='utf8') fin:     line in fin:         line =rm_control_chars(line)         cleanfile.append(line) out=io.open(fnam + '_out1.txt', 'w') out.write(''.join(cleanfile)) out.close() print time() - t0  # using set , checking character character all_chars = (unichr(i) in xrange(0x110000)) control_chars = set(c c in all_chars if unicodedata.category(c)[0] == 'c') def rm_control_chars_1(s):     return ''.join(c c in s if not c in control_chars)  t0 = time() cleanfile = [] io.open(fnam, 'r', encoding='utf8') fin:     line in fin:         line = rm_control_chars_1(line)         cleanfile.append(line) out=io.open(fnam + '_out2.txt', 'w') out.write(''.join(cleanfile)) out.close() print time() - t0 

the output is:

114.625444174 0.0149750709534 

i tried on file of 1gb (only second one) , lasted 186s.

i wrote other version of same script, faster (176s), , more memory efficient (for large files not fitting in ram):

t0 = time() out=io.open(fnam + '_out5.txt', 'w') io.open(fnam, 'r', encoding='utf8') fin:     line in fin:         out.write(rm_control_chars_1(line)) out.close() print time() - t0