Il modo migliore per 'ripulire' testo HTML

Question

ho il seguente testo:

"It's the show your only friend and pastor have been talking about! 
<i>Wonder Showzen</i> is a hilarious glimpse into the black 
heart of childhood innocence! Get ready as the complete first season of MTV2's<i> Wonder Showzen</i> tackles valuable life lessons like birth, 
nature, diversity, and history &#8211; all inside the prison of 
your mind! Where else can you..." 

Quello che voglio fare con questo è rimuovere i tag HTML e codificare in unicode. Attualmente sto facendo:

def remove_tags(text): 
    return TAG_RE.sub('', text) 

Quale solo strisce il tag. Come dovrei codificare correttamente quanto sopra per l'archiviazione del database?

Answer 1

Si potrebbe provare a passare il testo attraverso un parser HTML. Ecco un esempio utilizzando BeautifulSoup:

from bs4 import BeautifulSoup 

text = '''It's the show your only friend and pastor have been talking about! 
<i>Wonder Showzen</i> is a hilarious glimpse into the black 
heart of childhood innocence! Get ready as the complete first season of MTV2's<i> Wonder Showzen</i> tackles valuable life lessons like birth, 
nature, diversity, and history &#8211; all inside the prison of 
your mind! Where else can you...''' 

soup = BeautifulSoup(text) 

>>> soup.text 
u"It's the show your only friend and pastor have been talking about! \nWonder Showzen is a hilarious glimpse into the black \nheart of childhood innocence! Get ready as the complete first season of MTV2's Wonder Showzen tackles valuable life lessons like birth, \nnature, diversity, and history \u2013 all inside the prison of \nyour mind! Where else can you..." 

Si dispone ora di una stringa unicode con le entità HTML convertito in unicode sfuggito caratteri, vale a dire – è stato convertito a \u2013.

Rimuove anche i tag HTML.