Argthtjtr

s = 'ಆತಂಕವಾದಿಗಳಿಗೆ ವಿಶೇಷ ರಕ್ಷಣೆ ನೀಡುತ್ತದೆ, 24 ಕ್ಕೂ ಹೆಚ್ಚು ಹಿಂದೂ ಕಾರ್ಯಕರ್ತರ ಹತ್ಯೆಯಾದರೂ I am working on this'

words = re.findall(r'w+',s)

print(words)

I expected the above code to return only english words, but i am getting something as below.

['ಆತ', 'ಕವ', 'ದ', 'ಗಳ', 'ಗ', 'ವ', 'ಶ', 'ಷ', 'ರಕ', 'ಷಣ', 'ನ', 'ಡ', 'ತ', 'ತದ',

    '24', 'ಕ', 'ಕ', 'ಹ', 'ಚ', 'ಚ', 'ಹ', 'ದ', 'ಕ', 'ರ', 'ಯಕರ', 'ತರ', 'ಹತ', 'ಯ',

    'ಯ', 'ದರ', 'I', 'am', 'working', 'on', 'this']

Could someone explain how this is working?

I don't know why you expected w+ to only match English words. It doesn't even do that in ASCII mode. It matches any sequence of w characters, and the docs describe the actual behavior of w:

For Unicode (str) patterns:

Matches Unicode word characters; this includes most characters that can be part of a word in any language, as well as numbers and the underscore. If the ASCII flag is used, only [a-zA-Z0-9_] is matched.

For 8-bit (bytes) patterns:

Matches characters considered alphanumeric in the ASCII character set; this is equivalent to [a-zA-Z0-9_]. If the LOCALE flag is used, matches characters considered alphanumeric in the current locale and the underscore.

The docs unfortunately don't get any more specific than that, but w definitely isn't restricted to English.

If you wanted [a-zA-Z0-9_], you can write out your intended character class explicitly, or you can use the re.ASCII flag. If you wanted [a-zA-Z], write that out explicitly.

I cannot reproduce your observations, see the demo. Perhaps there is some encoding issue on your end, which is why w is picking up on Tamil characters. But, one workaround you could use here would be to just explicitly spell out of what the character class w actually consists:

words = re.findall(r'[A-Za-z0-9_]+', s)

print(words)

words = re.findall(r'w+',s)

The reason w+ doesn't pick up what you want is that it's missing the Unicode flag. The other answers here ignore encoding by simply saying which specific letters they are looking for

w

When the LOCALE and UNICODE flags are not specified, matches any
alphanumeric character and the underscore; this is equivalent to the
set [a-zA-Z0-9_]. With LOCALE, it will match the set [0-9_] plus
whatever characters are defined as alphanumeric for the current
locale. If UNICODE is set, this will match the characters [0-9_] plus
whatever is classified as alphanumeric in the Unicode character
properties database.

That is why.

Modify your code as given below to know why it is printing like that:

s = u'ಆತಂಕವಾದಿಗಳಿಗೆ ವಿಶೇಷ ರಕ್ಷಣೆ ನೀಡುತ್ತದೆ, 24 ಕ್ಕೂ ಹೆಚ್ಚು ಹಿಂದೂ ಕಾರ್ಯಕರ್ತರ ಹತ್ಯೆಯಾದರೂ I am 

working on this'

words = re.findall(r'w+',s)

print(words)





for letter in s:

    print(letter)

OUTPUT

['ಆತ', 'ಕವ', 'ದ', 'ಗಳ', 'ಗ', 'ವ', 'ಶ', 'ಷ', 'ರಕ', 'ಷಣ', 'ನ', 'ಡ', 'ತ', 'ತದ', '24', 'ಕ', 'ಕ', 'ಹ', 'ಚ', 'ಚ', 'ಹ', 'ದ', 'ಕ', 'ರ', 'ಯಕರ', 'ತರ', 'ಹತ', 'ಯ', 'ಯ', 'ದರ', 'I', 'am', 'working', 'on', 'this']

ಆ

ತ

ಂ 

ಕ

ವ

ಾ  

ದ



ಗ

ಳ



ಗ





ವ



ಶ

ೇ

ಷ



ರ

ಕ



ಷ

ಣ





ನ

ೀ

ಡ

ು

ತ



ತ

ದ



,



2

4



ಕ



ಕ

ೂ



ಹ



ಚ



ಚ

ು



ಹ



ಂ

ದ

ೂ



ಕ

ಾ

ರ



ಯ

ಕ

ರ



ತ

ರ



ಹ

ತ



ಯ



ಯ

ಾ

ದ

ರ

ೂ



I



a

m



w

o

r

k

i

n

g



o

n



t

h

i

s

Those circles are kind of spaces [ As understood by the code ]

Look at @itzMEonTV's suggestion:

In [46]: rex=re.compile(r'w+')                                                                                               

In [47]: rex                                                                                                                  

Out[47]: re.compile(r'w+', re.UNICODE)