Python - how to read txt file every xxx time and add to multiprocess if new text added
So basically I am trying to do something that I am not sure if it is possible to do nor not.
My idea is that I am using a multiprocessing which means it runs its separated process on the background for each "task" done. Similar to:
names.txt
Oskar Baldwin
Khalil Whittle
Jevon Burn
Paddy Wilkinson
Jocelyn Weiss
Ishaq Glenn
Zahraa Macfarlane
Marianna Roy
Humera Schultz
Luther Pugh
import json, time, sys, os, timeit, random, multiprocessing.dummy, re
import threading
def main(names):
print(names)
time.sleep(5)
if __name__ == '__main__':
try:
jobs =
for names in [line.rstrip('n') for line in open('names.txt')]:
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
p.start()
except KeyboardInterrupt:
print('Keyboard - Interrupted)
sys.exit()
So the output of this will be that each process will take care of each of names for themselves so process 1 will use the first line in txt file and the 2nd process will use the second name and so on.
My question is how can I move on and be able to check the names.txt xxx time and check if there is a new name added then add it to a new process and if there is no new names then just continue the process and sleep again until another xxx time while the other process is running in the background?
python multithreading process multiprocessing
asked Nov 20 '18 at 11:24
HellosiroverthereHellosiroverthere
10818
add a comment |
So basically I am trying to do something that I am not sure if it is possible to do nor not.
My idea is that I am using a multiprocessing which means it runs its separated process on the background for each "task" done. Similar to:
names.txt
Oskar Baldwin
Khalil Whittle
Jevon Burn
Paddy Wilkinson
Jocelyn Weiss
Ishaq Glenn
Zahraa Macfarlane
Marianna Roy
Humera Schultz
Luther Pugh
import json, time, sys, os, timeit, random, multiprocessing.dummy, re
import threading
def main(names):
print(names)
time.sleep(5)
if __name__ == '__main__':
try:
jobs =
for names in [line.rstrip('n') for line in open('names.txt')]:
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
p.start()
except KeyboardInterrupt:
print('Keyboard - Interrupted)
sys.exit()
So the output of this will be that each process will take care of each of names for themselves so process 1 will use the first line in txt file and the 2nd process will use the second name and so on.
My question is how can I move on and be able to check the names.txt xxx time and check if there is a new name added then add it to a new process and if there is no new names then just continue the process and sleep again until another xxx time while the other process is running in the background?
python multithreading process multiprocessing
asked Nov 20 '18 at 11:24
HellosiroverthereHellosiroverthere
10818
add a comment |
So basically I am trying to do something that I am not sure if it is possible to do nor not.
My idea is that I am using a multiprocessing which means it runs its separated process on the background for each "task" done. Similar to:
names.txt
Oskar Baldwin
Khalil Whittle
Jevon Burn
Paddy Wilkinson
Jocelyn Weiss
Ishaq Glenn
Zahraa Macfarlane
Marianna Roy
Humera Schultz
Luther Pugh
import json, time, sys, os, timeit, random, multiprocessing.dummy, re
import threading
def main(names):
print(names)
time.sleep(5)
if __name__ == '__main__':
try:
jobs =
for names in [line.rstrip('n') for line in open('names.txt')]:
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
p.start()
except KeyboardInterrupt:
print('Keyboard - Interrupted)
sys.exit()
So the output of this will be that each process will take care of each of names for themselves so process 1 will use the first line in txt file and the 2nd process will use the second name and so on.
My question is how can I move on and be able to check the names.txt xxx time and check if there is a new name added then add it to a new process and if there is no new names then just continue the process and sleep again until another xxx time while the other process is running in the background?
python multithreading process multiprocessing
asked Nov 20 '18 at 11:24
HellosiroverthereHellosiroverthere
10818
So basically I am trying to do something that I am not sure if it is possible to do nor not.
My idea is that I am using a multiprocessing which means it runs its separated process on the background for each "task" done. Similar to:
names.txt
Oskar Baldwin
Khalil Whittle
Jevon Burn
Paddy Wilkinson
Jocelyn Weiss
Ishaq Glenn
Zahraa Macfarlane
Marianna Roy
Humera Schultz
Luther Pugh
import json, time, sys, os, timeit, random, multiprocessing.dummy, re
import threading
def main(names):
print(names)
time.sleep(5)
if __name__ == '__main__':
try:
jobs =
for names in [line.rstrip('n') for line in open('names.txt')]:
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
p.start()
except KeyboardInterrupt:
print('Keyboard - Interrupted)
sys.exit()
So the output of this will be that each process will take care of each of names for themselves so process 1 will use the first line in txt file and the 2nd process will use the second name and so on.
My question is how can I move on and be able to check the names.txt xxx time and check if there is a new name added then add it to a new process and if there is no new names then just continue the process and sleep again until another xxx time while the other process is running in the background?
python multithreading process multiprocessing
python multithreading process multiprocessing
asked Nov 20 '18 at 11:24
HellosiroverthereHellosiroverthere
10818
asked Nov 20 '18 at 11:24
HellosiroverthereHellosiroverthere
10818
asked Nov 20 '18 at 11:24
HellosiroverthereHellosiroverthere
10818
asked Nov 20 '18 at 11:24
HellosiroverthereHellosiroverthere
10818
asked Nov 20 '18 at 11:24
HellosiroverthereHellosiroverthere
10818
10818
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Quick and dirty way of doing it. I didn't research whether there's a way to query an item in the jobs list for its "args" attribute.
Beware that it does not correctly handle the case where a duplicate name is added after that process has finished executing. You'll have to come up with a way to remove finished processes from the used_names list.
jobs =
used_names =
for i in range(0, 10): # repeat as often as necessary
try:
for names in [line.rstrip('n') for line in open('names.txt')]:
if names not in used_names: # dont spawn new process if one already exists
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
used_names.append(p)
p.start()
time.sleep(0.1) # arbitrary sleep
except KeyboardInterrupt:
print('Keyboard - Interrupted')
sys.exit(0)
answered Nov 20 '18 at 22:43
JavonJavon
513
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53391973%2fpython-how-to-read-txt-file-every-xxx-time-and-add-to-multiprocess-if-new-text%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Quick and dirty way of doing it. I didn't research whether there's a way to query an item in the jobs list for its "args" attribute.
Beware that it does not correctly handle the case where a duplicate name is added after that process has finished executing. You'll have to come up with a way to remove finished processes from the used_names list.
jobs =
used_names =
for i in range(0, 10): # repeat as often as necessary
try:
for names in [line.rstrip('n') for line in open('names.txt')]:
if names not in used_names: # dont spawn new process if one already exists
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
used_names.append(p)
p.start()
time.sleep(0.1) # arbitrary sleep
except KeyboardInterrupt:
print('Keyboard - Interrupted')
sys.exit(0)
answered Nov 20 '18 at 22:43
JavonJavon
513
add a comment |
Quick and dirty way of doing it. I didn't research whether there's a way to query an item in the jobs list for its "args" attribute.
Beware that it does not correctly handle the case where a duplicate name is added after that process has finished executing. You'll have to come up with a way to remove finished processes from the used_names list.
jobs =
used_names =
for i in range(0, 10): # repeat as often as necessary
try:
for names in [line.rstrip('n') for line in open('names.txt')]:
if names not in used_names: # dont spawn new process if one already exists
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
used_names.append(p)
p.start()
time.sleep(0.1) # arbitrary sleep
except KeyboardInterrupt:
print('Keyboard - Interrupted')
sys.exit(0)
answered Nov 20 '18 at 22:43
JavonJavon
513
add a comment |
Quick and dirty way of doing it. I didn't research whether there's a way to query an item in the jobs list for its "args" attribute.
Beware that it does not correctly handle the case where a duplicate name is added after that process has finished executing. You'll have to come up with a way to remove finished processes from the used_names list.
jobs =
used_names =
for i in range(0, 10): # repeat as often as necessary
try:
for names in [line.rstrip('n') for line in open('names.txt')]:
if names not in used_names: # dont spawn new process if one already exists
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
used_names.append(p)
p.start()
time.sleep(0.1) # arbitrary sleep
except KeyboardInterrupt:
print('Keyboard - Interrupted')
sys.exit(0)
answered Nov 20 '18 at 22:43
JavonJavon
513
Quick and dirty way of doing it. I didn't research whether there's a way to query an item in the jobs list for its "args" attribute.
Beware that it does not correctly handle the case where a duplicate name is added after that process has finished executing. You'll have to come up with a way to remove finished processes from the used_names list.
jobs =
used_names =
for i in range(0, 10): # repeat as often as necessary
try:
for names in [line.rstrip('n') for line in open('names.txt')]:
if names not in used_names: # dont spawn new process if one already exists
p = multiprocessing.dummy.Process(target=main, args=(names,))
jobs.append(p)
used_names.append(p)
p.start()
time.sleep(0.1) # arbitrary sleep
except KeyboardInterrupt:
print('Keyboard - Interrupted')
sys.exit(0)
answered Nov 20 '18 at 22:43
JavonJavon
513
answered Nov 20 '18 at 22:43
JavonJavon
513
answered Nov 20 '18 at 22:43
JavonJavon
513
answered Nov 20 '18 at 22:43
JavonJavon
513
513
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53391973%2fpython-how-to-read-txt-file-every-xxx-time-and-add-to-multiprocess-if-new-text%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
