Compare values between one text file and another for matching
I have a text file (results.txt) that contains a list of Hashes, URLs, IPs.
Trying to:
1.extract only the URLs in the "match" value where there is also key "type":"url"
2.to compare all URLs with another local text file that contains a list of known domains(Alexa.txt).
3.If there is a match in the compare process then to delete the URL line from the results.txt file
text file example:
{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}
So the goal is to compare: firefox.com and amazon.com
against the Alexa.txt . if there is a match then delete the URL line from the results.txt file.
My code is very undeveloped, quite lost...
python
asked Nov 22 '18 at 13:23
bugnet17bugnet17
667
add a comment |
I have a text file (results.txt) that contains a list of Hashes, URLs, IPs.
Trying to:
1.extract only the URLs in the "match" value where there is also key "type":"url"
2.to compare all URLs with another local text file that contains a list of known domains(Alexa.txt).
3.If there is a match in the compare process then to delete the URL line from the results.txt file
text file example:
{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}
So the goal is to compare: firefox.com and amazon.com
against the Alexa.txt . if there is a match then delete the URL line from the results.txt file.
My code is very undeveloped, quite lost...
python
asked Nov 22 '18 at 13:23
bugnet17bugnet17
667
Can you show some code of what you tried?
– Yserbius
Nov 22 '18 at 13:27
There is no python in this question. In fact there is no question in this question (other than the implicit "can you write this for me?"). See How to Ask and how to create a Minimal, Complete, and Verifiable example.
– Peter Wood
Nov 22 '18 at 13:34
Oh, forget adding my code. sorry
– bugnet17
Nov 22 '18 at 13:36
The "JSON file" you provided is not formatted as one piece of JSON. Instead, it contains several JSON blobs seperated by newlines. Yet your sample code is trying to parse the entire data file as one? For starters, You would need to load each line as json (or fix your data file so it's one nested JSON string)
– Corey Goldberg
Nov 22 '18 at 14:05
I edited my question
– bugnet17
Nov 22 '18 at 14:58
add a comment |
I have a text file (results.txt) that contains a list of Hashes, URLs, IPs.
Trying to:
1.extract only the URLs in the "match" value where there is also key "type":"url"
2.to compare all URLs with another local text file that contains a list of known domains(Alexa.txt).
3.If there is a match in the compare process then to delete the URL line from the results.txt file
text file example:
{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}
So the goal is to compare: firefox.com and amazon.com
against the Alexa.txt . if there is a match then delete the URL line from the results.txt file.
My code is very undeveloped, quite lost...
python
asked Nov 22 '18 at 13:23
bugnet17bugnet17
667
I have a text file (results.txt) that contains a list of Hashes, URLs, IPs.
Trying to:
1.extract only the URLs in the "match" value where there is also key "type":"url"
2.to compare all URLs with another local text file that contains a list of known domains(Alexa.txt).
3.If there is a match in the compare process then to delete the URL line from the results.txt file
text file example:
{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"}
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}
So the goal is to compare: firefox.com and amazon.com
against the Alexa.txt . if there is a match then delete the URL line from the results.txt file.
My code is very undeveloped, quite lost...
python
python
asked Nov 22 '18 at 13:23
bugnet17bugnet17
667
asked Nov 22 '18 at 13:23
bugnet17bugnet17
667
edited Nov 22 '18 at 14:58
bugnet17
asked Nov 22 '18 at 13:23
bugnet17bugnet17
667
asked Nov 22 '18 at 13:23
bugnet17bugnet17
667
asked Nov 22 '18 at 13:23
bugnet17bugnet17
667
667
Can you show some code of what you tried?
– Yserbius
Nov 22 '18 at 13:27
There is no python in this question. In fact there is no question in this question (other than the implicit "can you write this for me?"). See How to Ask and how to create a Minimal, Complete, and Verifiable example.
– Peter Wood
Nov 22 '18 at 13:34
Oh, forget adding my code. sorry
– bugnet17
Nov 22 '18 at 13:36
The "JSON file" you provided is not formatted as one piece of JSON. Instead, it contains several JSON blobs seperated by newlines. Yet your sample code is trying to parse the entire data file as one? For starters, You would need to load each line as json (or fix your data file so it's one nested JSON string)
– Corey Goldberg
Nov 22 '18 at 14:05
I edited my question
– bugnet17
Nov 22 '18 at 14:58
add a comment |
Can you show some code of what you tried?
– Yserbius
Nov 22 '18 at 13:27
There is no python in this question. In fact there is no question in this question (other than the implicit "can you write this for me?"). See How to Ask and how to create a Minimal, Complete, and Verifiable example.
– Peter Wood
Nov 22 '18 at 13:34
Oh, forget adding my code. sorry
– bugnet17
Nov 22 '18 at 13:36
The "JSON file" you provided is not formatted as one piece of JSON. Instead, it contains several JSON blobs seperated by newlines. Yet your sample code is trying to parse the entire data file as one? For starters, You would need to load each line as json (or fix your data file so it's one nested JSON string)
– Corey Goldberg
Nov 22 '18 at 14:05
I edited my question
– bugnet17
Nov 22 '18 at 14:58
Can you show some code of what you tried?
– Yserbius
Nov 22 '18 at 13:27
Can you show some code of what you tried?
– Yserbius
Nov 22 '18 at 13:27
There is no python in this question. In fact there is no question in this question (other than the implicit "can you write this for me?"). See How to Ask and how to create a Minimal, Complete, and Verifiable example.
– Peter Wood
Nov 22 '18 at 13:34
There is no python in this question. In fact there is no question in this question (other than the implicit "can you write this for me?"). See How to Ask and how to create a Minimal, Complete, and Verifiable example.
– Peter Wood
Nov 22 '18 at 13:34
Oh, forget adding my code. sorry
– bugnet17
Nov 22 '18 at 13:36
Oh, forget adding my code. sorry
– bugnet17
Nov 22 '18 at 13:36
The "JSON file" you provided is not formatted as one piece of JSON. Instead, it contains several JSON blobs seperated by newlines. Yet your sample code is trying to parse the entire data file as one? For starters, You would need to load each line as json (or fix your data file so it's one nested JSON string)
– Corey Goldberg
Nov 22 '18 at 14:05
The "JSON file" you provided is not formatted as one piece of JSON. Instead, it contains several JSON blobs seperated by newlines. Yet your sample code is trying to parse the entire data file as one? For starters, You would need to load each line as json (or fix your data file so it's one nested JSON string)
– Corey Goldberg
Nov 22 '18 at 14:05
I edited my question
– bugnet17
Nov 22 '18 at 14:58
I edited my question
– bugnet17
Nov 22 '18 at 14:58
add a comment |
1 Answer
1
active
oldest
votes
This is rather broad with a lot of questions rather than just 1.
For starters you need a valid JSON string e.g.
[{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}]
With this structure you can do the test on if type == URL
import json
import pandas as pd
with open(r'C:UsersUserDesktoppyJSON.json') as datafile:
data = json.load(datafile)
print([f for f in data if f['type']=="URL"])
Hopefully, that moves things a long a little.
answered Nov 22 '18 at 13:54
QHarrQHarr
33.8k82043
I edited my question
– bugnet17
Nov 22 '18 at 14:58
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53431976%2fcompare-values-between-one-text-file-and-another-for-matching%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
This is rather broad with a lot of questions rather than just 1.
For starters you need a valid JSON string e.g.
[{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}]
With this structure you can do the test on if type == URL
import json
import pandas as pd
with open(r'C:UsersUserDesktoppyJSON.json') as datafile:
data = json.load(datafile)
print([f for f in data if f['type']=="URL"])
Hopefully, that moves things a long a little.
answered Nov 22 '18 at 13:54
QHarrQHarr
33.8k82043
I edited my question
– bugnet17
Nov 22 '18 at 14:58
add a comment |
This is rather broad with a lot of questions rather than just 1.
For starters you need a valid JSON string e.g.
[{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}]
With this structure you can do the test on if type == URL
import json
import pandas as pd
with open(r'C:UsersUserDesktoppyJSON.json') as datafile:
data = json.load(datafile)
print([f for f in data if f['type']=="URL"])
Hopefully, that moves things a long a little.
answered Nov 22 '18 at 13:54
QHarrQHarr
33.8k82043
I edited my question
– bugnet17
Nov 22 '18 at 14:58
add a comment |
This is rather broad with a lot of questions rather than just 1.
For starters you need a valid JSON string e.g.
[{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}]
With this structure you can do the test on if type == URL
import json
import pandas as pd
with open(r'C:UsersUserDesktoppyJSON.json') as datafile:
data = json.load(datafile)
print([f for f in data if f['type']=="URL"])
Hopefully, that moves things a long a little.
answered Nov 22 '18 at 13:54
QHarrQHarr
33.8k82043
This is rather broad with a lot of questions rather than just 1.
For starters you need a valid JSON string e.g.
[{"path": "pastebin.com", "match": "http://firefox.com/eiKMths", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "http://amazon.com/m7GTLj59x7", "type": "URL", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "109.228.9.122", "type": "IP", "page": 1, "file": "pastebin.com"},
{"path": "pastebin.com", "match": "ED9DF1625C1D981FE54490FAB7934BE36322E4C5C88A19F4C244307DF2523E52", "type": "SHA256", "page": 1, "file": "pastebin.com"}]
With this structure you can do the test on if type == URL
import json
import pandas as pd
with open(r'C:UsersUserDesktoppyJSON.json') as datafile:
data = json.load(datafile)
print([f for f in data if f['type']=="URL"])
Hopefully, that moves things a long a little.
answered Nov 22 '18 at 13:54
QHarrQHarr
33.8k82043
answered Nov 22 '18 at 13:54
QHarrQHarr
33.8k82043
answered Nov 22 '18 at 13:54
QHarrQHarr
33.8k82043
answered Nov 22 '18 at 13:54
QHarrQHarr
33.8k82043
33.8k82043
I edited my question
– bugnet17
Nov 22 '18 at 14:58
add a comment |
I edited my question
– bugnet17
Nov 22 '18 at 14:58
I edited my question
– bugnet17
Nov 22 '18 at 14:58
I edited my question
– bugnet17
Nov 22 '18 at 14:58
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53431976%2fcompare-values-between-one-text-file-and-another-for-matching%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Can you show some code of what you tried?
– Yserbius
Nov 22 '18 at 13:27
There is no python in this question. In fact there is no question in this question (other than the implicit "can you write this for me?"). See How to Ask and how to create a Minimal, Complete, and Verifiable example.
– Peter Wood
Nov 22 '18 at 13:34
Oh, forget adding my code. sorry
– bugnet17
Nov 22 '18 at 13:36
The "JSON file" you provided is not formatted as one piece of JSON. Instead, it contains several JSON blobs seperated by newlines. Yet your sample code is trying to parse the entire data file as one? For starters, You would need to load each line as json (or fix your data file so it's one nested JSON string)
– Corey Goldberg
Nov 22 '18 at 14:05
I edited my question
– bugnet17
Nov 22 '18 at 14:58