Scrape CNNs Twiter page for tweets mentioning 'Donald Trump' in Python
I am currently working on a school project where I am comparing two news networks social media pages (CNN & FOX) for tweets mentioning 'Donald Trump' and exporting them into a CSV file to perform a sentiment analysis on. Which I will then analysis and asks certain questions to determine who has a more negative sentiment analysis. as well as some more analysis to be decided later.
I have been able to extract the tweets from CNN and export them into a CSV file, then perform the sentiment analysis on that CSV file. My problem is only grabbing tweets that specifically mention 'Donald Trump'
Here is my python code to scrape the tweets from CNN's twitter page:
(I have taken out my Twitter Auth Keys)
(file name: tweet_dumper.py)
import tweepy
import csv
#Twitter API credentials
consumer_key = "................."
consumer_secret = ".........................."
access_key = "............................"
access_secret = "......................."
def get_all_tweets(screen_name):
#Twitter only allows access to users most recent 3240 tweets with
this method
#authorize twitter, initialize tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
#initialize a list to hold all the tweepy Tweets
alltweets =
#make initial request for most recent tweets (200 is the maximum
allowed count)
new_tweets = api.user_timeline(screen_name = screen_name,count=200)
#save most recent tweets
alltweets.extend(new_tweets)
#save the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
#keep grabbing tweets until there are no tweets left to grab
while len(new_tweets) > 0:
print "getting tweets before %s" % (oldest)
#all subsiquent requests use the max_id param to prevent duplicates
new_tweets = api.user_timeline(screen_name =
screen_name,count=200,max_id=oldest)
#save most recent tweets
alltweets.extend(new_tweets)
#update the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
print "...%s tweets downloaded so far" % (len(alltweets))
#transform the tweepy tweets into a 2D array that will populate the csv
outtweets = [[tweet.id_str, tweet.created_at, tweet.text.encode("utf-
8")] for tweet in alltweets]
#write the csv
with open('%s_tweets.csv' % screen_name, 'wb') as f:
writer = csv.writer(f)
writer.writerow(["id","created_at","text"])
writer.writerows(outtweets)
pass
if __name__ == '__main__':
#pass in the username of the account you want to download
get_all_tweets("CNN")
What I need to do now is to modify this python code to get tweets mentioning only 'Donald Trumps' name, not just all the tweets from CNN (as the code does currently). Any suggetions would be appreciated.
Link to author for tweet scraper: https://gist.github.com/yanofsky/5436496
python python-2.7 twitter
add a comment |
I am currently working on a school project where I am comparing two news networks social media pages (CNN & FOX) for tweets mentioning 'Donald Trump' and exporting them into a CSV file to perform a sentiment analysis on. Which I will then analysis and asks certain questions to determine who has a more negative sentiment analysis. as well as some more analysis to be decided later.
I have been able to extract the tweets from CNN and export them into a CSV file, then perform the sentiment analysis on that CSV file. My problem is only grabbing tweets that specifically mention 'Donald Trump'
Here is my python code to scrape the tweets from CNN's twitter page:
(I have taken out my Twitter Auth Keys)
(file name: tweet_dumper.py)
import tweepy
import csv
#Twitter API credentials
consumer_key = "................."
consumer_secret = ".........................."
access_key = "............................"
access_secret = "......................."
def get_all_tweets(screen_name):
#Twitter only allows access to users most recent 3240 tweets with
this method
#authorize twitter, initialize tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
#initialize a list to hold all the tweepy Tweets
alltweets =
#make initial request for most recent tweets (200 is the maximum
allowed count)
new_tweets = api.user_timeline(screen_name = screen_name,count=200)
#save most recent tweets
alltweets.extend(new_tweets)
#save the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
#keep grabbing tweets until there are no tweets left to grab
while len(new_tweets) > 0:
print "getting tweets before %s" % (oldest)
#all subsiquent requests use the max_id param to prevent duplicates
new_tweets = api.user_timeline(screen_name =
screen_name,count=200,max_id=oldest)
#save most recent tweets
alltweets.extend(new_tweets)
#update the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
print "...%s tweets downloaded so far" % (len(alltweets))
#transform the tweepy tweets into a 2D array that will populate the csv
outtweets = [[tweet.id_str, tweet.created_at, tweet.text.encode("utf-
8")] for tweet in alltweets]
#write the csv
with open('%s_tweets.csv' % screen_name, 'wb') as f:
writer = csv.writer(f)
writer.writerow(["id","created_at","text"])
writer.writerows(outtweets)
pass
if __name__ == '__main__':
#pass in the username of the account you want to download
get_all_tweets("CNN")
What I need to do now is to modify this python code to get tweets mentioning only 'Donald Trumps' name, not just all the tweets from CNN (as the code does currently). Any suggetions would be appreciated.
Link to author for tweet scraper: https://gist.github.com/yanofsky/5436496
python python-2.7 twitter
add a comment |
I am currently working on a school project where I am comparing two news networks social media pages (CNN & FOX) for tweets mentioning 'Donald Trump' and exporting them into a CSV file to perform a sentiment analysis on. Which I will then analysis and asks certain questions to determine who has a more negative sentiment analysis. as well as some more analysis to be decided later.
I have been able to extract the tweets from CNN and export them into a CSV file, then perform the sentiment analysis on that CSV file. My problem is only grabbing tweets that specifically mention 'Donald Trump'
Here is my python code to scrape the tweets from CNN's twitter page:
(I have taken out my Twitter Auth Keys)
(file name: tweet_dumper.py)
import tweepy
import csv
#Twitter API credentials
consumer_key = "................."
consumer_secret = ".........................."
access_key = "............................"
access_secret = "......................."
def get_all_tweets(screen_name):
#Twitter only allows access to users most recent 3240 tweets with
this method
#authorize twitter, initialize tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
#initialize a list to hold all the tweepy Tweets
alltweets =
#make initial request for most recent tweets (200 is the maximum
allowed count)
new_tweets = api.user_timeline(screen_name = screen_name,count=200)
#save most recent tweets
alltweets.extend(new_tweets)
#save the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
#keep grabbing tweets until there are no tweets left to grab
while len(new_tweets) > 0:
print "getting tweets before %s" % (oldest)
#all subsiquent requests use the max_id param to prevent duplicates
new_tweets = api.user_timeline(screen_name =
screen_name,count=200,max_id=oldest)
#save most recent tweets
alltweets.extend(new_tweets)
#update the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
print "...%s tweets downloaded so far" % (len(alltweets))
#transform the tweepy tweets into a 2D array that will populate the csv
outtweets = [[tweet.id_str, tweet.created_at, tweet.text.encode("utf-
8")] for tweet in alltweets]
#write the csv
with open('%s_tweets.csv' % screen_name, 'wb') as f:
writer = csv.writer(f)
writer.writerow(["id","created_at","text"])
writer.writerows(outtweets)
pass
if __name__ == '__main__':
#pass in the username of the account you want to download
get_all_tweets("CNN")
What I need to do now is to modify this python code to get tweets mentioning only 'Donald Trumps' name, not just all the tweets from CNN (as the code does currently). Any suggetions would be appreciated.
Link to author for tweet scraper: https://gist.github.com/yanofsky/5436496
python python-2.7 twitter
I am currently working on a school project where I am comparing two news networks social media pages (CNN & FOX) for tweets mentioning 'Donald Trump' and exporting them into a CSV file to perform a sentiment analysis on. Which I will then analysis and asks certain questions to determine who has a more negative sentiment analysis. as well as some more analysis to be decided later.
I have been able to extract the tweets from CNN and export them into a CSV file, then perform the sentiment analysis on that CSV file. My problem is only grabbing tweets that specifically mention 'Donald Trump'
Here is my python code to scrape the tweets from CNN's twitter page:
(I have taken out my Twitter Auth Keys)
(file name: tweet_dumper.py)
import tweepy
import csv
#Twitter API credentials
consumer_key = "................."
consumer_secret = ".........................."
access_key = "............................"
access_secret = "......................."
def get_all_tweets(screen_name):
#Twitter only allows access to users most recent 3240 tweets with
this method
#authorize twitter, initialize tweepy
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
api = tweepy.API(auth)
#initialize a list to hold all the tweepy Tweets
alltweets =
#make initial request for most recent tweets (200 is the maximum
allowed count)
new_tweets = api.user_timeline(screen_name = screen_name,count=200)
#save most recent tweets
alltweets.extend(new_tweets)
#save the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
#keep grabbing tweets until there are no tweets left to grab
while len(new_tweets) > 0:
print "getting tweets before %s" % (oldest)
#all subsiquent requests use the max_id param to prevent duplicates
new_tweets = api.user_timeline(screen_name =
screen_name,count=200,max_id=oldest)
#save most recent tweets
alltweets.extend(new_tweets)
#update the id of the oldest tweet less one
oldest = alltweets[-1].id - 1
print "...%s tweets downloaded so far" % (len(alltweets))
#transform the tweepy tweets into a 2D array that will populate the csv
outtweets = [[tweet.id_str, tweet.created_at, tweet.text.encode("utf-
8")] for tweet in alltweets]
#write the csv
with open('%s_tweets.csv' % screen_name, 'wb') as f:
writer = csv.writer(f)
writer.writerow(["id","created_at","text"])
writer.writerows(outtweets)
pass
if __name__ == '__main__':
#pass in the username of the account you want to download
get_all_tweets("CNN")
What I need to do now is to modify this python code to get tweets mentioning only 'Donald Trumps' name, not just all the tweets from CNN (as the code does currently). Any suggetions would be appreciated.
Link to author for tweet scraper: https://gist.github.com/yanofsky/5436496
python python-2.7 twitter
python python-2.7 twitter
asked Nov 23 '18 at 22:51
User051593User051593
31114
31114
add a comment |
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53453661%2fscrape-cnns-twiter-page-for-tweets-mentioning-donald-trump-in-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53453661%2fscrape-cnns-twiter-page-for-tweets-mentioning-donald-trump-in-python%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown