Discoverying Friend List Changes on Facebook with Python
Unfriendfinder was a Firefox plugin which allowed Facebook users to detect when people left their friend list or deactivated their accounts. After three years of development, Facebook requested the removal of the extension due to violation of their terms of service. The author chose not to fight the request. In response, I’ve created sumfriender, a Python script that can detect friend list changes as well as import previous friend lists from the Unfriendfinder Firefox plugin and grease monkey scripts.
Facebook claims that Unfriendfinder violated their terms of service because it altered the interface between the user and Facebook to add in new unauthorized features. But Unfriendfinder isn’t a piece of malware. It started off as a Greasemonkey script, for which there are thousands out there designed to change the way people interact with websites that they use. It’s something users choose to install to alter the way the information they receive is presented to them or interacted with.
Update: As of May 1, 2015, this script no longer works. Facebook changed their API so apps could only list friends who used the same app. They added taggable friends to their API, but my attempts to use it inplace of the current implementation have failed. The following is just for reference.
The following script accomplishes the same ends. In order to use it, you will need a web server to place the token capture file, Python 3 and a basic understanding of how Python scripts and Facebook authorization work. First, place the fb_token.html file on a web server. When you authorize this script, Facebook will redirect you to this file and its simple Javascript will display the token you’ll need for authorization.
<!DOCTYPE html> <html lang="en"> <body> <p>FB Response <span id="token"></span></p> </body> </html>
Facebook recommends using https://www.facebook.com/connect/login_success.html as the redirect URL for authorization of desktop apps. Although the auth token is added to the URL, the login_success.html immediately redirects to block it out. It’s meant to be run within an embedded web browser in a desktop application where the token can be picked up programmatically before the redirect. It goes by too fast for a human to copy, so for a command line app, we need to use the fb_token.html to capture it.
Upload this file to a web server and then place the URL in the sumfriender.config file. Make sure this URL is setup in your Facebook app as a valid OAuth redirect. You will also need to add your Facebook API key and secrete as APP_ID and APP_KEY respectively:
[FB_API] APP_ID=1212121212121212 APP_KEY=abababababababababababababababab OAUTH_TOKEN= REDIRECT_URI=https://example.com/fb_token.html
In the advanced section of your Facebook application settings, you’ll need to add the address to the fb_token.html file on your web server.
If you used the previous Unfriendfinder Greasemonkey extension or the Firefox plugin, you can import your old friend database. To do so, search for either a prefs.js file or a greasemonkey-prefs.uff.js in your Firefox application data directory. This is typically in ~/.mozilla/firefox/xxxxxxxx.default on Linux or C:\Users\[Username]\AppData\Mozilla\Firefox\Profiles\xxxxxxxx.default on Windows. Run the preferences javascript through extract_uff.py to extract the Unfriendfinder data into text files.
Source code for extract_uff.py on GitHub
#!/usr/bin/env python3
#
# extract_uff.py - Extracts friends/unfriends list from previous
# versions of the UnfriendFinder GreaseMonkey script
# and FireFox Plugin
#
# Sumit Khanna <sumit@penguindreams.org> - /tech/
#
# License: Free for non-commercial use
#
import sys
import re
import json
import time
import os
def format_json(json_obj):
obj = json.loads(json_obj)
ret = []
for f in obj.items():
ret.append( "{0:15} {1}".format(f[0], str(f[1]['name'].encode('utf-8'),'ascii','ignore') ) )
return ret
def save(name,lst):
i = 1
while os.path.exists(name):
name = '{0}.{1}'.format(name,i)
i += 1
print("Writing {0}".format(name))
fd = open(name,'w')
for i in lst:
fd.write("{0}\n".format(i))
fd.close()
def user_pref(key,json_obj):
section = key.split('_')[-1:][0]
if section == 'unfriends':
save('unfriends.txt',format_json(json_obj))
if section == 'deactivated':
save('deactivated.txt',format_json(json_obj))
if section == 'friends':
save('friends.txt',format_json(json_obj))
if __name__ == '__main__':
if len(sys.argv) < 2:
print('Usage: extract_uff.py <prefs.js|greasemonkey-prefs.uff.js>')
exit(2)
fd = open(sys.argv[1], "r", encoding='utf-8')
for line in fd:
eline = line
if re.search('extensions.greasemonkey.scriptvals.unfriend_finder',eline):
eval(eline.strip().split(';')[0])
Running extract_uff.py
$ ./extract_uff.py ~/.mozilla/firefox/abcdefg.default/prefs.js Writing deactivated.txt Writing friends.txt Writing unfriends.txt Writing deactivated.txt.1 Writing friends.txt.1 Writing unfriends.txt.1
If you had multiple people using the same web browser while logging into Facebook, you might get multiple files as shown above. The friends.txt is what will be used going forward, so select the correct one for your account and move it to friends.txt in the directory you will execute sumfriends.py from. The other files are there for your reference.
sumfriends.py is the script that will grab your friends list. Upon the first run, it will open a web browser for you to authorize the application. If you’ve setup your application, redirect URL and fb_token.html correctly, you will get a token you can place in the configuration file. This token will expire very quickly, so ever subsequent time the script is run, it will exchange the token for one with a longer expiration time and save it in the configuration file. It will then show you any friends that are no longer in your list, output their details to the screen, save those details to the status.txt file and update the friends.txt file with your current friends list.
Source code for sumfriends.py on GitHub
#!/usr/bin/env python3
"""
sumfriender.py - A script for detecting changes in your Facebook friend list
Copyright 2013 - Sumit Khanna - PenguinDreams.org
Free for non-commercial use
"""
import time
import configparser
import urllib.request
import urllib.parse
import webbrowser
import json
import argparse
import os
import sys
import time
class Facebook(object):
def __init__(self,config_file):
config = configparser.ConfigParser()
config.read(config_file)
self.fb_app = config.get('FB_API','APP_ID')
self.fb_key = config.get('FB_API','APP_KEY')
self.redirect_uri = config.get('FB_API','REDIRECT_URI')
self.oauth_token = config.get('FB_API','OAUTH_TOKEN')
self.config_file = config_file
self.config_parser = config
def requires_auth(self):
return self.oauth_token.strip() == ''
def __fb_url(self,path,vars):
return 'https://graph.facebook.com/{0}?{1}'.format(
path,
urllib.parse.urlencode(vars))
def __make_request(self,url):
#print(url)
with urllib.request.urlopen(url) as html:
return str( html.read() , 'ascii' , 'ignore' )
def login(self):
webbrowser.open(self.__fb_url('oauth/authorize',
{ 'type' : 'user_agent' ,
'client_id' : self.fb_app ,
'redirect_uri' : self.redirect_uri,
'response_type' : 'token' ,
'scope' : 'user_friends'
}
))
def __friends_as_dict(self,obj):
ret = {}
for f in obj:
ret[f['id']] = f['name']
return ret
def friend_list(self):
obj = json.loads((self.__make_request(self.__fb_url('me/friends',
{ 'access_token' : self.oauth_token }
))))
friends = self.__friends_as_dict(obj['data'])
while 'paging' in obj and 'next' in obj['paging']:
obj = json.loads(self.__make_request(obj['paging']['next']))
if len(obj['data']) > 0:
friends.update(self.__friends_as_dict(obj['data']))
return friends
def user_active(self,uid):
try:
obj = json.loads(self.__make_request(self.__fb_url(uid,
{ 'access_token' : self.oauth_token }
)))
return 'id' in obj
except urllib.error.HTTPError:
return False
def extend_token(self):
"Requests a new OAUTH token with extended expiration time and saves it to the config file"
token = urllib.parse.parse_qs(self.__make_request(self.__fb_url('oauth/access_token',{
'client_id' : self.fb_app ,
'client_secret' : self.fb_key ,
'grant_type' : 'fb_exchange_token' ,
'fb_exchange_token' : self.oauth_token
})))['access_token'][0]
self.config_parser.set('FB_API','OAUTH_TOKEN',token)
fd = open(self.config_file,'w+')
self.config_parser.write(fd)
fd.close()
class StatusWriter(object):
def __init__(self,status_file,stdout=False):
self.__fd = open(status_file,'a')
self.__screen = stdout
def write(self,line):
self.__fd.write('{0}\n'.format(line))
if self.__screen:
print(line)
def __del__(self):
self.__fd.close()
def load_old_friends(data_file):
oldfriend = open(data_file,'r')
data = {}
for of in oldfriend:
parts = of.split(" ")
data[parts[0]] = " ".join(parts[1:]).strip()
return data
def save_friends(data_file,list):
fd = open(data_file,'w')
for f in list:
fd.write( "{0:15} {1}\n".format(f, str(list[f].encode('utf-8'),'ascii','ignore') ) )
fd.close()
if __name__ == '__main__':
parser = argparse.ArgumentParser(
description='A script to scan for changes in Facebook friend statuses',
epilog='Copyright 2013 Sumit Khanna. Free for non-commercial use. PenguinDreams.org')
#usage='%prog [-c config file] [-f friends file] [-s status file]'
parser.add_argument('-v','--version', action='version', version='%(prog)s 0.1')
parser.add_argument('-c',help='configuration file with API/AUTH keys [default: %(default)s]',
default='sumfriender.config',metavar='config')
parser.add_argument('-f',help='friend list db file [default: %(default)s]',
default='friends.txt',metavar='friend_db')
parser.add_argument('-l',help='status log file [default: %(default)s]',
default='status.txt',metavar='log')
parser.add_argument('-s',help='supress writing status to standard out', action='store_true')
args = parser.parse_args()
if not os.path.exists(args.c):
print('Configuration file {0} does not exist'.format(args.c), file=sys.stderr)
sys.exit(2)
fb = Facebook(args.c)
if fb.requires_auth():
print("You need a login key. Copy your access token to the OAUTH_TOKEN field in the configuration file.",file=sys.stderr)
fb.login()
sys.exit(3)
else:
#Let's renew our token so it doesn't expire
fb.extend_token()
cur_friends = fb.friend_list()
if not os.path.exists(args.f):
print("{0} not found. Creating initial friends list".format(args.f))
save_friends(args.f,cur_friends)
else:
old_friends = load_old_friends(args.f)
out = StatusWriter(args.l, not args.s)
heading = False
for uid in old_friends:
if uid not in cur_friends:
if not heading:
date = time.strftime("%Y-%m-%d %H:%M:%S")
out.write(date)
out.write('----------------------')
heading = True
status = 'is no longer in your friends list' if fb.user_active(uid) else 'has been deactivated or has disabled application access'
output = "Friend {0} ({1}) {2}".format(old_friends[uid],uid,status)
out.write(output)
if heading:
out.write('')
save_friends(args.f,cur_friends,)
I do not distribute any application keys or secretes. You’ll have to set those up yourself. Be aware you might be violating Facebook’s terms of service by using this script. You’ll want to run this script at regular intervals using a scheduler such as cron. Since it only displays output when there are changes to your friends, you can combine it with my Ruby E-mail Script to send you e-mail notifications on changes.
So you might be asking, why do I care if someone removes me as a friend on Facebook? Am I placing putting too much importance on my on-line life? Well honestly, I rarely use Facebook for anything other than Instant Message and promoting my own websites and projects. I just find it interesting what lengths Facebook would go to ensure users only experience their service in the way Facebook intended.
Facebook’s primary source of income is their ad revenue. Everything they do is carefully engineered to increase your interaction time. Don’t like a new interface change? It was probably put before a focus group to make sure you dislike it enough that you spend time trying to learn how to use it, but not so much you stop interacting with it entirely. Even your feed is filtered to avoid content from friends with opinions contrary to your own, in what Eli Pariser calls The Filter Bubble.
What bothers me is that Facebook would even request a take-down from Unfriendfinder. Is writing software that changes the way we interact with a website a violation of terms of service? How can it possibly infringe on intellectual property? If I built a custom web browser that added lots of additional context to the content of web pages, do the owners of those web pages have any right to demand I stop distributing my custom web browser? What does this imply about ad blocking software? What about augmented reality displays? Could a building owner sue for an advertisement a pair of glasses overlays on the building? If data is sent to your computer, whether it be audio, video or web pages, what software you chose to view and interpret that data should be up to you.
With the way technology is moving, I think that in the future, we’ll see more people move off closed private networks like Facebook, Google and Twitter and to more public and open platforms. We’ll see more open source solutions, with standard publishing interfaces, that will allow more control of what people post and share and more portability for moving that data between services.