Python 3 scripts to convert chat logs from various messaging platforms into Panda DataFrames. Can also generate ggplot histograms and word clouds from fetched chat logs.
21 Oct 2018: Updated Facebook Messenger and Google Hangouts parsers to make them work with the new exported file formats.
9 Feb 2018: Telegram support added thanks to @bmwant.
24 Oct 2016: Initial release supporting Facebook Messenger and Google Hangouts.
|Platform||Direct Chat||Group Chat|
Data exported for each message regardless of the platform:
|conversationId||A conversation ID, unique by platform|
|conversationWithName||Name of the other people in a direct conversation, or name of the group conversation|
|senderName||Name of the sender|
|text||Text of the message|
|language||Language of the conversation as inferred by langdetect|
|datetime||The proleptic Gregorian ordinal (= number of days since 01/01/0001)|
How to use it?
1. Download your chat logs
Use Google Takeout: https://takeout.google.com/settings/takeout
Request an archive containing your Hangouts chat logs. Extract the file called
Hangouts.json and put it in the
raw folder of Chatistics.
Google switched from "Google Talk" to "Google Hangouts" mid-2013. Sadly you will only get your Hangouts logs using Takeout.
- Go to the "Settings" page: https://www.facebook.com/settings
- Click on "Download a copy of your Facebook data" at the bottom of the General section.
- Click on "Start My Archive". It will take Facebook a while to generate it.
- Once it is done download and extract the archive, then move the
messagesfolder in the
rawfolder of Chatistics.
2. Parse the logs into DataFrames
Install the required Python packages:
virtualenv Chatistics source Chatistics/bin/activate pip install -r requirements.txt
You will need to give your own name to the parsers so it can make sense of the conversations. Use the exact same format as you have on Messenger or Hangouts.
python parsers/hangouts.py --own-name "John Doe"
python parsers/messenger.py --own-name "John Doe"
- Create your Telegram application to access chat logs (instructions).
You will need
- Paste these values to
parsers/config.pyinto corresponding variables.
- Grab your message history data
The pickle files will now be ready for analysis in the
For more options use the
-h argument on the parsers.
Chatistics can plot the chat logs as histograms, showing how many messages each interlocutor sent. It can also generate word clouds based on word density and a base image.
Plot all messages with:
python analyse.py --data data/*
You can filter messages as needed:
--filter-conversation FILTERCONVERSATION only keep messages sent in a conversation with this sender --filter-sender FILTERSENDER only keep messages sent by this sender --remove-sender REMOVESENDER remove messages sent by this sender
Eg to see all the messages sent between you and Jane Doe:
python analyse.py --data data/* --filter-conversation "Jane Doe"
To see the messages sent to you by the top 10 people with whom you talk the most:
python analyse.py --data data/* --remove-sender "Your Name" -n 10
You can also plot the conversation densities using the
You will need a mask file to render the word cloud. The white bits of the image will be left empty, the rest will be filled with words using the color of the image. See the WordCloud library documentation for more information.
python cloud.py --data data/* -m img/mask_image.jpg
You can filter which messages to use using the same flags as with histograms.
- Parsers for more chat platforms: WhatsApp? Pidgin? ...
- Handle group chats on more platforms.
- Figure out
Pull requests are welcome!
ValueError: unknown locale: UTF-8
export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8
LXML incompatible library
ImportError: dlopen(/Users/flaurent/Sites/Chatistics/Chatistics/lib/python2.7/site-packages/lxml/etree.so, 2): Library not loaded: @rpath/libxml2.2.dylib Referenced from: /Users/flaurent/Sites/Chatistics/Chatistics/lib/python2.7/site-packages/lxml/etree.so Reason: Incompatible library version: etree.so requires version 12.0.0 or later, but libxml2.2.dylib provides version 10.0.0
This will fix it: https://stackoverflow.com/a/31607751/318557
ModuleNotFoundError: No module named 'parsers'