This Python script uses the Gmail API to authenticate a user and retrieve the last 1000 emails from their Gmail account. The email data is saved as a JSON file on the local system. Below is a breakdown of the process and the complete script.
token.json, they are reused. Otherwise, a new login prompt is initiated.last_1000_emails.json with readable formatting.#!/usr/bin/env python
import os
import json
from google.oauth2.credentials import Credentials
from google_auth_oauthlib.flow import InstalledAppFlow
from googleapiclient.discovery import build
# Define the OAuth scope for read-only Gmail access
SCOPES = ['https://www.googleapis.com/auth/gmail.readonly']
# Authenticate and create Gmail API service
def authenticate_gmail():
creds = None
if os.path.exists('token.json'):
creds = Credentials.from_authorized_user_file('token.json', SCOPES)
# If there are no valid credentials, authenticate the user
if not creds or not creds.valid:
flow = InstalledAppFlow.from_client_secrets_file('credentials.json', SCOPES)
creds = flow.run_local_server(port=0)
# Save credentials for future runs
with open('token.json', 'w') as token_file:
token_file.write(creds.to_json())
return build('gmail', 'v1', credentials=creds)
# Fetch the last 1000 emails and save them to a JSON file
def fetch_and_dump_emails():
service = authenticate_gmail()
results = service.users().messages().list(userId='me', maxResults=1000).execute()
messages = results.get('messages', [])
emails = []
for msg in messages:
msg_detail = service.users().messages().get(userId='me', id=msg['id']).execute()
email_data = {
'id': msg_detail['id'],
'snippet': msg_detail.get('snippet'),
'internalDate': msg_detail.get('internalDate'),
'payload': msg_detail.get('payload', {})
}
emails.append(email_data)
# Save the emails to a JSON file
with open('last_1000_emails.json', 'w') as f:
json.dump(emails, f, indent=2)
print("Successfully dumped the last 1000 emails to 'last_1000_emails.json'.")
# Run the script
if __name__ == "__main__":
fetch_and_dump_emails()
| ID | Snippet | Internal Date | Payload |
|---|---|---|---|
| 1782345678abcdef | This is an example snippet from an email. | 1689465678901 | { ... } |
| 1456fghij789klm | Another email snippet with important data. | 1697462877654 | { ... } |
This script provides a simple way to automate the retrieval of email data from Gmail using the Gmail API. The data is saved locally as a JSON file, which can be used for further analysis or archival purposes.
The script is the ingestion step of a personal email search pipeline. It signs in to a Google account with OAuth 2.0 using the read-only Gmail scope, lists recent messages, downloads each message's metadata and payload, and writes the results to a JSON file. That file can then be parsed, chunked, embedded and indexed in a vector store so that past email can be searched by meaning rather than by exact keyword. Read-only scope means the script cannot send, delete or modify mail, which keeps the blast radius small if the token leaks.
Google APIs use OAuth for user data. A desktop OAuth client is created in the Google Cloud console and downloaded as credentials.json. On first run, InstalledAppFlow.run_local_server opens a browser, the user approves access, and Google redirects to a temporary local web server with an authorisation code that the library exchanges for tokens. The tokens are stored in token.json. Later runs reuse them; the access token is short-lived, and the refresh token lets the library obtain a new one without another browser prompt.
authenticate_gmail() loads saved credentials and runs the browser flow only when no valid token exists, then builds a service object with googleapiclient.discovery.build. fetch_and_dump_emails() calls users().messages().list with userId='me', which returns lightweight stubs containing only IDs. For each ID it calls messages().get, which by default returns the full format: headers and a MIME tree in payload, with bodies base64url-encoded. It keeps four fields and writes the list with json.dump. Note that the original script refreshes expired tokens only by re-running the full browser flow; the version below refreshes silently when a refresh token exists.
The payload field is the most complex part of the output. It is a tree of MIME parts: a simple email has one text/plain part, while a typical modern message is multipart/alternative with plain-text and HTML versions, possibly nested inside multipart/mixed with attachments. Headers such as From, To, Subject and Date live in payload.headers as a list of name/value pairs. The internalDate field is the time Gmail received the message, in milliseconds since the epoch, which is more reliable for sorting than the Date header supplied by the sender.
pip install google-api-python-client google-auth-httplib2 google-auth-oauthlib
# place credentials.json (Desktop OAuth client) next to the script
python gmail_fetch.py
chmod 600 token.json # it grants mailbox read access
import base64
from google.auth.transport.requests import Request
def get_creds():
creds = Credentials.from_authorized_user_file("token.json", SCOPES) if os.path.exists("token.json") else None
if creds and creds.expired and creds.refresh_token:
creds.refresh(Request()) # no browser needed
elif not creds or not creds.valid:
creds = InstalledAppFlow.from_client_secrets_file("credentials.json", SCOPES).run_local_server(port=0)
with open("token.json", "w") as f:
f.write(creds.to_json())
return creds
def list_ids(service, limit=1000, query=None):
ids, token = [], None
while len(ids) < limit:
resp = service.users().messages().list(
userId="me", q=query, pageToken=token,
maxResults=min(500, limit - len(ids))).execute() # API caps a page at 500
ids += [m["id"] for m in resp.get("messages", [])]
token = resp.get("nextPageToken")
if not token:
break
return ids
def body_text(payload):
"""Return the first text/plain part, decoded."""
if payload.get("mimeType") == "text/plain" and payload.get("body", {}).get("data"):
return base64.urlsafe_b64decode(payload["body"]["data"]).decode("utf-8", "replace")
for part in payload.get("parts", []) or []:
text = body_text(part)
if text:
return text
return ""
The paginated helper matters because a single list call returns at most 500 messages, so asking for 1000 in one request silently returns fewer. The q parameter accepts Gmail search syntax such as newer_than:30d -category:promotions to limit what is downloaded.
| Choice | Benefit | Cost |
|---|---|---|
| format=full (default) | Complete MIME tree | Large payloads, slower |
| format=metadata | Headers only, fast | No body text for embedding |
| One get per message | Simple code | Many HTTP calls; use batch requests for scale |
| JSON file output | Easy to inspect | No incremental sync; consider historyId for updates |
token.json and credentials.json are secrets; keep them out of git and web-served directories.