On September 16th, we discovered and resolved a security incident involving unauthorized access to a caching server. 

We found no evidence that any customer data, API keys, or accounts were exported or misused. But some information was viewable by third parties, including a small percentage of API keys. All customers whose API keys may have been visible received an email on September 17th with recommended actions.

This post contains a summary of the incident and steps we encourage customers to take. If you have any questions or concerns, please email us.

This incident only affected Geocodio's Self-Serve infrastructure. Geocodio's Enterprise platform runs on entirely separate infrastructure with its own network security controls and was never reachable during this incident.

What happened

On September 16th, a short spike in errors at 18:29 UTC recovered on its own within minutes. Later that evening, while investigating, we found that a firewall on two servers had been left disabled after a cancelled deployment on September 7th.

The disabled firewall exposed a route to an internal caching server (Redis), and our monitoring systems failed to catch it initially.

What we know

A caching server is effectively a temporary filing cabinet. Every entry is a drawer with two parts: a label on the front, and the contents inside. 

During this incident, the unauthorized party viewed the labels of 3.2% of the entries. Labels can include API keys. 

API keys are only cached for a few minutes while requests are actively being made, meaning any exposure was limited to short windows rather than the full period.

We verified that the commands that export data in bulk were never used, and no outside system connected to replicate it.

What was not exposed:

• Geocoding queries. The addresses and coordinates you send are stored only as a hash, never in readable form. Results we return are cached for up to five minutes and are not linked to any account or user.

• Billing information. All billing information, including credit card numbers, was safe.

• Spreadsheet files and their contents

• Passwords

The IP addresses that connected to the caching server during this incident have never been used to access the Geocodio API, before, during, or since. 

What we've done

1. Incident discovery and resolution. Geocodio engineers began investigating after the increase in error rates on September 16th at 20:58 UTC. The firewall was restored and the incident was resolved at 23:24 UTC, about 2 hours and 26 minutes after we began investigating.

2. Root cause analysis and remediation. On September 17th, we conducted a root cause analysis and remediated the immediate cause. 

3. Reset dashboard sessions. As a precautionary step, we reset all dashboard sessions (i.e., logged out all users), even though there was no evidence of improper access.

4. Customer impact analysis. As part of our investigation, we looked for anomalies in each potentially-impacted account's API traffic compared to typical patterns. We did not find evidence that there was unauthorized use of any API keys.

What we’re doing

1. Further network security and monitoring changes. We’ve already added additional hardening to our network security and monitoring, and further improvements are underway as a result of this incident. 

2. Moved up infrastructure audit. We are conducting our infrastructure audit ahead of its normal schedule.

3. Public post-mortem. In the coming weeks, we will publish a public post-mortem with further details on causes and remediation steps.

What you can do

All customers whose API keys were potentially visible received an email on September 17th with recommended actions. For additional precautions, you can:

1. Rotate your Geocodio API keys. You can create new API keys and delete your current keys on the dashboard.

2. Check your usage log. During our investigation, we did not find evidence that any API keys were used in an abnormal way. However, we encourage you to regularly look at your usage log.

3. Add two-factor authentication to your account. During our investigation, we did not find evidence that there was any improper access to accounts. However, as a general best practice, we encourage you to add two-factor authentication to your account. This can be done on the dashboard.

If you have any questions, please email us.