albertlane.net / bot

LaneBot

LaneBot is the web crawler for ALDI Search™, the search index run by Albert Lane Digital Infrastructure. It fetches public pages and stores a short text record of each one so they can be searched.

How it identifies itself

LaneBot/1.0 (+https://albertlane.net/bot)

Requests come from Albert Lane’s own machine. The robots.txt token is LaneBot.

How it behaves

robots.txt	Read before crawling any site and obeyed, including `Crawl-delay` (up to 10 seconds). If robots.txt cannot be read because of a server error, LaneBot does not crawl that site.
Speed	At most one request per second to any one site, or slower if robots.txt asks.
Private content	Skips login, account, admin, checkout and similar pages, sites that need a login to read, links that carry session or token values, and anything answering 401 or 403.
noindex / nofollow	Pages marked `noindex` (meta tag or X-Robots-Tag) are not stored; `nofollow` links are not followed.
Personal data	E-mail addresses and phone numbers are removed from stored text. LaneBot does not collect contact lists, track people or contact anyone.
What it fetches	HTML, JSON and plain-text pages only (no images, video, PDFs or downloads), up to 3 MB each.
Scope	Starts from the albertlane.net sitemap and follows outbound links two steps out, with a few thousand pages in total and a small limit per site.
What is stored	Address, title, description, a short text excerpt, crawl time (Pacific Time), HTTP status and the SHA-256 of the page.

How to block or limit LaneBot

Add this to your robots.txt:

User-agent: LaneBot
Disallow: /

To slow it down instead, add Crawl-delay: 10 under User-agent: LaneBot. To have pages removed from the index, e-mail lane.albert@pm.me with the addresses.

Crawl runs

1,551 pages from 222 websites in the index.

Run	Finished (PT)	Indexed	Failed	Blocked by robots.txt	Skipped as private	Sites
`lanebot-20261009-080803`	2026-10-09 08:08	10	0	1	0	1
`lanebot-20261009-074830`	2026-10-09 08:02	1530	28	85	256	220