This lesson's interactive features are locked, please to keep using them

Concurrency

Why write a web crawler with Go and not use concurrency? Let's speed this bad boy up with some goroutines!

To coordinate the goroutines, we'll use:

A Mutex to protect the shared pages map.
A buffered channel of empty structs to limit concurrent requests. Each request sends a token when it starts and receives one when it finishes. Sends block when the buffer is full.
A WaitGroup to keep main from exiting until all requests finish.

Assignment

Create a struct to share the pages map, base URL, and synchronization primitives between goroutines. Here's an example of what I used:

type config struct {
    pages              map[string]PageData
    baseURL            *url.URL
    mu                 *sync.Mutex
    concurrencyControl chan struct{}
    wg                 *sync.WaitGroup
}

Update crawlPage to use the shared struct and store page data instead of counts:
```
func (cfg *config) crawlPage(rawCurrentURL string)
```
- Fetch each normalized URL at most once, even when multiple goroutines discover it.
- Store the result of extractPageData for each successfully fetched page, using the current page's URL, and crawl its OutgoingLinks.
- Keep access to the shared pages map safe with the mutex.
Update main to initialize the shared struct and print discovered URLs instead of counts.
Test it (I tested it manually again). I recommend setting your maxConcurrency (the buffer size of the concurrencyControl channel) to 1 to start to just make sure everything still works. Then bump it up to 2. Make sure it still works, and that it's about twice as fast. Then bump it up to 5 or 10. Make sure you still get the same data, but it should be much faster.

When you're satisfied with the results, you can move on.

Tips

I created this method to call as a helper inside of crawlPage:

func (cfg *config) addPageVisit(normalizedURL string) (shouldVisit bool)

Make sure you're not crawling the same page multiple times. That's why my addPageVisit method returns a boolean: to indicate whether we should crawl the page.
I defer a wg.Done() and a receive from the (<-cfg.concurrencyControl) channel. This ensures that the wg is decremented and the channel is emptied even when the goroutine errors and returns early.
Make sure you wg.Add before you spawn a goroutine, but that you send into the channel after but still at the start of a new goroutine.
Because all the recursive calls share a wait group, you should only need to wg.Wait() once in the main function.