This tutorial will explain how to fix Broadcom 4328 v3 wireless problem in ubuntu karmic
Try unloading the module (sudo modprobe -r wl) then reload the module (sudo modprobe wl). You should now connect, however it may freeze... the broadcom module is a PITA and I had consistent, yet random system lock ups.
The instructions below will compile the most current version of the bcmwl driver and replace the currently supplied version.
This method has the advantage that in the event there is a update, your newer file will simply be replaced during the upgrade. Keep in mind however that in the event there is a kernel update you will have to repeat the process as the update will overwrite your custom wl.ko.
Also keep in mind that since this a closed source binary "blob" YMMV.
Procedure to follow
First you need to download latest broadcom drivers from here
Make a directory and extract the files in the archive
tar xvf ‘hybrid-portsrc-x86_32-v188.8.131.52.3.tar (1).gz' -C bcmwl
Compile the source
You will need to install the build-essential and kernel headers packages
sudo apt-get install build-essential kernel-header-`uname -r`
To Compile the driver
replace the current driver file with the one that you just compiled.
sudo mv ./wl.ko /lib/modules/`uname -r`/updates/dkms/wl.ko
Credit goes here