This is actually by design.
The review queue indicator is a lot smarter than the old one, if you remember it. It used to say how many items were in the queue and people were constantly annoyed that the number of things and the number of reviews didn't match up. So, we got rid of the numbers.
The new indicator does the best we can have it do, which is - it's 100% correct that there's something to review, it's just that you already reviewed those items. The indicator only knows what reviews are available while the reviews page tells you how many items are available for you to review.
We are aware of the discrepancy and apologise for any confusion it causes. Unfortunately, fixing it so that it reflects what you specifically can review would be quite expensive for us to do on the page.
The status of this has probably changed since yesterday but this is what the queue looks like for me right now:

I've not reviewed anything, so I see the full count of posts waiting on review for the various reasons. You may (probably) see different numbers.
For more information on how this works, please see the original announcement about the review queue indicators. The last paragraph there addresses this:
Finally, all active tasks are counted for each queue when determining if a queue is in “danger”. Thus, there may be times when the indicator will light up but you’ll be unable to do anything about it—for example, if you’ve already done all the reviews you’re allowed to do for the current day in that queue, or if all the items to be reviewed are your own work.