To His Coy Mistress

I enjoyed the poem To His Coy Mistress but there was one line in particular that kind of confused me. It is line 11:
"My vegetable love should grow".
The phrase vegetable love didn't make sense to me. I just couldn't find the meaning of it as I read. My first thought was actually of the vegetative state of some who don't have brain function. But that idea was contradicted by the word grow. When a person is vegetative, they are not moving forward or changing. They cannot grow. But the speaker is hoping for his love to grow. These opposing ideas confused me.